Streaming/Tailing data over SSH using C# with SSH.NET leaks memory

Question

I'm trying to tail a file over SSH using C#. This file with be read from the beginning and then continue to be monitored for hours at a time maintaining the SSH connection. I'm using SSH.NET library to provide the functionality to SSH. The file size can be anywhere up to ~2GB. The current implementation is working but the memory usage is quite bad.

Testing: To test this functionality I am using Visual Studio 2012, targeting .NET framework 4.5, to create a small console application with the code below. I am tailing a static file that is ~127MB.

Issue: Functionally this works fine but the memory usage is quite bad. The application will use ~7MB before shellStream.WriteLine is called and then rapidly increase and plateau using ~144MB (settles when all current file content has been read from the stream).

Below is the code that I am trying to use.

private SshClient sshClient;
private ShellStream shellStream;
//Command being executed to tail a file.
private readonly string command = "tail -f -n+1 {0}";
//EventHandler that is called when new data is received.
public EventHandler<string> DataReceived;

public void TailFile(string server, int port, string userName, string password, string file)
{
   sshClient = new SshClient(server, port, userName, password);
   sshClient.Connect();

   shellStream = sshClient.CreateShellStream("Tail", 0, 0, 0, 0, 1024);

   shellStream.DataReceived += (sender, dataEvent) =>
   {
      if (DataReceived != null)
      {
         DataReceived(this, Encoding.Default.GetString(dataEvent.Data));
      }
   };

   shellStream.WriteLine(string.Format(command, file));
}

Is there something that is missing to prevent memory increasing as much as it is, or any other solutions that could accomplish the same goal?

Answer 1

You do not consume the data from the stream, so it accumulates.

See how the ShellStream.DataReceived event is implemented:

private void Channel_DataReceived(object sender, ChannelDataEventArgs e)
{
    lock (this._incoming)
    {
        // this is where the memory "leaks" as the _incoming is never consumed
        foreach (var b in e.Data)
            this._incoming.Enqueue(b);
    }

    if (_dataReceived != null)
        _dataReceived.Set();

    this.OnDataReceived(e.Data);
}

Instead of using the ShellDataEventArgs.Data, use ShellStream.Read:

 shellStream.DataReceived += (sender, dataEvent) =>
 {
    if (DataReceived != null)
    {
       DataReceived(this, shellStream.Read());
    }
 };