StreamReader ReadLine throwing disposed exception rather than returning null

Question

I'm trying to fetch a text file from an FTP server, then read it line by line, adding each line to a list.

My code seems to fit the standard pattern for this:

            var response = (FtpWebResponse)request.GetResponse();
            using (var responseStream = response.GetResponseStream())
            {
                using (var reader = new StreamReader(responseStream))
                {
                    string line;
                    while((line = reader.ReadLine()) != null)
                    {
                        lines.Add(line);
                    }
                }
            }

But, for some reason, when reader.ReadLine() is called on the very last line of the file, it throws an exception, saying "Cannot access a disposed object".

This is really weirding me out. If I remember correctly, the final line of a stream when there is no further data is null, right?

In addition (while I'm not certain about this), this only seems to be happening locally; the live version of this service seems to be pootling along fine (albeit with some issues I'm trying to get to the bottom of). I certainly don't see this issue in my logs.

Anyone have an ideas?

EDIT: Here's the full text of the exception.

System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Net.Sockets.NetworkStream'.
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.FtpDataStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.IO.StreamReader.ReadBuffer()
   at System.IO.StreamReader.ReadLine()
   at CENSORED.CatalogueJobs.Common.FtpFileManager.ReadFile(String fileName, String directory) in C:\\Projects\\CENSORED_CatalogueJobs\\CENSORED.CatalogueJobs.Common\\FtpFileManager.cs:line 104
   at CENSORED.CatalogueJobs.CENSOREDDispatchService.CENSOREDDispatchesProcess.<Process>d__12.MoveNext() in C:\\Projects\\CENSORED_CatalogueJobs\\CENSORED.CatalogueJobs.CENSOREDDispatches\\CENSOREDDispatchesProcess.cs:line 95"

Type is "System.ObjectDisposedException". Sorry for censorship, exception contains my client's name.

EDIT 2: Here's the code now, after being expanded out and removing a layer of usings (I think I've done it right).

            var response = (FtpWebResponse)request.GetResponse();
            using (var reader = new StreamReader(response.GetResponseStream()))
            {
                string line = reader.ReadLine();
                while(line != null)
                {
                    lines.Add(line);
                    line = reader.ReadLine();
                }
            }

EDIT 3: A slightly more wide view of the code (temporarily reverted for my sanity). This is essentially everything in the function.

            var request = (FtpWebRequest)WebRequest.Create(_settings.Host + directory + fileName);

            request.Method = WebRequestMethods.Ftp.DownloadFile;
            request.Credentials = new NetworkCredential(_settings.UserName, _settings.Password);
            request.UsePassive = false;
            request.UseBinary = true;

            var response = (FtpWebResponse)request.GetResponse();
            using (var responseStream = response.GetResponseStream())
            {
                using (var reader = new StreamReader(responseStream))
                {
                    string line;
                    while((line = reader.ReadLine()) != null)
                    {
                        lines.Add(line);
                    }
                }
            }

Answer 1

Checking the source of the internal FtpDataStream class shows that its Read method will close the stream all by itself if there are no more bytes:

    public override int Read(byte[] buffer, int offset, int size) {
        CheckError();
        int readBytes;
        try {
            readBytes = m_NetworkStream.Read(buffer, offset, size);
        } catch {
            CheckError();
            throw;
        }
        if (readBytes == 0)
        {
            m_IsFullyRead = true;
            Close();
        }
        return readBytes;
    }

Stream.Close() is a direct call to Dispose :

    public virtual void Close()
    {
        /* These are correct, but we'd have to fix PipeStream & NetworkStream very carefully.
        Contract.Ensures(CanRead == false);
        Contract.Ensures(CanWrite == false);
        Contract.Ensures(CanSeek == false);
        */

        Dispose(true);
        GC.SuppressFinalize(this);
    }

That's not how other streams, eg FileStream.Read work.

It looks like StreamReader.ReadLine is trying to read more data, resulting in an exception. This could be because it's trying to decode a UTF8 or UTF16 character at the end of the file.

Instead of reading line by line from the network stream, it would be better to copy it into a MemoryStream or FileStream with Stream.CopyTo before reading it

UPDATE

This behaviour, while totally unexpected, isn't unreasonable. The following is more of an educated guess based on the FTP protocol itself, the FtpWebRequest.cs and FtpDataStream.cs sources and painful experience trying to download multiple files.

FtpWebRequest is a (very) leaky abstraction on top of FTP. FTP is a connection oriented protocol with specific commands like LIST, GET and the missing MGET. That means, that once the server has finished sending data to the client in response to LIST or GET, it goes back to waiting for commands.

FtpWebRequest tries to hide this by making each request appear connectionless. This means that once the client finishes reading data from the Response stream there's no valid state to return to - an FtpWebResponse command can't be used to issue further commands. It can't be used to retrieve multiple files with MGET either, which is a major pain. There's only supposed to be one response stream after all.

With .NET Core and an increased need (to put it mildly) to use unsupported protocols like SFTP it may be a very good idea to find a better FTP client library.