I have a question regarding the ftp library from C#. I need to download 9000 txt files from a ftp server. Station.ToUpper() is the file name, so for every file I need a new ftp connection. For one file it takes around one second. The txt files contain two lines. So for all files it takes around one and a half hour. Is there a better / faster solution?
// Get the object used to communicate with the server.
FtpWebRequest request = (FtpWebRequest)WebRequest.Create(ftpAddress + station.ToUpper());
//request.UsePassive = false;
request.Method = WebRequestMethods.Ftp.DownloadFile;
// This example assumes the FTP site uses anonymous logon.
request.Credentials = new NetworkCredential("anonymous", "[email protected]");
FtpWebResponse response = (FtpWebResponse)request.GetResponse();
Stream responseStream = response.GetResponseStream();
StreamReader reader = new StreamReader(responseStream);
There's not much you're doing wrong in this code, except for the fact you're not calling Dispose() on your streams or response objects. Do that first to make sure you're not somehow running out of resources on the client or something.
Other than that, you don't have too many options here, and a lot depends upon what you can do on the server side.
First, you could try to use threading to download a bunch of files at once. You'll need to experiment with how this affects your throughput. It will probably scale linearly for a while, then fall off. If you open up too many connections, you could anger the maintainer of the server, or it could start denying you connections. Be conservative.
Optimally, the files would be zipped (.ZIP or .TGZ) on the server. This will likely not be an option if you don't have more control over the process.