Transferring files from FTP directly to Azure Blob storage in C#

Question

I have written a function which move files from FTP server to Azure Blob storage. I want to pass the stream from FTP to blob so that I can upload the files. I am running a while loop for every file and trying to move the file to blob storage using UploadFromStreamAsync(). But when I came to this call, my stream object gets disposed because of which file is getting transfer to blob but without any content. I do not want to dispose my stream object till all files are transfer. Can anyone tell me what's wrong going on??

string ftpPath = ConfigurationSettings.AppSettings.Get("ftpPath");
string ftpUserName = ConfigurationSettings.AppSettings.Get("ftpUserName");
string ftpPassword = ConfigurationSettings.AppSettings.Get("ftpPassword");

FtpWebRequest request = (FtpWebRequest)WebRequest.Create(ftpPath);
request.Method = WebRequestMethods.Ftp.ListDirectory;
request.Credentials = new NetworkCredential(ftpUserName, ftpPassword);
FtpWebResponse response = (FtpWebResponse)request.GetResponse();
Stream responseStream = response.GetResponseStream();
StreamReader reader = new StreamReader(responseStream);

string connectionString = ConfigurationSettings.AppSettings.Get("connectionString");
string folderName = "Inbox/";
string file = reader.ReadLine();
while (!string.IsNullOrEmpty(file))
{
    string fileName = Path.GetFileNameWithoutExtension(file);
    string guid = Guid.NewGuid().ToString();
    string extension = Path.GetExtension(file);
    try
    {
        Stream fileForBlobStorage = reader.BaseStream;
        if (CloudStorageAccount.TryParse(connectionString, out storageAccount))
        {
            CloudBlobClient cloudBlobClient = storageAccount.CreateCloudBlobClient();
            CloudBlobContainer cloudBlobContainer = cloudBlobClient.GetContainerReference("falcon");
            BlobContainerPermissions permissions = new BlobContainerPermissions
            {
                PublicAccess = BlobContainerPublicAccessType.Blob
            };
            await cloudBlobContainer.SetPermissionsAsync(permissions);

            CloudBlockBlob cloudBlockBlob = cloudBlobContainer.GetBlockBlobReference(folderName + fileName + "-" + '[' + guid + ']' + guid + extension.ToString());

            await cloudBlockBlob.UploadFromStreamAsync((Stream  )fileForBlobStorage);
        }
        else
        {
            Console.WriteLine("Connection string not defined.");
        }
    }
    catch (Exception e)
    {
        string message = e.Message;
        Console.WriteLine(message);
    }
    file = reader.ReadLine();
}

Answer 1

You are requesting directory listing of a folder on FTP server. And with the listing you are, at the same time:

• Reading the listing line-by-line (file-by-file) – somehow trying to process individual lines/files.
• Yet you are trying to upload the listing (the same stream) to the blob.

That can never work. And moreover it makes no sense.

I assume that you actually want to upload the files, not the listing.

For that you need to start downloading the individual files from the FTP server in your loop:

FtpWebRequest fileRequest = (FtpWebRequest)WebRequest.Create(ftpPath + file);
fileRequest.Method = WebRequestMethods.Ftp.DownloadFile;
fileRequest.Credentials = new NetworkCredential(ftpUserName, ftpPassword);
FtpWebResponse fileResponse = (FtpWebResponse)fileRequest.GetResponse();
Stream fileStream = fileResponse.GetResponseStream();

await cloudBlockBlob.UploadFromStreamAsync(fileStream);