Reading text from Amazon s3 stream

Question

I use the following code to read text file from Amazon S3, and processing it line by line. This code works but the problem is it is slow.

 GetObjectRequest getObjRequest = new GetObjectRequest()
    .WithBucketName(amazonSettings.BucketName)
    .WithKey(_fileKey);
using (AmazonS3 client = AWSClientFactory.CreateAmazonS3Client(
    amazonSettings.AccessKey, 
    amazonSettings.SecretAccessKey))
using (GetObjectResponse getObjRespone = client.GetObject(getObjRequest))
using (Stream amazonStream = getObjRespone.ResponseStream)
{                        
    StreamReader amazonStreamReader = new StreamReader(amazonStream);
    tempGsContact = new GSContact();
    while ((_fileLine = amazonStreamReader.ReadLine()) != null)
    {
        if (_fileLine.Equals("END:VCARD"))
        {
            // Make process 1
        }
        else if (!_fileLine.Equals(string.Empty))
        {
            //Make process 2
        }
    }                        
}

The question is: can I get more sufficient way to reduce the time cost?

Answer 1

There is a similar performance bottleneck on HTTPWebResponse in .NET, which is probably what the AmazonS3 class they made is wrapping.

It's caused by the object taking a long time to resolve proxy settings, there are a few potential solutions listed here, but the easiest option might be to add the following to your app.config file:

<system.net>
  <defaultProxy enabled="false">
    <proxy/>
    <bypasslist/>
    <module/>
  </defaultProxy>
</system.net>

Alternatively you could replace the call here:

AWSClientFactory.CreateAmazonS3Client(amazonSettings.AccessKey, amazonSettings.SecretAccessKey)

with a call to an overload that accepts a third parameter of 'AmazonS3Config', where you can specify a null proxy via 'AmazonS3Config.ProxyHost = null' - which should effectively be the same as the above configuration change for only that request.