"The connection with the server was terminated abnormally" when uploading to to S3 TransferUtility.UploadAsync on EC2 instance

Question

The error I'm getting is:

The thread 0xffc has exited with code 0 (0x0).
Exception thrown: 'System.IO.IOException' in System.Private.CoreLib.dll
Exception thrown: 'System.IO.IOException' in Anjuna1.dll
Exception thrown: 'System.IO.IOException' in System.Private.CoreLib.dll
Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware:Error: An unhandled exception has occurred: The write operation failed, see inner exception.

System.IO.IOException: The write operation failed, see inner exception. ---> 
System.Net.Http.WinHttpException: The connection with the server was 
terminated abnormally
   --- End of inner exception stack trace ---

Complete stack trace here: https://pastebin.com/BGny5ULG

The code (.net core 2) is as follows:

    private async Task S3UploadFileAsync(string fileName, Stream stream)
    {
        using (IAmazonS3 client = new AmazonS3Client(Amazon.RegionEndpoint.EUWest2))
        {
            var uploadRequest = new TransferUtilityUploadRequest
            {
                InputStream = stream,
                BucketName = "******-*****",
                CannedACL = S3CannedACL.AuthenticatedRead,
                Key = fileName
            };
            TransferUtility fileTransferUtility = new TransferUtility(client);
            try
            {

                await fileTransferUtility.UploadAsync(uploadRequest);
            }
            catch (System.Exception e)
            {
                throw e;
            }

        }
    }

The above code works fine when running on my local machine (Windows 10), just not on the EC2 instance (Windows Server 2016).

I've tried making the S3 bucket public, thinking it was a permission issue (still not working).

I thought it might be an issue with the windows firewall or the EC" instance security groups so I installed the AWS CLI and can upload to the S3 instance using that with no problems.

It doesn't matter what I upload, I've tried large, small and even empty files.

It takes around 30 seconds before the exception is caught, so it feels like something is timing out. Changing the code to a non-existent BucketName results in the same symptoms making me think it's got nothing to do with the bucket configuration (I've starred out the name).

Any help with suggestions for things to try would be appreciated - I've been working on this all day and I'm running out of ideas... Thanks!

Update: similar errors here suggest it's permissions related: https://matt40k.uk/2017/09/s3-errors/

Answer 1

The issue appears to be related to permissions on the S3 bucket, if I comment out the following line the upload works:

CannedACL = S3CannedACL.AuthenticatedRead,

It's still not perfect, as without that line my app can't then download the file it's just uploaded - but at least I now know where to focus my efforts!

Ultimately this is looking like an issue with the AWS .NET SDK for not providing better error messages. I've opened an issue here: https://github.com/aws/aws-sdk-net/issues/890

Update: The underlying permission issue was eventually fixed by adding the AmazonS3FullAccess Policy to the aws-elasticbeanstalk-ec2-role as suggested in a comment here: Grant S3 access to Elastic Beanstalk instances