Get image from URL and upload to Amazon S3

Question

I'd like to load an image directly from a URL but without saving it on the server, I want to upload it directly from memory to Amazon S3 server.

This is my code:

Dim wc As New WebClient
Dim fileStream As IO.Stream = wc.OpenRead("http://www.domain.com/image.jpg")

Dim request As New PutObjectRequest()
request.BucketName = "mybucket"
request.Key = "file.jpg"
request.InputStream = fileStream 
client.PutObject(request)

The Amazon API gives me the error "Could not determine content length". The stream fileStream ends up as "System.Net.ConnectStream" which I'm not sure if it's correct.

The exact same code works with files from the HttpPostedFile but I need to use it in this way now.

Any ideas how I can convert the stream to become what Amazon API is expecting (with the length intact)?

Answer 1

I had the same problem when I'm using the GetObjectResponse() method and its propertie ResponseStream to copy a file from a folder to another in same bucket. I noted that the AWS SDK (2.3.45) have some faults like a another method called WriteResponseStreamToFile in GetObjectResponse() that simply doesn't work. These lacks of functions needs some workarounds.

I solved the problem openning the file in array of bytes and putting it in a MemoryStream object.

Try this (C# code)

WebClient wc = new WebClient();
Stream fileStream =  wc.OpenRead("http://www.domain.com/image.jpg");

byte[] fileBytes = fileStream.ToArrayBytes();

PutObjectRequest request  = new PutObjectRequest();
request.BucketName = "mybucket";
request.Key = "file.jpg";
request.InputStream = new MemoryStream(fileBytes);
client.PutObject(request);

The extesion method

public static byte[] ToArrayBytes(this Stream input)
        {
            byte[] buffer = new byte[16 * 1024];
            using (MemoryStream ms = new MemoryStream())
            {
                int read;
                while ((read = input.Read(buffer, 0, buffer.Length)) > 0)
                {
                    ms.Write(buffer, 0, read);
                }
                return ms.ToArray();
            }
        }

You can also create a MemoryStream without an array of bytes. But after the first PutObject in S3, the MemoryStream will be discarted. If you need to put others objects, I recommend the first option

 WebClient wc = new WebClient();
 Stream fileStream =  wc.OpenRead("http://www.domain.com/image.jpg");

 MemoryStream fileMemoryStream = fileStream.ToMemoryStream();  

 PutObjectRequest request  = new PutObjectRequest();
 request.BucketName = "mybucket";
 request.Key = "file.jpg";
 request.InputStream = fileMemoryStream ;
 client.PutObject(request);

The extesion method

    public static MemoryStream ToMemoryStream(this Stream input)
    {
        byte[] buffer = new byte[16 * 1024];

        int read;
        MemoryStream ms = new MemoryStream();
        while ((read = input.Read(buffer, 0, buffer.Length)) > 0)
        {
            ms.Write(buffer, 0, read);
        }
        return ms;
    }