I'm trying to upload a bunch of images I create using RMagick on the fly to S3.
Locally, I can create around 50 images and upload all of them in 30 seconds.
The code is pretty simple:
#outside the loop
s3 = AWS::S3.new
bucket = s3.buckets["bucket_name"]
#inside the loop
s3_object = bucket.objects[image_name]
s3_object.write(image.to_blob) # image from rmagick
It runs inside of Sidekiq job on a single worker dyno.
Unfortunately, uploading the same images on Heroku takes ~300-400 seconds. 10 times longer!
Looking at the logs I see a lot of retries for images that are fairly small.
2013-10-11T04:35:22.145747+00:00 app[worker.1]: [AWS S3 200 0.197307 0 retries] put_object(:bucket_name=>"bucket_name",:content_length=>109459,:data=>#<StringIO:0x007f2365b64240>,:key=>"key")
2013-10-11T04:35:25.874405+00:00 app[worker.1]: [AWS S3 200 0.361312 0 retries] put_object(:bucket_name=>"bucket_name",:content_length=>95877,:data=>#<StringIO:0x007f2365b804e0>,:key=>"key")
2013-10-11T04:35:29.132112+00:00 app[worker.1]: [AWS S3 200 0.234044 0 retries] put_object(:bucket_name=>"bucket_name",:content_length=>139513,:data=>#<StringIO:0x007f2365bb2468>,:key=>"key")
2013-10-11T04:35:32.344850+00:00 app[worker.1]: [AWS S3 200 0.2355 0 retries] put_object(:bucket_name=>"bucket_name",:content_length=>170072,:data=>#<StringIO:0x007f2365bf7090>,:key=>"key")
2013-10-11T04:35:36.945883+00:00 app[worker.1]: [AWS S3 200 1.333778 0 retries] put_object(:bucket_name=>"bucket_name",:content_length=>99901,:data=>#<StringIO:0x007f2365facdc0>,:key=>"key")
2013-10-11T04:35:40.425480+00:00 app[worker.1]: [AWS S3 200 27.044513 1 retries] put_object(:bucket_name=>"bucket_name",:content_length=>41198,:data=>#<StringIO:0x007f236586b430>,:key=>"key")
2013-10-11T04:35:43.188034+00:00 app[worker.1]: [AWS S3 200 0.619067 0 retries] put_object(:bucket_name=>"bucket_name",:content_length=>105656,:data=>#<StringIO:0x007f2377d37e98>,:key=>"key")
2013-10-11T04:36:01.464658+00:00 app[worker.1]: [AWS S3 200 23.727376 1 retries] put_object(:bucket_name=>"bucket_name",:content_length=>41198,:data=>#<StringIO:0x007f23668d57a8>,:key=>"key")
2013-10-11T04:36:05.773317+00:00 app[worker.1]: [AWS S3 200 20.765285 1 retries] put_object(:bucket_name=>"bucket_name",:content_length=>77195,:data=>#<StringIO:0x007f2378bc2350>,:key=>"key")
2013-10-11T04:36:24.315209+00:00 app[worker.1]: [AWS S3 200 20.947173 1 retries] put_object(:bucket_name=>"bucket_name",:content_length=>105656,:data=>#<StringIO:0x007f23757d1f00>,:key=>"key")
2013-10-11T04:36:28.069586+00:00 app[worker.1]: [AWS S3 200 20.963508 1 retries] put_object(:bucket_name=>"bucket_name",:content_length=>126600,:data=>#<StringIO:0x007f2366aa5fd8>,:key=>"key")
Any ideas?
Workaround: upgrading to the 2x worker dyno reduced the timeouts.