Sorl-thumbnail + DigitalOcean + Amazon S3 slowness

Question

Due to AWS EC2 being kind of expensive, I have tried to migrate to DigitalOcean, but I found out that rendering a page with thumbnails are kind of slow from DigitalOcean with Amazon S3 being the file storage. (Running from EC2 is pretty fast I assume the traffic within Amazon)

Symptoms:

1. Initial load was very very slow, due to thumbnail creation
2. Subsequent load was very slow also, but not as slow as initial load

I have the following settings:

STATICFILES_STORAGE = DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'

THUMBNAIL_KVSTORE = values.Value('sorl.thumbnail.kvstores.cached_db_kvstore.KVStore')

and I can see sorl-thumbnail generates cache values in the KVStore in the database. But it seems to me that it still checks for file existence on S3 before rendering the thumbnail. This is contradict to the documentation:

It is worth noting that sorl-thumbnail does not check if source or thumbnail exists if the thumbnail key is found in the Key Value Store.

I have searched on SO and google and saw some related posts, but they were 4 years old and don't seem to have conclusive answers.

Answer 1

After much debugging, I finally nailed it, the problem is buried deep inside s3boto.py inside django-storages

Problem:

def url(self, name):
    name = self._normalize_name(self._clean_name(name))
    if self.custom_domain:
        return "%s//%s/%s" % (self.url_protocol,
                              self.custom_domain, name)
    return self.connection.generate_url(self.querystring_expire,
        method='GET', bucket=self.bucket.name, key=self._encode_name(name),
        query_auth=self.querystring_auth, force_http=not self.secure_urls) 

I didn't have custom_domain setup so for every thumbnail it was trying to access S3 API to construct the URL.

After defining AWS_S3_CUSTOM_DOMAIN and AWS_S3_URL_PROTOCOL it's working lightening fast.