URL Serialization Performance Issue in Sorl-Thumbnail

Question

I'm having a bit of a difficult time with performance using sorl-thumbnail. I'm hosting images on S3 with the S3Boto3Storage storage backend, I'm using Redis as my Key-Value Store and all queries have been cached inside it already. I'm using the following dependencies:

boto3==1.17.103
botocore==1.20.105
Django==3.2.4
django-redis==4.10.0
django-storages==1.11.1
redis==3.3.11
sorl-thumbnail==12.7.0

Now I run & time the following script:

from sorl.thumbnail import get_thumbnail

images = Image.objects.all()  # 7 images
for image in images:
    x = image.file
    y = x.url                                           # 1
    x = get_thumbnail(x, '800x600', crop='center')      # 2
    x = x.url                                           # 3

1. Step 1: When I comment out #2 & #3, the script takes 0.193s
2. Step 2: When I comment out #1 & #3, the script takes 0.0153s
3. Step 3: When I comment out #1, the script takes 0.669s (!)

The odd thing is that when I debug the 3rd step, x seems to already have the attribute "url", so I don't know why it is taking so long to access it.

The connection to Redis is working and the query towards it is already activated in the 2nd step. Like I said earlier, the thumbnails have already been generated, the lookups are cached in Redis, so it is (I believe) unrelated to S3 thumbnail generation issues.

In production I have frequent queries for around 50 images at once... that amounts to more than 4s of url serialisation... Would appreciate any help! :)

Answer 1

Okay, figured out what was wrong. I had set AWS_S3_CUSTOM_DOMAIN in settings, but had accidentally overwritten it in my storage class:

from storages.backends.s3boto3 import S3Boto3Storage


class DefaultFileStorage(S3Boto3Storage):
     location = settings.AWS_DEFAULT_FILES
     file_overwrite = False
     custom_domain = False  # <-- Bottleneck! 

After removing custom_domain = False, performance went up by 10x (literally).