I'm using Django's cache framework with the Database backend. In my settings I have USE_TZ=True, and the TIME_ZONE of my databases is set to: America/New_York. In my dev/testing environment, I'm using SQLite and Oracle in production both of which don't support time zones, which should mean that Django will read/write date-times as America/New York time.
The problem I see is that the cache backend always determines the expiry to be in UTC - which is great - but then before the insert it simply converts this to a string without localizing and that's what gets set in the database. This means if I set the expiry as 120 seconds at 5:00 AM in America/New_York time: Django will look at the UTC time, adds the 120 seconds (so the expiry will be 9:02 AM in UTC) and it will simply insert 9:02 into the database.
However when Django reads this value back, it will be read as 9:02 in America/New_York TZ, hence there are additional 4 hours to the intended 2 mins of expiry.
Stepping through the source code I think the issue is that the logic for determining "exp" (for the insert) is different than for "current_expires" (read operation for expiry of existing record). For "exp" the value will be set as a naive timestamp with the datetime.utcfromtimestamp function and when it is adapted, it will be simply converted to a string.
Below is where timeout will be converted to a naive utc timestamp (in _base_set function of DatabaseCache):
if timeout is None:
exp = datetime.max
elif settings.USE_TZ:
exp = datetime.utcfromtimestamp(timeout)
else:
exp = datetime.fromtimestamp(timeout)
..then later will be 'adapted'
exp = connection.ops.adapt_datetimefield_value(exp)
However, this simply will return the UTC value as a string as the value is a naive timestamp:
def adapt_datetimefield_value(self, value):
if value is None:
return None
# Expression values are adapted by the database.
if hasattr(value, 'resolve_expression'):
return value
# SQLite doesn't support tz-aware datetimes
if timezone.is_aware(value):
if settings.USE_TZ:
value = timezone.make_naive(value, self.connection.timezone)
else:
raise ValueError("SQLite backend does not support timezone-aware datetimes when USE_TZ is False.")
return str(value)
Whereas for "current_expires" the database's TZ is correctly considered thanks to the "convert_datatimefield_value" db converter that get's called eventually in the _base_set function:
def convert_datetimefield_value(self, value, expression, connection):
if value is not None:
if not isinstance(value, datetime.datetime):
value = parse_datetime(value)
if settings.USE_TZ and not timezone.is_aware(value):
value = timezone.make_aware(value, self.connection.timezone)
return value
I think the two different approaches cause this additional 4 hours of difference in interpreting the expiry. I would expect this to be consistent for writing and read - is there something that I'm missing?
Currently, I'm using Django 2.2 and in the middle of upgrading to the 4.2 version - but I have checked, and seems to me the main logic stayed the same for this part of the source code.
I'm posting this in case someone runs into the same issue: starting from Django 4, the above scenario should work perfectly, as expected.
This is thanks to the fix that was applied for determining "exp" which is the expiry of the 'to-be-inserted' record. From Django 4 the corresponding code fragment reads as (_base_set method of DatabaseCache):
if timeout is None:
exp = datetime.max
else:
tz = timezone.utc if settings.USE_TZ else None
exp = datetime.fromtimestamp(timeout, tz=tz)
which means "exp" will be time aware in case USE_TZ is True, so when the adapt_datetimefield_value function is called (e.g. for sqlite):
# SQLite doesn't support tz-aware datetimes
if timezone.is_aware(value):
if settings.USE_TZ:
value = timezone.make_naive(value, self.connection.timezone)
else:
raise ValueError(
"SQLite backend does not support timezone-aware datetimes when "
"USE_TZ is False."
)
return str(value)
...this will be correctly adapted to the time zone of the database (America/New_York in my case), before the insert.