Enumerating keys in Django Database Cache

Question

I'm caching some calculation results in a Django application using Django Database Cache (https://docs.djangoproject.com/en/1.8/topics/cache/#database-caching).

What is the way to enumerate all keys stored in the cache at a given moment. Occasionally I need to invalidate (delete) parts of the cache before expiration (because I'm debugging). The cache keys are generated by elaborate calculation and I don't want to repeat that computation. I know the prefix of the cache keys I would like to delete but I don't know the full key strings.

I don't see right away on the cache API how could I do that. I can get entries, create keys, delete entries and clear the whole cache: https://docs.djangoproject.com/en/1.8/topics/cache/#the-low-level-cache-api

Right now I have to extract the keys with SQL statements which is a PITA. I'd like to write a management command I can use to invalidate sections.

---

Example:

settings.py:

CACHES = {
    'default': {
         'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
         'LOCATION': 'default-cache',
    },
    'staticfiles': {
         'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
         'LOCATION': 'static-files',
    },
    'bla_stats': {
        'BACKEND': 'django.core.cache.backends.db.DatabaseCache',
        'LOCATION': 'django_bla_stats_cache',
    }
}

I created the cache as explained in the Django documentation I linked. Placing some data into the cache (no expiration: I control the entries).

from django.core.cache import caches

cache = caches['bla_stats']
cache.set("a_d3e6a1e1-0565-4d20-8887-4fda47186299", "foo", None)
cache.set("a_e79a1e0d-bfe1-4a04-8db3-42495c09e780", "bar", None)
cache.set("b_390d42ec-2b70-436d-8600-404034b07fe9", "fiz", None)
cache.set("b_a2d3cb52-8941-4812-8186-676ee3de0ec3", "baz", None)

And here is the question: how can I find all the keys in the cache which has the key prefix "b_" at any given moment?

Answer 1

Assuming you are using MySQL as your DB backend this sub class of the DatabaseCache should work to return a dictionary of all the results from a LIKE type query on cache keys.

class DatabaseCacheExtended(DatabaseCache):
    def get_where(self, query, default=None, version=None):
        db = router.db_for_read(self.cache_model_class)
        table = connections[db].ops.quote_name(self._table)

        with connections[db].cursor() as cursor:
            cursor.execute("SELECT cache_key, value, expires FROM %s "
                           "WHERE cache_key LIKE %%s" % table, [query])
            rows = cursor.fetchall()
        if len(rows) < 1:
            return {}
        return_d ={}
        for row in rows:
            value = connections[db].ops.process_clob(row[1])
            return_d[row[0]] = pickle.loads(base64.b64decode(force_bytes(value)))
        return return_d

Then you just need to change the registered backend in your settings.py

    'bla_stats': {
        'BACKEND': 'path.to.DatabaseCacheExtended',
        'LOCATION': 'django_bla_stats_cache',
     }

Example:

>>> from django.core.cache import caches
>>> cache = caches['bla_stats']
>>> cache.get_where("b_%")
... {"b_key1":"val1", "b_key2":"val2"}