Does Clojure's `memoize` function useful in caching slow sql queries and complex computations?

Question

My project has a mysql database and a backend which is written by Clojure.

Some table of the database is updated only one time every day, to query the newest information we will use.

But:

1. the database is very large and network is slow, so every query will take seconds.

2. we need to do complex computations after each query.

3. we need to do many kinds of sql queries so it is not realistic to saving the results of every sql query.

4. we need to change the computation functions very frequently for the purpose of debugging.

All things are too slow for us now in the case of slow sql queries.

Fortunately, our data will only be updated one time every day, and that some of our queries from the database are very frequently used.

So we want to cache the frequently used sql queries and intermediate results.

Does Clojure's memoize function useful for this kind of jobs? My afraid is that sql queries are not pure so memoize should not cache their results. But during one day the results of our sql queries must be identical.

So can I memoize the results during one day and automatically update the results the next day?

Thank you for your help!

Answer 1

You should rather use a cache for that, e. g. clojure.core.cache.

To clarify regarding comment: memoization helps with pure functions: if your function always returns the same output given the same input, you can just store it after calculating it once.

If the output changes over time, you need to do invalidation. Memoization with invalidation (and a few other concerns such as the size of remembered things) is called "caching".

If you use a memoization mechanism for caching, what you are actually doing is implement a cache on top of it. It is much less work to just use a caching library.

In this case, you'd need to implement something like "invalidation at midnight". See https://github.com/clojure/core.cache/wiki for pointers how to do that.

EDIT: It might look somewhat like this (untested, bring your own today):

(defcache MidnightCache [cache dates]
  CacheProtocol
  (lookup [this key]
    (lookup this key nil))
  (lookup [this key not-found]
    (if (has? this key)
      (get cache key)
      not-found))
  (has? [this key]
    (let [d (get dates key)]
      (and d
           (= d (today)))))
  (hit [this key]
    this)
  (miss [this key new-value]
    (MidnightCache. (assoc (dissoc cache key)
                           key new-value)
                    (assoc (dissoc dates key)
                           key (today))))
  (evict [this key]
    (MidnightCache. (dissoc cache key)
                    (dissoc dates key)))
  (seed [this base]
    (MidnightCache. base
                    (zipmap (keys base)
                            (iterate identity (today))))))