Search code examples
pythonmemcacheddistributedpython-memcached

Any Python memcached client that batches and parallelizes multiple get to multiple servers?

Very often I need to get the values for a long list of N keys such as 

[1267, 56578, ... , 9800]

Those values are stored on M different Memcached servers.

I don't want to send out a get() request, wait for a response, and then repeat N times.

Is there any Python memcached clients that will let me call a single multiple_get for the entire list of N keys, split those keys into their respective servers, then send one batched request to each of the M servers, gather the results from all the servers, and doing all this in parallel?

If so, how do I use such a feature?

Solution

  • Use either python-memcached: Python or pylibmc - Python client for memcached, both which has a get_mult() method for that purpose. That will cause the get operations to run asyncronously in parallell to each server.

    See also Batch your requests with get_multi.