Search code examples
pythonmongodbpymongomonkeypatchinggevent

pymongo + gevent: throw me a banana and just monkey_patch?


Quickie here that needs more domain expertise on pymongo than I have right now:

Are the "right" parts of the pymongo driver written in python for me to call gevent monkey_patch() and successfully alter pymongo's blocking behavior on r/w within gevent "asynchronous" greenlets?

If this will require a little more leg work on gevent and pymongo -- but it is feasible -- I would be more than willing to put in the time as long as i can get a little guidance over irc.

Thanks!

Note: At small scale mongo writes are not a big problem because we are just queuing a write "request" before unblocking. BUT talking to fiorix about his twisted async mongo driver (https://github.com/fiorix/mongo-async-python-driver), even mongo's quick write (requests) can cause problems in asyncronous applications at scale. (And of course, non-blocking reads could cause problems from the start!)


Solution

  • I have used PyMongo with Gevent and here are a few things you need to watch out for:

    1. Instantiate only one pymongo.Connection object, preferrably as a global or module-level variable. This is important because Connection has within itself a pool!
    2. Monkey patch everything, or at least BOTH socket and threading. Due to the use of thread locals in Connection, patching socket alone is not enough.
    3. Remember to call end_request to return the connection to the pool.

    The answer to your question is go ahead, PyMongo works just fine with Gevent.