Search code examples
pythonsqlalchemyeventlet

Using eventlet with SQLAlchemy

I'm using eventlet to build a simple website crawler starting from this example. I would like to use SQLAlchemy to store web pages' content and metadata for further processing. It's possible to use SQLAlchemy along with eventlet? Could anyone provide a simple example?

Solution

  • Sorry for late reply.

    It would really depend on database connection library you use.

    • For C extension library, like MySQLdb, Eventlet has db_pool module as used in OpenStack example. Basically, it creates a pool of OS threads to execute blocking DB operations. So you can't really have a lot of parallel queries, but most probably your database would not survive them anyway, so that's not an issue.
    • For pure Python library, like myconnpy, pg8000, then just call eventlet.monkey_patch().
    • As a special case, Eventlet supports monkey patching for psycopg2. It is a C extension but it is awesome enough to provide IO hooks, so you get best of both worlds. Again, simple call to monkey_patch() would do the trick.