Implementing tornado.locks.Lock over multiple processes

I've been thinking about how I can implement tornado locks over multiple processes.

I have multiple processes that have the same co-routines. I never want C2 to run while C1 is running or vice versa (even across processes)

Does anyone have any ideas on how I can implement this using the tornado.locks.Lock?

p1     p2     p3

C1     C1     C1


C2     C2     C2

EDIT

I've been reading about multiprocessing.managers.SyncManager I started a server then in my tornado init i tried to connect to it with:

  m = SyncManager(address=("", 50000), authkey=b'abc')
  m.connect()

I then acquired a lock with:

check=lock.acquire(blocking=False)

I started another tornado process and did the same thing. However the Lock() class creates a separate instance for each process... How do i share this same object over multiple processes?

Solution

You can't do this with tornado.lock.Lock; this is a single-process synchronization primitive (and it doesn't know anything about the multiprocessing SyncManager. You could use SyncManager.Lock to get a multi-process lock, but it's synchronous so you can't easily use it from an async Tornado app).

You'll need to talk to some external coordinating process to handle locking across multiple processes, and even then you'll have to handle the possibility that a process dies while holding the lock (so you need to allow for locks to expire somehow, and handle the case where the lock expires while the process holding it is not actually dead...). I recommend using something like etcd or ZooKeeper as your locking service.