Running long blocking calculations in parallel in twisted

Question

I am trying to learn twisted framework. But, I am not able to get a handle of it.

Say, I have this function.

def long_blocking_call(arg1, arg2):
     # do something
     time.sleep(5) # simulate blocking call
     return result

results = []
for k, v in args.iteritems():
      r = long_blocking_call(k,v)
      results.append(r)

But, I was wondering how can I leverage deferToThread (or something else in twisted world) to run the long_blocking_call in "parallel"

I found this example: Periodically call deferToThread But, I am not exactly sure if that is running things in parallel?

Answer 1

deferToThread uses Python's built-in threading support to run the function passed to it in a separate thread (from a thread pool).

So deferToThread has all of the same properties as the built-in threading module when it comes to parallelism. On CPython, threads can run in parallel as long as only one of them is holding the Global Interpreter Lock.

Since there is no universal cause of "blocking" there is also no universal solution to "blocking" - so there's no way to say whether deferToThread will result in parallel execution or not in general. However, a general rule of thumb is that if the blocking comes from I/O it probably will and if it comes from computation it probably won't.

Of course, if it comes from I/O, you might be better off using some other feature from Twisted instead of multithreading.