Below is my testbed
from twisted.internet import defer, task
@defer.inlineCallbacks
def foo():
yield
print 'after yield in foo'
@defer.inlineCallbacks
def main(reactor):
d = foo()
yield
print 'after yield in main'
yield d
task.react(main)
I exepect that the yield statement will make the function to "yield the exeuction context" (whatever that means in Twisted) and let another deferred to take over the execution. For that particular example I expect that main() starts the execution, calls foo(), which being wrapped with inlineCallbacks is turned into a deferred, then yields the execution letting foo() to eventually start. Then foo() in its turn yields the execution too, so ultimately the order of printed lines should be
after yield in main
after yield in foo
For some reason the output is
after yield in foo
after yield in main
What is the right way to implement cooperative multi-tasking in Twisted and let the execution context go to another deferred in line?
I exepect that the yield statement will make the function to "yield the exeuction context" (whatever that means in Twisted) and let another deferred to take over the execution.
From your observations, I take it you have been disabused of this expectation by now. To be perfectly clear, this is not what yield does in a generator function decorated by inlineCallbacks.
What yield does in such a function is pass a value out to a trampoline. If the value is a Deferred, the trampoline suspends execution of the generator until the Deferred fires. If the value is not a Deferred, or when the Deferred fires with a value, the generator is resumed with that value sent in to it.
So, since yield is the same as yield None and None is not a Deferred, these yield expressions are just an expensive way to say None.
This also gives you some idea of how to achieve what you say you're after. To suspend execution, yield a Deferred that does not fire until you want execution to resume.
What is the right way to implement cooperative multi-tasking in Twisted and let the execution context go to another deferred in line?
There are many possibly correct ways to do this. The specifics depend more on your specific application requirements.
Unfortunately, there are some easily explained cheesy answers which might seem correct but which will probably ultimately lead to poor performance and costly maintenance. For example, you can suspend for "a reactor iteration" (to the extent such a thing exists, which may be less so than you believe). To do this, yield twisted.internet.task.deferLater(reactor, 0.0, lambda: None). This expression makes a Deferred that fires no sooner than "zero seconds" from now but also with the constraint that it doesn't fire right now.
However, this lets the whole reactor implementation have a chance at doing work before your generator is resumed - even if there is no other work to do. Thus, you're paying a large CPU cost for the possibility of being cooperative. Furthermore, it makes the function difficult to test by introducing time-based scheduling interactions.
An alternative that Twisted offers to try to cope with some of these difficulties is twisted.internet.task.cooperate which drops inlineCallbacks in favor of bare generators. These actually do use yield to offer suspend-points but do so in a way that doesn't give the whole reactor a chance to run before resuming. This addresses some CPU concerns and possibly some maintenance difficulties, though ultimately it still introduces some time-based dependencies. However, it is at least possible to operate on the bare generators without involving cooperate which mitigates this difficulty to some extent (compare this with inlineCallbacks where idiomatically written code destroys the underlying generator and only offers a function that returns a Deferred up for testing).