Why does Twisted's inlineCallbacks return a "Deferred' object?

Question

I am trying to understand how inlineCallbacks that makes asynchronous code looks like synchronous code work. I am using the Twisted's implementation as reference here.

Normal function: Input → Output

inlineCallbacks decorated generator function: Input → Deferred

It gives back a Deferred i.e an Object that can be registered with callback.

from twisted.internet import defer

intermediate_deferred = None

@defer.inlineCallbacks
def example():
    global intermediate_deferred
    print("Started")
    intermediate_deferred = defer.Deferred()
    result = yield intermediate_deferred
    print("Resuming for the first time with %s" % result)
    intermediate_deferred = defer.Deferred()
    result = yield intermediate_deferred
    print("Resuming for the second time with %s" % result)

generator_deferred = example()
intermediate_deferred.callback("One")
intermediate_deferred.callback("Two")

1. What is the real need to return a Deferred object i.e who the consumes it?
2. Who invokes it with the final value?
3. What happens if it not returned?

In a repl I don't see why I need this return value. My fundamental problems seems to arise from the fact that I am still not thinking in terms of the consumer of these functions?

So let take some web framework and imagine how the code should be:

# Assume it returns 'example' function object.
function_to_call = get_routing_function(request)
d = function_to_call()
# But who takes care of calling it back? The Event Loop has to do that?
# So the Event Loop has one important job of monitoring and calling back
# such Deferred's to complete the full cycle?
d.addCallback(handle_http_response)

@defer.inlineCallbacks
def get_user_details(user_idn):
    result = yield get_user_details_from_cache(user_idn)
    if not result:
       result = yield get_user_details_from_db(user_idn)
    result = post_process_user_details(user_idn)
    return result

Answer 1

What is the real need to return a Deferred object i.e who the consumes it?

The code which triggers such an inline callback function can register its post processing actions here. One example could be: d.addCallback(handle_http_response).

Who invokes it with the final value?

The inline callback decorator machinery does this internally. It fires this Deferred when the generator raises a StopIteration i.e when the generator has finally returned something for outside world. In constrast,yielded values are only used internally within the generator.

What happens if it not returned?

You can use these generator functions only for the side-effects i.e you can't do anything with output value of these functions.