How to execute code asynchronously in Twisted Klein?

Question

I have two functions in my python Twisted Klein web service:

@inlineCallbacks
def logging(data):
    ofile = open("file", "w")
    ofile.write(data)
    yield os.system("command to upload the written file")

@APP.route('/dostuff')
@inlineCallbacks
def dostuff():
    yield logging(data)
    print "check!" 
    returnValue("42")

When os.system("command to upload the written file") runs, it will show message saying "start uploading" then "upload complete". I want to make the logging function asynchronous so that processing in logging handler happens after dostuff handler prints out "check!". (I actually want processing to happen after returnValue("42"), but both of those are making the logging function async I think?)

I thought the yield statement will make it non-blocking but it seems not the case, the "check!" always got printed after "start uploading" and "upload complete". I'll appreciate if anyone can give me some feedback on it since I'm new to async coding and got blocked on this for a while...

Answer 1

To make your code async you need to use Twisted Deferreds as described here. Deferreds give you an API for asynchronous code execution, they allow you to attach callbacks to your functions and they execute code in Twisted event loop managed by reactor object.

I see two potential ways to use Deferreds in your case.

1) Execute task in background with reactor.callLater()

This is ok if dostuff handler doesn't care about result. You can use reactor.callLater(). This way your async function will execute after you return value from doStuff.

So something like this:

from klein import run, route, Klein
from twisted.internet import defer, task, reactor
import os

app = Klein()


def logging(data):
    ofile = open("file", "w")
    ofile.write(data)
    result = os.system("ls")
    print(result)


@route('/')
def dostuff(request):
    reactor.callLater(0, logging, "some data")
    print("check!")
    return b'Hello, world!'

run("localhost", 8080)

The order of events with this code is following, first "check" is printed, then "hello world" response is returned and in the end async call suceeds and prints results of running os.system().

2016-08-11 08:52:33+0200 [-] check!
2016-08-11 08:52:33+0200 [-] "127.0.0.1" - - [11/Aug/2016:06:52:32 +0000] "GET / HTTP/1.1" 200 13 "-" "curl/7.35.0"
a.py  file

2) Execute task in background and get result with task.deferLater()

If you care about results of your 'logging' function you can also attach callback to this object and use twisted.internet.task API. If you want to go this way you need to refactor your handler to work like this

@route('/')
def dostuff(request):
    def the_end(result):
        print("executed at the end with result: {}".format(result))
    dfd = task.deferLater(reactor, 0, logging, "some data")
    dfd.addCallback(the_end)
    print("check!")
    return b'Hello, world!'

This way order of events will be same as above, but the_end function will be executed at the end after your logging function finishes.

2016-08-11 08:59:24+0200 [-] check!
2016-08-11 08:59:24+0200 [-] "127.0.0.1" - - [11/Aug/2016:06:59:23 +0000] "GET / HTTP/1.1" 200 13 "-" "curl/7.35.0"
a.py  file
2016-08-11 08:59:24+0200 [-] executed at the end with result: some result