Multithread call inside Twisted _delayedRender of request

Question

I have simple Twisted webserver application serving my math requests. Everything working fine (I hide big code pieces which is not related to my question):

#import section ...

class PlsPage(Resource):
    isLeaf = True

    def render_POST(self, request):
        reactor.callLater(0, self._delayedRender, request)
        return NOT_DONE_YET

    def _delayedRender(self, request):

        #some actions before
        crossval_scores = cross_validation.cross_val_score(pls1, X, y=numpy.asarray(Y), scoring=my_custom_scorer, cv=KFold(700, n_folds=700))
        #some actions after

        request.finish()

reactor.listenTCP(12000, server.Site(PlsPage()))
reactor.run()

When I try to speed up cross_validation calculation by setting n_jobs for example to 3.

crossval_scores = cross_validation.cross_val_score(pls1, X, y=numpy.asarray(Y), scoring=my_custom_scorer, cv=KFold(700, n_folds=700), n_jobs=3)

and after that I got exactly 3 exceptions:

twisted.internet.error.CannotListenError: Couldn't listen on any:12000: [Errno 10048] Only one usage of each socket address (protocol/network address/port) is normally permitted.

For some reasons I can't call cross_val_score with n_jobs > 1 inside _delayedRender. Here is a traceback of exception, for some reasons reactor.listenTCP trying to start 3 times too.

Any ideas how to get it work?

UPD1. I create file PLS.py and moved all the code here, except last 2 lines:

from twisted.web import server
from twisted.internet import reactor, threads
import PLS

reactor.listenTCP(12000, server.Site(PLS.PlsPage()))
reactor.run()

But the problem still persists. I also found that this problem persists only on Windows. My Linux machine run this scripts well.

Answer 1

scikit_learn apparently uses the multiprocessing module in order to achieve concurrency. The multiprocessing transmits data between processes using pickle, which, among other... idiosyncratic problems that it causes, will cause some of the modules imported in your parent process to be imported in your worker processes.

Your PLS_web.py "module", however, is not actually a module, it's a script; since you have put reactor.listenTCP and reactor.run at the bottom of it, it actually does stuff when you import it rather than just loading its code.

This particular error is because since your web server is being run 4 times (once for the controller process, once for each of the three jobs), each of the 3 times beyond the first encounter an error because the first server is already listening on port 12000.

You should remove the reactor.run/reactor.listenTCP lines elsewhere, into a top level script. A good rule of thumb is that these lines should never appear in the same file as a class or def statement; define your code in one place and start it up in another. Once you've moved it to a file that doesn't get imported (and you might want to even put it in a file whose name isn't a legal module identifier, like run-my-server.py) then multiprocessing might be able to import all the code it needs and do its job.

Better yet, don't write those lines at all, write a twisted application plugin and run your program with twistd. If you don't have to put the reactor.run statement in any place, you can't put it in the wrong place :).