Search code examples
pythoncrashtornado

tornado multi process crash with nltk

I'm getting some very strange behaviour in a tornado application. I'm running a lot of processes each which have their own HTTP servers and running on a different port.

I added a new process to the system which is another TCPServer class that listens on an entirely different port and doesn't have any interaction with the other processes.

I bring the new server up as follows:

def runSimService(port):
   sim=SimService()
   sim.listen(port)
   currentIOLoop = tornado.ioloop.IOLoop.current()
   currentIOLoop.start()

class SimService(TCPServer):
   def __init__(self,host='localhost',motorport=27017):

      TCPServer.__init__(self)
      self.log=logging.getLogger("tornado.access")

      # Needs to contain a User class log. 
      self.con=motor.MotorClient(host,motorport)
      self.db=self.con.pDB
      self.col=self.db.pCol

Basically that's the only code I left when debugging. The crash i get isn't a normal python exception crash which is worrying me.

enter image description here

I'm developing on a mac the moment. Can someone please explain is this crash is something wrong with my code or is there something else happening here?'

UPDATE:

Ok this is really bizare it only seems to happen when I import the following:

from nltk.stem.snowball import SnowballStemmer

OR

from nltk import word_tokenize, pos_tag

or nltk in general...

Could there be some weird interaction between libraries happening? I'm stuck

Code to create processes

if __name__ =='__main__':
   AsyncHTTPClient.configure("tornado.curl_httpclient.CurlAsyncHTTPClient",max_clients=2000)

   processes=[]

   processes.append(Process(target=runSimServer,args=(...,)))
   processes.append(Process(target=runServer,args=(...)))
   processes.append(Process(target=runServer,args=(...)))
   processes.append(Process(target=runServer,args=(...)))
   processes.append(Process(target=runServer,args=(...)))
   processes.append(Process(target=runServer,args=(...)))

   # write pids to pid/ directory for use with the shutdown program
   with open("pid/"+__file__.split(".")[0]+".pid","w") as f:
      for p in processes:
         p.start()
         f.write(str(p.pid)+"\n")

Thanks

Solution

  • The key part of that crash message is "multi-threaded process forked". If you are going to use both threads and processes, you must fork all your processes before creating any threads. It looks like nltk is creating some threads when you import it. If you are also using multiple processes (it doesn't look like you are from the code you quoted, but that is obviously incomplete), you must not import nltk until after all processes have been started.