Why does creating a neo4j.GraphDatabase from within a Paste app cause a segfault?

Question

The following code causes Java to segfault:

import os.path
import neo4j
from paste import httpserver, fileapp
import tempfile
from webob.dec import wsgify
from webob import Response, Request

HOST = '127.0.0.1'
PORT = 8080

class DebugApp(object):
    @wsgify
    def __call__(self, req):

        # db = neo4j.GraphDatabase(tempfile.mkdtemp())
        db = neo4j.GraphDatabase(os.path.join(os.path.dirname(os.path.abspath(__file__)), 'data'))
        return Response(body='it worked')

def main():
    app = DebugApp()
    httpserver.serve(app, host=HOST, port=PORT)

if __name__ == '__main__':
    main()

To reproduce, first save that code into a file (say, app.py), and then run python app.py. Then try http://localhost:8080 in your browser; you should see the Java crash handler.

The top of the Java stack trace looks like this:

Stack: [0xb42e7000,0xb4ae8000],  sp=0xb4ae44f0,  free space=8181k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [_jpype.so+0x26497]  JPJavaEnv::NewObjectA(_jclass*, _jmethodID*, jvalue*)+0x37
C  [_jpype.so+0x3c0e8]  JPMethodOverload::invokeConstructor(_jclass*, std::vector<HostRef*, std::allocator<HostRef*> >&)+0x178
C  [_jpype.so+0x3a417]  JPMethod::invokeConstructor(std::vector<HostRef*, std::allocator<HostRef*> >&)+0x47
C  [_jpype.so+0x1beba]  JPClass::newInstance(std::vector<HostRef*, std::allocator<HostRef*> >&)+0x2a
C  [_jpype.so+0x67b9c]  PyJPClass::newClassInstance(_object*, _object*)+0xfc
C  [python+0x96822]  PyEval_EvalFrameEx+0x4332
C  [python+0x991e7]  PyEval_EvalCodeEx+0x127

I believe that's neo4j.GraphDatabase in Python triggering JPype to go looking for EmbeddedGraphDatabase in neo4j, under Java.

Running this code in an interactive Python session doesn't segfault:

>>> import webob
>>> import app
>>> debug_app = app.DebugApp()
>>> response = debug_app(webob.Request.blank('/'))
>>> response.body
'it worked'

Presumably that's because I'm avoiding Paste altogether in that example. Perhaps this has something to do with Paste's use of threads getting in the way of neo4j? I noted a somewhat similar problem in the neo4j forums: http://neo4j-community-discussions.438527.n3.nabble.com/Neo4j-CPython-Pylons-and-threading-td942435.html

...but that only occurs on shutdown.

Answer 1

The issue is not with Paste per se, but with the neo4j Python bindings, which use JPype. Paste creates threads to handle incoming requests; neo4j is supposed to be thread-safe, but JPype comes with this caveat from the documentation (1):

"For the most part, python threads based on OS level threads (i.e posix threads), will work without problem. The only thing to remember is to call jpype.attachThreadToJVM() in the thread body to make the JVM usable from that thread. For threads that you do not start yourself, you can call isThreadAttachedToJVM() to check."

I couldn't find the code that does this, but I think that some of the Java code in the neo4j bindings may call attachThreadToJVM at import time. If so, when a request is handed to a worker thread by paste, and that thread then goes to fetch data from neo4j, it is crossing thread boundaries, and the JVM attachment rule may not be satisfied.

You can avoid the crash by only running import neo4j from within a single thread. In the case above, this is the callable targeted by threading.Thread.

Unfortunately, this means that even though neo4j is thread-safe, it must be constrained to a single thread when used from Python. But that's not too disappointing, considering.

Update: the maintainers responded(2) and investigated the problem, and checked in a fix. I don't know which release of neo4j this was available in, and I can no longer find the commit to their github repo(3), so this stands for re-testing.