Search code examples
javaneo4jcypher

Neo4j parallel CREATE queries stuck

I've a Java application that creates Neo4j relationships between existing nodes. the app has about 16 threads (on 16 CPU cores) and each uses some Cypher like:

// smRelRows comes from java, as run() parameter
UNWIND $smRelRows AS relRow\s
MATCH ( gene:Gene { appId: relRow.geneId } ),
        ( concept:Concept { appId: relRow.conceptId } )
CREATE (gene) - [:hasMotifLink{ graphDistance: relRow.graphDistance }] -> (concept)

The app queues a few thousand tasks doing the above, but its parallelism is limited to only 16 threads. Each query receives 2500 values in smRelRows and is run in its own transaction.

The problem is everything is completely stuck, in the Neo4j server, SHOW TRANSACTIONS shows the first 16 queries running since the app started.

I suspect some deadlock, due to the lock of relation-shared nodes, but then I don't know I to fix the problem. In the past, I've written similar multi-thread CREATE-based updates without problems.

I've added appId's indexes, but no change.

Solution

  • Nodes are not locked anymore when you're creating relationships.

    For eg this block of code works without issues and produces generally around 65 concurrent transactions.

       private final String query = """
            MATCH (a:Node {id: 1})
            MATCH (b:Node {id: 2})
            CREATE (a)-[r:REL {type: 'demo'}]->(b)
            """;
   @Test
    void parallel_rels() {
        try (Driver driver = GraphDatabase.driver("bolt://localhost:7687", AuthTokens.basic("neo4j", "password"))) {
            IntStream.range(1, 10000)
                    .parallel()
                    .forEach(i -> {
                        try (Session session = driver.session(SessionConfig.forDatabase("locking2"))) {
                            session.executeWriteWithoutResult((tx) -> {
                                tx.run(query).consume();
                            });
                        }
                    });
        }
    }

    However, this the list of active locks taken when you're creating a relationship :

    ╒═══════════╤════════════════════════════════╤═══════════════════╕
│mode       │resourceType                    │resourceId         │
╞═══════════╪════════════════════════════════╪═══════════════════╡
│"EXCLUSIVE"│"RELATIONSHIP"                  │0                  │
├───────────┼────────────────────────────────┼───────────────────┤
│"SHARED"   │"INDEX_ENTRY"                   │649269460893700912 │
├───────────┼────────────────────────────────┼───────────────────┤
│"SHARED"   │"INDEX_ENTRY"                   │-890913567585829783│
├───────────┼────────────────────────────────┼───────────────────┤
│"SHARED"   │"LABEL"                         │0                  │
├───────────┼────────────────────────────────┼───────────────────┤
│"SHARED"   │"RELATIONSHIP_TYPE"             │0                  │
├───────────┼────────────────────────────────┼───────────────────┤
│"SHARED"   │"RELATIONSHIP_TYPE"             │2147483647         │
├───────────┼────────────────────────────────┼───────────────────┤
│"EXCLUSIVE"│"RELATIONSHIP_DELETE"           │0                  │
├───────────┼────────────────────────────────┼───────────────────┤
│"SHARED"   │"NODE_RELATIONSHIP_GROUP_DELETE"│0                  │
├───────────┼────────────────────────────────┼───────────────────┤
│"SHARED"   │"NODE_RELATIONSHIP_GROUP_DELETE"│1                  │
├───────────┼────────────────────────────────┼───────────────────┤
│"SHARED"   │"DEGREES"                       │0                  │
├───────────┼────────────────────────────────┼───────────────────┤
│"SHARED"   │"DEGREES"                       │1                  │
└───────────┴────────────────────────────────┴───────────────────┘

    The following locks however can happen :

    • RelationshipGroup : if multiple threads would compete to update the head of the relationship group
    • Degrees : If you have other transactions that are updating labels on the same nodes, locks will be taken to ensure consistent count stores
    • Writes : If you're trying to update properties of the same nodes in other transactions, for sparse nodes ( nodes having less than 50 relationships ) a lock will be taken, for dense nodes not

    The best way to investigate what is happening is while all your transactions are running, check the status of the transactions with SHOW TRANSACTIONS YIELD * where you can investigate if transactions are blocked by other transactions and which ones

    enter image description here

    Then, couple of questions :

    • Are you really using CREATE for creating the relationship or you're using MERGE ?
    • What are the constraints in your database -> output of SHOW CONSTRAINTS