Search code examples
neo4jneo4j-javascript

Neo4j 10k MERGE statements fail


I am running a batch merge operation in Neo4j, but it keeps on failing. I increased the heap size and heap limit but is seems like my operations is perhaps not supported or there's a more appropriate method of doing this.

I have 10k of such MERGE statements

var transaction = `
                   MERGE (n0: Account {sfRecordId:$Id0})
                         ON CREATE SET ...
                         ON MATCH SET ...
                   MERGE (n1: Account {sfRecordId:$Id1})
                         ON CREATE SET ...
                         ON MATCH SET ...
                   MERGE ... //10k

// then send to Neo4j via the Javascript driver:
 tx.run(transaction, bindParams)

My heap size is set to

server.memory.heap.initial_size=2G
server.memory.heap.max_size=3G

I have tried sending them all at once, I have tried batching them into batches of 100, 500, 1, and 10,000.

None of these seems to get my 10k to be inserted.

The first random 500-4k get in, but then the server crashes with an out of memory error.


Solution

  • Generating a very large statement to bulk insert data is not recommended. A good way of inserting such bulk data is to prepare a list of objects, and use UNWIND to iterate through it:

    UNWIND $data as item
    MERGE (n:Account {sfRecordId:item.Id})
                        ON CREATE SET ...
                        ON MATCH SET ...