Search code examples
javaperformanceluceneneo4jcypher

How to tweak lucene maxClauseCount from Neo4j?

Following is our cypher for a find by ID kind of service:

START n=node:PATIENTS('MEMBER_PLAN_ID:(1 2)') return n

Where 1 2 are the ids passed. When we pass around 2000 ids following error occurs:

java.lang.RuntimeException: org.apache.lucene.queryParser.ParseException: Cannot parse 'MEMBER_PLAN_ID:(1 2)': too many boolean clauses
        at org.neo4j.index.impl.lucene.IndexType.query(IndexType.java:304)
        at org.neo4j.index.impl.lucene.LuceneIndex.query(LuceneIndex.java:227)
        at org.neo4j.index.impl.lucene.LuceneIndex.query(LuceneIndex.java:238)
        at org.neo4j.cypher.internal.spi.gdsimpl.GDSBackedQueryContext$$anon$1.indexQuery(GDSBackedQueryContext.scala:87)
        at org.neo4j.cypher.internal.executionplan.builders.IndexQueryBuilder$$anonfun$getNodeGetter$2.apply(IndexQueryBuilder.scala:83)
        at org.neo4j.cypher.internal.executionplan.builders.IndexQueryBuilder$$anonfun$getNodeGetter$2.apply(IndexQueryBuilder.scala:81)
        at org.neo4j.cypher.internal.pipes.StartPipe$$anonfun$internalCreateResults$1.apply(StartPipe.scala:36)
        at org.neo4j.cypher.internal.pipes.StartPipe$$anonfun$internalCreateResults$1.apply(StartPipe.scala:35)
        at scala.collection.Iterator$$anon$13.__AW_hasNext(Iterator.scala:371)
        at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala)
        at org.neo4j.cypher.internal.ClosingIterator$$anonfun$hasNext$1.apply$mcZ$sp(ClosingIterator.scala:36)
        at org.neo4j.cypher.internal.ClosingIterator$$anonfun$hasNext$1.apply(ClosingIterator.scala:35)
        at org.neo4j.cypher.internal.ClosingIterator$$anonfun$hasNext$1.apply(ClosingIterator.scala:35)
        at org.neo4j.cypher.internal.ClosingIterator.failIfThrows(ClosingIterator.scala:86)
        at org.neo4j.cypher.internal.ClosingIterator.hasNext(ClosingIterator.scala:35)
        at org.neo4j.cypher.PipeExecutionResult.hasNext(PipeExecutionResult.scala:157)
        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
        at scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:29)
        at org.neo4j.cypher.PipeExecutionResult$$anon$1.hasNext(PipeExecutionResult.scala:73)
        at net.ahm.graph.dao.PatientDAO.__AW_findPatients(PatientDAO.java:376)


Caused by: org.apache.lucene.queryParser.ParseException: Cannot parse 'MEMBER_PLAN_ID:(1 2)': too many boolean clauses
        at org.apache.lucene.queryParser.QueryParser.parse(QueryParser.java:221)
        at org.neo4j.index.impl.lucene.IndexType.query(IndexType.java:300)
        ... 38 more
Caused by: org.apache.lucene.search.BooleanQuery$TooManyClauses: maxClauseCount is set to 1024
        at org.apache.lucene.search.BooleanQuery.add(BooleanQuery.java:136)
        at org.apache.lucene.queryParser.QueryParser.getBooleanQuery(QueryParser.java:958)
        at org.apache.lucene.queryParser.QueryParser.getBooleanQuery(QueryParser.java:933)
        at org.apache.lucene.queryParser.QueryParser.Query(QueryParser.java:1281)
        at org.apache.lucene.queryParser.QueryParser.Clause(QueryParser.java:1323)
        at org.apache.lucene.queryParser.QueryParser.Query(QueryParser.java:1245)
        at org.apache.lucene.queryParser.QueryParser.TopLevelQuery(QueryParser.java:1234)
        at org.apache.lucene.queryParser.QueryParser.parse(QueryParser.java:206)
        ... 39 more

We see this in the stack trace: maxClauseCount is set to 1024

Is there a way to configure this limit from Neo4j while using Cypher?

Solution

  • However, if i add this line to our service, things work fine:

    BooleanQuery.setMaxClauseCount(20000);

    We did some load tests where 1000 concurrent users hit the service with 10,000 ids each. Not seeing service crash / any unexpected performance problems afterwards.