Search code examples
javamysqlsqlhibernatestack-overflow

Stackoverflow with Hibernate using sql IN (id, id, id, id..id)


I am getting the error below saying there was a stack overflow. This is happening because a SQL statement with IN (id, id, id...id) has a ton of parameters. Is there anyway to fix this? This is happening in my local environment with Eclipse.

JPA

@Query(value="SELECT p FROM PendingCourseRegistration p WHERE p.sisId IN ?1 AND p.testId = ?2")
List<PendingCourseRegistration> findPendingCourseRegistrationInSisIdsAndTestId(List<String> sisIds, Long testID);

Error

java.lang.StackOverflowError: null
    at java.lang.Abstract witingBuilder.append(AbstractStringBuilder.java:416) ~[na:1.7.0_17]
    at java.lang.StringBuffer.append(StringBuffer.java:237) ~[na:1.7.0_17]
    at antlr.BaseAST.toStringList(BaseAST.java:341) ~[antlr-2.7.7.jar:na]
    at antlr.BaseAST.toStringList(BaseAST.java:347) ~[antlr-2.7.7.jar:na]
    at antlr.BaseAST.toStringList(BaseAST.java:347) ~[antlr-2.7.7.jar:na]
    at antlr.BaseAST.toStringList(BaseAST.java:347) ~[antlr-2.7.7.jar:na]

Hibernate Query

2:26.763 [ocPifScheduler-1] DEBUG o.s.o.j.SharedEntityManagerCreator$SharedEntityManagerInvocationHandler - Creating new EntityManager for shared EntityManager invocation
09:52:26.788 [Scheduler-1] DEBUG org.hibernate.hql.internal.ast.QueryTranslatorImpl - parse() - HQL: SELECT p FROM com.test.PendingCourseRegistration p WHERE p.sisId IN (:x10_, :x11_, :x12_, :x13_, :x14_, :x15_, :x16_, :x17_, :x18_, :x19_, :x110_, :x111_, :x112_, :x113_, :x114_, :x115_, :x116_, :x117_, :x118_, :x119_, ...:xN) AND p.id = ?2
09:52:26.891 [Scheduler-1] DEBUG org.hibernate.hql.internal.ast.QueryTranslatorImpl - --- HQL AST ---
 \-[QUERY] Node: 'query'
    +-[SELECT_FROM] Node: 'SELECT_FROM'
    |  +-[FROM] Node: 'FROM'
    |  |  \-[RANGE] Node: 'RANGE'
    |  |     +-[DOT] Node: '.'
    |  |     |  +-[DOT] Node: '.'
    |  |     |  |  +-[DOT] Node: '.'
    |  |     |  |  |  +-[DOT] Node: '.'
    |  |     |  |  |  |  +-[DOT] Node: '.'
    |  |     |  |  |  |  |  +-[DOT] Node: '.'
    |  |     |  |  |  |  |  |  +-[IDENT] Node: 'com'
    |  |     |  |  \-[IDENT] Node: 'model'
    |  |     |  \-[IDENT] Node: 'PendingCourseRegistration'
    |  |     \-[ALIAS] Node: 'p'
    |  \-[SELECT] Node: 'SELECT'
    |     \-[IDENT] Node: 'p'
    \-[WHERE] Node: 'WHERE'
       \-[AND] Node: 'AND'
          +-[IN] Node: 'in'
          |  +-[DOT] Node: '.'
          |  |  +-[IDENT] Node: 'p'
          |  |  \-[IDENT] Node: 'sisId'
          |  \-[IN_LIST] Node: 'inList'
          |     +-[COLON] Node: ':'
          |     |  \-[IDENT] Node: 'x10_'
          |     +-[COLON] Node: ':'
          |     |  \-[IDENT] Node: 'x11_'
          |     +-[COLON] Node: ':'
          |     |  \-[IDENT] Node: 'x12_'
          |     +-[COLON] Node: ':'
          |     |  \-[IDENT] Node: 'x13_'
          |     +-[COLON] Node: ':'
          |     |  \-[IDENT] Node: 'x14_'
          |     +-[COLON] Node: ':'
          |     |  \-[IDENT] Node: 'x15_'
          |     +-[COLON] Node: ':'
          |     |  \-[IDENT] Node: 'x16_'
          |     +-[COLON] Node: ':'
          |     |  \-[IDENT] Node: 'x17_'

Solution

  • In our Grails project (2.3.6) with Hibernate 4.3.1.Final, we never had that error but we encountered another error due to query buffer size restriction:

    Since you are using in (?, ..., ?) you write as much ?, as you have items in your list, which means that for big list (says 50000) you may write a query of 100000 characters, and you may have this exception (here with pgSQL drivers):

    Foobar.executeQuery("select f from Foobar f where f.id in (:ids)", 
                     [ids: 1L..100000L]); // Groovy way of creating a list of 100000 items.
    

    And the error:

    SqlExceptionHelper:146 - An I/O error occured while sending to the backend. 
    SqlExceptionHelper:146 - This connection has been closed.
    

    That why I think you might need to

    1. Either split your id list into smaller list (like 500 items or so) and coalesced the result by hand, which is more work to do than the default spring-data you are using.

    2. Either store the id list into a temporary table (which could prove to be a painful task using JPA). The temporary table would be a tuple of (key, sids). You would generate a temporary key (for the session), batch insert the ids into that table with the key, flush so that hibernate push the change onto the database, use that table using a subquery (p.sidIds in (select sisIds from IdTable where key = ?1)), delete data from that table. While being a painful task, it may produce a boost in performance.