Search code examples
solrdatastax-enterprise

DSE 2 solr nodes, both go down when 1 is unavailable

I'm currently migrating my existing SOLR cloud implementation to a Datastax Enterprise (3.2.1) Solr implementation.

I have 8 DSE nodes, 6 cassandra nodes and 2 solr nodes. When either of the solr nodes goes down, the other one also fails to do queries with the following exception:

ERROR [http-8983-98] 2013-11-22 20:03:32,447 SolrDispatchFilter.java (line 641) Error request exception: java.io.IOException: Unavailable shards for ranges: [(-6148914691236517206,-3074457345618258604], (-3074457345618258604,-2], (-9223372036854775807,-6148914691236517206], (-2,0]]
java.lang.RuntimeException: java.io.IOException: Unavailable shards for ranges: [(-6148914691236517206,-3074457345618258604], (-3074457345618258604,-2], (-9223372036854775807,-6148914691236517206], (-2,0]]
    at org.apache.solr.handler.component.CassandraHttpShardHandler.checkDistributed(CassandraHttpShardHandler.java:40)
    at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:178)
    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
    at org.apache.solr.core.SolrCore.execute(SolrCore.java:1839)
    at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:667)
    at com.datastax.bdp.cassandra.index.solr.CassandraDispatchFilter.execute(CassandraDispatchFilter.java:176)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:359)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:155)
    at com.datastax.bdp.cassandra.index.solr.CassandraDispatchFilter.doFilter(CassandraDispatchFilter.java:139)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.datastax.bdp.cassandra.audit.SolrHttpAuditLogFilter.doFilter(SolrHttpAuditLogFilter.java:194)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.datastax.bdp.cassandra.index.solr.auth.CassandraAuthorizationFilter.doFilter(CassandraAuthorizationFilter.java:95)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.datastax.bdp.cassandra.index.solr.auth.DseAuthenticationFilter.doFilter(DseAuthenticationFilter.java:102)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
    at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
    at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
    at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
    at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.IOException: Unavailable shards for ranges: [(-6148914691236517206,-3074457345618258604], (-3074457345618258604,-2], (-9223372036854775807,-6148914691236517206], (-2,0]]
    at com.datastax.bdp.cassandra.index.solr.dht.OptimizedShardSelectionStrategy.select(OptimizedShardSelectionStrategy.java:59)
    at com.datastax.bdp.cassandra.index.solr.dht.ShardRouter.endpointsToContact(ShardRouter.java:579)
    at com.datastax.bdp.cassandra.index.solr.dht.ShardRouter.route(ShardRouter.java:184)
    at org.apache.solr.handler.component.CassandraHttpShardHandler.checkDistributed(CassandraHttpShardHandler.java:36)
    ... 29 more

I can't figure out why this happens, it should be fault tolerant right?

Anyone has some ideas how to fix this?

Solution

  • Do you have replication factor of one. If so the error is saying that the node cannot be queried therefore complete results cannot be returned.