I am using Solr 4.8 on Windows Server 2008 R2.
Earlier Solr was working fine and numbers of documents has been indexed as well (almost 80MB of data when I check data folder).
But suddenly indexing has stopped and moreover Solr Core is not working. (note that the older blank core is working fine, but the one I was using is no longer working.)
More shocking thing is I'm not getting any error in solr.log or catalina log file.
Check the screenshot on what happens when I try to ping it: http://imageupper.com/i/?S0200010040011M14013429922007657
I have restarted the Solr after clearing old logging file, and here is the new catalina:
May 29, 2014 8:21:42 AM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: C:\Program Files (x86)\Apache Software Foundation\Tomcat 7.0\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\Microsoft SQL Server\110\DTS\Binn\;C:\Program Files (x86)\Microsoft SQL Server\110\Tools\Binn\;C:\Program Files\Microsoft SQL Server\110\Tools\Binn\;C:\Program Files (x86)\Microsoft SQL Server\110\Tools\Binn\ManagementStudio\;C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\PrivateAssemblies\;C:\Program Files (x86)\Microsoft SQL Server\110\DTS\Binn\;C:\Program Files\Microsoft\Web Platform Installer\;;.
May 29, 2014 8:21:43 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-8983"]
May 29, 2014 8:21:43 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["ajp-bio-8009"]
May 29, 2014 8:21:43 AM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 774 ms
May 29, 2014 8:21:43 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Catalina
May 29, 2014 8:21:43 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.53
May 29, 2014 8:21:43 AM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deploying web application archive C:\Program Files (x86)\Apache Software Foundation\Tomcat 7.0\webapps\solr.war
May 29, 2014 8:21:49 AM org.apache.catalina.util.SessionIdGenerator createSecureRandom
INFO: Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [356] milliseconds.
May 29, 2014 8:21:52 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory C:\Program Files (x86)\Apache Software Foundation\Tomcat 7.0\webapps\docs
May 29, 2014 8:21:53 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory C:\Program Files (x86)\Apache Software Foundation\Tomcat 7.0\webapps\manager
May 29, 2014 8:21:53 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory C:\Program Files (x86)\Apache Software Foundation\Tomcat 7.0\webapps\ROOT
May 29, 2014 8:21:53 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-8983"]
May 29, 2014 8:21:53 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["ajp-bio-8009"]
May 29, 2014 8:21:53 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 10179 ms
Here are the last few lines from solr.log, note that there is no error logged here as well.
INFO - 2014-05-29 08:40:22.188; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/admin/ping params={action=status&_=1401342022112&wt=json} status=503 QTime=0
INFO - 2014-05-29 08:40:22.202; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/admin/file/ params={file=admin-extra.html&_=1401342022110} status=0 QTime=1
INFO - 2014-05-29 08:40:22.203; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/admin/system params={_=1401342022107&wt=json} status=0 QTime=2
INFO - 2014-05-29 08:40:22.204; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/admin/luke params={numTerms=0&_=1401342022098&show=index&wt=json} status=0 QTime=3
INFO - 2014-05-29 08:40:22.205; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/replication params={command=details&_=1401342022105&wt=json} status=0 QTime=4
INFO - 2014-05-29 08:40:30.037; org.apache.solr.core.SolrCore; [newcore] webapp=/solr path=/admin/file/ params={file=admin-extra.html&_=1401342030030} status=0 QTime=1
INFO - 2014-05-29 08:40:30.037; org.apache.solr.core.SolrCore; [newcore] webapp=/solr path=/admin/system params={_=1401342030027&wt=json} status=0 QTime=2
INFO - 2014-05-29 08:40:30.038; org.apache.solr.core.SolrCore; [newcore] webapp=/solr path=/admin/ping params={action=status&_=1401342030032&wt=json} status=503 QTime=0
INFO - 2014-05-29 08:49:03.477; org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null path=/admin/info/properties params={_=1401342543470&wt=json} status=0 QTime=0
INFO - 2014-05-29 08:56:08.640; org.apache.solr.update.processor.LogUpdateProcessor; [newcore] webapp=/solr path=/update params={version=2.2} {} 0 213
INFO - 2014-05-29 08:56:08.661; org.apache.solr.update.processor.DistributedUpdateProcessor; Ignoring commit while not ACTIVE - state: REPLAYING replay:0
INFO - 2014-05-29 08:56:08.661; org.apache.solr.update.processor.LogUpdateProcessor; [newcore] webapp=/solr path=/update params={version=2.2} {commit=} 0 1
INFO - 2014-05-29 08:57:52.693; org.apache.solr.update.processor.LogUpdateProcessor; [newcore] webapp=/solr path=/update params={version=2.2} {add=[61942 (1469414713419890688), 61943 (1469414713510068224), 61944 (1469414713521602560), 61945 (1469414713532088320), 61947 (1469414713541525504), 61946 (1469414713550962688), 61949 (1469414713559351296), 61948 (1469414713566691328), 61950 (1469414713572982784), 61951 (1469414713578225664), ... (100 adds)]} 0 372
INFO - 2014-05-29 08:57:52.696; org.apache.solr.update.processor.DistributedUpdateProcessor; Ignoring commit while not ACTIVE - state: REPLAYING replay:0
INFO - 2014-05-29 08:57:52.696; org.apache.solr.update.processor.LogUpdateProcessor; [newcore] webapp=/solr path=/update params={version=2.2} {commit=} 0 0
INFO - 2014-05-29 08:58:07.425; org.apache.solr.update.DirectUpdateHandler2; start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
INFO - 2014-05-29 08:58:09.901; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onCommit: commits: num=2
commit{dir=NRTCachingDirectory(SimpleFSDirectory@C:\solr\newcore\data\index lockFactory=NativeFSLockFactory@C:\solr\newcore\data\index; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1lo,generation=2076}
commit{dir=NRTCachingDirectory(SimpleFSDirectory@C:\solr\newcore\data\index lockFactory=NativeFSLockFactory@C:\solr\newcore\data\index; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1lp,generation=2077}
INFO - 2014-05-29 08:58:09.901; org.apache.solr.core.SolrDeletionPolicy; newest commit generation = 2077
INFO - 2014-05-29 08:58:09.905; org.apache.solr.search.SolrIndexSearcher; Opening Searcher@140a194[newcore] realtime
INFO - 2014-05-29 08:58:09.906; org.apache.solr.update.DirectUpdateHandler2; end_commit_flush
INFO - 2014-05-29 09:01:15.092; org.apache.solr.update.processor.LogUpdateProcessor; [newcore] webapp=/solr path=/update params={version=2.2} {add=[62042 (1469414925835173888), 62043 (1469414925839368192), 62044 (1469414925841465344), 62046 (1469414925843562496), 62045 (1469414925845659648), 62047 (1469414925846708224), 62048 (1469414925848805376), 62049 (1469414925849853952), 62050 (1469414925851951104), 62052 (1469414925854048256), ... (100 adds)]} 0 179
INFO - 2014-05-29 09:01:15.096; org.apache.solr.update.processor.DistributedUpdateProcessor; Ignoring commit while not ACTIVE - state: REPLAYING replay:0
INFO - 2014-05-29 09:01:15.097; org.apache.solr.update.processor.LogUpdateProcessor; [newcore] webapp=/solr path=/update params={version=2.2} {commit=} 0 1
INFO - 2014-05-29 09:01:29.919; org.apache.solr.update.DirectUpdateHandler2; start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
INFO - 2014-05-29 09:01:31.990; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onCommit: commits: num=2
commit{dir=NRTCachingDirectory(SimpleFSDirectory@C:\solr\newcore\data\index lockFactory=NativeFSLockFactory@C:\solr\newcore\data\index; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1lp,generation=2077}
commit{dir=NRTCachingDirectory(SimpleFSDirectory@C:\solr\newcore\data\index lockFactory=NativeFSLockFactory@C:\solr\newcore\data\index; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1lq,generation=2078}
INFO - 2014-05-29 09:01:31.990; org.apache.solr.core.SolrDeletionPolicy; newest commit generation = 2078
INFO - 2014-05-29 09:01:31.993; org.apache.solr.search.SolrIndexSearcher; Opening Searcher@a0a929[newcore] realtime
INFO - 2014-05-29 09:01:31.995; org.apache.solr.update.DirectUpdateHandler2; end_commit_flush
Whats most problematic is, I don't get any error logged anywhere! What could be the problem. I didn't changed Solr-config or schema. Moreover, same solr schema and config is working fine on many other servers in production use and works fine on test server as well so I don't think the issue is with that. Moreover, note that Solr was working fine and was indexing document too.
Note that I have reviewed and taken suggested changes discussed here: How to fix exceeded limit of maxWarmingSearchers? but still it doesn't helped.
Any advise on how to proceed with this?
Just an update that after downgrading the same Solr instance to Solr 4.7, everything worked fine. So the issue seems to be specifically with Solr 4.8. However, not sure what was the issue. Recently tested with Solr 4.9 and it worked too.