ruby-on-rails solr sunspot-solr engineyard

Solr with Rails - rake sunspot:reindex is not working

I am stuck into a strange issue where I am looking for your inputs.

My problem is:

After deploying my application on Production using Capistrano, when I am doing solr re-indexing, it is giving me below error:

$ bundle exec rake sunspot:reindex --trace
** Invoke sunspot:reindex (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute sunspot:reindex
Skipping progress bar: for progress reporting, add gem 'progress_bar' to your Gemfile
rake aborted!
RSolr::Error::Http - 404 Not Found
Error:     Not Found

Request Data: "<?xml version=\"1.0\" encoding=\"UTF-8\"?><delete>query>type:OccupationData</query></delete>"
Backtrace: /data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rsolr-1.0.9/lib/rsolr/client.rb:268:in `adapt_response'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rsolr-1.0.9/lib/rsolr/client.rb:175:in `execute'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rsolr-1.0.9/lib/rsolr/client.rb:161:in `send_and_receive'
(eval):2:in `post'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rsolr-1.0.9/lib/rsolr/client.rb:67:in `update'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rsolr-1.0.9/lib/rsolr/client.rb:131:in `delete_by_query'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/indexer.rb:55:in `remove_all'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/session.rb:181:in `block in remove_all'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/session.rb:181:in `each'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/session.rb:181:in `remove_all'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/session_proxy/abstract_session_proxy.rb:11:in `remove_all'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rsolr-1.0.9/lib/rsolr/client.rb:268:in `adapt_response'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rsolr-1.0.9/lib/rsolr/client.rb:175:in `execute'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rsolr-1.0.9/lib/rsolr/client.rb:161:in `send_and_receive'
(eval):2:in `post'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rsolr-1.0.9/lib/rsolr/client.rb:67:in `update'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rsolr-1.0.9/lib/rsolr/client.rb:131:in `delete_by_query'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/indexer.rb:55:in `remove_all'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/session.rb:181:in `block in remove_all'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/session.rb:181:in `each'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/session.rb:181:in `remove_all'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/session_proxy/abstract_session_proxy.rb:11:in `remove_all'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/session_proxy/retry_5xx_session_proxy.rb:17:in `method_missing'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/session_proxy/abstract_session_proxy.rb:11:in `remove_all'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot.rb:464:in `remove_all'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot_rails-1.3.3/lib/sunspot/rails/searchable.rb:178:in `solr_remove_all_from_index'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot_rails-1.3.3/lib/sunspot/rails/searchable.rb:197:in `solr_reindex'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot_rails-1.3.3/lib/sunspot/rails/tasks.rb:58:in `block (3 levels) in <top (required)>'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/class_set.rb:16:in `each'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/class_set.rb:16:in `each'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot_rails-1.3.3/lib/sunspot/rails/tasks.rb:57:in `block (2 levels) in <top (required)>'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/task.rb:246:in `call'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/task.rb:246:in `block in execute'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/task.rb:241:in `each'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/task.rb:241:in `execute'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/task.rb:184:in `block in invoke_with_call_chain'
/usr/lib64/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/task.rb:177:in `invoke_with_call_chain'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/task.rb:170:in `invoke'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/application.rb:143:in `invoke_task'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/application.rb:101:in `block (2 levels) in top_level'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/application.rb:101:in `each'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/application.rb:101:in `block in top_level'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/application.rb:110:in `run_with_threads'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/application.rb:95:in `top_level'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/application.rb:73:in `block in run'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/application.rb:160:in `standard_exception_handling'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/application.rb:70:in `run'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/bin/rake:33:in `<top (required)>'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/bin/rake:19:in `load'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/bin/rake:19:in `<main>'
Tasks: TOP => sunspot:reindex

To investigate it more, when I looked at sunspot production log, I saw:

INFO: Creating SolrCore 'collection1' using instanceDir: /data/APP_NAME/releases/20130325112326/solr/collection1
Apr 1, 2013 2:38:16 AM org.apache.solr.core.SolrResourceLoader <init>
INFO: new SolrResourceLoader for directory: '/data/APP_NAME/releases/20130325112326/solr/collection1/'
Apr 1, 2013 2:38:16 AM org.apache.solr.core.CoreContainer recordAndThrow
SEVERE: Unable to create core: collection1
org.apache.solr.common.SolrException: Could not load config for solrconfig.xml
    at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:991)
    at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1051)
    at org.apache.solr.core.CoreContainer$3.call(CoreContainer.java:634)
    at org.apache.solr.core.CoreContainer$3.call(CoreContainer.java:629)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Can't find resource 'solrconfig.xml' in classpath or '/data/APP_NAME/releases/20130325112326/solr/collection1/conf/', cwd=/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/bundler/gems/sunspot-87e29212527f/sunspot_solr/solr
    at org.apache.solr.core.SolrResourceLoader.openResource(SolrResourceLoader.java:318)
    at org.apache.solr.core.SolrResourceLoader.openConfig(SolrResourceLoader.java:283)
    at org.apache.solr.core.Config.<init>(Config.java:103)
    at org.apache.solr.core.Config.<init>(Config.java:73)
    at org.apache.solr.core.SolrConfig.<init>(SolrConfig.java:117)
    at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:989)
    ... 11 more
Apr 1, 2013 2:38:16 AM org.apache.solr.common.SolrException log
SEVERE: null:org.apache.solr.common.SolrException: Unable to create core: collection1
    at org.apache.solr.core.CoreContainer.recordAndThrow(CoreContainer.java:1672)
    at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1057)
    at org.apache.solr.core.CoreContainer$3.call(CoreContainer.java:634)
    at org.apache.solr.core.CoreContainer$3.call(CoreContainer.java:629)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.solr.common.SolrException: Could not load config for solrconfig.xml
    at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:991)
    at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1051)
    ... 10 more
Caused by: java.io.IOException: Can't find resource 'solrconfig.xml' in classpath or '/data/APP_NAME/releases/20130325112326/solr/collection1/conf/', cwd=/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/bundler/gems/sunspot-87e29212527f/sunspot_solr/solr
    at org.apache.solr.core.SolrResourceLoader.openResource(SolrResourceLoader.java:318)
    at org.apache.solr.core.SolrResourceLoader.openConfig(SolrResourceLoader.java:283)
    at org.apache.solr.core.Config.<init>(Config.java:103)
    at org.apache.solr.core.Config.<init>(Config.java:73)
    at org.apache.solr.core.SolrConfig.<init>(SolrConfig.java:117)
    at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:989)
    ... 11 more
Apr 1, 2013 2:38:16 AM org.apache.solr.servlet.SolrDispatchFilter init
INFO: user.dir=/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/bundler/gems/sunspot-87e29212527f/sunspot_solr/solr
Apr 1, 2013 2:38:16 AM org.apache.solr.servlet.SolrDispatchFilter init
INFO: SolrDispatchFilter.init() done

From the logs, it is clear that re-indexing is failing because of 2 reasons:

A. Unable to create core: collection1
B. Can't find resource 'solrconfig.xml' in classpath

Based on the errors above, below are the tricks I tried:

A. Checked the folder permission for solr
B. Manually created collection1 inside solr and copied solrconfig.xml inside /solr/collection1/conf.
C. copied the lib and conf folders from /data/APP_NAME/shared/bundled_gems/ruby/1.9.1/bundler/gems/sunspot-87e29212527f/sunspot_solr/solr and pasted it in APP_NAME/solr/sunspot_rails folder.

But none of the solutions worked. :-(

Kindly suggest.

[UPDATED]: My Gemfile looks like this:

gem "sunspot", "~> 1.3.3"
gem 'sunspot_rails', '~> 1.3.0'
gem 'sunspot_cell', :git => 'git://github.com/zheileman/sunspot_cell.git'
gem 'sunspot_cell_jars', :git => 'https://github.com/mrcsparker/sunspot_cell_jars.git'
gem 'sunspot_solr', :git => "git://github.com/sunspot/sunspot.git"

[UPDATED]: As of 4/4/2013: I tried with the steps below, but still no luck: Please check the details, maybe this will help you to understand if I am heading towards the right direction.

1. Stopped solr <br />
${PATH_TO_APP}: bundle exec rake sunspot:solr:stop

2. remove solr folder
${PATH_TO_APP}: rm -rf solr

3. Start solr
${PATH_TO_APP}: bundle exec rake sunspot:solr:start
--backtrace
java version "1.6.0_30"
Java(TM) SE Runtime Environment (build 1.6.0_30-b12)
Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode)
Copying /data/APP_NAME/shared/bundled_gems/ruby/1.9.1/bundler/gems/sunspot-87e29212527f/sunspot_solr/solr/solr/conf/solrconfig.xml => /data/APP_NAME/releases/20130402102051/solr/conf/solrconfig.xml
Copying /data/APP_NAME/shared/bundled_gems/ruby/1.9.1/bundler/gems/sunspot-87e29212527f/sunspot_solr/solr/solr/conf/spellings.txt => /data/APP_NAME/releases/20130402102051/solr/conf/spellings.txt
Copying /data/APP_NAME/shared/bundled_gems/ruby/1.9.1/bundler/gems/sunspot-87e29212527f/sunspot_solr/solr/solr/conf/synonyms.txt => /data/APP_NAME/releases/20130402102051/solr/conf/synonyms.txt
Copying /data/APP_NAME/shared/bundled_gems/ruby/1.9.1/bundler/gems/sunspot-87e29212527f/sunspot_solr/solr/solr/conf/elevate.xml => /data/APP_NAME/releases/20130402102051/solr/conf/elevate.xml
Copying /data/APP_NAME/shared/bundled_gems/ruby/1.9.1/bundler/gems/sunspot-87e29212527f/sunspot_solr/solr/solr/conf/admin-extra.html => /data/APP_NAME/releases/20130402102051/solr/conf/admin-extra.html
Copying /data/APP_NAME/shared/bundled_gems/ruby/1.9.1/bundler/gems/sunspot-87e29212527f/sunspot_solr/solr/solr/conf/schema.xml => /data/APP_NAME/releases/20130402102051/solr/conf/schema.xml
Copying /data/APP_NAME/shared/bundled_gems/ruby/1.9.1/bundler/gems/sunspot-87e29212527f/sunspot_solr/solr/solr/conf/protwords.txt => /data/APP_NAME/releases/20130402102051/solr/conf/protwords.txt
Copying /data/APP_NAME/shared/bundled_gems/ruby/1.9.1/bundler/gems/sunspot-87e29212527f/sunspot_solr/solr/solr/conf/scripts.conf => /data/APP_NAME/releases/20130402102051/solr/conf/scripts.conf
Copying /data/APP_NAME/shared/bundled_gems/ruby/1.9.1/bundler/gems/sunspot-87e29212527f/sunspot_solr/solr/solr/conf/stopwords.txt => /data/APP_NAME/releases/20130402102051/solr/conf/stopwords.txt
Copying /data/APP_NAME/shared/bundled_gems/ruby/1.9.1/bundler/gems/sunspot-87e29212527f/sunspot_solr/solr/solr/conf/mapping-ISOLatin1Accent.txt => /data/APP_NAME/releases/20130402102051/solr/conf/mapping-ISOLatin1Accent.txt
Copying /data/APP_NAME/shared/bundled_gems/ruby/1.9.1/bundler/gems/sunspot-87e29212527f/sunspot_solr/solr/solr/solr.xml => /data/APP_NAME/releases/20130402102051/solr
Successfully started Solr ...

4. Ensure config file is correct (solr/conf/schema.xml)
I used this link for schema.xml: http://stackoverflow.com/a/14700918/649868 and overwrite schema.xml file under {PATH_TO_APP}/solr/conf

5. Run Sunspot re-index
${PATH_TO_APP}: bundle exec rake sunspot:reindex

--backtrace
rake aborted!
RSolr::Error::Http - 404 Not Found
Error:     Not Found

Request Data: "<?xml version=\"1.0\" encoding=\"UTF-8\"?><delete><query>type:JobSeeker</query></delete>"

One thing I noticed is, when I executed bundle exec rake sunspot:solr:start command and checked the log files, I found this:

SEVERE: null:org.apache.solr.common.SolrException: Unable to create core: development
Caused by: org.apache.solr.common.SolrException: Cannot create directory: /data/APP_NAME/current/solr/data/index

But when I visited this directory index folder was there.

Interesting to see that solr creates solr.xml file in Production mode, whereas it's not there in my staging environment. Please see solr.xml (after changing it):

<?xml version="1.0" encoding="UTF-8" ?>
<solr persistent="true">
<cores adminPath="/admin/cores" host="${host:}" hostPort="${jetty.port:}">
    <core name="default"     instanceDir="/data/APP_NAME/current/solr" dataDir="/data/APP_NAME/current/solr/data"/>
    <core name="development" instanceDir="/data/APP_NAME/current/solr" dataDir="/data/APP_NAME/current/solr/data"/>
    <core name="test"        instanceDir="/data/APP_NAME/current/solr" dataDir="/data/APP_NAME/current/solr/data"/>
</cores>
</solr>

Any idea about this?

[UPDATED]: As on 5th-April-2013 I deployed a fresh build on production instance and after that, when I do:

$ RAILS_ENV=production bundle exec rake sunspot:reindex --trace

I got this:

** Invoke sunspot:reindex (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute sunspot:reindex
Skipping progress bar: for progress reporting, add gem 'progress_bar' to your Gemfile

rake aborted!
RSolr::Error::Http - 404 Not Found
Error:     Not Found

Request Data: "<?xml version=\"1.0\" encoding=\"UTF-8\"?><delete>query>type:OccupationData</query></delete>"
Backtrace: /data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rsolr-1.0.8/lib/rsolr/client.rb:230:in `adapt_response'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rsolr-1.0.8/lib/rsolr/client.rb:167:in `execute'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rsolr-1.0.8/lib/rsolr/client.rb:161:in `send_and_receive'
(eval):2:in `post'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rsolr-1.0.8/lib/rsolr/client.rb:67:in `update'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rsolr-1.0.8/lib/rsolr/client.rb:131:in `delete_by_query'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/indexer.rb:55:in `remove_all'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/session.rb:181:in `block in remove_all'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/session.rb:181:in `each'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/session.rb:181:in `remove_all'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/session_proxy/abstract_session_proxy.rb:11:in `remove_all'/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rsolr-1.0.8/lib/rsolr/client.rb:230:in `adapt_response'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rsolr-1.0.8/lib/rsolr/client.rb:167:in `execute'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rsolr-1.0.8/lib/rsolr/client.rb:161:in `send_and_receive'
(eval):2:in `post'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rsolr-1.0.8/lib/rsolr/client.rb:67:in `update'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rsolr-1.0.8/lib/rsolr/client.rb:131:in `delete_by_query'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/indexer.rb:55:in `remove_all'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/session.rb:181:in `block in remove_all'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/session.rb:181:in `each'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/session.rb:181:in `remove_all'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/session_proxy/abstract_session_proxy.rb:11:in `remove_all'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/session_proxy/retry_5xx_session_proxy.rb:17:in `method_missing'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/session_proxy/abstract_session_proxy.rb:11:in `remove_all'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot.rb:464:in `remove_all'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot_rails-1.3.3/lib/sunspot/rails/searchable.rb:178:in `solr_remove_all_from_index'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot_rails-1.3.3/lib/sunspot/rails/searchable.rb:197:in `solr_reindex'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot_rails-1.3.3/lib/sunspot/rails/tasks.rb:58:in `block (3levels) in <top (required)>'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/class_set.rb:16:in `each'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/class_set.rb:16:in `each'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot_rails-1.3.3/lib/sunspot/rails/tasks.rb:57:in `block (2levels) in <top (required)>'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/task.rb:246:in `call'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/task.rb:246:in `block in execute'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/task.rb:241:in `each'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/task.rb:241:in `execute'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/task.rb:184:in `block in invoke_with_call_chain'
/usr/lib64/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/task.rb:177:in `invoke_with_call_chain'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/task.rb:170:in `invoke'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/application.rb:143:in `invoke_task'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/application.rb:101:in `block (2 levels) in top_level'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/application.rb:101:in `each'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/application.rb:101:in `block in top_level'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/application.rb:110:in `run_with_threads'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/application.rb:95:in `top_level'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/application.rb:73:in `block in run'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/application.rb:160:in `standard_exception_handling'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/application.rb:70:in `run'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/bin/rake:33:in `<top (required)>'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/bin/rake:19:in `load'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/bin/rake:19:in `<main>'
Tasks: TOP => sunspot:reindex

Below is my model:

class OccupationData < ActiveRecord::Base
    attr_accessible :onetsoc_code, :title, :description

    searchable :auto_index => false do
        text :title
        text :description
    end
end

Tried to run all the sunspot rake tasks by specifying RAILS_ENV=production and cross checked if SOLR is running with ROOT or deploy, and got this:

/data/APP_NAME/current $ ps -ef | grep solr
deploy    2342     1  0 Apr04 ?        00:00:30 /usr/lib/jvm/sun-jdk-1.6/bin/java -Djetty.port=8983 -Dsolr.data.dir=/data/APP_NAME/releases/20130404095808/solr/data/production -Dsolr.solr.home=/data/APP_NAME/releases/20130404095808/solr -Djava.util.logging.config.file=/tmp/logging.properties20130404-2342-17k2tsp -jar start.jar
deploy    9770  9461  0 06:42 pts/0    00:00:00 grep --colour=auto solr

It tells me that SOLR is running on deploy. I am not even in a situation to provide SOLR logs, because failing sunspot:reindex is not generating any log level information as well. :-(

9th-April-2013 - [UPDATE] [SOLUTION]

After struggling a lot with re-indexing, I finally found a solution of this. The error was occurring due to wrong gem mentioned in Gemfile.

Here's the old copy of my Gemfile:

gem "sunspot", "~> 1.3.3"
gem 'sunspot_rails', '~> 1.3.0'
gem 'sunspot_cell', :git => 'git://github.com/zheileman/sunspot_cell.git'
gem 'sunspot_cell_jars', :git => 'https://github.com/mrcsparker/sunspot_cell_jars.git'
gem 'sunspot_solr', :git => "git://github.com/sunspot/sunspot.git"

When I browsed this link: https://github.com/mrcsparker/sunspot_cell_jars, I came to know that I have to use the updated sunspot_solr gem in order to keep it work.

so I am using:

gem 'sunspot_solr', :git => "https://github.com/mrcsparker/sunspot.git"

and then I created JAR files again with "rails g command" and when ran sunspot:reindex everything worked. Hope this will help someone.

Solution

Had the same problem slightly different on error logging. In my case it was like:

2013-07-18 10:26:47.750:INFO:oejw.StandardDescriptorProcessor:NO JSP Support for /solr, did not find org.apache.jasper.servlet.JspServlet
Null identity service, trying login service: null
Finding identity service: null

What I'm using is Solr 4, which has a better Admin interface, so into the Gemfile:

gem 'sunspot_solr', github: 'sunspot/sunspot', branch: 'master'
gem 'sunspot_rails', github: 'sunspot/sunspot', branch: 'master'

And that's it, no more. Just notice the github: part. I did something similar to what you did: deleted the solr directory, then,

bundle install

then,

rails generate sunspot_rails:install

and then,

bundle exec rake sunspot:solr:start

and eventually it started to work.

Next, run

bundle exec rake sunspot:reindex

One thing I have to admit, is that before the above procedure, since I have Solr 4, I replaced my xml files with these from here. Not sure whether this is critical.

Hope this helps someone.