Search code examples
ruby-on-railsherokuunicornthinking-sphinx

Flying Sphinx Error on Heroku - FATAL: Failed to lock.. Resource temporarily unavailable

I have a Rails (3.2) app running on Heroku (Unicorn server). I am also using Flying Sphinx for searching.

When I attempt to index or rebuild, I use

heroky run bundle exec flying-sphinx configure
heroku run bundle exec flying-sphinx index (or rebuild)

Some of my indices will output the following error during the process:

FATAL: failed to lock /mnt/local/flying-sphinx/4188b791429cbac91/indexes/user_delta.spl: Resource temporarily unavailable, will not index. Try --rotate option.

enter image description here

Even after trying config, rebuild, and/or index I'm getting this error for some indices.

In addition, the Flying Sphinx status page on Heroku states that Sphinx is not running. I'm not sure if this is related. Here is a screenshot of the status page:

enter image description here

Please let me know if I can supply anything else.

EDIT

I just noticed a "No Space Left on Device" error in the output of my rebuild as well, not sure if it's also related:

ERROR: index 'search_proxy_core': /mnt/local/flying-sphinx/4188b791429cbac91/indexes/search_proxy_core.tmp.spd: write error: No space left on device.

enter image description here

Solution

  • This was a combination of a few things:

    • Using delta indices but not running a full index regularly, so the delta indices got really, really big and slow to process.
    • Not having a database index on each of the foreign keys used by Sphinx's SQL query.
    • A corruption of data in one of the index files, which made Sphinx crash on every query to them, and couldn't be fixed without reprocessing the indices.

    We got the database indices in place, ran a rebuild, and then everything was working smoothly again.