ElasticSearch indices become empty randomly, but most of times, it occurs after deploying the application built with Rails.
Here some info about ElastiSearch:
curl -XGET 'http://localhost:9200/_nodes?pretty'
curl -XGET 'http://localhost:9200/_cluster/health?pretty'
curl 'localhost:9200/_cat/indices?v'
Recurring stack trace in /var/log/elasticsearch/elasticsearch.log:
[2014-11-21 16:46:05,204][WARN ][cluster.service ] [Joe Cartelli] failed to reconnect to node [Joe Cartelli][AvwiuMNRRUywfTyii7gI0A][ip-172-31-19-137][inet[/172.31.19.137:9300]]
org.elasticsearch.transport.ConnectTransportException: [Joe Cartelli][inet[/172.31.19.137:9300]] connect_timeout[30s]
at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:727)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:656)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:624)
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:146)
at org.elasticsearch.cluster.service.InternalClusterService$ReconnectToNodes.run(InternalClusterService.java:518)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.net.ConnectException: Connection refused: /172.31.19.137:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
Obs: The 172.31.19.137 IP address is the private IP of EC2 instance on AWS.
EDIT:
The erros stills occurs and I had new information about the issue.
After 'data gone', I looked at elasticsearch.log and Stack Trace was printed many times. I don't know if this error happens before or after the 'data gone'. Note that the index name is products_production_20141130122307375.
Running the command to list all Elasticsearch indexes curl 'localhost:9200/_cat/indices?v', printed the products index but with different number: products_production_20141XXXXXXXXXX (I didn't wrote down the exact number).
So, why the name of indexes weren't same?
It's possible, for some strange reason for me, Rails/Searchkick recreate the index and lose the data?
EDIT 2
The problem occurs after ElasticSearch log this:
[2014-12-02 00:47:10,698][INFO ][cluster.metadata ] [Tyr] [products_production_20141202004040313] update_mapping [product] (dynamic)
[2014-12-02 00:58:02,173][INFO ][cluster.metadata ] [Tyr] [products_production_20141202005801257] creating index, cause [api], shards [5]/[1], mappings [_default_]
[2014-12-02 00:58:46,064][INFO ][cluster.metadata ] [Tyr] [products_production_20141202005801257] update_mapping [product] (dynamic)
[2014-12-02 00:58:48,371][INFO ][cluster.metadata ] [Tyr] [products_production_20141202004040313] deleting index
It looks like that searchkick recreates the index and loses the products of older index.
Maybe this issue could be related to your problem: