I have a cassandra cluster of 18 prod nodes in 1 DC1 and 12 backup nodes in DC2 data center, few days before all backup nodes went down and crossed gc_grace period. now i am trying to make all Backup nodes up so have removed all data from backup nodes and trying to rebuild but it is getting halted with FileNotFoundException:.
Rebuild commands is : nohup nodetool rebuild DC1 &
(DC1 is prod data center )
Error in nohup.out file :
Error while rebuilding node: Stream failed
-- StackTrace --
java.lang.RuntimeException: Error while rebuilding node: Stream failed
at org.apache.cassandra.service.StorageService.rebuild(StorageService.java:1076)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Error in system.log:
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.FileNotFoundException: /data1/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-tmp-ka-62-Data.db (No such file or directory)
at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-16.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-16.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-16.0.jar:na]
at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:372) ~[apache-cassandra-2.1.16.jar:2.1.16]
... 12 common frames omitted
Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: /data1/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-tmp-ka-62-Data.db (No such file or directory)
at org.apache.cassandra.io.util.SequentialWriter.<init>(SequentialWriter.java:82) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.io.compress.CompressedSequentialWriter.<init>(CompressedSequentialWriter.java:67) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:124) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:130) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.db.Memtable$FlushRunnable.createFlushWriter(Memtable.java:414) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:351) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:335) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.16.jar:2.1.16]
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.jar:na]
at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1134) ~[apache-cassandra-2.1.16.jar:2.1.16]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_79]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_79]
... 5 common frames omitted
Caused by: java.io.FileNotFoundException: /data1/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-tmp-ka-62-Data.db (No such file or directory)
Your problem is not the FileNotFound exception. It's the fact that you are streaming system tables. System tables will be created on the node locally when it's started up. All the data should be streamed EXCEPT the system tables data. /data1/cassandra/data/system/
Which Cassandra version are you using?
If you didn't change anything that forced Cassandra to stream the system tables, I would say this is a bug.