Search code examples
databasecassandranodetool

Cassandra nodes are not equal

We have two nodes. Node1 was down for a long time. During this time Node2 raised 1 TB capacity when Node1 has 100 GB.

We tried to repair Node1 with nodetool repair but nothing changed. After that we started nodetool repair in Node2, it took 5 days to compaction but nothing changed either.

Actual status here: 

Datacenter: dc1
===============
Status=Up/Down

|/ State=Normal/Leaving/Joining/Moving
--  Address       Load       Tokens       Owns (effective)  Host ID    Rack

UN  172.x.y.z     149.46 GB  256          100.0%            xxx  rack1
UN  172.x.y.k     1.04 TB    256          100.0%            xyz  rack1

Nodes are in AWS. What should we do?

Solution

  • We had a solution but don't know how it happened. Here what we did:

    • Took snapshot of both servers in case of data loss over AWS
    • We detached Volume where data is stored. (for instance 172.x.y.k)
    • We formatted instance and loaded Cassandra with new version. Then we attached Volume where data is stored.
    • Finally we started nodetool repair --full in terminal and it took 4 days.

    Now our nodes are equal.