Search code examples
dockercassandrafleetdatabasenosql

Cassandra fails to initialize with error "Cannot add table 'role_members' to non existing keyspace 'system_auth'"

I am running a Cassandra cluster in Docker containers using fleet for management. I am able to get the cluster up and running, but if I bring the units down with fleet and then back up again, the containers fail. The Cassandra logs has this entry on the second start.

Cannot add table 'role_members' to non existing keyspace 'system_auth'.
Fatal configuration error; unable to start server.  See log for stacktrace.
INFO  20:59:34 InetAddress /172.17.8.102 is now DOWN
ERROR 20:59:34 Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: Cannot add table 'role_members' to non existing keyspace 'system_auth'.
        at org.apache.cassandra.service.MigrationManager.announceNewColumnFamily(MigrationManager.java:284) ~[apache-cassandra-2.2.0.jar:2.2.0]
        at org.apache.cassandra.service.MigrationManager.announceNewColumnFamily(MigrationManager.java:275) ~[apache-cassandra-2.2.0.jar:2.2.0]
        at org.apache.cassandra.service.StorageService.maybeAddTable(StorageService.java:1046) ~[apache-cassandra-2.2.0.jar:2.2.0]
        at org.apache.cassandra.service.StorageService.doAuthSetup(StorageService.java:1034) ~[apache-cassandra-2.2.0.jar:2.2.0]
        at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:967) ~[apache-cassandra-2.2.0.jar:2.2.0]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:698) ~[apache-cassandra-2.2.0.jar:2.2.0]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:581) ~[apache-cassandra-2.2.0.jar:2.2.0]
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:291) [apache-cassandra-2.2.0.jar:2.2.0]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:481) [apache-cassandra-2.2.0.jar:2.2.0]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:588) [apache-cassandra-2.2.0.jar:2.2.0]

I can't find any information on this particular error, and I really have no idea why it's happening. The closest information I can find is that the system_auth table needs to be configured specially if you are not using the default AllowAllAuthenticator, but I am using this. I haven't changed it in the cassandra.yaml file.

Does anyone know why this might be happening?

Solution

  • This turned out to be a rather unique configuration issue I had. I was mapping /var/lib/cassandra on the host to /var/lib/cassandra inside my docker container. But I was also inadvertently mapping /var/lib/cassandra/data to an auto-generated Docker directory on the host. As such when I stopped and restarted the containers, the data directory would disappear and Cassandra would fail as it tried to recreate data from the commitlog directory.