Server Errors While Writing With Python Cassandra Driver

code=1000 [Unavailable exception] message="Cannot achieve consistency level ONE" info={'required_replicas': 1, 'alive_replicas': 0, 'consistency': 'ONE'}
code=1100 [Coordinator node timed out waiting for replica nodes' responses] message="Operation timed out - received only 0 responses." info={'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'}
code=1200 [Coordinator node timed out waiting for replica nodes' responses] message="Operation timed out - received only 0 responses." info={'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'}

I am inserting into Cassandra Cassandra 2.0.13(single node for testing) by python cassandra-driver version 2.6

The following are my keyspace and table definitions:

CREATE KEYSPACE test_keyspace WITH replication = {   'class': 'SimpleStrategy',   'replication_factor': '1' };


 CREATE TABLE test_table (
  key text PRIMARY KEY,
  column1 text,
  ...,
  column17 text
 ) WITH COMPACT STORAGE AND
  bloom_filter_fp_chance=0.010000 AND
  caching='KEYS_ONLY' AND
  comment='' AND
  dclocal_read_repair_chance=0.000000 AND
  gc_grace_seconds=864000 AND
  read_repair_chance=0.100000 AND
  replicate_on_write='true' AND
  populate_io_cache_on_flush='false' AND
  compaction={'class': 'SizeTieredCompactionStrategy'} AND
  compression={'sstable_compression': 'SnappyCompressor'};

What I tried:

1) multiprocessing(protocol version set to 1) each process has its own cluster, session(default_timeout set to 30.0)

def get_cassandra_session():
    """creates cluster and gets the session base on key space"""
    # be aware that session cannot be shared between threads/processes
    # or it will raise OperationTimedOut Exception
    if CLUSTER_HOST2:
        cluster = cassandra.cluster.Cluster([CLUSTER_HOST1, CLUSTER_HOST2])
    else:
        # if only one address is available, we have to use older protocol version
        cluster = cassandra.cluster.Cluster([CLUSTER_HOST1], protocol_version=1)

    session = cluster.connect(KEY_SPACE)
    session.default_timeout = 30.0
    return session

2) batch insert (protocol version set to 2 because BatchStatement is enabled on Cassandra 2.X)

def batch_insert(session, batch_queue, batch):
    try:
        insert_user = session.prepare("INSERT INTO " + db.TABLE + " (" + db.COLUMN1 + "," + db.COLUMN2 + "," + db.COLUMN3 +
                                      "," + db.COLUMN4 + ") VALUES (?,?,?,?)")
        while batch_queue.qsize() > 0:
            '''batch queue size is 1000'''
            row_tuple = batch_queue.get()
            batch.add(insert_user, row_tuple)

        session.execute(batch)
    except Exception as e:
        logger.error("batch insert fail.... %s", e)

the above function is invoked by:

 batch = BatchStatement(consistency_level=ConsistencyLevel.ONE)
 batch_insert(session, batch_queue, batch)

tuples are stored in batch_queue.

3) synchronizing execution Several days ago I post another question Cassandra update fails , cassandra was complaining about TimeOut issue. I was using synchronize execution for updating.

Can anyone help, is this my code issue or python cassandra-driver issue or Cassandra itself ?

Thanks a million!

Solution

If your question is about those errors at the top, those are server-side error responses.

The first says that the coordinator you contacted cannot satisfy the request at CL.ONE, with the nodes it believes are alive. This can happen if all replicas are down (more likely with a low replication factor).

The other two errors are timeouts, where the coordinator didn't get responses from 'live' nodes in a time configured in the cassandra.yaml.

All of these indicate that the cluster you're connected to is not healthy. This could be because it is overwhelmed (high GC pauses), or experiencing network issues. Check the server logs for clues.