Search code examples
cassandrabigdatacqldata-migration

How to Copy/Move data between columnfamilies with same structure in cassandra


I'm looking for a method to copy/move data in a column family to another with same structure, in same keyspace.

I had already tried the COPY command to import as csv & export the same to destination column family.

Since the data is large, I'm getting timeouts while using COPY.

One method is to write an application to copy data using a client api and add to the other column family.

Is there any tool to copy/move data between different column family? Or any other way?


Solution

  • I would recommend using Spark for this kind of bulk migration. It's also a useful tool for general maintenance of C*.

    https://github.com/datastax/spark-cassandra-connector

    With spark the command

    sc.cassandraTable("ks1","table").saveToCassandra("ks2","table") 
    

    you would move your tables.

    If you aren't interested in Spark I think a custom java program or Brian Hess's Bulkloader tool would be useful

    https://github.com/brianmhess/cassandra-loader