Error while overwriting Cassandra table from PySpark

Question

I am attempting to OVERWRITE data in cassandra with a PySpark dataframe. I get this error: keyword can't be an expression

I am able to append the data by df.write.format("org.apache.spark.sql.cassandra").options(keyspace="ks",table="testtable").mode("append").save()

However, overwriting is throwing error df.write.format("org.apache.spark.sql.cassandra").options(keyspace="ks",table="testtable", confirm.truncate="true").mode("overwrite").save()

Error: keyword can't be an expression

Answer 1

I found the solution.

df.write.format("org.apache.spark.sql.cassandra")
.mode("overwrite").option("confirm.truncate","true")
.options(keyspace="ks",table="testtable")
.save()