Search code examples
apache-sparkcassandraamazon-emrspark-cassandra-connectordatastax-astra

Get correct path for sparkdid.cassandra.connection.config.cloud.path in AWS EMR


I checked the entire web. but could not found the solution.

I am trying to connect astra cassandra using bundle in AWS EMR its able to download the bundle file but not loading it.

spark.conf.set("sparkdid.cassandra.connection.config.cloud.path", "secure-connect-app.zip")

this is how I am giving path I know this is wrong path since its not loading the correct config returning connection refused with localhost.

I don't what is the correct path in EMR.


Solution

  • If you look into documentation, then you see that the file could be either specified as URL of file on S3, or you can use —files option when submitting with spark-submit or spark-shell , then it will be available as just a file name, like you’re doing