Search code examples
scalacsvamazon-s3

How to upload a CSV file to a given s3 bucket in scala?

I want to upload a CSV file to a given s3 bucket, this CSV file is transfered from a dataframe by using the df.csv(path). For now, I saved the file locally, I'm wondering if there is a way to upload that file to a S3 bucket if given a s3 bucket name?

Solution

  • Something like this should work

    dataframe
  .write
  .option("header","true")
  .csv("s3a://bucket/path/file.csv")

    More examples on https://sparkbyexamples.com/spark/write-read-csv-file-from-s3-into-dataframe/