Search code examples
rapache-sparksparkr

Sparkr write DF as file csv/txt

Hi I'm working on sparkR in yarn mode.

I need to write a sparkr df to a csv/txt file.

I saw that there is write.df but it writes parquet files.

I tried to do this things 

RdataFrame<-collect(SparkRDF)
write.table(RdataFrame, ..)

But I got many WARN and some ERROR on contextCleaner.

Is there any way ?

Solution

  • Spark 2.0+

    You can use write.text function:

    Save the content of the SparkDataFrame in a text file at the specified path. The SparkDataFrame must have only one column of string type with the name "value". Each row becomes a new line in the output file.

    write.text(df, path)

    or write.df with built-in SparkR csv writer:

    write.df(df, path, source="csv")

    Spark 1.x

    You can use spark-csv package:

    write.df(SparkRDF, "foo.csv", "com.databricks.spark.csv", ...)

    It can be added for example with packages argument to SparkR / spark-submit:

    sparkR --packages com.databricks:spark-csv_2.10:1.3.0 # For Scala 2.10
sparkR --packages com.databricks:spark-csv_2.11:1.3.0 # For Scala 2.11

    For other options see the official documentation