Search code examples
pythonapache-sparkpysparkaws-glue

How to create a text file with windows line ending (CRLF) using pyspark?

I have a dynamic frame created with the required data. I'm using pyspark in aws glue job. I'm creating a file as below

df_final_data.coalesce.write.format("text").option("mode", "append").save(var_bucket)

But this is creating a file with LF line endings.

How can I create a file with CRLF line endings or is there a way to replace these LF to CRLF after the file is created?

Solution

  • Try using the lineSep parameter:

    df_final_data.coalesce.write.format("text").option("lineSep", "\r\n").option("mode", "append").save(var_bucket)