Trouble with formatting when writing a csv to s3 with s3fs

Question

I'm pushing a dataframe to an s3 bucket using s3fs with the following code:

s3fs = s3fs.S3FileSystem(anon=False)

with s3fs.open(f"bucket-name/csv-name.csv",'w') as f:
      my_df.to_csv(f)

The action is completed successfully, but the csv has every other row empty:

I'm sure this is not an issue with the dataframe since I've also tried to push the csv to s3 with a different method and the csv was properly formatted.

The code for it:

s3_res.Object(bucket_name, s3_object_name).put(Body=csv_buffer.getvalue())

Is there a setting I can use to fix or mitigate this?

Answer 1

It seems the s3fs package adds /r/n to the end of each line. Adding newline=" to the .open() method should solve it.

with s3fs.open(f"bucket-name/csv-name.csv",'w',newline='') as f: my_df.to_csv(f)