How to rename output files written by aws glue script to a s3 location? using pyspark

Question

I am looking to rename the output files written to s3 using aws glue in pyspark.

If there's a code to refer to renaming files in s3 after the glue job run, that would be really helpful

Answer 1

This is unfortunately not possible. Glue is using Spark under the hood which assigns those names to your files.

The only thing you can do is to rename it after writing.