Search code examples
amazon-web-servicespysparketlaws-glue

How can we rename the generated/output parquet file in PYSPARK or Dynamic Frames in AWS Glue?


This is a generated output parquet file in S3 from AWS Glue with PySpark, we want to give a specific name like abcd.parquet not auto-generated characters. Any help would be great. Thanks! Image


Solution

  • This is unfortunately not possible. Glue is using Spark under the hood which assigns those names to your files.

    The only thing you can do is to rename it after writing.