How do i write log data into file in AWS Glue?

Question

I am using AWS Glue with pyspark for my code.I have enabled contious log in AWS Glue.I am not sure where will the contious log store the log data. I am using the below code,

logger = logging.getLogger('My-code')
logger.setLevel(logging.INFO)
logger.info("Hello")

How do i redirect this data (i.e. logger.info("Hello")) into a file?

Answer 1

When you enable continuous monitoring you can watch the complete logs of your job in the cloudwatch.

To view the logs you need to enable cloudwatch for glue jobs then you can see all the logs and also enable the retention of those job logs.

And if you want glue jobs to generate the logs directly to s3, you cannot do it using file protocol or logger package instead you can leverage the rdd approach which is a bit hacky.

Create a list with the log message you want and in the end convert that list to rdd using parallelize then save that rdd to s3 path.

But cloudwatch is more recommended for getting the logs.