I'm starting to build out my glue job however I'd like to add a custom logger (and other files) to it so that it's cleaner code. How do i break up my files and re import them into the glue job. Say a main.py
, utils.py
and logging.py
?
The answer is included here:
In terraform you zip up your modules and an __init__.py
file and deposit them in s3. You then add their location in s3 in the --extra-py-files
argument.
In terraform you have an argument available to you:
"--extra-py-files" = "s3://${aws_s3_object.foo.bucket}/${aws_s3_object.foo.key},s3://${aws_s3_object.barr.bucket}/${aws_s3_object.barr.key}"