Search code examples
pythonamazon-web-servicesaws-glue

How do I break up my glue code into multiple files?


I'm starting to build out my glue job however I'd like to add a custom logger (and other files) to it so that it's cleaner code. How do i break up my files and re import them into the glue job. Say a main.py, utils.py and logging.py?


Solution

  • The answer is included here: In terraform you zip up your modules and an __init__.py file and deposit them in s3. You then add their location in s3 in the --extra-py-files argument. In terraform you have an argument available to you:

    "--extra-py-files"            = "s3://${aws_s3_object.foo.bucket}/${aws_s3_object.foo.key},s3://${aws_s3_object.barr.bucket}/${aws_s3_object.barr.key}"