How to Connect to RDS Instance from AWS Glue Python Shell?

Question

I am trying to access RDS Instance from AWS Glue, I have a few python scripts running in EC2 instances and I currently use PYODBC to connect, but while trying to schedule jobs for glue, I cannot import PYODBC as it is not natively supported by AWS Glue, not sure how drivers will work in glue shell as well.

Answer 1

From: Introducing Python Shell Jobs in AWS Glue announcement:

Python shell jobs in AWS Glue support scripts that are compatible with Python 2.7 and come pre-loaded with libraries such as the Boto3, NumPy, SciPy, pandas, and others.

The module list doesn't include pyodbc module, and it cannot be provided as custom .egg file because it depends on libodbc.so.2 and pyodbc.so libraries.

I think you have 2 options:

1. Create a jdbc connection to your DB from Glue's console, and use Glue's internal methods to query it. This will require code changes of course.
2. Use Lambda function instead. You'll need to pack pyodbc and the required libs along with your code in a zip file. Someone has already compiled those libs for AWS Lambda, see here.

Hope it helps