After the ETL Job is done, What is the best way to call stored procedure in AWS Glue script?
I am using PySpark to fetch the data from S3 and storing in staging table. After this process, need to call a stored procedure. This stored procedure loads data from the staging table into the appropriate MDS tables.
If I have to call a Stored Procedure after ETL Job is done, what is the best way? If I consider AWS lambda, is there any way that lambda can be notified after the ETL.
You can use the boto sdk to fire off your lambda at the end of your glue ETL job. Since you are writing to a staging table, you will also need to have a NAT Gateway on the connection's subnet for the boto call to work. The example at this link uses boto to fire an ECS task, which would be a similar process: https://github.com/full360/glue-sneaql-demo/blob/master/Glue-Job-Script.