run a job after multiple dependent jobs completed in aws glue

Question

I have multiple Glue jobs J1, J2, J3 which are triggered via AWS Lamda and Lambda is triggered via the files coming on S3 buckets.

Sequence is below:-

S3(Bucket B1)--file added--> L1(Lambda triggered)-->J1(Glue Jobs)-->insert data T1 table( RedSfhift)

S3(Bucket B2)--file added--> L2(Lambda triggered)-->J2(Glue Jobs)-->insert data T2 table( RedSfhift)

S3(Bucket B3)--file added--> L3(Lambda triggered)-->J3(Glue Jobs)-->insert data T3 table( RedSfhift)

Now, After completing all those 3 jobs I want to trigger a separate Glue Job(JFinal), which will process the data from T1, T2 and T3 and does some calculation then insert in TFinal Table.

Now how we can trigger the job JFinal. The J1, J2, J3 jobs are independent and the execution sequence can be different.

Answer 1

AWS Step functions would be a good solution here. You have to configure your step machine where your 3 jobs can be run independently, still the JFinal will wait until all the other jobs are completed. Refer: Step Functions Getting Started