Search code examples
pythonazure-devopssnowflake-cloud-data-platformwebhooksaws-step-functions

How to stop build agent from waiting in the pipeline process in Azure devops

I have been able to put together a pipeline that trigger a step function in AWS through a python script which estimated to run for 7 hours and the next job in the pipeline depends on the successful completion of the step function. The next job is also a python script that generate tables in snowflakes for 30 hours. This is working as expected, but there is need for improvement. The build agent in the Azure devops pipeline is waiting for 30 hours which is not good. I am looking at efficient way to trigger the python script that call the snowflakes and exit the pipeline to stop the build agent from waiting.

What are the options available?

  1. Is it possible to use lambda to call this snowflakes and exit the pipeline and how would the pipeline get notification if anything goes wrong ?
  2. There is suggestion to use webbook and have not used webhook before, is there any pointer to use this?
Solution
    1. Is it possible to use lambda to call this snowflakes and exit the pipeline and how would the pipeline get notification if anything goes wrong ?

    Lambda Functions timeout after 15 minutes. So, just putting your script into Lambda Function won't work.

    1. There is suggestion to use webbook and have not used webhook before, is there any pointer to use this?

    Webhook is just a way of invoking another service via HTTP call. When you call Lambda Function or Step Function via API Gateway, you can consider it as a webhook. There might be also a webhook on Snowflake side to perform some actions. This doesn't solve the script execution part though.

    I suggest to use Step Function with Lambda Functions in the following way:

    1. Extract SQL queries from your Python, let the first step of Step Function call one Lambda function per query.
    2. Parallelize query execution of Lambda Functions.
    3. Call your Step Function from your pipeline, don't wait for the execution to finish.
    4. Use Amazon SNS to send an email to notify when Step Function has finished.

    You can see your individual execution results in CloudWatch – useful for debugging and statistics.

    This will work if each SQL query runs under 15 minutes. If not, you can consider running the script in container environment, EC2 Spot instance, or AWS Batch.