Search code examples
airflowamazon-sqsairflow-2.xmwaa

Polling multiple SQS messages using Airflow SQSSensor

I am using this SQSSensoe settings to poll messages

    fetch_sqs_message = SQSSensor(
        task_id="...",
        sqs_queue="...",
        aws_conn_id="aws_default",
        max_messages=10,
        wait_time_seconds=30,
        poke_interval=60,
        timeout=300,
        dag=dag
    )

I would assume everytime it polls it should poll up to 10 messages. Which my queue has around 5 when I tested this. But each time I trigger the dag, it only polls 1 message at a time, which I found out from the SQS message count. Why is it doing this? How can I to get it poll as much messages as possible?

Solution

  • Recently, a new feature has been added to SQSSensor so that the sensor can polls SQS multiple times instead of only once.

    You can check out this merged PR

    For example, if num_batches is set to 3, SQSSensor will poll the queue 3 times before returning the results.

    Disclaimer: I contributed to this feature.