Search code examples
pythonorchestrationprefect

prefect warning message regarding task performance for longer running tasks

i have been getting the below warning message with tasks with larger run time most of my tasks take more than 10 seconds

it says

Try wrapping large task parameters with `prefect.utilities.annotations.quote` for increased performance

how can i do that for better performance

Below is the actual waring

Task parameter introspection took 3676.513 seconds , exceeding PREFECT_TASK_INTROSPECTION_WARN_THRESHOLD of 10.0. Try wrapping large task parameters with prefect.utilities.annotations.quote for increased performance, e.g. my_task(quote(param)). To disable this message set PREFECT_TASK_INTROSPECTION_WARN_THRESHOLD=0.

Solution

  • The official docs explain it quite well. Simply import the quote function with:

    from prefect.utilities.annotations import quote

    Then, within your flow use:

    def my_flow():
    # ...
    large_item = task_1()
    result = task_2(quote(large_item))
    # ...

    instead of directly passing large_item to the next task.