Search code examples
pythonprefect

How can I run a flow several times using different parameters using Prefect?

How can I run a flow several times using different parameters? I'm trying to use create_flow_run.map like this:

dump_to_gcs_flow = create_flow_run.map(
    flow_name=unmapped(utils_constants.FLOW_DUMP_TO_GCS_NAME.value),
    project_name=unmapped(constants.PREFECT_DEFAULT_PROJECT.value),
    parameters=tables_to_zip,
    labels=unmapped(current_flow_labels),
    run_name=unmapped("Dump to GCS"),
)

except for parameters all the other arguments are constants, so I use unmapped on them. tables_to_zip is a list of dictionaries containing the parameters values for each table to be zip. However, this didn't work. I'm currently receiving the (somewhat crypt) error:

prefect.exceptions.ClientError: [{'message': 'parsing UUID failed, expected String, but encountered Array', 'locations': [{'line': 2, 'column': 5}], 'path': ['flow_run'], 'extensions': {'path': '$.selectionSet.flow_run.args.where.id._eq', 'code': 'parse-failed', 'exception': {'message': 'parsing UUID failed, expected String, but encountered Array'}}}]

what am I doing wrong here?

Solution

  • It would need to be a list of dictionaries with the key being parameter name - this Discourse topic discusses this problem in much more depth: https://discourse.prefect.io/t/how-to-map-over-flows-with-various-parameter-values/365/2

    LMK, if you still have issues implementing this pattern after reading this

    (btw Prefect 2 is the default version now and makes this much easier: https://docs.prefect.io)