Search code examples
pythonprefect

How to deploy locally multiple flows using Prefect 2.0?

I'm reading Prefect documentation and trying to understand how local deployment works. I can deploy a flow locally following the below steps.

First, I build the flow:

prefect deployment build ./log_flow.py:log_flow -n log-simple -q test

Where ./log_flow.py:log_flow are, respectively, the flow's location and entrypoint. log-simple is the name of deployment and test is the work queue

Second, I start the worker using:

prefect agent start -q 'test'

To apply the deployment, I use python running the below snippet:

from log_flow import log_flow
from prefect.deployments import Deployment

deployment = Deployment.build_from_flow(
    flow=log_flow,
    name="log-simple",
    parameters={"name": "Marvin"},
    infra_overrides={"env": {"PREFECT_LOGGING_LEVEL": "DEBUG"}},
    work_queue_name="test",
)

if __name__ == "__main__":
    deployment.apply()

Well, that works fine for a single flow. But how can I deploy several flows at once? I can repeat the above process for every flow, but it looks a bit unpractical to me since each build step generates another YAML file. I think would be more practical if my deployment generates a single YAML file for all flows.

Is there a way to deploy several flows at once in Prefect 2.0?

Solution

  • Yes, absolutely. You could loop over all flows from Python, bash or from CI/CD. Here are a couple of examples showing how you could approach it: