Search code examples
deploymentserverorchestrationprefect

Prefect: Is is possible to access storage like NAS with multiple machines in Prefect?


I have set up a Prefect backend server on a remote machine. I was able to connect local agents from different other machines to the server by modifying the config.toml in the .prefect folder:

[server]
endpoint = "http://server_ip:port/graphql"

  [server.ui]
    apollo_url = "http://server_ip:port/graphql"

As it stands, I can create a local agent on each machine, register flows and run them on the respective machines. Now I would like to have a central computer where I can develop and register my flows. Unfortunately, when I run a flow on Machine B, registered on Machine A, I get a "Module not Found" error message. I have read that the error comes from machines only looking for the flows in their local storage. Without using Git, GCS, etc., is it possible to use, for example, a NAS where all flows are stored and which all machines can use to access the flows? if so, how must flows, agents, and storage be configured? Unfortunately, I have not found any good documentation on this. Many applications use Docker agents and have similar problems, or use remote storage directly.


Solution

  • I was able to find a solution to my answer. The prerequisite is shared storage (e.g. a NAS), which is accessible on all machines under the same path. In this storage, the flows are stored in the form of .py files. Flows and used local Agents do not need any special preparations. I simply registered my flows with

    prefect register --project "PREFECT_PROJECT_NAME" --path "PATH_TO_.py"
    

    in CLI. I was able to deploy all my flows from machine A and execute them from/schedule them on any other machine