Search code examples
rayrllib

open file inside Ray

I'm using RAY, and created a custom env. However, the custom env needs to open a file, and ray creates workers in a different location. Therefore, I can't access the file.

when printing the worker location I'm getting : city_v1/DQN/DQN_CityFlows_074cc_00000_0_2022-04-20_13-09-56

file exists in:

examples/1x1/file.json
Solution

  • As you mentioned, Ray worker processes may run in different locations, even different machines.

    The recommended solution for making files available to workers is to explicitely specify them via runtime environments. https://docs.ray.io/en/latest/ray-core/handling-dependencies.html

    In your case you may want to do

    runtime_env = {"working_dir": "./"}

    to copy the local working directory.

    Note that the these directories are sent from the head node to all the workers, which could be a slow process if your directory is large (in which case, you should try to minimize the number of files that need to be uploaded).