Search code examples
kuberneteskubernetes-pod

Automatically transfer files between containers using Kubernetes

I want to make a container that is able to transfer files between itself and other containers on the cluster. I have multiple containers that are responsible for executing a task, and they are waiting to get an input file to do so. I want a separate container to be responsible for handling files before and after the task is executed by the other containers. As an example:

  1. have all files on the file manager container.
  2. let the file manager container automatically copy a file to a task executing container.
  3. let task executing container run the task.
  4. transfer the output of the task executing container to the file manager container.

And i want to do this automatically, so that for example 400 input files can be processed to output files in this way. What would be the best way to realise such a process with kubernetes? Where should I start?

Solution

  • A simple approach would be to set up the NFS or use the File system like AWS EFS or so.

    You can mount the File system or NFS directly to POD which will be in ReadWriteMany access method.

    ReadWriteMany - Multiple POD can access the single file system.

    If you don't want to use the Managed service like EFS or so you can also set up the file system on K8s checkout the MinIO : https://min.io/

    All files will be saved in the File system and as per POD requirement, it can simply access it from the file system.

    You can create different directories to separate the outputs.

    If you want only read operation, meaning all PODs can read the files only you can also set up the ReadOnlyMany access mode.

    If you are GCP you can checkout this nice document : https://cloud.google.com/filestore/docs/accessing-fileshares