Search code examples
kubernetesspring-batchbatch-processingspring-cloud-dataflow

Spring Batch Restartability on Kubernetes for File Operations

I want to learn what is the proper way to reach the processed files when restarting the spring batch application on Kubernetes. Especially if the target type is file, it is being deleted together with the pod after the job failed.

We are considering to use persistent volume or backing up the created file somewhere such as DB or sftp server by implementing a listener.

Is there anyone have the experience of persistent volume usage(nfs or other solutions) for file operations. We are concerned about the performance and unexpected problems. Do you have any suggestions?

Thank you.

Solution

  • You should not rely on the ephemeral file system of a Pod for files that should persist and survive a Job (Pod) failure.

    You need to use a persistent volume for that, so that Spring Batch can find the (incomplete) output file in a restart scenario and resume writing where it left off.