Search code examples
kubernetesnextflow

Running nextflow as a job vs. a pod in kubernetes

I'm new to nextflow. We would like to build our workflow using nextflow and have nextflow deploy the workflow to a large mulit-institution Kubernetes cluster that we use.

In this cluster we don't have admin permission, we have a namespace we work in. Also, pods in our cluster have limited resources, but jobs have unlimited resources.

In looking at the documentation for nextflow + kubernetes and it says that the workflow runs under a Kubernetes pod, which raises red flags for me because of the limitation on pods in our cluster.

Is there a way to execute nextflow workflows as a kubernetes jobs instead of a pod? What are my options in this area?

Solution

  • There might be a new feature with nextflow 22.04. Quoting Ben Sherman's nextflow k8s best practices page:

    In Nextflow v22.04 and later, the k8s executor can be configured to use Jobs instead of Pods directly.