Search code examples
apache-sparkkubernetespysparkspark-operator

PVC-oriented executor pod allocation with spark-on-k8s-operator

I see in the documentation that there is an option of PVC-oriented executor pod allocation that from my understanding should help with spot instances decommission. I couldn't find how to use it with the spark-on-k8s-operator.

Is it even possible?

Solution

  • It is accualy not really a problem and the configuration can be added under sparkConf.

      sparkConf:
    spark.kubernetes.driver.ownPersistentVolumeClaim: "true"
    spark.kubernetes.driver.reusePersistentVolumeClaim: "true"
    spark.kubernetes.driver.waitToReusePersistentVolumeClaim: "true"