Search code examples
kubernetesnodesgcloudpool

gcloud kubernetes node pool: high priority for preemptible VM nodes possible

i need a solution for that:

i have 2 node pools in gcloud kubernetes, first is preemptible and autoscaling, second is only autoscaling.

Jobs should be started on the first one ( with preemptible VMs ), but when no resources on the first pool are available Jobs should be started on the second one.

How can i realize that, maybe with Taints and Tolerations?

Solution

  • I don't think you can get exactly what you want with the Cluster Autoscaler but I'll hopefully give you a couple options and pointers to further explore.

    • The Cluster Autoscaler has the notion of Expanders which can help determine which node group to scale up when a scaling event happens. The price expander seems to be close to what you want, but based on the description of it, it doesn't look like it has support for preemptible VMs yet. You could explore that further, and possibly submit a feature request to add support for preemptible node pools.

    • When choosing a mixture of preemptible and non-preemptible nodes, whenever there is a stock out on GCP and preemptible nodes are not available, it's very likely that non-preemptible nodes will also not be available. In that case you may find yourself with a small number of non-preemptible nodes in the cluster and without the ability to create new ones.

      It may be a better idea to have a fixed minimum size of non-preemptible nodes, and auto-scale a preemptible node pool on top of that using the Cluster Autoscaler.