Search code examples
kubernetesgoogle-kubernetes-engineautoscaling

GKE autoscaling

I have three node pools in my cluster each of them have autoscaling enabled to go from 1-100 nodes. Minimum nodes are 1 for all. I am having something weird happening with autoscaling.

Scale down works fine for all pools. Scale up seems to create a new node pool instead of scaling the corresponding node pools but since that node pool is missing the labels we need nothing gets scheduled and eventually gets destroyed.

I swear I am missing some information to enable it to scale the right node-pool, Any suggestions on what to look at and where to change? I do not use/have GCE auto-scaling

Solution

  • GKE starts new nodes only from user-created node pools. With Node auto-provisioning enabled, the cluster autoscaler can extend node pools automatically. Node auto-provisioning automatically manages a set of node pools on the user's behalf. Since the nodepools here don't have labels, Node auto-provisioning is creating the new nodepools with required labels.

    Node auto-provisioning might create node pools with labels and taints if all the following conditions are met:

    • A pending Pod requires a node with a specific label key and value.
    • The Pod has a toleration for a taint with the same key.
    • The toleration is for the NoSchedule effect, NoExecute effect, or all effects.

    You can update node labels and node taints for the existing nodepools by disabling the autoscaling on the node pool. After the labels or taints are updated, re-enable autoscaling.

    To update node labels for a existing node pool, use the following command:

    gcloud beta container node-pools update NODEPOOL_NAME \
        --node-labels=[NODE_LABEL,...] \
        [--cluster=CLUSTER_NAME] [--region=REGION | --zone=ZONE]
        [GCLOUD_WIDE_FLAG …]

    Note: The cluster autoscaler is automatically enabled when using node auto-provisioning.

    Refer to Node auto-provisioning for more information.