Updating the memory for each worker node in OKE k8s cluster node pool using terraform is not working

Question

I have an OKE k8s cluster in oracle cloud. It has a node pool pool2 with only one worker node. This pool was created using terraform.

Now I'm trying to update the memory and cpu for each worker node in the nodepool. This is the terraform script that I'm using to create the node pool:

resource "oci_containerengine_node_pool" "oke-node-pool" {

    # Required variables
    compartment_id = var.compartment_ocid
    cluster_id = var.oke_cluster_ocid
    kubernetes_version = var.node_pool_kubernetes_version
    name = var.node_pool_name
    node_shape = var.node_pool_node_shape
    # subnet_ids = var.node_pool_subnet_ids

    # Specify the source to use to launch nodes in the node pool. Currently, image is the only supported source.
    node_source_details {
         image_id = var.node_image_ocid[var.region]
         source_type = "IMAGE"
    }

    # The configuration of nodes in the node pool.
    node_config_details {
        placement_configs {
            availability_domain = var.node_pool_availability_domain
            subnet_id = var.node_pool_subnet_id
        }
        size = var.node_pool_size
    }


    # The shape configuration of the nodes.
    node_shape_config {
        memory_in_gbs = var.node_shape_config_memory
        ocpus = var.node_shape_config_ocpus
    }

}

I updated the values for var.node_shape_config_memory and var.node_shape_config_ocpus ( which modifies the shape configuration of the nodes) in terraform.tfvars file and performed terraform apply.

The apply operation works without any error and it says: Terraform will perform the following actions:

  # oci_containerengine_node_pool.oke-node-pool will be updated in-place
  ~ resource "oci_containerengine_node_pool" "oke-node-pool" {
        id                  = "ocid1.nodepool.oc1.***.aaaaaaaakc67a"
        name                = "pool2"
        # (11 unchanged attributes hidden)



      ~ node_shape_config {
          ~ memory_in_gbs = 8 -> 16
          ~ ocpus         = 1 -> 2
        }

        # (4 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

The section in this output says that node configs are updated, and that's true, I can see it in node pool details in OCI console.:

  ~ node_shape_config {
      ~ memory_in_gbs = 8 -> 16
      ~ ocpus         = 1 -> 2
    }

But the problem is, when I check the individual node configurations ( via Instances page of OCI console) it is clear that existing nodes have the old ocpu and memory configurations. They are not updated.

Is there a case that updates to the node pool will reflect only on the nodes that are created after the update? They are not applicable to existing nodes?

Is there any fix to this problem?

Answer 1

The Oracle docs says that:

You can use Container Engine for Kubernetes to modify the properties of node pools and worker nodes in existing Kubernetes clusters.

You can change:

• the version of Kubernetes to run on new worker nodes
• the image to use for new worker nodes
• the shape to use for new worker nodes
• the boot volume size to use for new worker nodes

Also note the following:

Any changes you make to worker node properties will only apply to new worker nodes. You cannot change the properties of existing worker nodes.

That means, the changes are applied to only new worker nodes in the node pool.

In situations, when we want to update properties of all the worker nodes in a node pool simultaneously (like, upgrading all worker nodes to a new version of Oracle Linux), then we should create a new node pool with worker nodes that have the required properties, and shift work from the original node pool to the new node pool using the kubectl drain command and pod disruption budgets.