Search code examples
rediskubernetesgcloudgoogle-cloud-memorystore

GoogleCloud Kubernetes node cannot connect to Redis Memorystore possibly due to overlap in IP ranges

I have a GoogleCloud Kubernetes cluster consisting of multiple nodes and a GoogleCloud Redis Memorystore. Distributed over these nodes are replicas of a pod containing a container that needs to connect to the Redis Memorystore. I have noticed that one of the nodes is not able to connect to Redis, i.e. any container in a pod on that node cannot connect to Redis.

The Redis Memorystore has the following properties:

  • IP address: 10.0.6.12
  • Instance IP address range: 10.0.6.8/29 (10.0.6.8 - 10.0.6.15)

The node from which no connection to Redis can be made has the following properties:

  • Internal IP: 10.132.0.5
  • PodCIDR: 10.0.6.0/24 (10.0.6.0 - 10.0.6.255)

I assume this problem is caused by the overlap in IP ranges of the Memorystore and this node. Is this assumption correct?

If this is the problem I would like to change the IP range of the node. I have tried to do this by editing spec.podCIRD in the node config:

$ kubectl edit node <node-name>

However this did not work and resulted in the error message:

# * spec.podCIDR: Forbidden: node updates may not change podCIDR except from "" to valid
# * []: Forbidden: node updates may only change labels, taints, or capacity (or configSource, if the DynamicKubeletConfig feature gate is enabled)

Is there another way to change the IP range of an existing Kubernetes node? If so, how?

Sometimes I need to temporarily increase the number of pods in a cluster. When I do this I want to prevent Kubernetes from creating a new node with the IP range 10.0.6.0/24. Is it possible to tell the Kubernetes cluster to not create new nodes with the IP range 10.0.6.0/24? If so, how?

Thanks in advance!

Solution

    • Not for a node. The podCidr gets defined when you install your network overlay in initial steps when setting up a new cluster.

    • Yes for the cluster. but it's not that easy. You have to change the podCidr for the network overlay in your whole cluster. It's a tricky process that can be done, but if you are doing that you might as well deploy a new cluster. Keep in mind that some network overlays require a very specific PodCidr. For example, Calico requires 192.168.0.0/16

    You could:

    1. Create a new cluster with a new cidr and move your workloads gradually.
    2. Change the IP address cidr where your GoogleCloud Redis Memorystore lives.

    Hope it helps!