Search code examples
kubernetesetcd

See wrong client URL when listing the etcd member

I have a Stacked master K8s cluster (etcd is also local/internal) with three master and 9 worker nodes. And my cluster version is currently 1.12.3, while going through etcd commands, i tried listing the etcd member, executing 

ETCDCTL_API=3 etcdctl member list

, and found that the client Url's of master2 and master3 is wrong.

Below is the image,

As per my understanding ip for peers and client should be same, but as I can see IP is 127.0.0.1 in case of master2 and master3.

When I check the endpoint status I get below error as,

Failed to get the status of endpoint :2379 (context deadline exceeded)

while I am successfully getting the status for master1,

Could anyone please help me out in solving this.

Things I tried:

1) Edited the manifest file, etcd pods got restarted, but still nothing changed when I listed the member.

2) I have also successfully removed and added master3 in the etcd cluster, and this worked (IP's got corrected and getting the status of master3), but when I did the same for master2 getting error as

"error validating peerURLs {{ID: xyz, PeerUrls:xyz, clienturl:xyz},{&ID:xyz......}}: member count is unequal"

Solution

  • Editing etcd manifest file and correcting the IP worked for me. Previously it wasn't working because there was one etcd.yml.bkp file present in the manifest folder (probably i took the backup of etcd manifest there it self before upgrading) and found that etcd pods referring to that yml file, removing that yml file from manifest folder resolved the issue.

    Also found IP mentioned in the kube-apiserver.yml files was incorrect, for correcting it tried below two methods both worked:

    • Manually edited the file and corrected the IP
    • Or, We can generate a new manifest file for kube-api server executing kubeadm init phase control-plane apiserver --kubernetes-version 1.14.5