Search code examples
etcd

Have troubles while adding new etcd members

I'm planning to add new members to a single instance of etcd, but am faced with problems.

I started the first etcd member with the following command: 

nohup etcd \
  --advertise-client-urls=https://192.168.22.34:2379 \
  --cert-file=/etc/kubernetes/pki/etcd/server.crt \
  --client-cert-auth=true \
  --data-dir=/var/lib/etcd \
  --initial-advertise-peer-urls=https://192.168.22.34:2380 \
  --initial-cluster=test-master-01=https://192.168.22.34:2380 \
  --key-file=/etc/kubernetes/pki/etcd/server.key \
  --listen-client-urls=https://0.0.0.0:2379 \
  --listen-peer-urls=https://192.168.22.34:2380 \
  --name=test-master-01 \
  --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt \
  --peer-client-cert-auth=true \
  --peer-key-file=/etc/kubernetes/pki/etcd/peer.key \
  --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt \
  --snapshot-count=10000 \
  --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt &

Then I checked the health of the cluster and it seems to be healthy:

member f13d668ae0cba84 is healthy: got healthy result from https://192.168.22.34:2379
cluster is healthy

I also checked the members:

f13d668ae0cba84: name=test-master-01 peerURLs=http://192.168.22.34:2380 clientURLs=https://192.168.22.34:2379 isLeader=true

Then I tried to add second member:

etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --ca-file=/etc/kubernetes/pki/etcd/ca.crt \
  --cert-file=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key-file=/etc/kubernetes/pki/etcd/healthcheck-client.key \
  member add test-master-02 https://192.168.22.37:2380

Added member named test-master-02 with ID 65bec874cca265d8 to cluster ETCD_NAME="test-master-02"
ETCD_INITIAL_CLUSTER="test-master-01=http://192.168.22.34:2380,test-master-02=https://192.168.22.37:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"

Then started the second etcd member with the following command:

etcd \
  --name test-master-02 \
  --listen-client-urls https://192.168.22.37:2379 \
  --advertise-client-urls https://192.168.22.37:2379 \
  --listen-peer-urls https://192.168.22.37:2380 \
  --cert-file=/etc/kubernetes/pki/etcd/server.crt \
  --client-cert-auth=true \
  --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt \
  --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt \
  --peer-client-cert-auth=true \
  --peer-key-file=/etc/kubernetes/pki/etcd/peer.key \
  --key-file=/etc/kubernetes/pki/etcd/server.key \
  --initial-cluster-state=existing \
  --initial-cluster=test-master-01=https://192.168.22.34:2380,test-master-02=https://192.168.22.37:2380

But I got an error:

etcdmain: error validating peerURLs {ClusterID:bc8c76911939f2de Members:[&{ID:f13d668ae0cba84 RaftAttributes:{PeerURLs:[http://192.168.22.34:2380]} Attributes:{Name:test-master-01 ClientURLs:[https://192.168.22.34:2379]}} &{ID:65bec874cca265d8 RaftAttributes:{PeerURLs:[https://192.168.22.37:2380]} Attributes:{Name: ClientURLs:[]}}] RemovedMemberIDs:[]}: unmatched member while checking PeerURLs

Update Looks like I don't have such problem while starting cluster from scratch without restoring from snapshot.

Solution

  • Figured out that before adding new members I needed to update my main etcd member, because instead of etcd config, member list command returned 127.0.0.1 on peerurl