Search code examples
ssldockerkubernetescoreosetcd

Kubernetes 1.2.2: api-server fails: can't find mounted certs for TLS on etcd

I'm been struggling to get api-server 1.2.2 to run with etcd secured with TLS.

I am upgrading from 1.1.2 to 1.2.2

In 1.1.2 I was using the --etcd-config flag and had a file that looked like:

{
  "cluster": {
    "machines": [
      "https://XXX.XXX.XXX.XXX:2379",
      "https://XXX.XXX.XXX.XXY:2379",
      "https://XXX.XXX.XXX.XXZ:2379"
    ]
  },
  "config": {
    "certFile": "/etc/ssl/etcd/etcd-peer.cert.pem",
    "keyFile": "/etc/ssl/etcd/private/etcd-peer.key.pem",
    "caCertFiles": [
      "/etc/ssl/etcd/ca-chain.cert.pem"
    ],
    "consistency": "STRONG_CONSISTENCY"
  }
}

now this is no longer supported and I switched to using the flags:

--etcd-cafile="/etc/ssl/etcd/ca-chain.cert.pem"
--etcd-certfile="/etc/ssl/etcd/etcd-peer.cert.pem"
--etcd-keyfile="/etc/ssl/etcd/private/etcd-peer.key.pem"    
--etcd-servers="https://XXX.XXX.XXX.XXX:2379, https://XXX.XXX.XXX.XXY:2379,https://XXX.XXX.XXX.XXZ:2379"

now I am getting this error:

F0421 00:54:40.133777       1 server.go:291] Invalid storage version or misconfigured etcd: open "/etc/ssl/etcd/etcd-peer<nodeIP>.cert.pem": no such file or directory

So, it seems like it cannot find the cert file. The file paths and names are the same as before, and they are mounted with hostPath the exact same way as with v1.1.2, so I don't understand why api-server would not not find them.

I have been trying to figure what is going on with the file paths by simply switching the command in the pod from

- /hyperkube
- api-server
...

to

- /bin/sleep
- 60

but kubelet won't start this pod for some reason I don't understand.

Does it have to do with the yaml file name or something?

I don't understand what is happening why kubelet won't run with this command.

Any help with this would be greatly appreciated.

Thanks

UPDATE

I was able to get into the running container after replacing the command with /hyperkube scheduler

i can cat the files that apiserver is complaining about, so I don't understand why they're not found.

Solution

  • Well, the culprit was as simple as ""

    --etcd-cafile="/etc/ssl/etcd/ca-chain.cert.pem"
--etcd-certfile="/etc/ssl/etcd/etcd-peer.cert.pem"
--etcd-keyfile="/etc/ssl/etcd/private/etcd-peer.key.pem"    
--etcd-servers="https://XXX.XXX.XXX.XXX:2379, https://XXX.XXX.XXX.XXY:2379,https://XXX.XXX.XXX.XXZ:2379"

    is WRONG

    but this works:

    --etcd-cafile=/etc/ssl/etcd/ca-chain.cert.pem
--etcd-certfile=/etc/ssl/etcd/etcd-peer.cert.pem
--etcd-keyfile=/etc/ssl/etcd/private/etcd-peer.key.pem   
--etcd-servers=https://XXX.XXX.XXX.XXX:2379,https://XXX.XXX.XXX.XXY:2379,https://XXX.XXX.XXX.XXZ:237