I installed successfully Kubernetes 1.1.2 on CoreOS alpha (877.1.0) will the service files as link https://gist.github.com/thanhson1085/5a005e92245cb2288dee
After that, I want to run SkyDNS AddOn for my Kubernetes as a Service Discovery. And I followed this guide: https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dns
But it does not work:
core@coreos-1 ~/projects $ sudo kubectl exec busybox -- nslookup kubernetes.default.cluster.local
Server: 10.100.100.100
Address 1: 10.100.100.100
nslookup: can't resolve 'kubernetes.default.cluster.local'
error: error executing remote command: Error executing command in container: Error executing in Docker Container: 1
core@coreos-1 ~/projects $ sudo kubectl exec busybox -- nslookup kubernetes.default
Server: 10.100.100.100
Address 1: 10.100.100.100
nslookup: can't resolve 'kubernetes.default'
error: error executing remote command: Error executing command in container: Error executing in Docker Container: 1
core@coreos-1 ~/projects $ sudo kubectl exec busybox -- nslookup kubernetes.default.svc.cluster.local
Server: 10.100.100.100
Address 1: 10.100.100.100
nslookup: can't resolve 'kubernetes.default.svc.cluster.local'
error: error executing remote command: Error executing command in container: Error executing in Docker Container: 1
It works if i lookup global domain:
core@coreos-1 ~/projects $ sudo kubectl exec busybox -- nslookup google.com
Server: 10.100.100.100
Address 1: 10.100.100.100
Name: google.com
Address 1: 2404:6800:4003:c02::64 sc-in-x64.1e100.net
Address 2: 74.125.200.138 sa-in-f138.1e100.net
Address 3: 74.125.200.139 sa-in-f139.1e100.net
Address 4: 74.125.200.100 sa-in-f100.1e100.net
Address 5: 74.125.200.113 sa-in-f113.1e100.net
Address 6: 74.125.200.102 sa-in-f102.1e100.net
Address 7: 74.125.200.101 sa-in-f101.1e100.net
core@coreos-1 ~/projects $ sudo docker logs k8s_skydns.ed0ae89c_kube-dns-v9-a5afs_kube-system_b45f88f2-9684-11e5-9f5a-04018a53d701_32d98a24
2015/11/29 10:47:00 skydns: falling back to default configuration, could not read from etcd: 100: Key not found (/skydns) [2]
2015/11/29 10:47:00 skydns: ready for queries on cluster.local. for tcp://0.0.0.0:53 [rcache 0]
2015/11/29 10:47:00 skydns: ready for queries on cluster.local. for udp://0.0.0.0:53 [rcache 0]
kube2sky logs:
Failed to list *api.Service: couldn't get version/kind; json parse error: invalid character '<' looking for beginning of value
Failed to list *api.Endpoints: couldn't get version/kind; json parse error: invalid character '<' looking for beginning of value
And the below is the information for my installation:
core@coreos-1 ~/projects $ sudo kubectl get endpoints -a --all-namespaces
NAMESPACE NAME ENDPOINTS AGE
default frontend 10.100.21.7:80,10.100.21.9:80 1d
default kubernetes 128.199.134.19:6443 1d
default redis-master 10.100.21.2:6379 1d
default redis-slave 10.100.21.5:6379,10.100.21.6:6379 1d
kube-system kube-dns 51m
core@coreos-1 ~/projects $ sudo kubectl get pods -a --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default busybox 1/1 Running 0 51m
default frontend-1w89q 1/1 Running 0 1d
default frontend-w44qx 1/1 Running 0 1d
default redis-master-qde8g 1/1 Running 0 1d
default redis-slave-3m0t3 1/1 Running 0 1d
default redis-slave-m6fc8 1/1 Running 0 1d
kube-system kube-dns-v9-zrd22 3/4 Running 96 52m
core@coreos-1 ~/projects $ sudo kubectl get services -a --all-namespaces
NAMESPACE NAME CLUSTER_IP EXTERNAL_IP PORT(S) SELECTOR AGE
default frontend 10.100.192.188 nodes 80/TCP name=frontend 1d
default kubernetes 10.100.0.1 <none> 443/TCP <none> 1d
default redis-master 10.100.180.191 <none> 6379/TCP name=redis-master 1d
default redis-slave 10.100.146.91 <none> 6379/TCP name=redis-slave 1d
kube-system kube-dns 10.100.100.100 <none> 53/UDP,53/TCP k8s-app=kube-dns 52m
core@coreos-1 ~/projects $ sudo kubectl get rc -a --all-namespaces
NAMESPACE CONTROLLER CONTAINER(S) IMAGE(S) SELECTOR REPLICAS AGE
default frontend php-redis gcr.io/google_samples/gb-frontend:v3 name=frontend 2 1d
default redis-master master redis name=redis-master 1 1d
default redis-slave worker gcr.io/google_samples/gb-redisslave:v1 name=redis-slave 2 1d
kube-system kube-dns-v9 etcd gcr.io/google_containers/etcd:2.0.9 k8s-app=kube-dns,version=v9 1 52m
kube2sky gcr.io/google_containers/kube2sky:1.11
skydns gcr.io/google_containers/skydns:2015-10-13-8c72f8c
healthz gcr.io/google_containers/exechealthz:1.0
See Kubelet Service Status:
core@coreos-1 ~ $ sudo systemctl status kubelet
● kubelet.service - Kubernetes Kubelet
Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2015-11-28 07:17:28 UTC; 52min ago
Main PID: 27007 (kubelet)
Memory: 2.9M
CPU: 944ms
CGroup: /system.slice/kubelet.service
├─27007 /opt/bin/kubelet --port=10250 --cluster-dns=10.100.100.100 --cluster-domain=cluster.local --hostname-override=coreos-1 --api-servers=http://127.0.0.1:8080 --logtostderr=true
└─27018 journalctl -k -f
Nov 28 08:09:26 coreos-1 kubelet[27007]: E1128 08:09:26.690587 27007 fs.go:211] Stat fs failed. Error: no such file or directory
Nov 28 08:09:27 coreos-1 kubelet[27007]: E1128 08:09:27.689255 27007 fs.go:211] Stat fs failed. Error: no such file or directory
Nov 28 08:09:28 coreos-1 kubelet[27007]: E1128 08:09:28.696916 27007 fs.go:211] Stat fs failed. Error: no such file or directory
Nov 28 08:09:29 coreos-1 kubelet[27007]: E1128 08:09:29.697705 27007 fs.go:211] Stat fs failed. Error: no such file or directory
Nov 28 08:09:30 coreos-1 kubelet[27007]: E1128 08:09:30.691816 27007 fs.go:211] Stat fs failed. Error: no such file or directory
Nov 28 08:09:31 coreos-1 kubelet[27007]: E1128 08:09:31.684655 27007 fs.go:211] Stat fs failed. Error: no such file or directory
Nov 28 08:09:32 coreos-1 kubelet[27007]: I1128 08:09:32.008987 27007 manager.go:1769] pod "kube-dns-v9-zrd22_kube-system" container "skydns" is unhealthy (probe result: failure), it will be killed and re-created.
Nov 28 08:09:32 coreos-1 kubelet[27007]: E1128 08:09:32.708600 27007 fs.go:211] Stat fs failed. Error: no such file or directory
Nov 28 08:09:33 coreos-1 kubelet[27007]: E1128 08:09:33.708741 27007 fs.go:211] Stat fs failed. Error: no such file or directory
Nov 28 08:09:34 coreos-1 kubelet[27007]: E1128 08:09:34.685334 27007 fs.go:211] Stat fs failed. Error: no such file or directory
Please help me to fix it. And can I use Consul to place of SkyDNS in this case?
I found the solution from @Tim Hockin comment.
I got logs from kube2sky service:
Failed to list *api.Service: couldn't get version/kind; json parse error: invalid character '<' looking for beginning of value
Failed to list *api.Endpoints: couldn't get version/kind; json parse error: invalid character '<' looking for beginning of value
With this error, I just need to add:
- -kube_master_url=http://10.130.162.60:8080 # IP Kubernets Api Server
To skydns-rc.yaml file.
All the details about my service configuration files, please see: https://gist.github.com/thanhson1085/5a005e92245cb2288dee
After that, restart dns service containers. It works:
core@coreos-1 ~/projects $ sudo kubectl exec busybox -- nslookup kubernetes.default
Server: 10.100.100.100
Address 1: 10.100.100.100
Name: kubernetes.default
Address 1: 10.100.0.1