fork/exec /usr/bin/rkt: permission denied

I'm trying to install Kubernetes.

configuration details:

Controler

OS: Container Linux by CoreOS alpha (1339.0.0)
LAN IP: 192.168.1.2
FQDN: coreos-2.tux-in.com

environment variables used for installation:

export MASTER_FQDN=coreos-2.tux-in.com
export MASTER_IP=192.168.1.2
export WORKER_IP=192.168.1.3
export WORKER_FQDN=coreos-3.tux-in.com
export ETCD_ENDPOINTS=http://127.0.0.1:4001
export ADVERTISE_IP=192.168.1.2
export K8S_VER=v1.5.4_coreos.0
export HYPERKUBE_IMAGE_REPO=quay.io/coreos/hyperkube
export POD_NETWORK=10.2.0.0/16
export SERVICE_IP_RANGE=10.3.0.0/24
export K8S_SERVICE_IP=10.3.0.1
export DNS_SERVICE_IP=10.3.0.10
export USE_CALICO=true
export CONTAINER_RUNTIME=rkt
export EMAIL="[email protected]"
export uuid_file="/var/run/kubelet-pod.uuid"

worker

OS: Container Linux by CoreOS alpha (1339.0.0)
LAN IP: 192.168.1.3
FQDN: coreos-3.tux-in.com

environment variables used for installation

export ETCD_ENDPOINTS=http://127.0.0.1:4001
export CONTROLLER_ENDPOINT=https://coreos-2.tux-in.com
export ADVERTISE_IP=192.168.1.3
export K8S_VER=v1.5.4_coreos.0
export HYPERKUBE_IMAGE_REPO=quay.io/coreos/hyperkube
export POD_NETWORK=10.2.0.0/16
export DNS_SERVICE_IP=10.3.0.10
export USE_CALICO=true
export CONTAINER_RUNTIME=rkt
export EMAIL="[email protected]"
export uuid_file="/var/run/kubelet-pod.uuid"
export  CALICO_OPTS="--volume cni-bin,kind=host,source=/opt/cni/bin \
            --mount volume=cni-bin,target=/opt/cni/bin"

I used my fork of coreos-kubernetes to install the controller and worker (https://github.com/kfirufk/coreos-kubernetes).

the controller node is running fine, but the worker node complains permission denied when trying to run rkt on every image that kubernetes tells it to execute.

for example:

Mar 11 21:34:58 coreos-3.tux-in.com kubelet-wrapper[10203]: E0311 21:34:58.654536   10203 pod_workers.go:184] Error syncing pod ac3615637dcb614ace172dc5c5dd85f3, skipping: failed to SyncPod: failed to run [prepare --quiet --pod-manifest /tmp/manifest-kube-proxy-192.168.1.3-986457484 --stage1-name=coreos.com/rkt/stage1-fly]: fork/exec /usr/bin/rkt: permission denied

any ideas ?

Solution

it was a simple problem.

I point kubelet.service to run rkt with /opt/bin/host-rkt that contains:

#!/bin/sh
# This is bind mounted into the kubelet rootfs and all rkt shell-outs go
# through this rkt wrapper. It essentially enters the host mount namespace
# (which it is already in) only for the purpose of breaking out of the chroot
# before calling rkt. It makes things like rkt gc work and avoids bind mounting
# in certain rkt filesystem dependancies into the kubelet rootfs. This can
# eventually be obviated when the write-api stuff gets upstream and rkt gc is
# through the api-server. Related issue:
# https://github.com/coreos/rkt/issues/2878
exec nsenter -m -u -i -n -p -t 1 -- /usr/bin/rkt "$@"

and it wasn't with executable permission. that's it.. a simple chmod +x /opt/bin/host-rkt solved it :)