Search code examples
kubernetesrpcnfspersistent-volumes

Kubernetes, cannot mount NFS share via DNS

I am trying to mount a NFS share (outside of k8s cluster) in my container via DNS lookup, my config is as below 

apiVersion: v1
kind: Pod
metadata:
  name: service-a
spec:
  containers:
  - name: service-a
    image: dockerregistry:5000/centOSservice-a
    command: ["/bin/bash"]
    args: ["/etc/init.d/jboss","start"]
    volumeMounts:
      - name: service-a-vol
        mountPath: /myservice/por/data
  volumes:
    - name: service-a-vol
      nfs:
        server: nfs.service.domain
        path: "/myservice/data"
  restartPolicy: OnFailure

nslookup of nfs.service.domin works fine from my container. This is achiveded via StubDomain . However when creating the container it fails to resolve the nfs server. Error:

Warning  FailedMount  <invalid>  kubelet, worker-node-1  MountVolume.SetUp failed for volume "service-a-vol" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/44aabfb8-2767-11e8-bcf9-fa163ece9426/volumes/kubernetes.io~nfs/service-a-vol --scope -- mount -t nfs nfs.service.domain:/myservice/data /var/lib/kubelet/pods/44aabfb8-2767-11e8-bcf9-fa163ece9426/volumes/kubernetes.io~nfs/service-a-vol
Output: Running scope as unit run-27293.scope.
mount.nfs: Failed to resolve server nfs.service.domain: Name or service not known
mount.nfs: Operation already in progress

If i modify server: nfs.service.domain to server: 10.10.1.11 this works fine! So to summarise

  1. DNS resolution of the service works fine
  2. Mounting via DNS resolution does not
  3. Mounting via specific IP address works
  4. I have tried Headless Service instead of StubDomain but the same issue exists

Any help much appreciated

Update 1: If i add an entry in the /etc/hosts files of worker/master nodes 10.10.1.11 nfs.service.domain then my configuration above server: nfs.service.domain works. This is obviously not a desired workaround...

Solution

  • As pointed out by @Giorgio Cerruti and as referenced in this github ticket among others this is currently not possible as the node needs to be able to resolve the DNS entry and it does not resolve kube-dns. Two possible solutions are:

    1. Update /etc/hosts of each kubernetes node to resolve the NFS endpoint (as per update above). This is a primitive solution.

    2. A more robust fix that would work for this NFS service and any other remote service in the same domain (as NFS) is to add the remote DNS server to the kubernetes nodes resolv.conf

      someolddomain.org service.domain xx.xxx.xx nameserver 10.10.0.12 nameserver 192.168.20.22 nameserver 8.8.4.4