Kubernetes NFS server pod mount works with pod ip but not with kubernetes service

Question

I created a nfs server in a pod to use it as a volume. When creating another pod with a volume, the volume mount does work with the ip of the nfs pod. Since this ip is not guaranteed to stay the same, I added a service for my nfs pod and added a fixed cluster ip. When starting the container with the volume mount, it always fails with the following error:

Unable to mount volumes for pod "nginx_default(35ecd8ec-a077-11e8-b7bc-0cc47a9aec96)": timeout expired waiting for volumes to attach or mount for pod "default"/"nginx". list of unmounted volumes=[nfs-demo]. list of unattached volumes=[nfs-demo nginx-test-account-token-2dpgg]

    apiVersion: v1
    kind: Pod
    metadata:
      name: nfs-server
      labels:
        name: nfs-server
    spec:
      containers:
      - name: nfs-server
        image: my-nfs-server:v1
        args: ["/exports"]
        securityContext:
          privileged: true
    ---
    kind: Service
    apiVersion: v1
    metadata:
      name: nfs-service
    spec:
      selector:
        name: nfs-server
      clusterIP: "10.96.0.3"
      ports:
        - name: nfs
          port: 2049
          protocol: UDP
        - name: mountd
          port: 20048
          protocol: UDP   
        - name: rpcbind
          port: 111
          protocol: UDP
        - name: nfs-tcp
          port: 2049
          protocol: TCP
        - name: mountd-tcp
          port: 20048
          protocol: TCP
        - name: rpcbind-tcp
          port: 111
          protocol: TCP

My pod trying to mount the server:

    apiVersion: v1
    kind: Pod
    metadata:
      name: nginx
      labels:
        name: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        volumeMounts:
        - mountPath: "/exports"
          name: nfs-demo
        securityContext:
          privileged: true
      securityContext:
        supplementalGroups: [100003]
      serviceAccountName: nginx-test-account
      volumes:
      - name: nfs-demo
        nfs:
          server: 10.96.0.3
          path: "/exports"
          readOnly: false

I used this as a base for my nfs server image:

https://github.com/cpuguy83/docker-nfs-server

https://medium.com/@aronasorman/creating-an-nfs-server-within-kubernetes-e6d4d542bbb9

Does anyone have an idea why the mount ist working with the pod ip but not with the service ip?

Answer 1

I found the solution to my problem:

There were ports missing in my service, not the pod. To find the ports I needed, I opened a console to my pod (kubectl exec) and used the "rpcinfo -p" command to list the ports needed for the service.

It does fix the connection problem, but only temporarily. These ports are not static, so it is not better than using the port IP itself. I do think it is possible to configure static ports though.

If anyone with a similar problem needs further reading:

http://tldp.org/HOWTO/NFS-HOWTO/security.html

https://wiki.debian.org/SecuringNFS

The second problem I encountered: the mount only worked if the nfs-server pod and the pod mounting it were on the same node. I could fix it when updating to kubernetes version 1.11.

Since my original problem is solved, I consider my question answered though.