google container engine having problems pulling image from container registry

Question

I'm trying to create a deployment on GKE (running 1.6.0) which is looking like this:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: my-api
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: api
          image: eu.gcr.io/<PROJECT>/<IMAGE>:latest
          imagePullPolicy: Always
          ports:
            - containerPort: 3000
          env:
            - name: NODE_ENV
              value: production
          resources:
            requests:
              cpu: 100m

Creating this fails with the following error message:

Failed to pull image "eu.gcr.io/<PROJECT>/<IMAGE>:latest": rpc error: code = 2 desc = failed to register layer: rename /var/lib/docker/image/overlay/layerdb/tmp/layer-629814250 /var/lib/docker/image/overlay/layerdb/sha256/bd2793152ee77e9d503e981352ff16122b220968ce9df1cc3b49b9704d7dfe28: directory not empty
Error syncing pod, skipping: failed to "StartContainer" for "api" with ErrImagePull: "rpc error: code = 2 desc = failed to register layer: rename /var/lib/docker/image/overlay/layerdb/tmp/layer-629814250 /var/lib/docker/image/overlay/layerdb/sha256/bd2793152ee77e9d503e981352ff16122b220968ce9df1cc3b49b9704d7dfe28: directory not empty"

Other deployments that look almost identical but use a different image are working as expected. What is wrong with the image I'm trying to pull? And how can I debug/fix this?

Answer 1

This may be caused by a known docker bug where shutdown occurs before the content is synced to disk on layer creation. The fix is included in docker v1.13.

One temporary workaround sugguested is to remove the empty files in the directory, and re-pull the image.