I'm running docker-registry inside a deployment with ingress setup (nginx-ingress), and I use cloudflare. I started getting issues when trying to push images larger then 1GB if a layer is bit larger then that I just get "Retrying in x", and it begins from 0. Strange enough pushing any layer below that threshold just passes without issue and push succeeds.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: {{ .Values.name }}
annotations:
kubernetes.io/ingress.class: nginx
cert-manager.io/cluster-issuer: {{ .Values.certManager.name }}
nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.org/client-max-body-size: "0"
nginx.ingress.kubernetes.io/proxy-buffering: "off"
nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
nginx.ingress.kubernetes.io/proxy_ignore_headers: "X-Accel-Buffering"
nginx.ingress.kubernetes.io/connection-proxy-header: "keep-alive"
nginx.ingress.kubernetes.io/proxy-connect-timeout: "600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
nginx.ingress.kubernetes.io/proxy-next-upstream-timeout: "600"
nginx.ingress.kubernetes.io/proxy-next-upstream-tries: "10"
nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
nginx.ingress.kubernetes.io/proxy-body-size: "8192m"
kubernetes.io/tls-acme: 'true'
nginx.ingress.kubernetes.io/configuration-snippet: |
more_set_headers "proxy_http_version 1.1";
more_set_headers "X-Forwarded-For $proxy_add_x_forwarded_for";
more_set_headers "Host $http_host";
more_set_headers "Upgrade $http_upgrade";
more_set_headers "Connection keep-alive";
more_set_headers "X-Real-IP $remote_addr";
more_set_headers "X-Forwarded-For $proxy_add_x_forwarded_for";
more_set_headers "X-Forwarded-Proto: https";
more_set_headers "X-Forwarded-Ssl on";
labels:
app: {{ .Values.name }}
spec:
tls:
- hosts: {{- range .Values.certificate.dnsNames }}
- {{ . }}
{{- end}}
secretName: {{ .Values.certificate.secretName }}
rules:
- host: {{ .Values.certManager.mainHost }}
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: {{ .Values.service.name }}
port:
number: {{ .Values.service.port }}
I want to be able to upload any size image as long as storage is available.
First, verify you are using nginx-ingress and not ingress-nginx, which uses a different configuration for the body size:
nginx.ingress.kubernetes.io/proxy-body-size: "0"
Next, track down where the connection is getting dropped by checking the proxy and registry logs. This includes cloudflare, the nginx pod, and the registry pod. There's no need to debug all three simultaneously, so figure out which one of these is rejecting the large put requests. If the issue is cloudflare, e.g. if you don't see any logs in your nginx ingress instance or registry containers, then consider pushing directly to your nginx ingress rather than cloudflare. Those logs may also indicate if the issue is based on time rather than size, which would be a different setting to adjust.
And finally, as a workaround if you can't push with one large blob, there is an option to do a chunked blob put to a registry, which breaks the upload up into smaller requests, each of which should be below the proxy limits. Docker by default does a chunked upload but with only a single chunk, and I'm not aware of any way to change it's settings. My own project is regclient, and it can copy images in an OCI layout or exported from the docker engine to a registry. With regclient/regctl working with an exported OCI Layout or docker save output, that could be implemented with the following:
regctl registry login $registry
# when a blob exceeds 500M, push it as 50M chunks, note each chunk is held in ram
regctl registry set --blob-max 500000000 --blob-chunk 50000000 $registry
regctl image import $registry/$repo:$tag $file_tar