Search code examples
dockerkubernetesout-of-memoryminikubeminio

Minio OOM(Out Of Memory) when uploading a file

I have a local Kubernetes cluster inside Minikube on my Mac. I deploy a Minio standalone server as a single container with resource limits specified. When I upload a file which is larger than container memory limit, the container is terminated with reason OOMKilled. On Ubuntu with the same setup file is uploaded with no errors.

Minikube is running in VirtualBox and configured to use 4GB of memory. I also use Heapster and Metric Server to check memory usage over time.

$ minikube config set memory 4096
$ minikube addons enable heapster
$ minikube addons enable metrics-server
$ minikube start

minikube in vb

I use a slightly modified version of Kubernetes confiuration for Minio standalone setup provided in Minio documentation. I create PV and PVC for storage, Deployment and Service for Minio server. Container configuration:

  • Using official Minio Docker image
  • Using initContainer to create a bucket

Resource limits are set to have a Guaranteed QoS. Container is limited to 256 MB of memory and 0.5 CPU.

resources:
  requests:
    cpu: '500m'
    memory: '256Mi'
  limits:
    cpu: '500m'
    memory: '256Mi'

Full videos-storage.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: videos-storage-pv
  labels:
    software: minio
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /data/storage-videos/

---

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: videos-storage-pv-claim
spec:
  storageClassName: ''
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
  selector:
    matchLabels:
      software: minio

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: videos-storage-deployment
spec:
  selector:
    matchLabels:
      app: videos-storage
  template:
    metadata:
      labels:
        app: videos-storage
    spec:
      initContainers:
      - name: init-minio-buckets
        image: minio/minio
        volumeMounts:
        - name: data
          mountPath: /data/storage-videos
        command: ['mkdir', '-p', '/data/storage-videos/videos']
      containers:
      - name: minio
        image: minio/minio
        volumeMounts:
        - name: data
          mountPath: /data/storage-videos
        args:
        - server
        - /data/storage-videos
        env:
        - name: MINIO_ACCESS_KEY
          value: 'minio'
        - name: MINIO_SECRET_KEY
          value: 'minio123'
        ports:
        - containerPort: 9000
        resources:
          requests:
            cpu: '500m'
            memory: '256Mi'
          limits:
            cpu: '500m'
            memory: '256Mi'
        readinessProbe:
          httpGet:
            path: /minio/health/ready
            port: 9000
          initialDelaySeconds: 5
          periodSeconds: 20
        livenessProbe:
          httpGet:
            path: /minio/health/live
            port: 9000
          initialDelaySeconds: 5
          periodSeconds: 20
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: videos-storage-pv-claim

---

apiVersion: v1
kind: Service
metadata:
  name: videos-storage-service
spec:
  type: LoadBalancer
  ports:
    - port: 9000
      targetPort: 9000
      protocol: TCP
  selector:
    app: videos-storage

I deploy the config to the cluster:

$ kubectl apply -f videos-storage.yaml

and access Minio dashboard using Minikube, the following command opens it in browser:

$ minikube service videos-storage-service

Then I select a videos bucket and upload a 1 GB file. After about ~250 MB are uploaded, I get an error in Minio dashboard. Playing with limits and analyzing Heapster graphs I can see a correlation between file size and memory usage. Container uses exactly the same amount of memory equal to file size, which is weird to me. My impression is that file won't be stored directly in memory during file upload.

minio dashboard error

Describe pod

Name:           videos-storage-deployment-6cd94b697-p4v8n
Namespace:      default
Priority:       0
Node:           minikube/10.0.2.15
Start Time:     Mon, 22 Jul 2019 11:05:53 +0300
Labels:         app=videos-storage
                pod-template-hash=6cd94b697
Annotations:    <none>
Status:         Running
IP:             172.17.0.8
Controlled By:  ReplicaSet/videos-storage-deployment-6cd94b697
Init Containers:
  init-minio-buckets:
    Container ID:  docker://09d75629a39ad1dc0dbdd6fc0a6a6b7970285d0a349bccee2b0ed2851738a9c1
    Image:         minio/minio
    Image ID:      docker-pullable://minio/minio@sha256:456074355bc2148c0a95d9c18e1840bb86f57fa6eac83cc37fce0212a7dae080
    Port:          <none>
    Host Port:     <none>
    Command:
      mkdir
      -p
      /data/storage-videos/videos
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 22 Jul 2019 11:08:45 +0300
      Finished:     Mon, 22 Jul 2019 11:08:45 +0300
    Ready:          True
    Restart Count:  1
    Environment:    <none>
    Mounts:
      /data/storage-videos from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-zgs9q (ro)
Containers:
  minio:
    Container ID:  docker://1706139f0cc7852119d245e3cfe31d967eb9e9537096a803e020ffcd3becdb14
    Image:         minio/minio
    Image ID:      docker-pullable://minio/minio@sha256:456074355bc2148c0a95d9c18e1840bb86f57fa6eac83cc37fce0212a7dae080
    Port:          9000/TCP
    Host Port:     0/TCP
    Args:
      server
      /data/storage-videos
    State:          Running
      Started:      Mon, 22 Jul 2019 11:08:48 +0300
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    0
      Started:      Mon, 22 Jul 2019 11:06:06 +0300
      Finished:     Mon, 22 Jul 2019 11:08:42 +0300
    Ready:          True
    Restart Count:  1
    Limits:
      cpu:     500m
      memory:  256Mi
    Requests:
      cpu:      500m
      memory:   256Mi
    Liveness:   http-get http://:9000/minio/health/live delay=5s timeout=1s period=20s #success=1 #failure=3
    Readiness:  http-get http://:9000/minio/health/ready delay=5s timeout=1s period=20s #success=1 #failure=3
    Environment:
      MINIO_ACCESS_KEY:  minio
      MINIO_SECRET_KEY:  minio123
    Mounts:
      /data/storage-videos from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-zgs9q (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  videos-storage-pv-claim
    ReadOnly:   false
  default-token-zgs9q:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-zgs9q
    Optional:    false
QoS Class:       Guaranteed
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

Minikube logs in dmesg:

[  +3.529889] minio invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=-998
[  +0.000006] CPU: 1 PID: 8026 Comm: minio Tainted: G           O     4.15.0 #1
[  +0.000001] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
[  +0.000000] Call Trace:
[  +0.000055]  dump_stack+0x5c/0x82
[  +0.000010]  dump_header+0x66/0x281
[  +0.000006]  oom_kill_process+0x223/0x430
[  +0.000002]  out_of_memory+0x28d/0x490
[  +0.000003]  mem_cgroup_out_of_memory+0x36/0x50
[  +0.000004]  mem_cgroup_oom_synchronize+0x2d3/0x310
[  +0.000002]  ? get_mem_cgroup_from_mm+0x90/0x90
[  +0.000002]  pagefault_out_of_memory+0x1f/0x4f
[  +0.000002]  __do_page_fault+0x4a3/0x4b0
[  +0.000003]  ? page_fault+0x36/0x60
[  +0.000002]  page_fault+0x4c/0x60
[  +0.000002] RIP: 0033:0x427649
[  +0.000001] RSP: 002b:000000c0002eaae8 EFLAGS: 00010246
[  +0.000154] Memory cgroup out of memory: Kill process 7734 (pause) score 0 or sacrifice child
[  +0.000013] Killed process 7734 (pause) total-vm:1024kB, anon-rss:4kB, file-rss:0kB, shmem-rss:0kB

Initially the problem occurred when I did not have any resource limits. When I tried to upload a big file, container with Minio would use all of the memory available in the node, and because there is no memory left, Kubernetes services become unresponsive and it started killing other containers, like apiserver; and file won't upload either. After that I added resource limits to Minio container and cluster itself became stable, but Minio container still dies.

I would like Minio to function within provided limits and not consume memory equal to file size. I am not sure on what side the problem might be, whether it is Minio or Minikube or VirtualBox or Docker or Kubernetes. I am not familiar with how memory works in this setup as well. As I said, the same setup worked fine on Ubuntu 18.04.

Versions:

  • MacOS Mojave 10.14.5, 2.6 GHz Intel Core i5, 8 GB 1600 MHz DDR3
  • Minikube v1.2.0
  • VirtualBox 5.2.18 r124319
  • kubectl v1.15.0
  • Docker version 18.09.2, build 6247962
  • Minio 2019-07-17T22:54:12Z (https://hub.docker.com/r/minio/minio)
Solution

  • I've also posted this issue to Minio repository, got a response that 256mb is too low. After increasing the memory to 512mb it worked fine.