I logged into the Linux bastion host where kubectl is installed for connecting to the Kubernetes cluster.
On the Bastion host when I run any kubectl command like the one below:
kubectl get pods
I get the error below:
fatal error: runtime: out of memory
runtime stack:
runtime.throw(0x192cb47, 0x16)
/usr/local/go/src/runtime/panic.go:617 +0x72 fp=0x7fff534ae560 sp=0x7fff534ae530 pc=0x42d482
runtime.sysMap(0xc000000000, 0x4000000, 0x2d178b8)
/usr/local/go/src/runtime/mem_linux.go:170 +0xc7 fp=0x7fff534ae5a0 sp=0x7fff534ae560 pc=0x417d07
runtime.(*mheap).sysAlloc(0x2cfef80, 0x2000, 0x2cfef90, 0x1)
/usr/local/go/src/runtime/malloc.go:633 +0x1cd fp=0x7fff534ae648 sp=0x7fff534ae5a0 pc=0x40ae0d
runtime.(*mheap).grow(0x2cfef80, 0x1, 0x0)
/usr/local/go/src/runtime/mheap.go:1222 +0x42 fp=0x7fff534ae6a0 sp=0x7fff534ae648 pc=0x425022
runtime.(*mheap).allocSpanLocked(0x2cfef80, 0x1, 0x2d178c8, 0x0)
/usr/local/go/src/runtime/mheap.go:1150 +0x37f fp=0x7fff534ae6d8 sp=0x7fff534ae6a0 pc=0x424f0f
runtime.(*mheap).alloc_m(0x2cfef80, 0x1, 0x2a, 0x0)
/usr/local/go/src/runtime/mheap.go:977 +0xc2 fp=0x7fff534ae728 sp=0x7fff534ae6d8 pc=0x424562
runtime.(*mheap).alloc.func1()
/usr/local/go/src/runtime/mheap.go:1048 +0x4c fp=0x7fff534ae760 sp=0x7fff534ae728 pc=0x456a4c
runtime.(*mheap).alloc(0x2cfef80, 0x1, 0x1002a, 0x0)
/usr/local/go/src/runtime/mheap.go:1047 +0x8a fp=0x7fff534ae7b0 sp=0x7fff534ae760 pc=0x42483a
runtime.(*mcentral).grow(0x2cffd80, 0x0)
/usr/local/go/src/runtime/mcentral.go:256 +0x95 fp=0x7fff534ae7f8 sp=0x7fff534ae7b0 pc=0x417785
runtime.(*mcentral).cacheSpan(0x2cffd80, 0x7fd1a382d000)
/usr/local/go/src/runtime/mcentral.go:106 +0x2ff fp=0x7fff534ae858 sp=0x7fff534ae7f8 pc=0x41728f
runtime.(*mcache).refill(0x7fd1a382d008, 0x2a)
/usr/local/go/src/runtime/mcache.go:135 +0x86 fp=0x7fff534ae878 sp=0x7fff534ae858 pc=0x416d26
runtime.(*mcache).nextFree(0x7fd1a382d008, 0x2cf6e2a, 0x7fd1a382d008, 0x7fd1a382d000, 0x8)
/usr/local/go/src/runtime/malloc.go:786 +0x88 fp=0x7fff534ae8b0 sp=0x7fff534ae878 pc=0x40b648
runtime.mallocgc(0x180, 0x190fb40, 0x1, 0x2d17920)
/usr/local/go/src/runtime/malloc.go:939 +0x76e fp=0x7fff534ae950 sp=0x7fff534ae8b0 pc=0x40bf5e
runtime.newobject(0x190fb40, 0x4000)
/usr/local/go/src/runtime/malloc.go:1068 +0x38 fp=0x7fff534ae980 sp=0x7fff534ae950 pc=0x40c368
runtime.malg(0x74ac0e00008000, 0x2d015f0)
/usr/local/go/src/runtime/proc.go:3220 +0x31 fp=0x7fff534ae9c0 sp=0x7fff534ae980 pc=0x436871
runtime.mpreinit(...)
/usr/local/go/src/runtime/os_linux.go:311
runtime.mcommoninit(0x2cf9240)
/usr/local/go/src/runtime/proc.go:618 +0xc2 fp=0x7fff534ae9f8 sp=0x7fff534ae9c0 pc=0x430222
runtime.schedinit()
/usr/local/go/src/runtime/proc.go:540 +0x74 fp=0x7fff534aea50 sp=0x7fff534ae9f8 pc=0x42feb4
runtime.rt0_go(0x7fff534aea88, 0x3, 0x7fff534aea88, 0x0, 0x0, 0x3, 0x7fff534aee07, 0x7fff534aee0f, 0x7fff534aee13, 0x0, ...)
/usr/local/go/src/runtime/asm_amd64.s:195 +0x11a fp=0x7fff534aea58 sp=0x7fff534aea50 pc=0x458c4a
I was able to fix the issue, it was due to insufficient memory on the bastion host.
Here's how I fixed it:
First I ran the command below to check the memory utilization:
free -m
which gave me the following output which confirmed that the insufficient memory:
total used free shared buff/cache available
Mem: 3.8G 3.7G 93M 988K 34M 27M
Swap: 0B 0B 0B
You can temporarily run the command below to free up little memory space:
free && sync && echo 3 > /proc/sys/vm/drop_caches && free
What I did next was to upgrade the EC2 Instance type of the Bastion Host since it's an AWS EC2 Instance from t2.medium (4GB) to t2.large (8GB) using the steps stated here: Change the instance type. Basically, all you have to do is to:
Wait for a few minutes and everything should take effect. Please take note that your Public IP address for the Bastion Host might change after the upgrade.
Reference: Kubeflow deployment error "fatal error: runtime: out of memory" #3803