Background:
I run a java process in my docker container and I take histo dumps using jmap to a file at /home/heapdump.txt inside container. I get this file from the container for further processing.
Now, I do this at an interval of 5 minutes. However, after 20 mins meaning, 4 heapdumps, when I try to get this file, I get the below error:
{"message":"mount/:/var/lib/docker/overlay2/<container_id>/merged/hostroot, flags: 0x5001: no space left on device"}
I don't understand what no space left on device means in this case. 😕😕😕
Your storage is mapped to default /var. Which I believe will hold much less space unless you have manually allotted more.
Do a df -kh on your device and see the status of the device mapped to /var. You would have run out of space.
To fix this find a disk with good space - remember this will be used by docker to store all its image and volume data. and make the docker use it.
You need to configure this in daemon.json file as a data-root config like below.
{
“data-root”: “/new/data/root/path”
}
Remember to reload the daemon and restart docker service. Once done you will see docker beautifully copies its image and volume data to the new directory.
once you test you can clean up the var/lib/docker.
Hope this helps