Background
I am trying to build a docker image from Dockerfile on a VM. VM is running Redhat 7.1 (kernel 3.10) and Docker is 1.10.2
The Dockerfile has following content
FROM rhel
MAINTAINER MyName<me@email.com>
RUN #yum install wget and other tools (less than 500 MB)
COPY entitlementfile /opt/entitlementfile
RUN wget -O /opt/installer.bin https://installer.com/installer.bin \
&& chmod +x /opt/* \
&& /opt/installer.bin --quiet \
&& rm -f /opt/*.bin
USER admin
My build VM has around 16G space available
[root@xrh701 DockerImage]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rhel-root 18G 2.1G 16G 12% /
...
The installer is around 3G and the installed package is around 8G. This adds up to 11G maximum, slightly exceeds docker defaults base device size which is 10G.
So I manually bring up docker daemon with a larger dm.size (15G) to work around this issue.
docker daemon --storage-opt dm.basesize=15G
Since docker is based on Union FS, the images are layered on top of each other. So my understanding is that
(1) the maximum size that my image could get is 11G (Installer in one layer 3G, and the package layer added on top 8G)
(2) and if I wget the installer, run it, then delete the installer in the same RUN command, the image should only be 8G ( since 3G installer is deleted)
Either way, the bottom line is, 16G space should be more than enough.
Problem
But my current observation is that during my docker build process, it would always hang since it had consumed all the available spaces
[root@xrh701 DockerImage]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rhel-root 18G 18G 20K 100% /
...
I can see two images
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
<none> <none> fa09e98656ba About an hour ago 258.1 MB
rhel latest 32f8a1d5f019 9 days ago 203.2 MB
"docker inspect " shows that the < none > image is the intermediate image after COPY the entitlement file.
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
232245023f90 fa09e98656ba "/bin/sh -c 'wget -O " About an hour ago Exited (0) About an hour ago lonely_curie
This intermediate container has completed the RUN wget line, exited successfully, but did not commit to docker image due to lack of space
What's worse, I cannot remove the container nor delete /var/lib/docker to restore the space back
# docker rm -f lonely_curie
Failed to remove container (lonely_curie): Error response from daemon: Driver devicemapper failed to remove root filesystem 232245023f90b42a4dbd19a78bf32836f9f8618d7dbcba54159c3df029b5b114: mount still active
Question
Why has docker used up all the space? According to the calculation, the disk space (16G) should be more than enough for the target image (8G). [I own this VM so I can guarantee no one else or any other process is consuming hard disk space ]
How do I force remove the container in my current situation so I can restore the space back?
How should I build this 8G image (or 11G image if you count the installer) on a 16G VM? If this is not possible, what is the minimum space to successfully build it? I am currently applying for a 32G VM from the lab.
TL;DR
Assume the installer is 3G and the installed package is 8G.
If I release the Dokcerfile, then the minimum disk requirement would be at least 22 G [ (3+8)*2 = 22 ] to build this image.
If I release the image and have pushed to Dockerhub, then user only needs 11 G to pull the image and run a container based on it.
==================================================================
Build
I got a machine about 50G in size to rerun the build and monitored the disk consumption.
Before the build starts
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_01-lv_root 59G 6G 53G 9% /
...
After the installation has completed
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_01-lv_root 59G 19G 38G 34% /
...
After the image is commited
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_01-lv_root 59G 27G 30G 48% /
...
So to answer my own question: (1) It used up all the disk because it requires at least that much disk space. Previous calculation had wishfully assumed the running container and image to be built will share the same layer.
(2) Haven't figure out this part yet. Right now I am just throwing away the VM and let the reclaim it.
(3) The minimum disk required would be (3G + 8G) * 2 = 22 G. So I guess for future reference, I should reserve twice as much of theoretically calculated image size, since the layer seems to be a copied instead of shared when committing a running container to a image. [Building a Dockerfile is essentially the same as the manually running a container and committing to image.]
=================================================================
Run
And to follow up, after I commit the image and delete the container, the disk can be reclaimed
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_01-lv_root 59G 15G 42G 26% /
....
And from then on, bringing up a running container will not increase the disk consumption (significantly).
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_01-lv_root 59G 15G 42G 26% /
...