NGINX Docker container growing too big

Question

Having a hard time figuring this one out. I have a docker container that just keeps growing in size.

Running sudo du -h / | grep '[0-9\.]\+G' shows that the container e917b9b06 is taking up almost 60G of space.

If I do docker ps is looks like that container is my nginx container.

I thought perhaps it was logs, but all the nginx logs are symlinks to stdout. There's nothing in the tmp directory.

Any fancy tools to figure out what is happening here? ncdu inside the container doesn't show anything of value.

Answer 1

I suspect it is indeed the logs.

While you are correct that nginx logs are streamed to stdout, that would means docker store the logs directly as container logs. You have several options (pick any, but only one):

1. Just delete/recreate the containers. It will create a fresh one. You'll be safe as long as you already put the persistent data in the correct volume.

2. Go to the /var/lib/docker/containers/<id>. You will find the stdout logs there, usually in the format: <id>-json.log depending on your docker version. Just check where the big file is. Confirm the content with tail <filename>. If this is the logs, delete or truncate it: truncate -s 0 <filename> or simply : > <filename> if you want to use redirection.

Now that we fixed the issue, we should try to prevent it from happening again.

I can see that you already running these containers for 3 years. Congratulations. Container-based approach would mean that it should be trivial if you just delete or create the new container, by the orchestrator.

If you plan on running this for a long time, I suggest you read about docker log rotation. The default one, the json log driver, doesn't have rotation by default so you should specify the limit either in your docker run command or docker compose. This is the relevant documentation about json log driver. Or you can use other log driver like syslog.