Setup: [Varnish] <-> [Nginx] <-> [PHP FPM] <-> [PostgreSQL] Varnish, Nginx and PHP FPM run as Docker containers.
When we push new code, we need to update at least the PHP container with the new build. Our CI/CD pipeline also triggers Nginx to restart docker exec -i $CONTAINER_ID_API_NGINX nginx -s reload so it will fetch the (internal) ip of the newly created PHP Container.
However sometimes the Nginx container has to be restarted as well, for example if there are new static assets. In this case we also need to restart Varnish and this is where our troubles lie, and where we have some downtime on our API.
Things we've tried:
1. Starting second Nginx
2. Starting second Varnish
3. Killing first Nginx
4. Killing first Varnish
Problem: The second Varnish Container complains that there are 2 IPs for the backend and goes in a restart cycle until the first Nginx container is shut down.
1. Starting second Nginx
2. Killing first Nginx
3. Starting second Varnish
4. Killing first Varnish
Problem: Before the second varnish is accepting connections, API request still go to the first Varnish instance, it tries to forward them to the first (now killed) Nginx instance
Question: Is there a way to get Varnish to switch it's backend host to the new Nginx container, with or without a restart? Our VCL file currently contains
backend default {
.host = "api-api";
.port = "80";
}
In an ideal world, we would start a second Nginx instance, Varnish still has the DNS cached to the first Nginx instance, we kill the first Nginx instance and trigger a DNS refresh on the Varnish instance without needing to kill the Varnish container itself.
Reloading the VCL can be done using the varnishreload command.
This command will load a new VCL, compile it, deactivate the previous VCL and activate the newly added one.
Any backend DNS resolution will happen while the new VCL file is parsed and compiled.
The varnishreload command runs multiple varnishadm commands. See https://github.com/varnishcache/pkg-varnish-cache/blob/master/systemd/varnishreload for more information on the code.
You can choose to either run varnishreload inside your container to automate the VCL reload, which also resolves the DNS of your backend endpoint.
You can also call varnishadm remotely, but that requires some extra measures.
See http://varnish-cache.org/docs/6.0/reference/varnishadm.html for varnishadm docs.
See http://varnish-cache.org/docs/6.0/reference/varnish-cli.html for more information about the Varnish CLI which varnishadm wraps around.