Caching with Varnish (5.2) is working fine, but when changing something on a site, the cache does not get refreshed and stale content keeps being shown. I can see that there is a purge queue, but no BAN 's are being processed. Drupal (8) with the Varnish Purge module forms a queue to get cache tags invalidation in Varnish and drush processes it:
drush p:queue-work
------------------------ -----
Succeeded 0
Failed 100
Currently invalidating 0
Not supported 0
------------------------ -----
In QueueCommands.php line 529: Over 40% failed, please check the logs!
And then this message in the logs:
purger_varnish_84f7afee13: item failed due GuzzleHttp\Exception\ConnectException, details (JSON): {"msg":"cURL error 28: Operation timed out after 1000 milliseconds with 0 out of 0 bytes received (see https:\/\/curl.haxx.se\/libcurl\/c\/libcurl-errors.html)","uri":"https:\/\/127.0.0.1:6081\/","method":"BAN","guzzle_opt":{"http_errors":true,"connect_timeout":1,"timeout":1,"verify":false},"headers":{"user-agent":"varnish_purger module for Drupal 8.","cache-tags":"config:contact.form.personal"}}
Manually purging with cURL does work:
curl -X BAN http://127.0.0.1:6081/ -H "Cache-Tags: node:1042"
What could be the problem, it looks like Varnish doesn't accept a connection or something like that?
The contents of /etc/varnish/usr.vcl:
vcl 4.0;
backend default {
.host = "127.0.0.1";
.port = "8080";
}
acl purge {
"127.0.0.1";
}
# Respond to incoming requests.
sub vcl_recv {
# Add an X-Forwarded-For header with the client IP address.
if (req.restarts == 0) {
if (req.http.X-Forwarded-For) {
set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip;
}
else {
set req.http.X-Forwarded-For = client.ip;
}
}
# Only allow PURGE requests from IP addresses in the 'purge' ACL.
if (req.method == "PURGE") {
if (!client.ip ~ purge) {
return (synth(405, "Not allowed."));
}
return (hash);
}
# Only allow BAN requests from IP addresses in the 'purge' ACL.
if (req.method == "BAN") {
# Same ACL check as above:
if (!client.ip ~ purge) {
return (synth(403, "Not allowed."));
}
# Logic for the ban, using the Cache-Tags header. For more info
# see https://github.com/geerlingguy/drupal-vm/issues/397.
if (req.http.Cache-Tags) {
ban("obj.http.Cache-Tags ~ " + req.http.Cache-Tags);
}
else {
return (synth(403, "Cache-Tags header missing."));
}
# Throw a synthetic page so the request won't go to the backend.
return (synth(200, "Ban added."));
}
if (req.method == "URIBAN") {
ban("req.http.host == " + req.http.host + " && req.url == " + req.url);
# Throw a synthetic page so the request won't go to the backend.
return (synth(200, "Ban added."));
}
# Only cache GET and HEAD requests (pass through POST requests).
if (req.method != "GET" && req.method != "HEAD") {
return (pass);
}
# Pass through any administrative or AJAX-related paths.
if (req.url ~ "^/status.php$" ||
req.url ~ "^/update.php$" ||
req.url ~ "^/admin$" ||
req.url ~ "^/admin/.*$" ||
req.url ~ "^/flag/.*$" ||
req.url ~ "^.*/ajax/.*$" ||
req.url ~ "^.*/ahah/.*$") {
return (pass);
}
# Removing cookies for static content so Varnish caches these files.
if (req.url ~ "(?i).(pdf|asc|dat|txt|doc|xls|ppt|tgz|csv|png|gif|jpeg|jpg|ico|swf|css|js)(\?.*)\?$") {
unset req.http.Cookie;
}
if (req.http.Cookie) {
set req.http.Cookie = ";" + req.http.Cookie;
set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";");
set req.http.Cookie = regsuball(req.http.Cookie, ";(SESS[a-z0-9]+|SSESS[a-z0-9]+|NO_CACHE)=", "; \1=");
set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", "");
set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", "");
if (req.http.Cookie == "") {
unset req.http.Cookie;
}
else {
return (pass);
}
}
}
# Set a header to track a cache HITs and MISSes.
sub vcl_deliver {
# Remove ban-lurker friendly custom headers when delivering to client.
unset resp.http.X-Url;
unset resp.http.X-Host;
# Comment these for easier Drupal cache tag debugging in development.
#unset resp.http.Cache-Tags;
#unset resp.http.X-Drupal-Cache-Contexts;
if (obj.hits > 0) {
set resp.http.Cache-Tags = "HIT";
}
else {
set resp.http.Cache-Tags = "MISS";
}
}
# Instruct Varnish what to do in the case of certain backend responses (beresp).
sub vcl_backend_response {
# Set ban-lurker friendly custom headers.
set beresp.http.X-Url = bereq.url;
set beresp.http.X-Host = bereq.http.host;
# Cache 404s, 301s, at 500s with a short lifetime to protect the backend.
if (beresp.status == 404 || beresp.status == 301 || beresp.status == 500) {
set beresp.ttl = 10m;
}
# Don't allow static files to set cookies.
# (?i) denotes case insensitive in PCRE (perl compatible regular expressions).
# This list of extensions appears twice, once here and again in vcl_recv so
# make sure you edit both and keep them equal.
if (bereq.url ~ "(?i).(pdf|asc|dat|txt|doc|xls|ppt|tgz|csv|png|gif|jpeg|jpg|ico|swf|css|js)(\?.*)?$") {
unset beresp.http.set-cookie;
}
# Allow items to remain in cache up to 6 hours past their cache expiration.
set beresp.grace = 6h;
}
I use an NGINX proxy for enabling SSL with Varnish, the contents of the nginx server configuration:
server {
listen 443 ssl http2;
server_name test.example.com;
port_in_redirect off;
ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
ssl_ciphers EECDH+CHACHA20:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:!MD5;
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
location / {
proxy_pass http://127.0.0.1:6081;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header HTTPS "on";
proxy_set_header If-Modified-Since $http_if_modified_since;
proxy_buffering on;
proxy_buffer_size 128k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 256k;
}
}
server {
listen 8080;
server_name test.example.com;
root /home/example/domains/test/public_html/web;
index index.php index.html index.htm index.nginx-debian.html;
port_in_redirect off;
location / {
try_files $uri $uri/ /index.php?$query_string;
}
location ~ \.php$ {
include snippets/fastcgi-php.conf;
fastcgi_pass 127.0.0.1:9000;
}
}
server {
listen 80;
if ($host = test.example.com) {
return 301 https://$host$request_uri;
}
server_name test.example.com;
return 404;
}
Drush cannot connect to varnish on port 6081 over TLS.
Your log line indicates that you're connecting to Varnish using the following URL: https://127.0.0.1:6081/.
The open source version of Varnish doesn't support native TLS, that's why you're terminating it in Nginx.
2 possible solutions:
http://127.0.0.1:6081 to use plain HTTPhttps://127.0.0.1 to use HTTPS via NginxI'd go for the former if I were you, and just use plain HTTP for invalidation. It happens internally, it is not exposed to the internet, so it's pretty safe to use regular HTTP.