I hope i can get some assistance in resolving this issue with varnish:
I have the following configuration:
Configuration Overview:
HTTP Traffic:
Apache listens on port 8080.
Varnish listens on port 80 and forwards requests to Apache on port 8080.
HTTPS Traffic:
Apache handles SSL termination directly on port 443.
Varnish listens on port 80 for HTTP requests.
Adminpanel: Virtualmin 7 GPL
OS: Ubuntu 22.04 LTS
Apache is configured correctly, it serves the correct website as per DocumentRoot in the apache conf file for the VirtualHost/DomainName
As soon as i install varnish, then a problem arises. The DocumentRoot becomes /var/www/html, as if apache/varnish is not recognizing the virtualhost or can't match the hostname, and therefore a alternative DocumentRoot is used(/var/www/html). I can't wrap my head around why this is happening. I have double checked the apache configuration and all is configured correctly.
The DocumentRoot should be: /home/dewebshop/magento2/pub , as it is configured in the apache conf files. And It works properly from apache, but not from Varnish then it uses /var/www/html
I tried adding the following code to the /etc/varnish/default.vcl file, but that did not solve the problem.:
sub vcl_recv {
if (req.http.host) {
set req.http.Host = req.http.host;
}
}
As a consequence of the wrong DocumentRoot used(/var/www/html), the health check generates a backend error because the file returns a 404.
.probe = {
.url = "/health_check.php";
.timeout = 2s;
.interval = 5s;
.window = 10;
.threshold = 5;
}
I tried replacing the /etc/varnish/default.vcl with a very basic version, but the wrong DocumentRoot is still used, that implies that the problem might not be in this vcl file.
/etc/varnish/default.vcl
backend default {
.host = "127.0.0.1";
.port = "8080";
.first_byte_timeout = 600s;
.probe = {
# .url = "/health_check.php";
.url = "/";
.timeout = 2s;
.interval = 5s;
.window = 10;
.threshold = 5;
}
}
/lib/systemd/system/varnish.service
ExecStart=/usr/sbin/varnishd \
-a :80 \
-a localhost:8443,PROXY \
-p feature=+http2 \
-p http_resp_hdr_len=64k \
-p http_resp_size=128k \
-f /etc/varnish/default.vcl \
-s malloc,256m
ExecReload=/usr/sbin/varnishreload
I set apache to listen to port 8080
/etc/apache2/ports.conf
Listen 8080
Listen 443
Here is the relevant apache virtualhost lines for the host:
<VirtualHost 149.210.243.33:8080 [2a01:7c8:aabb:70:5054:ff:fe36:327f]:8080>
DocumentRoot /home/dewebshop/magento2/pub
</VirtualHost>
<VirtualHost 149.210.243.33:443 [2a01:7c8:aabb:70:5054:ff:fe36:327f]:443>
DocumentRoot /home/dewebshop/magento2/pub
ProxyPreserveHost On
# Forward requests to Varnish
# Forward requests to Varnish
ProxyPass / http://127.0.0.1:80/
ProxyPassReverse / http://127.0.0.1:80/
RequestHeader set X-Forwarded-Proto "https"
RequestHeader set X-Forwarded-Port "443"
</VirtualHost>
Here is apachectl -S
root@ubuntu:~$ sudo apachectl -S
VirtualHost configuration:
[2a01:7c8:aabb:70:5054:ff:fe36:327f]:8080 dewebshop.online (/etc/apache2/sites-enabled/dewebshop.online.conf:1)
[2a01:7c8:aabb:70:5054:ff:fe36:327f]:443 dewebshop.online (/etc/apache2/sites-enabled/dewebshop.online.conf:51)
149.210.243.33:8080 dewebshop.online (/etc/apache2/sites-enabled/dewebshop.online.conf:1)
149.210.243.33:443 dewebshop.online (/etc/apache2/sites-enabled/dewebshop.online.conf:51)
ServerRoot: "/etc/apache2"
Main DocumentRoot: "/var/www/html"
Main ErrorLog: "/var/log/apache2/error.log"
Mutex fcgid-pipe: using_defaults
Mutex authdigest-opaque: using_defaults
Mutex watchdog-callback: using_defaults
Mutex proxy-balancer-shm: using_defaults
Mutex rewrite-map: using_defaults
Mutex ssl-stapling-refresh: using_defaults
Mutex authdigest-client: using_defaults
Mutex fcgid-proctbl: using_defaults
Mutex ssl-stapling: using_defaults
Mutex proxy: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/var/run/apache2/" mechanism=default
PidFile: "/var/run/apache2/apache2.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
Define: ENABLE_USR_LIB_CGI_BIN
User: name="www-data" id=33
Group: name="www-data" id=33
This is the /etc/varnish/default.vcl (Created by magento admin panel)
# VCL version 5.0 is not supported so it should be 4.0 even though actually used Varnish version is 6
vcl 4.0;
import std;
# The minimal Varnish version is 6.0
# For SSL offloading, pass the following header in your proxy server or load balancer: 'X-Forwarded-Proto: https'
backend default {
.host = "127.0.0.1";
.port = "8080";
.first_byte_timeout = 600s;
.probe = {
# .url = "/health_check.php";
.url = "/";
.timeout = 2s;
.interval = 5s;
.window = 10;
.threshold = 5;
}
}
acl purge {
"localhost";
}
sub vcl_recv {
#added because magento sucks
# if (req.http.host) {
# set req.http.Host = req.http.host;
# }
if (req.restarts > 0) {
set req.hash_always_miss = true;
}
if (req.method == "PURGE") {
if (client.ip !~ purge) {
return (synth(405, "Method not allowed"));
}
# To use the X-Pool header for purging varnish during automated deployments, make sure the X-Pool header
# has been added to the response in your backend server config. This is used, for example, by the
# capistrano-magento2 gem for purging old content from varnish during it's deploy routine.
if (!req.http.X-Magento-Tags-Pattern && !req.http.X-Pool) {
return (synth(400, "X-Magento-Tags-Pattern or X-Pool header required"));
}
if (req.http.X-Magento-Tags-Pattern) {
ban("obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern);
}
if (req.http.X-Pool) {
ban("obj.http.X-Pool ~ " + req.http.X-Pool);
}
return (synth(200, "Purged"));
}
if (req.method != "GET" &&
req.method != "HEAD" &&
req.method != "PUT" &&
req.method != "POST" &&
req.method != "TRACE" &&
req.method != "OPTIONS" &&
req.method != "DELETE") {
/* Non-RFC2616 or CONNECT which is weird. */
return (pipe);
}
# We only deal with GET and HEAD by default
if (req.method != "GET" && req.method != "HEAD") {
return (pass);
}
# Bypass customer, shopping cart, checkout
if (req.url ~ "/customer" || req.url ~ "/checkout") {
return (pass);
}
# Bypass health check requests
if (req.url ~ "^/(pub/)?(health_check.php)$") {
return (pass);
}
# Set initial grace period usage status
set req.http.grace = "none";
# normalize url in case of leading HTTP scheme and domain
set req.url = regsub(req.url, "^http[s]?://", "");
# collect all cookies
std.collect(req.http.Cookie);
# Remove all marketing get parameters to minimize the cache objects
if (req.url ~ "(\?|&)(gclid|cx|ie|cof|siteurl|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|_bta_[a-z]+)=") {
set req.url = regsuball(req.url, "(gclid|cx|ie|cof|siteurl|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|_bta_[a-z]+)=[-_A-z0-9+()%.]+&?", "");
set req.url = regsub(req.url, "[?|&]+$", "");
}
# Static files caching
if (req.url ~ "^/(pub/)?(media|static)/") {
# Static files should not be cached by default
return (pass);
# But if you use a few locales and don't use CDN you can enable caching static files by commenting previous line (#return (pass);) and uncommenting next 3 lines
#unset req.http.Https;
#unset req.http.X-Forwarded-Proto;
#unset req.http.Cookie;
}
# Bypass authenticated GraphQL requests without a X-Magento-Cache-Id
if (req.url ~ "/graphql" && !req.http.X-Magento-Cache-Id && req.http.Authorization ~ "^Bearer") {
return (pass);
}
return (hash);
}
sub vcl_hash {
if ((req.url !~ "/graphql" || !req.http.X-Magento-Cache-Id) && req.http.cookie ~ "X-Magento-Vary=") {
hash_data(regsub(req.http.cookie, "^.*?X-Magento-Vary=([^;]+);*.*$", "\1"));
}
# To make sure http users don't see ssl warning
if (req.http.X-Forwarded-Proto) {
hash_data(req.http.X-Forwarded-Proto);
}
if (req.url ~ "/graphql") {
call process_graphql_headers;
}
}
sub process_graphql_headers {
if (req.http.X-Magento-Cache-Id) {
hash_data(req.http.X-Magento-Cache-Id);
# When the frontend stops sending the auth token, make sure users stop getting results cached for logged-in users
if (req.http.Authorization ~ "^Bearer") {
hash_data("Authorized");
}
}
if (req.http.Store) {
hash_data(req.http.Store);
}
if (req.http.Content-Currency) {
hash_data(req.http.Content-Currency);
}
}
sub vcl_backend_response {
set beresp.grace = 3d;
if (beresp.http.content-type ~ "text") {
set beresp.do_esi = true;
}
if (bereq.url ~ "\.js$" || beresp.http.content-type ~ "text") {
set beresp.do_gzip = true;
}
if (beresp.http.X-Magento-Debug) {
set beresp.http.X-Magento-Cache-Control = beresp.http.Cache-Control;
}
# cache only successfully responses and 404s that are not marked as private
if ((beresp.status != 200 && beresp.status != 404) || beresp.http.Cache-Control ~ "private") {
set beresp.uncacheable = true;
set beresp.ttl = 86400s;
return (deliver);
}
# validate if we need to cache it and prevent from setting cookie
if (beresp.ttl > 0s && (bereq.method == "GET" || bereq.method == "HEAD")) {
# Collapse beresp.http.set-cookie in order to merge multiple set-cookie headers
# Although it is not recommended to collapse set-cookie header,
# it is safe to do it here as the set-cookie header is removed below
std.collect(beresp.http.set-cookie);
# Do not cache the response under current cache key (hash),
# if the response has X-Magento-Vary but the request does not.
if ((bereq.url !~ "/graphql" || !bereq.http.X-Magento-Cache-Id)
&& bereq.http.cookie !~ "X-Magento-Vary="
&& beresp.http.set-cookie ~ "X-Magento-Vary=") {
set beresp.ttl = 0s;
set beresp.uncacheable = true;
}
unset beresp.http.set-cookie;
}
# If page is not cacheable then bypass varnish for 2 minutes as Hit-For-Pass
if (beresp.ttl <= 0s ||
beresp.http.Surrogate-control ~ "no-store" ||
(!beresp.http.Surrogate-Control &&
beresp.http.Cache-Control ~ "no-cache|no-store") ||
beresp.http.Vary == "*") {
# Mark as Hit-For-Pass for the next 2 minutes
set beresp.ttl = 120s;
set beresp.uncacheable = true;
}
# If the cache key in the Magento response doesn't match the one that was sent in the request, don't cache under the request's key
if (bereq.url ~ "/graphql" && bereq.http.X-Magento-Cache-Id && bereq.http.X-Magento-Cache-Id != beresp.http.X-Magento-Cache-Id) {
set beresp.ttl = 0s;
set beresp.uncacheable = true;
}
return (deliver);
}
sub vcl_deliver {
if (obj.uncacheable) {
set resp.http.X-Magento-Cache-Debug = "UNCACHEABLE";
} else if (obj.hits) {
set resp.http.X-Magento-Cache-Debug = "HIT";
set resp.http.Grace = req.http.grace;
} else {
set resp.http.X-Magento-Cache-Debug = "MISS";
}
# Not letting browser to cache non-static files.
if (resp.http.Cache-Control !~ "private" && req.url !~ "^/(pub/)?(media|static)/") {
set resp.http.Pragma = "no-cache";
set resp.http.Expires = "-1";
set resp.http.Cache-Control = "no-store, no-cache, must-revalidate, max-age=0";
}
if (!resp.http.X-Magento-Debug) {
unset resp.http.Age;
}
unset resp.http.X-Magento-Debug;
unset resp.http.X-Magento-Tags;
unset resp.http.X-Powered-By;
unset resp.http.Server;
unset resp.http.X-Varnish;
unset resp.http.Via;
unset resp.http.Link;
}
sub vcl_hit {
if (obj.ttl >= 0s) {
# Hit within TTL period
return (deliver);
}
if (std.healthy(req.backend_hint)) {
if (obj.ttl + 300s > 0s) {
# Hit after TTL expiration, but within grace period
set req.http.grace = "normal (healthy server)";
return (deliver);
} else {
# Hit after TTL and grace expiration
return (restart);
}
} else {
# server is not healthy, retrieve from cache
set req.http.grace = "unlimited (unhealthy server)";
return (deliver);
}
}
The request does pass through varnish:
sudo varnishlog
* << Request >> 18
- Begin req 17 rxreq
- Timestamp Start: 1724754038.213498 0.000000 0.000000
- Timestamp Req: 1724754038.213498 0.000000 0.000000
- VCL_use boot
- ReqStart 127.0.0.1 57130 a0
- ReqMethod GET
- ReqURL /
- ReqProtocol HTTP/1.1
- ReqHeader Host: dewebshop.online
- ReqHeader Cache-Control: max-age=0
- ReqHeader Sec-Ch-Ua: "Not)A;Brand";v="99", "Google Chrome";v="127", "Chromium";v="127"
- ReqHeader Sec-Ch-Ua-Mobile: ?0
- ReqHeader Sec-Ch-Ua-Platform: "Windows"
- ReqHeader Dnt: 1
- ReqHeader Upgrade-Insecure-Requests: 1
- ReqHeader User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36
- ReqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
- ReqHeader Sec-Fetch-Site: none
- ReqHeader Sec-Fetch-Mode: navigate
- ReqHeader Sec-Fetch-User: ?1
- ReqHeader Sec-Fetch-Dest: document
- ReqHeader Accept-Encoding: gzip, deflate, br, zstd
- ReqHeader Accept-Language: en-US,en;q=0.9,nl;q=0.8,tr;q=0.7
- ReqHeader Cookie: private_content_version=5bbc2e20c413c9fb6072b732fb16c494; lang=en; s_fid=406394D596AD0356-05F76B80A46CF07E; s_cc=true; redirect=1; testing=1; sid=c60abda9a8bdc8b9730fa90cedae6544; form_key=pNkAyrsXTQKRwQmt; apt.uid=AP-XD7ZED5OKDHG-2-1-17245986379
- ReqHeader Priority: u=0, i
- ReqHeader X-Forwarded-Proto: https
- ReqHeader X-Forwarded-Port: 443
- ReqHeader X-Forwarded-For: 87.210.82.16
- ReqHeader X-Forwarded-Host: dewebshop.online
- ReqHeader X-Forwarded-Server: dewebshop.online
- ReqHeader Connection: Keep-Alive
- ReqUnset X-Forwarded-For: 87.210.82.16
- ReqHeader X-Forwarded-For: 87.210.82.16, 127.0.0.1
- VCL_call RECV
- ReqHeader grace: none
- ReqURL /
- VCL_return hash
- ReqUnset Accept-Encoding: gzip, deflate, br, zstd
- ReqHeader Accept-Encoding: gzip
- VCL_call HASH
- VCL_return lookup
- Hit 3 -1597.631370 259200.000000 0.000000
- VCL_call HIT
- VCL_return restart
- Timestamp Restart: 1724754038.213624 0.000125 0.000125
- Link req 19 restart
- End
* << BeReq >> 20
- Begin bereq 19 fetch
- VCL_use boot
- Timestamp Start: 1724754038.213698 0.000000 0.000000
- BereqMethod GET
- BereqURL /
- BereqProtocol HTTP/1.1
- BereqHeader Host: dewebshop.online
- BereqHeader Sec-Ch-Ua: "Not)A;Brand";v="99", "Google Chrome";v="127", "Chromium";v="127"
- BereqHeader Sec-Ch-Ua-Mobile: ?0
- BereqHeader Sec-Ch-Ua-Platform: "Windows"
- BereqHeader Dnt: 1
- BereqHeader Upgrade-Insecure-Requests: 1
- BereqHeader User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36
- BereqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
- BereqHeader Sec-Fetch-Site: none
- BereqHeader Sec-Fetch-Mode: navigate
- BereqHeader Sec-Fetch-User: ?1
- BereqHeader Sec-Fetch-Dest: document
- BereqHeader Accept-Language: en-US,en;q=0.9,nl;q=0.8,tr;q=0.7
- BereqHeader Cookie: private_content_version=5bbc2e20c413c9fb6072b732fb16c494; lang=en; s_fid=406394D596AD0356-05F76B80A46CF07E; s_cc=true; redirect=1; testing=1; sid=c60abda9a8bdc8b9730fa90cedae6544; form_key=pNkAyrsXTQKRwQmt; apt.uid=AP-XD7ZED5OKDHG-2-1-17245986379
- BereqHeader Priority: u=0, i
- BereqHeader X-Forwarded-Proto: https
- BereqHeader X-Forwarded-Port: 443
- BereqHeader X-Forwarded-Host: dewebshop.online
- BereqHeader X-Forwarded-Server: dewebshop.online
- BereqHeader X-Forwarded-For: 87.210.82.16, 127.0.0.1
- BereqHeader Accept-Encoding: gzip
- BereqHeader grace: none
- BereqHeader X-Varnish: 20
- VCL_call BACKEND_FETCH
- VCL_return fetch
- Timestamp Fetch: 1724754038.213723 0.000024 0.000024
- Timestamp Connected: 1724754038.213793 0.000094 0.000069
- BackendOpen 29 default 127.0.0.1 8080 127.0.0.1 42864 connect
- Timestamp Bereq: 1724754038.213859 0.000160 0.000066
- Timestamp Beresp: 1724754038.214681 0.000982 0.000821
- BerespProtocol HTTP/1.1
- BerespStatus 200
- BerespReason OK
- BerespHeader Date: Tue, 27 Aug 2024 10:20:38 GMT
- BerespHeader Server: Apache
- BerespHeader Upgrade: h2,h2c
- BerespHeader Connection: Upgrade
- BerespHeader Last-Modified: Mon, 26 Aug 2024 22:31:01 GMT
- BerespHeader ETag: "29b0-6209dae040bba-gzip"
- BerespHeader Accept-Ranges: bytes
- BerespHeader Vary: Accept-Encoding
- BerespHeader Content-Encoding: gzip
- BerespHeader Content-Length: 3121
- BerespHeader Content-Type: text/html
- TTL RFC 120 10 0 1724754038 1724754038 1724754038 0 0 cacheable
- VCL_call BACKEND_RESPONSE
- TTL VCL 120 259200 0 1724754038 cacheable
- VCL_return deliver
- Timestamp Process: 1724754038.214747 0.001048 0.000066
- Filters gunzip esi_gzip
- BerespUnset Content-Encoding: gzip
- BerespUnset Content-Length: 3121
- BerespUnset ETag: "29b0-6209dae040bba-gzip"
- BerespHeader ETag: W/"29b0-6209dae040bba-gzip"
- BerespHeader Content-Encoding: gzip
- Storage malloc s0
- Fetch_Body 3 length -
- Gzip G F E 10672 3138 80 25024 25034
- Gzip U F - 3121 10672 80 80 24903
- BackendClose 29 default recycle
- Timestamp BerespBody: 1724754038.215567 0.001868 0.000820
- Length 3138
- BereqAcct 1935 0 1935 305 3121 3426
- End
* << Request >> 19
- Begin req 18 restart
- Timestamp Start: 1724754038.213624 0.000125 0.000000
- ReqStart 127.0.0.1 57130 a0
- ReqMethod GET
- ReqURL /
- ReqProtocol HTTP/1.1
- ReqHeader Host: dewebshop.online
- ReqHeader Cache-Control: max-age=0
- ReqHeader Sec-Ch-Ua: "Not)A;Brand";v="99", "Google Chrome";v="127", "Chromium";v="127"
- ReqHeader Sec-Ch-Ua-Mobile: ?0
- ReqHeader Sec-Ch-Ua-Platform: "Windows"
- ReqHeader Dnt: 1
[cut off becau8se max 40.000 chars per post]
To recap: The problem is: Apache is running a virtualdomain from /home/dewebshop/magento2/pub As soon as i install and configure varnish, the document served comes from /var/www/html
I am tempted to make a symbolic link for /var/www/html to /homr/dewebshop/magento2/pub but i think there must be someone wiser than me with a better solution ;)
I see that your Varnish backend is proxying to 127.0.0.1, whereas the relevant virtual hosts are listening in IP 149.210.243.33.
There are 2 potential solutions:
.host property to 149.210.243.33.127.0.0.1 match those virtual hosts.