Search code examples
nginxmagentovarnish

Magento 2 with Varnish and Nginx as SSL termination - backend server is serving


I am setting up a Magento server with Nginx SSL termination and varnish nginx and varnish 5.1 are installed in dedicated host 192.168.1.251 (ubuntu) & Magento on 192.168.1.250 (ubuntu)

nginx 1.6 (192.168.1.251:443 or https://mywebsite.com/) + varnish (127.0.0.1:6081) -> magento 2.3 (192.168.1.250:8080)

problem is content like jpg,svg etc are served from 192.168.1.250 i.e my backend server directly and scripts are blocked due to CORS ref below image chrome DevTools

if I access 192.168.1.251:6081 i.e varnish host & port all the contents are coming from the backend server Chrome Devtool

nginx ssl termination config:

server {
        server_name mywebsite.com;
        listen 80;
        return 301 https://mywebsite.com$request_uri;
    }

server {
    listen 443 ssl http2;
    server_name mywebsite.com;
    ssl_certificate /etc/nginx/ssl/cert.pem;
    ssl_certificate_key /etc/nginx/ssl/key.pem;
    ssl_dhparam /etc/ssl/certs/dhparam.pem;
    ssl_session_timeout 1d;
    ssl_session_cache shared:SSL:50m;
    ssl_session_tickets off;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM- 
     SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS;
        ssl_prefer_server_ciphers on;

        resolver 8.8.8.8 8.8.4.4 valid=300s;
    resolver_timeout 30s;
    keepalive_timeout 300s;

        location / {
        proxy_pass http://127.0.0.1:6081;
       proxy_set_header Host $http_host;
      proxy_set_header X-Forwarded-Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header Ssl-Offloaded 1;
       proxy_set_header X-Forwarded-Proto https;
       proxy_set_header X-Forwarded-Port 443;
proxy_headers_hash_max_size 512;
      proxy_headers_hash_bucket_size 128;
      #proxy_hide_header X-Varnish;
      #proxy_hide_header Via;
      proxy_set_header X-Forwarded-Proto $scheme;
    }
}

varnish config:

# VCL version 5.0 is not supported so it should be 4.0 even though actually used Varnish version is 5
vcl 4.0;

import std;
# The minimal Varnish version is 5.0
# For SSL offloading, pass the following header in your proxy server or load balancer: 'X-Forwarded-Proto: https'

backend default {
    .host = "192.168.1.250";
    .port = "8080";
    .first_byte_timeout = 600s;
    .probe = {
        .url = "/pub/health_check.php";
        .timeout = 2s;
        .interval = 5s;
        .window = 10;
        .threshold = 5;
   }
}

acl purge {
    "localhost";
    "192.168.1.250";
}

sub vcl_recv {
    if (req.method == "PURGE") {
        if (client.ip !~ purge) {
            return (synth(405, "Method not allowed"));
        }
        # To use the X-Pool header for purging varnish during automated deployments, make sure the X-Pool header
        # has been added to the response in your backend server config. This is used, for example, by the
        # capistrano-magento2 gem for purging old content from varnish during it's deploy routine.
        if (!req.http.X-Magento-Tags-Pattern && !req.http.X-Pool) {
            return (synth(400, "X-Magento-Tags-Pattern or X-Pool header required"));
        }
        if (req.http.X-Magento-Tags-Pattern) {
          ban("obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern);
        }
        if (req.http.X-Pool) {
          ban("obj.http.X-Pool ~ " + req.http.X-Pool);
        }
        return (synth(200, "Purged"));
    }

    if (req.method != "GET" &&
        req.method != "HEAD" &&
        req.method != "PUT" &&
        req.method != "POST" &&
        req.method != "TRACE" &&
        req.method != "OPTIONS" &&
        req.method != "DELETE") {
          /* Non-RFC2616 or CONNECT which is weird. */
          return (pipe);
    }

    # We only deal with GET and HEAD by default
    if (req.method != "GET" && req.method != "HEAD") {
        return (pass);
    }

    # Bypass shopping cart, checkout and search requests
    if (req.url ~ "/checkout" || req.url ~ "/catalogsearch") {
        return (pass);
    }

    # Bypass health check requests
    if (req.url ~ "/pub/health_check.php") {
        return (pass);
    }

    # Set initial grace period usage status
    set req.http.grace = "none";

    # normalize url in case of leading HTTP scheme and domain
    set req.url = regsub(req.url, "^http[s]?://", "");

    # collect all cookies
    std.collect(req.http.Cookie);

    # Compression filter. See https://www.varnish-cache.org/trac/wiki/FAQ/Compression
    if (req.http.Accept-Encoding) {
        if (req.url ~ "\.(jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|flv)$") {
            # No point in compressing these
            unset req.http.Accept-Encoding;
        } elsif (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
        } elsif (req.http.Accept-Encoding ~ "deflate" && req.http.user-agent !~ "MSIE") {
            set req.http.Accept-Encoding = "deflate";
        } else {
            # unknown algorithm
            unset req.http.Accept-Encoding;
        }
    }

    # Remove all marketing get parameters to minimize the cache objects
    if (req.url ~ "(\?|&)(gclid|cx|ie|cof|siteurl|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|_bta_[a-z]+)=") {
        set req.url = regsuball(req.url, "(gclid|cx|ie|cof|siteurl|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|_bta_[a-z]+)=[-_A-z0-9+()%.]+&?", "");
        set req.url = regsub(req.url, "[?|&]+$", "");
    }

    # Static files caching
    if (req.url ~ "^/(pub/)?(media|static)/") {
        # Static files should not be cached by default
        return (pass);

        # But if you use a few locales and don't use CDN you can enable caching static files by commenting previous line (#return (pass);) and uncommenting next 3 lines
        #unset req.http.Https;
        #unset req.http.X-Forwarded-Proto;
        #unset req.http.Cookie;
    }

    return (hash);
}

sub vcl_hash {
    if (req.http.cookie ~ "X-Magento-Vary=") {
        hash_data(regsub(req.http.cookie, "^.*?X-Magento-Vary=([^;]+);*.*$", "\1"));
    }

    # For multi site configurations to not cache each other's content
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }

    # To make sure http users don't see ssl warning
    if (req.http.X-Forwarded-Proto) {
        hash_data(req.http.X-Forwarded-Proto);
    }
    

    if (req.url ~ "/graphql") {
        call process_graphql_headers;
    }
}

sub process_graphql_headers {
    if (req.http.Store) {
        hash_data(req.http.Store);
    }
    if (req.http.Content-Currency) {
        hash_data(req.http.Content-Currency);
    }
}

sub vcl_backend_response {

    set beresp.grace = 3d;

    if (beresp.http.content-type ~ "text") {
        set beresp.do_esi = true;
    }

    if (bereq.url ~ "\.js$" || beresp.http.content-type ~ "text") {
        set beresp.do_gzip = true;
    }

    if (beresp.http.X-Magento-Debug) {
        set beresp.http.X-Magento-Cache-Control = beresp.http.Cache-Control;
    }

    # cache only successfully responses and 404s
    if (beresp.status != 200 && beresp.status != 404) {
        set beresp.ttl = 0s;
        set beresp.uncacheable = true;
        return (deliver);
    } elsif (beresp.http.Cache-Control ~ "private") {
        set beresp.uncacheable = true;
        set beresp.ttl = 86400s;
        return (deliver);
    }

    # validate if we need to cache it and prevent from setting cookie
    if (beresp.ttl > 0s && (bereq.method == "GET" || bereq.method == "HEAD")) {
        unset beresp.http.set-cookie;
    }

   # If page is not cacheable then bypass varnish for 2 minutes as Hit-For-Pass
   if (beresp.ttl <= 0s ||
       beresp.http.Surrogate-control ~ "no-store" ||
       (!beresp.http.Surrogate-Control &&
       beresp.http.Cache-Control ~ "no-cache|no-store") ||
       beresp.http.Vary == "*") {
        # Mark as Hit-For-Pass for the next 2 minutes
        set beresp.ttl = 120s;
        set beresp.uncacheable = true;
    }

    return (deliver);
}

sub vcl_deliver {
    if (resp.http.X-Magento-Debug) {
        if (resp.http.x-varnish ~ " ") {
            set resp.http.X-Magento-Cache-Debug = "HIT";
            set resp.http.Grace = req.http.grace;
        } else {
            set resp.http.X-Magento-Cache-Debug = "MISS";
        }
    } else {
        unset resp.http.Age;
    }

    # Not letting browser to cache non-static files.
    if (resp.http.Cache-Control !~ "private" && req.url !~ "^/(pub/)?(media|static)/") {
        set resp.http.Pragma = "no-cache";
        set resp.http.Expires = "-1";
        set resp.http.Cache-Control = "no-store, no-cache, must-revalidate, max-age=0";
    }
    
    unset resp.http.X-Magento-Debug;
    unset resp.http.X-Magento-Tags;
    unset resp.http.X-Powered-By;
    unset resp.http.Server;
    unset resp.http.X-Varnish;
    unset resp.http.Via;
    unset resp.http.Link;
}

sub vcl_hit {
    if (obj.ttl >= 0s) {
        # Hit within TTL period
        return (deliver);
    }
    if (std.healthy(req.backend_hint)) {
        if (obj.ttl + 300s > 0s) {
            # Hit after TTL expiration, but within grace period
            set req.http.grace = "normal (healthy server)";
            return (deliver);
        } else {
            # Hit after TTL and grace expiration
            return (miss);
        }
    } else {
        # server is not healthy, retrieve from cache
        set req.http.grace = "unlimited (unhealthy server)";
        return (deliver);
    }
}

magento nginx config:

upstream fastcgi_backend {
    server unix:/run/php/php7.2-fpm.sock;
}

server {
    server_name mywebsite.com;
    listen 8080;
    add_header Strict-Transport-Security "max-age=15768000;includeSubdomains;preload";
    add_header X-Frame-Options SAMEORIGIN;
    add_header X-Content-Type-Options nosniff;
    set $MAGE_ROOT /var/www/html/magento;
    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;
    include /var/www/html/magento/nginx.conf.sample;
}

nginx.conf.sample

## Example configuration:
# upstream fastcgi_backend {
#    # use tcp connection
#    # server  127.0.0.1:9000;
#    # or socket
#    server   unix:/var/run/php/php7.0-fpm.sock;
# }
# server {
#    listen 80;
#    server_name mage.dev;
#    set $MAGE_ROOT /var/www/magento2;
#    set $MAGE_DEBUG_SHOW_ARGS 1;
#    include /vagrant/magento2/nginx.conf.sample;
# }
#
## Optional override of deployment mode. We recommend you use the
## command 'bin/magento deploy:mode:set' to switch modes instead.
##
## set $MAGE_MODE default; # or production or developer
##
## If you set MAGE_MODE in server config, you must pass the variable into the
## PHP entry point blocks, which are indicated below. You can pass
## it in using:
##
## fastcgi_param  MAGE_MODE $MAGE_MODE;
##
## In production mode, you should uncomment the 'expires' directive in the /static/ location block

root $MAGE_ROOT/pub;

index index.php;
autoindex off;
charset UTF-8;
error_page 404 403 = /errors/404.php;
#add_header "X-UA-Compatible" "IE=Edge";


# Deny access to sensitive files
location /.user.ini {
    deny all;
}

# PHP entry point for setup application
location ~* ^/setup($|/) {
    root $MAGE_ROOT;
    location ~ ^/setup/index.php {
        fastcgi_pass   fastcgi_backend;

        fastcgi_param  PHP_FLAG  "session.auto_start=off \n suhosin.session.cryptua=off";
        fastcgi_param  PHP_VALUE "memory_limit=756M \n max_execution_time=600";
        fastcgi_read_timeout 600s;
        fastcgi_connect_timeout 600s;

        fastcgi_index  index.php;
        fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
        include        fastcgi_params;
    }

    location ~ ^/setup/(?!pub/). {
        deny all;
    }

    location ~ ^/setup/pub/ {
        add_header X-Frame-Options "SAMEORIGIN";
    }
}

# PHP entry point for update application
location ~* ^/update($|/) {
    root $MAGE_ROOT;

    location ~ ^/update/index.php {
        fastcgi_split_path_info ^(/update/index.php)(/.+)$;
        fastcgi_pass   fastcgi_backend;
        fastcgi_index  index.php;
        fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
        fastcgi_param  PATH_INFO        $fastcgi_path_info;
        include        fastcgi_params;
    }

    # Deny everything but index.php
    location ~ ^/update/(?!pub/). {
        deny all;
    }

    location ~ ^/update/pub/ {
        add_header X-Frame-Options "SAMEORIGIN";
    }
}

location / {
    try_files $uri $uri/ /index.php$is_args$args;
}

location /pub/ {
    location ~ ^/pub/media/(downloadable|customer|import|custom_options|theme_customization/.*\.xml) {
        deny all;
    }
    alias $MAGE_ROOT/pub/;
    add_header X-Frame-Options "SAMEORIGIN";
}

location /static/ {
    # Uncomment the following line in production mode
    # expires max;

    # Remove signature of the static files that is used to overcome the browser cache
    location ~ ^/static/version {
        rewrite ^/static/(version\d*/)?(.*)$ /static/$2 last;
    }

    location ~* \.(ico|jpg|jpeg|png|gif|svg|js|css|swf|eot|ttf|otf|woff|woff2|html|json)$ {
        add_header Cache-Control "public";
        add_header X-Frame-Options "SAMEORIGIN";
        include /etc/nginx/cors.conf;
        expires +1y;

        if (!-f $request_filename) {
            rewrite ^/static/(version\d*/)?(.*)$ /static.php?resource=$2 last;
        }
    }
    location ~* \.(zip|gz|gzip|bz2|csv|xml)$ {
        add_header Cache-Control "no-store";
        add_header X-Frame-Options "SAMEORIGIN";
        include /etc/nginx/cors.conf;
        expires    off;

        if (!-f $request_filename) {
           rewrite ^/static/(version\d*/)?(.*)$ /static.php?resource=$2 last;
        }
    }
    if (!-f $request_filename) {
        rewrite ^/static/(version\d*/)?(.*)$ /static.php?resource=$2 last;
    }
    add_header X-Frame-Options "SAMEORIGIN";
        include /etc/nginx/cors.conf;
}

location /media/ {
    try_files $uri $uri/ /get.php$is_args$args;

    location ~ ^/media/theme_customization/.*\.xml {
        deny all;
    }

    location ~* \.(ico|jpg|jpeg|png|gif|svg|js|css|swf|eot|ttf|otf|woff|woff2)$ {
        add_header Cache-Control "public";
        add_header X-Frame-Options "SAMEORIGIN";
        include /etc/nginx/cors.conf;
        expires +1y;
        try_files $uri $uri/ /get.php$is_args$args;
    }
    location ~* \.(zip|gz|gzip|bz2|csv|xml)$ {
        add_header Cache-Control "no-store";
        add_header X-Frame-Options "SAMEORIGIN";
        include /etc/nginx/cors.conf;
        expires    off;
        try_files $uri $uri/ /get.php$is_args$args;
    }
    add_header X-Frame-Options "SAMEORIGIN";
        include /etc/nginx/cors.conf;
}

location /media/customer/ {
    deny all;
}

location /media/downloadable/ {
    deny all;
}

location /media/import/ {
    deny all;
}

location /media/custom_options/ {
    deny all;
}

location /errors/ {
    location ~* \.xml$ {
        deny all;
    }
}

# PHP entry point for main application
location ~ ^/(index|get|static|errors/report|errors/404|errors/503|health_check)\.php$ {
    try_files $uri =404;
    fastcgi_pass   fastcgi_backend;
    fastcgi_buffers 1024 4k;

    fastcgi_param  PHP_FLAG  "session.auto_start=off \n suhosin.session.cryptua=off";
    fastcgi_param  PHP_VALUE "memory_limit=756M \n max_execution_time=18000";
    fastcgi_read_timeout 600s;
    fastcgi_connect_timeout 600s;

    fastcgi_index  index.php;
    fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
    include        fastcgi_params;
}

gzip on;
gzip_disable "msie6";

gzip_comp_level 6;
gzip_min_length 1100;
gzip_buffers 16 8k;
gzip_proxied any;
gzip_types
    text/plain
    text/css
    text/js
    text/xml
    text/javascript
    application/javascript
    application/x-javascript
    application/json
    application/xml
    application/xml+rss
    image/svg+xml;
gzip_vary on;

# Banned locations (only reached if the earlier PHP entry point regexes don't match)
location ~* (\.php$|\.phtml$|\.htaccess$|\.git) {
    deny all;
}

cors.conf

add_header 'Access-Control-Allow-Origin' '*' 'always';
if ($request_method = 'OPTIONS') {
    add_header 'Access-Control-Allow-Origin' '*' 'always';
    add_header 'Access-Control-Allow-Headers' 'x-requested-with' 'always';
    add_header 'Access-Control-Max-Age' 86400 'always';
    add_header 'Content-Length' 0 'always';
    return 204;
}

please help me. I am ready to assist you with required information

Varnish log for 192.168.1.251

    @ThijsFeryn  varnishlog -g request -q "ReqUrl eq '/'"
*   << Request  >> 32772
-   Begin          req 32771 rxreq
-   Timestamp      Start: 1595910698.277032 0.000000 0.000000
-   Timestamp      Req: 1595910698.277032 0.000000 0.000000
-   ReqStart       192.168.1.29 65243
-   ReqMethod      GET
-   ReqURL         /
-   ReqProtocol    HTTP/1.1
-   ReqHeader      Host: 192.168.1.251:6081
-   ReqHeader      Connection: keep-alive
-   ReqHeader      Upgrade-Insecure-Requests: 1
-   ReqHeader      User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36
-   ReqHeader      Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
-   ReqHeader      Accept-Encoding: gzip, deflate
-   ReqHeader      Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
-   ReqHeader      X-Forwarded-For: 192.168.1.29
-   VCL_call       RECV
-   ReqHeader      grace: none
-   ReqURL         /
-   ReqUnset       Accept-Encoding: gzip, deflate
-   ReqHeader      Accept-Encoding: gzip
-   VCL_return     hash
-   VCL_call       HASH
-   VCL_return     lookup
-   Hit            8 86252.364472 259200.000000 0.000000
-   VCL_call       HIT
-   VCL_return     deliver
-   RespProtocol   HTTP/1.1
-   RespStatus     200
-   RespReason     OK
-   RespHeader     Date: Tue, 28 Jul 2020 04:29:08 GMT
-   RespHeader     Server: Apache/2.4.29 (Ubuntu)
-   RespHeader     Expires: Wed, 29 Jul 2020 04:29:09 GMT
-   RespHeader     Cache-Control: max-age=86400, public, s-maxage=86400
-   RespHeader     Pragma: cache
-   RespHeader     X-Magento-Tags: store,cms_b,cms_b_1,cms_b_footer_links_block,cms_b_14,cms_b_home-page-block,cms_p_2,cat_p
-   RespHeader     Content-Security-Policy-Report-Only: font-src 'self' 'unsafe-inline'; form-action geostag.cardinalcommerce.com geo.cardinalcommerce.com 1eafstag.cardinalcommerce.com 1eaf.cardinalcommerce.com centinelapistag.cardinalcommerce.com centinelapi.cardinalcomme
-   RespHeader     X-Content-Type-Options: nosniff
-   RespHeader     X-XSS-Protection: 1; mode=block
-   RespHeader     X-Frame-Options: SAMEORIGIN
-   RespHeader     Vary: Accept-Encoding
-   RespHeader     X-UA-Compatible: IE=edge
-   RespHeader     Content-Type: text/html; charset=UTF-8
-   RespHeader     Content-Encoding: gzip
-   RespHeader     X-Varnish: 32772 8
-   RespHeader     Age: 147
-   RespHeader     Via: 1.1 varnish (Varnish/5.2)
-   VCL_call       DELIVER
-   RespUnset      Age: 147
-   RespUnset      Pragma: cache
-   RespHeader     Pragma: no-cache
-   RespUnset      Expires: Wed, 29 Jul 2020 04:29:09 GMT
-   RespHeader     Expires: -1
-   RespUnset      Cache-Control: max-age=86400, public, s-maxage=86400
-   RespHeader     Cache-Control: no-store, no-cache, must-revalidate, max-age=0
-   RespUnset      X-Magento-Tags: store,cms_b,cms_b_1,cms_b_footer_links_block,cms_b_14,cms_b_home-page-block,cms_p_2,cat_p
-   RespUnset      Server: Apache/2.4.29 (Ubuntu)
-   RespUnset      X-Varnish: 32772 8
-   RespUnset      Via: 1.1 varnish (Varnish/5.2)
-   VCL_return     deliver
-   Timestamp      Process: 1595910698.277253 0.000221 0.000221
-   RespHeader     Accept-Ranges: bytes
-   RespHeader     Transfer-Encoding: chunked
-   RespHeader     Connection: keep-alive
-   Link           req 32773 esi
-   Timestamp      Resp: 1595910698.277633 0.000602 0.000380
-   ReqAcct        437 0 437 2366 12979 15345
-   End
**  << Request  >> 32773
--  Begin          req 32772 esi
--  ReqURL         /page_cache/block/esi/blocks/%5B%22catalog.topnav%22%5D/handles/WyJkZWZhdWx0IiwiY21zX2luZGV4X2luZGV4IiwiY21zX3BhZ2VfdmlldyJd/
--  ReqUnset       Host: 192.168.1.251:6081
--  ReqHeader      Host: 192.168.1.250:8080
--  ReqUnset       Accept-Encoding: gzip, deflate
--  ReqHeader      Accept-Encoding: gzip
--  Timestamp      Start: 1595910698.277385 0.000000 0.000000
--  ReqStart       192.168.1.29 65243
--  ReqMethod      GET
--  ReqURL         /page_cache/block/esi/blocks/%5B%22catalog.topnav%22%5D/handles/WyJkZWZhdWx0IiwiY21zX2luZGV4X2luZGV4IiwiY21zX3BhZ2VfdmlldyJd/
--  ReqProtocol    HTTP/1.1
--  ReqHeader      Connection: keep-alive
--  ReqHeader      Upgrade-Insecure-Requests: 1
--  ReqHeader      User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36
--  ReqHeader      Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
--  ReqHeader      Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
--  ReqHeader      Host: 192.168.1.250:8080
--  ReqHeader      Accept-Encoding: gzip
--  ReqHeader      X-Forwarded-For: 192.168.1.29
--  VCL_call       RECV
--  ReqHeader      grace: none
--  ReqURL         /page_cache/block/esi/blocks/%5B%22catalog.topnav%22%5D/handles/WyJkZWZhdWx0IiwiY21zX2luZGV4X2luZGV4IiwiY21zX3BhZ2VfdmlldyJd/
--  ReqUnset       Accept-Encoding: gzip
--  ReqHeader      Accept-Encoding: gzip
--  VCL_return     hash
--  VCL_call       HASH
--  VCL_return     lookup
--  Hit            10 3455.551462 259200.000000 0.000000
--  VCL_call       HIT
--  VCL_return     deliver
--  RespProtocol   HTTP/1.1
--  RespStatus     200
--  RespReason     OK
--  RespHeader     Date: Tue, 28 Jul 2020 04:29:10 GMT
--  RespHeader     Server: Apache/2.4.29 (Ubuntu)
--  RespHeader     Expires: Tue, 28 Jul 2020 05:29:13 GMT
--  RespHeader     Cache-Control: max-age=3600, public, s-maxage=3600
--  RespHeader     Pragma: cache
--  RespHeader     X-Magento-Tags: cat_c,cat_c_38,cat_c_20,cat_c_11,cat_c_3,cat_c_9,cat_c_37
--  RespHeader     Content-Security-Policy-Report-Only: font-src 'self' 'unsafe-inline'; form-action geostag.cardinalcommerce.com geo.cardinalcommerce.com 1eafstag.cardinalcommerce.com 1eaf.cardinalcommerce.com centinelapistag.cardinalcommerce.com centinelapi.cardinalcomme
--  RespHeader     X-Content-Type-Options: nosniff
--  RespHeader     X-XSS-Protection: 1; mode=block
--  RespHeader     X-Frame-Options: SAMEORIGIN
--  RespHeader     Vary: Accept-Encoding
--  RespHeader     X-UA-Compatible: IE=edge
--  RespHeader     Content-Type: text/html; charset=UTF-8
--  RespHeader     Content-Encoding: gzip
--  RespHeader     X-Varnish: 32773 10
--  RespHeader     Age: 144
--  RespHeader     Via: 1.1 varnish (Varnish/5.2)
--  VCL_call       DELIVER
--  RespUnset      Age: 144
--  RespUnset      Pragma: cache
--  RespHeader     Pragma: no-cache
--  RespUnset      Expires: Tue, 28 Jul 2020 05:29:13 GMT
--  RespHeader     Expires: -1
--  RespUnset      Cache-Control: max-age=3600, public, s-maxage=3600
--  RespHeader     Cache-Control: no-store, no-cache, must-revalidate, max-age=0
--  RespUnset      X-Magento-Tags: cat_c,cat_c_38,cat_c_20,cat_c_11,cat_c_3,cat_c_9,cat_c_37
--  RespUnset      Server: Apache/2.4.29 (Ubuntu)
--  RespUnset      X-Varnish: 32773 10
--  RespUnset      Via: 1.1 varnish (Varnish/5.2)
--  VCL_return     deliver
--  Timestamp      Process: 1595910698.277408 0.000024 0.000024
--  RespHeader     Accept-Ranges: bytes
--  RespHeader     Content-Length: 738
--  Timestamp      Resp: 1595910698.277482 0.000097 0.000073
--  ReqAcct        0 0 0 0 723 723
--  END

Solution

  • The problem

    The issue you're experiencing is probably related to the fact that your Magento base URL is set to 192.168.1.250:8080.

    Magento will enforce that value if it notices the Host header (or the protocol scheme) doesn't match its own.

    So in your case, you're sending the following host header to Magento through Varnish:

    Host: 192.168.1.251:6081
    

    Magento doesn't recognize this, and starts using its own hostname in the URLs its generating for images, CSS, JS, and other related resources.

    These subordinate requests, result in the following Host header being sent:

    Host: 192.168.1.250:8080
    

    This fact supports your claim that you start from Varnish, but get a response back from the backend server. Although it's not 100% the case, the change of hostname in the URL, will have a similar effect.

    Side effect

    Another side effect from the URL change is that the protocol scheme might also differ. It is possible that you're entering through Varnish using HTTP, but that the Magento URLs return an HTTPS URL.

    Although this is not the original problem, it will contributed to mixed content, which your browser will block.

    The solution

    The best way to solve this problem is to work with a hostname.

    You suggested www.magento.test, that's a good idea. Set it as the Magento base URL, but make sure the hostname resolves to your Varnish IP.

    In the end it's all about what your HTTP request's Host header will send, and which protocol you're using.

    If you're HTTPS, make sure you have a TLS proxy in front of Varnish that terminates the TLS connection before passing it to Varnish.

    There's also a lazy way of solving the problem and that is by forcing the IP address of Magento as the Host header.

    Here's an example:

    curl -H "Host: 192.168.1.250:8080" http://192.168.1.251:6081
    

    You're sending this request to Varnish, but you're forcing the Host header of Magento. This is not really a viable solution if you're using a web browser, but for testing purposes it should prove the point.

    Conclusion

    • Please set your Magento base URL to the right value
    • Consider using a hostname
    • If the URL is an https:// URL, terminate TLS before entering Varnish.