Mixed content error when accessing HTTPS website using Hitch as a TLS Proxy with Varnish on WordPress?

The https version of my website is getting a mixed content error.

I have the following setup

Traffic on Port 443 is running the Hitch TLS Proxy which is forwarded to Varnish and finally ends in Apache Web Server.

What I have figured is the first request on my websites loads via HTTPS but all the subsequent request are processed directly by Varnish in HTTP.

I need help figuring out how to load all the urls over TLS on the client side.

Here is my Hitch TLS Proxy Configuration File

# Upstream server address.
backend = "[127.0.0.1]:8443"

pem-file = {
        cert = "/etc/letsencrypt/live/staging1.sainikbiswas.com/fullchain.pem"
        private-key = "/etc/letsencrypt/live/staging1.sainikbiswas.com/privkey.pem"
}

ocsp-dir = "/var/lib/hitch-ocsp"

ciphersuites = "TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256"

# List of allowed TLS ciphers.

ciphers = "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHAA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RRSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384"   
                            
# Enforce server cipher list order
#
# type: boolean
prefer-server-ciphers = off

tls-protos = TLSv1.2 TLSv1.3
alpn-protos = "h2, http/1.1"

workers = 4

backlog = 100
keepalive = 3600

chroot = ""
user = "hitch"
group = "hitch"

syslog = on
syslog-facility = "daemon"
daemon = on

write-ip = off
write-proxy-v1 = off
write-proxy-v2 = on
proxy-proxy = off
sni-nomatch-abort = off

Apache 2 Configuration File

<VirtualHost *:8080>
        ServerName staging1.sainikbiswas.com
        ServerAdmin [email protected]

        DocumentRoot "/home/sainikbiswas/domains/sainikbiswas-com/public"

        <Directory /home/sainikbiswas/domains/sainikbiswas-com/public>
            #Disable .htaccess file
            AllowOverride None

            #WordPress Rewrite Rules
            <IfModule mod_rewrite.c>
                RewriteEngine On
                RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
                RewriteBase /
                RewriteRule ^index\.php$ - [L]
                RewriteCond %{REQUEST_FILENAME} !-f
                RewriteCond %{REQUEST_FILENAME} !-d
                RewriteRule . /index.php [L]
            </IfModule>

            #Disable File Index
            Options -Indexes -FollowSymLinks +SymLinksIfOwnerMatch

            #Enable Access to the Document Root
            Require all granted
        </Directory>

</VirtualHost>

Varnish Configuration File

vcl 4.1;

import proxy;
import std;

# Default backend definition. Set this to point to your content server.
backend default {
    .host = "127.0.0.1";
    .port = "8080";
}

# Add hostnames, IP addresses and subnets that are allowed to purge content
acl purge {
    "localhost";
    "127.0.0.1";
    "::1";
}


sub vcl_recv {
        if ((req.http.X-Forwarded-Proto && req.http.X-Forwarded-Proto != "https") ||
            (req.http.Scheme && req.http.Scheme != "https")) {
                return (synth(750));
        } elseif (!req.http.X-Forwarded-Proto &&
                  !req.http.Scheme && !proxy.is_ssl()) {
                return (synth(750));
   }

# WordPress Varnish Configuration
# Remove empty query string parameters
    # e.g.: www.example.com/index.html?
    if (req.url ~ "\?$") {
        set req.url = regsub(req.url, "\?$", "");
    }

    # Remove port number from host header
    set req.http.Host = regsub(req.http.Host, ":[0-9]+", "");

    # Sorts query string parameters alphabetically for cache normalization purposes
    set req.url = std.querysort(req.url);

    # Remove the proxy header to mitigate the httpoxy vulnerability
    # See https://httpoxy.org/
    unset req.http.proxy;

# Add X-Forwarded-Proto header when using https

        if(!req.http.X-Forwarded-Proto) {
        if (proxy.is_ssl()) {
            set req.http.X-Forwarded-Proto = "https";
        } else {
            set req.http.X-Forwarded-Proto = "http";
        }
    }

    # Purge logic to remove objects from the cache.
    # Tailored to the Proxy Cache Purge WordPress plugin
    # See https://wordpress.org/plugins/varnish-http-purge/
    if(req.method == "PURGE") {
        if(!client.ip ~ purge) {
            return(synth(405,"PURGE not allowed for this IP address"));
        }
        if (req.http.X-Purge-Method == "regex") {
            ban("obj.http.x-url ~ " + req.url + " && obj.http.x-host == " + req.http.host);
            return(synth(200, "Purged"));
        }
        ban("obj.http.x-url == " + req.url + " && obj.http.x-host == " + req.http.host);
        return(synth(200, "Purged"));
    }


 # Only handle relevant HTTP request methods
    if (
        req.method != "GET" &&
        req.method != "HEAD" &&
        req.method != "PUT" &&
        req.method != "POST" &&
        req.method != "PATCH" &&
        req.method != "TRACE" &&
        req.method != "OPTIONS" &&
        req.method != "DELETE"
    ) {
        return (pipe);
    }

    # Remove tracking query string parameters used by analytics tools
    if (req.url ~ "(\?|&)(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteur
l)=") {
        set req.url = regsuball(req.url, "&(utm_source|utm_medium|utm_campaign|utm_content|gcli
d|cx|ie|cof|siteurl)=([A-z0-9_\-\.%25]+)", "");
        set req.url = regsuball(req.url, "\?(utm_source|utm_medium|utm_campaign|utm_content|gcl
id|cx|ie|cof|siteurl)=([A-z0-9_\-\.%25]+)", "?");
        set req.url = regsub(req.url, "\?&", "?");
        set req.url = regsub(req.url, "\?$", "");
    }

    # Only cache GET and HEAD requests
    if (req.method != "GET" && req.method != "HEAD") {
        set req.http.X-Cacheable = "NO:REQUEST-METHOD";
        return(pass);

 # Mark static files with the X-Static-File header, and remove any cookies
    # X-Static-File is also used in vcl_backend_response to identify static files
    if (req.url ~ "^[^?]*\.(7z|avi|bmp|bz2|css|csv|doc|docx|eot|flac|flv|gif|gz|ico|jpeg|jpg|js
|less|mka|mkv|mov|mp3|mp4|mpeg|mpg|odt|ogg|ogm|opus|otf|pdf|png|ppt|pptx|rar|rtf|svg|svgz|swf|t
ar|tbz|tgz|ttf|txt|txz|wav|webm|webp|woff|woff2|xls|xlsx|xml|xz|zip)(\?.*)?$") {
        set req.http.X-Static-File = "true";
        unset req.http.Cookie;
        return(hash);
    }

    # No caching of special URLs, logged in users and some plugins
    if (
        req.http.Cookie ~ "wordpress_(?!test_)[a-zA-Z0-9_]+|wp-postpass|comment_author_[a-zA-Z0
-9_]+|woocommerce_cart_hash|woocommerce_items_in_cart|wp_woocommerce_session_[a-zA-Z0-9]+|wordp
ress_logged_in_|comment_author|PHPSESSID" ||
        req.http.Authorization ||
        req.url ~ "add_to_cart" ||
        req.url ~ "edd_action" ||
        req.url ~ "nocache" ||
        req.url ~ "^/addons" ||
        req.url ~ "^/bb-admin" ||
        req.url ~ "^/bb-login.php" ||
        req.url ~ "^/bb-reset-password.php" ||
        req.url ~ "^/cart" ||
        req.url ~ "^/checkout" ||
        req.url ~ "^/control.php" ||
        req.url ~ "^/login" ||
        req.url ~ "^/logout" ||
        req.url ~ "^/lost-password" ||
        req.url ~ "^/my-account" ||
        req.url ~ "^/product" ||
        req.url ~ "^/register" ||
        
req.url ~ "^/register.php" ||
        req.url ~ "^/server-status" ||
        req.url ~ "^/signin" ||
        req.url ~ "^/signup" ||
        req.url ~ "^/stats" ||
        req.url ~ "^/wc-api" ||
        req.url ~ "^/wp-admin" ||
        req.url ~ "^/wp-comments-post.php" ||
        req.url ~ "^/wp-cron.php" ||
        req.url ~ "^/wp-login.php" ||
        req.url ~ "^/wp-activate.php" ||
        req.url ~ "^/wp-mail.php" ||
        req.url ~ "^/wp-login.php" ||
        req.url ~ "^\?add-to-cart=" ||
        req.url ~ "^\?wc-api=" ||
        req.url ~ "^/preview=" ||
        req.url ~ "^/\.well-known/acme-challenge/"
    ) {
             set req.http.X-Cacheable = "NO:Logged in/Got Sessions";
             if(req.http.X-Requested-With == "XMLHttpRequest") {
                     set req.http.X-Cacheable = "NO:Ajax";
             }
        return(pass);
    }

    # Remove any cookies left
    unset req.http.Cookie;
    return(hash);
}


sub vcl_synth {
        if (resp.status == 750) {
                set resp.status = 301;
                set resp.http.location = "https://" + req.http.Host + req.url;
                set resp.reason = "Moved";
                return (deliver);
        }
}

sub vcl_hash {
#WordPress Varnish Configuration

    if(req.http.X-Forwarded-Proto) {
        # Create cache variations depending on the request protocol
        hash_data(req.http.X-Forwarded-Proto);
    }
}

sub vcl_backend_response {
if(beresp.http.Vary) {
        set beresp.http.Vary = beresp.http.Vary + ", X-Forwarded-Proto";
    } else {
        set beresp.http.Vary = "X-Forwarded-Proto";
    }


# Inject URL & Host header into the object for asynchronous banning purposes
    set beresp.http.x-url = bereq.url;
    set beresp.http.x-host = bereq.http.host;

    # If we dont get a Cache-Control header from the backend
    # we default to 1h cache for all objects
    if (!beresp.http.Cache-Control) {
        set beresp.ttl = 1h;
        set beresp.http.X-Cacheable = "YES:Forced";
    }

    # If the file is marked as static we cache it for 1 day
    if (bereq.http.X-Static-File == "true") {
        unset beresp.http.Set-Cookie;
        set beresp.http.X-Cacheable = "YES:Forced";
        set beresp.ttl = 1d;
    }

        # Remove the Set-Cookie header when a specific Wordfence cookie is set
    if (beresp.http.Set-Cookie ~ "wfvt_|wordfence_verifiedHuman") {
            unset beresp.http.Set-Cookie;
         }

    if (beresp.http.Set-Cookie) {
        set beresp.http.X-Cacheable = "NO:Got Cookies";
    } elseif(beresp.http.Cache-Control ~ "private") {
        set beresp.http.X-Cacheable = "NO:Cache-Control=private";
    }
}


sub vcl_deliver {
    # Debug header
    if(req.http.X-Cacheable) {
        set resp.http.X-Cacheable = req.http.X-Cacheable;
    } elseif(obj.uncacheable) {
        if(!resp.http.X-Cacheable) {
            set resp.http.X-Cacheable = "NO:UNCACHEABLE";
        }
    } elseif(!resp.http.X-Cacheable) {
        set resp.http.X-Cacheable = "YES";
    }

    # Cleanup of headers
    unset resp.http.x-url;
    unset resp.http.x-host;
}

I tried implementing this from the Varnish Tutorial. I thought this would forward all HTTP requests to HTTPS. But most probably only the first request was redirected to HTTPS with the consequents links on the page being loaded via HTTP which gets blocked by the browser as mixed content errors.

vcl 4.1;

import proxy;

backend default {
    .host = "127.0.0.1";
    .port = 8080;
}

sub vcl_recv {
    if ((req.http.X-Forwarded-Proto && req.http.X-Forwarded-Proto != "https") || 
        (req.http.Scheme && req.http.Scheme != "https")) {
        return (synth(750));
    } elseif (!req.http.X-Forwarded-Proto && !req.http.Scheme && !proxy.is_ssl()) {
        return (synth(750));
    }
}

sub vcl_synth {
    if (resp.status == 750) {
        set resp.status = 301;
        set resp.http.location = "https://" + req.http.Host + req.url;
        set resp.reason = "Moved";
        return (deliver);
    }
}

I tried setting up the following constants in my wp-config.php file to see if forcing the URL from the application would make a change and that did not work.

define( 'WP_HOME', 'https://staging1.sainikbiswas.com' );
define( 'WP_SITEURL', 'https://staging1.sainikbiswas.com');

Solution

By the looks of it, you're doing everything right. There's a couple of things you need to do to avoid the mixed content when you're using a TLS proxy:

Terminate TLS in Hitch
Ensure you use the PROXY protocol in hitch to pass along the original protocol
Ensure that Varnish is listening on a port with PROXY support (e.g. -a :8443,PROXY)
Check whether an X-Forwarded-Proto headers was set along the way
Use the proxy VMOD in VCL to call proxy.is_ssl() in case there's no X-Forwarded-Proto header and set the header appropriately
Add the X-Forwarded-Proto header value to the hash to create protocol awareness (e.g. hash_data("X-Forwarded-Proto"))
Optionally enforce HTTPS through redirections in Varnish

Again, it seems like you've done all these things.

Just to set the record straight: Varnish Cache (open source) doesn't handle HTTPS natively and has TLS awareness through the PROXY protocol when TLS is offloaded elsewhere.

That's why the X-Forwarded-Proto header is so useful: it announces the protocol of the initial connection. You could add the following PHP code to your WordPress setup to force support for X-Forwarded-Proto:

if (strpos($_SERVER['HTTP_X_FORWARDED_PROTO'], 'https') !== false)
    $_SERVER['HTTPS']='on';
}

If that doesn't help, please provide some relevant log output through varnishlog for me to examine.

Assuming the homepage is what we're monitoring, you'll provide the output of the following command:

sudo varnishlog -g request -q "ReqUrl eq '/'"

Please ensure your cache is empty when running this command, otherwise we have no idea how the backend responds to your request.