Varnish error "Uncached req.body can only be consumed once" by extracting response header from subrequest and add it to the original request

I have an api with jwt authentication (bearer token). The jwt is sent on every api request. For validating the jwt I have a specific route in my backend (GET /_jwt_user_sub). A request to this route with a valid jwt returns a X-User response header with code 200 and Content-Type: application/vnd.user-sub. I configured Varnish to make on every api call (GET and POST) a sub request to this route, extract the X-User header from the response and add it to the originally api request. This jwt user id response should be cached by varnish. For api requests with GET method this works fine but not for api calls with POST method because Varnish by default do all backend requests with GET. So I override this behaviour in the vcl_backend_fetch sub routine (see following vcl configuration).

With my vcl configuration I get a Error 503 Backend fetch failed. By debugging with varnishlog I see that a vcl error is thrown: VCL_Error Uncached req.body can only be consumed once.. I am not sure how to rewrite the configuration correctly, as the error says at one point the configuration tries to consume a request body twice without caching it. Varnish should only cache GET api calls, no POST calls.

This is my vcl configuration:

vcl 4.1;
import std;
import xkey;

backend app {
    .host = "nginx";
    .port = "8080";
}

sub vcl_recv {
    set req.backend_hint = app;

    // Retrieve user id and add it to the forwarded request.
    call jwt_user_sub;

    // Don't cache requests other than GET and HEAD.
    if (req.method != "GET" && req.method != "HEAD") {
        return (pass);
    }

    return (hash);
}

sub vcl_hit {
   if (obj.ttl >= 0s) {
       return (deliver);
   }

   if (obj.ttl + obj.grace > 0s) {
       if (!std.healthy(req.backend_hint)) {
           return (deliver);
       } else if (req.http.cookie) {
           return (miss);
       }

       return (deliver);
   }

   return (miss);
}

// Called when the requested object has been retrieved from the backend
sub vcl_backend_response {
    if (bereq.http.accept ~ "application/vnd.user-sub"
        && beresp.status >= 500
    ) {
        return (abandon);
    }
}

// Sub-routine to get jwt user id
sub jwt_user_sub {
    // Prevent tampering attacks on the mechanism
    if (req.restarts == 0
        && (req.http.accept ~ "application/vnd.user-sub"
            || req.http.x-user
        )
    ) {
        return (synth(400));
    }

    if (req.restarts == 0) {
        // Backup accept header, if set
        if (req.http.accept) {
            set req.http.x-fos-original-accept = req.http.accept;
        }
        set req.http.accept = "application/vnd.user-sub";

        // Backup original URL
        set req.http.x-fos-original-url = req.url;
        set req.http.x-fos-original-method = req.method;
        set req.url = "/_jwt_user_sub";
        set req.method = "GET";

        return (hash);
    }

    // Rebuild the original request which now has the hash.
    if (req.restarts > 0
        && req.http.accept == "application/vnd.user-sub"
    ) {
        set req.url = req.http.x-fos-original-url;
        set req.method = req.http.x-fos-original-method;
        unset req.http.x-fos-original-url;
        if (req.http.x-fos-original-accept) {
            set req.http.accept = req.http.x-fos-original-accept;
            unset req.http.x-fos-original-accept;
        } else {
            // If accept header was not set in original request, remove the header here.
            unset req.http.accept;
        }

        if (req.http.x-fos-original-method == "POST") {
            return (pass);
        }

        return (hash);
    }
}

sub vcl_backend_fetch {
    if (bereq.http.x-fos-original-method == "POST") {
        set bereq.method = "POST";
    }
}

sub vcl_deliver {
    // On receiving the hash response, copy the hash header to the original
    // request and restart.
    if (req.restarts == 0
        && resp.http.content-type ~ "application/vnd.user-sub"
    ) {
        set req.http.x-user = resp.http.x-user;

        return (restart);
    }
}

Solution

The built-in VCL says that only GET and HEAD requests are cacheable by default. You can override this and cache POST calls, but unless you perform explicit caching of the request body (which is stored in req.body), the body itself will only be processed once.

Because you perform restart calls in your VCL, the request body is processed multiple times. Unfortunately, at the 2nd attempt it is no longer available.

To tackle this issue, you can use the std.std.cache_req_body(BYTES size) function to explicitly cache the request body. Most of the times this will be for POST calls but the HTTP spec also allows you to have request bodies for GET calls.

See https://varnish-cache.org/docs/6.0/reference/vmod_generated.html#func-cache-req-body for more information.

Here's how to implement this function in your vcl_recv subroutine:

sub vcl_recv {
    set req.backend_hint = app;
     
    // Cache the request body for POST calls
    if(req.method == "POST") {
        std.cache_req_body(10K);
    }    
 
    // Retrieve user id and add it to the forwarded request.
    call jwt_user_sub;

    // Don't cache requests other than GET and HEAD.
    if (req.method != "GET" && req.method != "HEAD") {
        return (pass);
    }

    return (hash);
}