Is it possible to create two layers of multi-tiered cache based on memory -> disk cache layers on the same machine?

Question

Is it possible to create two layers of multi-tiered cache based on memory -> disk cache layers on the same machine? We would like to create two layers of tiered cache based on memory -> disk with the open sourced version of varnish 6+. The idea is to create two storage back-ends (which is possible to do) with memory and disk on the same server. The disk tier is about 1TB of very fast SSD and the memory tier is about 256G.

My idea is to create some sort of hot (memory - for extremely popular content) and cold (disk - for less popular content) cache on the same server, which have different TTLs. For example, if an item has too many hits on specific file, it will move to memory for at least 5 to 10 seconds and if the item doesn't exist on the memory tier, it will go to the local disk and fetch it from the SSD tier (cold).

The problem is that I can't find the best way to implement the logic inside vcl_backend_responce. It is possible to set a condition based on host or files types, but I can't find anywhere how to redirect the request from memory storage to disk storage before it goes out the remote HTTP server to fetch the file.

Below is the idea of how I think it should be implemented, the only missing part is what condition should be inside the if statements. I would appreciate your inputs or ideas if there are any.

The place the code should be placed -> vcl_backend_responce

if (obj.hits exsit on beresp.storage_hint ="memory") {
    set beresp.storage_hint ="memory"
    set beresp.ttl = 5s;
}elseif (obj.hits exsit on beresp.storage_hint ="disk") {
    set beresp.storage_hint ="disk"
    set beresp.ttl = 2m;
}

enter image description here

File -->Disk --Memory --> client
         2M (DISK TTL )      10S (Memory TTL) 

Answer 1

You may use multiple varnish ports to achieve that.

    ExecStart=/usr/sbin/varnishd \
      -a :6081 \
      -a :6181 \
      -T localhost:6082 \
      -f /etc/varnish/default.vcl \
      -P %t/%N/varnishd.pid \
      -p feature=+http2 \
      -s memory=malloc,5G \
      -s file=file,/mnt/cache/data,100G

Above command allows Varnish to run on both 6081 and 6181. Then yo can manage storage usage in your vcl with simple if/else condition.

sub vcl_recv
{
  if (std.port(server.ip) == 6081) {
    set req.backend_hint = cluster.backend();

    # If backend hits, pass to layer2Cache which is 6181
    if (req.backend_hint == server.identity) { 
      set req.backend_hint = layer2Cache;
    } else {
      return(pass);
   }
  } else {
    // if it is layer2Cache, pass to origin
    set req.backend_hint = origin; 
  }
}

sub vcl_backend_response {
  if (std.port(server.ip) == 6081) {
    set beresp.storage = storage.memory;
      
    set beresp.ttl = 60s;
    set beresp.grace = 1s * 60 * 5;
  } else {
    set beresp.storage = storage.file;
    
    set beresp.ttl = 240s;
    set beresp.grace = 1s * 60 * 60 * 24;
  }
}

We are using this method in our cache servers, but please note that the code above may not work with direct copy/paste.