HAPROXY jwt_verify() always returns Unknown Certificate (-5)

I am creating an HAProxy instance in a docker container (image: "haproxytech/haproxy-alpine:latest"). This uses haproxy version 2.8 (with OpenSSL installed), so the jwt_verify() function is available.

Here is a redacted (comments,corporate info) version of my haproxy.cfg file:

global
    log stdout format raw local0
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy

defaults
    mode        http
    log         global
    option      httplog
    option      log-separate-errors
    option      dontlognull
    option      http-server-close
    option      forwardfor except 127.0.0.0/8
    option      redispatch
    retries     3
    maxconn     3000
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s

frontend web-http
    bind :80
    http-request set-header X-Forwarded-Proto http

    # JWT checks for api requests
    acl is_api hdr(host) -i -m end api.myapiserver.com

    http-request deny content-type 'text/html' string 'Missing Authorization HTTP header' if is_api !{ req.hdr(authorization) -m found }
    http-request set-var(txn.alg) http_auth_bearer,jwt_header_query('$.alg') if is_api
    http-request set-var(txn.kid) http_auth_bearer,jwt_header_query('$.kid') if is_api
    http-request set-var(txn.aud) http_auth_bearer,jwt_payload_query('$.aud') if is_api

    http-request deny content-type "text/html" string "Incorrect JWT alg" if is_api !{ var(txn.alg) -m str "RS256" }

    http-request set-var(txn.verify) http_auth_bearer,jwt_verify(txn.alg,"/usr/local/haproxy/jwt.pem"),debug('VERIFY',stdout)

    http-request deny content-type "text/html" string "Failed to validate JWT" if is_api !{ var(txn.verify) -m int 1 }

    log-format  %[var(txn.aud)]\ %[var(txn.kid)]\ %[var(txn.jwt_verify)]\ %[var(txn.alg)]


    default_backend api


backend offline
    server   offline1 maintenance:8080 check

backend api
        server   api1 192.168.10.1:80 check
        server   offline2 maintenance:8080 check backup

Here is the haproxy log output when i call the server with a POST request via curl:

proxy-haproxy-1      | [debug] VERIFY: type=sint <-5>
proxy-haproxy-1      | https://api.myapiserver.com o1ynZps8fCXTIjCPtBR0x - RS256

The curl command always returns:

Failed to validate JWT

As you can see in trying to debug this I have added the debug() function to the jwt_verify() function, which returns -5

The log-format line returns valid data such as aud and kid so that tells me haproxy is correctly processing the bearer data.

The JWT itself is via microsoft and is valid if I go to jwt.io, and paste it there. The jwt and signature are valid.

To try and validate the signature I get the JWKS data from the relevant microsoft endpoint which gives me a public key certificate in the x5c Attribute.

I then use openssl on that attribute to get the public key in an x509 format to be used by haproxy:

$ openssl x509 -pubkey -noout -in ./x5c.pem > jwt.pem

I send up with the following as the contents of the jwt.pem:

However all I ever get is "Unknown Certificate"

See the following for error codes: https://docs.haproxy.org/2.8/configuration.html#7.3.1-jwt_verify

Things I have tried:

I have tried changing the location of the jwt.pem file in the container, I have tried changing the permissions of the file.

I have tried other combinations of public key, and certificate in jwt.pem file.

I have tried other methods of generating (node library, bash scripts) the jwt.pem file.

I have tried generating an RSA version of the key.

I have tried creating a JWT, and public/private key via Auth0 (following the haproxy instructions), same error.

I am running out of ideas, can anyone help?

Solution

In case someone else comes across this issue.

As I was always getting the same error I considered the fact that it might be something to do with the container setup, so I changed the folder that held the key in the container to:

/certs/jwt.pem

and it all worked!

config file:

http-request set-var(txn.verify) http_auth_bearer,jwt_verify(txn.alg,"/certs/jwt.pem"),debug('VERIFY',stdout)

output:

proxy-haproxy-1      | [debug] VERIFY: type=sint <1>
proxy-haproxy-1      | https://api.myapiserver.com o1ynZps8fCXTIjCPtBR0x - RS256
proxy-maintenance-1  | 192.168.10.1 - - [19/Sep/2023:07:39:05 +0000] "GET / HTTP/1.1" 200 800 "-" "curl/8.1.2" "192.168.10.10"

As you can see verify is now returning 1, and the request is being passed to the backend.

Whats more concerning now, is why this change worked.