Trying to configure nginx to proxy-pass to an AWS Lambda with resolver

Question

so I've set up an nginx frontend webserver behind cloudflare which I use to forward requests to backend services. Everything is basically working, but now I added a Lambda service which is being used for authentication and also signing requests for the other APIs.

However, after some time the initial requests to the lambda behind the API gateway fail with HTTP error code 499. I was able to figure out it's due to API Gateway DNS IP changes, which could theoretically be tackled with nginx resolvers (https://medium.com/driven-by-code/dynamic-dns-resolution-in-nginx-22133c22e3ab).

However, any way I tried to fix it either caused one of 2 error cases:

• Auth API not working or missing Header parameters at the backend (I send along an API key but that gets removed)
• Nothing is working anymore (Wrong config for auth API kills the other APIs as well)

I believe I'm likely just missing a small detail, so I'm sharing my code here, hopefully someone knows what is missing.

This works until AWS Changes IP of the API Gateway

server {
    listen 443 ssl;
    server_name auth.mywebsite.com;

    ssl_certificate /etc/nginx/ssl/mywebsite.pem;
    ssl_certificate_key /etc/nginx/ssl/mywebsite.key;

    location / {
        return 301 https://mywebsite.com$request_uri;
    }

    location /v1/ {
        proxy_pass https://{AWS-GATEWAY-ID}.execute-api.us-west-1.amazonaws.com/main/;
        proxy_set_header Host $proxy_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

This breaks Api key Header forwarding functionality at the backend:

server {
    listen 443 ssl;
    server_name auth.mywebsite.com;

    ssl_certificate /etc/nginx/ssl/mywebsite.pem;
    ssl_certificate_key /etc/nginx/ssl/mywebsite.key;

    location / {
        return 301 https://mywebsite.com$request_uri;
    }

    location /v1/ {
        resolver 8.8.8.8 valid=30s;
        set $endpoint {AWS-GATEWAY-ID}.execute-api.us-west-1.amazonaws.com;
        
        proxy_pass https://$endpoint/main/;
        proxy_set_header Host $proxy_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

Any help is really appreciated

Answer 1

Alright so I was able to get this resolved now with a two-step approach.

On the nginx side, I ended up with the following code:

server {
    listen 443 ssl;
    server_name auth.mywebsite.com;

    ssl_certificate /etc/nginx/ssl/mywebsite.pem;
    ssl_certificate_key /etc/nginx/ssl/mywebsite.key;

    # Use Google's DNS servers for resolution
    resolver 8.8.8.8 8.8.4.4 valid=30s;
    resolver_timeout 10s;

    location / {
        return 301 https://mywebsite.com$request_uri;
    }

    location /v1/ {
        # Make sure to use a variable in the proxy_pass for dynamic resolution
        set $backend {AWS-GATEWAY-ID}.execute-api.us-west-1.amazonaws.com;
        proxy_pass  https://$backend/main$request_uri;
        
        proxy_set_header Host $proxy_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

This allowed requests to go through to the Lambda, with one restriction:

The previous config which did the proxy-pass directly but failed resolving after some time, would remove the /v1 from the full URI which was sent to the lambda as a request path.

The new config would alwasy include the /v1. After spending like 2 hours on trying to change the nginx config so the /v1 is not part of the request path being sent to the backend, I gave up and instead changed the lambda code to expect the API version to be part of the request as well.

Now everything has been working expected for the last week.