Search code examples
amazon-web-servicesaws-lambdaamazon-cloudfrontserverlessaws-lambda-edge

Serverless AWS Lambda@Edge: How to debug

I have a python Lambda@Edge function deployed with serverless which is working OK using the test feature:

enter image description here

FYI it's setup as viewer-request:

enter image description here

functions:
  cfLambda:
    handler: handler.lambda_handler
    events:
      - cloudFront:
          eventType: viewer-request

And if I go to CloudWatch, I can see the logs:

enter image description here

Now when I test with cURL it fails with a 503:

HTTP/2 503 
content-type: text/html
content-length: 1019
server: CloudFront
date: Mon, 05 Apr 2021 07:24:45 GMT
x-cache: LambdaExecutionError from cloudfront
via: 1.1 XXXXXXXXXXXXXXXXXX.cloudfront.net (CloudFront)
x-amz-cf-pop: AMS50-C1
x-amz-cf-id: 4vYpBnOGd6yfgowoSpiCyBkh5cbV1g3IJf1H2Eheln89MpEnScL-1g==

However this time I get no logs in CloudWatch. Q1: How can I have traces of my Lambda@Edge CloudFront calls visible in CloudWatch?

If I read the Lambda@Edge debug guide it says that 503 status code is either:

  • too many executions (not my case, there is 0 traffic to that cloudfront, it's purely test)
  • function exceeded the Lambda function timeout quota

If I look at the console tests, they only consume ~220ms and ~75MB so I think we're way below the 5-second / 128MB limit for viewer request

enter image description here

If I look at the CloudFront logs they seem useless as they just confirm the 503:

E2HX7F6YEZN897.2021-04-04-16.a77a21e1:2021-04-04    16:34:12    SEA19-C3    389 35.247.33.169   HEAD    XXXXXXXX.cloudfront.net /   503 -   Mozilla/5.0%20(Windows%20NT%205.1)%20AppleWebKit/537.36%20(KHTML,%20like%20Gecko)%20Chrome/41.0.2224.3%20Safari/537.36  -   -   LambdaExecutionError    AcM5SX3ggB53fmjXO83xND_Lw3-eHXd8dlIZGEO53XaDMjuctRw==   example.org https   223 0.021   -   TLSv1.3 TLS_AES_128_GCM_SHA256  LambdaExecutionError    HTTP/1.1    -   -   51810   0.021   LambdaExecutionError    text/html   1019    -   -

Q2: Is there a way to increase the verbosity of the CloudFront logs (I couldn't find one)?

Q3: If I can't get my CloudFront Lambda@Edge calls in CloudWatch (Q1=no) and I can't increase verbosity of CloudFront Logs (Q2=no), how can I debug this further?

Solution

    • The Lambda@Edge function must be deployed to the us-east-1 region.

    • The x-amz-cf-pop header gives a hint about where the request was executed. You can refer to this unofficial list here.

    • For us-east-1 the logs can be found in CloudWatch under the group /aws/lambda/<Your-function-name>. For any other region the log group would be /aws/lambda/us-east-1.<Your-function-name>. If you know the region, then select the appropriate region. Go to CloudWatch and search the appropriate log group. enter image description here

    • You can also navigate to the appropriate logs from the CloudFront page. Go to the Monitoring section -> Choose your Distribution -> View Distribution Metrics -> Lambda@Edge Errors. The graph would display the errors from all the regions when you hover over the data points. Once you know the region where the error is happening, you can select the same followed by the Lambda function and finally click on View logs. Refer to the below image. enter image description here

    • Take a look at some official examples here. For the viewer-request event, the ones manipulating the request are relevant.

    • The CloudFront logs can be accessed at /aws/cloudfront/LambdaEdge/<YourDistributionId>

    Hopefully this helps you proceed further.