Search code examples
amazon-web-servicesaws-lambdalaravel-vapor

Clear DNS cache in AWS Lambda


How to clear the DNS cache of an instance in AWS Lambda?

Context:
We backup the database to an external S3 provider. It has worked for few years now.
Recently they did a maintenance and added some proxies servers and renewed their SSL certificates (20 april) and since then (5 days later) I'm still getting the error:

Copying zip failed because: Could not connect to disk s3-infomaniak because: GuzzleHttp\Exception\ConnectException: cURL error 6: Could not resolve host: s3.swiss-backup02.infomaniak.com (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://s3.swiss-backup02.infomaniak.com/default?list-type=2&prefix=database-backup%2F&delimiter=%2F

We suppose that the DNS cache is still pointing to the old destination.

  • The website is managed with Laravel vapor: https://vapor.laravel.com
  • I confirm the S3 is resolvable from our AWS EC2 and from my machine
  • The backups works from the EC2 and from my machine
  • With or without the NAT, it doesn't change anything
  • Redeploying didn't change anything

Solution

  • I figured out the issue, and it was not the cache.

    1. I could replicate the issue by creating a sub domain with 24 A records (the same amount of records that have been added to s3.swiss-backup02.infomaniak.com)
    2. Then I could still replicate the issue by (and only by) using an Alpine based image on lambda.

    Apparently, Alpine seems to have some DNS issues on Kubernetes cluster due to the usage of musl-libc instead of g-libc1. Having a "large" DNS answer seems to be the case with lambda.

    Since Laravel vapor is using Alpine based image for their Dockerfiles2 (You're not concerned if you use native runtimes) I did a fork3 of their image with php:8.2-fpm-buster and now the URL is reachable.


    1 https://stackoverflow.com/a/65593511/8068675
    2 https://github.com/laravel/vapor-dockerfiles/blob/master/php82.Dockerfile
    3 https://github.com/melba-ch/vapor-dockerfiles-buster