Why does AWS Lambda experience a consistent 10-second delay for an unresolvable domain using requests.get?

Question

I'm experiencing a peculiar behavior when trying to make an HTTP request to unresolved domain names from an AWS Lambda function using the requests library in Python.

When I attempt to make a request using:

response = requests.get('https://benandjerry.com', timeout=(1,1))

In AWS Lambda, it consistently takes around 10 seconds before it throws an error. However, when I run the same code on my local environment, it's instant. I've verified this using logs and isolated tests.

I've considered potential issues like Lambda's cold starts, Lambda runtime differences, and even VPC configurations, but none seem to be the root cause.

I also tried using curl to access the domain, and it instantly returned with Could not resolve host: benandjerry.com.

Last point, this is happening on specific unresolved domain names, not all of them.

Here's a sample:

• http://seen.ma/
• benandjerry.com
• clenyabeauty.com
• gong.com

FYI, you can easily replicate the issue by creating a python3.9 Lambda on AWS & adding the following code:

import json
from botocore.vendored import requests
import urllib.request
import os

def lambda_handler(event, context):
    # TODO implement
    url = 'http://benandjerry.com'
    try:
        response = requests.get(url, proxies=None,verify=False)
    except Exception as e:
        print(e)

    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

Questions:

1. What could be causing this consistent 10-second delay in AWS Lambda for an unresolvable domain using requests?
2. How can I get AWS Lambda to instantly recognize that the domain is unresolvable, similar to the behavior on my local machine?

Answer 1

The issue you're seeing is due to the AWS DNS taking up to 10 seconds trying to resolve the domain. If you want more control over the DNS resolution, you can implement a custom requests Transport Adapter to do the DNS resolution yourself—which allows you to better customize the timeout.

pip install dnspython

import urllib
import requests
import dns.resolver


class CustomDnsResolverHttpAdapter(requests.adapters.HTTPAdapter):
    def resolve(self, hostname):
        resolver = dns.resolver.Resolver(configure=False)
        resolver.timeout = 5
        resolver.lifetime = 5

        resolver.nameservers = [
            "1.1.1.1"  # cloudflare dns
            , "8.8.8.8" # google dns
            # , "169.254.78.1"  # aws dns
        ]

        answer = resolver.resolve(hostname, "A", lifetime=5)

        if len(answer) == 0:
            return None

        return str(answer[0])

    def send(self, request, **kwargs):
        connection_pool_kwargs = self.poolmanager.connection_pool_kw

        result = urllib.parse.urlparse(request.url)
        resolved_ip = self.resolve(result.hostname)

        if result.scheme == "https" and resolved_ip:
            request.url = request.url.replace(
                "https://" + result.hostname,
                "https://" + resolved_ip,
            )
            connection_pool_kwargs["server_hostname"] = result.hostname  # SNI
            connection_pool_kwargs["assert_hostname"] = result.hostname

            # overwrite the host header
            request.headers["Host"] = result.hostname
        else:
            # clear these headers if they were set in a previous TLS request
            connection_pool_kwargs.pop("server_hostname", None)
            connection_pool_kwargs.pop("assert_hostname", None)

            # overwrite the host header
            request.headers["Host"] = result.hostname

        return super(CustomDnsResolverHttpAdapter, self).send(request, **kwargs)



http_agent = requests.Session()
http_agent.mount("http://", CustomDnsResolverHttpAdapter())
http_agent.mount("https://", CustomDnsResolverHttpAdapter())

response = http_agent.get("https://benandjerry.com")