AWS Lambda requests.get not responding - ConnectTimeout: HTTPSConnectionPool

Question

I'm trying to run my code in AWS Lambda and part of the code is requesting the URL "https://api.hypixel.net/skyblock/auctions" which should kick back a json. Instead the code runs for about 250 seconds then send the following error as if the URL doesn't exist.

ConnectTimeout: HTTPSConnectionPool(host='api.hypixel.net', port=443): Max retries exceeded with url: /skyblock/auctions (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f513a00e590>, 'Connection to api.hypixel.net timed out. (connect timeout=None)'))
Traceback (most recent call last):

The Code:

import requests

print("Start")
AH_request = requests.get("https://api.hypixel.net/skyblock/auctions",verify=False)
print("SC: ",AH_request.status_code)
AH_data = AH_request.json()
print("AH_data: ", AH_data)

Output: Output

Things i've tried:

1. The code works in visual studio code on my PC. So the problem is localized to AWS Lambda
2. I tried using verify=false
3. I tried different APIs such as: https://www.boredapi.com/api/activity and https://api.publicapis.org/entries - All return the same error
4. I tried using from botocore.vendored import requests but it would say '.get' is not part of the 'botocore.vendored' module. Maybe I messed up something there?

My guesses:

1. Maybe my AWS Lambda function doesnt have the correct permissions? Advice for this would be appreciated.

Other Info:

1. Something to keep in mind is that I am pretty new to AWS in general. Learning mostly from youtube tutorials and whatnot. So I might've overlooked something small and/or very basic.
2. I do have a VPC and EFS connected to the Lambda. Although Allow IPv6 traffic = false I don't know if that is important

Answer 1

Break it down and think about the error you're getting back. You're getting a timeout, it's unable to hit any of the APIs over the internet - and you have tried multiple.

This does like something has gone awry with your setup as you initially thought. You mentioned you have an AWS VPC configured in the Lambda. Does that have an Internet Gateway available to be able to talk externally?

I tried the code (and changed a few things):

import requests

def lambda_handler(event, context):

    AH_request = requests.get("\nhttps://api.hypixel.net/skyblock/auctions", verify=False)
    print("\nStatus Code: ", AH_request.status_code)
    
    AH_data = AH_request.json()

    return AH_data

Which returns as you expect a JSON payload - as you desired and are getting locally (Python 3.7 Runtime).

Take it back to basics. Try removing the AWS VPC and set instead set it to 'None'. Often we only need to set a VPC in a Lambda to talk to other services - which it sounds like you want to do eventually via the AWS EFS you mentioned. However, set it to None, and figure out if that works as you expect.

The issue probably lies on the AWS VPC setup and egress or routing settings. Hope this helps!