API call script fails in cron job but works fine when run manually

API call script fails in cron job but works fine when run manually. My Python script is making a call to https://stat-xplore.dwp.gov.uk/webapi/rest/v1/schema The error I get is:

Request failed: HTTPSConnectionPool(host='stat-xplore.dwp.gov.uk', port=445): Max retries exceeded with url: /webapi/rest/v1/schema (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0j0iefhfnfj9>, 'Connection to stat-xplore.dwp.gov.uk timed out. (connect timeout=30)'))

I have tried increasing timeout but it didn't make a difference. Cron jobs that don't involve making API call work fine.

#!/usr/bin/python3
import json
import requests
import os


url = 'https://stat-xplore.dwp.gov.uk/webapi/rest/v1/schema'

# Set your API key in the environment variable
os.environ['APIKey'] = 'myapikey'

#I have also tried this without setting environment variable
#api_key='myapikey'

# Get the API key from the environment variable
api_key = os.environ['APIKey']
headers = {'Accept': 'application/json', 'APIKey': api_key}

try:
    response = requests.get(url, headers=headers, timeout=30)
    print(response)
    # Process the response...
except requests.RequestException as e:
    print("Request failed:", e)

My cron job

42 18 * * * /usr/bin/python3 /home/user/bin/bash/statxplore_api.py

Things I have tried:

Set environment variables, Checked file permissions to ensure it can be executed, Tested network conn by pinging to https://stat-xplore.dwp.gov.uk

Solution

I solved the issue by adding proxies to my script. The issue occurred because my user has proxy installed under my profile but contab does not.

#!/usr/bin/python3

import json
import requests
import os


# Define your proxy server
proxies = {
    'http': 'http://yourproxy:0303',
    'https': 'http://yourproxy:0303'
}


url = 'https://stat-xplore.dwp.gov.uk/webapi/rest/v1/schema'

# Set your API key in the environment variable
os.environ['APIKey'] = 'myapikey'

# Get the API key from the environment variable
api_key = os.environ['APIKey']
headers = {'Accept': 'application/json', 'APIKey': api_key}

try:
    response = requests.get(url, headers=headers, proxies=proxies, timeout=30)
    print(response)
    # Process the response...
except requests.RequestException as e:
    print("Request failed:", e)

To find out what proxies are in use you can run the code below

import os

proxy_env_vars = ['http_proxy', 'https_proxy']

# Check for proxy settings
for var in proxy_env_vars:
    proxy_val = os.getenv(var)
    if proxy_val:
        print(f"{var}: {proxy_val}")
    else:
        print(f"{var}: Not set")