Error initializing boto3 session in Django application using DigitalOcean Spaces

Question

I'm having an issue with my Django application when trying to configure it to use DigitalOcean Spaces for static and media files. Here is the relevant part of my settings.py file:

import boto3
from botocore.exceptions import NoCredentialsError, PartialCredentialsError
from botocore.client import Config
from os import getenv

AWS_ACCESS_KEY_ID = getenv('SPACES_KEY')
AWS_SECRET_ACCESS_KEY = getenv('SPACES_SECRET')
AWS_STORAGE_BUCKET_NAME = getenv('BUCKET_NAME')
AWS_S3_REGION_NAME = "region"
AWS_S3_ENDPOINT_URL = f"https://{AWS_S3_REGION_NAME}.digitaloceanspaces.com"

AWS_S3_PARAMETERS = {
    'CacheControl': 'max-age=86400',
}

AWS_DEFAULT_ACL = 'public-read'
AWS_S3_SIGNATURE_VERSION = 's3v4'

STATIC_URL = f"{AWS_S3_ENDPOINT_URL}/{AWS_STORAGE_BUCKET_NAME}/static/"
MEDIA_URL = f"{AWS_S3_ENDPOINT_URL}/{AWS_STORAGE_BUCKET_NAME}/media/"

STATICFILES_STORAGE = 'name.custom_storages.StaticStorage'
DEFAULT_FILE_STORAGE = 'name.custom_storages.MediaStorage'

# Ensure the credentials are explicitly set for boto3
session = boto3.session.Session()
s3_client = session.client(
    's3',
    region_name=AWS_S3_REGION_NAME,
    endpoint_url=AWS_S3_ENDPOINT_URL,
    aws_access_key_id=AWS_ACCESS_KEY_ID,
    aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
    config=Config(signature_version='s3v4')
)

# Test connection to S3
def test_s3_connection():
    try:
        print("Connecting to S3...")
        response = s3_client.list_objects_v2(Bucket=AWS_STORAGE_BUCKET_NAME)
        print("Connection successful. Contents:")
        for obj in response.get('Contents', []):
            print(obj['Key'])
    except NoCredentialsError:
        print("Error: No credentials found.")
    except PartialCredentialsError:
        print("Error: Incomplete credentials.")
    except Exception as e:
        print(f"Error: {e}")

test_s3_connection()

When I run the server using python3 manage.py runserver, I get the following error:

Traceback (most recent call last):
  ...
  File "/Users/User/.aws/credentials"
botocore.exceptions.ConfigParseError: Unable to parse config file: /Users/User/.aws/credentials

I don't want boto3 to use the credentials from the ~/.aws/credentials file because I'm not using AWS, I'm using DigitalOcean Spaces.

Any help or suggestions on how to resolve this issue would be greatly appreciated. Thanks in advance!

Verified the AWS credentials file:

1. Checked for syntax errors: I made sure there are no syntax errors or formatting issues in the credentials file.

2. Environment variables: I confirmed that the environment variables (SPACES_KEY, SPACES_SECRET, BUCKET_NAME) are set correctly in my environment.

3. Updated boto3 and botocore: I made sure I have the latest versions of boto3 and botocore installed in my virtual environment.

4. Debugging logs: Added print statements and logging to verify that the environment variables are correctly loaded and passed to boto3.

5. I expected the Django application to initialize the boto3 session using only the provided environment variables and not reference the AWS credentials file. However, I keep encountering the ConfigParseError related to the credentials file, and I am unable to proceed further.

Any insights or suggestions on how to resolve this issue would be greatly appreciated.

Answer 1

It sounds like you may have a file at ~/.aws/credentials that has some sort of invalid content in it.

In any case, your problem is that you are calling boto3.session.Session() without any arguments, so it is trying to use the default behavior to find credentials. You are waiting till you create the S3 client to "override" the default boto3 credential settings with your custom credentials for the S3 client. Instead you should just be passing those credentials into the Session call.