Search code examples
pythonlakefs

boto3 change the endpoint of session

I want to use LakeFS configurations for my boto session. I tried using this, but still, it gives me the default one

s3 = boto3.resource('s3', 
    endpoint_url=lakefsEndPoint,
    aws_access_key_id=lakefsAccessKey,
    aws_secret_access_key=lakefsSecretKey
)


s3_session = boto3.Session(
    aws_access_key_id=lakefsAccessKey,
    aws_secret_access_key=lakefsSecretKey
)

How can I get the new session? I want to write the data using aws wrangler, that's why I need to get the BOTO session.

Solution

  • Get the resource by using the session directly:

    import boto3

session = boto3.Session(...)

resource = session.resource('s3', endpoint_url=lakefsEndPoint)

    However, in awswrangler, you can pass the session directly with the boto3_session kwarg and set the endpoint_url using the config:

    import awswrangler as wr

wr.config.s3_endpoint_url = lakefsEndPoint

df = wr.s3.read_parquet('s3://bucket/key.parquet', boto3_session=session)