Search code examples
pythonboto3

s3fs custom endpoint url


How do I pass a custom endpoint url to s3fs.S3FileSystem ?

I've tried:

kwargs = {'endpoint_url':"https://s3.wasabisys.com",
          'region_name':'us-east-1'}
self.client = s3fs.S3FileSystem(key=AWS_ACCESS_KEY_ID, 
                                secret=AWS_SECRET_ACCESS_KEY,
                                use_ssl=True,
                                **kwargs)

However I get the error:

  File "s3fs/core.py", line 215, in connect
    **self.kwargs)
TypeError: __init__() got an unexpected keyword argument 'endpoint_url'

I've also tried passing kwargs as the parameter config_kwargs and s3_additional_kwargs with similar errors.

I can verify boto3 is working with the following:

client = boto3.client("s3",
        aws_access_key_id=AWS_ACCESS_KEY_ID,
        aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
        endpoint_url="https://s3.wasabisys.com",
        use_ssl=True,
        region_name="us-east-1", 
        api_version=None,verify=None, config=None)

Solution

  • There is an undocumented parameter in the current verison client_kwargs={'endpoint_url': 'https:...'} which is the desired one.

    https://github.com/dask/s3fs/issues/119