Search code examples
amazon-s3botoluigifakes3

How to test Luigi with FakeS3?

I'm trying to test my Luigi pipelines inside a vagrant machine using FakeS3 to simulate my S3 endpoints. For boto to be able to interact with FakeS3 the connection must be setup with the OrdinaryCallingFormat as in:

from boto.s3.connection import S3Connection, OrdinaryCallingFormat
conn = S3Connection('XXX', 'XXX', is_secure=False, 
                    port=4567, host='localhost',
                    calling_format=OrdinaryCallingFormat())

but when using Luigi this connection is buried in the s3 module. I was able to pass most of the options by modifying my luigi.cfg and adding an s3 section as in

[s3]
host=127.0.0.1
port=4567
aws_access_key_id=XXX
aws_secret_access_key=XXXXXX
is_secure=0

but I don't know how to pass the required object for the calling_format.

Now I'm stuck and don't know how to proceed. Options I can think of:

  1. Figure out how to pass the OrdinaryCallingFormat to S3Connection through luigi.cfg
  2. Figure out how to force boto to always use this calling format in my Vagrant machine, by setting an unknown option to me either in .aws/config or boto.cfg
  3. Make FakeS3 to accept the default calling_format used by boto that happens to be SubdomainCallingFormat (whatever it means).

Any ideas about how to fix this?

Solution

  • Can you not pass it into the constructor as kwargs for the S3Client?

    client = S3Client(aws_access_key, aws_secret_key,
                  {'calling_format':OrdinaryCallingFormat()})
target = S3Target('s3://somebucket/test', client=client)