how to test boto3 resource download file raising 404 error using mock?

Question

I want to test s3 resource download_file

Here is the code I want to test

def logfile_downloader():
    s3 = boto3.resource('s3')
    bucket = s3.Bucket(bucket)
    for object in bucket.objects.filter(Prefix='logs/access_2018'):
        try:
            bucket.download_file(object.key, 'logs/' + save_path + '/' + object.key.split('/')[-1])
        except botocore.exceptions.ClientError as e:
            if e.response['Error']['Code'] == "404":
                click.echo(click.style("The object does not exist.", bg="white", fg="red"))
            else:
                raise

When I test using python mock, it passes:

@mock.patch('boto3.resource')
    def test_log_downloader(mock_resource):
    logfinder._log_downloader()
    assert mock_resource.called

but, coverage is not 100% because botocore.exceptions.ClientError was not tested

So I create a test

@mock.patch('s3.Bucket.download_file')
def test_log_downloader_404(mock_download_file):
    mock_download_file.return_value = 404
    logfinder.log_downloader()
    assert mock_download_file.called

but it failed with

ModuleNotFoundError: No module named 's3'

I think mock raises error when running download_file function.

I found download_file documented here:
http://boto3.readthedocs.io/en/latest/guide/s3-example-download-file.html#more-info

but in the test, I can't import s3 module

Answer 1

s3 is not a module, boto3 is. I wanted to do the same as you, mocking a 500 response botocore.exceptions.ClientError object. Here's how I did (I updated my code to match yours as it was quite similar):

import botocore

def test_log_downloader_500():
    with mock.patch('boto3.s3.transfer.S3Transfer.download_file') as download_file:
        error_response = {'Error': {'Code': '500'}}
        side_effect = botocore.errorfactory.ClientError(error_response, 'unexpected')
        download_file.side_effect = side_effect

        with pytest.raises(botocore.errorfactory.ClientError):
            logfinder.log_downloader()

This will cover the else raise part. Just do the same for a 404 error by replacing above values and you'll cover the 404 condition 👍