Search code examples
python-3.xamazon-web-servicesamazon-s3aws-lambdaaws-serverless

Is there a way to download a csv file from a website and upload it directly to Amazon S3 using Lambda?

I'm making an app that cleans-etc-search a CSV that is updated daily on a website. I was using an EC2 to download the file using python pandas.read_csv(url) to an EBS, but now I want to make the app serverless. I want to automate the download from 'https://sam.gov/api/prod/fileextractservices/v1/api/download/Contract%20Opportunities/datagov/ContractOpportunitiesFullCSV.csv?privacy=Public' and upload it to S3 serverless. I'm not sure if is possible to do it serverless. Is there a better way to do it? The file size is about 500 MB.

Solution

  • A lambda is exactly what you would want to use for this kind of scenario. Do the following:

    • Create the S3 bucket
    • Write the lambda function
    • Configure an IAM role to give lambda permission to write to the S3 bucket
    • Configure an EventBridge task to trigger the lambda function daily