Search code examples
amazon-web-servicesfileamazon-s3transfer

Using AWS Lambda to copying S3 files to on-premise LAN folder


Problem: We need to perform a task under which we have to transfer all files ( CSV format) stored in AWS S3 bucket to a on-premise LAN folder using the Lambda functions. This will be a scheduled tasks which will be carried out after every 1 hour, and the file will again be transferred from S3 to on-premise LAN folder while replacing the existing ones. Size of these files is not large (preferably under few MBs).

I am not able to find out any AWS managed service to accomplish this task.

I am a newbie to AWS, any solution to this problem is most welcome.

Thanks,


Solution

  • Actually, I am looking for a solution by which I can push S3 files to on-premise folder automatically

    For that you need to make the on-premise network visible to the logic (lambda, whatever..) "pushing" the content. The default solution is using the AWS site-to-site VPN.

    There are multiple options for setting up the VPN, you could choose based on the needs.

    Then the on-premise network will look just like another subnet.

    However - VPN has its complexity and cost. In most of the cases it is much easier to "pull" data from the on-premise environment.

    To sync data there are multiple options. For a managed service, I could point out the S3 Gateway which based on your description sounds like an insane overkill.

    Maybe you could start with a simple cron job (or a task timer if working with windows) and run a CLI command to sync the S3 content or just copy specified files.