Search code examples
amazon-web-servicesaws-lambdaaws-glue

Trigger a Custom Function Every X Hours in AWS

I am looking to trigger code every 1 Hour in AWS.

The code should: Parse through a list of zip codes, fetch data for each of the zip codes, store that data somewhere in AWS.

  • Is there a specific AWS service would I use for parsing through the list of zip codes and call the api for each zip code? Would this be Lambda?
  • How could I schedule this service to run every X hours? Do I have to use another AWS Service to call my Lambda function (assuming that's the right answer to #1)?
  • Which AWS service could I use to store this data?

I tried looking up different approaches and services in AWS. I found I could write serverless code in Lambda which made me think it would be the answer to my first question. Then I tried to look into how that could be ran every x time, but that's where I was struggling to know if I could still use Lambda for that. Then knowing where my options were to store the data. I saw that Glue may be an option, but wasn't sure.

Solution

  • Yes, you can use Lambda to run your code (as long as the total run time is less than 15 minutes).

    You can use Amazon EventBridge Scheduler to trigger the Lambda every 1 hour.

    Which AWS service could I use to store this data?

    That depends on the format of the data and how you will subsequently use it. Some options are

    If you choose S3, you can still do SQL-like queries on the data using Amazon Athena