I get files from sources into a S3 bucket.The text files can range in size from 1-100 MB. I need to extract some stuff from files and write them to dynamoDB. So far, the implementation is such that an incoming file into S3 bucket triggers a lambda, which processes the files and writes them to dynamoDB. So far so good, but there is a problem.
The problem is, I have no control over rate of files coming into S3. It ranges from few per seconds to hundreds per second under certain circumstance. There are two problems: 1) a large number of lambdas are triggered. I want to limit it to say N lambdas, where N corresponds to max active connections allowed by dynamoDB.
Can I use Kinesis somehow between S3 and dynamoDB in this case? Or, what could be a good solution?
I find kinesis might not be the right fit, since you cannot really control how fast items are processed. Another option, where you have a more precise control of the speed of processing might be SQS. Just put every event in a queue and have a lambda trigged every few minutes which reads from the queue. However keep in mind that you might also trigger the lambda, when no event is in the queue thus wasting resources.