Search code examples
amazon-web-servicesaws-lambda

Does AWS-Lamda support multi-threading?


I am writing an AWS-lambda function that reads past 1-minute data from a DB, converts it to JSON, and then pushes it to a Kafka topic. The lambda will run every 1 minute. So if a scenario comes like this: at t1, a lambda process is invoked let's say P1, it is reading data from t1 to t1-1. at t2, if P1 is not finished then will a new lambda process will be invoked or we will wait for P1 to finish so that we can invoke another process? I understand that lambda support up to 1000 parallel processes in a region but in this situation, the lambda function will already be running in a process.


Solution

  • Lambda does support multi-threading and multi-process in the same execution (see an example). However, from the context that you gave, that doesn't seem to be the underlying question.

    Concurrency allows you to configure how many instances of a single Lambda function can be in execution at a given time. Quoting the relevant part of the documentation:

    Concurrency is the number of requests that your function is serving at any given time. When your function is invoked, Lambda allocates an instance of it to process the event. When the function code finishes running, it can handle another request. If the function is invoked again while a request is still being processed, another instance is allocated, which increases the function's concurrency. The total concurrency for all of the functions in your account is subject to a per-region quota.

    Triggers define when a Lambda is executed. If you have multiple events coming (from SQS for example) to a lambda that has concurrency>1, then it's likely that there will be multiples instances of that given Lambda running at the same time.

    With concurrency=1, if you trigger the Lambda every 1 minute and it takes more than 1 minute to execute and finish, then your processing will lag behind. In other words, future Lambdas will be processing t-2, t-3, and so on.

    With concurrency=1, if you want something to be processed every 1 minute, you have to make sure it doesn't take more than 1 minute to process it. With extra concurrency it can take longer.