Calling API inside a lambda function fails when there is lot of requests

Question

In a document upload process, there is an API endpoint which use to update the database. That endpoint exposes through a Aws API gateway and it has been pointed to a AWS SQS to process requests. That Queue triggers a lambda function and call an API method to update the database inside the lambda. When there is large number of requests,(15-20 document upload requests) that lambda function fails throwing 'Response status code does not indicate success : 400' (400 error). Its working normally when there is small number of requests. What would be the reason?

Lambda Code.

 public async Task FunctionHandler(SQSEvent evnt, ILambdaContext context)
    {
        try
        {
            HttpClient client = new();

            foreach (var message in evnt.Records)
            {
                await ProcessMessageAsync(message, context, client);
            }
        }
        catch (Exception ex)
        {
            throw new UploaderException(ex.Message);
        }
    }

//Private method
private async Task ProcessMessageAsync(SQSEvent.SQSMessage message, ILambdaContext context, HttpClient client) {
 
        string item = string.Empty;
        string methodName = string.Empty;
        string httpMethod = string.Empty;

        foreach (var attribute in message.MessageAttributes)
        {
            if (attribute.Key.ToLower() == "item")
            {
                item = attribute.Value.StringValue;
            }
            if (attribute.Key.ToLower() == "methodname")
            {
                methodName = attribute.Value.StringValue;
            }
            if (attribute.Key.ToLower() == "httpmethod")
            {
                httpMethod = attribute.Value.StringValue;
            }

            if (attribute.Key.ToLower() != "methodname" || attribute.Key.ToLower() != "httpmethod")
            {
                client.DefaultRequestHeaders.Add(attribute.Key, attribute.Value.StringValue);
            }
        }

        if (string.IsNullOrWhiteSpace(item))
        {
            throw new UploaderException("Could not find item");
        }

        string baseUrl = Environment.GetEnvironmentVariable(item.ToUpper());
        var content = new StringContent(message.Body, System.Text.Encoding.UTF8, "application/json");

        context.Logger.LogLine($"URL: {baseUrl}{methodName}");

        HttpResponseMessage response;
        if (httpMethod.ToUpper() == "POST")
        {
            response = await client.PostAsync($"{baseUrl}{methodName}", content);
        }
        else
        {
            response = await client.PutAsync($"{baseUrl}{methodName}", content);
        }
        
        response.EnsureSuccessStatusCode();
        context.Logger.LogLine("Document upload success");

        await Task.CompletedTask;
 }

Answer 1

There are many different processes modifying the same client at the same time (client.DefaultRequestHeaders.Add(..)) - that could be an issue.

I would suggest creating a separate headers-object per message/HTTP request, and not rely on the default headers at all (unless they are shared across all requests).