AWS API Gateway Slowness

Question

This has me perturbed. I have a basic API gateway that is supposed to be capped at 10,000 requests per second with 5,000 request bursts. However, when hooked up to Lambdas, best I can hit currently is ~70 requests / second.

The end-points I have are basic Lambda proxies created with Serverless framework (HTTP EDGE).

I know that the lambda itself is not the bottleneck as I have the same issue when I replace the lambda with an empty function. I have 100+ allocated concurrency for the lambda, but the lambda never appears to hit the limit.

functions:
  loadtest:
    handler: loadtest/index.handler
    reservedConcurrency: 200
    events:
      - http: POST load_test

I'm wondering if there's something that I am overlooking here. My test runs for a minute and attempts to hit 200 req / sec (works fine with other so it's not my bandwidth). The delays grow to be as much as 20-30s at some point, so clearly something is choking up.

If it's a warm up issue - how long am I expected to run such load until everything is running warm?

Any ideas on where to look or additional information that I could share?

[Edit] I am using node12.x and I even tried with this code:

const AWS = require('aws-sdk');

AWS.config.update({region: '<my-region>'});

var sqs = new AWS.SQS({apiVersion: '2012-11-05'});

exports.handler = async (event, context) => {
  return {"status":"ok", ... }
};

The results were basically the same. I'm not sure where the bottle neck is, to be honest. I can try further testing with concurrency on the lambda side, but going from 100 to 200 had no effect - the completed requests clocks at around 70/s for an empty function.

Also, I'm using loadtest npm package to perform the loadtest and this is what the output looks like:

{ totalRequests: 8200,
  totalErrors: 0,
  totalTimeSeconds: 120.00341689999999,
  rps: 68,
  meanLatencyMs: 39080.6,
  maxLatencyMs: 78490,
  minLatencyMs: 427,
  percentiles: { '50': 38327, '90': 70684, '95': 74569, '99': 77679 },
  errorCodes: {},
  instanceIndex: 0 }

Here's a picture of how provisioned concurrency looked like over that period of time. I ran this over 2 minutes with the target at 200 req/sec.

Answer 1

Appears that this is actually an issue with WSL2 and NodeJS. The exact nature of it is still unclear, but it is not an issue with the API gateway itself. I demonstrated this by running it on MacBook and everything worked fine and request counts were high.

There are posts suggesting that this is an issue with the Node HTTP client & DNS, so perhaps that's a good starting point, but the above question is moot.