Search code examples
mysqlnode.jsperformancenginxperformance-testing

Perfomance issue (Nginx, NodeJs, Mysql)

I have the following problem. Using REST, I am getting binary content (BLOBs) from a MySql database via a NodeJS Express app. All works fine, but I am having issues scaling the solution.

I increased the number of NodeJS instances to 3 : they are running ports 4000,4001,4002. On the same machine I have Nginx installed and configured to do a load balancing between my 3 instances. I am using Apache Bench to do some perf testing. Please see attached pic.

Perf testing

Assuming I have a dummy GET REST that goes to the db, reads the blob (roughly 600KB in size) and returns it back (all http), I am making 300 simultaneous calls. I would have thought that using nginx to distribute the requests would make it faster, but it does not. enter image description here

Why is this happening? I am assuming it has to do with MySql? My NodeJs app is using a connection pool with a limit set to 100 connections. What should be the relation between this value and the max connection value in Mysql? If I increase the connection pool to a higher number of connections, I get worse results.

Any suggestion on how to scale?

Thanks!

Solution

  • "300 simultaneous" is folly. No one (today) has the resources to effectively do more than a few dozen of anything.

    • 4 CPU cores -- If you go much beyond 4 threads, they will be stumbling over each over trying to get CPU time.
    • 1 network -- Have you check to see whether your big blobs are using all the bandwidth, thereby being the bottleneck?
    • 1 I/O channel -- Again, lots of data could be filling up the pathway to disk.

    (This math is not quite right, but it makes a point...) You cannot effectively run any faster than what you can get from 4+1+1 "simultaneous" connections. (In reality, you may be able to, but not 300!)

    The typical benchmarks try to find how many "connections" (or whatever) leads to the system keeling over. Those hard-to-read screenshots say about 7 per second is the limit.

    I also quibble with the word "simultaneous". The only thing close to "simultaneous" (in your system) is the ability to use 4 cores "simultaneously". Every other metric involves sharing of resources. Based on what you say, ...

    • If you start about 7 each second, some resource will be topped out, but each request will be fast (perhaps less than a second)
    • If you start 300 all at once, they will stumble over each other, some of them taking perhaps minutes to finish.

    There are two interesting metrics:

    • How many per second you can sustain. (Perhaps 7/sec)
    • How long the average (and, perhaps, the 95% percentile) takes.

    Try 10 "simultaneous" connections and report back. Try 7. Try some other small numbers like those.