Load balancing with nginx lower RPS than single node

Question

I setup a nginx load balancer for 4 blockchain clients and am getting lower requests per second by load testing with k6 through the load balancer than I do if I target an individual node. All of the nodes are responding to equal amount of traffic and none of them seem to be responding slower than the next.

The load balancer is setup with the bitnami nginx helm chart and the blockchain clients are running on their own VMs.

**nginx.conf**

    # Based on https://www.nginx.com/resources/wiki/start/topics/examples/full/#nginx-conf
    # user              www www;  ## Default: nobody;

    load_module modules/ngx_http_geoip2_module.so;
    load_module modules/ngx_stream_geoip2_module.so;

    worker_processes  auto;
    error_log         "/opt/bitnami/nginx/logs/error.log";
    pid               "/opt/bitnami/nginx/tmp/nginx.pid";

    events {
        worker_connections  1024;
    }

    http {
        include       mime.types;
        default_type  application/octet-stream;
        log_format    main '$remote_addr - $remote_user [$time_local] '
                          '"$request" $status  $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for"';
        access_log    "/opt/bitnami/nginx/logs/access.log";
        add_header    X-Frame-Options SAMEORIGIN;

        client_body_temp_path  "/opt/bitnami/nginx/tmp/client_body" 1 2;
        proxy_temp_path        "/opt/bitnami/nginx/tmp/proxy" 1 2;
        fastcgi_temp_path      "/opt/bitnami/nginx/tmp/fastcgi" 1 2;
        scgi_temp_path         "/opt/bitnami/nginx/tmp/scgi" 1 2;
        uwsgi_temp_path        "/opt/bitnami/nginx/tmp/uwsgi" 1 2;

        sendfile           on;
        tcp_nopush         on;
        tcp_nodelay        off;
        gzip               on;
        gzip_http_version  1.0;
        gzip_comp_level    2;
        gzip_proxied       any;
        gzip_types         text/plain text/css application/javascript text/xml application/xml+rss;
        keepalive_timeout  65;
        ssl_protocols      TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
        ssl_ciphers        HIGH:!aNULL:!MD5;
        client_max_body_size 80M;
        server_tokens off;

        include  "/opt/bitnami/nginx/conf/server_blocks/*.conf";
    }

**serverBlock**

upstream backend {
  server 1.2.3.4:8000;
  server 2.2.3.4:8000;
  server 3.2.3.4:8000;
  server 4.2.3.4:8000;
}
server {
  listen 0.0.0.0:8080;
  location / {
      proxy_pass http://backend;
  }

  location /status {
    stub_status on;
    access_log   off;
    allow 127.0.0.1;
    deny all;
  }
}

Nothing is standing out from the nginx metrics or from inspecting the resource uses of the containers.

Does anyone have any suggestions for what to look into next to debug this or what the culprit could be?

Thank you for your help.

Answer 1

Ended up not being nginx at all as I wrongly assumed that it was the bottleneck. Setup a little hello world example which after load tests pointed me to the real bottleneck in my API gateway.