Testing Minio with K6, but despite large number of VUs sending, the number of requests is very small

Question

I'm currently using K6 to load test a simple API that downloads objects from a Minio server. I am using NestJS to test the API. The main function being called is downloadObject, which downloads objects from Minio using the stream API provide by nodejs. Here is my NestJS service and k6 configuration file:

@Injectable()
export class S3Service {
  private readonly minioClientCache: Map<string, minio.Client> = new Map();

  async getS3Client() {
    const endpoint = this.configService.get('s3.endpoint');

    if (!this.s3ClientCache.has(endpoint)) {
      const s3 = new S3({
        apiVersion: '2006-03-01',
        forcePathStyle: true,
        tls: false,
        region: 'us-west-1',
        endpoint: this.configService.get('s3.endpoint'),
        credentials: {
          accessKeyId: this.configService.get('s3.accessKey'),
          secretAccessKey: this.configService.get('s3.secretKey'),
        },
      });
      this.s3ClientCache.set(endpoint, s3);
    }
    return this.s3ClientCache.get(endpoint);
  }

  generateRandomString(length: number) {
    let result = '';
    const characters =
      'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789';
    for (let i = 0; i < length; i++) {
      result += characters.charAt(
        Math.floor(Math.random() * characters.length),
      );
    }
    return result;
  }

  async downloadObject(s3Path: string) {
    const s3Client = await this.getS3Client();

    const command = new GetObjectCommand({
      Bucket: 'artifacts',
      Key: s3Path,
    });

    try {
      const response = await s3Client.send(command);
      const inputStream = response.Body;
      const subFolderName = this.generateRandomString(10);
      if (!fs.existsSync(`./s3_objects/${subFolderName}`)) {
        fs.mkdirSync(`./s3_objects/${subFolderName}`);
      }

      const downloadPath = `./s3_objects/${subFolderName}/${basename(s3Path)}`;

      const outputStream = fs.createWriteStream(downloadPath);

      if (inputStream instanceof Readable) {
        console.log('inputStream has been piped');
        inputStream.pipe(outputStream);
      }
      outputStream.on('finish', function () {
        console.log('File downloaded successfully');
        return HttpStatus.CREATED;
      });
      outputStream.on('error', function () {
        console.log('Error downloading file');
        return HttpStatus.INTERNAL_SERVER_ERROR;
      });
    } catch (err) {
      console.error(err);
    }
  }
}

import http from 'k6/http';
import { sleep, check } from 'k6';

export const options = {
  discardResponseBodies: true,
  scenarios: {
    load: {
      executor: 'ramping-vus',
      startVUs: 0,
      stages: [
        { duration: '5m', target: 50 },
        { duration: '5m', target: 200 },
        { duration: '5m', target: 0 },
      ],
      tags: { test_type: 'load' },
      gracefulRampDown: '5m',
    },
  },
  thresholds: {
    'http_req_duration{test_type:load}': ['p(99)<180000'],
  },
};

export default async function () {
  const url = 'http://localhost:4000/api/s3/object/download';
  const data = {
    s3Path: 'TEST_PATH/2463/test.zip',
  };
  const headers = {
    'Content-Type': 'application/x-www-form-urlencoded',
  };
  await http.asyncRequest('POST', url, data, {
    headers,
    timeout: '240s',
  });
  sleep(1);
}

I'm not sure why, but K6 is only able to send 1-4 requests at a time, despite the large number of VUs executing the javascript file. I was thinking this may be because of how I implemented file streaming, but I'm not sure. Any thoughts? Here is what the data looks like:

Answer 1

The number of VUs doesn't tell you anything about the request rate. A single VU will perform a single request, wait until that request is finished, then start the next request. Since your requests seem to take more than 16 seconds, judging from the screenshot; that means a single VU will send less than 4 requests per minute. With 60 VUs that will correspond to 4 requests per second.

If you want to control the request rate, don't use a VU executor, but use an arrival-rate executor instead, e.g. the ramping-arrival-rate executor. Arrival-rate executors will inject additional VUs to achieve the desired request rate.

Of course, that won't help much if your service is already overloaded with a small number of VUs/requests. Looking at the screenshot again, your service's minimum latency is already 10+ seconds with less than 20 concurrent users.

As to why your service is so slow, that should probably be a separate post (only one question per post: either "why does k6 only send a low number of requests" or "why is my minio service only handling a low number of concurrent requests").