How to generate a constant request rate in k6 with the new scenarios API?

📖What you will learn

• How to generate a constant request rate in k6
• How to utilize the `scenarios` api to configure executors

Overview

This v0.27.0 release includes a new execution engine and lots of new executors that cater to your specific needs. It also includes the new scenarios API with lots of different options to configure and model the load on the system under test (SUT). This is the result of 1.5 years of work on the infamous #1007 PR 😂.

For generating constant request rates, we may use the constant-arrival-rate executor. This executor starts the test with iterations at a fixed rate for a specified duration. This allows k6 to dynamically change the amount of active VUs during a test run, with the goal of achieving the specified amount of iterations per time unit. In this article, I am going to explain how to use this scenario to generate constant request rates.

Configuring a scenario with the constant-arrival-rate executor

Let's look at different terms used in k6 to describe a test configuration in a scenario that uses a constant-arrival-rate executor:

executor

Executors are the workhorses of the k6 execution engine. Each one schedules VUs and iterations differently, and you'll choose one depending on the type of traffic you want to model to test your services.

rate and timeUnit

k6 tries to start rate iterations every timeUnit period. For example:

• rate: 1, timeUnit: '1s' means "try to start 1 iteration every second"
• rate: 1, timeUnit: '1m' means "try to start 1 iteration every minute"
• rate: 90, timeUnit: '1m' means "try to start 90 iterations per minute", i.e. 1.5 iterations/s or try to start a new iteration every 667ms
• rate: 50, timeUnit: '1s' means "try to start 50 iterations every second", i.e. 50 RPS if we have 1 request in our iteration, i.e. try to start a new iteration every 20ms

duration

The total duration of the scenario, excluding gracefulStop.

preAllocatedVUs

The number of VUs to pre-allocate before the test starts.

maxVUs

The maximum number of VUs to allow during the test run.

Together, these terms form a scenario, which is part of the test configuration options. The code snippet below is an example of a constant-arrival-rate scenario.

In this configuration, we have a constant_request_rate scenario, which is a unique identifier used as a label for the scenario. This scenario uses the constant-arrival-rate executor and executes for 1 minute. Each second (timeUnit), 1 iteration will be made (rate). The pool of pre-allocated virtual users contains 20 instances and may go up to 100, depending on the number of requests and iterations.

Keep in mind that initializing VUs mid-test could be a CPU-heavy task and might skew your test results. In general, it's better for the preAllocatedVUs to be enough to run the load test. So, make sure to allocate more VUs depending on the number of requests you have in your test and the rate at which you want to run the test.

export const options = {
  scenarios: {
    constant_request_rate: {
      executor: 'constant-arrival-rate',
      rate: 1,
      timeUnit: '1s',
      duration: '1m',
      preAllocatedVUs: 20,
      maxVUs: 100,
    },
  },
};

Generating a constant request rate with constant-arrival-rate

In the previous tutorial, we demonstrated how to calculate a constant request rate. Let's run through it again, taking into account how scenarios work:

Suppose that you expect your SUT to handle 1000 requests per second on an endpoint. Pre-allocating 100 VUs (with a maximum of 200) allows each VU to send roughly 5~10 requests (based on 100~200 VUs). If each request takes more than 1 second to complete, you'll end up making fewer requests than expected (more dropped_iterations), which is a sign of either performance issues with your SUT or having unrealistic expectations. Thus, fix performance issues and test again, or lower your expectations by adjusting the timeUnit.

In this scenario, each pre-allocated VU will make 10 requests (rate divided by preAllocatedVUs). If the requests don't make it in 1 second, e.g. the response took more than 1 second to receive or your SUT took more than 1 second to complete the task, k6 will increase the number of VUs to account for missing requests. The following test generates 1000 requests per second and runs for 30 seconds, which roughly generates 30,000 requests, as you can see below in the output: http_reqs and iterations. Also, k6 has only used 148 VUs from the pool of 200 VUs.

import http from 'k6/http';

export const options = {
  scenarios: {
    constant_request_rate: {
      executor: 'constant-arrival-rate',
      rate: 1000,
      timeUnit: '1s', // 1000 iterations per second, i.e. 1000 RPS
      duration: '30s',
      preAllocatedVUs: 100, // how large the initial pool of VUs would be
      maxVUs: 200, // if the preAllocatedVUs are not enough, we can initialize more
    },
  },
};

export default function () {
  http.get('http://test.k6.io/contacts.php');
}

The result of executing this script is as follows:

$ k6 run test.js

          /\      |‾‾|  /‾‾/  /‾/
     /\  /  \     |  |_/  /  / /
    /  \/    \    |      |  /  ‾‾\
   /          \   |  |‾\  \ | (_) |
  / __________ \  |__|  \__\ \___/ .io

  execution: local
     script: test.js
     output: -

  scenarios: (100.00%) 1 executors, 200 max VUs, 1m0s max duration (incl. graceful stop):
           * constant_request_rate: 1000.00 iterations/s for 30s (maxVUs: 100-200, gracefulStop: 30s)


running (0m30.2s), 000/148 VUs, 29111 complete and 0 interrupted iterations
constant_request_rate ✓ [======================================] 148/148 VUs  30s  1000 iters/s


    data_received..............: 21 MB  686 kB/s
    data_sent..................: 2.6 MB 85 kB/s
    *dropped_iterations.........: 889    29.454563/s
    http_req_blocked...........: avg=597.53µs min=1.64µs  med=7.28µs   max=152.48ms p(90)=9.42µs   p(95)=10.78µs
    http_req_connecting........: avg=561.67µs min=0s      med=0s       max=148.39ms p(90)=0s       p(95)=0s
    http_req_duration..........: avg=107.69ms min=98.75ms med=106.82ms max=156.54ms p(90)=111.73ms p(95)=116.78ms
    http_req_receiving.........: avg=155.12µs min=21.1µs  med=105.52µs max=34.21ms  p(90)=147.69µs p(95)=190.29µs
    http_req_sending...........: avg=46.98µs  min=9.81µs  med=41.19µs  max=5.85ms   p(90)=53.33µs  p(95)=67.3µs
    http_req_tls_handshaking...: avg=0s       min=0s      med=0s       max=0s       p(90)=0s       p(95)=0s
    http_req_waiting...........: avg=107.49ms min=98.62ms med=106.62ms max=156.39ms p(90)=111.52ms p(95)=116.51ms
    *http_reqs..................: 29111  964.512705/s
    iteration_duration.........: avg=108.54ms min=99.1ms  med=107.08ms max=268.68ms p(90)=112.09ms p(95)=118.96ms
    *iterations.................: 29111  964.512705/s
    vus........................: 148    min=108 max=148
    vus_max....................: 148    min=108 max=148

Considerations

There are some things to consider while writing your test script:

1. Since k6 follows redirects, the number of redirects add to the total number of RPS in the result output. If you don't want that, you can disable it globally by setting maxRedirects: 0 in your options. You can also configure the maximum redirects on the http request itself, which will override the global maxRedirects.

2. Complexity adds to the mix. So, keep the function being executed simple, preferably only executing a few requests, avoiding additional processing or sleep() calls where possible.

3. You need a fair amount of VUs to achieve the desired results, otherwise, you'll encounter warning message(s) like the following. In this case, just increase the preAllocatedVUs and/or maxVUs, but keep in mind that you will reach the maximum capacity of the machine running the test at some point, where neither preAllocatedVUs, nor maxVUs, will make any difference.

   WARN[0005] Insufficient VUs, reached 100 active VUs and cannot initialize more  executor=constant-arrival-rate scenario=constant_request_rate

   Copy

4. As you can see in the above results, there are dropped_iterations and the number of iterations and http_reqs is less than the specified rate. Having dropped_iterations set means that there weren't enough initialized VUs to execute some of the iterations. The issue can generally be solved by increasing preAllocatedVUs. The precise value requires a bit of trial and error since it depends on different factors, including the endpoint response time, network throughput, and other related latencies.

5. While testing, you may encounter the following warning message(s) that signify you have reached your operating system limits. So, consider fine-tuning your operating system:

   WARN[0008] Request Failed error="Get \"http://test.k6.io/contacts.php\": dial tcp 3.227.130.174:80: socket: too many open files"

   Copy

6. Beware that the scenarios API deprecates global usages of duration, vus and stages, although they can still be used. This also means that you can't use them together with scenarios.

Conclusion

Before the release of k6 v0.27.0, there was insufficient support for generating constant request rates. Therefore, we introduced a JavaScript workaround by calculating the time it takes for requests to exhaust each iteration of the script. With v0.27.0, this is no longer needed.

In this article, I've explained how k6 can achieve a constant request rate by using the new scenarios API with the constant-arrival-rate executor. This executor simplifies the code and provides the means to achieve fixed RPS. This is in contrast with the previous version of the same article which I described another method to achieve pretty much the same results by calculating the number of VUs, iterations and duration using a formula and some boilerplate JavaScript code. Fortunately, this new approach works as intended and we don't need to use any hacks anymore.

I hope you enjoyed reading this article and I'd be happy to hear your feedback.