Thresholds

What are thresholds?

Thresholds are pass/fail criteria that specify the performance expectations of the system under test.

For example, you can use thresholds to test that your system meets the following expectations:

Less than 1% of requests return an error.
95% of requests have a response time below 200ms.
99% of requests have a response time below 400ms.
A specific endpoint always responds within 300ms.
Any conditions for a custom metric.

Thresholds analyze the performance metrics and determine whether the final results passed or failed the test. Thresholds are essential for load-testing automation.

Here is a sample script that specifies two thresholds. One threshold evaluates the rate of HTTP errors (http_req_failed metric). The other evaluates whether 95 percent of responses happen within a certain duration (the http_req_duration metric).

1import http from 'k6/http';
2
3export const options = {
4  thresholds: {
5    http_req_failed: ['rate<0.01'], // http errors should be less than 1%
6    http_req_duration: ['p(95)<200'], // 95% of requests should be below 200ms
7  },
8};
9
10export default function () {
11  http.get('https://test-api.k6.io/public/crocodiles/1/');
12}

In other words, when you define your threshold, specify an expression for a pass criteria. If that expression evaluates to false at the end of the test, k6 considers the whole test a fail.

After executing that script, k6 outputs something similar to this:

   ✓ http_req_duration..............: avg=151.06ms min=151.06ms med=151.06ms max=151.06ms p(90)=151.06ms p(95)=151.06ms
       { expected_response:true }...: avg=151.06ms min=151.06ms med=151.06ms max=151.06ms p(90)=151.06ms p(95)=151.06ms
   ✓ http_req_failed................: 0.00%  ✓ 0 ✗ 1

In this case, the test met the criteria for both thresholds. k6 considers this test a pass and exits with an exit code 0.

If any of the thresholds had failed, the little green checkmark ✓ next to the threshold name (http_req_failed, http_req_duration) would have been a red cross ✗, and k6 would have generated a non-zero exit code.

Copy-paste threshold examples

The quickest way to start with thresholds is to use the standard, built-in k6 metrics. Here are a few copy-paste examples that you can start using right away.

For more specific threshold examples, refer to the Counter, Gauge, Trend and Rate pages.

A percentile of requests finishes in a specified duration

1import http from 'k6/http';
2import { sleep } from 'k6';
3
4export const options = {
5  thresholds: {
6    // 90% of requests must finish within 400ms.
7    http_req_duration: ['p(90) < 400'],
8  },
9};
10
11export default function () {
12  http.get('https://test-api.k6.io/public/crocodiles/1/');
13  sleep(1);
14}

Error rate is lower than 1 percent

1import http from 'k6/http';
2import { sleep } from 'k6';
3
4export const options = {
5  thresholds: {
6    // During the whole test execution, the error rate must be lower than 1%.
7    http_req_failed: ['rate<0.01'],
8  },
9};
10
11export default function () {
12  http.get('https://test-api.k6.io/public/crocodiles/1/');
13  sleep(1);
14}

Multiple thresholds on a single metric

You can also apply multiple thresholds for one metric. This threshold has different duration requirements for different request percentiles.

1import http from 'k6/http';
2import { sleep } from 'k6';
3
4export const options = {
5  thresholds: {
6    // 90% of requests must finish within 400ms, 95% within 800, and 99.9% within 2s.
7    http_req_duration: ['p(90) < 400', 'p(95) < 800', 'p(99.9) < 2000'],
8  },
9};
10
11export default function () {
12  const res1 = http.get('https://test-api.k6.io/public/crocodiles/1/');
13  sleep(1);
14}

Threshold on group duration

You can set thresholds per Group. This code has groups for individual requests and batch requests. For each group, there are different thresholds.

1import http from 'k6/http';
2import { group, sleep } from 'k6';
3
4export const options = {
5  thresholds: {
6    'group_duration{group:::individualRequests}': ['avg < 400'],
7    'group_duration{group:::batchRequests}': ['avg < 200'],
8  },
9  vus: 1,
10  duration: '10s',
11};
12
13export default function () {
14  group('individualRequests', function () {
15    http.get('https://test-api.k6.io/public/crocodiles/1/');
16    http.get('https://test-api.k6.io/public/crocodiles/2/');
17    http.get('https://test-api.k6.io/public/crocodiles/3/');
18  });
19
20  group('batchRequests', function () {
21    http.batch([
22      ['GET', `https://test-api.k6.io/public/crocodiles/1/`],
23      ['GET', `https://test-api.k6.io/public/crocodiles/2/`],
24      ['GET', `https://test-api.k6.io/public/crocodiles/3/`],
25    ]);
26  });
27
28  sleep(1);
29}

Threshold Syntax

To use a threshold, you must define at least one threshold_expression:

1export const options = {
2  thresholds: {
3    metric_name1: ['threshold_expression', `...`], // short format
4    metric_name2: [
5      {
6        threshold: 'threshold_expression',
7        abortOnFail: true, // boolean
8        delayAbortEval: '10s', // string
9      },
10    ], // full format
11  },
12};

This declaration configures thresholds for the metrics metric_name1 and metric_name2. To determine whether the threshold passes or fails, the script evaluates the 'threshold_expression'.

The 'threshold_expression' must follow the format:

aggregation_method operator value

Examples:

avg < 200 // average duration must be less than 200ms
count >= 500 // count must be larger than or equal to 500
p(90) < 300 // 90% of samples must be below 300

A threshold expression evaluates to true or false.

Each of the four metric types included in k6 provides a set of aggregation methods that you can use in threshold expressions.

Metric type	Aggregation methods
Counter	count and rate
Gauge	value
Rate	rate
Trend	avg, min, max, med and p(N) where N is a number between 0.0 and 100.0 meaning the percentile value to look at, e.g. p(99.99) means the 99.99th percentile. The unit for these values is milliseconds.

Here is a (slightly contrived) sample script that uses all different types of metrics, setting different types of thresholds for each:

1import http from 'k6/http';
2import { Trend, Rate, Counter, Gauge } from 'k6/metrics';
3import { sleep } from 'k6';
4
5export const TrendRTT = new Trend('RTT');
6export const RateContentOK = new Rate('Content OK');
7export const GaugeContentSize = new Gauge('ContentSize');
8export const CounterErrors = new Counter('Errors');
9export const options = {
10  thresholds: {
11    'Errors': ['count<100'],
12    'ContentSize': ['value<4000'],
13    'Content OK': ['rate>0.95'],
14    'RTT': ['p(99)<300', 'p(70)<250', 'avg<200', 'med<150', 'min<100'],
15  },
16};
17
18export default function () {
19  const res = http.get('https://test-api.k6.io/public/crocodiles/1/');
20  const contentOK = res.json('name') === 'Bert';
21
22  TrendRTT.add(res.timings.duration);
23  RateContentOK.add(contentOK);
24  GaugeContentSize.add(res.body.length);
25  CounterErrors.add(!contentOK);
26
27  sleep(1);
28}

We have these thresholds:

A counter metric that keeps track of the total number of times that the content response was not OK. The success criteria here is that content cannot be bad more than 99 times.
A gauge metric that contains the latest size of the returned content. The success criteria for this metric is that the returned content is smaller than 4000 bytes.
A rate metric that keeps track of how often the content returned was OK. This metric has one success criteria: content must have been OK more than 95% of the time.
A trend metric that is fed with response time samples and which has the following threshold criteria:
- 99th percentile response time must be below 300 ms
- 70th percentile response time must be below 250 ms
- Average response time must be below 200 ms
- Median response time must be below 150 ms
- Minimum response time must be below 100 ms

⚠️ Common mistake Do not specify multiple thresholds for the same metric by repeating the same object key:

1export const options = {
2  thresholds: {
3    // avoid using the same metric more than once here
4    // metric_name: [ 'count<100' ],
5    // metric_name: [ 'rate<50' ],
6  },
7};

Since thresholds are defined as the properties of a JavaScript object, it's not possible to specify multiple ones with the same property name. Only the last one will remain. The rest will be silently ignored. Instead, specify them with an array for the same key.

Thresholds on tags

It's often useful to specify thresholds on a single URL or specific tag. In k6, tagged requests create sub-metrics that you can use in thresholds:

export const options = {
  thresholds: {
    'metric_name{tag_name:tag_value}': ['threshold_expression'],
  },
};

And here's a full example.

1import http from 'k6/http';
2import { sleep } from 'k6';
3import { Rate } from 'k6/metrics';
4
5export const options = {
6  thresholds: {
7    'http_req_duration{type:API}': ['p(95)<500'], // threshold on API requests only
8    'http_req_duration{type:staticContent}': ['p(95)<200'], // threshold on static content only
9  },
10};
11
12export default function () {
13  const res1 = http.get('https://test-api.k6.io/public/crocodiles/1/', {
14    tags: { type: 'API' },
15  });
16  const res2 = http.get('https://test-api.k6.io/public/crocodiles/2/', {
17    tags: { type: 'API' },
18  });
19
20  const responses = http.batch([
21    ['GET', 'https://test-api.k6.io/static/favicon.ico', null, { tags: { type: 'staticContent' } }],
22    [
23      'GET',
24      'https://test-api.k6.io/static/css/site.css',
25      null,
26      { tags: { type: 'staticContent' } },
27    ],
28  ]);
29
30  sleep(1);
31}

Aborting a test when a threshold is crossed

If you want to abort a test as soon as a threshold is crossed, before the test finishes, there's an extended threshold specification format:

1export const options = {
2  thresholds: {
3    metric_name: [
4      {
5        threshold: 'p(99) < 10', // string
6        abortOnFail: true, // boolean
7        delayAbortEval: '10s', // string
8        /*...*/
9      },
10    ],
11  },
12};

In this example, the threshold specification has been extended to alternatively support a JS object with parameters to control the abort behavior. The fields are as follows:

Name	Type	Description
threshold	string	This is the threshold expression string specifying the threshold condition to evaluate.
abortOnFail	boolean	Whether to abort the test if the threshold is evaluated to false before the test has completed.
delayAbortEval	string	If you want to delay the evaluation of the threshold to let some metric samples to be collected, you can specify the amount of time to delay using relative time strings like 10s, 1m and so on.

Here is an example:

1import http from 'k6/http';
2
3export const options = {
4  vus: 30,
5  duration: '2m',
6  thresholds: {
7    http_req_duration: [{ threshold: 'p(99) < 10', abortOnFail: true }],
8  },
9};
10
11export default function () {
12  http.get('https://test-api.k6.io/public/crocodiles/1/');
13}

⚠️ Evaluation delay in the cloud
When k6 runs in the cloud, thresholds are evaluated every 60 seconds, therefore the abortOnFail feature may be delayed by up to 60 seconds.

Failing a load test using checks

Checks are nice for codifying assertions, but unlike thresholds, checks do not affect the exit status of k6.

If you use only checks to verify that things work as expected, you can't fail the whole test run based on the check results.

It's often useful to combine checks and thresholds, to get the best of both:

1import http from 'k6/http';
2import { check, sleep } from 'k6';
3
4export const options = {
5  vus: 50,
6  duration: '10s',
7  thresholds: {
8    // the rate of successful checks should be higher than 90%
9    checks: ['rate>0.9'],
10  },
11};
12
13export default function () {
14  const res = http.get('http://httpbin.test.k6.io');
15
16  check(res, {
17    'status is 500': (r) => r.status == 500,
18  });
19
20  sleep(1);
21}

In this example, the threshold is configured on the checks metric, establishing that the rate of successful checks is higher than 90%.

Additionally, you can use tags on checks if you want to define a threshold based on a particular check or group of checks. For example:

1import http from 'k6/http';
2import { check, sleep } from 'k6';
3
4export const options = {
5  vus: 50,
6  duration: '10s',
7  thresholds: {
8    'checks{myTag:hola}': ['rate>0.9'],
9  },
10};
11
12export default function () {
13  let res;
14
15  res = http.get('http://httpbin.test.k6.io');
16  check(res, {
17    'status is 500': (r) => r.status == 500,
18  });
19
20  res = http.get('http://httpbin.test.k6.io');
21  check(
22    res,
23    {
24      'status is 200': (r) => r.status == 200,
25    },
26    { myTag: 'hola' }
27  );
28
29  sleep(1);
30}

Thresholds in k6 Cloud Results

In k6 Cloud Results Thresholds are available in their own tab for analysis.

You can also see how the underlying metric compares to a specific threshold throughout the test. The threshold can be added to the analysis tab for further comparison against other metrics.

Learn more about analyzing results in the k6 Cloud Results docs.

Getting started

Using k6

Test authoring

Results visualization

Test Types

Testing Guides

Misc