Thresholds are a pass/fail criteria used to specify the performance expectations of the system under test.
Example expectations (Thresholds):
- System doesn't produce more than 1% errors
- Response time for 95% of requests should be below 200ms.
- Response time for 99% of requests should be below 400ms.
- Specific endpoint must always respond within 300ms.
Thresholds analyze the performance metrics and determine the final test result (pass/fail). Thresholds are a essential for load-testing automation.
Here is a sample script that specifies two thresholds, one using a custom Rate metric, and one using a standard http_req_duration metric.
The failed requests threshold specifies that we want our load test to be considered a failure (resulting in k6 exiting with a non-zero exit code) if 10% or more of requests resulted in the server returning anything else than a 200-response. The http_req_duration threshold specifies that 95% of requests must complete within 500ms.
In other words, you specify the pass criteria when defining your threshold, and if that expression evaluates to false at the end of the test, the whole test will be considered a fail.
In the above case, criteria for both thresholds were met. The whole load test is considered to be a pass, which means that k6 will exit with exit code zero.
If any of the thresholds had failed, the little green checkmark✓ next to the threshold name (`failed requests`, `http_req_duration`) would have been a red cross ✗ instead, and k6 would have generated a non-zero exit code.
The quickest way to start with thresholds is to use the standard, built-in k6 metrics.
Here are a few copy-paste examples that you can start using right away.
Thresholds can be specified in a short or full format.
The above declaration inside a k6 script means that there will be a threshold configured for the metric metric_name1. To determine if the threshold has failed or passed, the string 'threshold_expression' will be evaluated. The 'threshold_expression' must follow the following format.
aggregation_method operator value
- avg < 200 // average duration can't be larger than 200ms
- count >= 500 // count must be larger or equal to 500
- p(90) < 300 // 90% of samples must be below 300
A threshold expression evaluates to true or false.
Each of the four metric types included in k6 provide its own set of aggregation methods usable in threshold expressions.
|Metric type||Aggregation methods|
|Counter||count and rate|
|Trend||avg, min, max, med and p(N) where N is a number between 0.0 and 100.0 meaning the percentile value to look at, e.g. p(99.99) means the 99.99th percentile. The unit for these values is milliseconds.|
Here is a (slightly contrived) sample script that uses all different types of metrics, and sets different types of thresholds for them:
We have these thresholds:
- A trend metrics that is fed with response time samples, and which has the following threshold
- 99th percentile response time must be below 300 ms
- 70th percentile response time must be below 250 ms
- Average response time must be below 200 ms
- Median response time must be below 150 ms
- Minimum response time must be below 100 ms
- A rate metric that keeps track of how often the content returned was OK. This metric has one success criteria: content must have been OK more than 95% of the time.
- A gauge metric that contains the latest size of the returned content. The success criteria for this metric is that the returned content must be smaller than 4000 bytes.
- A counter metric that keeps track of the total number of times content returned was not OK. The success criteria here implies that content can't have been bad more than 99 times.
It's often useful to specify thresholds only on a single URL or a specific tag. In k6, tagged requests create sub-metrics that can be used in thresholds as shown below.
And here's a full example.
If you want to abort a test as soon as a threshold is crossed, before the test has completed, there's an extended threshold specification format that looks like this:
As you can see in the example above the threshold specification has been extended to alternatively support a JS object with parameters to control the abort behavior. The fields are as follows:
|threshold||string||This is the threshold expression string specifying the threshold condition to evaluate.|
|abortOnFail||boolean||Whether to abort the test if the threshold is evaluated to false before the test has completed.|
|delayAbortEval||string||If you want to delay the evaluation of the threshold for some time, to allow for some metric samples to be collected, you can specify the amount of time to delay using relative time strings like 10s, 1m and so on.|
When k6 runs in the cloud, thresholds are evaluated every 60 seconds, therefore the "abortOnFail" feature may be delayed by up to 60 seconds.
Checks are nice for codifying assertions, but unlike thresholds, checks will not affect the exit status of k6.
If you only use checks to verify that things work as expected, you will not be able to fail the whole test run based on the results of those checks.
It can often be useful to combine checks and thresholds, to get the best of both:
In this example, the threshold is configured on the checks metric - establishing that the rate of successful checks should be higher than 90%.
Additionally, you can use tags on checks if you want to define a threshold based on a particular check or group of checks. For example:
You can also see how the underlying metric compares to a specific threshold throughout the test. The threshold can be added to the analysis tab for further comparison against other metrics.
Learn more about analyzing results in the k6 Cloud Results docs.