k6 v0.31.0 released

k6 v0.31.0 is here! 🎉 It's a smaller release with some significant performance improvements, as well as some really exciting new features. For additionall details, see the full release notes on GitHub.

Outputs

Major cleanup

The state of k6's output packages has been a development pain point for a long time, which made it difficult to add new outputs in a consistent way. In the refactor done in v0.31.0, this has been mostly addressed and output implementations now follow a much simpler and cleaner model.

Output Extensions

In addition to this, it is now possible to implement custom k6 output extensions in Go with xk6! This is very useful if you use a system that's currently not supported by the built-in outputs, or need some custom handling of the metrics k6 produces.

Writing output extensions is done very similarly to how JS module extensions are currently written, but with some minor adjustments. To learn more about this, see the full release notes on GitHub.

We are working on the proper documentation for this, as well as the overdue xk6 documentation about JS extensions, so keep a lookout for those here as well as on k6.io/docs.

Marking requests as failed

It's now possible to declare expected HTTP response statuses for either the entire test or for individual HTTP requests, and k6 will emit a new http_req_failed metric as well as tag HTTP metrics with expected_response: <bool>. By default, k6 will now fail requests that return HTTP 4xx/5xx response codes.

For example:

import http from 'k6/http';

// Set expected statuses globally for all requests.
http.setResponseCallback(http.expectedStatuses({ min: 200, max: 399 }, 418));

export default function () {
  // This request will be marked as failed.
  http.get('https://httpbin.test.k6.io/status/400');
  // This request will be considered as "passed" because of the responseCallback override.
  http.get('https://httpbin.test.k6.io/status/400', {
    responseCallback: http.expectedStatuses(400),
  });
}

Running this script will produce a summary like:

http_req_duration..............: avg=204.57ms min=203.31ms med=204.57ms max=205.82ms p(90)=205.57ms p(95)=205.7ms
  { expected_response:true }...: avg=203.31ms min=203.31ms med=203.31ms max=203.31ms p(90)=203.31ms p(95)=203.31ms
http_req_failed................: 50.00% ✓ 1   ✗ 1

Note the new http_req_duration sub-metric for expected responses only, and the new http_req_failed Rate metric. This new metric and metric tag have many potential use cases, and one of the most important ones is the ability to set better thresholds.

For example:

• 'http_req_failed': ['rate<0.1'], i.e. fail the test if more than 10% of requests fail.
• 'http_req_duration{expected_response:true}': ['p(95)<300', 'p(99.9)<500'] - fail the test if the the 95th percentile HTTP request duration is above 300ms or the 99.9th percentile is above 500ms; specifying expected_response:true here may be important, because a lot of times failed requests have a significantly different response time to successful ones, thus skewing the results.

If the response callback is not specified, the default expected statuses will be {min: 200, max: 399}. The previous behavior of not emitting anything can be achieved by setting the callback to null, i.e. http.setResponseCallback(null).

Additionally, the expected_response tag can be disabled by removing it from the default list of system tags, e.g.

$ k6 run --system-tags 'proto,subproto,status,method,url,name,group,check,error,error_code,tls_version,scenario,service'

The http.setResponseCallback() is planned to allow arbitrary JS functions to process responses in the future, but for now only the http.expectedStatuses() callback is supported.

Other enhancements and UX improvements

JavaScript Runtime

• Because of some awesome improvements to goja, the JS runtime k6 uses, it's no longer necessary for k6 to load core.js to polyfill missing JS features when using the default --compatibility-mode=extended. So in v0.31.0 core.js has been dropped entirely, yielding some significant CPU and memory usage improvements. The actual numbers will depend on the use case, but for simple tests users can expect a memory drop of about 2MB per VU (from ~2.7MB to ~600KB), and a slight CPU decrease of about 5-10%. For more complex tests with a lot of JS code this benefit won't be as pronounced. Another benefit of this change is that initializing VUs and starting a test is substantially faster than before! (#1824)

• These improvements in Goja also allowed us to disable some previously used Babel plugins, contributing to the performance increase.

• Expanded ArrayBuffer support in most internal modules, so now you can pass ArrayBuffer to http.file(), in k6/encoding and k6/crypto functions. This makes working with binary files more efficient as it doesn't require string translations. In upcoming versions we plan to expand this to the WebSocket module, as well as make some potentially breaking changes for APIs that currently return an array of integers or string.

⚠️ Breaking changes

While we don't expect the core.js removal and Babel changes to impact the vast majority of users, those were substantial changes in how k6 interprets JS and a minority of users might experience issues with their tests.

Please report any unexpected JavaScript errors by creating a GitHub issue. In particular Promise is now undefined, and some unused Babel plugins like transform-es2015-for-of and transform-regenerator were also removed.

This means that some workarounds like the ones mentioned here and here also won't work as is and will need additional polyfills and plugins to work properly.

• The Docker image base was updated to Alpine 3.13. Thanks @andriisoldatenko for your contribution!

• The Debian package now includes ca-certificates as a dependency. Thanks @Bablzz for your contribution!

Planned future breaking changes

The following are not breaking changes in this release, but we'd like to announce them so users can prepare for them in upcoming releases (likely k6 v0.32.0).

• JS: The ArrayBuffer changes in this release are backwards compatible and shouldn't cause any issues, but in v0.32.0 some JS APIs that currently return an array of integers or string for binary data will return ArrayBuffer instead. This is the case for open() when used with the 'b' argument, response bodies for requests that specify responseType: 'binary', crypto.randomBytes(), hasher.digest('binary'), and encoding.b64decode(). Response.json() and Response.html() will also probably stop working when used with requests that specify responseType: 'binary'. These changes shouldn't be a problem for most users that were simply using these values to pass them to other internal modules (e.g. opening a binary file and passing it to http.post()), but if the scripts modified the binary data or depended on the current array of integers or string values they will need to be adapted to use typed arrays instead. You can follow the discussion in PR #1841 and issue #1020.
• As part of the rebranding of Load Impact to k6, the k6 GitHub repository will be moved from loadimpact/k6 to k6io/k6 . Additionally because of Go's usage of URLs in package imports, the URL will be changed throughout the codebase. Since GitHub will maintain a redirect from the old location we don't expect this to impact a lot of users, or even external k6 contributors and xk6 developers, but be aware that the new URL should be used moving forward.
• Because of the sunsetting of Bintray, the DEB, RPM, MSI and Chocolatey package repositories currently hosted on Bintray will be moved to a self-hosted solution sometime in the upcoming weeks. We'll communicate these changes via our blog as well as the official documentation.