k6 as alternative to Azure load testing and Visual Studio load test

At the end of 2018, Microsoft stated that Visual Studio 2019 is the last version with load testing features. A few months later they formally announced that Azure load testing is closing down on March 31st, 2020.

This announcement came as a bomb to the community. But for those who follow this market, the news was not a complete surprise.

We believe open source tools and other vendors are providing better performance testing alternatives, and that was one of the main reasons for Microsoft to shut down Azure load testing and deprecate Visual Studio load test.

When I look around at other offerings in this space (open source as well as commercial offerings that sometimes include consulting services) I honestly feel they are now better suited to meet the needs of our customers.

Jamie Cool, Director of Program Management, Azure DevOps

Following the announcement, a large number of users, who were doing performance testing with Visual Studio or Azure DevOps, have been asking questions about our load testing platform.

This post intends to present how to use the k6 platform as an alternative to Azure load testing and Visual Studio web test.

k6 OSS and Grafana Cloud k6

Before digging into how k6 compares, let’s start with an introduction of the main components of our platform: k6 Open Source and the Grafana Cloud k6.

k6 OSS is an open source load testing tool that has proven to be a very capable, modern, alternative to JMeter. Key benefits include:

• Test scripts are written in Javascript.
• Easy-to-use CLI and robust set of built-in APIs.
• Automation friendly, to easily facilitate the inclusion of your performance tests into CI/CD pipelines.
• Effectively enables developers to also take part in performance testing efforts, due to it's code-first approach.

You can install k6 and start running performance tests in your machine for free. By default, k6 prints the result of your test to the console.

Additionally, multiple alternatives (InfluxDB, Kafka and DataDog to name a few) are available for storing your test results, allowing for even better visualization and analysis of your performance tests.

We also provide Grafana Cloud k6, our managed performance testing platform that complements the open-source k6 tool. If we may say so ourselves, the cloud offering does a great job at further improving your experience. Benefits include:

• Run and scale your tests in the cloud from multiple geographic locations.
• Avoid managing your own load testing infrastructure.
• Provide a User-Friendly Interface to organize, configure, and run your tests.
• Manage your projects, team members and permissions.
• Provide a convenient and powerful interface for the visualization of your test result.
• Leverage intelligent algorithms to identify performance issues.
• Analyze and compare the performance of your application over time.

After the k6 intro, we are going to describe how to run the same type of Azure load tests using the Grafana Cloud k6.

Azure load testing

Azure DevOps Load Testing Service has the following options to create and run your load tests:

• URL test
• HTTP Archive test
• JMeter test
• Visual Studio load test

URL load test

A URL test is one of the simplest ways to create load tests; you add your URLs and configure the HTTP method, headers, and query string values for each URL. The URL Load Test will then access each of these URLs using the specified parameters multiple times depending on your load test settings.

Grafana Cloud k6 provides the same mechanism for configuring your load tests with the Test Builder Interface. These tests are configured through the web app, utilizing an interface that is very familiar to Azure’s offering.

The Test Builder is an excellent tool to run a load test without writing any Javascript code.

You can use the Test Builder GUI to:

• Configure ramping, VUs, and duration of the test.
• Create HTTP requests.
• Capture variables to correlate to the next request.
• Create Checks.
• More options available on the Test Builder documentation.

The main goal of the test builder is to help you generate a k6 script. By clicking the View script icon at the top right corner, you will find the k6 script related to your test configuration.

HTTP Archive load test

The HTTP Archive, also known as HAR, is a format for logging a user interaction of a website.

Generating a load test from a recorded user session is a common practice for testing websites that involves complex user interactions.

If you intend to load test from a recorded user session, we recommend reading our guide for load testing websites.

Recording and downloading a HAR file is a straightforward process. Here, is a guide to generate a HAR file from a browser session.

Note that in Chrome, the Preserve log option set up the network request log to not be cleared upon a page change. After we download the HAR file, we are ready to start creating our load test.

Azure import HAR file

In Azure, you have to select New HTTP Archive based test and the web interface will convert the HAR file content into individual HTTP requests on the HTTP URL interface.

The k6 ecosystem provides a few options to generate a load test from a HAR file or browser session.

• Import a HAR file to the cloud.
• Use the k6 Browser Recorder.
• Convert the HAR file using the har-to-k6 converter.

Import a HAR file to the cloud

The cloud also provides the ability to create a load test from a HAR file. The following image shows the view from you could upload the HAR file, configure other test settings, and run your load test.

k6 Browser Recorder

The k6 Browser Recorder will capture every single HTTP(s) request performed by the browser as you click – including ads, images, documents. Just press “record”, start browsing and when complete, the script will automatically upload to your cloud account.

har-to-k6 converter

For users preferring CLI tools or those without a cloud account, we also provide an alternative with the open-source har-to-k6 tool to convert HAR files to k6 scripts.

# install the converter
npm install -g har-to-k6

# convert HAR file
har-to-k6 my-user-session.har -o loadtest.js

# local execution
k6 run loadtest.js

# cloud execution
k6 cloud loadtest.js

JMeter test

Azure allows you to run JMeter tests on its Load Testing Cloud Service. This is done by selecting the New Apache JMeter test, uploading your jmx file, and running your test from the web interface.

JMeter is the most popular open source load testing tool on the market. The first release was launched in 1998 and it has been the standard tool for performance testing.

We thank the JMeter community for providing a great load testing tool and paving the way for performance testing in the software industry, but we built k6 because we believe an easy-to-use alternative with a best in class developer experience is necessary in today’s world.

For users having implemented their performance tests in JMeter, we have built a tool to convert JMeter tests into k6 scripts.

# install the converter
npm install -g jmeter-to-k6

# convert JMeter load test
jmeter-to-k6 loadtest.jmx -o loadtest

Now, you could use k6 to run a local or cloud test of the converted k6 script.

# local execution
k6 run ~/loadtest/test.js


# cloud execution
k6 cloud ~/loadtest/test.js

Visual Studio load test

Visual Studio web test (.webtest files) is a Microsoft technology to perform web tests simulating many users accessing a server at the same time. While it was not designed for API testing, users have also used them for running API tests.

A web test project provides many options to configure your load test. Comparing both technologies deserves its dedicated blog post having in-depth explanations.

The following section quickly describes how some webtest options can be configured in k6:

• Add artificial human interaction pauses in your scenario
• Specify the number of virtual users for your scenario
• Configure test iteration settings for your scenario
• Configure the probability of a virtual user running a test in the scenario
• Configure the desired network mix for your scenario
• Select the appropriate Web browser mix for your scenario
• Specify a threshold rule using counters for your load test
• Use context parameters

Add artificial human interaction pauses in your scenario

You can use the sleep function to simulate think-time by suspending the test execution for a period of time.

import { sleep } from 'k6';
import http from 'k6/http';

export default function () {
  http.get('https://k6.io');
  sleep(Math.random() * 30);
  http.get('https://k6.io/open-source');
}

Specify the number of virtual users for your scenario

You can specify the number of Virtual Users and duration of your load test in your k6 script, via the CLI or using the web interface.

k6 run --vus 10 --duration 30s script.js

export const options = {
  vus: 10,
  duration: '3m',
};

Configure test iteration settings for your scenario

You can configure the number of iterations a VU can make on a similar way than duration and VUs.

export const options = {
  vus: 10,
  iterations: 100,
};

You can read more about all the k6 options.

Configure the probability of a virtual user running a test in the scenario

There is Work In Progress for providing the ability to configure probability for different scenarios. The current workaround is to use conditional statements in your k6 script.

Configure the desired network mix for your scenario

k6 does not yet provide the ability to simulate network conditions. Some users have been using chaos engineering tools while running their Load Tests to replicate the same type of network conditions.

Select the appropriate Web browser mix for your scenario

Set the User Agent header on the HTTP requests of your k6 script to simulate a particular browser generating the requests.

const headers = {
  'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0',
};

http.get('https//test.k6.io', { headers: headers });

Additionaly, the batch option configures the maximum number of simultaneous/parallel connections in total that an http.batch() call in a VU can make.

export const options = {
  batch: 15,
};

Specify a threshold rule using counters for your load test

In k6, you can use Thresholds to specify performance expectations in the form of Pass/Fail criteria.

Thresholds analyze the performance metrics and determine the final test result. For example:

• 99th percentile response time must be below 300 ms.
• Minimum response time must be below 100 ms
• The response content must have been OK more than 95% of the time.
• ...

export const options = {
  thresholds: {
    'failed requests': ['rate<0.1'], // threshold on a custom metric
    'http_req_duration': ['p(95)<500'], // threshold on a standard metric
  },
};

Additionally, you can use Checks to define assertions that will be reported but will never fail the test.

check(res, {
  'is status 200': (r) => r.status === 200,
});

Use context parameters

You can use Environment variables to pass parameters to your load tests.

k6 run -e MY_HOSTNAME=test.k6.io script.js

For further questions on this or any other topic, we encourage you to reach out to support or the k6 Community forum

Reviewing test result visualizations

The web interface of the Azure DevOps load testing service and Visual Studio load test allows you to analyze the result of your load. You can visualize the test summary, configuration, errors and more, analyze individual load metrics, graphs and trends, create reports, compare tests, and much more.

In the beginning of this post, we mentioned that k6 OSS offers multiple choices for visualizing your test results.

With the --out option of the k6 CLI, you can choose to output your load test results to different sources. For example:

# json
k6 run --out json=my_test_result.json script.js

# InfluxDB/Grafana
k6 run --out influxdb=http://localhost:8086/k6 script.js

# Apache Kafka
k6 run --out kafka=brokers=broker_host:8000,topic=k6

# statsd
k6 run --out statsd script.js

# datadog
k6 run --out datadog script.js

Grafana Cloud k6 test result view

There are multiple ways to visualize your test results in Grafana Cloud k6:

• Running a test from Grafana Cloud.
• Running a cloud test from the k6 CLI (k6 cloud test.js).
• Running your test locally and stream the results to the cloud (k6 run --out test.js).

In all these cases, Grafana Cloud k6 provides pre-built dashboards for the visualization of your load test results.

There is a lot to be said of the Grafana Cloud k6 dashboards. If you are interested, we recommend you to try it out or read more on the documentation.

Running k6 tests in Azure Pipelines

One of the main reasons to run load tests as part of your CI pipeline is to ensure that you'll catch any performance regressions that occur when your application or infrastructure changes.

If you want to learn more about this topic, we recommend reading our guide about automated performance testing.

Azure Pipelines can execute k6 local tests and cloud tests from the k6 CLI.

k6 run my-test.js

k6 cloud my-test.js

You have to install k6 in the Azure Pipeline Virtual Machine and run the k6 command to execute your load test.

For example, here is a minimal azure-pipelines YAML file that configures a job to execute a k6 test using the k6 Docker image.

# azure-pipelines.yml

pool:
  vmImage: 'ubuntu-18.04'

steps:
- script: |
    docker run -v `pwd`:/src -i grafana/k6 run /src/loadtest.js
  displayName: Run k6

If you want to read more detailed instructions, we wrote a tutorial to Integrate Load Testing with k6 and Azure Pipelines.

Conclusion

We have presented k6 and the Grafana Cloud k6 as an alternative to the Azure load testing service. This post intends to provide a high-level overview of both platforms to show you how you can run the same type of load tests using k6.

If you are evaluating load testing alternatives, the best way is to try the tools and services running your performance tests.

k6 is OSS and free. The easiest way to get started with performance testing is with Grafana Cloud. There are plans for every use case, including a free forever plan that comes with up to 500 Virtual User Hours each month. Sign up for a free account today.