At the end of 2018, Microsoft stated that Visual Studio 2019 is the last version with load testing features. A few months later they formally announced that Azure load testing is closing down on March 31st, 2020.
This announcement came as a bomb to the community. But for those who follow this market, the news was not a complete surprise.
We believe open source tools and other vendors are providing better performance testing alternatives, and that was one of the main reasons for Microsoft to shut down Azure load testing and deprecate Visual Studio load test.
When I look around at other offerings in this space (open source as well as commercial offerings that sometimes include consulting services) I honestly feel they are now better suited to meet the needs of our customers.
Jamie Cool, Director of Program Management, Azure DevOps
Following the announcement, a large number of users, who were doing performance testing with Visual Studio or Azure DevOps, have been asking questions about our load testing platform.
This post intends to present how to use the k6 platform as an alternative to Azure load testing and Visual Studio web test.
k6 OSS and k6 Cloud
Before digging into how k6 compares, let’s start with an introduction of the main components of our platform: k6 Open Source and the k6 Cloud.
k6 OSS is an open source load testing tool that has proven to be a very capable, modern, alternative to JMeter. Key benefits include:
- Easy-to-use CLI and robust set of built-in APIs.
- Automation friendly, to easily facilitate the inclusion of your performance tests into CI/CD pipelines.
- Effectively enables developers to also take part in performance testing efforts, due to it's code-first approach.
You can install k6 and start running performance tests in your machine for free. By default, k6 prints the result of your test to the console.
Additionally, multiple alternatives (InfluxDB, Kafka and DataDog to name a few) are available for storing your test results, allowing for even better visualization and analysis of your performance tests.
We also provide k6 Cloud load testing service as a complement to the open-source k6 tool. If we may say so ourselves, the cloud offering does a great job at further improving your experience. Benefits include:
- Run and scale your tests in the cloud from multiple geographic locations.
- Avoid managing your own load testing infrastructure.
- Provide a User-Friendly Interface to organize, configure, and run your tests.
- Manage your projects, team members and permissions.
- Provide a convenient and powerful interface for the visualization of your test result.
- Leverage intelligent algorithms to identify performance issues.
- Analyze and compare the performance of your application over time.
After the k6 intro, we are going to describe how to run the same type of Azure load tests using the k6 Cloud.
Azure load testing
Azure DevOps Load Testing Service has the following options to create and run your load tests:
URL load test
A URL test is one of the simplest ways to create load tests; you add your URLs and configure the HTTP method, headers, and query string values for each URL. The URL Load Test will then access each of these URLs using the specified parameters multiple times depending on your load test settings.
The k6 Cloud provides the same mechanism for configuring your load tests with the Test Builder Interface. These tests are configured through the web app, utilizing an interface that is very familiar to Azure’s offering.
You can use the Test Builder GUI to:
- Configure ramping, VUs, and duration of the test.
- Create HTTP requests.
- Capture variables to correlate to the next request.
- Create Checks.
- More options available on the Test Builder documentation.
The main goal of the test builder is to help you generate a k6 script. By clicking the View script icon at the top right corner, you will find the k6 script related to your test configuration.
HTTP Archive load test
The HTTP Archive, also known as HAR, is a format for logging a user interaction of a website.
Generating a load test from a recorded user session is a common practice for testing websites that involves complex user interactions.
If you intend to load test from a recorded user session, we recommend reading our guide for load testing websites.
Recording and downloading a HAR file is a straightforward process. Here, is a guide to generate a HAR file from a browser session.
Note that in Chrome, the Preserve log option set up the network request log to not be cleared upon a page change. After we download the HAR file, we are ready to start creating our load test.
Azure import HAR file
In Azure, you have to select New HTTP Archive based test and the web interface will convert the HAR file content into individual HTTP requests on the HTTP URL interface.
The k6 ecosystem provides a few options to generate a load test from a HAR file or browser session.
- Import a HAR file to the k6 Cloud.
- Use the k6 Browser Recorder.
- Convert the HAR file using the har-to-k6 converter.
Import a HAR file to the k6 Cloud
The k6 Cloud also provides the ability to create a load test from a HAR file. The following image shows the view from you could upload the HAR file, configure other test settings, and run your load test.
k6 Browser Recorder
The k6 Browser Recorder will capture every single HTTP(s) request performed by the browser as you click – including ads, images, documents. Just press “record”, start browsing and when complete, the script will automatically upload to your k6 Cloud account.
For users preferring CLI tools or those without a k6 Cloud account, we also provide an alternative with the open-source har-to-k6 tool to convert HAR files to k6 scripts.
Azure allows you to run JMeter tests on its Load Testing Cloud Service. This is done by selecting the New Apache JMeter test, uploading your jmx file, and running your test from the web interface.
JMeter is the most popular open source load testing tool on the market. The first release was launched in 1998 and it has been the standard tool for performance testing.
We thank the JMeter community for providing a great load testing tool and paving the way for performance testing in the software industry, but we built k6 because we believe an easy-to-use alternative with a best in class developer experience is necessary in today’s world.
For users having implemented their performance tests in JMeter, we have built a tool to convert JMeter tests into k6 scripts.
Now, you could use k6 to run a local or cloud test of the converted k6 script.
Visual Studio load test
Visual Studio web test (.webtest files) is a Microsoft technology to perform web tests simulating many users accessing a server at the same time. While it was not designed for API testing, users have also used them for running API tests.
A web test project provides many options to configure your load test. Comparing both technologies deserves its dedicated blog post having in-depth explanations.
The following section quickly describes how some webtest options can be configured in k6:
- Add artificial human interaction pauses in your scenario
- Specify the number of virtual users for your scenario
- Configure test iteration settings for your scenario
- Configure the probability of a virtual user running a test in the scenario
- Configure the desired network mix for your scenario
- Select the appropriate Web browser mix for your scenario
- Specify a threshold rule using counters for your load test
- Use context parameters
Add artificial human interaction pauses in your scenario
You can use the sleep function to simulate think-time by suspending the test execution for a period of time.
Specify the number of virtual users for your scenario
You can specify the number of Virtual Users and duration of your load test in your k6 script, via the CLI or using the web interface.
Configure test iteration settings for your scenario
You can configure the number of iterations a VU can make on a similar way than duration and VUs.
You can read more about all the k6 options.
Configure the probability of a virtual user running a test in the scenario
There is Work In Progress for providing the ability to configure probability for different scenarios. The current workaround is to use conditional statements in your k6 script.
Configure the desired network mix for your scenario
k6 does not yet provide the ability to simulate network conditions. Some users have been using chaos engineering tools while running their Load Tests to replicate the same type of network conditions.
Select the appropriate Web browser mix for your scenario
Set the User Agent header on the HTTP requests of your k6 script to simulate a particular browser generating the requests.
Additionaly, the batch option configures the maximum number of simultaneous/parallel connections in total that an http.batch() call in a VU can make.
Specify a threshold rule using counters for your load test
In k6, you can use Thresholds to specify performance expectations in the form of Pass/Fail criteria.
Thresholds analyze the performance metrics and determine the final test result. For example:
- 99th percentile response time must be below 300 ms.
- Minimum response time must be below 100 ms
- The response content must have been OK more than 95% of the time.
Additionally, you can use Checks to define assertions that will be reported but will never fail the test.
You can use Environment variables to pass parameters to your load tests.
For further questions on this or any other topic, we encourage you to reach out to the k6 Cloud support or the k6 Community forum
Reviewing test result visualizations
The web interface of the Azure DevOps load testing service and Visual Studio load test allows you to analyze the result of your load. You can visualize the test summary, configuration, errors and more, analyze individual load metrics, graphs and trends, create reports, compare tests, and much more.
In the beginning of this post, we mentioned that k6 OSS offers multiple choices for visualizing your test results.
With the --out option of the k6 CLI, you can choose to output your load test results to different sources. For example:
k6 Cloud test result view
There are multiple ways to visualize your test results in the k6 Cloud application:
- Running a test from the k6 Cloud application.
- Running a cloud test from the k6 CLI (k6 cloud test.js).
- Running your test locally and uploading your results to the k6 Cloud (k6 run --out test.js).
In these cases, the k6 Cloud application will give you a convenient interface for the visualization of your load test results.
There is a lot to be said of the k6 Cloud test result view. If you are interested, we recommend you to try it out or read more on the documentation.
Running k6 tests in Azure Pipelines
One of the main reasons to run load tests as part of your CI pipeline is to ensure that you'll catch any performance regressions that occur when your application or infrastructure changes.
If you want to learn more about this topic, we recommend reading our guide about automated performance testing.
Azure Pipelines can execute k6 local tests and k6 cloud tests from the k6 CLI.
You have to install k6 in the Azure Pipeline Virtual Machine and run the k6 command to execute your load test.
For example, here is a minimal azure-pipelines YAML file that configures a job to execute a k6 test using the k6 Docker image.
If you want to read more detailed instructions, we wrote a tutorial to Integrate Load Testing with k6 and Azure Pipelines.
We have presented k6 and the k6 Cloud service as an alternative to the Azure load testing service. This post intends to provide a high-level overview of both platforms to show you how you can run the same type of load tests using k6.
If you are evaluating load testing alternatives, the best way is to try the tools and services running your performance tests.
k6 is OSS and free. The k6 Cloud offers a trial to run 50 cloud tests for free, which we encourage you to use by signing up.