- Prerequisites
- Write Your Performance Test Script
- Configure Thresholds
- CircleCI Pipeline Configuration
- Setting up the repository
- Add project and run the test
- Running k6 Cloud Tests
- Nightly builds
- Storing test results as artifacts
- Using the JSON output for time-series data
- Using handleSummary callback for test summary
- Running k6 extensions
- Conclusion
In this tutorial, we will look into how to integrate k6 tests into CircleCI. By integrating performance tests into your CI pipelines, you can catch performance issues earlier and ship reliable applications to production.
Prerequisites
- k6, an open-source load testing tool for testing the performance and reliability of websites, APIs, microservices, and system infrastructure.
- CircleCI is a Continuous Integration and Delivery tool to automate your development process.
The examples in this tutorial can be found here for best experience you can clone the repository and execute the tests.
Write Your Performance Test Script
For the sake of this tutorial, we will create a simple k6 test for our demo API. Feel free to change this to any of the API endpoints you are looking to test.
The following test will run 50 VUs (virtual users) continuously for one minute. Throughout this duration, each VU will generate one request, sleep for 3 seconds, and then start over.
You can run the test locally using the following command. Just make sure to install k6 first.
This produces the following output:
Configure Thresholds
The next step is to add your service-level objectives (SLOs) for the performance of your application. SLOs are a vital aspect of ensuring the reliability of your systems and applications. If you do not currently have any defined SLAs or SLOs, now is a good time to start considering your requirements.
You can then configure your SLOs as pass/fail criteria in your test script using thresholds. k6 evaluates these thresholds during the test execution and informs you about its results.
If a threshold in your test fails, k6 will finish with a non-zero exit code, which communicates to the CI tool that the step failed.
Now, we add two thresholds to our previous script to validate that the 95th percentile response time is below 500ms and also that our error rate is less than 1%. After this change, the script will be as in the snippet below:
Thresholds are a powerful feature providing a flexible API to define various types of pass/fail criteria in the same test run. For example:
- The 99th percentile response time must be below 700 ms.
- The 95th percentile response time must be below 400 ms.
- No more than 1% failed requests.
- The content of a response must be correct more than 95% of the time.
- Your condition for pass/fail criteria (SLOs)
CircleCI Pipeline Configuration
There are two steps to configure a CircleCI pipeline to fetch changes from your repository:
- Create a repository that contains a .circleci/config.yml configuration with your k6 load test script and related files.
- Add the project (repository) to CircleCI so that it fetches the latest changes from your repository and executes your tests.
Setting up the repository
In the root of your project, create a folder named .circleci and inside that folder, create a configuration file named config.yml. This file will trigger the CI to build whenever a push to the remote repository is detected. If you want to know more about it, please visit these links:
Proceed by adding the following YAML code into your config.yml file. This configuration file does the following:
- Uses grafana/k6 orb which encapsulates all test execution logic.
- Runs k6 with your performance test script passed to the script in the job.
Note: A CircleCI orb is a reusable package of YAML configuration that condenses repeated pieces of config into a single line of code. In our case above the orb is grafana/k6@1.1.3 and this orb will specify all the test steps that our configuration file above will follow to execute our loadtest. Use the latest Orb version.
Add project and run the test
To enable CircleCI build for your repository, use the "Add Project" button. While you have everything setup, head over to your dashboard to watch the build that was just triggered by pushing the changes to GitHub. You might see the build at the top of the pipeline page:
On this page, you will have your build running. Just click on the specific build step, i,e. the highlighted grafana/k6, to watch the test run and see the results when it has finished. Below is the build pages for a successful build:
Running k6 Cloud Tests
There are two common ways to run k6 tests as part of a CI process:
- k6 run to run a test locally on the CI server.
- k6 cloud to run a test on the k6 Cloud from one or multiple geographic locations.
You might want to trigger cloud tests in these common cases:
- If you want to run a test from one or multiple geographic locations (load zones).
- If you want to run a test with high-load that will need more compute resources than provisioned by the CI server.
If any of those reasons fits your needs, then running k6 cloud tests is the way to go for you.
Before we start with the CircleCI configuration, it is good to familiarize ourselves with how cloud execution works, and we recommend you to test how to trigger a cloud test from your machine. Check out the guide to running cloud tests from the CLI to learn how to distribute the test load across multiple geographic locations and more information about the cloud execution.
We will now show how to trigger cloud tests from CircleCI. If you do not have an account with k6 Cloud already, register for a trial account. After that, go to the API token page on Account Settings in k6 Cloud and copy your API token.
Next, navigate to the project's settings in CircleCI and select the Environment Variables page. Add a new environment variable:
- K6_CLOUD_TOKEN: its value is the API token we got from k6 Cloud.
If you need to run the cloud test in a particular project, you need your project ID to be set as an environment variable or on the k6 script. The project ID is visible under the project name in k6 Cloud project page.
Then, the .circleci/config.yml file should look like the following:
Again, due to the power of orbs, since we already have an existing written configuration, all we need to do is to pass in the configuration cloud: true and CircleCI will be able to use our K6_CLOUD_TOKEN, that we configured in our environment variables and use it to send our k6 run output to our k6 Cloud dashboard.
Note: It is mandatory to set the K6_CLOUD_TOKEN for the cloud configuration on Circleci to work with the k6 orb, this is because once we have a cloud:true option in our configuration, CircleCI expects to also have a valid token that can be used to run our tests on the k6 cloud.
With that done, we can now go ahead and push the changes we've made in .circleci/config.yml to our GitHub repository. This subsequently triggers CircleCI to build our new pipeline using the new config file. Just keep in mind that for keeping things tidy, we've created a directory named cloud-example which contains the CircleCI config file to run our cloud tests in the cloned repository.
It is essential to know that CircleCI prints the output of the k6 command, and when running cloud tests, k6 prints the URL of the test result in the k6 Cloud, which is highlighted in the above screenshot. You could navigate to this URL to see the result of your cloud test.
We recommend that you define your performance thresholds in the k6 tests in a previous step. If you have configured your thresholds properly and your test passes, there should be nothing to worry about. But when the test fails, you want to understand why. In this case, navigate to the URL of the cloud test to analyze the test result. The result of the cloud service will help you quickly find the cause of the failure.
Nightly builds
It's common to run some performance tests during the night when users do not access the system under test. For example, to isolate larger tests from other types of tests or to generate a performance report periodically. Below is the first example configuring a scheduled nightly build that runs at midnight (UTC) everyday.
This example is using the crontab syntax to schedule execution. To learn more, we recommend reading the scheduling a CircleCI workflow guide.
Storing test results as artifacts
Using the JSON output for time-series data
We can store the k6 results as artifacts in CircleCI so that we can inspect results later.
This excerpt of the CircleCI config file shows how to do this with the JSON output and the k6 Docker image.
In our snippet above, we have defined the output to the CircleCI artifact with > /tmp/artifacts which CircleCI will interpret it as the volume in the orb for storing artifacts.
The full.json file will provide all the metric points collected by k6. The file size can be huge. If you don't need to process raw data, you can store the k6 end-of-test summary as a CircleCI artifact.
Using handleSummary callback for test summary
k6 can also report the general overview of the test results (end of the test summary) in a custom file. To achieve this, we will need to export a handleSummary function as in the code snippet below of our performance-test.js script file:
In the handleSummary callback, we have specified the summary.json file to save the results. To store the summary result as an CircleCI artifact, we use the same running command as our testing command but define an output path where CircleCI will store our output file as shown below:
Looking at the execution, we can observe that our console statement appeared as an INFO message. We can also verify that a summary.json file was created after we finished executing our test as shown below. For additional information, you can read about the handleSummary callback function here.
On observation, we can verify that the summary.json is an overview of all the data that k6 uses to curate the end of the test summary report including the metrics gathered, test execution state and also test configuration.
Running k6 extensions
k6 extensions enable users to be able to extend the usage of k6 with Go based k6 extensions that can then be imported as js modules. With extensions, not only are we able to build functionality on top of k6 but also a door for limitless opportunities of extending k6 to custom needs.
For this section, we will set up a docker environment to run our k6 extension. In our case we will utilize a simple extension xk6-counter which we set up on docker with the following snippet.
We will also have a file counter.js on the root directory that will demonstrate how we can extend k6 using the extensions.
To execute our tests with the extension, we will create a docker-compose file, as the above will build our image and not execute our tests. For the purposes of the tutorial, we will build our image as xk6-counter:latest. Once this is done, we can use the image in our docker-compose file to run the counter.js file located under the loadtests folder in the root directory as below:
The snippet above can also be found in the cloned repository in the docker-compose.yml file and executed with the following command:
This command should then execute the counter.js file which makes use of the xk6 counter extension. The dockerfile above and the docker-compose file can be utilized to execute other extensions and should only serve as a template.
To execute tests with extensions on CircleCI, we will push our docker image to docker Hub then use that image when executing our tests in CircleCI. We can then use the image to execute the tests as shown below in the snippet:
Conclusion
In this tutorial we have learned how to execute a basic k6 test, how to integrate k6 with CircleCI and execute tests on the CI platform, how to execute cloud tests and also executing k6 extensions with xk6. We also learned how we can use other workflows such as nightly builds and k6 outputs using the summary flag or the new k6 handleSummary callback option. With all that covered, we hope you enjoyed reading this article as we did creating it. We'd be happy to hear your feedback.
- Prerequisites
- Write Your Performance Test Script
- Configure Thresholds
- CircleCI Pipeline Configuration
- Setting up the repository
- Add project and run the test
- Running k6 Cloud Tests
- Nightly builds
- Storing test results as artifacts
- Using the JSON output for time-series data
- Using handleSummary callback for test summary
- Running k6 extensions
- Conclusion