In this tutorial, we will look into how to integrate k6 tests into CircleCI. By integrating performance tests into your CI pipelines, you can catch performance issues earlier and ship more stable and performant applications to production.
- k6, an open-source load testing tool for testing the performance of APIs, microservices, and websites.
- CircleCI is a Continuous Integration and Delivery tool to automate your development process.
- Write Your Performance Test Script
- Configure Thresholds
- CircleCI Pipeline Configuration
- Running k6 Cloud Tests
- Load Testing Behind The Firewall
- Other CI/CD Workflows
The examples in this tutorial can be found here.
Write Your Performance Test Script
For the sake of this tutorial, we will create a simple k6 test for our demo API. Feel free to change this to any of the API endpoints you are looking to test.
The following test will run 50 VUs (virtual users) continuously for one minute. Throughout this duration, each VU will generate one request, sleep for 3 seconds, and then start over.
You can run the test locally using the following command. Just make sure to install k6 first.
This produces the following output:
The next step is to add your service-level objectives (SLOs) for the performance of your application. SLOs are a vital aspect of ensuring the reliability of your systems and applications. If you do not currently have any defined SLAs or SLOs, now is a good time to start considering your requirements.
You can then configure your SLOs as pass/fail criteria in your test script using thresholds. k6 evaluates these thresholds during the test execution and informs you about its results.
If a threshold in your test fails, k6 will finish with a non-zero exit code, which communicates to the CI tool that the step failed.
Now, we add one threshold to our previous script to validate that the 95th percentile response time is below 500ms. After this change, the script will look like this:
Thresholds are a powerful feature providing a flexible API to define various types of pass/fail criteria in the same test run. For example:
- The 99th percentile response time must be below 700 ms.
- The 95th percentile response time must be below 400 ms.
- No more than 1% failed requests.
- The content of a response must be correct more than 95% of the time.
- Your condition for pass/fail criteria (SLOs)
CircleCI Pipeline Configuration
There are two steps to configure a CircleCI pipeline to fetch changes from your repository:
- Create a repository that contains a .circleci/config.yml, your k6 load test script and related files.
- Add the project (repository) to CircleCI so that it fetches the latest changes from your repository.
Setting up the repository
In the root of your project, create a folder named .circleci and inside that folder, create a configuration file named config.yml. This file will trigger the CI to build whenever a push to the remote repository is detected. If you want to know more about it, please visit these links:
Proceed by adding the following YAML code into your config.yml file. This configuration file does the following:
- Downloads your project from your remote repository.
- Pulls the latest stable version of k6 from Docker Hub.
- Runs k6 with your performance test script.
Add project to CircleCI
To enable CircleCI build for your repository, use the "Add Project" button. While you have everything setup, head over to your dashboard to watch the build that was just triggered by pushing the changes to GitHub. You might see the build at the top of the pipeline page:
On this page, you will have your build running. Just click on the specific build step, i,e. the highlighted run_performance_tests, to watch the test run and see the results when it has finished. Below is the build pages for a successful build:
Running k6 Cloud Tests
There are two common ways to run k6 tests as part of a CI process:
- k6 run to run a test locally on the CI server.
- k6 cloud to run a test on the k6 Cloud from one or multiple geographic locations.
You might want to trigger cloud tests in these common cases:
- If you want to run a test from one or multiple geographic locations (load zones).
- If you want to run a test with high-load that will need more compute resources than provisioned by the CI server.
If any of those reasons fits your needs, then running k6 cloud tests is the way to go for you.
Before we start with the CircleCI configuration, it is good to familiarize ourselves with how cloud execution works, and we recommend you to test how to trigger a cloud test from your machine.
Check out the guide to running cloud tests from the CLI to learn how to distribute the test load across multiple geographic locations and more information about the cloud execution.
Now, we will show how to trigger cloud tests from CircleCI. If you do not have an account with k6 Cloud already, you should go register for a trial account here. After that, go to the API token page on Account Settings in k6 Cloud and copy your API token. You also need your project ID to be set as an environment variable. The project ID is visible under the project name in k6 Cloud project page.
Next, navigate to the project's settings in CircleCI and select the Environment Variables page. Add two new environment variable:
- K6_CLOUD_TOKEN: its value is the API token we got from k6 Cloud.
- K6_CLOUD_PROJECT_ID: its value is the project ID we got from k6 Cloud.
Then, the .circleci/config.yml file should look like the following:
The changes, compared to the previous configuration, are:
- Replace the k6 run command for the k6 cloud command to trigger cloud tests.
- Add K6_API_TOKEN environment variable to specify the k6 Cloud API token.
- Add K6_CLOUD_PROJECT_ID environment variable to specify the k6 Cloud project ID.
With that done, we can now go ahead and push the changes we've made in .circleci/config.yml to our GitHub repository. This subsequently triggers CircleCI to build our new pipeline using the new config file. Just keep in mind that for keeping things tidy, we've created a branch named k6-cloud to have a separate CircleCI config file.
It is essential to know that CircleCI prints the output of the k6 command, and when running cloud tests, k6 prints the URL of the test result in the k6 Cloud, which is highlighted in the above screenshot. You could navigate to this URL to see the result of your cloud test.
We recommend that you define your performance thresholds in the k6 tests in a previous step. If you have configured your thresholds properly and your test passes, there should be nothing to worry about. But when the test fails, you want to understand why. In this case, navigate to the URL of the cloud test to analyze the test result. The result of the cloud service will help you quickly find the cause of the failure.
Load Testing Behind The Firewall
If the system under test is behind a firewall, we have to configure the firewall to allow IPs of the k6 instances to access the system.
For cloud tests
If you are running a k6 cloud test, you will be utilizing k6 Cloud infrastructure. Check out this guide to open your firewall to the k6 Cloud service.
When the test runs in the CI server
We need to grant CircleCI access to our system by adding the necessary IP ranges to the firewall. If you're using AWS you can temporarily grant access by adding a security group rule pre-test. Make sure you have created an AWS user with the ec2* role to allow CircleCI edit rights to VPC security group. When the user is created, copy the access key, secret key ID, and security group ID to CircleCI environment variables:
The configuration of this example is different than the previous ones, CircleCI will setup a Docker executor for our CircleCI instance and will install k6 from the apt package manager instead of using the k6 Docker image.
To set this up, CircleCI will run the following script to install k6 and the aws cli to your CircleCI instance running the tests.
The following script will be executed to grant access to the public IP of the CircleCI machine:
And, we must not forget to remove the grant access after the test execution:
Below the final CircleCI configuration that glues all together:
If you want to see the code of this example, the CircleCI configuration and bash scripts are available here.
Other CI/CD Workflows
In the previous example, we showed how to setup a simple CI configuration. But teams usually run their software operations with a more sophisticated development process. You could integrate your performance tests in different ways. For example:
Run all tests you need to run with exception from performance tests. These tests will include unit tests, e2e tests, integration tests, etc.
Deploy to staging or test environment or whichever environment you have set up for performance tests. If you do not have that set up yet, I suggest you do as it would not be wise to performance test a live production environment that is being used by real users, unless you know what you are doing.
After the deployment definition, run your performance tests against your staging or test environment deployment.
When that passes, we can now consider our workflow a success. We can now go ahead and merge the branch you were running on to your production branch.
The above is another simple example. In a real environment, you start deciding which performance tests to execute in your CI/CD pipelines and how to run them (either sequentially or in parallel) and configure your workflow to adapt to the environments, branching, jobs, and testing of your software development and DevOps process.
It's common to run some performance tests during the night when users do not access the system under test. For example, to isolate larger tests from other types of tests or to generate a performance report periodically.
Below is the first example configuring a scheduled nightly build that runs at midnight (UTC) everyday.
Storing result output as artifacts
We can store the k6 result output as artifacts in CircleCI so that we can inspect them later. This excerpt of the CircleCI config file shows how to do this. If you want to know more, please visit the storing build artifacts guide.
Hope you enjoyed reading this article. We'd be happy to hear your feedback.