Load testing with CircleCI

📖What you will learn

• How to integrate load testing with k6 into CircleCI
• How to detect performance issues earlier
• Different implementation paths, and when to use each

Introduction

In this tutorial, we will look into how to integrate k6 tests into CircleCI. By integrating performance tests into your CI pipelines, you can catch performance issues earlier and ship more stable and performant applications to production.

Prerequisites

• k6, an open-source load testing tool for testing the performance of APIs, microservices, and websites.
• CircleCI is a Continuous Integration and Delivery tool to automate your development process.

The examples in this tutorial can be found here.

Write Your Performance Test Script

For the sake of this tutorial, we will create a simple k6 test for our demo API. Feel free to change this to any of the API endpoints you are looking to test.

The following test will run 50 VUs (virtual users) continuously for one minute. Throughout this duration, each VU will generate one request, sleep for 3 seconds, and then start over.

import { sleep } from 'k6';
import http from 'k6/http';

export let options = {
  duration: '1m',
  vus: 50,
};

export default function () {
  http.get('http://test.k6.io/contacts.php');
  sleep(3);
}

You can run the test locally using the following command. Just make sure to install k6 first.

$ k6 run performance-test.js

This produces the following output:

          /\      |‾‾|  /‾‾/  /‾/
     /\  /  \     |  |_/  /  / /
    /  \/    \    |      |  /  ‾‾\
   /          \   |  |‾\  \ | (_) |
  / __________ \  |__|  \__\ \___/ .io

  execution: local
     script: performance-test.js
     output: -

  scenarios: (100.00%) 1 executors, 50 max VUs, 1m30s max duration (incl. graceful stop):
           * default: 50 looping VUs for 1m0s (gracefulStop: 30s)


running (1m02.5s), 00/50 VUs, 1000 complete and 0 interrupted iterations
default ✓ [======================================] 50 VUs  1m0s


    data_received..............: 711 kB 11 kB/s
    data_sent..................: 88 kB  1.4 kB/s
    http_req_blocked...........: avg=8.97ms   min=1.37µs   med=2.77µs   max=186.58ms p(90)=9.39µs   p(95)=8.85ms
    http_req_connecting........: avg=5.44ms   min=0s       med=0s       max=115.8ms  p(90)=0s       p(95)=5.16ms
    http_req_duration..........: avg=109.39ms min=100.73ms med=108.59ms max=148.3ms  p(90)=114.59ms p(95)=119.62ms
    http_req_receiving.........: avg=55.89µs  min=16.15µs  med=37.92µs  max=9.67ms   p(90)=80.07µs  p(95)=100.34µs
    http_req_sending...........: avg=15.69µs  min=4.94µs   med=10.05µs  max=109.1µs  p(90)=30.32µs  p(95)=45.83µs
    http_req_tls_handshaking...: avg=0s       min=0s       med=0s       max=0s       p(90)=0s       p(95)=0s
    http_req_waiting...........: avg=109.31ms min=100.69ms med=108.49ms max=148.22ms p(90)=114.54ms p(95)=119.56ms
    http_reqs..................: 1000   15.987698/s
    iteration_duration.........: avg=3.11s    min=3.1s     med=3.1s     max=3.3s     p(90)=3.12s    p(95)=3.15s
    iterations.................: 1000   15.987698/s
    vus........................: 50     min=50 max=50
    vus_max....................: 50     min=50 max=50

Configure Thresholds

The next step is to add your service-level objectives (SLOs) for the performance of your application. SLOs are a vital aspect of ensuring the reliability of your systems and applications. If you do not currently have any defined SLAs or SLOs, now is a good time to start considering your requirements.

You can then configure your SLOs as pass/fail criteria in your test script using thresholds. k6 evaluates these thresholds during the test execution and informs you about its results.

If a threshold in your test fails, k6 will finish with a non-zero exit code, which communicates to the CI tool that the step failed.

Now, we add one threshold to our previous script to validate that the 95th percentile response time is below 500ms. After this change, the script will look like this:

import { sleep } from 'k6';
import http from 'k6/http';

export let options = {
  duration: '1m',
  vus: 50,
  thresholds: {
    http_req_duration: ['p(95)<500'], // 95 percent of response times must be below 500ms
  },
};

export default function () {
  http.get('http://test.k6.io/contacts.php');
  sleep(3);
}

Thresholds are a powerful feature providing a flexible API to define various types of pass/fail criteria in the same test run. For example:

• The 99th percentile response time must be below 700 ms.
• The 95th percentile response time must be below 400 ms.
• No more than 1% failed requests.
• The content of a response must be correct more than 95% of the time.
• Your condition for pass/fail criteria (SLOs)

CircleCI Pipeline Configuration

There are two steps to configure a CircleCI pipeline to fetch changes from your repository:

1. Create a repository that contains a .circleci/config.yml, your k6 load test script and related files.
2. Add the project (repository) to CircleCI so that it fetches the latest changes from your repository.

Setting up the repository

In the root of your project, create a folder named .circleci and inside that folder, create a configuration file named config.yml. This file will trigger the CI to build whenever a push to the remote repository is detected. If you want to know more about it, please visit these links:

• Sample config.yml files
• Configuring CircleCI

Proceed by adding the following YAML code into your config.yml file. This configuration file does the following:

1. Downloads your project from your remote repository.
2. Pulls the latest stable version of k6 from Docker Hub.
3. Runs k6 with your performance test script.

default: &defaults
  parallelism: 1
  working_directory: ~/my-project

k6_performance_tests: &k6_performance_tests
  run:
    name: Running k6 tests
    # Download the k6 docker image. Alternatively, download the k6 release binary
    # Mount a volume to access the folder and run the test
    command: |
      docker pull loadimpact/k6:latest
      docker run -i -v $HOME/my-project:/ci/ loadimpact/k6:latest run /ci/loadtests/performance-test.js

version: 2
jobs:
  run_performance_tests:
    <<: *defaults
    # Use `machine` executor because the Docker executor cannot mount volumes
    machine: true
    steps:
      - checkout
      - *k6_performance_tests

workflows:
  version: 2
  build-and-test:
    jobs:
      - run_performance_tests

Add project to CircleCI

To enable CircleCI build for your repository, use the "Add Project" button. While you have everything setup, head over to your dashboard to watch the build that was just triggered by pushing the changes to GitHub. You might see the build at the top of the pipeline page:

On this page, you will have your build running. Just click on the specific build step, i,e. the highlighted run_performance_tests, to watch the test run and see the results when it has finished. Below is the build pages for a successful build:

Running k6 Cloud Tests

There are two common ways to run k6 tests as part of a CI process:

• k6 run to run a test locally on the CI server.
• k6 cloud to run a test on the k6 Cloud from one or multiple geographic locations.

You might want to trigger cloud tests in these common cases:

• If you want to run a test from one or multiple geographic locations (load zones).
• If you want to run a test with high-load that will need more compute resources than provisioned by the CI server.

If any of those reasons fits your needs, then running k6 cloud tests is the way to go for you.

Before we start with the CircleCI configuration, it is good to familiarize ourselves with how cloud execution works, and we recommend you to test how to trigger a cloud test from your machine.

Check out the guide to running cloud tests from the CLI to learn how to distribute the test load across multiple geographic locations and more information about the cloud execution.

Now, we will show how to trigger cloud tests from CircleCI. If you do not have an account with k6 Cloud already, you should go register for a trial account here. After that, go to the API token page on Account Settings in k6 Cloud and copy your API token. You also need your project ID to be set as an environment variable. The project ID is visible under the project name in k6 Cloud project page.

Next, navigate to the project's settings in CircleCI and select the Environment Variables page. Add two new environment variable:

1. K6_CLOUD_TOKEN: its value is the API token we got from k6 Cloud.
2. K6_CLOUD_PROJECT_ID: its value is the project ID we got from k6 Cloud.

Then, the .circleci/config.yml file should look like the following:

default: &defaults
  parallelism: 1
  working_directory: ~/my-project

k6_performance_tests: &k6_performance_tests
  run:
    name: Running k6 tests
    command: |
      docker pull loadimpact/k6:latest
      docker run -i -e K6_CLOUD_TOKEN=$K6_CLOUD_TOKEN -e K6_CLOUD_PROJECT_ID=$K6_CLOUD_PROJECT_ID -v $HOME/my-project:/ci/ loadimpact/k6:latest cloud /ci/loadtests/performance-test.js

version: 2
jobs:
  run_performance_tests:
    <<: *defaults
    machine: true
    steps:
      - checkout
      - *k6_performance_tests

workflows:
  version: 2
  build-and-test:
    jobs:
      - run_performance_tests

The changes, compared to the previous configuration, are:

1. Replace the k6 run command for the k6 cloud command to trigger cloud tests.
2. Add K6_API_TOKEN environment variable to specify the k6 Cloud API token.
3. Add K6_CLOUD_PROJECT_ID environment variable to specify the k6 Cloud project ID.

With that done, we can now go ahead and push the changes we've made in .circleci/config.yml to our GitHub repository. This subsequently triggers CircleCI to build our new pipeline using the new config file. Just keep in mind that for keeping things tidy, we've created a branch named k6-cloud to have a separate CircleCI config file.

It is essential to know that CircleCI prints the output of the k6 command, and when running cloud tests, k6 prints the URL of the test result in the k6 Cloud, which is highlighted in the above screenshot. You could navigate to this URL to see the result of your cloud test.

We recommend that you define your performance thresholds in the k6 tests in a previous step. If you have configured your thresholds properly and your test passes, there should be nothing to worry about. But when the test fails, you want to understand why. In this case, navigate to the URL of the cloud test to analyze the test result. The result of the cloud service will help you quickly find the cause of the failure.

Load Testing Behind The Firewall

If the system under test is behind a firewall, we have to configure the firewall to allow IPs of the k6 instances to access the system.

For cloud tests

If you are running a k6 cloud test, you will be utilizing k6 Cloud infrastructure. Check out this guide to open your firewall to the k6 Cloud service.

When the test runs in the CI server

We need to grant CircleCI access to our system by adding the necessary IP ranges to the firewall. If you're using AWS you can temporarily grant access by adding a security group rule pre-test. Make sure you have created an AWS user with the ec2* role to allow CircleCI edit rights to VPC security group. When the user is created, copy the access key, secret key ID, and security group ID to CircleCI environment variables:

• $AWS_ACCESS_KEY
• $AWS_SECRET_ACCESS_KEY
• $AWS_SECURITY_GROUP_ID

The configuration of this example is different than the previous ones, CircleCI will setup a Docker executor for our CircleCI instance and will install k6 from the apt package manager instead of using the k6 Docker image.

To set this up, CircleCI will run the following script to install k6 and the aws cli to your CircleCI instance running the tests.

#!/bin/bash

set -ex
# set -o pipefail

update_cache() {
    apt-get update
}

install_k6() {
    sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C5AD17C747E3415A3642D57D77C6C491D6AC1D69
    echo "deb https://dl.k6.io/deb stable main" | sudo tee /etc/apt/sources.list.d/k6.list
    sudo apt-get update
    sudo apt-get install k6
}

install_aws_cli() {
    apt-get install curl unzip python2.7 python-pip sudo -y
    curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
    unzip awscli-bundle.zip
    ./awscli-bundle/install -b ~/bin/aws
}

configuring_aws() {
    /root/bin/aws configure set aws_access_key_id $AWS_ACCESS_KEY
    /root/bin/aws configure set aws_secret_access_key $AWS_SECRET_ACCESS_KEY
    /root/bin/aws configure set default.region us-east-1
    /root/bin/aws configure set default.output json
}

main() {
    update_cache
    install_aws_cli
    configuring_aws
    update_cache
    install_k6
}

main "$@"

The following script will be executed to grant access to the public IP of the CircleCI machine:

#!/bin/bash
public_ip_address=$(curl -q http://checkip.amazonaws.com)

/root/bin/aws ec2 authorize-security-group-ingress --group-id $AWS_SECURITY_GROUP_ID --ip-permissions "[{\"IpProtocol\": \"tcp\", \"FromPort\": 443, \"ToPort\": 443, \"IpRanges\": [{\"CidrIp\": \"${public_ip_address}/32\"}]}]"

And, we must not forget to remove the grant access after the test execution:

#!/bin/bash
public_ip_address=$(curl -q http://checkip.amazonaws.com)

/root/bin/aws ec2 revoke-security-group-ingress --group-id $AWS_SECURITY_GROUP_ID --ip-permissions "[{\"IpProtocol\": \"tcp\", \"FromPort\": 443, \"ToPort\": 443, \"IpRanges\": [{\"CidrIp\": \"${public_ip_address}/32\"}]}]"

Below the final CircleCI configuration that glues all together:

default: &defaults
  parallelism: 1
  docker:
    - image: ubuntu:latest

setup: &setup
  run:
    name: Installing aws cli and setting it up
    command: |
      chmod 777 setup.sh
      sh setup.sh
authorize_circleci_through_firewall: &authorize_circleci_through_firewall
  run:
    name: Adding firewall rule to allow circleci through aws security group
    command: |
      chmod 777 permit.sh
      sh permit.sh
revoke_circleci_through_firewall: &revoke_circleci_through_firewall
  run:
    name: Remove firewall rule to deny circleci aws security group access
    command: |
      chmod 777 revoke.sh
      sh revoke.sh
k6_performance_tests: &k6_performance_tests
  run:
    name: Running Load Tests Using K6
    command: |
      k6 run loadtests/performance-test.js
version: 2
jobs:
  setup_authorize_test_revoke:
    <<: *defaults
    steps:
      - checkout
      - *setup
      - *authorize_circleci_through_firewall
      - *k6_performance_tests
      - *revoke_circleci_through_firewall

workflows:
  version: 2
  build-workflow:
    jobs:
      - setup_authorize_test_revoke

If you want to see the code of this example, the CircleCI configuration and bash scripts are available here.

Other CI/CD Workflows

In the previous example, we showed how to setup a simple CI configuration. But teams usually run their software operations with a more sophisticated development process. You could integrate your performance tests in different ways. For example:

• Run all tests you need to run with exception from performance tests. These tests will include unit tests, e2e tests, integration tests, etc.

• Deploy to staging or test environment or whichever environment you have set up for performance tests. If you do not have that set up yet, I suggest you do as it would not be wise to performance test a live production environment that is being used by real users, unless you know what you are doing.

• After the deployment definition, run your performance tests against your staging or test environment deployment.

• When that passes, we can now consider our workflow a success. We can now go ahead and merge the branch you were running on to your production branch.

The above is another simple example. In a real environment, you start deciding which performance tests to execute in your CI/CD pipelines and how to run them (either sequentially or in parallel) and configure your workflow to adapt to the environments, branching, jobs, and testing of your software development and DevOps process.

Nightly build

It's common to run some performance tests during the night when users do not access the system under test. For example, to isolate larger tests from other types of tests or to generate a performance report periodically.

Below is the first example configuring a scheduled nightly build that runs at midnight (UTC) everyday.

default: &defaults
  parallelism: 1
  working_directory: ~/my-project

k6_performance_tests: &k6_performance_tests
  run:
    name: Running k6 tests
    command: |
      docker pull loadimpact/k6:latest
      docker run -i -v $HOME/my-project:/ci/ loadimpact/k6:latest run /ci/loadtests/performance-test.js

version: 2
jobs:
  run_performance_tests:
    <<: *defaults
    machine: true
    steps:
      - checkout
      - *k6_performance_tests

workflows:
  version: 2
  nightly:
    triggers:
      - schedule:
          cron: '0 0 * * *'
          filters:
            branches:
              only:
                - master
    jobs:
      - run_performance_tests
  build-and-test:
    jobs:
      - run_performance_tests

This example is using the crontab syntax to schedule execution. To learn more, we recommend reading the scheduling a CircleCI workflow guide.

Storing result output as artifacts

We can store the k6 result output as artifacts in CircleCI so that we can inspect them later. This excerpt of the CircleCI config file shows how to do this. If you want to know more, please visit the storing build artifacts guide.

k6-load-tests:
  machine: true
  steps:
    - checkout
    - run:
        name: Running k6 tests
        command: |
          mkdir -p /results
          docker run \
            -v $(pwd)/test.js:/test.js \
            -v /results:/results \
            loadimpact/k6:latest \
            run \
            --out json=/results/full.json \
            --summary-export=/results/summary.json \
            /test.js
    - store_artifacts:
        path: /results/

Hope you enjoyed reading this article. We'd be happy to hear your feedback.