Load testing with CircleCI

In this tutorial, we will look into how to integrate k6 tests into CircleCI.

k6 OSS is an open-source load testing tool for testing the performance of APIs, microservices and websites. Developers use k6 to test the performance of a system under a particular load to catch performance regressions or errors.

CircleCI is a Continuous Integration and Delivery tool to automate your development process.

Continuous Integration is a practice that encourages developers to integrate their code into a main branch of a shared repository early and often. Continuous Delivery extends CI to ensure the code of a branch is correct and deploy it to users.

Integrating performance tests into your CI pipelines helps catch performance issues earlier and shipping more stable and performant applications to production.

The roadmap of this tutorial is going to follow the outline below:

• Write your performance test
• Configure Thresholds
• Write a CircleCI config file
• Running k6 cloud tests
• Load testing behind the firewall
• Other CI/CD Workflows

The examples of this tutorial can be found here.

Prerequisites

To be able to follow through this tutorial, you will need to have the following installed on your computer:

• Git
• k6
• CircleCI (version 2.x)

Write your performance test

To start, we will create a simple k6 test for our demo API, but you could write more realistic tests simulating other API interactions.

The following test will run 50 Virtual Users continuously for a period of one minute, each VU execution generates one request and sleep for a period of 3 seconds.

// performance-test.js

import { sleep } from"k6";
import http from "k6/http";

export let options = {
  duration: "1m",
  vus: 50
};



export default function() {
  http.get("https://test-api.loadimpact.com/public/crocodiles/");
  sleep(3);
}

If you have installed k6 in your local machine, you could run your test locally in your terminal using the command: k6 run performance-test.js

What type of performance tests should run as part of my CI pipelines?

There is no one answer for the type of performance tests to integrate into your CI/CD pipeline. While planning and developing your performance tests, you often decide which performance tests are good candidates to be run regularly as part of your automation pipelines.

Tip: It’s common to reuse the same test with varying load configurations or for testing different environments. You could use Environment variables in your k6 script to modify the test behavior depending on the test execution context.

Configure Thresholds

The next step is to define your service level objectives, or SLOs around your application performance. SLOs are a vital aspect of ensuring the reliability of your systems and applications.

k6 allows you to define SLOs as Pass/Fail criteria with Thresholds. You should configure them based on the particular test and the expected performance of your application. For example:

• The 99th percentile response time must be below 700 ms.
• No more than 1% failed requests.

In k6, Thresholds provides a powerful and flexible API to define various types of Pass/Fail criteria. And you will be able to easily visualize and get notified about the status of the Thresholds in your test result.

Now, we will add one Threshold to our previous example to ensure than the 95th percentile response time must be below 500ms.

export let options = {
  duration: "1m",
  vus: 50,
  thresholds: {
    http_req_duration: ["p(95)<500"]
  }
};

If a Threshold in your test fails, k6 will end with a non-zero exit code, which in turn indicates a failed build step to our CI tool.

Write a CircleCI config file

The Continuous Integration tool we are going to use in this tutorial is CircleCI.

If you have not enabled CircleCI for your repository yet, do so by using the "Add Project” button.

In the root of your project folder, create a folder with this name .circleci and inside that folder, create a config.yml file. Those exact names should be used else CircleCI will not get triggered when you push to your remote repository.

Add the code below into the config.yml file:

default: &defaults
  parallelism: 1
  working_directory: ~/my-project

k6_performance_tests: &k6_performance_tests
  run:
    name: Running k6 tests
    # Download the k6 docker image. Alternatively, download the k6 release binary
    # Mount a volume to access the folder and run the test
    command: |
      docker pull loadimpact/k6:latest
      docker run -i -v $HOME/my-project:/ci/ loadimpact/k6:latest run /ci/loadtests/performance-test.js

version: 2
jobs:
  run_performance_tests:
    <<: *defaults
    # Use `machine` executor because the Docker executor cannot mount volumes
    machine: true
    steps:
      - checkout
      - *k6_performance_tests

workflows:
  version: 2
  build-and-test:
    jobs:
      - run_performance_tests

After adding the above code, git add , git commit and eventually git push it to your cloud repository. Now, head over to circleci.com to checkout the build that we just triggered with the git push to github.

Click on the JOBS section of the sidebar menu and select your project.

On this page you will have your initiated build running, click on the specific build to watch the test run and see the results when it has finished. Below is the build pages for the succeeded build:

Running k6 cloud tests

There are two common execution modes to run k6 tests as part of the CI process.

• k6 run to run a test locally on the CI server.
• k6 cloud to run a test on the k6 Cloud SaaS from one or multiple geographic locations.

You might want to trigger cloud tests in these common cases:

• If you want to run a test from one or multiple geographic locations (load zones).
• If you want to run a test with high-load that will need more compute resources than provisioned by the CI server.

If any of those reasons fits your needs, then running k6 cloud tests is the way to go for you.

Before we start with the CircleCI configuration, it is good to familiarize ourselves with how cloud execution works, and we recommend you to test how to trigger a cloud test from your machine.

Check out the Cloud Execution Guide to learn how to distribute the test load across multiple geographic locations and more information about the cloud execution.

Now, we will show how to trigger cloud tests from CircleCI. If you do not have an account with LoadImpact already, you should go here and register for one as well as start the free trial.

After that, go to the API token page in LoadImpact and copy your API token.

Navigate to the project's settings in CircleCI and select the Environment variables page. Add a new Environment Variable with name K6_CLOUD_TOKEN whose value must be your API token.

When CircleCI executes a k6 command, the K6_CLOUD_TOKEN environment variable will automatically authenticate you to the LoadImpact Cloud Service:

Now, we can configure the CircleCI to trigger cloud tests. If you have a CircleCI config file running k6 tests, you would only have to replace the k6 run command for k6 cloud command and pass your API Token to the Docker container. The change is:

docker run -i -e K6_CLOUD_TOKEN=$K6_CLOUD_TOKEN -v $HOME/my-project:/ci/ loadimpact/k6:latest cloud /ci/loadtests/performance-test.js

With that done, now we can go ahead and git add, git commit and git push the changes we have made in the .circleci/config.yml file to github and initiate the circleci build.

When all is done and good, we should see a screen like this from circleci build page:

It is essential to know that CircleCI prints the output of the k6 command, and when running cloud tests, k6 prints the URL of the test result in the LoadImpact Cloud Service. You could navigate to this URL to see the result of your cloud test.

We recommend that you define your performance thresholds in the k6 tests in a previous step. If you have configured your thresholds properly and your test passes, there should be nothing to worry about. But when the test fails, you want to understand why.

In this case, navigate to the URL of the cloud test to analyze the test result. The result of the cloud service will help you quickly find the cause of the failure.

Load testing behind the firewall

If the system under test is behind a firewall, we have to enable the IPs of the k6 instances to access it.

For cloud tests

If you are running a k6 cloud test, you will be utilizing LoadImpact’s infrastructure. Check out this guide to open your firewall to the LoadImpact Cloud service.

When the test runs in the CI server

We need to grant CircleCI access to our system by adding the necessary IP ranges to the firewall. If you're using AWS you can temporarily grant access by adding a security group rule pre-test. Make sure you have created an aws user with the ec2* role to allow CircleCI edit rights to VPC security group. When the user is created, copy the access key, secret key ID, and security group ID to CircleCI environment variables:

• $AWS_ACCESS_KEY
• $AWS_SECRET_ACCESS_KEY
• $AWS_SECURITY_GROUP_ID

The configuration of this example is different than the previous ones, CircleCI will setup a Docker executor for our CircleCI instance and will install k6 from the apt package manager instead of using the k6 Docker image.

To set this up, CircleCI will run the following script to install k6 and the aws cli to your CircleCI instance running the tests.

#!/bin/bash

set -ex
# set -o pipefail

update_cache() {
    apt-get update
}

install_k6() {
    sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 379CE192D401AB61
    echo "deb https://dl.bintray.com/loadimpact/deb stable main" | sudo tee -a /etc/apt/sources.list
    sudo apt-get update
    sudo apt-get install k6
}

install_aws_cli() {
    apt-get install curl unzip python2.7 python-pip sudo -y
    curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
    unzip awscli-bundle.zip
    ./awscli-bundle/install -b ~/bin/aws

}

configuring_aws() {
    /root/bin/aws configure set aws_access_key_id $AWS_ACCESS_KEY
    /root/bin/aws configure set aws_secret_access_key $AWS_SECRET_ACCESS_KEY
    /root/bin/aws configure set default.region us-east-1
    /root/bin/aws configure set default.output json
}

main() {
    update_cache
    install_aws_cli
    configuring_aws
    update_cache
    install_k6
}

main "$@"

The following script will be executed to grant access to the public IP of the CircleCI machine:

#!/bin/bash
public_ip_address=$(curl -q http://checkip.amazonaws.com)

/root/bin/aws ec2 authorize-security-group-ingress --group-id $AWS_SECURITY_GROUP_ID --ip-permissions "[{\"IpProtocol\": \"tcp\", \"FromPort\": 443, \"ToPort\": 443, \"IpRanges\": [{\"CidrIp\": \"${public_ip_address}/32\"}]}]"

And, we must not forget to remove the grant access after the test execution:

#!/bin/bash
public_ip_address=$(curl -q http://checkip.amazonaws.com)

/root/bin/aws ec2 revoke-security-group-ingress --group-id $AWS_SECURITY_GROUP_ID --ip-permissions "[{\"IpProtocol\": \"tcp\", \"FromPort\": 443, \"ToPort\": 443, \"IpRanges\": [{\"CidrIp\": \"${public_ip_address}/32\"}]}]"

Below the final CircleCI configuration that glues all together:

default: &defaults
  parallelism: 1
  docker:
    - image: ubuntu:latest

setup: &setup
  run:
    name: Installing aws cli and setting it up
    command: |
      chmod 777 setup.sh
      sh setup.sh
authorize_circleci_through_firewall: &authorize_circleci_through_firewall
  run:
    name: Adding firewall rule to allow circleci through aws security group
    command: |
      chmod 777 permit.sh
      sh permit.sh
revoke_circleci_through_firewall: &revoke_circleci_through_firewall
  run:
    name: Remove firewall rule to deny circleci aws security group access
    command: |
      chmod 777 revoke.sh
      sh revoke.sh
k6_performance_tests: &k6_performance_tests
  run:
    name: Running Load Tests Using K6
    command: |
      k6 run loadtests/performance-test.js
version: 2
jobs:
  setup_authorize_test_revoke:
    <<: *defaults
    steps:
      - checkout
      - *setup
      - *authorize_circleci_through_firewall
      - *k6_performance_tests
      - *revoke_circleci_through_firewall

workflows:
  version: 2
  build-workflow:
    jobs:
      - setup_authorize_test_revoke

If you want to see the code of this example, the CircleCI configuration and bash scripts are available here.

Other CI/CD Workflows

In the previous example, we showed how to setup a simple CI configuration. But teams usually run their software operations with a more sophisticated development process. You could integrate your performance tests in different ways. For example:

• Run all tests you need to run with exception from performance tests.

  These tests will include unit tests, e2e tests, integration tests, etc.

• Deploy to staging or test environment or whichever environment you have set up for performance tests. If you do not have that set up yet, I suggest you do as it would not be wise to performance test a live production environment that is being used by real users.

• After the deployment definition, run your performance tests against your staging or test environment deployment.

• When that passes, we can now consider our workflow a success. We can now go ahead and merge whatever that branch you were running on to the master or development branch.

The above is another simple example. In a real environment, you start deciding which performance tests to execute in your CI/CD pipelines and how to run them (sequentially or parallelly) and configure your workflow to adapt to the environments, branching, jobs, and testing of your software development and DevOps process.

Nightly build

It's common to run some performance tests during the night when users do not access the system under test. For example, to isolate larger tests from other types of testing or to generate periodically a performance report.

Triggering a subset of performance tests at a specific time is considered a best practice for automating your performance testing.

Below is the first example configuring a scheduled nightly build that runs at midnight (UTC) of everyday.

default: &defaults
  parallelism: 1
  working_directory: ~/my-project

k6_performance_tests: &k6_performance_tests
  run:
    name: Running k6 tests
    command: |
      docker pull loadimpact/k6:latest
      docker run -i -v $HOME/my-project:/ci/ loadimpact/k6:latest run /ci/loadtests/performance-test.js

version: 2
jobs:
  run_performance_tests:
    <<: *defaults
    machine: true
    steps:
      - checkout
      - *k6_performance_tests

workflows:
  version: 2
  nightly:
    triggers:
      - schedule:
          cron: "0 0 * * *"
          filters:
            branches:
              only:
                - master
    jobs:
      - run_performance_tests
  build-and-test:
    jobs:
      - run_performance_tests

The CircleCI configuration could be familiar, as is using the crontab syntax. To learn more, we recommend reading the Scheduling a CircleCI workflow guide.