Tutorials 30 September 2018

Load testing with GitLab

Slava Karhaltsev

Load testing with GitLab

In this tutorial, we will look into how to integrate performance testing in your development process with GitLab and k6.

k6 is a free and open-source performance testing tool for testing the performance of APIs, microservices and websites. Developers use k6 to test the performance of a system under a particular load to catch performance regressions or errors.

GitLab is a complete DevOps Platform including a CI/CD toolchain and many more features.

The roadmap of this tutorial is going to follow the outline below:

The examples of this tutorial can be found here.

Write your performance test

To start, we will create a simple k6 test for our demo API; the following test will run 50 Virtual Users continuously for a period of one minute, each VU execution generates one request and sleep for a period of 3 seconds.

By design, this is a very simple script for illustrative purposes.

// performance-test.js

import { sleep } from"k6";
import http from "k6/http";

export let options = {
  duration: "1m",
  vus: 50
};

export default function() {
  http.get("https://test-api.loadimpact.com/public/crocodiles/");
  sleep(3);
}

If you have installed k6 in your local machine, you could run your test locally in your terminal using the command: k6 run performance-test.js

What type of performance tests should run as part of my CI pipelines?

There is no one answer for the type of performance tests to integrate into your CI/CD pipeline. While planning and developing your performance tests, you often decide which performance tests are good candidates to be run regularly as part of your automation pipelines.

Configure Thresholds

Now, it is a good time to start considering your service level objectives, or SLOs around your application performance. SLOs are a vital aspect of ensuring the reliability of your systems and applications.

You can define SLOs as Pass/Fail criteria with Thresholds in your k6 script. k6 evaluates them during the test execution and inform about the Threshold results.

If a Threshold in your test fails, k6 will end with a non-zero exit code, which indicates a failed build step to the CI tool.

Now, we add one Threshold to validate than the 95th percentile response time must be below 500ms.


export let options = {
  duration: "1m",
  vus: 50,
  thresholds: {
    http_req_duration: ["p(95)<500"]
  }
};

Thresholds provide a flexible API to define various types of Pass/Fail criteria in the same test run. For example:

  • The 99th percentile response time must be below 700 ms and the 95th below 400 ms.
  • No more than 0.5% failed requests.
  • The content of a response must be correct more than 99% of the time.
  • ...

Read the Thresholds documentation for extended information about the API and its usage.

Write a GitLab config file

The Continuous Integration tool we are going to use in this tutorial is GitLab.

In the root of your project folder, create a file with this name .gitlab-ci.yml, add the code below into it:

stages:
  - build
  - deploy
  - loadtest-local

image:
  name: ubuntu:latest

build:
  stage: build
  script:
    - echo "building my application in ubuntu container..."

deploy:
  stage: deploy
  script:
    - echo "deploying my application in ubuntu container..."

loadtest-local:
  image:
    name: loadimpact/k6:latest
    entrypoint: [""]
  stage: loadtest-local
  script:
    - echo "executing local k6 in k6 container..."
    - k6 run ./loadtests/performance-test.js

After adding the above code, git add , git commit and eventually git push it to your cloud repository. Now, head over to GitLab repository to checkout the job that we just triggered with the git push.

Click on the CI / CD -> Jobs section of the sidebar menu.

GitLab Job list

On this page you will have your initiated jobs running, click on the specific job to watch the test run and see the results when it has finished. Below is the succeeded job:

GitLab Job running a k6 load test

Running k6 cloud tests

There are two common execution modes to run k6 tests as part of the CI process.

  • k6 run to run a test locally on the CI server.
  • k6 cloud to run a test on the LoadImpact cloud service from one or multiple geographic locations.

You might want to trigger cloud tests in these common cases:

  • If you want to run a test from one or multiple geographic locations (load zones).
  • If you want to run a test with high-load that will need more compute resources than provisioned by the CI server.

If any of those reasons fits your needs, then running k6 cloud tests is the way to go for you.

Before we start with the GitLab configuration, it is good to familiarize ourselves with how cloud execution works, and we recommend you to test how to trigger a cloud test from your machine.

Check out the Cloud Execution Guide to learn how to distribute the test load across multiple geographic locations and more information about the cloud execution.

Now, we will show how to trigger cloud tests from GitLab. If you do not have an account with LoadImpact already, you should go here and register for one as well as start the free trial.

After that, go to the API token page in LoadImpact and copy your API token.

Navigate to Settings -> CI / CD -> Variables of GitLab repository and expand variables. Add a new Environment Variable with name K6_CLOUD_TOKEN whose value must be your API token.

When GitLab runs a job, the K6_CLOUD_TOKEN environment variable will automatically authenticate you to the LoadImpact Cloud Service:

GitLab Environment Variable for k6 API Token

 

Now, we can configure the GitLab to trigger cloud tests. If you have a GitLab config file running k6 tests, you would only have to replace the k6 run command for k6 cloud command and pass your API Token to the Docker container. So in order to add k6 cloud execution we would add a loadtest-cloud job in .gitlab-ci.yml file:

stages:
  - build
  - deploy
  - loadtest-local
  - loadtest-cloud

image:
  name: ubuntu:latest

build:
  stage: build
  script:
    - echo "building my application in ubuntu container..."

deploy:
  stage: deploy
  script:
    - echo "deploying my application in ubuntu container..."

loadtest-local:
  image:
    name: loadimpact/k6:latest
    entrypoint: [""]
  stage: loadtest-local
  script:
    - echo "executing local k6 in k6 container..."
    - k6 run ./loadtests/performance-test.js

loadtest-cloud:
  image:
    name: loadimpact/k6:latest
    entrypoint: [""]
  stage: loadtest-cloud
  script:
    - echo "executing cloud k6 in k6 container..."
    - k6 cloud ./loadtests/performance-test.js

With that done, now we can go ahead and git add, git commit and git push the changes we have made in the .gitlab-ci.yml file and initiate the GitLab job.

When all is done and good, we should see a screen like this from GitLab job page:

GitLab Job running a k6 cloud test

 

It is essential to know that GitLab prints the output of the k6 command, and when running cloud tests, k6 prints the URL of the test result in the LoadImpact Cloud Service. You could navigate to this URL to see the result of your cloud test.

We recommend that you define your performance thresholds in the k6 tests in a previous step. If you have configured your thresholds properly and your test passes, there should be nothing to worry about. But when the test fails, you want to understand why.

In this case, navigate to the URL of the cloud test to analyze the test result. The result of the cloud service will help you quickly find the cause of the failure.

k6 cloud test result

Nightly Triggers

It's common to run some performance tests during the night when users do not access the system under test. For example, to isolate larger tests from other types of testing or to generate periodically a performance report.

Triggering a subset of performance tests at a specific time is considered a best practice for automating your performance testing.

To configure scheduled nightly pipeline that runs at a given time of a given day or night in GitLab, head over to CI / CD -> Schedules section of your repository. Click New schedule and configure it.

For example, to trigger our jobs at midnight everyday the cron value of 0 0 * * * will do the job. The screenshot below shows the same:

GitLab Pipeline Schedule

After clicking button Save pipeline schedule you will see that schedule is created and when next time pipeline will be triggered:

GitLab Pipeline Schedule Created

To learn more about GitLab pipeline schedules, we recommend reading the Pipeline schedules documentation.

< Back to all posts