Tutorials 29 September 2018

Load testing with Jenkins

Load Impact

Load testing with Jenkins

In this tutorial, we will look into how to integrate performance testing with k6 in your Jenkins Pipeline setup.

k6 is a free and open-source performance testing tool for testing the performance of APIs, microservices and websites.Jenkins is an open source and extensible automation server for Continuous Integration and Continuous Delivery.
Integrating performance tests into your CI pipelines helps catch performance issues earlier and shipping more stable and performant applications to production.

The roadmap of this tutorial is going to follow the outline below:

The examples of this tutorial can be found here.

Write your performance test

To start, we will create a simple k6 test for our demo API. Your tests should be focused on endpoints and architecture that are important to your business.

The following test will run 50 Virtual Users continuously for a period of one minute, each VU execution generates one request and sleep for a period of 3 seconds. By design, this is a very simple script for illustrative purposes.

// performance-test.js

import { sleep } from"k6";
import http from "k6/http";

export let options = {
  duration: "1m",
  vus: 50
};



export default function() {
  http.get("https://test-api.loadimpact.com/public/crocodiles/");
  sleep(3);
}

If you have installed k6 in your local machine, you could run your test locally in your terminal using the command: k6 run performance-test.js

Configure Thresholds

The next step is to define your service level objectives, or SLOs around your application performance. SLOs are a vital aspect of ensuring the reliability of your systems and applications. If you do not currently have any defined SLAs or SLOs, now is a good time to start considering your requirements.

You can define SLOs as Pass/Fail criteria with Thresholds in your k6 script. k6 evaluates them during the test execution and inform about the Threshold results.

If a Threshold in your test fails, k6 will end with a non-zero exit code, which communicates to the CI tool that the step has failed.

Now, we will add one Threshold to our previous script to validate than the 95th percentile response time must be below 500ms.


export let options = {
  duration: "1m",
  vus: 50,
  thresholds: {
    http_req_duration: ["p(95)<500"]
  }
};

Thresholds is a powerful feature providing a flexible API to define various types of Pass/Fail criteria in the same test run. For example:

  • The 99th percentile response time must be below 700 ms.
  • The 95th percentile response time must be below 400 ms.
  • No more than 1% failed requests.
  • The content of a response must be correct more than 95% of the time.
  • ...

Check out the Thresholds documentation for extended information about the API and its usage.

Jenkins Pipeline Configuration

The Continuous Integration tool we are going to use in this tutorial is Jenkins.

For this tutorial, Jenkins is installed on a google compute instance on Google Cloud Platform. It's pretty easy to get this up and running, use the Google Cloud deployment manager. Simply search for "jenkins” in the top search bar and select the jenkins option with "click to deploy” which is the 3rd option in the screenshot below. This will install jenkins on an ubuntu linux google compute instance.

Install Jenkins with Google Cloud deployment manager

 

With that said, let’s go ahead and configure the Jenkins pipeline.

In the root of your project folder, create a file named Jenkinsfile. This is Jenkins’ configuration file that will trigger CI builds when a push to the remote repository is made. If you want to know more about it, check out the tutorial: Using a Jenkinsfile.

Alongside the Jenkinsfile, I've added setup_k6.sh script which installs k6 via Debian packages into the machine running Jenkins. For other ways to install k6, check out the k6 installation documentation.

#!/bin/bash
set -ex

install_k6() {
    apt-get update
    apt-get install dirmngr --install-recommends
    apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 379CE192D401AB61
    echo "deb https://dl.bintray.com/loadimpact/deb stable main" | sudo tee -a /etc/apt/sources.list
    apt-get update
    apt-get install k6
}

main() {
    install_k6
}

main "$@"

We then execute the setup_k6.sh script in the Jenkinsfile. Below is the Jenkinsfile configuration that ties everything together.

pipeline {
    agent any
    stages {
        stage('Performance Testing') {
            steps {
                echo 'Running K6 performance tests...'
                sh 'sudo chmod +x setup_k6.sh'
                sh 'sudo ./setup_k6.sh'
                sh 'k6 run loadtests/performance-test.js'
                echo 'Completed Running K6 performance tests!'
            }
        }
    }
}

After adding the above files to your local git repository, git add , git commit and eventually git push it to your cloud repository (bitbucket for this tutorial).

After you push to bitbucket, head over to your jenkins instance url to take a look at the build that we just triggered with the git push.

While on the jenkins dashboard, head over to the pipeline you created for your CI/CD pipeline and check out the running pipeline test logs.

Jenkins status - Performance Testing stage

 

Running k6 cloud tests

There are two common execution modes to run k6 tests as part of the CI process.

  • k6 run to run a test locally on the CI server.
  • k6 cloud to run a test on the LoadImpact cloud service from one or multiple geographic locations.

You might want to trigger cloud tests in these common cases:

  • If you want to run a test from one or multiple geographic locations (load zones).
  • If you want to run a test with high-load that will need more compute resources than provisioned by the CI server.

If any of those reasons fits your needs, then running k6 cloud tests is the way to go for you.

Before we start with the Jenkins configuration, it is good to familiarize ourselves with how cloud execution works, and we recommend you to test how to trigger a cloud test from your machine.

Check out the Cloud Execution Guide to learn how to distribute the test load across multiple geographic locations and more information about the cloud execution.

Now, we will show how to trigger cloud tests from Jenkins. If you do not have an account with LoadImpact already, you should go here and register for one as well as start the free trial.

After that, go to the API token page in LoadImpact and copy your API token.

Head over to https://jenkins.yourdomain.com/credentials/ and add K6_API_TOKEN to jenkins as environment variable.

Jenkins Credentials for k6 API Token

 

Now, you have to update the previous Jenkinsfile. It will look something like:

pipeline {
    agent any

    environment {
        K6_API_TOKEN=credentials("K6_API_TOKEN")
    }
    stages {
        stage('Performance Testing') {
            steps {
                echo 'Running K6 performance tests...'
                sh 'sudo chmod +x setup_k6.sh'
                sh 'sudo ./setup_k6.sh'
                sh 'k6 login cloud --token ${K6_API_TOKEN}'
                sh 'k6 cloud loadtests/performance-test.js'
                echo 'Completed Running K6 performance tests!'
            }
        }
    }
}

The changes of the previous configuration are:

  1. Authenticate to the LoadImpact Cloud Service using K6_API_TOKEN environment variable when running the command k6 login cloud --token ${K6_API_TOKEN}.

  2. Replace the k6 run command for k6 cloud command to trigger cloud tests.

With that done, now we can go ahead and git add, git commit and git push the changes we have made in the Jenkinsfile file to Bitbucket and initiates the Jenkins build.

When all is done and good, we should see a screen like this from jenkins build page:

Jenkins Console Output for load test

 

It is essential to know that Jenkins prints the output of the k6 command, and when running cloud tests, k6 prints the URL of the test result in the LoadImpact Cloud Service. You could navigate to this URL to see the result of your cloud test.

Jenkins Console Output for k6 cloud load test

 

We recommend that you define your performance thresholds in the k6 tests in a previous step. If you have configured your thresholds properly and your test passes, there should be nothing to worry about. But when the test fails, you want to understand why.

In this case, navigate to the URL of the cloud test to analyze the test result. The result of the cloud service will help you quickly find the cause of the failure.

k6 cloud test result

Nightly Triggers

Triggering a subset of performance tests at a specific time is a best practice for automating your performance testing.

It's common to run some performance tests during the night when users do not access the system under test. For example, to isolate larger tests from other types of testing or to generate periodically a performance report.

To configure scheduled nightly build that runs at a given time of a given day or night, head over to your jenkins dashboard and click on your pipeline and then click configure. While here, head over to the Build Triggers section and select the checkbox and fill in a cron-like value for the time you wish to trigger the pipeline execution at.

For example, to trigger a build at midnight everyday the cron value of H H * * * will do the job. The screenshot below shows the same. You should be aware that the time zone is relative to the location where your jenkins hosted machine is running. For this example it is in UTC.

Jenkins Nightly Build - Performance tests
< Back to all posts