How to Perform Load Testing with k6 using GitHub Actions

In this tutorial, we will look into how to integrate performance testing in your development process with GitHub Actions and k6. We will use a simple static website to learn how to set it up.

k6 is a free and open-source performance testing tool for testing the performance of APIs, microservices and websites. Developers use k6 to test the performance of a system under a particular load to catch performance regressions or errors.

GitHub is the world's most popular Git repository hosting service offering numerous collaboration features, access control and tools such as issue tracking, wikis and pages. Quite recently, GitHub launched GitHub Actions which is a new tool that enables developers to create custom software development life cycle(SDLC) workflows directly inside a GitHub repository.

The outline of this tutorial is as follows:

• Setting up Demo Project
• Write your performance test
• Configure Thresholds
• GitHub Actions Pipeline Configuration
• GitHub Actions for Windows and macOS Runners
• Running k6 cloud tests
• Nightly Triggers

You can find a collection of k6 scripts and GitHub workflows referenced in this tutorial here.

Prerequisites

To be able to follow through this tutorial, you will need to have the following installed on your computer:

• Git
• k6
• Now CLI

Some knowledge on how to work with GitHub actions is recommended. The following links will provide you with good reference guides:

• Workflow syntax for GitHub Actions
• Events that trigger workflows

Setting up Demo Project

As mentioned earlier, I have prepared a repository containing a starter-project which is simply a static site we will deploy and perform load testing on. We will use Zeit's free serverless hosting to deploy our project. If you wish to follow along, you'll need to setup the demo project on your workspace in one of the following ways:

Method 1:

Duplicate the repository to your account by following these instructions on how to mirror a repository. I recommend you change the project name to something unique. This way when you deploy the project, you site will automatically be assigned a public URL that will match your project name: <project-name>.now.sh.

Method 2:

Fork the project to your GitHub account, then download it to your workspace. Do note that a random URL will will likely be assigned when you deploy. This is not a problem and you can proceed with the tutorial as is. If you want, you can change the domain to something different by visiting the Zeit's deployment dashboard for the project.

When it comes to load testing, you should always do it in a pre-production area. Testing in production is risking interruption to business, and should only be done if your processes are mature enough to support it, eg. you’re already doing chaos engineering ;)

The pre-production area a.k.a "staging" should match the production environment as much as possible in order for test results to be reliable. Luckily, we don't need to configure any environments as it's handled automatically for us when launching deployments:

# Deploy to Staging
now --target staging

# Deploy to Production (a.k.a now --target production)
now --prod

In my case, my staging area URL is https://static-bootstrap-site.brandiqa.now.sh. This is the URL I'll use to perform load testing. If the test passes, I'll perform a production deployment where the site will be published on https://static-bootstrap-site.now.sh. I'll show you how to automate this process using GitHub Actions and a simple branch strategy.

Automating Deployments

We'll setup production deployments to only occur on master branch which will be triggered by the push event. We'll employ a simple branching strategy where we'll use feature branches to introduce new code to the project. When a pull request is created, deployment to the staging area will launch. If the process is successful, GitHub will put a checkmark that the new feature branch can safely merge with the master branch. Once you commit this merge, the production deployment will be launched.

Note: A pull request is a request initiated from a feature branch to merge changes with the master branch.

Let's start by creating our production deployment workflow first. Create the file .github/workflows/production-deploy.yml and copy the following:

name: Deploy to Production
on:
   push:
    branches:
      - master

jobs:
  deploy:
    name: Deploy to Zeit
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v1

      - name: Deploy to Zeit
        uses: amondnet/now-deployment@v1
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          zeit-token: ${{ secrets.ZEIT_TOKEN }}
          now-args: '--prod'

We are using Zeit Now Deployment GitHub Action to deploy our code straight from our remote repository. Take note of the now-args that specifies we are deploying to production. GITHUB_TOKEN will be set automatically by GitHub when the workflow is run. For ZEIT_TOKEN, you will need to generate one from your Zeit account. Once acquired, set it in Secrets page located in your GitHub repository settings.

The next workflow we need to create is for automating stage deployment. Create the file .github/workflows/stage-deploy.yml and copy the following:

name: Stage Deploy Website
on:
   pull_request:
    branches:
      - master

jobs:
  deploy:
    name: Deploy to Zeit
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v1

      - name: Deploy to Zeit
        uses: amondnet/now-deployment@v1
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          zeit-token: ${{ secrets.ZEIT_TOKEN }}
          now-args: '--target staging'

This workflow will only trigger when we create a pull request. Take note of the arguments specified in now-args. With that set, you can commit both files and make a push. Only the deployment workflow should run.

If you click on the active workflow link, you can view the livestream logs. Confirm on the Zeit dashboard that your site has been deployed. In the next section, we will write the performance test and learn how to automate it as well using GitHub actions.

Write your performance test

Before we write our performance test, we need to determine our staging URL. Simply execute the command now --target staging:

In my case, the staging URL I'll use to perform load testing on is https://static-bootstrap-site.brandiqa.now.sh. With that determined, we'll create a simple k6 test.

The following test will run 50 Virtual Users continuously for a period of one minute, each VU execution generates one request and sleep for a period of 3 seconds. Make sure to specify your staging URL in the place I've indicated.

// scripts/performance-test.js

import { sleep } from"k6";
import http from "k6/http";

export let options = {
  duration: "1m",
  vus: 50
};

export default function() {
  http.get("https://<your staging URL>.now.sh"); // Place your staging URL here
  sleep(3);
}

If you have installed k6 in your local machine, you can run your test locally in your terminal using the command: k6 run scripts/performance-test.js.

Configure Thresholds

The next step is to define your service level objectives, or SLOs around your application performance. SLOs are a vital aspect of ensuring the reliability of your systems and applications. If you do not currently have any defined SLAs or SLOs, now is a good time to start considering your requirements.

You can define SLOs as Pass/Fail criteria with Thresholds in your k6 script. k6 evaluates them during the test execution and inform about the Threshold results.

If a Threshold in your test fails, k6 will end with a non-zero exit code, which communicates to the CI tool that the step has failed.

Now, we will add one Threshold to our previous script to validate than the 95th percentile response time must be below 500ms.

export let options = {
  duration: "1m",
  vus: 50,
  thresholds: {
    http_req_duration: ["p(95)<500"]
  }
};

Thresholds is a powerful feature providing a flexible API to define various types of Pass/Fail criteria in the same test run. For example:

• The 99th percentile response time must be below 700 ms.
• The 95th percentile response time must be below 400 ms.
• No more than 1% failed requests.
• The content of a response must be correct more than 95% of the time.
• …

Check out the Thresholds documentation for extended information about the API and its usage.

Configure GitHub Actions for load testing with k6

To integrate performance testing in our workflow, open .github/workflows/stage-deploy.yml and append the following:

  k6_load_test:
      name: k6 Load Test
      runs-on: ubuntu-latest
      container: docker://loadimpact/k6:latest

      steps:
      - name: Checkout
        uses: actions/checkout@v1

      - name: Local k6 test
        run: k6 run scripts/performance-test.js

The k6_load_test job specified above will run right after stage deployment has completed successfully. We'll need to commit and push the new changes to the master branch first.

Next, let's confirm that our automated workflows are working. Start by creating a new branch:

git checkout -b feature-1

Open public/index.html and update the content in the Jumbotron header:

<!-- Jumbotron Header -->
    <header class="jumbotron my-4">
      <h1 class="display-3">Load Testing : Feature #01</h1>
      ....
    </header>

Commit this file and push the change. In the terminal, you should see the URL to visit to create a pull request:

Visit the page and and click the Create pull request button. This should initialize the stage-deployment workflow:

In about a minute or so, the checks will resolve to green indicating the stage deployment and load testing was successful.

Here are the logs of the k6 performance that just completed:

If you visit your staging URL site, you should note that the home page title has changed. However, the site for the production site should remain unchanged. The next step is to merge the feature branch with master. Simply click the 'Merge pull request' button. Feel free to put in a comment in the description box that appears. Click the 'Confirm merge' button. You should get the message 'Pull request successfully merged and closed'.

Do note that this is a push event which should trigger production-deploy.yml workflow. In less than a minute, the main site should update:

Now that we've covered the main workflows of using GitHub Actions with k6, let's look at some options.

GitHub Actions for Windows and macOS Runners

GitHub provides Windows and macOS environments to run your workflows. You can also setup custom runners that operate on your premises or your cloud infrastructure. The k6-load-testing workflow we have used above is based on a docker image, k6:latest, maintained by Docker. Using GitHub workflow syntax, you can build your own container using any Linux distro as base. Then, you can follow these instructions to perform k6 installation on your Linux container. From there, you can call k6 commands as usual.

The following are workflow examples for Windows and macOS.

Windows:

on: [push]

jobs:  
  k6_local_test:
    name: k6 local test run on windows
    runs-on: windows-latest
  
    steps:
    - name: Checkout
      uses: actions/checkout@v1

    - name: download and extract k6 release binaries
      run: |
        curl -L https://github.com/loadimpact/k6/releases/download/v0.25.1/k6-v0.25.1-win64.zip -o k6.zip
        7z.exe e k6.zip
      shell: bash

    - name: k6 test
      run: ./k6.exe run scripts/performance-test.js
      shell: bash

Here is the most up-to-date for k6 Windows installation instructions. I would recommend using Chocolatey Package Manager to ensure your script grabs the latest k6 version.

macOS:

on: [push]

jobs:  
  k6_local_test:
    name: k6 local test on macos
    runs-on: macos-latest
  
    steps:
    - name: Checkout
      uses: actions/checkout@v1

    - name: Install k6 by homebrew
      run: brew install k6

    - name: Local k6 test
      run: k6 run scripts/performance-test.js

The brew package manager is the best tool for grabbing and installing the latest version of k6 whenever the workflow is run.

Passing environment variables

If you would like to use environment variables instead of hard coding values in your script, you can set it up like this:

// performance test

export default function() {
    var r = http.get(`http://${__ENV.MY_HOSTNAME}/`);
    check(r, {
        "status is 200": (r) => r.status === 200
    });
    sleep(5);
}

The environment variable, __ENV.MY_HOSTNAME will need to be specified on your GitHub Action yaml file like this:

on: [push]

jobs:  
  k6_local_test:
    name: k6 local test run - env vars example
    runs-on: ubuntu-latest
    container: docker://loadimpact/k6:latest
  
    steps:
    - name: Checkout
      uses: actions/checkout@v1

    - name: Local k6 test with env vars
      env:
        MY_HOSTNAME: test.loadimpact.com
      run: k6 run scripts/performance-test.js

Running k6 cloud tests

There are two common execution modes to run k6 tests as part of the CI process.

k6 run to run a test locally on the CI server. k6 cloud to run a test on the LoadImpact cloud service from one or multiple geographic locations. You might want to trigger cloud tests in these common cases:

• If you want to run a test from one or multiple geographic locations (load zones).
• If you want to run a test with high-load that will need more compute resources than provisioned by the CI server.

If any of those reasons fits your needs, then running k6 cloud tests is the way to go for you.

Before we start with the configuration, it is good to familiarize ourselves with how cloud execution works, and we recommend you to test how to trigger a cloud test from your machine. Check out the Cloud Execution Guide to learn how to distribute the test load across multiple geographic locations and more information about the cloud execution.

Now, we will show how to trigger cloud tests using GitHub actions. If you do not have an account with LoadImpact already, you should go here and register for one as well as start the free trial.

After that, go to the API token page in LoadImpact and copy your API token. Add this token to your GitHub's project Secrets page.

If you would like to follow along with the tutorial, create a second feature branch using the command git checkout -b feature-2. To run the cloud tests, you can update the existing script scripts/performance-test.js or create a new one: scripts/cloud_performance_test.js:

import { check, sleep } from "k6";
import http from "k6/http";

export let options = {
    duration: "1m",
    vus: 50,
    thresholds: {
        http_req_duration: ["p(95)<500"]
    }
};

export default function () {
    var r = http.get("https://static-bootstrap-site.brandiqa.now.sh/");
    check(r, {
        "status is 200": (r) => r.status === 200
    });
    sleep(1);
}

Creating a second script is not necessary as you can still run the first performance script in the cloud as well. Next, replace k6_local_test job in .github/workflows/stage-deploy.yml workflow with the following:

  k6_cloud_test:
    name: k6 cloud test run
    runs-on: ubuntu-latest
    container: docker://loadimpact/k6:latest
  
    steps:
    - name: Checkout
      uses: actions/checkout@v1

    - name: Cloud k6 test
      env:
        K6_CLOUD_TOKEN: ${{ secrets.k6_cloud_token }}
      run: k6 cloud scripts/cloud-performance-test.js

Feel free to update the title in public/index.html. After you have saved the files. Commit and push the new branch: git push -u origin feature-2. Just like before, create a new pull request to trigger stage deployment and load testing.

Wait for the performance test to complete. Then copy the URL highlighted below to a new tab:

And here are the results as you can clearly see, our performance test has passed the threshold we specified earlier.

Nightly build

Triggering a subset of performance tests at a specific time is a best practice for automating your performance testing.

It’s common to run some performance tests during the night when users do not access the system under test. For example, to isolate larger tests from other types of testing or to generate periodically a performance report.

To configure scheduled nightly build that runs at a given time of a given day or night, head over to your GitHub action workflow and update the on section. Here is an example that triggers the workflow every 15 minutes:

on:
  schedule:
    # * is a special character in YAML so you have to quote this string
    - cron:  '*/15 * * * *'

You'll to use POSIX cron syntax to schedule a workflow to run at specific UTC times. Here is a cheat sheet for crontab scheduling expressions.

Simply save, commit and push the file. GitHub will take care of running the workflow at the time intervals you specified.