08 November 2022

Performance Testing in Keptn using k6

Jainam Shah

This tutorial will demonstrate Keptn, a CNCF Incubator Project’s integration with k6. Keptn is a cloud-native application delivery and operations platform. We will use the Job Executor Service to execute k6 performance testing in a Keptn project. We'll start with running a k6 script and how the logs look. And then we'll modify the k6 script to see the behavior when it fails.

A demo of this tutorial can be found in YouTube.

About Keptn

Keptn is an event-driven orchestration engine connecting observability with cloud-native applications operations. The project uses a declarative approach to build scalable automation for delivery and operations, evaluates Service Level Indicators (SLOs), and provides a dashboard, alerts, and auto-remediation for them. It allows us to define multi-stage delivery pipelines declaratively.

Keptn allows you to pick a use case and automate & integrate it in a more general way. Depending on the use case, you bring a configuration. For instance:

  • Quality Gates <> SLI/SLO config
  • Progressive Delivery <> Shipyard file
  • Remediation <> Runbook for fixing system

Keptn & k6 Test Workflow

This will the workflow of events in this tutorial:

Keptn & k6 Test Workflow


  • Shipyard: a Shipyard file is declared on a project Level in Keptn. All the services in the project will share the same shipyard definition. Shipyard defines the stages which each deployment would have to go through, e.g. dev, staging, production stage. It allows us to declare multi-stage delivery workflows by defining dedicated strategies in them.

  • Stage: A stage is declared by its name. This name will be used for the branch in the Git repository and Kubernetes namespace to which services at this stage will be deployed to.

  • Sequence: After defining stages, sequences can be added to a stage. A sequence is an ordered list of tasks that are triggered sequentially. K6 will use the **test(()) sequence reserved in Keptn for testing any microservice.

  • Job Executor Service: Job Executor Service allows us to run customizable tasks with Keptn as Kubernetes Jobs. This will allow us to integrate k6 testing easily with Keptn. You can read more about it at keptn-contrib/job-executor-service.

Setup

Install Keptn

Setup Keptn from the quickstart guide.

Install Job Executor Service

The Job Executor Service can be installed using this command:

JES_VERSION="0.2.0"
JES_NAMESPACE="keptn-jes"
TASK_SUBSCRIPTION="sh.keptn.event.test.triggered" # Events used in current tutorial
helm upgrade --install --create-namespace -n ${JES_NAMESPACE} \
job-executor-service "https://github.com/keptn-contrib/job-executor-service/releases/download/${JES_VERSION}/job-executor-service-${JES_VERSION}.tgz" \
--set remoteControlPlane.autoDetect.enabled="true",remoteControlPlane.topicSubscription="${TASK_SUBSCRIPTION}",remoteControlPlane.api.token="",remoteControlPlane.api.hostname="",remoteControlPlane.api.protocol=""

For more information regarding the latest version of Job Executor Service, please follow this keptn-contrib/job-executor-service. In this tutorial, the Job Executor Service should listen to sh.keptn.event.test.triggered CloudEvent, which can be configured in the shipyard file.

If you have Job Executor Service installed already, please directly subscribe to sh.keptn.event.test.triggered event from Bridge after creating Keptn Project.

Local Files Setup

Clone this repository keptn-sandbox/k6-service from GitHub, and find all the files here in docs/k6-jes-example folder. Please enter into that directory to find all the files path in commands given in this tutorial.

Creating Project

Create a new Keptn project using the following command:

keptn create project k6-jes --shipyard=./shipyard.yaml --git-user=<GIT_USER> --git-token=<GIT_TOKEN> --git-remote-url=<UNINTIALIZED_GIT_REPO_URL>

This command will create the project k6-jes and in the mentioned GIT_REPO, have a shipyard.yaml file in the master branch and initialize the production branch based on the stage mentioned in the file.

Creating Service

Create a microserviceA service using the command:

keptn create service microserviceA --project k6-jes -y

This command will create a microserviceA service. It will have a job config file for the k6 testing command and file path.

Adding Resources

Next, we'll add config files for microserviceA services using the commands:

keptn add-resource --project k6-jes --service microserviceA --stage production --resource ./production/microserviceA/job/config.yaml --resourceUri job/config.yaml
keptn add-resource --project k6-jes --service microserviceA --stage production --resource ./production/microserviceA/files/k6_test.js --resourceUri files/k6_test.js

This will add config.yaml and k6 test file to the production branch on GIT_REPO.

Make sure the resources have been added successfully to the git repo for the execution of the test.

Alternative Approach

Users can skip this step and upload to Git directly if they choose. Users need to upload into the {branch-name-which-matches-the-stage} and path {keptn-service-name}/{resourceUri-path}.

For example: Inside the production branch, in the folder microserviceA/job/config.yaml.

Understanding Resources

Config

The config.yaml for service microserviceA looks like:

apiVersion: v2
actions:
- name: "Run k6"
events:
- name: "sh.keptn.event.test.triggered"
tasks:
- name: "Run k6 with Keptn"
files:
- /files
image: "grafana/k6"
cmd: ["k6"]
args: ["run", "--duration", "30s", "--vus", "10", "/keptn/files/k6_test.js"]

K6 docker image is pulled from grafana/k6 and used for execution using the k6 run command. Inside the container, files are placed inside /keptn/ path. So, <resource-uri> in Git becomes accessible from /keptn/<resource-uri>.

Any custom k6 Docker image could be used here, along with k6 binary created using k6 extensions. A common example would be grafana/xk6-output-prometheus-remote. We'll take a look at k6 extensions in the next tutorial.

K6 test

A simple k6 test is used here:

import http from 'k6/http';
export const options = {
thresholds: {
http_req_failed: ['rate<0.01'], // http errors should be less than 1%
http_req_duration: ['p(95)<500'], // 95% of requests should be below 500ms
},
};
export default function () {
http.get('https://test-api.k6.io/public/crocodiles/1/');
}

This k6 test would be used for performance testing.

Trigger Sequence

Let's trigger the sequence using the command:

keptn trigger sequence --sequence testMyService --project k6-jes --service microserviceA --stage production

You can trigger the sequence from Keptn Bridge or using Keptn API too.

Success Trigger

The Sequence has been successfully triggered can be seen by the log of job-executor-service started in the below image:

Success Trigger

K6 Execution Logs

The microserviceA service will execute with a zero exit code, therefore it will be finished successfully. We can view the logs given by the k6 run command.

K6 Execution Logs

Failing a k6 Test

We'll see how the service fails when the k6 performance test fails. In the k6_test.js file, we'll change the threshold from 500ms to 5ms... Forcing it to fail!

export const options = {
thresholds: {
http_req_failed: ['rate<0.01'], // http errors should be less than 1%
http_req_duration: ['p(95)<5'], // 95% of requests should be below 5ms... Force Fail :)
},
};

We'll have to update the resources on Git Upstream. You can do that manually in the production branch or by using this command:

keptn add-resource --project k6-jes --service microserviceA --stage production --resource ./production/microserviceA/files/k6_test_fail.js --resourceUri files/k6_test.js

Here, the k6_test_fail.js will replace the k6_test.js in the Git repo.

Trigger Sequence Again

Let's re-run the Service using the same command to trigger the sequence:

keptn trigger sequence --sequence testMyService --project k6-jes --service microserviceA --stage production

K6 Fail Logs

Due to the change in the k6 test script, now the service microserviceA will execute with a non-zero exit code, hence it will be finished as a failure.

K6 Fail Logs

Debug

Some of the steps which might help in debugging some common issues are:

  • Make sure the resources are in the correct location in the git repo. As reference, you can use jainammm/keptn-k6-jes-tutorial.

  • View the logs of Job Executor Service by using the command:

    kubectl -n keptn-jes logs deployment/job-executor-service -f job-executor-service

See also

This tutorial has shown how to run on Keptn, a standard k6 test with the core k6 functionality. But you can also enrich k6 with other testing protocols or result outputs by incorporating k6 extensions.

Check out the following tutorials that provide instructions for other cases:

< Back to all posts