Load Testing Your API with Swagger/OpenAPI and k6

📖What you will learn

• How to generate a load test from a OpenAPI specification
• How to run the load test generated by the converter
• How to further elaborate on the generated script

Overview

Throughout the years, there has been many attempts to devise a universal format for defining Web API specifications. The objective was (and still is) to help stakeholders of the system to work with those APIs, without having access to the source code. Each new "universal" format came with the promise of being ubiquitous and all-encompassing, but eventually faded away due to various reasons, like OData and WSDL.

One of the major players in this field is Swagger, developed by Tony Tam in 2011, which later was hired by the SmartBear Software, effectively acquiring the rights to the software. In 2015, the same company created a new organization under the sponsorship of the Linux Foundation and called it the OpenAPI Initiative. The new initiative has renamed the Swagger specification to OpenAPI Specification (OAS) in 2016. This specification is generally used for developing, interacting and documenting APIs.

There are many use-cases to OpenAPI. The first is to use it as a documentation for your APIs. Usually it is generated from the request handlers along with the schema for your database models and fields. The second is to use it as a means to generate server-side code, that is the actual API. This is particularly useful for those who want to go spec-first, rather than code-first. The third is to use as a means to generate client-side code. With this method, you'll have stubs generated for all your requests, which can be used to make use of or even test your APIs.

Overall, the OpenAPI specification and the openapi-generator project can help you generate a lot of integrations out of the box from your API specification. Imagine you could have a specification document that can help you generate your server-side code to serve API endpoints, your client-side code for testing and the documentation of your API for developers and testers.

Swagger/OpenAPI load testing

Formerly, the idea of Swagger/OpenAPI load testing was taken less seriously by the developer community. It was due to the fact that only QA people used to work on performance testing. k6 takes load testing to a whole another level by letting developers write their own load test scripts in JavaScript. With the introduction of new tools, like k6, and then the widespread use of Swagger/OpenAPI for API design and documentation, we felt the need that tools for transforming Swagger/OpenAPI specifications to scripts to be used for performance testing should exist.

We thought it would be a good idea to have a tool to generate a load test script out of an OpenAPI specification document. This new generator will then help you easily integrate load testing in your infrastructure. Therefore, we have added a new k6 generator to the openapi-generator project.

By using this tool, as a Swagger/OpenAPI test generator, you can now generate your k6 load test scripts using the same specification documents you used for your APIs. We tried our best to convert almost all the specification to k6 script, but the generated script should still be modified to make it runnable and suitable for your use case.

Once you have a working script, you can easily use it in your Continuous Integration (CI) platform to automate load testing. You can also use Grafana Cloud k6, that gives you the ability to run your load tests in a distributed environment from various regions, plus giving you detailed insights on the results of your test(s).

API load testing with Swagger/OpenAPI specification

We have written a guide for API load testing and since OpenAPI is concerned with APIs, we advise you to take a look at the guide. It will help you understand why you should load test your APIs. The guide gives you an overview of different approaches to API load testing. It includes various ways to create load test scripts, from writing one yourself, to using your existing Postman collections or HAR files and converting them to load test scripts. It also provides you with considerations you need to know while load testing your APIs.

Generating load test script from Swagger/OpenAPI specification

There are various ways to install the openapi-generator. It is distributed in source code, so that you can build it yourself. You have lots of options to use binaries for your operating system. We recommend using the Docker image, which is going to be explained in this article.

I assume you have docker installed, otherwise have a look at the installation instructions for your operating system. The following commands pulls the Docker image for you. You can also omit this command in favor of the next one, because it pulls the latest image for you, if you don't have it on your machine.

$ docker pull openapitools/openapi-generator-cli

The next step is to run the command inside a container to convert the OpenAPI specification file for you. Assuming you are running Docker on Linux-based operating systems, the following command mounts your present working directory (e.g. /home/user/Desktop) to /local on the container. Also note that after conversion, the container is removed. The container runs the command inside it with the generate option, along with its parameters.

$ docker run --rm -v "${PWD}:/local" openapitools/openapi-generator-cli generate \
    -i https://httpbin.test.k6.io/spec.json \
    -g k6 \
    -o /local/k6-test/ \
    --skip-validate-spec

With the above command, we've converted the API specification document of the httpbin.org instance, that is hosted on our servers. The parameters are:

• The -i parameter is used to feed the specification document to the generator. You can use a file for that purpose. This way you should make sure that it resides in the same directory on your host (/home/user/Desktop/myspec.json), and you address it from the container side: /local/myspec.json. You also have the option to specify a URL that points to a file. Note that the file can be in either JSON or YAML format.
• The -g parameter specifies the generator to use, that is k6.
• The -o parameter specifies the directory to store the generated files. After conversion, the files can be accessed from /home/user/Desktop/k6-test/ on your host.
• In case you get a lot of errors from validation, just skip them with --skip-validate-spec. These errors are often related to differences between Swagger and OpenAPI specification.

Load testing using the generated script

The script should definitely be cleaned up after conversion. The original script is too long to fit into this article, thus the following example is an snippet of the generated script. Extra comments and unneeded code has been removed for demonstration purposes and the n variable is initialized.

import http from 'k6/http';
import { group, check, sleep } from 'k6';

const BASE_URL = 'https://httpbin.org/';
const SLEEP_DURATION = 0.1;

export default function () {
  group('/absolute-redirect/{n}', () => {
    const n = 2;
    const url = BASE_URL + `/absolute-redirect/${n}`;
    // Request No. 1
    const request = http.get(url);
    sleep(SLEEP_DURATION);
  });
}

Running the above script has produced the following output:

Considerations for the generated script

Take these into consideration while converting your OpenAPI specification documents to k6 scripts:

• The order of the requests follow the order in the specification document and may not always be correct. Replace them to match your request/response flow. For example, you first create a resource with POST, then you read it with GET.
• All requests belonging to the same path are grouped together with the k6 group feature.
• A global sleep duration is defined with the SLEEP_DURATION variable and for each request, the value is applied using the k6 sleep function. Change them to match the delay you need between the requests.
• Make sure the BASE_URL is correct.
• Initialize all variables that their value starts with "TODO_EDIT_THE_...".
• Body parameters are in the form of { variable: "datatype" }. Change the datatype to the desired value. Nested body parameters follow the same pattern.
• For file upload endpoints that use multipart requests, replace the file name with your own.
• You may encounter undefined variables being used. Just add them manually. It may be due to differences between OpenAPI and Swagger specifications.
• We only use checks for the default and 200 OK responses for now. It is up to you to check for the rest.
• If you found any issues, please file it on issues.

Conclusion

The k6 generator for OpenAPI was written to onboard users to k6. The tool is going to help users quickly generate a load test script out of their existing Swagger/OpenAPI specification documents. This auto-generation of the load test script will help streamline the API testing process, keeping on par with the latest changes to their APIs and specifications.