⚠️ This tutorial is outdated
The instructions in this tutorial work with v0.8.0 or below.
To use the latest xk6-kafka version, check out the changes on the API documentation and examples.
Recently, k6 started supporting k6 extensions to extend k6 capabilities for other cases required by the community. The community has already built plenty of extensions. k6 extensions are written in Go, and many of them are reusing existing Go libraries.
This makes k6 to be a versatile tool to test different protocols and adapt to multiple cases. This post is the third part of my series of articles testing various systems using k6:
Let's look in this post how we test the popular Kafka project. Apache Kafka is a powerful event streaming platform that provides the following features:
- Write and read streams of events
- Store streams of events for as long as you want to
- Process streams of events in parallel retrospectively
It works by having client applications write events to the Kafka server. We term this type of application as Producers. Client applications which read and process events from the Kafka server are called Consumers.
Kafka itself is capable of handling hundreds to millions of events per second seamlessly on simple setup. But what if you wanted to test and observe how your Kafka service behaves before going live?
The xk6-kafka extension provides some convenient functions for interacting with Kafka Producers and Consumers. It serves as a producer that can send a high volume of messages per second, allowing you to monitor the system under test (SUT) and test how the application will keep up with the load.
At the time of this writing, the xk6-kafka extension provides the following APIs:
|consume(reader, limit)||Consume messages from the Kafka server.|
|createTopic(address, topic)||Create a new topic.|
|listTopics(address)||Return a unique set of topics.|
|produce(writer, messages)||Produce messages to the Kafka server.|
|reader(brokers, topic)||Instantiate a new Reader instance.|
|writer(brokers, topic)||Instantiate a new Writer instance.|
Some of the APIs mentioned above do accept additional optional parameters meant for authentication and message compression. Refer to more examples for additional information.
Building k6 with the kafka extension
By default, k6 does not support testing Kafka. Building k6 with the xk6-kafka extension creates a k6 version with the capabilities to test Kafka producers and consumers.
Make sure you have the following installed and ready before you continue:
- Go (>=1.7)
Next, continue the installation by running the following command in your terminal to install the xk6 module:
Once the command finishes successfully, you can start making your own custom k6 binary for Kafka as follows:
It will take some time for the process to create a new k6 binary in your working directory.
Troubleshooting: If you experienced issues installing xk6 extension, kindly head over to the Github repository to download the pre-compiled binaries instead.
The recommended approach is to use docker as manual installation is quite complicated and prone to errors. You can pull the following image by lensesio from DockerHub. It contains the complete Kafka setup for development.
After that, run the following to start the docker in detached mode:
Visit http://localhost:3030 to get into the fast-data-dev environment.
Continue by appending the following initialization code:
The code will initialize both the writer and reader instances based on the configuration specified. If you are using a different IP/host address and port for your Kafka server, kindly modify it accordingly.
Next, call the createTopic function to create a new topic. Rest assured that this function will do nothing if the topic already exists.
Let’s create a function which generates a random integer as a unique identifier for each message later on. Please be noted that this is optional and not a mandatory requirement to do load testing.
As for the default function, define it as follows:
The code block above works as follows:
- Initialize a list of messages
- Call produce function to publish the messages
- Check if messages are successfully sent
Once you are done with it, create a teardown function and close the connections:
Run the test
Save the file and run the following command on your terminal:
You should see the following output:
Scale the load
You can easily scale the load by increasing the number of vus. For example, the following command uses 500 vus to load test for a minute:
If you are new to k6, check out how to configure the load options in the script or run a stress test with k6.
Extend the test
The script above is all about producing messages to your Kafka server. In fact, you can easily modify the code into a test which produces and consumes messages.
Simply add the following code below the for loop code:
The code will read 10 messages each time. Simply modify the value to something higher if you wish to consume more messages.
The output is as follows when you run it with the same command:
Kafka metrics in k6
By default, k6 has its own built-in metrics which are collected automatically. Apart from that, you can create your own custom metrics. Custom metrics can be categorized into the following types:
- Counter: A metric that cumulatively sums added values.
- Gauge: A metric that stores the min, max and last values added to it.
- Rate: A metric that tracks the percentage of added values that are non-zero.
- Trend: A metric that allows for calculating statistics on the added values (min, max, average and percentiles).
Besides k6, k6 extensions can collect metrics and report them as part of the k6 results output. In this case, xk6-kafka collects individual stats for both reader and writer.
Let’s have a look at the metrics meant for the reader.
|kafka.reader.dial.count||Counter||Total number of times the reader tries to connect to Kafka.|
|kafka.reader.error.count||Counter||Total number of errors occurred when reading from Kafka.|
|kafka.reader.fetches.count||Counter||Total number of times the reader fetches batches of messages from Kafka.|
|kafka.reader.message.bytes||Counter||Total bytes consumed.|
|kafka.reader.message.count||Counter||Total number of messages consumed.|
|kafka.reader.rebalance.count||Counter||Total number of rebalances of a topic in a consumer group (deprecated) .|
|kafka.reader.timeouts.count||Counter||Total number of timeouts occurred when reading from Kafka|
As for the writer, the metrics are as follows:
|kafka.writer.dial.count||Counter||Total number of times the writer tries to connect to Kafka.|
|kafka.writer.error.count||Counter||Total number of errors occurred when writing to Kafka.|
|kafka.writer.message.bytes||Counter||Total bytes produced.|
|kafka.writer.message.count||Counter||Total number of messages produced.|
|kafka.writer.rebalance.count||Counter||Total number of rebalances of a topic (deprecated).|
|kafka.writer.write.count||Counter||Total number of times the writer writes batches of messages to Kafka.|
There are more available kafka metrics, as you can find them here. However, the extension does not collect all metrics yet. You can follow this GitHub issue to track the progress of their additions.
Moreover, the xk6-kafka repository provides a few test scripts that work out-of-the-box for new users. At the time of this writing, it comes with the following tests:
- test_avro.js : Tests Kafka with 200 Avro messages per iteration.
- test_avro_with_schema_registry.js : Tests Kafka with 200 Avro messages per iteration using schema registry
- test_json.js : Tests Kafka with 200 JSON messages per iteration.
- test_json_with_snappy_compression.js : Tests Kafka with 200 JSON messages per iteration using snappy compression
- test_sasl_auth.js : Tests Kafka with 200 JSON messages per iteration and based on SASL authentication.
- test_topics.js : List topics on all Kafka partitions and create random topics.
Feel free to experiment with them and modify the code accordingly based on your own use cases. If you bump into issues, report them on GitHub.
In conclusion, load testing Apache Kafka is now a lot easier with k6. k6 provides the foundation to create and scale your load tests, and the xk6-kafka extension brings a convenient API to interact with a Kafka server.
If you wish to find out more on other available k6 extensions, simply head over to the bundle builder page. The page also allows you to generate the corresponding command for building your own custom k6 binary.
If you have any questions or are interested in building an extension, join the k6 community on Slack.