📖What you will learn
- How to setup and start Apache Kafka and ZooKeeper on your local machine
- How to configure Kafka to handle k6 output
- How to configure a k6 test to send the test output to Kafka
Apache Kafka is an open-source event streaming platform. It can be used to store and process millions of messages per second, making it a good fit for distributed real-time data processing solutions. Although it provides features like stream-processing and horizontal scalability, it remains easy to use: you send your messages to it (produce) and later pick them up (consume) for further processing.
If you want read more about Kafka, I recommend you to have a look at the official documentation.
k6 is an open-source load testing tool. It can produce hundreds of messages per second, as a result of testing a system or platform. These messages can be sent to a number of different destinations, like a JSON file or, as in this case, Apache Kafka.
To be able to send messages from k6 to Apache Kafka, we’d go with a very simple setup, with only one instance. In our scenario, k6 would act as a simple producer of JSON messages and we'll try to consume message on terminal, using the built-in simple kafka consumer. Here I will use the official binary provided by Apache and would leave the docker setup up to you to try. To give you a clue, using Lenses.io's fast-data-dev, you can have a complete Docker environment with Kafka, ZooKeeper, Schema Registry, Kafka-Connect, Landoop Tools and more than 20 connectors in an easy to use package.
Since we're using the binary package, no special installation is needed, and you can directly invoke the binary from the command line. I have used Debian 10.2 to run these commands. Simply follow the instructions below:
Download and extract Apache Kafka
The current version as of this writing is 2.3.0, but you can choose your own desired version.
Starting ZooKeeper and Kafka
As you can see, when you start the ZooKeeper, it will start listening on all interfaces (0.0.0.0) on port 2181.
Now start Kafka server in a new terminal. It will connect to your local ZooKeeper instance on port 2181 and will start listening for new connections on port 9092.
Creating a k6 topic
On another terminal, create a topic, for example k6-output, to receive k6 messages on. This topic would be a single partition topic with no replication.
Executing our test
Running k6 with Kafka output
Using the following script in a file named scenario.js, we’ll start k6 to send its output to Kafka:
To view messages of k6 output on your terminal, use the following built-in consumer:
As you can see, we’ve sent our messages with JSON format to Kafka, so you'll now be able to build a JSON consumer in your own desired programming language (e.g. Python) and start consuming and using the data provided by k6 to Kafka.
There are other possibilities, like using platforms that can import or consume data from Apache Kafka, like Apache Spark or Kafka Connect from Confluent Platform, to name a few. You can also use visualization platforms that has support for Kafka as input. I recommend reading the following articles about it:
- Just Enough Kafka for the Elastic Stack, Part 1 and Part 2
- How the k6 Load Testing Tool is Leveraging Grafana
I hope you've found this article helpful and were able to send your k6 output to Apache Kafka, an industry-proven technology.
If you have any suggestions or comments, I'd really like to hear from you.