Comparing LoadRunner VS k6

At k6, we get frequent requests to compare k6 vs. LoadRunner, and since I am an experienced user with LoadRunner (even certified), I will try to put both tools head to head.

But comparing these tools is difficult as their components do not match 1:1. Each is a robust ecosystem with different parts, some doing multiple things that make straight comparisons more complicated. And on top of that, we constantly get new features that match up with many other components.

To ease things, let's start describing each as a platform and as they are at the moment of writing this post (beginnings of 2023).

LoadRunner Suite overview

LoadRunner is an old solution (in internet years), born in 1993. It has been growing and developing since then. Nowadays, it has evolved into four main variations of the solution with shared components and unique components/capabilities. Let's look at them.

LoadRunner Professional (LRP)

• Original LoadRunner solution for distributed load testing
• On-premise desktop solution based on three core components:
  • Virtual User Generator (VUGen)
    • IDE for generating and editing scripts.
  • Load generator
    • Generates the load by following the script.
  • Controller
    • Control and launch load generator instances to execute load tests.
    • Set up agents to enable machines to be controlled and generate load.
    • Advanced scenario creation features.
    • Monitors hardware of the solution under test.
    • Displays live test results and consolidates results and measurements.
  • Analysis tool
    • It analyzes raw results produced by the controller.
    • It helps find correlations between events and metrics.
    • Can generate reports in multiple formats.

LoadRunner Enterprise (LRE)

• LRP as an on-premise and cloud-based web solution, plus:
• Enables teams to upload scripts, design scenarios, schedule, execute, etc. within a shared repository
• Displays a central dashboard where teams can view live results, analyze past results, and generate test result reports.

LoadRunner Cloud (LRC)

• LRE as a SaaS solution
• Allows the use of cloud and on-premise load generators

LoadRunner Developer (LRD)

• LRD is a free (for up to 50 virtual users) solution independent of LRP, LRE, and LRC. For distributed test executions, testers must upload scripts to other LR solutions (sold separately).
• Scripting is in JavaScript, enabling work on any text editor.
• It is oriented at devs to execute from the command line.
• It does not include scenario design inside of the script.
• Uses Grafana for visualizing results of local tests.

NOTE: This article compares each solution as a whole. It will delve little into specific components and suites, especially LRD, as at the moment of writing this (beginnings of 2023) is relatively new, the documentation still seems like WIP, and it doesn't have significant adoption. Stay tuned as an LRD vs. k6OSS comparison is in the backlog.

Grafana k6 overview

The k6 solutions bring a tone of communities and collaborative software. It is lean, open-source, and focuses on breaking adoption and sharing barriers. It brings a different spirit to the testing realm, making it an evolving entity that moves with the community.

k6 OSS (Open Source Software)

k6 OSS is a standalone open-source solution that you can easily download, install and execute from any computer.

• It is a lightweight, developer-oriented, and super extensible application for performance, load testing, and even SRE tasks.
• Scripts developed using JavaScript on any text platform.
• It can reuse script sections as functions in other scripts.
• Lean solution with blazing performance.
• Natively supports HTTP 1 & 2, WebSockets, and gRPC.
• Increases support through org/community-built extensions (browser automations, results outputs, queue protocols, Kubernetes, tracing, chaos, etc.)
• Easy to integrate into CI/CD platforms through powerful command line options.

Grafana Cloud k6

The cloud version of OSS makes it super easy to execute larger tests (no LGs hassle), distribute them worldwide, schedule them in a snap, and store results without managing them all by yourself.

• k6 SaaS solution.
• It removes local dependencies and maintenance, like the load generators.
• Run scripts/tests in the cloud.
• Initiate executions locally or upload and run them on the web platform.
• Create scripts and scenarios through the web interface, even "codeless."
• Schedule test executions.
• Analyze test data.
• Trigger Alerts on thresholds.
• Publicly share results.
• Much more!

k6 operator, browser, extensions, Grafana, and more!

The k6 ecosystem consists of many other pieces, is open source, is constantly growing, extending, and provides multiple capabilities:

• Community-oriented: They allow communities to contribute, add, change, and extend k6.
• A vast set of extended functionalities (and growing.)
• Automate browser sessions through xk6-Browser.
• Manage and run Kubernetes services with xk6-kubernetes.
• Use a result repository such as Prometheus and visualize results with Grafana.
• And much more. Check extensions.

Comparing the tools

As these tools have different components, a 1-to-1 comparison is impossible. At the very least, it would be unfair. The comparison points below will focus on each ecosystem's philosophies, principles, and intended approaches from a broad perspective.

This analysis is my personal and non-respectable opinion on what each solution has in its guts, soul, and purpose. Take this info with a pinch of salt.

Birthplaces

Applications and solutions get most of their philosophies and foundations from the day and age when they were born (created). Let's look into the main differences between them.

LoadRunner

• Born in the waterfall days and intended for phased projects (1993.)
• Created for testers, the only ones who know/can use it.
• Heavy dependence on the LR GUI suite (Vu Gen, Controller, analysis, etc.)
• It focuses on the GUIs to access and work on tests.
• Scripts created by recorder in the GUI in ANSI C.
• Limits access to platforms, tools, and some functions with licenses.

k6

• Born for agile, continuous, and modern projects (2017.)
• Accessible to the whole team, especially developers and testers.
• Independent of a GUI and usable for anyone with basic JavaScript knowledge and any text editor.
• Intended to be triggered and worked from the command line.
• It is Open Source, oriented at communities, and with all features accessible as is.

Automation types

Teams can do load and performance automations at different application levels and through different methods. Some depend on long and complex front-end automations, and others divide them granularly with protocol requests. These differences change each tool's efficiency, speed, complexity, and capabilities.

LoadRunner

• Designed to record front-end steps in diverse front-ends of monolithic solutions.
• Oriented at recording long sets of steps per script/test case.
• Complex scenarios that mix the above-mentioned long scripts.
• Useful for monolithic and front-end heavy solutions without service tiers.
• It can be used on service tiers but excels at the above-mentioned.
• Used after development finished to reverse engineer front-end to back-end communication.

k6

• Tool designed for modern solutions based on (micro)service and API architectures.
• The web recorder is not the intended first choice, as modern teams list services, and their utilizations, enabling devs to automate.
• Automations focused on working in minor frequent service releases rather than a long set of steps and multiple functionalities at once.
• It can automate long sets of steps but excels in those mentioned above.
• Designed for both quick-continuous runs in pipelines and massive-complex load tests.

Ease of use

How easy is it to work with the tool? We define that ease by things such as: start using it, downloading it, being able to open it, learning it, and enabling team members–that includes sharing any work related to the automations such as creation, maintenance, tuning, and triggering executions.

LoadRunner

• VuGen is (still) the cornerstone of the solution, but multiple components must be installed to design, run, and analyze a test.
• Users must know how to use VuGen to create, update, and tune scripts.
• Team members must have access, licensing, and knowledge of the LR Controller or the web platform to design/execute tests.
• Only one execution at a time through the controller and LRE, LRC
• It requires knowledge of the GUI, programmatic knowledge, technical skills, performance understanding, and basic code development skills.

k6

• Anyone can create or update, as there's no need for heavy or exclusive GUIs.
• Scripts and scenarios are in plain and understandable JS code.
• Anyone with basic CL and JS skills can start tests from any command line.
• Multiple teams can use it at the same time on different independent executions.
• It requires programmatic knowledge, technical skills, an understanding of performance practices, and basic code development skills.

Collab and reuse

Asides from the ease of use, teams must collaborate on their performance automations. Collaboration happens through repositories, version control systems, and file shares. Shared code among multiple automations can also diminish rework on widespread tasks/commands.

LoadRunner

• Scripts and scenarios consist of long lists of diverse files.
• Only users with VuGen and Controller can work and run scripts/scenarios.
• Executing above 50 virtual users requires licensing.
• Teams must share several files to collaborate on scripts and scenarios.
• Complex to manage in repositories.
• Teams can use the LRE platform to (transparently) share and collaborate.

k6

• Everyone in the team can work/share scripts and scenarios in a single file, especially developers.
• Everyone with a text editor and k6 installed can create, update and run tests/scenarios.
• No license needed for executions.
• Scripts/scenarios consist of more than one file only when there are external dependencies or if that is the intended use.
• Easy to update, version, manage and edit in repositories.
• Teams can use any versioning or repository platform to (easily) share and collaborate.

Command Line

CL or command line executions have become the backbone of agile teams and CI/CD. It is crucial to have an elegant triggering mechanism in a lean and fast way. Teams must integrate CL tests for multiple processes, such as compilations, builds, releases, and test automations.

LoadRunner

• Most indications are to use and execute using GUIs.
• It has CL options, but there's not much emphasis on it.
• Small CL executions are not frequent.
• The general indication is to trigger long-lasting tests in the GUI manually.

k6

• Designed to be run from the command line (even for k6 Cloud.)
• Multiple options and configurations from the CL.
• Test executions boot quickly from the CL and are easy to integrate into other processes.
• The CL execution provides quick result values for CI Pipelines.

Which protocols/features?

The means to automate diverse processes are better known as protocols. These means cover a wide range, from mimicking actual clicks on coordinates on the screen, interacting with objects in the applications, using owned mechanisms, forging messages, and even triggering processes inside the code itself. Let's see how each tool fares and the impacts of each added feature.

LoadRunner

• Can automate a massive list of "protocols" (even many outdated ones.)
• This large capacity makes the software weighty and bulky (huge installer and HW requirements.)
• Most do not need that many protocols nor use them all at once.
• Has a browser based test feature called TruClient which enables teams to automate browser tests.
• It is almost the only choice for specific protocols (SAP GUI, RDC, Citrix, etc.)

k6

• Out of the box supports the most popular protocols: HTTP 1 & 2, WebSockets, and gRPC.
• Focusing on the most common protocols makes it fast, lean, and installable in a snap.
• Specific protocols can be extended to only use the ones needed, keeping it lean.
• Extendable with xk6-browser to support front end browser based automations.
• It does not support specific thick client protocols (yet), such as SAP GUI, RDC, Citrix, etc.
• Other protocols are supported by third-party extensions (SQL, Kafka, ZeroMQ, Redis, and more).

Communities collaboration

The collaboration and support from and for communities will differ depending on the tool's licensing and functionality. Allowing or limiting the support and contributions from the community has different impacts. What does each tool do in this area?

LoadRunner

• LoadRunner is not open to community contributions.
• The vendor must release new features.
• Tight vendor control on new features.
• It suggests greater release control.

k6

• Open-source solution available for communities.
• Extensible by the k6 organization and the community.
• Thanks to the collaborations, it can be tailor suited for/by each organization, community, and team.
• Quality control on modules done by the organizations and the communities.

Sharing results

Organizations have different philosophies on how they want to treat performance test results (live or post.)

Some organizations want to gate-keep them for the testing team to own and provide interpretations, while others prefer to share them everywhere and with everyone. How does each tool handle these?

LoadRunner

• Teams must view live results only on LR platforms (Controller, LRE, and LRC.)
• Natively stores/work execution results in the execution machines (Controller, LRE, LRC.)
• The format of native test results is specific to the controller, allows limited changes, and is stored in multiple files.
• The vendor recently implemented results output to InfluxDB and JSON through an exporter process after runs.
• Teams require the controller or the analysis tools to view results and generate reports.
• Access to test results and native reports is limited to licensed users accessing the platforms or the tool.
• Troubleshooting performance defects is done after executions and interacting with the tool to analyze results.

k6

• Natively can output results to multiple possible destinations.
• The solution can share results to time series databases after completing runs or live during executions.
• Results are in plain, understandable formats.
• Supported output destinations include JSON, CSV, Prometheus, InfluxDB, and many more.
• Once results are stored, anyone and any platform in the organization can view & analyze them on multiple platforms (Grafana recommended.) Either live or after executions.
• Troubleshooting performance defects can be done live and by anyone accessing results

Test Scenarios

Scenarios are the behavior patterns of a performance/load test. They define elements like the distribution of processes among virtual users, the duration, the frequency, the number of iterations, network latency, changes in the patterns, and many more. They can be hyper-realistic or test specific patterns. How do our tools behave here?

LoadRunner

• It has excellent visual design capabilities for load scenarios.
• The controller, LRE, and LRC display load scenario progress as they run.
• Some load patterns must be manually defined (script pacing to reach desired throughputs.)
• Powerful scenario design capabilities, but may lead to excessive complexities (best practices suggest keeping load tests simple.)

k6

• Scenario design is codified and designed through executors.
• Executors provide great ways to reach load patterns automatically.
• Some may find it hard to visualize scenario patterns from the code.
• Better for simple load test patterns (best practice and recommended for CI/CD and agile.)

Where/when are they great?

Enough comparisons. As you may have noticed, it is hard to say which tool is "better" as every organization and team has different preferences and requirements. Instead, why don't we mention where each tool is unquestionably better?

LoadRunner

• Automating niche protocols such as SAP GUI, Citrix, RDP, FTP, POP, etc.
• Visualizing and creating load scenarios, even considerably complex ones.
• When teams and projects are not pursuing/achieving continuous production integration.
• Isolated testing teams from development teams who execute tests as a separate phase.
• When there is (absolutely) no other way to automate web platforms than using TruClient.
• Teams and organizations reliant on integrations with other MicroFocus/OpenText testing solutions.
• For teams looking for voluminous all-in-one solutions.

k6

• Lean automations for popular protocols such as APIs, HTTP, gRPC, WebSockets, and more through extensions.
• Collaboration among teams where Developers and Testers work/share together following Agile principles.
• Increases productivity by detaching teams from automation GUIs, enabling everyone in the team to create/update/execute/share/version performance tests.
• Ease of work and integration with code repositories.
• A lean tool that integrates well in CI/CD performance/load tests without becoming bottlenecks slowing down pipelines.
• Natively shares test metrics (live and post) to multiple platforms and people in the organization.
• Open source solution, enabling team members, community support and extensions.
• Load tests as code and shift-left approaches to testing and development.

Conclusion

In this comparison, we presented that each tool has different characteristics that can be desired or avoided by diverse teams and organizations. If you are asking for a "winner," there is no absolute answer here; as usual, the true answer is "it depends." 

Every project is different regarding needs, culture, processes, and much more. There is no right or wrong choice here; I would only recommend trying to pick what best fits your needs. You may even need to pick both solutions or more!

I am sorry if I disappoint anyone looking for tables, one-to-one comparisons, winners, and losers. I hope I have shown that each tool has elements different enough, making it pointless and even soul-crushing to try to match them on fairgrounds.

You may like to point out that some tools are developing into realms they did not work before. Even though this may be true, the tool's spirit is among the lines I have analyzed above in my personal and non-respectable opinion.

At the very least, I hope this was useful for you.

Knowing how rare it is, thank you for reaching the end of this post. I really appreciate it.

Leandro out!

-Besos <3

Leandro/SrPerf