Introducing browser automation and end-to-end web testing with k6

We’re excited to launch xk6-browser at Grafana ObservabilityCON today, an extension to k6 adding support for browser automation via the Chrome Devtools Protocol (CDP).

k6 was built because we weren’t satisfied with the developer experience offered by existing load testing solutions. We believe in shifting performance testing, and with it observability, to the left. We promote a proactive approach to building and operating reliable systems, uncovering issues before they’re released to the end user. We encourage the inclusion of load tests in your CI pipeline to prevent system updates from introducing performance regressions.

This public beta release of xk6-browser extends the possible use case beyond what is supported by the existing protocol-level capabilities. From a test authoring and scripting perspective, this means that you can script in terms of user actions (such as navigations, mouse and keyboard actions, taking screenshots) rather than by making HTTP requests, WebSocket messages, etc. This has advantages, improving the developer experience in use cases where scripting using protocol-level APIs would be cumbersome, fragile and difficult to maintain.

With xk6-browser, you can interact with the browser to test your web applications end-to-end while accessing all of the k6 core features, including protocol-level APIs and other k6 extensions. It's a single tool for both protocol and browser-level testing.

Why browser testing?

Well, for starters it's been the most requested feature for years. We had resisted adding browser-level testing support to k6 as we wanted to focus on testing backend systems first, making sure we provided a good developer experience for those use cases before tackling other areas of testing.

The golden rule of web performance states that 80-90% of the load time of a web page or application is spent in the frontend. It thus makes sense to start optimizing performance in the frontend. Our counter argument to this has always been that this ratio between load time spent in backend vs frontend very quickly can flip when a backend system approaches its concurrency limit (as illustrated by the graphic below). In our experience it's very common that teams aren't aware of what their system's concurrency limit is and thus have no idea of when the flip will happen.

xk6-browser allows for mixing browser-level and protocol-level APIs. You can now simulate the bulk of traffic with protocol-level scenario(s) as usual, and, at the same time, have a Virtual User interacting with your web site or application using a real browser to collect frontend metrics (DOM content loaded, load, first contentful paint, etc.). You can finally understand both sides of the coin with k6!

How to get started

We have initially built this browser-level capability as a k6 extension. This allows us to experiment without changes to the k6 codebase. To use browser-level APIs in your k6 test, you need a k6 binary built with the xk6-browser extension. You can either download one of the release binaries from GitHub, or build a custom version of k6 from source.

To build from source, first ensure you have the prerequisites:

• Go toolchain
• Git

Then:

1. Build k6 version.

   # Install xk6
   go install go.k6.io/xk6/cmd/xk6@latest
   # build k6 binary
   xk6 build v0.35.0 --output xk6-browser --with github.com/grafana/xk6-browser
   
   ... [INFO] Build environment ready
   ... [INFO] Building k6
   ... [INFO] Build complete: ./xk6-browser

   Copy

   xk6 creates the xk6-browser binary in the current working directory. This file can be used exactly the same as the main k6 binary, with the addition of being able to run xk6-browser scripts.

2. Create a k6 test and import the k6/x/browser module.

   import launcher from "k6/x/browser";
   
   export default function() {
      const browser = launcher.launch('chromium', { headless: false });
      const context = browser.newContext();
      const page = context.newPage();
   
      // Goto front page, find login link and click it
      page.goto('https://test.k6.io/', { waitUntil: 'networkidle' });
      const elem = page.$('a[href="/my_messages.php"]');
      elem.click();
   
      // Enter login credentials and login
      page.$('input[name="login"]').type('admin');
      page.$('input[name="password"]').type('123');
      page.$('input[type="submit"]').click();
   
      // Wait for next page to load
      page.waitForLoadState('networkidle');
   
      page.close();
      browser.close();
   }

   Copy

3. Run the k6 test using the xk6-browser binary created on the first step.

   ./xk6-browser run browser-test.js

   Copy

   When xk6-browser launches and interacts with the browser, it will automatically collect frontend metrics and report them as with other k6 metrics.

           /\      |‾‾| /‾‾/   /‾‾/   
      /\  /  \     |  |/  /   /  /    
     /  \/    \    |     (   /   ‾‾\  
    /          \   |  |\  \ |  (‾)  | 
   / __________ \  |__| \__\ \_____/ .io
   
   execution: local
      script: browser-test.js
      output: -
   
   scenarios: (100.00%) 1 scenario, 1 max VUs, 10m30s max duration (incl. graceful stop):
            * default: 1 iterations for each of 1 VUs (maxDuration: 10m0s, gracefulStop: 30s)
   
   running (00m03.3s), 0/1 VUs, 1 complete and 0 interrupted iterations
   default ✓ [======================================] 1 VUs  00m03.3s/10m0s  1/1 iters, 1 per VU
   
      browser_dom_content_loaded.......: avg=202.36ms min=166µs    med=202.36ms max=404.56ms p(90)=364.12ms p(95)=384.34ms
      browser_first_contentful_paint...: avg=409.48ms min=409.48ms med=409.48ms max=409.48ms p(90)=409.48ms p(95)=409.48ms
      browser_first_paint..............: avg=409.48ms min=409.48ms med=409.48ms max=409.48ms p(90)=409.48ms p(95)=409.48ms
      browser_loaded...................: avg=202.28ms min=4.54ms   med=202.28ms max=400.03ms p(90)=360.48ms p(95)=380.25ms
      http_req_connecting..............: avg=103.8ms  min=0s       med=0s       max=519ms    p(90)=311.4ms  p(95)=415.19ms
      http_req_duration................: avg=370.62ms min=184.01ms med=293.7ms  max=864.18ms p(90)=638.38ms p(95)=751.28ms
      http_req_receiving...............: avg=238.8ms  min=181ms    med=222ms    max=292ms    p(90)=291.2ms  p(95)=291.6ms 
      http_req_sending.................: avg=0s       min=0s       med=0s       max=0s       p(90)=0s       p(95)=0s      
      http_req_tls_handshaking.........: avg=75ms     min=0s       med=0s       max=375ms    p(90)=225ms    p(95)=299.99ms
      http_reqs........................: 5 1.531592/s
      vus..............................: 1 min=1      max=1
      vus_max..........................: 1 min=1      max=1
   
   ...

   Copy

With xk6-browser, k6 will now report the following list of new frontend metrics:

• browser_dom_content_loaded: Time from start of document load until DOMContentLoaded event is fired.
• browser_loaded: Time from start of document load until load event is fired.
• browser_first_paint: Time from start of document load until first paint event happens.
• browser_first_contentful_paint: Time from start of document load until first contentful paint happens.
• browser_first_meaningful_paint: Time from start of document load until first meaningful paint happens.

xk6-browser API

The xk6-browser API aims for rough compatibility with the Playwright API for NodeJS, meaning k6 users don't have to learn an entirely new API.

Note that because k6 does not currently run in NodeJS or support the Event Loop. xk6-browser APIs are synchronous and can slightly differ from their Playwright counterpart.

For browser support, the initial focus is on providing solid support for Chromium-based browsers. Over time we want to support Firefox and Webkit-based browsers as well.

What's next?

The team is working on project stability and reliability. End-to-end tests are sometimes known to be flaky and unreliable, producing failures and inconsistent results. In that case, the testing process can become very inefficient. Reducing flakiness is a primary goal, so you can be confident in your testing.

We’re adding more APIs and looking forward to feedback from users to aid us in prioritizing what to build next! Join the #xk6-browser channel on the k6 Slack and let us know what you think.

Learn more

You can find out more by reading the documentation and GitHub repository. If you get stuck or have any questions for the team, please get in touch with us on the k6 Forum, GitHub, or Slack.

Special acknowledgement to the authors of Playwright and Puppeteer. This project is heavily influenced, and in some regards, based on their code.