Prometheus remote write

Experimental module, use at your own risk

While we intend to keep this module as simple and stable as possible, we may need to add features or introduce breaking changes. This could happen at any time until we release this module as stable.

Feel free to provide user feedback, and open an issue or pull request if you have any suggestions.

Prometheus remote write is a protocol that makes it possible to reliably propagate data in real-time from a sender to a receiver. It has a defined specification and multiple implementations. For example, you can store the metrics in Prometheus. For other implementations, check the Prometheus Integrations guide.

With the Prometheus remote write output, k6 can send test-result metrics to a Prometheus remote write endpoint. The output during the k6 run execution gets all the generated data points for the built-in k6 metrics. It then generates the equivalent Prometheus remote write time series.

note

If you want to export cloud results to remote write, refer to Cloud to Prometheus RW.

About metrics mapping

All k6 metric types are converted into an equivalent Prometheus remote write type:

Trend metric conversion

By default, k6 trend metrics convert to primitive counters and gauges. To convert trend metrics to high-fidelity histograms, you can use Prometheus Native histogram instead.

Because k6 can't easily determine fixed buckets in advance, k6 metrics can't export to the original Prometheus histograms. So, the output maps a trend metric into primitive counter and gauges. Each value represents a math function (count, sum, min, max, avg, med, p(x)). This mapping has following drawbacks:

• It is impossible to aggregate some gauge values (especially percentiles).
• It uses a memory-expensive k6 data structure.

To resolve these limitations, you can map a trend as a Prometheus Native Histogram. To do this, you can use the K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM=true environment variable (or one of the other ways). The output then converts all the trend types into a dedicated Native Histogram.

Mapping trends as native histograms gives you more efficient storage and more precise queries. The drawback is that the feature is experimental, released in Prometheus v2.40.0.

note

Native Histogram is an experimental feature, so it has to be enabled (--enable-feature=native-histograms). Other remote write implementations might not support it yet.

Naming conventions

The output maps the metrics into time series with Name labels. As much as possible, k6 respects the naming best practices that the Prometheus project defines:

• All time series are prefixed with the k6_ namespace.
• All time series are suffixed with the base unit of the sample value (if k6 knows what the base unit is).
• Trends and rates have the relative suffixes, to make them more discoverable.

Stale trend metrics

This k6 output can mark the time series at the end of the test as stale. To enable the stale marker option, set the K6_PROMETHEUS_RW_STALE_MARKERS environment variable to true.

By default, the metrics are active for 5 minutes after the last flushed sample. They are automatically marked as stale after. For details about staleness, refer to the Prometheus docs.

Send test metrics to a remote write endpoint

Prometheus as remote write agent

To use remote write in Prometheus 2.x, use the --web.enable-remote-write-receiver flag. The xk6 extension repository has some docker compose examples in the example directory. For remote write storage options, refer to the Prometheus docs.

To send k6 metrics to a remote write endpoint, follow these steps:

1. Set up a running remote write endpoint with an endpoint that k6 can reach.

2. Run your k6 script, using the --out flag with experimental-prometheus-rw as the argument:

   k6 run -o experimental-prometheus-rw script.js

   Copy

All the time series have a k6_ prefix. In the Metric Explorer UI in Grafana, it looks something like this:

Authenticate

If the remote write endpoint requires authentication, the output supports the HTTP Basic authentication and it can be used with the following command:

    K6_PROMETHEUS_RW_USERNAME=foo \
    K6_PROMETHEUS_RW_PASSWORD=bar \
    ./k6 run script.js -o experimental-prometheus-rw

Feel free to request more authentication methods or provide your experience in the issue tracker.

Options

k6 has special options for remote write output.

Time series visualization

To visualize time series, you can use Grafana via explorer, importing the pre-built official dashboard or create a custom version. You can visualize them from Grafana Cloud integration or in a self-hosted instance.

If you want a local option, the xk6 extension repository includes a docker-compose setup with two pre-built Grafana dashboards. You can use these dashboards to visualize the generated time series with Prometheus configured as a data source:

• visualize the results of a test run
• list test runs

Docker compose example

Clone the repository to get started and follow these steps for using the docker-compose.yml file that starts Prometheus and Grafana:

note

The docker-compose.yml file has the Native Histogram mapping set as enabled.

1. Start the docker compose environment.

docker compose up -d prometheus grafana

# Output
Creating xk6-output-prometheus-remote_grafana_1     ... done
Creating xk6-output-prometheus-remote_prometheus_1  ... done

2. Use the k6 Docker image to run the k6 script and send metrics to the Prometheus container started on the previous step. You must set the testid tag as a wide test tag with a unique identifier to segment the metrics into discrete test runs for the pre-built Grafana dashboards.

K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM=true \
  k6 run -o experimental-prometheus-rw --tag testid=<SET-HERE-A-UNIQUE-ID> ./samples/http_get.js

testid can be any unique string that let you clearly identify the test. For convenience, a docker-run.sh file is available with a time-based testid, and it can be simply:

docker-run.sh ./samples/http_get.js

3. Visit http://localhost:3000 to view results in Grafana.