Data Parameterization

Scripting examples on how to parameterize data in a test script. Parameterization is typically necessary when Virtual Users (VUs) will make a POST, PUT, or PATCH request in a test.

Parameterization helps to prevent server-side caching from impacting your load test. This will, in turn, make your test more realistic.

warning!
Each VU in k6 is a separate JS VM so in order to not have multiple copies of the whole data file SharedArray was added. It does have some CPU overhead in accessing elements compared to a normal non shared array, but the difference is negligible compared to the time it takes to make requests. This becomes even less of an issue compared to not using it with large files, as k6 would otherwise use too much memory to run, which might lead to your script not being able to run at all or aborting in the middle if the system resources are exhausted.
For example, the Cloud service allocates 8GB of memory for every 300 VUs. So if your files are large enough and you are not using SharedArray, that might mean that your script will run out of memory at some point. Additionally even if there is enough memory, k6 has a garbage collector (as it's written in golang) and it will walk through all accessible objects (including JS ones) and figure out which need to be garbage collected. For big JS arrays copied hundreds of times this adds quite a lot of additional work.
A note on performance characteristics of SharedArray can be found within its API documentation.

From a JSON file

1{
2  "users": [
3    { "username": "test", "password": "qwerty" },
4    { "username": "test", "password": "qwerty" }
5  ]
6}

1import { SharedArray } from 'k6/data';
2// not using SharedArray here will mean that the code in the function call (that is what loads and
3// parses the json) will be executed per each VU which also means that there will be a complete copy
4// per each VU
5const data = new SharedArray('some data name', function () {
6  return JSON.parse(open('./data.json')).users;
7});
8
9export default function () {
10  const user = data[0];
11  console.log(data[0].username);
12}

From a CSV file

As k6 doesn't support parsing CSV files natively, we'll have to resort to using a library called Papa Parse.

You can download the library and import it locally like this:

1import papaparse from './papaparse.js';
2import { SharedArray } from 'k6/data';
3// not using SharedArray here will mean that the code in the function call (that is what loads and
4// parses the csv) will be executed per each VU which also means that there will be a complete copy
5// per each VU
6const csvData = new SharedArray('another data name', function () {
7  // Load CSV file and parse it using Papa Parse
8  return papaparse.parse(open('./data.csv'), { header: true }).data;
9});
10
11export default function () {
12  // ...
13}

Or you can grab it directly from jslib.k6.io like this.

1import papaparse from 'https://jslib.k6.io/papaparse/5.1.1/index.js';
2import { SharedArray } from 'k6/data';
3
4// not using SharedArray here will mean that the code in the function call (that is what loads and
5// parses the csv) will be executed per each VU which also means that there will be a complete copy
6// per each VU
7const csvData = new SharedArray('another data name', function () {
8  // Load CSV file and parse it using Papa Parse
9  return papaparse.parse(open('./data.csv'), { header: true }).data;
10});
11
12export default function () {
13  // ...
14}

Here's an example using Papa Parse to parse a CSV file of username/password pairs and using that data to login to the test.k6.io test site:

1/*  Where contents of data.csv is:
2username,password
3admin,123
4test_user,1234
5*/
6import http from 'k6/http';
7import { check, sleep } from 'k6';
8import { SharedArray } from 'k6/data';
9import papaparse from 'https://jslib.k6.io/papaparse/5.1.1/index.js';
10
11// not using SharedArray here will mean that the code in the function call (that is what loads and
12// parses the csv) will be executed per each VU which also means that there will be a complete copy
13// per each VU
14const csvData = new SharedArray('another data name', function () {
15  // Load CSV file and parse it using Papa Parse
16  return papaparse.parse(open('./data.csv'), { header: true }).data;
17});
18
19export default function () {
20  // Now you can use the CSV data in your test logic below.
21  // Below are some examples of how you can access the CSV data.
22
23  // Loop through all username/password pairs
24  for (const userPwdPair of csvData) {
25    console.log(JSON.stringify(userPwdPair));
26  }
27
28  // Pick a random username/password pair
29  const randomUser = csvData[Math.floor(Math.random() * csvData.length)];
30  console.log('Random user: ', JSON.stringify(randomUser));
31
32  const params = {
33    login: randomUser.username,
34    password: randomUser.password,
35  };
36  console.log('Random user: ', JSON.stringify(params));
37
38  const res = http.post('https://test.k6.io/login.php', params);
39  check(res, {
40    'login succeeded': (r) => r.status === 200 && r.body.indexOf('successfully authorized') !== -1,
41  });
42
43  sleep(1);
44}

Retrieving unique data

It is often a requirement not to use the same data more than once in a test. With the help of k6/execution, which includes a property scenario.iterationInTest, you can retrieve unique rows from your data set.

⚠️ Multiple scenarios
scenario.iterationInTest property is unique per scenario, not the overall test. That means if you have multiple scenarios in your test you might need to split your data per scenario.

1import { SharedArray } from 'k6/data';
2import { scenario } from 'k6/execution';
3
4const data = new SharedArray('users', function () {
5  return JSON.parse(open('./data.json')).users;
6});
7
8export const options = {
9  scenarios: {
10    'use-all-the-data': {
11      executor: 'shared-iterations',
12      vus: 10,
13      iterations: data.length,
14      maxDuration: '1h',
15    },
16  },
17};
18
19export default function () {
20  // this is unique even in the cloud
21  const user = data[scenario.iterationInTest];
22  console.log(`user: ${JSON.stringify(user)}`);
23}

Alternatively, if your use case requires using a unique data set per VU, you could leverage a property called vu.idInTest.

In the following example we're going to be using per-vu-iterations executor to ensure that every VU completes a fixed amount of iterations.

import { sleep } from 'k6';
import { SharedArray } from 'k6/data';
import { vu } from 'k6/execution';

const users = new SharedArray('users', function () {
  return JSON.parse(open('./data.json')).users;
});

export const options = {
  scenarios: {
    login: {
      executor: 'per-vu-iterations',
      vus: users.length,
      iterations: 20,
      maxDuration: '1h30m',
    },
  },
};

export default function () {
  // VU identifiers are one-based and arrays are zero-based, thus we need - 1
  console.log(`Users name: ${users[vu.idInTest - 1].username}`);
  sleep(1);
}

Generating data

See this example project on GitHub showing how to use faker.js to generate realistic data at runtime.

Old workarounds

The following section is here for historical reasons as it was the only way to lower the memory usage of k6 prior to v0.30.0 but after v0.27.0, but still have access to a lot of parameterization data with some caveats. All of the below should probably not be used as SharedArray should be sufficient.

After k6 version v0.27.0, while there was still no way to share memory between VUs, the __VU variable was now defined during the init context which means that we could split the data between the VUs during initialization and not have multiple copies of it during the test run. This is not useful now that SharedArray exists. Combining both will likely not bring any more performance benefit then using just the SharedArray.

1const splits = 100; // in how many parts are we going to split the data
2let data = [];
3
4if (__VU == 0) {
5  open('./data.json'); // we just open it so it is available in the cloud or if we do k6 archive
6} else {
7  const all_data = JSON.parse(open('./data.json')); // we load and parse the data in one go, no need for temp variables
8  const part_size = all_data.length / splits;
9  const index = part_size * (__VU % splits);
10  data = all_data.slice(index, index + part_size);
11}
12
13export default function () {
14  console.log(`VU=${__VU} has ${data.length} data`);
15}

With 100k lines like:

{ "username": "test", "password": "qwerty" },

and a total of 4.8MB the script uses 3.5GB to start 300 VUs, while without it for 100 VUs (with all the data for each VU) it requires nearly 10GB. For direct comparison 100VUs used near 2GB of memory.

Playing with the value for splits will give a different balance between memory used and the amount of data each VU has.

A second approach using another technique will be to pre-split the data in different files and load and parse only the one for each VU.

1import papaparse from 'https://jslib.k6.io/papaparse/5.1.1/index.js';
2import { sleep } from 'k6';
3
4const dataFiles = [
5  './data1.csv',
6  './data2.csv',
7  './data3.csv',
8  './data4.csv',
9  './data5.csv',
10  './data6.csv',
11  './data7.csv',
12  './data8.csv',
13  './data9.csv',
14  './data10.csv',
15];
16let csvData;
17if (__VU == 0) {
18  // workaround to collect all files for the cloud execution
19  for (let i = 0; i < dataFiles.length; i++) {
20    open(dataFiles[i]);
21  }
22} else {
23  csvData = papaparse.parse(open(dataFiles[__VU % dataFiles.length]), {
24    header: true,
25  }).data;
26}
27export default function () {
28  sleep(10);
29}

The files have 10k lines and are in total 128kb. Running 100VUs with this script takes around 2GB, while running the same with a single file takes upwards of 15GBs.

Either approach works for both JSON and CSV files and they can be combined, as that will probably reduce the memory pressure during the initialization even further.

Tutorials

Data Parameterization

From a JSON file

From a CSV file

Retrieving unique data

⚠️ Multiple scenarios

Generating data

Old workarounds

Product

Resources

Community

About