No results for

Powered byAlgolia

Inject HTTP faults into Pod

This example shows how PodDisruptor can be used for testing the effect of faults injected in the HTTP requests served by a pod.

You will find the complete source code at the end of this document. Next sections examine the code in detail.

The example uses httpbin, a simple request/response application that offers endpoints for testing different HTTP requests.

The test requires httpbin to be deployed in a cluster in the namespace httpbin and exposed with an external IP. the IP address is expected in the environment variable SVC_IP.

You will find the Kubernetes manifests and the instructions of how to deploy it to a cluster in the test setup section at the end of this document. You can learn more about how to get the external IP address in the expose your application section.

Initialization

The initialization code imports the external dependencies required by the test. The PodDisruptor class imported from the xk6-disruptor extension provides functions for injecting faults in pods. The k6/http module provides functions for executing HTTP requests.

import { PodDisruptor } from 'k6/x/disruptor';
import http from 'k6/http';

Test Load

The test load is generated by the default function, which executes a request to the httpbin pod using the IP obtained from the environment variable SVC_IP. The test makes requests to the endpoint delay/0.1 which will return after 0.1 seconds (100ms).

export default function(data) {
http.get(`http://${data.SVC_IP}/delay/0.1`);
}
note

The test uses the delay endpoint which return after the requested delay. It requests a 0.1s (100ms) delay to ensure the baseline scenario (see scenarios below) has meaningful statistics for the request duration. If we were simply calling a locally deployed http server (for example nginx), the response time would exhibit a large variation between a few microseconds to a few milliseconds. Having 100ms as baseline response time has proved to offer more consistent results.

Fault injection

The disrupt function creates a PodDisruptor using a selector that matches pods in the namespace httpbin with the label app: httpbin.

The http faults are injected by calling the PodDisruptor.injectHTTPFaults method using a fault definition that introduces a delay of 50ms on each request and an error code 500 in 10% of the requests.

export function disrupt(data) {
if (__ENV.SKIP_FAULTS == "1") {
return
}
const selector = {
namespace: namespace,
select: {
labels: {
app: "httpbin"
}
}
}
const podDisruptor = new PodDisruptor(selector)
// delay traffic from one random replica of the deployment
const fault = {
average_delay: 50,
error_code: 500,
error_rate: 0.1
}
podDisruptor.injectHTTPFaults(fault, 30)
}

Notice the following code snippet in the injectFaults function above:

if (__ENV.SKIP_FAULTS == "1") {
return
}

This code makes the function return without injecting faults if the SKIP_FAULTS environment variable is passed to the execution of the test with a value of "1". We will use this option to obtain a baseline execution without faults.

Scenarios

This test defines two scenarios to be executed. The load scenario applies the test load to the httpbin application for 30s invoking the default function. The disrupt scenario invokes the disrupt function to inject a fault in the HTTP requests of the target application.

scenarios: {
load: {
executor: 'constant-arrival-rate',
rate: 100,
preAllocatedVUs: 10,
maxVUs: 100,
exec: "default",
startTime: '0s',
duration: "30s",
},
disrupt: {
executor: 'shared-iterations',
iterations: 1,
vus: 1,
exec: "disrupt",
startTime: "30s",
}
}
note

Notice that the disrupt scenario uses a shared-iterations executor with one iteration and one VU. This setting ensures the disrupt function is executed only once. Executing this function multiples times concurrently may have unpredictable results.

Executions

note

The commands in this section assume the xk6-disruptor binary is available in your current directory. This location can change depending on the installation process and the platform. Refer to the installation section for details on how to install it in your environment.

Baseline execution

We will first execute the test without introducing faults to have an baseline using the following command:

xk6-disruptor run --env SKIP_FAULTS=1 --env SVC_IP=$SVC_IP disrupt-pod.js

Notice the argument --env SKIP_FAULT=1 which makes the disrupt function to return without injecting any fault as explained in the fault injection section above. Also notice the --env SVC_IP=$SVC_IP argument which passes the external IP used to access the httpbin application.

You should get an output similar to the one shown below (click Expand button to see all output).

/\ |‾‾| /‾‾/ /‾‾/
/\ / \ | |/ / / /
/ \/ \ | ( / ‾‾\
/ \ | |\ \ | (‾) |
/ __________ \ |__| \__\ \_____/ .io
execution: local
script: test.js
output: -
scenarios: (100.00%) 2 scenarios, 101 max VUs, 10m30s max duration (incl. graceful stop):
* disrupt: 1 iterations shared among 1 VUs (maxDuration: 10m0s, exec: disrupt, gracefulStop: 30s)
* load: 100.00 iterations/s for 30s (maxVUs: 10-100, exec: default, gracefulStop: 30s)
running (00m30.1s), 000/014 VUs, 2998 complete and 0 interrupted iterations
disrupt ✓ [======================================] 1 VUs 00m00.0s/10m0s 1/1 shared iters
load ✓ [======================================] 000/013 VUs 30s 100.00 iters/s
data_received..................: 1.4 MB 46 kB/s
data_sent......................: 267 kB 8.9 kB/s
dropped_iterations.............: 4 0.132766/s
http_req_blocked...............: avg=8.08µs min=2.36µs med=5.8µs max=543.79µs p(90)=8.68µs p(95)=10.5µs
http_req_connecting............: avg=1.25µs min=0s med=0s max=418.63µs p(90)=0s p(95)=0s
http_req_duration..............: avg=103.22ms min=101.65ms med=103.13ms max=121.7ms p(90)=104.01ms p(95)=104.4ms
{ expected_response:true }...: avg=103.22ms min=101.65ms med=103.13ms max=121.7ms p(90)=104.01ms p(95)=104.4ms
http_req_failed................: 0.00% ✓ 0 ✗ 2997
http_req_receiving.............: avg=133.5µs min=45.06µs med=131.23µs max=879.43µs p(90)=193.48µs p(95)=223.04µs
http_req_sending...............: avg=31.14µs min=11.46µs med=29.37µs max=171.68µs p(90)=40.48µs p(95)=47.53µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=103.05ms min=101.54ms med=102.98ms max=121.56ms p(90)=103.82ms p(95)=104.2ms
http_reqs......................: 2997 99.474844/s
iteration_duration.............: avg=103.34ms min=109.86µs med=103.28ms max=121.92ms p(90)=104.19ms p(95)=104.63ms
iterations.....................: 2998 99.508035/s
vus............................: 13 min=11 max=13
vus_max........................: 14 min=12 max=14

Fault injection

We repeat the execution injecting the faults. Notice we have removed the --env SKIP_FAULTS=1 argument.

xk6-disruptor run --env SVC_IP=$SVC_IP disrupt-pod.js
/\ |‾‾| /‾‾/ /‾‾/
/\ / \ | |/ / / /
/ \/ \ | ( / ‾‾\
/ \ | |\ \ | (‾) |
/ __________ \ |__| \__\ \_____/ .io
execution: local
script: disrupt-pod.js
output: -
scenarios: (100.00%) 2 scenarios, 101 max VUs, 10m30s max duration (incl. graceful stop):
* disrupt: 1 iterations shared among 1 VUs (maxDuration: 10m0s, exec: disrupt, gracefulStop: 30s)
* load: 100.00 iterations/s for 30s (maxVUs: 10-100, exec: default, gracefulStop: 30s)
running (00m31.1s), 000/018 VUs, 2995 complete and 0 interrupted iterations
disrupt ✓ [======================================] 1 VUs 00m31.1s/10m0s 1/1 shared iters
load ✓ [======================================] 000/017 VUs 30s 100.00 iters/s
data_received..................: 1.1 MB 34 kB/s
data_sent......................: 267 kB 8.6 kB/s
dropped_iterations.............: 7 0.224798/s
http_req_blocked...............: avg=9.81µs min=2.59µs med=5.93µs max=489.67µs p(90)=7.88µs p(95)=9.5µs
http_req_connecting............: avg=2.48µs min=0s med=0s max=367.63µs p(90)=0s p(95)=0s
http_req_duration..............: avg=142.15ms min=50.33ms med=153.79ms max=165.85ms p(90)=154.8ms p(95)=155.12ms
{ expected_response:true }...: avg=151.9ms min=101.81ms med=153.9ms max=165.85ms p(90)=154.86ms p(95)=155.17ms
http_req_failed................: 9.65% ✓ 289 ✗ 2705
http_req_receiving.............: avg=80.92µs min=28.32µs med=77.33µs max=352.19µs p(90)=105.09µs p(95)=123.68µs
http_req_sending...............: avg=30.43µs min=11.27µs med=29.37µs max=287.71µs p(90)=37.42µs p(95)=41.84µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=142.04ms min=50.25ms med=153.68ms max=165.76ms p(90)=154.69ms p(95)=155.01ms
http_reqs......................: 2994 96.149356/s
iteration_duration.............: avg=152.64ms min=50.43ms med=153.93ms max=31.12s p(90)=154.97ms p(95)=155.29ms
iterations.....................: 2995 96.18147/s
vus............................: 1 min=1 max=18
vus_max........................: 18 min=12 max=18

Comparison

Let's take a closer look at the results for the requests on each scenario. We can observe that he base scenario has an average of 103ms and an error rate of 0% while the faults scenario has a median around 151.9ms and an error rate of nearly 10%, matching the definition of the faults defined in the disruptor.

ExecutionAvg. ResponseFailed requests
Baseline103.22ms0.00%
Fault injection151.9 ms9.65%
note

Notice we have used the average response time reported as expected_response:true because this metric only consider successful requests while http_req_duration considers all requests, including those returning a fault.

Source Code

disrupt-pod.js
import { PodDisruptor } from 'k6/x/disruptor';
import http from 'k6/http';
export default function (data) {
http.get(`http://${__ENV.SVC_IP}/delay/0.1`);
}
export function disrupt(data) {
if (__ENV.SKIP_FAULTS == "1") {
return
}
const selector = {
namespace: "httpbin",
select: {
labels: {
app: "httpbin"
}
}
}
const podDisruptor = new PodDisruptor(selector)
// delay traffic from one random replica of the deployment
const fault = {
averageDelay: 50,
errorCode: 500,
errorRate: 0.1
}
podDisruptor.injectHTTPFaults(fault, 30)
}
export const options = {
scenarios: {
load: {
executor: 'constant-arrival-rate',
rate: 100,
preAllocatedVUs: 10,
maxVUs: 100,
exec: "default",
startTime: '0s',
duration: "30s",
},
disrupt: {
executor: 'shared-iterations',
iterations: 1,
vus: 1,
exec: "disrupt",
startTime: "0s",
},
}
}

Test setup

The tests requires the deployment of the httpbin application. The application must also be accessible using an external IP available in the SVC_IP environment variable.

The manifests below define the resources required for deploying the application and exposing it as a LoadBalancer service.

You can deploy the application using the following commands:

# Create Namespace
kubectl apply -f namespace.yaml
namespace/httpbin created
# Deploy Pod
kubectl apply -f pod.yaml
pod/httpbin created
# Expose Pod as a Service
kubectl apply -f service.yaml
service/httpbin created

You can retrieve the resources using the following command:

kubectl -n httpbin get all
NAME READY STATUS RESTARTS AGE
pod/httpbin 1/1 Running 0 1m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/httpbin LoadBalancer 10.96.169.78 172.18.255.200 80:31224/TCP 1m

You can retrieve the external IP address in the environment variable SVC_IP using the following command:

SVC_IP=$(kubectl -n httpbin get svc httpbin --output jsonpath='{.status.loadBalancer.ingress[0].ip}')

Manifests

namespace.yaml
apiVersion: "v1"
kind: Namespace
metadata:
name: httpbin
pod.yaml
kind: "Pod"
apiVersion: "v1"
metadata:
name: httpbin
namespace: httpbin
labels:
app: httpbin
spec:
containers:
- name: httpbin
image: kennethreitz/httpbin
ports:
- name: http
containerPort: 80
service.yaml
apiVersion: "v1"
kind: "Service"
metadata:
name: httpbin
namespace: httpbin
spec:
selector:
app: httpbin
type: "LoadBalancer"
ports:
- port: 80
targetPort: 80