No results for

Powered byAlgolia

Inject gRPC faults into Service

This example shows a way to use the ServiceDisruptor to test the effect of faults injected in the gRPC requests served by a service.

The complete source code is at the end of this document. The next sections examine the code in detail.

The example uses grpcpbin, a simple request/response application that offers endpoints for testing different gRPC requests.

The test requires grpcpbin to be deployed in a cluster in the namespace grpcbin and exposed with an external IP. The IP address is expected in the environment variable GRPC_HOST.

For the Kubernetes manifests and the instructions on how to deploy it to a cluster, refer to the test setup section at the end of this document. To learn more about how to get the external IP address, refer to Expose your application.

Initialization

The initialization code imports the external dependencies required by the test. The ServiceDisruptor class imported from the xk6-disruptor extension provides functions for injecting faults in services. The k6/net/grpc module provides functions for executing gRPC requests. The check function verifies the results from the requests.

import { ServiceDisruptor } from 'k6/x/disruptor';
import grpc from 'k6/net/grpc';
import { check } from 'k6';

We also create a grpc client and load the protobufs definitions for the HelloService service.

const client = new grpc.Client();
client.load(['pb'], 'hello.proto');

Test Load

The test load is generated by the default function, which connects to the grpcbin service using the IP and port obtained from the environment variable GRPC_HOST and invokes the SayHello method of the hello.HelloService service. Finally, The status code of the response is checked. When faults are injected, this check should fail.

export default function () {
client.connect(
__ENV.GRPC_HOST,
{
plaintext: true,
timeout: '1s'
}
);
const data = { greeting: 'Bert' };
const response = client.invoke(
'hello.HelloService/SayHello',
data,
{
timeout: '1s'
}
);
client.close()
check(response, {
'status is OK': (r) => r && r.status === grpc.StatusOK,
});
}

Fault injection

The disrupt function creates a ServiceDisruptor for the grpcbin service in the namespace grpcbin.

The gRPC faults are injected by calling the ServiceDisruptor.injectGrpcFaults method using a fault definition that introduces a delay of 300ms on each request and an error with status code 13 ("Internal error") in 10% of the requests.

export function disrupt() {
if (__ENV.SKIP_FAULTS == "1") {
return
}
const fault = {
averageDelay: "300ms",
statusCode: grpc.StatusInternal,
errorRate: 0.1,
port: 9000,
}
const disruptor = new ServiceDisruptor('grpcbin','grpcbin')
disruptor.injectGrpcFaults(fault, "30s")
}

Notice the following code snippet in the injectFaults function above:

if (__ENV.SKIP_FAULTS == "1") {
return
}

This code makes the function return without injecting faults if the SKIP_FAULTS environment variable is passed to the execution of the test with a value of "1". We will use this option to obtain a baseline execution without faults.

Scenarios

This test defines two scenarios to be executed. The load scenario applies the test load to the grpcpbin application for 30s invoking the default function. The disrupt scenario invokes the disrupt function to inject a fault in the gRPC requests to the target application.

scenarios: {
load: {
executor: 'constant-arrival-rate',
rate: 100,
preAllocatedVUs: 10,
maxVUs: 100,
exec: "default",
startTime: '0s',
duration: "30s",
},
disrupt: {
executor: 'shared-iterations',
iterations: 1,
vus: 1,
exec: "disrupt",
startTime: "0s",
}
}
note

The disrupt scenario uses a shared-iterations executor with one iteration and one VU. This setting ensures the disrupt function is executed only once. Executing this function multiples times concurrently may have unpredictable results.

Executions

note

The commands in this section assume the xk6-disruptor binary is available in your current directory. This location can change depending on the installation process and the platform. Refer to Installation for details on how to install it in your environment.

Baseline execution

We will first execute the test without introducing faults to have an baseline using the following command:

Linux/MacOS
Windows PowerShell
xk6-disruptor --env SKIP_FAULTS=1 --env GRPC_HOST=$GRPC_HOST run grpc-faults.js

Notice the argument --env SKIP_FAULT=1, which makes the disrupt function return without injecting any fault as explained in the fault injection section. Also notice the --env GRPC_HOST argument, which passes the external IP used to access the grpcbin application.

You should get an output similar to the following (use the Expand button to see all output).

/\ |‾‾| /‾‾/ /‾‾/
/\ / \ | |/ / / /
/ \/ \ | ( / ‾‾\
/ \ | |\ \ | (‾) |
/ __________ \ |__| \__\ \_____/ .io
execution: local
script: scripts/grpc-faults.js
output: -
scenarios: (100.00%) 2 scenarios, 101 max VUs, 10m30s max duration (incl. graceful stop):
* disrupt: 1 iterations shared among 1 VUs (maxDuration: 10m0s, exec: disrupt, gracefulStop: 30s)
* load: 100.00 iterations/s for 30s (maxVUs: 10-100, exec: default, gracefulStop: 30s)
running (00m30.0s), 000/011 VUs, 3001 complete and 0 interrupted iterations
disrupt ✓ [======================================] 1 VUs 00m00.0s/10m0s 1/1 shared iters
load ✓ [======================================] 000/010 VUs 30s 100.00 iters/s
✓ status is OK
checks...............: 100.00% ✓ 3000 ✗ 0
data_received........: 416 kB 14 kB/s
data_sent............: 630 kB 21 kB/s
grpc_req_duration....: avg=1.36ms min=512.43µs med=1.25ms max=26.62ms p(90)=1.75ms p(95)=2.09ms
iteration_duration...: avg=4.4ms min=52.86µs med=4.01ms max=66.26ms p(90)=5.32ms p(95)=6.35ms
iterations...........: 3001 100.028406/s
vus..................: 10 min=10 max=10
vus_max..............: 11 min=11 max=11

Fault injection

We repeat the execution injecting the faults. Notice we have removed the --env SKIP_FAULTS=1 argument.

Linux/MacOS
Windows PowerShell
xk6-disruptor --env GRPC_HOST=$GRPC_HOST run grpc-faults.js
/\ |‾‾| /‾‾/ /‾‾/
/\ / \ | |/ / / /
/ \/ \ | ( / ‾‾\
/ \ | |\ \ | (‾) |
/ __________ \ |__| \__\ \_____/ .io
execution: local
script: scripts/grpc-faults.js
output: -
scenarios: (100.00%) 2 scenarios, 101 max VUs, 10m30s max duration (incl. graceful stop):
* disrupt: 1 iterations shared among 1 VUs (maxDuration: 10m0s, exec: disrupt, gracefulStop: 30s)
* load: 100.00 iterations/s for 30s (maxVUs: 10-100, exec: default, gracefulStop: 30s)
running (00m50.6s), 000/033 VUs, 2975 complete and 0 interrupted iterations
disrupt ✓ [======================================] 1 VUs 00m50.6s/10m0s 1/1 shared iters
load ✓ [======================================] 000/032 VUs 30s 100.00 iters/s
✗ status is OK
↳ 89% — ✓ 2668 / ✗ 306
checks...............: 89.71% ✓ 2668 ✗ 306
data_received........: 405 kB 8.0 kB/s
data_sent............: 615 kB 12 kB/s
dropped_iterations...: 26 0.513628/s
grpc_req_duration....: avg=270.51ms min=502.12µs med=302.34ms max=310.48ms p(90)=303.25ms p(95)=303.57ms
iteration_duration...: avg=290.05ms min=1.96ms med=304.86ms max=50.61s p(90)=306.36ms p(95)=306.89ms
iterations...........: 2975 58.770842/s
vus..................: 1 min=1 max=33
vus_max..............: 33 min=25 max=33

Comparison

Let's take a closer look at the results for the requests on each scenario. We can observe that in the base scenario requests duration has an percentile 95 of 2.09ms and 100% of requests pass the check. The faults scenario has a percentile 96 of 303.57ms and only 89% of requests pass the check (or put in another way, 11% of requests failed), closely matching the fault definition.

ExecutionP95 Req. DurationPassed Checks
Baseline2.09ms100%
Fault injection303.57ms89%

Source Code

grpc-faults.js
import { ServiceDisruptor } from 'k6/x/disruptor';
import grpc from 'k6/net/grpc';
import { check } from 'k6';
const client = new grpc.Client();
client.load(['pb'], 'hello.proto');
export default function () {
client.connect(
__ENV.GRPC_HOST,
{
plaintext: true,
timeout: '1s'
}
);
const data = { greeting: 'Bert' };
const response = client.invoke(
'hello.HelloService/SayHello',
data,
{
timeout: '1s'
}
);
client.close()
check(response, {
'status is OK': (r) => r && r.status === grpc.StatusOK,
});
}
export function disrupt() {
if (__ENV.SKIP_FAULTS == "1") {
return
}
const disruptor = new ServiceDisruptor('grpcbin','grpcbin')
// inject errors in requests
const fault = {
averageDelay: "300ms",
statusCode: grpc.StatusInternal,
errorRate: 0.1,
port: 9000,
}
disruptor.injectGrpcFaults(fault, "30s")
}
export const options = {
scenarios: {
load: {
executor: 'constant-arrival-rate',
rate: 100,
preAllocatedVUs: 10,
maxVUs: 100,
exec: "default",
startTime: '0s',
duration: "30s",
},
disrupt: {
executor: 'shared-iterations',
iterations: 1,
vus: 1,
exec: "disrupt",
startTime: "0s",
}
}
}
pb/hello.proto
syntax = "proto2";
package hello;
service HelloService {
rpc SayHello(HelloRequest) returns (HelloResponse);
rpc LotsOfReplies(HelloRequest) returns (stream HelloResponse);
rpc LotsOfGreetings(stream HelloRequest) returns (HelloResponse);
rpc BidiHello(stream HelloRequest) returns (stream HelloResponse);
}
message HelloRequest {
optional string greeting = 1;
}
message HelloResponse {
required string reply = 1;
}

Test setup

The tests requires the deployment of the grpcbin application. The application must also be accessible using an external IP available in the GRPC_HOST environment variable.

The following manifests define the resources required for deploying the application and exposing it as a LoadBalancer service.

You can deploy the application using the following commands:

# Create Namespace
kubectl apply -f namespace.yaml
namespace/grpcbin created
# Deploy Pod
kubectl apply -f pod.yaml
pod/grpcbin created
# Expose Pod as a Service
kubectl apply -f service.yaml
service/grpcbin created

You must set the environment variable GRPC_HOST with the external IP address and port used to access the grpcbin service from the test script.

Learn more about how to get the external IP address in the Expose your application.

Manifests

namespace.yaml
apiVersion: "v1"
kind: Namespace
metadata:
name: grpcbin
pod.yaml
kind: "Pod"
apiVersion: "v1"
metadata:
name: grpcbin
namespace: grpcbin
labels:
app: grpcbin
spec:
containers:
- name: grpcbin
image: moul/grpcbin
ports:
- name: http
containerPort: 9000
service.yaml
apiVersion: "v1"
kind: "Service"
metadata:
name: grpcbin
namespace: grpcbin
spec:
selector:
app: grpcbin
type: "NodePort"
ports:
- port: 9000
targetPort: 9000