This example shows a way to use the ServiceDisruptor to test the effect of faults injected in the gRPC requests served by a service.
The complete source code is at the end of this document. The next sections examine the code in detail.
The example uses grpcpbin, a simple request/response application that offers endpoints for testing different gRPC requests.
The test requires grpcpbin to be deployed in a cluster in the namespace grpcbin and exposed with an external IP. The IP address is expected in the environment variable GRPC_HOST.
For the Kubernetes manifests and the instructions on how to deploy it to a cluster, refer to the test setup section at the end of this document. To learn more about how to get the external IP address, refer to Expose your application.
The initialization code imports the external dependencies required by the test. The ServiceDisruptor class imported from the xk6-disruptor extension provides functions for injecting faults in services. The k6/net/grpc module provides functions for executing gRPC requests. The check function verifies the results from the requests.
We also create a grpc client and load the protobufs definitions for the HelloService service.
The test load is generated by the default function, which connects to the grpcbin service using the IP and port obtained from the environment variable GRPC_HOST and invokes the SayHello method of the hello.HelloService service. Finally, The status code of the response is checked. When faults are injected, this check should fail.
The disrupt function creates a ServiceDisruptor for the grpcbin service in the namespace grpcbin.
The gRPC faults are injected by calling the ServiceDisruptor.injectGrpcFaults method using a fault definition that introduces a delay of 300ms on each request and an error with status code 13 ("Internal error") in 10% of the requests.
Notice the following code snippet in the injectFaults function above:
This code makes the function return without injecting faults if the SKIP_FAULTS environment variable is passed to the execution of the test with a value of "1". We will use this option to obtain a baseline execution without faults.
This test defines two scenarios to be executed. The load scenario applies the test load to the grpcpbin application for 30s invoking the default function. The disrupt scenario invokes the disrupt function to inject a fault in the gRPC requests to the target application.
The disrupt scenario uses a shared-iterations executor with one iteration and one VU. This setting ensures the disrupt function is executed only once. Executing this function multiples times concurrently may have unpredictable results.
The commands in this section assume the xk6-disruptor binary is available in your current directory. This location can change depending on the installation process and the platform. Refer to Installation for details on how to install it in your environment.
We will first execute the test without introducing faults to have an baseline using the following command:
Notice the argument --env SKIP_FAULT=1, which makes the disrupt function return without injecting any fault as explained in the fault injection section. Also notice the --env GRPC_HOST argument, which passes the external IP used to access the grpcbin application.
You should get an output similar to the following (use the Expand button to see all output).
We repeat the execution injecting the faults. Notice we have removed the --env SKIP_FAULTS=1 argument.
Let's take a closer look at the results for the requests on each scenario. We can observe that in the base scenario requests duration has an percentile 95 of 2.09ms and 100% of requests pass the check. The faults scenario has a percentile 96 of 303.57ms and only 89% of requests pass the check (or put in another way, 11% of requests failed), closely matching the fault definition.
|Execution||P95 Req. Duration||Passed Checks|
The tests requires the deployment of the grpcbin application. The application must also be accessible using an external IP available in the GRPC_HOST environment variable.
The following manifests define the resources required for deploying the application and exposing it as a LoadBalancer service.
You can deploy the application using the following commands:
You must set the environment variable GRPC_HOST with the external IP address and port used to access the grpcbin service from the test script.
Learn more about how to get the external IP address in the Expose your application.