API Testing 22 March 2017

Blueprint for API Performance Testing: Plan, Configure, and Execute

David Rosen 

APIs are everywhere. As I write this post, I know that the underlying technology enabling me to produce this text is wired together and magically works through the integration of a significant number of loosely and tightly coupled API interactions. As I use the Fitbit ecosystem to monitor my health and fitness, I know there are numerous APIs involved in delivering insights and information to me. As I file my business expenses via Expensify, I know the simple soup-to-nuts workflow from submission to payout would be impossible unless a number of different industry providers integrated seamlessly via APIs.

Functional capabilities and features aside, how does an API provider make sure it is fit for purpose and can deliver on defined Service Level Agreements, SLAs?
There are a number of non-functional needs to consider: Security, Reliability, Scalability and Performance. This post covers all but the first aspect and will detail an approach for planning, configuring and running load tests to make sure your APIs scale and perform in a reliable manner given a model of the traffic pattern that your business demands.

API performance testing planning

API performance testing goals

The first question to answer in the API performance testing planning phase is what the goals of your testing are. Is your primary interest to establish metrics for your business and technical operations through black-box testing, or more developer-focused where you're looking to understand the impact and effects on your system as you run what we call white-box tests?

For the metrics-minded: are you looking to validate performance, i.e., how fast (or slowly!) your API responds to requests? Or are you looking to validate scalability, i.e., at what levels of incoming traffic your system starts to fall behind? Or are you looking to understand how reliable your system is, i.e., after how long at known amounts of load your system starts to crumble?

For the developer: do you have enough system instrumentation and monitoring so you can derive actionable insights as your system responds to the load generated by the testing? Can you answer with certainty when and where your system hit bottlenecks or fragile architecture as it is exposed to load testing? If so, can you quantify what part of the system represents the largest degradation in serviceability, thus implying a priority of what performance bottleneck to address first?

System Under Test (SUT)

Next question to clarify and understand is what exactly to test in your API performance test. Are you looking to test serviceability of individual API endpoints? Are you instead looking to test broadly across all endpoints of an API? Or are you looking for a more complex test where you have any number of APIs interacting via or within some workflow or defined scenario?

Service Level Agreements (SLA)

The value of load and performance testing is directly proportional to how closely your test resembles real-life or expected usage. Knowing your business demands a particular API to support incoming throughput at 1 million r/s (requests per second, sometimes abbreviated RPS) obviously largely dictates the end goal of testing that API.

For other APIs where no SLA exists, we recommend understanding existing traffic patterns in order to derive reasonable levels of traffic and load. Application Performance Monitoring (APM) services such as New Relic, AppDynamics and Dynatrace should be able to help you understand:

  • Distribution of throughput in requests per second - average, peak, etc.
  • Distribution of response times and resource utilization at average and peak loads.
  • Throughput distribution by API endpoint. This is rarely evenly distributed, more commonly one or a few endpoints representing a significant majority of all traffic with the rest spread out across a long tail.
  • Throughput distribution by users. Typically we see a few users or accounts generating most traffic, with the rest spread out across a long tail.
  • Throughput distribution by geography.

Tooling and infrastructure

Success of load testing depends on proper choice of tooling, and sufficient infrastructure. At Load Impact with our SaaS performance testing service, we are obviously biased in the choice of tooling. Load Impact's SaaS offering shows our hard work to eliminate insufficient infrastructure being a factor for users as they plan and run their load testing. This all said, should you have an interest to further explore the load testing tooling space, here is our review of open source load testing tools.

API performance testing configuration

Modeling load testing throughput

As you move to configure your API load testing, it is important to consider the traffic pattern of your tests. Will your testing be satisfied by simple and repetitive "hammering" of API endpoints? Or will you need something more dynamic and sophisticated, typically accomplished through scripting of logic and execution flows? Or will you rather want to test using true replication and replay of real-world traffic?

Here, less is typically more. We encourage starting out by "exploring" the performance and scalability boundaries of your API and system. Once you have an understanding of your baseline, you can expand and explore more sophisticated strategies to meet your testing targets and goals.

As you move to make your API load testing more structured, define your testing of the various API endpoints in composable scenarios. Consider using traffic shaping to model ramp-ups/ramp-downs, adding multiple scenarios to your test, distributing the traffic proportionally according to your understanding of API endpoint distribution. Also, if your choice of load testing tooling and infrastructure supports global distribution of traffic, consider configuring load generators based upon your understanding of geographic distribution of your incoming traffic.

Load testing scenario creation and scripting

We recommend composing your API load tests into reusable modules or scenarios, where each scenario typically tests an individual API endpoint or distinct user workflow. This scenario can then be used standalone to validate serviceability of that particular endpoint, and/or be used as one of many scenarios in a more complex configuration where performance and scalability of the overall system is looking to be characterized when used in a more real-world production setup.

For an example of how a scenario can be defined in Load Impact to support controlled modeling of API load testing, see this article on How to load test an API and below code snippet:

API load testing snippet:

API performance testing execution

Exploratory API load testing

As mentioned previously, we recommend starting out by building a composable set of API load testing scenarios that you can test individually for baselines and boundaries. Look at this as manual, exploratory API load testing, using metrics generated from your load testing infrastructure and system monitoring to inform next steps as you move towards more structured, automated performance testing.

Image 2017 03 13 at 16 17 50

Automated API load testing in a CI/CD setup

Once you have a testing setup that models your SLAs and can satisfy your load testing goals, consider integrating execution of your tests into your existing CI/CD infrastructure. If you are using Load Impact for your load testing, see our main website for more details on how to integrate your performance testing with Jenkins, CircleCI, Team City and many other CI automation platforms.

Happy API Performance Testing!

< Back to all posts