Note: This blog post has graphics from our older performance testing solution, Load Impact, which is no longer available and was replaced by k6. You can check out the documentation and website for more information about k6.
One arena where web APIs are especially useful is the Internet of Things. These devices are often low-power network connected (wifi or ethernet) with minimal CPU capabilities.
We often test web APIs that run in highly scalable cloud environments where most customers will deploy their applications. When running on a low-power, limited platform, the efficiency of the API becomes much more important.
We decided to run this series of tests on the very popular Raspberry Pi platform running a python-based API. The API being tested is the Webiopi, which provides an HTTP and REST API interface to a wide variety of sensors and GPIO on the platform. Think of it as a web-based window into the physical world of automation and sensors.
Why anyone would load test such a low-power device?
The principals for testing APIs apply quite well to this test scenario. Being constrained on CPU resources forces developers to be efficient in their coding and API development, which is something that's important regardless of the platform.
This particular Raspberry Pi used in "production" at my home monitors a GPIO input for a rising edge, indicating an event has taken place. The event in question is a doorbell press, which triggers a current sensor that outputs a "1" when current is flowing.
When this event occurs a python script pulls the most recent still image from a security camera, packages it in an email and sends it to me. Because this is event-driven, requires no monitoring and doesn't bother me with constant alerts, it is extremely useful.
It's also very popular with visitors who know how the system works.
A note on the architecture: This testing was done directly from our cloud-based platform directly to the Raspberry Pi device (via port forwarding on my firewall). This is fine for temporary testing but not the most secure or scalable solution for production use.
In a future article we will explore some additional options to expose this functionality to the web, including Weaved, a new cloud-based service that securely and rapidly deploys IoT capabilities. Weaved has a direct integration to Webiopi that provides secure access to the API and web front end without port forwarding.
The API being tested is a REST API with an HTTP front end. REST APIs are very popular and can be a very efficient means of checking and updating different types of data.
The first test I ran was a very simple script that gets the status of GPIO #10 using the following HTTP string: http://petermcannell.com:82/GPIO/10/value
The user scenario (script) is very simple for this initial test and can easily be edited to test other functions:
This returns the value of that GPIO output and is either a zero or one. As you can see below this test ran with excellent results and very consistent response times. Only 39KB of total data transferred, so clearly using the API to get the status of the input we want is very efficient.
I also ran the above test configuration with an HTTP POST instead of GET, thus setting the value of GPIO #10 instead of getting the status. There was no difference in performance whatsoever.
If there was a significant difference in performance, or if the CPU or memory on the server was dramatically different, this could point toward potential issues with the API code itself. It could also mean that the function just takes more resources. Having the data and quantifying the differences is useful.
Because that test went so well, the next test was an HTTP GET to retrieve the full JSON response using the following URL http://petermcannell.com:82/* which is in the following format:
Note that this is all text — no images or other overhead for the HTTP server to deal with. Results continue to show excellent performance and consistent latency. This time, almost 10x the amount of data was transferred (300KB):
Because that test went so well, the next test was the same HTTP GET but ramping up to 1000 users. Note that response times were pretty consistent, and the CPU of the server doesn't get maxed out, but there were 178 failures (where no response was received).
The next test was to see how far I could push the single HTTP GET request for GPIO #10. This is a very lightweight response (just a zero or one) so we are really pushing the API's ability to deal with connections and execute on the low-power platform.
This test failed at approximately the same point that the previous test did, which was around 850 concurrent users. That's where I aborted the test:
The last test I ran differed from the above. This time I loaded the entire webpage that includes images and status of all the GPIO pins. Even though this is a simple page many more objects are loaded and the response time (for even a few users) is much higher.
This test failure occurred at a much lower number of concurrent users — approximately 75. At that point failed connections started to add up and response times spiked.
This series of tests illustrates how efficient web APIs can be when compared to making a request for a full webpage with the same data displayed on it. Using an API to only request the exact data you want and nothing else can minimize response times, bandwidth utilization and improve performance significantly.
We also demonstrated that testing different functions (HTTP GET vs POST) can help uncover potential problems or show the API is well built and exhibits consistent behavior under load.
Running these type of tests on an API as part of a continuous deployment methodology adds very little time and ensures that problems are caught early in the development cycle.