At ShipEngine, we’re constantly load testing. As the leading shipping API, our product allows users to shop rates, validate addresses, print labels, track shipments and more. We serve e-commerce marketplaces, online brands and third-party logistics companies — and we pride ourselves on stability, uptime and speed.
- ShipEngine processes millions of shipments each day.
- ShipEngine offers greater than 99.99% uptime.
- ShipEngine delivers sub-second response times.
And, that’s just on an average day. But, our users often can experience periods of extreme traffic, which means ShipEngine needs to be ready to accommodate any uptick in the volume of calls. Cyber Monday is a perfect example. In 2018, ShipEngine:
- Created 20,000 labels per minute at its peak.
- Maintained response times of no more than 600 milliseconds.
- Averaged response times of 500 milliseconds.
How was ShipEngine able to perform under the heavy stress of Cyber Monday?
Load testing is an important part of how we ensure this level of performance, no matter the day and no matter the time of year.
ShipEngine’s Infrastructure and Load Testing
ShipEngine is a cloud-native .NET application running in AWS. The core application runs on a series of EC2 instances and uses services running in a Kubernetes cluster also hosted in AWS.
As a software engineer I am hyper-focused on the quality of the code and the integrity of the processes. I used JMeter to test this in a previous role. And, while I found JMeter’s GUI to be helpful in getting comfortable with load testing when I was just starting out, it made it difficult to quickly see differences in test scripts.
At ShipEngine, I began running k6 locally to generate load. After importing the statistics into an InfluxDB for analysis — and getting the hang of the framework — I realized we needed a lot more computing power to generate the kind of load we needed to stress our application. We wanted the ability to generate Cyber Monday-like stress.
To simulate that environment, we began using k6 Cloud to run our k6 load scripts. This approach now gives us the ability to generate a sufficient load without needing to manage the client systems required to support it. Additionally, k6 Cloud presents the metrics and automates the results analysis so that we don’t have to maintain a time-series database.
ShipEngine’s Performance Testing Goals
It’s not just Cyber Monday that keeps us load testing. We have an extensive user base right now, and we expect continued growth in the coming weeks, months and years. Our primary focus is proactively scaling so that we stay ahead of the curve.
Similarly, our product also relies on the success of our partnership integrations with multiple carriers, all of which can vary widely in terms of their response times and behavior of their existing applications.
Performance testing (and testing tools like k6) allow us to evaluate the stability and responsiveness of our API under different load conditions.
We execute these tests regularly leading up to high-traffic days like Black Friday and Cyber Monday, but we also test before we release new features, new partners, and prior to onboarding new users that will bring a substantial increase in load.
We use k6 to generate load, and dotTrace to profile the .NET application while it’s running under load. The profiler lets us monitor the application at a granular level. We use it to identify how long a particular method takes to execute or whether we have threads stuck in a waiting state. Essentially, the profiler gives us hints on where to start investigating in the code.
While running under load and profiling, we monitor the health of our EC2 instances using PerfMon and AWS Cloudwatch. These tools combined allow us to nail down any bottlenecks.
- Do we need to scale our infrastructure?
- Do we have network latency?
- Is there a database contention or an inefficient code path that can be improved?
By running k6 scripts, we can mirror the exact same traffic pattern each time we make a change to our application stack until we’ve narrowed down the cause of any performance degradations.
Pro tip: make only one change at a time so that you can definitively correlate the change in performance to the change made within the application stack!
A few other tools we use to help monitor performance and infrastructure are New Relic APM, and the AWS Console.
Improvements: Our k6 Wishlist
There is one thing we’d like to see added to k6. We’d like the ability to log the full response whenever we encounter a non-200 series response code in our testing. I understand this feature is currently in process; it just hasn’t made it into the current version yet. Luckily we can get additional information about the request using Sumo Logic and elk in the meantime.
Give ShipEngine a Spin
If you’re interested in experiencing the fruits of our load testing, you can always sign up for a free API key through ShipEngine. We’d love to hear your feedback, too, so feel free to contact us about anything related to ShipEngine’s developer experience, documentation or the product itself.