Here in the U.S. it’s almost Thanksgiving. That means, of course, that Black Friday and Cyber Monday are not far behind. Black Friday is still a very big day for retail stores and ecommerce sites. But, as predicted, Black Friday has become "Black Fri-month” as retailers have been starting their promotions earlier and earlier to try to get a jump on the competition.
Some Thanksgiving, Black Friday and Turkey Related Stats:
- Black Friday online sales are about \$5 billion (USD) 88% of Americans eat turkey on Thanksgiving! (Source: National Turkey Federation)
- The U.S. poultry industry is about \$28 billion dollars (includes chicken, turkey, duck, geese and quail)
- Turkey products had a compound annual growth rate (CAGR) of almost 8% from 2011 to 2016 (Source: Packaged Facts)
As noted above, almost everyone here eats turkey for Thanksgiving. Also, in the U.S., turkeys have gotten a reputation for being dumb. My wife, a bird lover, would take strong exception to that idea. In any case, it’s probably related to the euphemism of "being a turkey,” that is, something that doesn’t perform very well.
If your business is an ecommerce site, now is the time when it’s got to perform. Peak traffic season is upon us. Hopefully you have been doing performance testing for several months, at least, and have confidence that your site will be up to the task.
If you work for a Software as a Service (SaaS) business, you’re probably concerned with performance and user experience (UX) throughout the year. This means regular performance testing as part of a continuous testing and agile development / DevOps methodology.
Some SaaS Industry Stats:
- Total Software as a Service (SaaS) sales are expected to be about \$55 billion in 2018-- so it’s about double the size of the poultry industry. (Source: FinancesOnline 2018 SaaS Industry Market Report)
- The SaaS industry is growing at about 33% CAGR-- again, we are handily beating the turkey industry! (Source: FinancesOnline 2018 Saas Industry Market Report) I say we, because Load Impact is a SaaS company.
Basically, whether you’re running an ecommerce site or a SaaS business, regular performance testing is good. Performance is a key component of overall software quality and significantly impacts the user experience.
Let’s take a look at how performance testing fits into the software development lifecycle.
Figure 1: Performance Testing Throughout the Software Development Lifecycle
Early in the development lifecycle, developers should run small, local load tests. By "local” we mean on their own machines. These can be ‘smoke tests’ that verify the load testing script. Or, tests to get quick performance feedback before pushing code to version control and your Continuous Integration (CI) tool. These tests would typically be run manually, from the command line. Output can either be to _stdout _or to an automated results analysis tool such as Load Impact Insights. As noted in this previous blog on 3 Phases of a Successful Performance Testing Methodology, the k6 load testing tool supports local execution from the command line.
In the next stage, automate your load tests by integrating into your CI pipeline. These tests can be run as part of your nightly build process. This promotes load testing as part of your continuous testing process. k6 has a feature called Thresholds that allows you to create a pass/fail result for your load test run. A non-zero exit code tells your CI tool that the load test failed.
Here is a sample JavaScript test script showing the use of Thresholds:
Here’s what the script does:
- The test will ramp up from 0 to 10 virtual users over 60s, stay flat at 10 virtual users for another 60s, before ramping down to 0 virtual users again over 60s
- We’ve set a goal that the 95th percentile response time should be below 500ms. This step, k6 thresholds, is essential; it’s how we define our goals which is the basis for the pass/fail mechanism to work that is built into k6. If a threshold fails, k6 will end with a non-zero exit code, which in turn indicates a failed build step to your CI tools (Jenkins, CircleCI, GitLab, Team City, etc.).
- In the load test "main” function we define a group Front page inside which we make a request, check the response status code and sleep for 10s before letting the user loop from the top of the function
(Learn more in this K6 article on integration with Jenkins.)
Running performance tests regularly allows you to see trends over multiple runs of the same tests. This lets you see when a performance regression occurs due to a code or configuration change.
In the last stage of the development lifecycle, QA teams can schedule larger load tests that will run less frequently than your nightly CI build process. These may be stress tests to ensure that your site or app can handle 2x or 3x the normal peak load, for example.
These later stage tests require more load testing infrastructure to run. Leverage services like Load Impact Cloud Execution to eliminate the burden of managing your own load testing infrastructure. Run load tests from multiple ‘load zones’ to test performance of users from many locations around the world.
Happy Thanksgiving Everyone!