Why Peak Season Testing Sparks Debate
Black Friday and other peak trading periods will always trigger one simple question. Should we be experimenting during peak when the “risks” are so high?
The main argument, and an understandable one is that a failed test could cost thousands in lost revenue. Others say it’s the best time to experiment, with bigger traffic volumes and the only chance for some ever 12 months to better understand this traffic set. The truth of course lies somewhere in the middle.
In this article, I’ve taken a look at the pros and cons of A/B testing during Black Friday, and share my thoughts on the approach for these key periods.
We Shouldn’t Test, it’s too risky!
Let’s start with the reasons I believe teams will ask this question, and why the decision can be to stop testing all together:
Higher stakes, bigger impact
A losing test in July might cost you a few thousand in missed sales. The same test in November could cost ten times as much. The financial impact of a “losing” experiment is much higher.
Noise and data volatility
Your peak period will differ from standard shopper behaviour as it brings in: gift buyers, more bargain hunters and the one-time seasonal shoppers. This will likely distort what we see as normal can make next step decisions more difficult, as can you guarantee these results outside of peak?
Short windows
For most Black Friday campaigns they only run for a few days, sometimes as things have shifted a couple of weeks. However, that may not be enough time for every test to reach statistical significance, especially for lower-traffic pages.
Teams are too busy
Having worked in a toy retailer during peak, I can relate to this one. When there are 101 jobs to do every day to implement promotions, price changes, marketing campaigns and the rest. Experimentation will often fall to the bottom of the pile.
But we could reap huge rewards if we win
Now the flip side, I personally believe it’s a mistake to fully stop testing during your peak and here’s why:
Bigger upside
A winning test during peak has the potential to generate much larger incremental revenues. If you also can roll this change to 100% quickly, you can reap the rewards immediately with the large traffic pools. The positives could outweigh the negatives here.
Everyday changes are untested anyway
During peak, hundreds of untested decisions are made across our sites from promotions, homepage banners, email campaigns, product sorting rules. These all carry risk too, each one of these has the potential to decrease conversion or revenue the same way testing can.
Faster learning cycles
The surge in traffic means tests reach significance quicker. What might take weeks in March could take just days in November.
My View: The Risks haven’t changed Only the Impact has
- The risks of A/B testing are present all year-round.
- A poor variant can underperform any month of the year.
- Traffic fluctuations can always distort results.
- Tests always require time, discipline, and monitoring.
What changes in Black Friday isn’t the type of risk, but the impact. A loss will cost more. But a win equally will deliver more too.
Best Practices for A/B Testing During Peak Season
If you decide to test during peak (and you should), approach it differently:
Prioritise ruthlessly
Focus only on tests with high potential impact or testing to understand the peak shopper better: recommendation algorithms, promotional content, delivery messaging, social proofing, etc
Monitor performance more regularly and set guardrails
Avoid “peaking” but keep your eyes open. If a test is underperforming and it’s slowly making its way to a level of loss you’re not comfortable with, pause it. You can always come back to it in the new year.
Shorten decision cycles
Be ready to pause or roll out variants faster than you would at other times of year. Peak windows are short, don’t forget Experimentation in ecommerce is to support decision making, we’re not testing the effectiveness of medicines.
Run “safer” tests
Stick to experiences unlikely to break core functionality and lower risk from a development perspective. This isn’t the time for risky experiments and 100’s of lines of code that could break with a simple merchandising change.
Plan ahead
Have experiments built, UAT’d, and ready to launch ahead of your peak season.
Conclusion: Should You Test During Black Friday?
Short answer: yes, I believe you should, if you can prioritise well, monitor closely and be prepared to pause losses or launch winners quickly.
However, this is only a question that truly each business can answer on its own, based on workload, team capacity and ability to turn changes round quickly. Hopefully, this article has given you some thoughts on how it could be approached.
Black Friday is when every conversion counts most. And that’s exactly when your testing plan should be at its sharpest.
Convinced? Us too – book some time with our expert team to discuss your strategy for Black Friday or peak trading times in general.
