Converio Infinity Logo

How can we help?

Tell us a little bit about your situation, so that we can do a little bit of homework before calling you right back.

We often do the following things, so if any of them particularly appeal to you then check the box and we’ll get something ready for you.

    We’re here:

    Somerset House,

    Strand,

    London WC2 1LA

    Statistical Significance, Simplified

    Statistical significance is more than a set of complex calculations—it's a critical tool in the optimisation toolkit.

    Image of Steve

    Steve

    22nd April 2024

    Tags

    In the world of Conversion Rate Optimisation (CRO), the term ‘statistical significance’ is foundational. Yet, it’s a concept that often gets tangled in a web of complex explanations and statistical jargon, leaving many digital professionals scratching their heads. This post aims to demystify statistical significance, breaking it down into simpler terms and highlighting its crucial role in decision-making processes.

    The Essence of Optimisation

    At its core, CRO is about enhancing website experiences for customers through methodical testing. The goal is straightforward: if one version of a website experience proves superior, it should become the default customer experience. However, determining which version is truly superior introduces the need for a more scientific approach, where statistical significance takes the spotlight.

    What is Statistical Significance?

    A Google search on statistical significance yields the following definition:

    “Statistical significance refers to the claim that a result from data generated by testing or experimentation is likely to be attributable to a specific cause. A high degree of statistical significance indicates that an observed relationship is unlikely to be due to chance.”

    While accurate, this definition can seem daunting with its technicality. Yet, the concept can be simplified:

    “Statistical significance is a tool to support decision making, by giving an understanding of the risk of implementing a particular change.”

    The Holy Grail of CRO: 95% Statistical Significance

    In the realm of CRO, achieving a 95% statistical significance is often considered the gold standard. This means there’s only a 5% likelihood that the observed results occurred by chance. In practical terms, this level of confidence suggests that the outcomes we see in testing would likely persist if implemented across the website.

    Factors to Consider

    • Sample Size: The amount of data you need to ensure your results are reliable. A larger sample size can provide more confidence in the findings.
    • Test Duration: It’s essential to run tests for a sufficient period, ideally spanning a few business cycles, to account for any fluctuations in customer behaviour.

    Once a satisfactory level of statistical significance is achieved, the decision to implement changes, iterate further, or abandon the initiative rests on several considerations:

    1. Potential Impact: Evaluating the observed uplift and how it could improve customer experience or business metrics.
    2. Implementation Feasibility: Considering the cost and time required to make the changes permanent.
    3. Data for Further Optimisation: Deciding if there’s enough evidence to suggest modifications could yield even better results.

    Why Statistical Significance Matters

    Employing statistical significance effectively in CRO empowers businesses to make more informed decisions. It offers a buffer against the randomness of chance, providing a layer of confidence when deciding to roll out a new website feature or tweak an existing one. In essence, it helps businesses mitigate risk, ensuring that changes are genuinely beneficial to both the customer and the company.

    Conclusion

    Statistical significance is more than a set of complex calculations—it’s a critical tool in the optimisation toolkit. By understanding and applying this concept, optimisation specialists can lead their businesses to make data-driven decisions that enhance customer experiences and, ultimately, drive success. While the science behind statistical significance can seem intricate, its application doesn’t have to be. Simplifying this concept down to its essence allows us to harness its power more effectively, making our digital environments better for users everywhere.

    In a landscape where data is king, understanding statistical significance is akin to holding the keys to the kingdom. It’s what allows us to discern between a fleeting anomaly and a genuine improvement, guiding our optimisation efforts towards meaningful, impactful changes.

    Subscribe

    Optimise your inbox - subscribe to our newsletter