Experimentation at Scale: Deep Statistical Concepts for Trustworthy A/B Testing in E-Commerce

Abstract

To stay competitive in the e-commerce space, retailers must always introduce new features to their website to enhance the user experience and improve operational performance. A/B Testing can help identify if these features have a positive impact by allowing retailers to conduct a controlled experimental design using statistical methods to experimentally compare a group of users who experience the new features against a group of users who do not experience the new features. When they are doing a test, a retailer must come up with a hypothesis that they will test, then measure and analyze by statistical methods a sample of users (for example, by setting up a sample size, choosing metrics and determining randomization procedures) to find statistically significant evidence that points to whether the test result is a real improvement or just caused by random chance. Besides this, if you fail to consider common pitfalls of the experimental design, like the novelty effect, selection bias, or problems related to multiple testing under the same conditions, the results of your experiments may be incorrect. Traditional A/B Testing is a reasonable tool for use in most experimental needs; however, advanced experimentation techniques provide additional capabilities when conducting experiments under more complex scenarios. Multi-Armed Bandit Algorithms can dynamically optimize the allocation of web traffic to different versions of a webpage while minimizing the cost associated with directing users to suboptimal experiences. Lastly, causal inference methods can enable businesses to measure the impact of changes on their websites without using randomization methodology, thereby allowing them to evaluate the effectiveness of changes across the entire platform. "Trustworthy Experimentation" combines a solid foundation of statistical methodology with experience gained from actual business decisions to help businesses learn and iterate more rapidly while reducing risk. The principles of valid experimentation provide a framework for businesses to implement valid experiments to support evidence-based decision-making as it relates to e-commerce.