Sequential Testing for Low-Volume A/B Tests

How to accelerate decision-making and improve accuracy when dealing with limited data

A/B testing is a simple way to reduce uncertainty in decision making by providing a data driven way to determine which version of a product is more effective. The concept of A/B testing is simple.

• Imagine you are at a friend’s birthday party. You’ve been painstakingly working on perfecting your cookie recipe. You think you’ve perfected it, but you don’t know if people will prefer the cookie with or without oats. In your opinion, oats give the cookie a nice chewy texture. However, you’re not sure if this is a mass opinion or just your individual preference.
• You end up showing up to the party with two different versions of the cookie, cookie A has oats and cookie B doesn’t. You randomly give half of your friends cookie A, and the other have get cookie B.
• You decide that the cookie that gets more “yums” is the better cookie.
• Once everyone has tasted the cookie, you find that cookie B got more “yums” and conclude that is the better cookie.