Table of contents

Child pages

Related pages

...

patio11 - Why You Don't A/B Test... And How You Can Start This August
- The single highest ROI tactic which I know of for software companies is conversion optimization.
- I used to consult at companies with $10 to $50 a million year in revenue. The median total number of A/B tests run in the firms' history was zero.
- "We Don't Have Enough Traffic"
  - Testing free trial signup or email submission?: You need 3,000 visits to detect a small change in your conversion rate.
  - Testing a usability improvement within the app?: You need about 1,500 users to detect improvements within your core workflow (good news: you'll notice almost any level of improvement).
  - Testing conversion to paid use from a free trial?: You need about 2,000 free trials to detect an epic win in conversion rate, or 5,000 free trials to detect a strong win. Drop a digit off of both these numbers if you require a credit card upfront.
- "That 'Math' Nonsense Isn't A Substitute For Vision"
  - Best Buy made $300 million in annual revenue by removing a single button and adding two sentences of copy.
  - Bingo Card Creator did a 60% yearly revenue increase in 2012 on the strength of a brief series of A/B tests. That was the only thing I changed about BCC in 2012.
- "We Don't Have Enough Time"
  - Graphical tools like Visual Website Optimizer or Optimizely make many genres of A/B tests take 5 to 10 minutes to set up.
  - People have made over a dozen open-source A/B testing libraries, covering almost every commonly used web programming stack.
- "We Don't Know What To Test"
  - Do my mini-course to find out.
2010.01.26 - VWO Blog - What you really need to know about mathematics of A/B split testing
- Main idea: Always, always, always use an A/B split testing calculator to determine significance of results before jumping to conclusions.

Positive A/B stories

https://signalvnoise.com/posts/2991-behind-the-scenes-ab-testing-part-3-final

Warnings / Criticisms regarding A/B testing

Browser not supported