Table of contents

Child pages

Related pages

• ...

• patio11 - Why You Don't A/B Test... And How You Can Start This August
  • The single highest ROI tactic which I know of for software companies is conversion optimization.
  • I used to consult at companies with $10 to $50 a million year in revenue. The median total number of A/B tests run in the firms' history was zero.
  • "We Don't Have Enough Traffic"
    • Testing free trial signup or email submission?: You need 3,000 visits to detect a small change in your conversion rate.
    • Testing a usability improvement within the app?: You need about 1,500 users to detect improvements within your core workflow (good news: you'll notice almost any level of improvement).
    • Testing conversion to paid use from a free trial?: You need about 2,000 free trials to detect an epic win in conversion rate, or 5,000 free trials to detect a strong win. Drop a digit off of both these numbers if you require a credit card upfront.
  • "That 'Math' Nonsense Isn't A Substitute For Vision"
    • Best Buy made $300 million in annual revenue by removing a single button and adding two sentences of copy.
    • Bingo Card Creator did a 60% yearly revenue increase in 2012 on the strength of a brief series of A/B tests. That was the only thing I changed about BCC in 2012.
  • "We Don't Have Enough Time"
    • Graphical tools like Visual Website Optimizer or Optimizely make many genres of A/B tests take 5 to 10 minutes to set up.
    • People have made over a dozen open-source A/B testing libraries, covering almost every commonly used web programming stack.
  • "We Don't Know What To Test"
    • Do my mini-course to find out.
• 2010.01.26 - VWO Blog - What you really need to know about mathematics of A/B split testing
  • Main idea: Always, always, always use an A/B split testing calculator to determine significance of results before jumping to conclusions.

Positive A/B stories

• https://signalvnoise.com/posts/2991-behind-the-scenes-ab-testing-part-3-final

Warnings / Criticisms regarding A/B testing

• https://www.widerfunnel.com/ab-testing-myths-debunked/
• https://conversionxl.com/blog/12-ab-split-testing-mistakes-i-see-businesses-make-all-the-time/