...
- patio11 - Why You Don't A/B Test... And How You Can Start This August
- The single highest ROI tactic which I know of for software companies is conversion optimization.
- I used to consult at companies with $10 to $50 a million year in revenue. The median total number of A/B tests run in the firms' history was zero.
- "We Don't Have Enough Traffic"
- Testing free trial signup or email submission?: You need 3,000 visits to detect a small change in your conversion rate.
- Testing a usability improvement within the app?: You need about 1,500 users to detect improvements within your core workflow (good news: you'll notice almost any level of improvement).
- Testing conversion to paid use from a free trial?: You need about 2,000 free trials to detect an epic win in conversion rate, or 5,000 free trials to detect a strong win. Drop a digit off of both these numbers if you require a credit card upfront.
- "That 'Math' Nonsense Isn't A Substitute For Vision"
- Best Buy made $300 million in annual revenue by removing a single button and adding two sentences of copy.
- Bingo Card Creator did a 60% yearly revenue increase in 2012 on the strength of a brief series of A/B tests. That was the only thing I changed about BCC in 2012.
- "We Don't Have Enough Time"
- Graphical tools like Visual Website Optimizer or Optimizely make many genres of A/B tests take 5 to 10 minutes to set up.
- People have made over a dozen open-source A/B testing libraries, covering almost every commonly used web programming stack.
- "We Don't Know What To Test"
- Do my mini-course to find out.
- 2010.01.26 - VWO Blog - What you really need to know about mathematics of A/B split testing
- Main idea: Always, always, always use an A/B split testing calculator to determine significance of results before jumping to conclusions.
...
Warnings / Criticisms
...
regarding A/B testing
- https://www.widerfunnel.com/ab-testing-myths-debunked/
- https://conversionxl.com/blog/12-ab-split-testing-mistakes-i-see-businesses-make-all-the-time/
...