A/B test calculator: statistical significance
Enter visitors and conversions for two versions of an ad or page. You get both conversion rates, the change, a p-value and a plain answer on whether the difference is likely real.
Likely a real difference: B converts better than A, at 95% confidence.
- Conversion rate A
- 10%
- Conversion rate B
- 13%
- Change from A to B
- +30%
- p-value
- 0.035
- z-score
- 2.10
Two-sided test at 95% confidence. Visitors can also be clicks, sessions or people who saw an ad.
Free, no sign-up. Runs in your browser; nothing you type is sent to us.
What this A/B test calculator tells you
When two versions of an ad, email or landing page get different conversion rates, the gap might be real or might be chance. This A/B test calculator runs a standard statistical significance test on your numbers and says, in plain words, whether there is enough evidence that the versions really differ.
How it works
It uses a two-proportion z-test, the usual test for comparing two conversion rates:
- Conversion rate of each version = conversions ÷ visitors
- Pooled rate p = (conversions A + conversions B) ÷ (visitors A + visitors B)
- Standard error = √(p × (1 - p) × (1 ÷ visitors A + 1 ÷ visitors B))
z = (rate B - rate A) ÷ standard error
The p-value is the chance of seeing a gap at least this big if the two versions were really the same. Below 0.05, the result is called significant at 95% confidence.
Worked example
Example with made-up round numbers: version A gets 1,000 visitors and 100 conversions (10%). Version B gets 1,000 visitors and 130 conversions (13%), a 30% relative lift.
The pooled rate is 230 ÷ 2,000 = 11.5%. The standard error is √(0.115 × 0.885 × 0.002), about 0.0143. So z is 0.03 ÷ 0.0143, about 2.10, and the p-value is about 0.036. That is below 0.05, so B's better rate is likely a real difference.
Mistakes that fool A/B tests
- Peeking. Checking every day and stopping the moment the result looks significant makes false wins much more likely. Decide the sample size or the end date first, and judge the test then.
- Small samples. With a handful of conversions, one or two extra sales swing the rate a lot. The calculator warns when either version has fewer than 10 conversions.
- Uneven traffic. If one version ran on a weekend or to a different audience, you are comparing more than the change you made.
- Too many versions. Testing many variants at once raises the chance that one looks like a winner by luck.
Reading the result
"Likely a real difference" means a gap this large would be unusual if the versions performed the same. It does not prove the size of the lift: the true difference could be smaller or larger than what you measured.
"Not enough evidence yet" does not mean the versions are the same. It means the data cannot tell them apart. More visitors, or a bigger change, may give a clear answer. Feed the winner's rate into the conversion rate calculator to plan traffic.
Questions
What does statistically significant mean?
That a difference this large would rarely happen by chance if the two versions really performed the same. This calculator uses the common 95% level, meaning a p-value below 0.05.
How many visitors do I need for an A/B test?
It depends on your conversion rate and how big a change you want to detect. Small changes need far more visitors than large ones. As a rough floor, wait for at least 10 conversions in each version.
Can I stop the test as soon as it is significant?
Better not. Checking often and stopping at the first significant result makes false wins more likely. Set the length or sample size before you start.
What is a p-value?
The chance of seeing a gap at least as big as yours if there were really no difference between A and B. The smaller it is, the stronger the evidence.
Can I use this for ad tests?
Yes. Use impressions or clicks as visitors and the results you care about, such as clicks or sales, as conversions. Make sure both ads ran at the same time to similar audiences.
Ask your ads anything
Add AskAds to ChatGPT or Claude and ask your first question. Nothing spends until you press Approve.
Free for 7 days. No card needed to start.
Updated . See all free tools.