A/B Test Significance Calculator
An A/B test significance calculator using a two-proportion z-test. Enter the visitors and conversions for a control and a variant to get each conversion rate, the relative uplift, the z-score and two-tailed p-value, and a clear verdict on whether the result is statistically significant. Explains what significance does and does not tell you.
Control (A)
4.00%
Variant (B)
5.25%
An A/B test never hands you a clean answer — it hands you two noisy numbers and a question: is the gap real? Variant B converting at 4.2% versus A’s 4.0% could be a genuine improvement, or it could be the same coin landing heads a few more times. Significance testing is how you tell those apart with a stated level of doubt.
The two-proportion z-test does it directly. It measures how far apart the two rates are in units of their combined sampling noise (the z-score), then reads off the p-value: the chance that two truly-identical variants would look at least this different by luck alone. Below 0.05 and you call it significant — the difference is hard to write off as randomness at this sample size. This calculator shows the rates, the uplift, the z-score and that p-value so you can see the whole picture, not just a verdict.
Two disciplines keep it honest. First, fix your sample size before you start and evaluate once — peeking and stopping the instant you cross 0.05 manufactures false positives, because a random walk crosses the line temporarily all the time. Second, significance is not importance: a giant sample makes trivial differences significant, so always read the effect size and ask whether the uplift is actually worth shipping. “Probably real” and “worth doing” are separate questions, and you need both.
How it works
- Two-proportion z-test on control vs variant conversions.
- Shows conversion rates, relative uplift, z-score and p-value.
- Flags significance at the 95% (p < 0.05) level.
- Reminds you that significance ≠ a meaningful effect size.
Frequently asked questions
What does statistical significance mean here?
It asks: if the two variants were actually identical, how often would random chance alone produce a difference at least this large? That probability is the p-value. A common threshold is 0.05 (5%) — if the p-value is below it, the result is called statistically significant, meaning the difference is unlikely to be pure noise. It is not proof the variant is better; it is evidence that the observed gap is hard to explain by chance at your sample size.
How is the p-value computed?
This uses a two-proportion z-test. It pools the two conversion rates, computes the standard error of their difference, and turns the observed difference into a z-score (how many standard errors apart they are). The two-tailed p-value is the probability of a z-score at least that extreme in either direction, read from the normal distribution. It is the standard test for comparing two conversion rates with reasonably large samples.
Why can a result flip if I keep peeking?
Because significance testing assumes you fixed your sample size in advance. If you check the p-value repeatedly and stop the moment it dips below 0.05, you dramatically inflate your false-positive rate — random walks cross the threshold temporarily all the time. Decide the sample size before the test, run to it, then evaluate once. Peeking-and-stopping is one of the most common ways teams ship changes that were actually noise.
Is significance the same as a meaningful result?
No. With a huge sample, a tiny, business-irrelevant difference can be highly significant; with a small sample, a large real effect can miss significance. Always look at the effect size (the actual uplift and its confidence interval), not just the p-value, and ask whether the change is big enough to matter. Statistical significance answers "is it probably real?", not "is it worth shipping?" — those are two different questions.