Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path
A/B Test Significance Calculator
An A/B test significance calculator using a two-proportion z-test. Enter the visitors and conversions for a control and a variant to get each conversion rate, the relative uplift, the z-score and two-tailed p-value, and a clear verdict on whether the result is statistically significant. Explains what significance does and does not tell you.
Control (A)
4.00%
Variant (B)
5.25%
An A/B test never hands you a clean answer. It hands you two noisy conversion rates and a question: is the gap real, or is it the same coin landing heads a few more times? This calculator answers that with the standard two-proportion z-test and shows every intermediate number, so the verdict is never a black box.
What it computes
Each group is a count of visitors n and a count of conversions c. The test measures how far apart the two rates are in units of their combined sampling noise:
p_A = c_A / n_A control conversion rate
p_B = c_B / n_B variant conversion rate
uplift = (p_B − p_A) / p_A relative change vs control
p_pool = (c_A + c_B) / (n_A + n_B)
SE = √( p_pool × (1 − p_pool) × (1/n_A + 1/n_B) )
z = (p_B − p_A) / SE
p = 2 × (1 − Φ(|z|)) two-tailed p-valuep_pool is the conversion rate you would expect in both groups if the variant changed nothing, which is the null hypothesis being tested. SE is the standard error of the difference under that assumption. Φ is the standard normal cumulative distribution, which the tool evaluates through a polynomial approximation of the error function accurate to within 0.00000015. The result is flagged significant when p is below 0.05.
A worked example
The defaults are 4,000 visitors with 160 conversions in control and 4,000 visitors with 210 conversions in the variant.
- Rates:
160 / 4,000 = 4.00%and210 / 4,000 = 5.25%, an absolute gap of 1.25 percentage points. - Uplift:
(0.0525 − 0.04) / 0.04 = 31.25%. Floating-point arithmetic lands a hair under that, so the widget prints 31.2%. - Pooled rate:
370 / 8,000 = 0.04625. - Standard error:
√(0.04625 × 0.95375 × 0.0005) = √0.00002206 = 0.004696. - z-score:
0.0125 / 0.004696 = 2.66. - p-value:
2 × (1 − Φ(2.66)) = 0.0078, below 0.05, so the verdict is significant.
The same two rates tell a different story at other sample sizes, which is the whole reason the test exists:
| Visitors per arm | Conversions A / B | z | p-value |
|---|---|---|---|
| 800 | 32 / 42 | 1.19 | 0.2339 |
| 2,000 | 80 / 105 | 1.88 | 0.0598 |
| 4,000 | 160 / 210 | 2.66 | 0.0078 |
| 8,000 | 320 / 420 | 3.76 | 0.0002 |
Assumptions and limits
- Independent, binary outcomes. Each visitor is counted once and either converts or does not. Revenue per visitor, or several conversions per user, needs a different test.
- Large enough counts. The normal approximation is reliable when each group has roughly ten or more conversions and ten or more non-conversions. With fewer, use an exact test.
- One fixed-horizon comparison. The test assumes the sample size was chosen in advance and read once, with one variant against one control. Testing several variants or metrics raises the chance that one looks significant by luck.
- Fixed threshold. The cut-off is 0.05, two-tailed, and cannot be changed in the widget. It does not compute power, a confidence interval or a required sample size.
- Valid inputs. Conversions must not exceed visitors; if they do, the rates pass 100% and the result is meaningless.
How to use the result
A significant p-value says the difference is unlikely to be noise if the variants were truly identical. It does not say there is a 95% chance the variant wins, and it says nothing about size. In the example, a quick 95% interval on the absolute gap runs from about 0.33 to 2.17 percentage points, so the true relative lift could plausibly be anywhere from roughly 8% to 54%. Plan from the low end, not the headline 31%.
- Size the test first. For 80% power at the 5% level, a common rule is
16 × p × (1 − p) / δ²visitors per arm, whereδis the absolute lift you care about. For this example that is about 3,900, which is why 4,000 per arm was just enough. - Ship when the result is significant, the effect is worth the cost of the change, and guardrail metrics such as refunds or latency did not get worse.
- Inconclusive is an answer. Do not keep adding traffic until the number crosses 0.05. Either run to a larger sample you committed to in advance, or record that the change made no detectable difference.
How it works
- Pooled two-proportion z-test on control vs variant conversions.
- Shows both conversion rates, relative uplift, z-score and two-tailed p-value.
- Flags significance at p < 0.05, the 95% confidence level.
- Reminds you that significance ≠ a meaningful effect size.
Frequently asked questions
What does statistical significance mean here?
It asks: if the two variants were actually identical, how often would random chance alone produce a difference at least this large? That probability is the p-value. The usual threshold is 0.05, and below it the result is called statistically significant, meaning the gap is hard to explain as noise at your sample size. It is not the probability that the variant is better, and it is not proof of a large effect. It is evidence that the observed difference is unlikely to be pure chance.
How is the p-value computed?
The calculator runs a pooled two-proportion z-test. It pools both groups into one conversion rate, uses that to compute the standard error of the difference between the two rates, and divides the observed difference by it to get a z-score. The two-tailed p-value is the probability of a z-score at least that extreme in either direction under the standard normal curve, which is 2 times (1 minus the normal CDF of the absolute z). It is the standard test for two conversion rates with reasonably large samples.
How many visitors do I need for an A/B test?
A common rule of thumb for 80% power at a 5% significance level is about 16 times p times (1 minus p), divided by the squared absolute difference you want to detect, per variant. For a 4% baseline and a lift to 5.25%, that is 16 times 0.0384 divided by 0.0125 squared, or roughly 3,900 visitors per arm. Halving the lift you want to detect quadruples the sample, which is why small improvements need very large tests. Fix the number before the test starts.
Why can a result flip if I keep peeking?
Because significance testing assumes you fixed your sample size in advance. If you check the p-value repeatedly and stop the moment it dips below 0.05, you inflate the false-positive rate well above 5%, since a random walk crosses the threshold temporarily all the time. Decide the sample size before the test, run to it, then evaluate once. If you need to monitor continuously, use a sequential method designed for it rather than re-reading this fixed-horizon test every day.
Is this a one-tailed or two-tailed test?
Two-tailed. The p-value counts differences as extreme as yours in either direction, so it asks whether the variant is different, not only whether it is better. That is the safer default, because a change can hurt conversion as easily as help it. A one-tailed test would halve the p-value for a lift in the expected direction, but it should only be chosen before the data arrive, never after seeing which way the result went.
Is significance the same as a meaningful result?
No. With a huge sample, a tiny, business-irrelevant difference can be highly significant; with a small sample, a large real effect can miss significance. Always look at the effect size, meaning the actual uplift and the range it plausibly covers, not just the p-value, and ask whether the change is big enough to matter. Statistical significance answers whether the gap is probably real, not whether it is worth shipping, and you need both answers before you decide.