Statistical Significance in A/B Testing: Is 94% Enough?

Your testing app says 94%. Now what?

You wanted a yes or a no. You got a number one point short of the line everybody told you to wait for. So you'll check again tomorrow, hoping it ticks up to 95.

Don't. Here's what statistical significance AB testing tools are reporting, in plain words: how surprised you should be by the gap between two versions if they actually sold the same. That's it. It measures surprise. It doesn't measure size.

So is 94% enough? If the test hit its planned stop date and the change is cheap to undo, I'd ship it. If it's day four, or the change is your price, keep running. The number alone can't make that call. You make it, in dollars.

What is statistical significance, in store owner words?

Think of it as a smoke alarm.

The alarm tells you something's probably going on. It can't tell you if that's burnt toast or the whole kitchen. You still have to walk in and look.

A test works the same way. It starts by assuming your two versions are identical, then asks how often pure chance would produce a gap this big.

That answer comes out as a p value. In an AB test, a p value of 0.05 means a gap this big or bigger would show up 5% of the time with no true difference at all. Most dashboards flip it and print 95% confidence.

Significance tells you how surprised to be. Dollars tell you how much to care.

What does a p value in an AB test not tell you?

This is where founders get burned. The word "confidence" is doing a terrible job.

The American Statistical Association got so tired of the mix-ups that it published six principles on p values. Principle five is the one to tape to your monitor: statistical significance "does not measure the size of an effect or the importance of a result." Principle three says business decisions shouldn't rest only on whether a number passes a threshold.

What it feels like it means	What it actually means
95% chance the variant is better	A gap this size would be rare if the versions were identical
The lift on the dashboard is the lift you'll get	The lift is one noisy estimate inside a range
A bigger number means a bigger win	A bigger number means a gap that's harder to blame on chance, at any size
Not significant means the change did nothing	The test didn't gather enough evidence either way

Is 94% confidence enough to call a test?

First find out what your 94 is. Because the same data can print two different numbers.

Run the math on a hypothetical store. Two versions of a page selling a $70 weighted blanket, 10,000 visitors each. Control gets 200 orders, a 2.0% conversion rate. The variant gets 232 orders, a 2.32% conversion rate.

Now read that one test two ways:

• A probability-to-win tool. A basic Bayesian model with flat starting assumptions puts the variant at about 94% to be better. (Each vendor's model has its own settings, so yours may read a little differently.)
• A classic significance calculator. The p value is 0.12, which prints as 88% confidence. Check it yourself in Evan Miller's two-sample calculator.

Same orders. Same visitors. 94 or 88, depending on the question the tool is answering.

Neither one is wrong. Intelligems describes its model as Bayesian, and reports the probability a test group beats control on the metric you pick. Shoplift calls its number Probability to Win, and makes it hold at a level for multiple days before it moves a test to the next stage. I went deeper on the two camps in Bayesian vs frequentist A/B testing.

Here's the part nobody wants to hear. If you've been refreshing that number all week, 94% is worth less than it looks. Evan Miller showed that stopping the first time a test looks significant can push your false positive rate to 26.1%, instead of the 5% you think you're running. His same article says that if you plan to peek 10 times, you'd need a reading of 1.0% significance, which is 99% confidence, to earn a true 5%.

So before you argue about 94 versus 95, answer this: did you set a stop date, and did you reach it? If not, go read how long to run an A/B test and come back.

Can a 99.96% winner be worth less than a 94% one?

Yes. Watch.

Same $70 product, two hypothetical tests.

Test A is the one above. Control converts at 2.0% with a $70 average order value, so revenue per visitor is $1.40. On 10,000 visitors, that's $14,000. The variant converts at 2.32%, so revenue per visitor is $1.624. On the same 10,000 visitors, that's $16,240. Reads 94% on a probability-to-win tool.

Test B is on a store with far more traffic, 500,000 visitors per version. Control gets 10,000 orders at 2.0%. The variant gets 10,500 orders at 2.1%, so revenue per visitor goes from $1.40 to $1.47. On 10,000 visitors, that's $14,700 instead of $14,000. Classic confidence: 99.96%, and a probability-to-win tool would read higher still.

Test A: 2.0% to 2.32% $2,240

Test B: 2.0% to 2.1% $700

$70 average order value held flat in both. Test A reads 94% probability to win. Test B reads 99.96% confidence.

The louder alarm is the smaller fire. Pile on enough traffic and a tiny lift can clear the 95% bar, because big samples make small gaps surprising.

Now, $700 per 10,000 visitors can still pay on a store that size. I'd take it. But the 99.96% didn't tell you that. The math did.

And one trap inside that math. A conversion rate test says nothing about order size. A variant can win on conversion and lose on order value, so point the significance reading at the right metric. At RevenueFlows AI I judge every test on revenue per visitor, for reasons I laid out in revenue per visitor vs conversion rate.

How do you turn the number into a store decision?

Stop reading the single number. Read the range.

Most testing tools can show you a confidence interval. For Test A at 95%, the true difference could sit anywhere from -0.08 to +0.72 percentage points. Multiply by $70 and by 10,000 visitors:

• Worst plausible case: about -$580 per 10,000 visitors
• Best plausible case: about +$5,060 per 10,000 visitors

So the bet on the table is risking $580 to make up to $5,060. On a headline you can put back in 30 seconds, I'd take that all day. On a price increase across your whole catalog, I wouldn't. Same 94%. Different call.

The confidence number never knew what you were testing. You do.

Here's the table I'd use. These are my rules, not a law of statistics, so set yours before the test starts:

Reading	Stop date reached?	Cost of being wrong	Move
95% or higher	Yes	Any	Ship it, after checking the dollar range
90% to 94%	Yes	Small, easy to undo (headline, image order, section order)	Ship it and watch the next 30 days
90% to 94%	Yes	Big or hard to undo (price, bundle structure, theme rebuild)	Keep running in full weeks, or rerun it
Under 90%	Yes	Any	No proven winner. Log it, test something bolder
Any number	No	Any	Keep running. Only stop early for harm

Run your own version through this. It's set to Test A, so change the inputs to match your dashboard:

What does "not statistically significant" actually mean?

It means the test didn't gather enough evidence. That's all. A quiet alarm doesn't prove there's no fire. Maybe it's mounted too far from the kitchen.

Test A is a good example. Miller's rule of thumb for visitors per version is 16 x p x (1 - p) / (the change you want to detect) squared. On a 2.0% conversion rate, a move to 2.32% is a change of 0.0032. That's 0.3136 divided by 0.00001024, or about 30,625 visitors per version. Test A ran 10,000.

So at 88% confidence on a classic test, you don't have proof of a win. You also don't have proof of nothing. Analytics Toolkit lists that exact mistake among its common misreadings. If your traffic can't get there, testing on low traffic has the methods that work.

What does AI change about significance?

AI made variants cheap. It didn't make evidence cheap.

AI can write the copy, design the images and build a page in less than 15 minutes, so you can spin up 20 versions of a product page in a workday. And that's exactly how you manufacture a fake winner.

Here's the math. Convert lays out the formula for the chance of at least one false positive across many comparisons at a 5% threshold: about 22.6% for 5, 40.1% for 10, 64.2% for 20.

1 comparison 5.0%

5 comparisons 22.6%

10 comparisons 40.1%

20 comparisons 64.2%

5% threshold, independent comparisons, 1 - 0.95 to the power of the number of comparisons.

Test 20 versions that do nothing and you've got better than even odds of crowning one anyway.

Three fixes:

1. Test fewer, bolder versions. Two or three big swings beat 20 small rewrites, and a bigger gap needs far fewer visitors.
2. Raise the bar. The Bonferroni correction divides your threshold by the number of comparisons. 5% across 20 versions becomes 0.25% each, or 99.75% confidence. Some tools, like Convert, have a setting for this. Check yours.
3. Rerun the winner alone. A fake winner has to get lucky twice. At a 5% threshold, that's 0.25%.

Where AI genuinely helps is the reading. Paste your four numbers into whichever AI assistant you use, ask for a two-proportion test with a confidence interval, then ask it to put that interval in dollars at your average order value. Then check its answer against Miller's calculator, because I wouldn't bet a price change on an unchecked model either.

The stores that win here aren't the ones with the most tests. On one bedding client's Cooling Bamboo Sheets page, conversion rate went from 1.0% to 4.3% and average order value from $125 to $254. Revenue per visitor went from $1.25 to $10.92. On 10,000 visitors, that's $109,200 instead of $12,500. See the full case study numbers. Real client numbers, not typical results, and not a promise of what your store will do. (That's a before and after, not a split test. But a swing that size doesn't need an argument over one percentage point of confidence.)

What to do next

Open the test on your dashboard right now and answer one question: did it reach the stop date you set before launch? If you never set one, that's your next move. Calculate it, write it down, and leave the number alone until then. The full system around it lives in the ecommerce A/B testing guide.

---

Book Your Profit Audit

A close call at 94% often means the change was too small for your traffic to prove, and the product page is where the big swings live. Book a free profit audit and we'll show you where your revenue per visitor is leaking, then show you how to rebuild a high-converting product sales page in less than 15 minutes.

Book Your Profit Audit →

Or go here to check it out → revenueflows.ai

P.S. The alarm tells you there's smoke. It's still your job to go see if it's toast or the kitchen.