Bayesian vs Frequentist A/B Testing for Store Owners
Your testing app says 93% probability to win. Your old calculator says not significant. Same data. Here's what each number answers, and which one matches the decision you're about to make.
Here's a hot take most statisticians won't love: for a store owner, the bayesian vs frequentist ab testing debate is a question about questions.
A frequentist test answers: "If these two pages were identical, how weird would this data look?" A Bayesian test answers: "What's the chance the new page is better?"
Only one of those is the question you're asking when you decide what to ship. It's the second one.
So when your testing app shows "probability to beat control: 93%," it means this: given the data so far, there's about a 93% chance the variant really is better on the metric you picked, and about a 7% chance it isn't. That's a decision number. A p-value is a courtroom number. It tells you whether you can reject "no difference," and nothing about how likely you are to be right if you ship.
A p-value tells you whether to be surprised. A probability to beat control tells you whether to ship. You run a store, not a journal.
What does each method actually tell you?
Think of it as two different employees reporting on the same test.
The frequentist analyst starts by assuming nothing changed. Then she asks how often you'd see a gap this big by pure luck. If that's under 5% of the time (p below 0.05), she calls it significant. If not, she says "inconclusive" and walks away. Her answer is yes or no.
The Bayesian analyst starts with a neutral guess and updates it as orders come in. She hands you a probability, a likely range for the lift, and sometimes the money you stand to lose if you pick wrong. Her answer is a bet with odds attached.
Same data. Different report. And here's the part almost every explainer skips: in a simple test like the one below, the numbers are close cousins. With a flat starting guess, a one-sided p-value of about 0.07 sits right next to a 93% probability to beat control. The math agrees. The framing doesn't.
Same test, two verdicts: which one do you trust?
Run the math on a store like this, a hypothetical. A candle brand tests a new product page. 10,000 visitors see each version.
- Control: 200 orders. Conversion rate 2.0%, average order value $80. That means revenue per visitor is $1.60. On 10,000 visitors, that's $16,000.
- Variant: 230 orders. Conversion rate 2.3%, average order value $80. That means revenue per visitor is $1.84. On the same 10,000 visitors, that's $18,400.
Conversion rate x average order value x visitors. Control: 2.0% x $80. Variant: 2.3% x $80.
Now the two reports.
The frequentist calculator runs a standard two-sided test and spits out a p-value of 0.14. Not significant. Verdict: inconclusive, keep testing or give up.
The Bayesian model, fed the exact same 430 orders, says the variant has a 93% probability to beat control. The likely lift runs from about 5% worse to about 39% better. Wide. But tilted hard toward better.
Here's the thing. The frequentist "inconclusive" doesn't mean the pages are equal. It means the test didn't clear a 95% bar that was set before anyone thought about your margins. A lot of founders read p = 0.14 and conclude "no difference." That's the wrong read, and it kills good pages.
Why is the Bayesian answer the one you actually act on?
Because a store owner's real question has a price tag on it.
Watch what happens when you add expected loss, the number Bayesian tools like VWO lean on. In the hypothetical above, if you ship the variant and you're wrong, the average cost works out to about 0.66 orders per 10,000 visitors. At $80 an order, that's roughly $53.
If you keep the control and the variant was better, the average cost is about 31 orders per 10,000 visitors. That's roughly $2,452.
$53 of downside versus $2,452 of upside left on the table. You don't need a 95% bar to make that call. You need the odds and the stakes, and the Bayesian report hands you both.
Wingify, the company behind VWO, builds its winner rule around exactly this. Its documentation on how it calculates a winning variation says a variation becomes a "Smart Decision" when its potential loss is below the threshold of caring and its chance to beat baseline is 95% or more. Odds plus stakes.
I'll be honest about my bias. I've watched more stores lose money by throwing out a 90%-plus winner as "inconclusive" than by shipping a false winner. Small stores feel this worst. At 10,000 visitors a version, a fixed 95% test can take months to call a real 15% lift. Months of the weaker page earning less, on every visitor.
Being stuck at "inconclusive" for 90 days costs money too. Nobody puts that line on the report.
Bayesian vs frequentist ab testing: the side-by-side
| Frequentist | Bayesian | |
|---|---|---|
| The question it answers | If nothing changed, how surprising is this data? | What's the chance B beats A? |
| What you see in the app | p-value, "statistically significant," confidence interval | Probability to beat control, probability to win, expected loss |
| Sample size | Set it before launch and stick to it (classic method) | No fixed number, but tools set minimums |
| Checking results early | Inflates false winners unless the tool uses sequential methods | Safer to look, still risky to stop the second it crosses a line |
| Small store, under 20,000 visitors a month | Often stuck on "inconclusive" | Gives a usable bet with the risk spelled out |
| Best for | Big permanent bets where a false winner is expensive | Everyday page, offer and layout decisions |
| How to explain it to your team | "We can reject no difference at 95%" | "93% chance it's better, about $53 at risk per 10,000 visitors" |
Which testing tools use which method?
I checked each vendor's own docs before writing this, because app listings blur it.
| Tool | Method (per its docs) | What it shows you | Stated rule or minimum |
|---|---|---|---|
| Shoplift | Bayesian | Probability to Win | Must hold for multiple days before a result stage is assigned |
| Intelligems | Bayesian with Monte Carlo simulation | Probability to beat control | 300+ orders per group and 7+ days |
| VWO (Wingify) | Bayesian (SmartStats) | Chance to beat baseline, potential loss | 95%+ chance and potential loss below threshold |
| Convert | Frequentist by default, plus Bayesian and sequential modes | p-values, or chance to win | Bayesian threshold defaults to 95% |
| Optimizely | Frequentist sequential (Stats Engine) | Significance and confidence intervals valid whenever you check | Controls false discovery rate across metrics |
Sources: Shoplift explains that a Probability to Win of 92 means the variant comes out ahead in about 92 of every 100 likely outcomes. Intelligems states its Bayesian model and the 300-order, 7-day minimums. Convert's statistical methods page lists all three engines with frequentist as the default. Optimizely's Stats Engine guide describes its core method as frequentist sequential testing.
For the full app-by-app breakdown, see the best Shopify A/B testing apps, ranked by rigor and the head-to-head in Shoplift vs Intelligems.
Can you stop a Bayesian test the moment it hits 95%?
This is where Bayesian fans oversell it.
The frequentist peeking problem is real. Evan Miller's classic essay on how not to run an A/B test shows a test on a change that does nothing. Stop it as soon as it hits 5% significance, or at 150 observations, and it wrongly finds a winner 26.1% of the time. More than five times the error rate you thought you signed up for. Miller also writes that with Bayesian design you can stop any time and make valid inferences.
But "valid" isn't the same as "safe from noise." A probability can spike to 96% on day two because 14 people bought a bundle from one email blast. That's why the vendors build in brakes. Shoplift makes the reading hold for multiple days. Intelligems says plainly that hitting its minimums doesn't guarantee significance. And the Mida glossary entry on chance to win names the most common mistake: reading chance to win as certainty.
So my rule: look whenever you want. Decide only after a full week, your tool's minimum orders, and a reading that's held steady for several days. We held our own numbers to the same rule in this ecommerce case study, where the first-week conversion rate was the best number and not the true one.
Which metric should the probability be about?
Wait. Before you trust any 93%, check what it's 93% about.
Most tools default to conversion rate. A variant can win on conversion rate and lose on money. A hypothetical: version B converts at 2.6% with a $58 average order value, so revenue per visitor is $1.51. Version A converts at 2.0% with an $80 average order value, so revenue per visitor is $1.60. B "wins" with high probability on conversion rate, and costs you $920 on every 10,000 visitors ($15,080 instead of $16,000).
Set the primary metric to revenue per visitor wherever your tool allows it. We break down why in revenue per visitor vs conversion rate. And if your tool is shifting traffic toward the leader while the test runs, that's a different machine with its own rules, covered in multi-armed bandit vs A/B testing.
What's the bigger lever than the stats engine?
What you put into the test.
A perfect Bayesian model comparing two spec-sheet pages will tell you, with 97% probability, which spec sheet is slightly less dull. Small lifts mean wide ranges and long waits, no matter which statistics you pick.
Real client numbers, not typical results, and not a promise of what your store will do: on one bedding product, Copper Bamboo Sheets, conversion rate went from 1.0% to 4.3% and average order value from $125 to $221. Revenue per visitor went from $1.25 to $9.50. On 10,000 visitors, that's $95,000 instead of $12,500. You can see the full case study numbers.
A gap that big doesn't need a statistician to call it. Any method sees it in days.
That's what AI changed. Building a strong challenger page used to take weeks of copy and design, so stores tested button colors because that's all they could afford to build. Now a full rebuild takes under 15 minutes, so the variant you test can be a real swing. The ecommerce A/B testing guide covers how to plan those swings and how much traffic each one needs.
What to do next
Open your testing app right now and find the last test you called "inconclusive." Look for the probability to beat control, or ask your tool for it. If it's above 90% and the expected loss is small, you may have shelved a winner. Re-run it with revenue per visitor as the goal, give it a full week, and decide on the odds.
Book Your Profit Audit
The stats engine decides how sure you are. The page decides how much there is to be sure about. Book a free profit audit and we'll show you where your revenue per visitor is leaking, then show you how to rebuild a high-converting product sales page in less than 15 minutes.
Or go here to check it out → revenueflows.ai
P.S. A p-value asks whether you should be surprised. A probability asks whether you should ship. Only one of those pays the rent.
Frequently asked questions
What is the difference between Bayesian and frequentist A/B testing?
Frequentist testing asks how surprising your data would be if the two versions were identical, and reports a p-value. Bayesian testing asks how likely it is that the new version beats the old one, and reports a probability like 93%. For a store owner, the Bayesian number maps straight onto the decision you're making.
What does probability to beat control mean?
It's the chance, given the data so far and the tool's model, that the variant is better than the control on the metric you picked. Intelligems defines it as the probability that a test group is better than the control group on the selected metric. It's a probability, so a 93% reading still loses about 7 times out of 100.
Is Bayesian A/B testing better than frequentist?
Neither is better at everything. Bayesian results are easier to act on and handle small stores better because they tell you how much you risk by shipping. Frequentist results with a fixed sample size give stronger protection against false winners when you need proof for a big permanent change.
Can I stop a Bayesian A/B test early?
You can look at it any time, but stopping the moment it crosses 95% still invites false winners from early noise. That's why Shoplift requires its Probability to Win to hold for multiple days and Intelligems recommends 300+ orders per group and 7+ days before you call it.
Which Shopify testing apps use Bayesian statistics?
Shoplift and Intelligems both use Bayesian models according to their own documentation. Convert defaults to frequentist but offers Bayesian and sequential modes. Optimizely uses frequentist sequential testing in its Stats Engine.