Multi-Armed Bandit vs A/B Test: Which Wins for Shopify?
A bandit earns more while the test runs. A fixed split teaches you more when it ends. Here's the math on a 7-day promo, and the one setting that decides whether the bandit makes you money or quietly loses it.
Every testing app now has a toggle that says it'll pick the winner for you.
Founders ask me if they should flip it on. My answer on the multi armed bandit vs ab testing question is short: a bandit earns, a fixed split learns. Pick the one that matches what you need from the next seven days.
A classic A/B test holds traffic at 50/50 until it reaches its sample size, then tells you with confidence which version is better. A multi-armed bandit starts at 50/50 and keeps moving visitors toward whichever version is winning while the test is still live. You make more money during the test. You get a fuzzier answer at the end.
So use a bandit for short windows where the money is now: a promo week, a launch, Black Friday. Use a fixed split for anything you plan to keep: a headline, a price, a new page.
A bandit is a cash register. An A/B test is a textbook. Don't ask the register to teach you, and don't ask the textbook to pay rent.
What does a bandit actually do with your traffic?
Here's the thing most app listings skip. "Auto-allocate" isn't magic. It's a decision rule running every few hours.
The most common rule is Thompson sampling. For each version, the tool keeps a range of believable conversion rates based on what it's seen so far. Every time a visitor arrives, it draws a guess from each range and sends the visitor to the version with the highest guess. Winners draw high guesses more often, so they get more visitors. Losers still get a trickle, in case they were just unlucky early.
Convert documents three rules you can choose from in its bandit auto-allocation settings: Thompson sampling (the default), epsilon-greedy and upper confidence bound. Wingify, the company behind VWO, says its bandit uses an epsilon-greedy layer on top of Thompson sampling, and describes the trade in one line: it suits optimizers who'd rather maximize a metric in a short time and can give up on statistical significance.
Adobe Target's version keeps a permanent exploration slice. According to Adobe's Auto-Allocate documentation, 20% of visitors are always assigned at random across every version, and the rest follow the leader.
That's the whole machine. Now watch what it does to a week of traffic.
How much more does a bandit earn during a 7-day promo?
Run the math on a store like this, a hypothetical. A skincare brand runs a 7-day bundle promo. 2,000 visitors a day, 14,000 total. Two versions of the promo page.
- Version A: conversion rate 2.0%, average order value $70. That means revenue per visitor is $1.40.
- Version B: conversion rate 2.5%, average order value $72. That means revenue per visitor is $1.80.
A fixed split never changes. A bandit starts even and leans toward B as the orders come in.
Fixed split holds 50% all week. The bandit row shows one plausible path as B pulls ahead.
Here's the math.
The fixed split sends 7,000 visitors to each version. A gets 140 orders at $70, which is $9,800. B gets 175 orders at $72, which is $12,600. Total: $22,400.
The bandit's daily shares add up to 10,000 visitors on B and 4,000 on A. B gets 250 orders at $72, which is $18,000. A gets 80 orders at $70, which is $5,600. Total: $23,600.
Version A: 2.0% x $70. Version B: 2.5% x $72. Bandit sends 10,000 of 14,000 visitors to B.
The bandit made $1,200 more from the same 14,000 visitors. Not a fortune. But look at it another way. If you'd known the winner on day one, you'd have made $25,200. The fixed split paid $2,800 to learn. The bandit paid $1,600. It cut the cost of learning by 43%.
And on a promo that ends Sunday, you don't need a lesson. You need the money.
When does a classic A/B test beat a bandit?
Whenever you're about to make a change you'll live with for a year.
Optimizely is blunt about this. Its documentation says bandits do not generate statistical significance and do not use a control. The report shows improvement over equal allocation instead. Adobe's own comparison of the two methods lists weaker statistical guarantees and unclear stopping rules as the bandit's main limits.
That matters because the losing version barely gets traffic. By day 7 in the example above, A was seeing 200 visitors a day. You can't tell from that whether A lost by 10% or 40%. You can't tell whether A would win with returning customers, or on mobile, or in January.
I'll admit I used to run bandits on everything. It felt efficient. Then a client asked me why we'd switched their headline, and all I had was "the algorithm sent traffic to it." That's not an answer you bet twelve months of revenue on.
The bandit tells you where the money went this week. Only a fixed split tells you why.
So for a new headline, a price change, a page rebuild or anything else you'll keep: hold the split, hit your sample size, read the result. The ecommerce A/B testing guide walks through how much traffic that takes, and how to A/B test a Shopify product page shows the setup step by step.
Which metric should the bandit chase?
This is where bandits quietly lose money, and almost nobody checks.
Most bandits chase conversion rate by default. Convert's settings list conversion rate as the default goal, with revenue per visitor as an option you have to pick. And once the bandit is live, you can't change the goal. Optimizely says the same: after you start, you can't change the primary metric.
Here's why that matters. Say you add Version C to the promo above, a hypothetical with a cheaper single-product offer. Conversion rate 2.8%, average order value $48. That means revenue per visitor is $1.34. Version A, at 2.0% and $70, earns $1.40.
A bandit chasing conversion rate sees C winning and pours traffic into it. On 10,000 visitors, C earns $13,440. A would have earned $14,000. The algorithm did exactly what you told it to, and it cost you $560.
The store didn't need a smarter algorithm. It needed the right target.
Intelligems gets this right in its guide to building a bandit on its API: the workflow compares versions on revenue per visitor. Worth knowing, though, that this is a do-it-yourself automation through their API, not a toggle in the app. We break down the rest of the differences in Shoplift vs Intelligems, and why conversion rate alone lies in revenue per visitor vs conversion rate.
Multi armed bandit vs ab testing: which one for which job?
| Situation | Use | Why |
|---|---|---|
| 7-day promo, launch week or flash sale | Bandit | The window closes before a fixed test finishes. Earn while it runs. |
| Black Friday to Cyber Monday offer test | Bandit, with a fast refresh | Four days. The money is now. |
| New headline or product page rebuild | Fixed A/B test | You'll keep it for months. You need proof. |
| Price change | Fixed A/B test | Price touches margin on every future order. |
| Always-on offer, like a free gift threshold | Bandit | Conditions keep shifting and you never "finish." |
| Three or more versions of one idea | Bandit first, then a fixed test of the top two | Cut the losers cheaply, then prove the winner. |
Why can a bandit fail on Black Friday?
Two reasons, both about time.
First, refresh speed. Optimizely says its bandit model updates hourly. The Intelligems API guide waits a minimum of 72 hours before moving any traffic, then checks every 4 hours. On a 4-day sale from November 27 to November 30, a 72-hour wait means the bandit only acts on Cyber Monday. Check how often your tool reallocates before you trust it with a short window.
Second, shoppers change by the hour. Adobe warns that conversion rates that shift with time of day or context can skew where a bandit sends traffic. Black Friday morning shoppers aren't Cyber Monday evening shoppers. A bandit that locks onto a Friday winner can keep feeding it on Monday after it's stopped winning. That's why Adobe keeps 20% exploring. Pick a tool that keeps some traffic on every version.
We cover the full holiday plan, including what to freeze, in A/B testing during Black Friday.
What's the bigger risk than picking the wrong method?
Testing two weak pages against each other.
A bandit can only route traffic between the versions you give it. If both pages are spec sheets, the bandit finds the slightly less dull spec sheet, faster. That's it.
Real client numbers, not typical results, and not a promise of what your store will do: a bedding brand we worked with started at a conversion rate of 1.0% and an average order value of $125. That means revenue per visitor was $1.25. On 10,000 visitors, that's $12,500. After the page rebuild, conversion rate was 3.5% and average order value was $231. Revenue per visitor: $8.10. On the same 10,000 visitors, that's $81,000. You can see the full case study numbers.
No traffic split found that. A better page did. The test just confirmed it.
AI is what changed the math here. Building a strong challenger used to take weeks of copy and design, so stores tested button colors because that's all they could afford to build. Now you can rebuild a full product page in under 15 minutes, which means the thing you put into the test can finally be worth testing. If you're picking an app to run it, the best Shopify A/B testing apps ranks them by how honestly they report. And if you only want to compare two whole themes, Shopify Rollouts now lets you A/B test two themes or checkout setups natively.
What to do next
Open your testing app and find the one test you're planning for November. Ask one question: will I keep this change after the sale? If no, flip it to a bandit and set the goal to revenue per visitor. If yes, run it now as a fixed split, so it finishes on normal traffic before the holiday rush muddies everything.
Book Your Profit Audit
Before you split traffic between two versions, make sure the page you're testing can actually sell. Book a free profit audit and we'll show you where your revenue per visitor is leaking, then show you how to rebuild a high-converting product sales page in less than 15 minutes.
Or go here to check it out → revenueflows.ai
P.S. A bandit earns this week. A fixed test teaches you for next year. Neither one can rescue a page that was never built to sell.
Frequently asked questions
What is the difference between a multi-armed bandit and an A/B test?
An A/B test holds a fixed split, usually 50/50, until it hits its sample size, so you learn how much better one version is. A multi-armed bandit shifts traffic toward the version that's winning while the test runs, so you lose fewer sales during the test but get a weaker answer at the end.
Is a multi-armed bandit better than A/B testing for Shopify?
For short windows like a 7-day promo or Black Friday weekend, yes, because a bandit earns more while it runs. For permanent changes like a new headline, price or page layout, no. Use a fixed split so you get a result you can trust for the next twelve months.
Do multi-armed bandits give statistical significance?
Usually not. Optimizely states plainly that its bandits do not generate statistical significance and report improvement over equal allocation instead. If you need proof to make a permanent change, run a classic A/B test.
What metric should a bandit test use on an ecommerce store?
Revenue per visitor, which is conversion rate times average order value. A bandit set to conversion rate can pour traffic into a version that converts at 2.8% with a $48 order ($1.34 per visitor) and away from one converting at 2.0% with a $70 order ($1.40 per visitor).
How much traffic do you need for a bandit test?
Enough for the tool to see a difference before your window closes. Some setups wait days before moving any traffic, so check how often your tool reallocates. On a 4-day sale, a tool that waits 72 hours only gets one day to earn.