A/B Test Prioritization Framework: Score Ideas Before Testing

Your test backlog is a wish list. Forty ideas, ranked by whoever talked loudest in the last meeting.

And your traffic can only pay for about six of them this year.

So here's the ab test prioritization framework I'd hand any Shopify founder with 40 ideas and limited traffic. Score every idea before it touches a visitor:

1. Revenue impact (1 to 10): how much could this change what the buyer decides?
2. Confidence (1 to 10): what evidence says the problem is real?
3. Ease (1 to 10): how fast can you ship the variant?
4. Multiply the three, then multiply by distance to the buy (1.0 for the buy box, 0.1 for the blog).
5. Run the traffic gate: if the test can't finish in about eight weeks, make it bigger or skip it.

Highest score runs first. Everything else waits.

A test roadmap doesn't fail because the ideas are bad. It fails because the bad ideas go first and eat the traffic the good ones needed.

Why do you need a framework when most tests lose anyway?

Because most tests lose. That's exactly the reason.

Ronny Kohavi, who ran experimentation at Microsoft, has said that over two-thirds of ideas tested there failed to move the metric they were built to move. In the same interview he describes Airbnb testing 250 ideas and finding 20 winners. That's 8%.

Microsoft can burn traffic on misses. You can't.

Run the math on a store like this, a hypothetical: a skincare brand selling a $48 vitamin C serum. Its hero product page gets 9,000 visitors a month. Conversion rate 2.2%, average order value $52. That means revenue per visitor is $1.14. On 10,000 visitors, that's $11,440.

Split 9,000 monthly visitors across two versions and each one sees about 1,038 a week. Every week a weak test runs is a week the strong one sits in the backlog, bleeding.

I've watched founders spend a quarter of their testing year on a homepage carousel. Nobody scored it. It just felt important.

What's wrong with ICE score testing and the PIE framework?

Nothing, as a start. They're just missing two questions.

Here's how the three famous models stack up, according to Noble Studios' breakdown of prioritization frameworks and Mida's comparison of ICE, PIE and PXL:

Framework	Who made it	What it scores	The gap for a Shopify store
ICE	GrowthHackers team	Impact, confidence, ease, 1 to 10	All three are guesses
PIE	Popularized by Widerfunnel	Potential, importance, ease, 1 to 10	Weak on brand-new pages with no data
PXL	CXL	Weighted yes/no questions, summed	Slower, harder to explain to a team
Distance-to-buy score (this post)	Built for small stores	Impact x confidence x ease x distance, plus a traffic gate	Still needs honest scoring

Noble's sharpest line on ICE: assigning numbers to guesses "disguises subjective assessments as objective data points." That's fair. A 7 for confidence means nothing if nobody can say where the 7 came from.

And neither ICE nor the PIE framework in CRO asks the two questions that decide whether a small store wins:

• How close does this change sit to the moment someone pays?
• Can the test actually finish on my traffic?

Even Mida's hybrid model scores impact, evidence and ease. None of those asks whether the test can finish. That's the hole. Here's how to fill it.

How do you score each idea on revenue impact, confidence and effort?

Score each one from 1 to 10. Then multiply, don't add.

Here's why that matters. Adding lets a cheap, pointless idea sneak up the list on ease alone. Multiplying means a 2 anywhere drags the whole score down. A button color with impact 2, confidence 3 and ease 10 scores 60. A headline rewrite with impact 8, confidence 8 and ease 9 scores 576.

Same store. Same week of work. Nearly ten times the score.

Revenue impact asks one thing: does this change what the buyer gets, pays, or believes? A bundle in the buy box changes what they get. A price anchor changes what they pay. A first screen that answers "will this burn my sensitive skin?" changes what they believe. A font changes none of those.

Confidence has to come from evidence, never vibes. Give a 7 or higher only when something points at the problem: the top complaint in your reviews, a scroll map showing most mobile visitors never reach the price, a support inbox full of the same question. A finding from a conversion audit counts too, as long as it names the gap on the page. Mida suggests a hard rule worth stealing: any idea scoring below 4 on evidence drops out of the top quarter, whatever its total.

Ease is days to ship the variant. A 10 is a copy swap in the theme editor. A 3 is a new photo shoot.

If you can't name the evidence behind a confidence score, the score is a 3. Write it down and move on.

Why should the closest change to the buy decision go first?

Because a visitor on the buy box is one tap from paying. A visitor on your About page is six.

So the score gets one more multiplier, distance to the buy:

Where the change lives	Multiplier
Product page buy box, price, offer, first screen	1.0
Product page below the fold, cart, cart drawer	0.8
Collection pages	0.5
Homepage	0.3
Blog, About, popups on content pages	0.1

Now watch what happens to the hypothetical skincare store's backlog:

Idea	Impact	Confidence	Ease	Raw score	Distance	Final
First screen answers the sensitive-skin fear	8	8	9	576	1.0	576
Serum plus moisturizer two-pack in the buy box	9	7	6	378	1.0	378
Free shipping progress bar at $60 in the cart	7	6	8	336	0.8	268.8
Add to cart button color	2	3	10	60	1.0	60
Collection page filters by skin type	5	4	5	100	0.5	50
Homepage hero video	6	4	3	72	0.3	21.6
Blog exit popup with a quiz	4	5	9	180	0.1	18

The homepage video, the one everyone in the meeting loved, finishes sixth. The blog popup looked easy and dies at 18.

That's the polarity in one table. Testing the lobby versus testing the checkout counter.

Can the test finish on your traffic?

This is the gate ICE and PIE skip, and it's where small stores lose whole quarters.

Evan Miller's rule of thumb in How Not to Run an A/B Test says the visitors you need per version equal 16 times the variance, divided by the square of the smallest difference you want to detect, with the sample size fixed in advance. For a conversion rate, the variance is the rate times one minus the rate.

Here's the math on the hypothetical serum page, 2.2% conversion rate, about 1,038 visitors per version per week:

Lift you need to detect	Visitors per version	Weeks to finish
10% (2.2% to 2.42%)	71,127	68.5
30% (2.2% to 2.86%)	7,903	7.6
50% (2.2% to 3.3%)	2,845	2.7

Detect a 10% lift 68.5

Detect a 30% lift 7.6

Detect a 50% lift 2.7

Two versions, 2.2% baseline conversion rate, Evan Miller's 16 x variance rule of thumb.

The button color test can only hope for a small lift. That's 68.5 weeks. It fails the gate before it starts. The headline rewrite and the bundle are big swings, so they can finish in under eight weeks.

My rule: if the gate says more than about eight weeks, either make the idea bolder or ship it without a test and watch revenue per visitor. The full traffic playbook is in how to A/B test on low traffic.

How is AI changing test prioritization?

It mostly changes two of the three scores.

Confidence used to be the expensive one. You needed a researcher to read 400 reviews. Now you can paste those reviews into any chat assistant and ask for the five objections buyers mention most, ranked by how often they show up. That turns a guessed 5 into an evidenced 8 in an afternoon. Testing apps are building this in too. Intelligems says its Atlas AI "surfaces high-impact experiment ideas based on your data", and recommends what to test next.

Ease is the other one. A full challenger page used to score a 2. With RevenueFlows AI you drop in a product URL and the AI writes the copy, designs the images and builds the page in less than 15 minutes. The bold test stops being the slow test.

But here's the thing. AI can fill your backlog with 200 ideas by lunch. It can't tell you which one sits closest to the buy. That call is still yours.

More ideas was never the problem. A store with 40 ideas and no scoring has the same problem as a store with 400.

What happens when the top-scored test wins?

Run the math on the hypothetical store if the first two tests both win. Conversion rate moves from 2.2% to 2.8%. Average order value moves from $52 to $63 with the two-pack. Revenue per visitor goes from $1.14 to $1.76. On the same 10,000 visitors, that's $17,640 instead of $11,440, a gap of $6,200 a month.

Before $11,440

After $17,640

Conversion rate x average order value x visitors. Before 2.2% x $52. After 2.8% x $63.

Now look at what a full rebuild does when you stop testing pieces and change everything close to the buy at once. Real client numbers, not typical results, and not a promise of what your store will do. On one bedding product, Cooling Bamboo Sheets, conversion rate went from 1.0% to 4.3% and average order value went from $125 to $254. Revenue per visitor went from $1.25 to $10.92. On 10,000 visitors, that's $109,200 instead of $12,500. You can see the full case study numbers.

That's the same principle at full volume. Every change sat on the product page.

Judge every winner on revenue per visitor, never conversion rate alone. The reason is in revenue per visitor vs conversion rate. For the full testing system, start with the ecommerce A/B testing guide. If your list is thin, borrow from 12 ecommerce A/B test ideas ranked by offer, then set up the winner with how to A/B test your Shopify product page.

What to do next

Open a blank sheet tonight. List every test idea you've got, then score just the top 10 you'd have run anyway. Multiply them out. If your number one isn't on the product page, ask yourself why it's first.

---

Book Your Profit Audit

A test roadmap is only as good as the idea at the top, and the top idea almost always lives on the product page. Book a free profit audit and we'll find where your revenue per visitor is leaking, then show you how to rebuild a high-converting product sales page in less than 15 minutes.

Book Your Profit Audit →

Or go here to check it out → revenueflows.ai

P.S. Testing the lobby feels busy. Testing the checkout counter pays the bills. Score your list and let the math pick.