CRO Agency Case Studies: What to Ask Before You Hire
A 150% lift means nothing until you know what it was 150% of. Here's how to read CRO agency case studies like game tape, and the questions that expose a highlight reel.
A 150% lift means nothing on its own.
150% of what? On how many visitors? Over how many days? And what did they change?
Here's the short answer. CRO agency case studies are only useful when they show four things: the baseline (conversion rate and average order value before), the traffic the test ran on, the time frame, and exactly what was changed. Miss any one of those and you can't check the claim. You're watching a highlight reel. What you want before you hire anyone is the game tape, and the checklist at the bottom of this page is how you ask for it.
I've read a lot of agency decks. The pretty ones are usually the thinnest.
Why do most CRO agency case studies tell you nothing?
Because a case study is a sales asset first. Nobody puts their losing tests on the website.
That's fine. Every coach shows the dunk, not the airball. But a highlight reel tells you the player can score. Game tape tells you how often, against who, and whether it holds up in the fourth quarter.
Run the math on a deck like this (a hypothetical, but you've seen this exact slide). "Conversion rate up 150% in 9 days." Sounds huge. Now ask for the inputs:
- The test page got 1,500 visitors.
- Before: conversion rate 0.4%. That's 6 orders.
- After: conversion rate 1.0%. That's 15 orders.
Nine extra orders. At a $60 average order value, revenue per visitor went from $0.24 to $0.60, and the whole "150%" is $540 of revenue. Nine orders can swing on one good email send or one influencer story.
Now compare a quieter slide from a different, also hypothetical, agency. "Conversion rate up 18% over 6 weeks." Inputs: 120,000 visitors, 2.0% to 2.36%, 2,400 orders to 2,832. At an $80 average order value, revenue per visitor moved from $1.60 to $1.89. On those 120,000 visitors, that's $192,000 becoming $226,560, a gap of $34,560.
The smaller percentage is the stronger evidence. By a mile.
A big percentage on small traffic is a coin flip with a press release. A small percentage on big traffic is a business result.
| The slide says | The inputs | Extra orders | What it proves |
|---|---|---|---|
| +150% in 9 days | 1,500 visitors, 0.4% to 1.0% | 9 | Almost nothing yet |
| +18% over 6 weeks | 120,000 visitors, 2.0% to 2.36% | 432 | A real, repeatable lift |
What should a real CRO case study show?
Four numbers, minimum. If an agency's case study has these, it's game tape. If it doesn't, ask for them on the call.
- The baseline. Conversion rate and average order value before anything changed.
- The traffic. Visitors or sessions on the tested page, both periods.
- The time frame. Start date, end date, and why it stopped then.
- The change. What was actually different on the page. A new hero, a new offer and a whole new format like a listicle landing page are three very different claims.
Then one more thing I always look for: the result reported per visitor. Conversion rate alone lies. A page can convert more people into smaller orders and make less money. Revenue per visitor (conversion rate times average order value) catches both levers at once, which is why I wrote a whole breakdown on revenue per visitor vs conversion rate.
And check the definition. Shopify's own behaviour reports count conversion rate as the percentage of sessions that end in a purchase. Some agencies quote add-to-cart rate or "micro-conversions" in the same font size. Different number. Ask which one it is.
How do you evaluate a CRO agency's results in 10 minutes?
Take their best case study. The one they lead with. Then run it through this table.
| What they claim | Ask for | Good answer | Red flag |
|---|---|---|---|
| "+X% conversion rate" | Visitors and orders, before and after | Exact counts, both periods | "We don't share client data" for every single case |
| "Revenue up $X" | Same traffic window, before and after | Per-visitor numbers | A monthly total during a sale month |
| "Tested for 2 weeks" | How the stop date was chosen | Picked before launch | "When it hit significance" |
| "90% win rate" | How many tests lost last year | A real number of losers | Nearly every test wins |
| "Ran a full redesign" | What exactly changed | A specific list | "Everything, it was a full refresh" |
That third row matters more than it looks. Evan Miller's classic piece on how not to run an A/B test shows that if you keep checking a test and stop the moment it looks significant, a result reported at 5% significance can carry a real false positive rate of 26.1%. So "we stopped when it won" is a confession, not a method.
The fourth row too. An agency-selection guide from upGrowth puts typical test win rates between 20% and 40% and flags 80%+ claims as a warning sign. Losing tests are normal. An agency with no losers is either testing button colors or hiding the tape.
What questions should you ask a CRO agency on the sales call?
These are the ones I'd ask. Short, specific, hard to dodge.
- "In your lead case study, what were the conversion rate and average order value before you started?"
- "How many visitors did that test run on?"
- "How did you decide when to stop it?"
- "How many of your tests lost or came back flat last quarter?"
- "Did the lift hold 60 days later?"
- "Was anything else running at the same time? A sale, a new ad campaign, a price change?"
- "What will you measure on my store, and what's the number we judge you on?"
Watch the answers, but watch the pause more. A team that runs real tests answers these from memory. A team selling a highlight reel starts talking about "holistic strategy."
Also be careful with anyone promising a fixed uplift before they've seen your numbers. The team at Blend Commerce makes the same point in their guide to vetting CRO agencies. Your traffic, product and price aren't the stores in their portfolio.
The agency that shows you its losing tests is the one that actually runs tests.
For a deeper look at running the test yourself so you can check any agency's work, here's how to A/B test a Shopify product page.
What does our own game tape look like?
Fair question. I'd be a hypocrite to write this and hide ours.
A Shopify bedding brand with 30+ products, stuck around $15,000 a month. We rebuilt the pages for their top 3 hero products.
Before: conversion rate 1.0%, average order value $125. That means revenue per visitor was $1.25. On 10,000 visitors, that's $12,500.
After: conversion rate 3.5%, average order value $231. Revenue per visitor $8.10. On the same 10,000 visitors, that's $81,000. A gap of $68,500 a month, and a 6.5x lift per visitor. The banked number on the actual traffic: $67,565 from 8,200 clicks. See the full before and after numbers on our results page. Real client numbers, not typical results, and not a promise of what your store will do.
Conversion rate x average order value. 1.0% x $125 before, 3.5% x $231 after.
And here's the part most case studies skip. That client's first week spiked higher, to a 4.5% conversion rate and a $250 average order value. We don't lead with that. We lead with the steady-state numbers, because a week-one spike is a highlight and the settled number is the tape.
Same rule on the Amazon side. One repellent product moved from Amazon to Shopify went from $3.83 net profit per sale to $48.27, after paying for ads and inventory. Profit per sale, with the costs left in. If you want more receipts like that, our ecommerce case studies library has every input, and this before and after conversion rate case study shows how to measure your own.
How is AI changing what you should ask?
Here's the thing. A polished case study used to take a designer and a copywriter a week. Now any AI writing tool drafts one in a minute. The deck got cheaper. The proof didn't.
Same with the pages themselves. We build product pages with AI in less than 15 minutes, and plenty of tools now spin up test variants fast (I compared a few in this roundup of AI CRO tools). Speed is great. But faster variants mean more tests, more peeking, more chances to stop early on a lucky week.
So the bottleneck moved. It used to be "can they build it?" Now it's "can they measure it honestly?" Every question on this page is a measurement question for that reason.
When anyone can make a beautiful deck in a minute, the only thing worth paying for is the honest math underneath it.
What to do next
Pick the agency you're closest to hiring. Email them one line today: "For your lead case study, can you send the visitors, orders, average order value and dates for both periods?" Their reply, or their silence, is your answer.
Book Your Profit Audit
On a profit audit we'll pull your real baseline first, conversion rate, average order value and revenue per visitor on your top pages, so any result after it can be checked. Then we'll show you how to rebuild a high-converting product sales page in less than 15 minutes.
Or go here to check it out → revenueflows.ai
P.S. Every agency has a highlight reel. Before you sign, ask for the game tape.
Frequently asked questions
What should a CRO agency case study include?
Four things: the baseline conversion rate and average order value, the traffic the test ran on, the time frame, and exactly what was changed. Without all four, a percentage lift can't be checked, so treat it as marketing rather than evidence.
How do you evaluate a CRO agency's results?
Ask for the raw inputs behind their best case study: visitors, orders, dates and the change made. Then recompute revenue per visitor before and after (conversion rate times average order value). If they can't hand you those numbers, you've learned what you needed to know.
What is a realistic A/B test win rate for a CRO agency?
One agency-selection guide puts typical win rates between 20% and 40% and flags claims of 80% or higher as a red flag. An agency that says almost every test wins is either testing tiny, safe changes or stopping tests early.
What questions should I ask a CRO agency before hiring?
Ask what the baseline was in their best case study, how much traffic it ran on, how they decided when to stop the test, how many tests lost, and whether the lift held 60 days later. Those five questions separate game tape from a highlight reel fast.
Should I trust a CRO agency that guarantees results?
Be careful. Your traffic, product and price are different from every store in their portfolio, so a fixed uplift promise can't be grounded in your numbers yet. Ask what they'll measure, how, and what happens if a test loses.