How Long Should You Run an A/B Test? The 2 Week Rule

I've called a test on day three. Once.

The variant was up big, the dashboard was green, and I was tired of waiting. I shipped it. Two weeks later the store's numbers were right back where they started, and I had no idea why.

So here's the answer I wish someone had handed me on how long to run an AB test. Run it for at least two full weeks, in whole 7 day blocks, and until each version hits the sample size you calculated before you launched. Whichever comes last is the stop date. Never stop on a good day. A good day is the most convincing liar in your analytics.

Why does a test that's winning on day three lie to you?

Because early data is mostly noise, and noise loves to look like a pattern.

Evan Miller ran the numbers on this years ago. He took a change that does nothing at all and showed what happens if you peek at the results and stop the first time they look significant. Your chance of crowning a fake winner jumps to 26.1%, more than five times the 5% you think you're running.

One in four "winners" you call early didn't win anything. You just stopped watching at the right moment.

That's the cage. Every testing app shows you a live chart. The live chart begs you to check it. And each check is another roll of the dice.

Here's the thing. You don't need to stop looking. You need to stop acting on what you see before the date you set.

How long should an A/B test run on a Shopify store?

Two rules, and both have to be true before you call it.

1. The time rule. At least 14 days, in full 7 day blocks. Shopify's own testing guide says to run at least one, ideally two, full business cycles. For a store selling to consumers, a business cycle is a week.
2. The sample rule. Each version reaches the visitor count your sample size math demanded, decided before launch.

Most founders only use one. The fast-traffic store uses the sample rule, hits it in four days, and ships a Monday-heavy fluke. The small store uses the time rule, stops at day 14, and ships a coin flip.

Use both. The later date wins.

And there's a ceiling too. The same Shopify guide warns that the longer a test runs, the more it's exposed to sample pollution: returning visitors who've seen both versions, a holiday, a sale, a new ad campaign. If your math says a test needs more than about eight weeks, that's your signal the change is too small for your traffic. (The full playbook for that situation lives in how to A/B test on low traffic.)

Why full weeks and not just "14 days"?

Because Tuesday and Saturday aren't the same store.

The Tuesday buyer is on her phone at lunch, browsing in a hurry. The Saturday buyer has coffee and twenty minutes to read every review. A variant that adds a long story section might lose on Tuesday and win on Saturday.

Run the math on a hypothetical. A store starts a test on a Thursday and stops after 11 days, on a Sunday. That window holds two Saturdays and two Sundays, but only one Tuesday and one Wednesday. If the variant is a weekend kind of page, it just got a head start the real calendar won't give it.

Full blocks of seven fix that. Start on any day you like. Stop on the same weekday, two or more weeks later. Every version sees every kind of day the same number of times.

How do you calculate the stop date before you start?

This is where most test plans are a guess. Don't guess. Evan Miller's rule of thumb is short enough to do on a napkin:

Visitors needed per version = 16 x p x (1 - p) / (the change you want to detect)². Here p is your current conversion rate.

Now watch what happens on a hypothetical store.

• The hero product page converts at 2.0%. That's p.
• You want to catch a 30% lift, so 2.0% to 2.6%. The change is 0.006.
• 16 x 0.02 x 0.98 = 0.3136. Divide by 0.006 squared (0.000036). That's about 8,711 visitors per version, or 17,422 for the whole test.
• The page gets 20,000 visitors a month, which is about 4,615 a week.
• 17,422 / 4,615 = 3.8 weeks. Round up to full weeks: 4 weeks.

Four weeks beats the two week floor, so four weeks it is. Write that date down before you hit launch.

Now try it with a timid change. Hunting a 10% lift (2.0% to 2.2%) needs about 78,400 visitors per version. On the same page that's 34 weeks. That test isn't a test. It's a hope with a progress bar.

Intelligems puts the same idea in orders instead of visitors: a couple hundred orders per variant is a loose guidepost before you trust a result, and a best seller doing 100 orders a week takes about a month to get there.

Catch a 50% lift 2

Catch a 30% lift 4

Catch a 20% lift 9

Catch a 10% lift 34

2.0% baseline conversion rate, Evan Miller's rule of thumb, rounded up to full weeks, never under 2.

When should you stop an A/B test early?

When it's hurting you. Almost never when it's helping you.

That asymmetry matters. A losing variant costs real orders every day it runs. A winning variant will still be winning on day 28 if it's real. So the rules for when to stop an AB test aren't symmetrical either.

Convert interviewed a group of testing practitioners on this. One stops for harm only when the variant stays significantly worse for three or more consecutive days. Another keeps a firm minimum of two weeks no matter when significance shows up. That's the shape I'd copy:

Situation	What to do
Variant up big on day 3	Keep running. Write nothing in the log yet.
Variant down badly for 3+ days straight	Stop. Revert. Log it as a loser.
Sample reached on day 9	Keep running to day 14, then stop on the same weekday.
Day 14 reached, sample not reached	Keep running in full weeks until it is.
Math says 8+ weeks	Don't run it. Make the change bigger, or ship it without a test.
A sale or holiday lands mid-test	Pause the read. Extend by a full week after it ends.

If you want to know why the dashboard's "probability to win" behaves differently from a classic significance number, I broke that down in Bayesian vs frequentist A/B testing. And if the calendar has a Black Friday on it, read what to test and freeze during Black Friday first.

Has AI changed how long a test needs to run?

Here's the honest part. AI changed the build. It didn't change the clock.

A year ago the slow step was making the variant: a copywriter, a designer, a developer, two weeks of back and forth. Now an AI system can write the copy, design the images, and build the page in less than 15 minutes. Shopify even ships testing in the admin now. Its Rollouts Experiments start with a 50/50 split and a default end date 90 days out, on the Grow plan or higher.

But statistics don't care how fast you built it. 17,422 visitors is 17,422 visitors. The traffic still has to show up, a week at a time.

So the bottleneck moved. It's no longer "can we build it?" It's "is this change big enough to be worth four weeks of our traffic?" That's a better problem to have. It pushes you toward bold variants: a rebuilt first screen, a new offer stack, a full page rewrite in a new section order. Small button tweaks don't survive the math.

And judge the winner on money, not clicks. A variant can win on conversion rate and lose on order size. Run a hypothetical: control converts at 2.0% with a $70 average order value. That means revenue per visitor is $1.40, and on 10,000 visitors that's $14,000. The variant converts at 2.5% with a $74 average order value. Revenue per visitor is $1.85, and on the same 10,000 visitors that's $18,500. That's the number that decides it. (More on why in revenue per visitor vs conversion rate.)

Control $14,000

Variant $18,500

Conversion rate x average order value x visitors. Control 2.0% x $70. Variant 2.5% x $74.

Big changes are where big numbers live. On a bedding client's Adjustable Pillows page, conversion rate went from 1.0% to 4.4% and average order value from $125 to $152. Revenue per visitor went from $1.25 to $6.68. On 10,000 visitors, that's $66,800 instead of $12,500. See the full case study numbers. Real client numbers, not typical results, and not a promise of what your store will do.

A swing like that doesn't need 34 weeks to prove itself.

What to do next

Open your analytics and grab two numbers for your hero product page: last month's visitors and its conversion rate. Plug them into the napkin formula for a 30% lift. If the answer comes out under eight weeks, you have a real test. Put the stop date in your calendar before you build anything.

For the full process around it, start with the ecommerce A/B testing guide, then follow the setup steps in how to A/B test your Shopify product page.

---

Book Your Profit Audit

The test that finishes fast is the one with a big enough change behind it, and a rebuilt product page is the biggest change most stores can make. Book a free profit audit and we'll show you where your revenue per visitor is leaking, then show you how to rebuild a high-converting product sales page in less than 15 minutes.

Book Your Profit Audit →

Or go here to check it out → revenueflows.ai

P.S. A good day feels like a winner. Fourteen days and a full sample tell you if it really is.