AB Testing During Black Friday: What to Test and Freeze

Every October a founder asks me the same question. "Should I pause my tests for Black Friday, or run them through it?"

My answer: split the store in two. Ab testing during Black Friday works when you test the offer and freeze the page. Test discount type, gift with purchase, bundle thresholds and deadline copy, because that's what holiday shoppers are deciding on. Freeze layout, checkout, navigation and theme code, because a broken page on the biggest weekend of the year costs more than any test can win. And never carry a holiday winner into January without retesting it on normal traffic.

This year Black Friday lands on November 27 and Cyber Monday on November 30. That gives you four days of the biggest sample size you'll see all year. It's also the most misleading one.

Why does Black Friday traffic break normal A/B tests?

Because the people are different. Same store, same products, different buyer.

A November shopper shows up with a deadline, a budget and three tabs open. A February shopper shows up curious and in no hurry. Put the same page in front of both and you'll get two different answers.

The volume is real. Adobe tracked $14.25 billion in U.S. online spending on Cyber Monday 2025, up 7.1% on the year, with $11.8 billion on Black Friday and $44.2 billion across the five days from Thanksgiving to Cyber Monday. The same report says 57.5% of Cyber Monday sales came through a phone, and traffic to retail sites from AI assistants rose 670% on the day.

So your holiday visitor is more mobile, more deal-driven and more likely to arrive pre-sold by a chatbot's comparison than the visitor your tests normally see.

The team at Conversion.com frames it well: during Black Friday, motivation spikes while the ability to buy drops because sites get crowded and slow. That's the whole game. Buyers already want the deal. Your job that weekend is to make saying yes easier, not to redesign the store they're trying to check out of.

A Black Friday winner tells you what a deal hunter wants on a deadline. It tells you almost nothing about what a full-price buyer wants on a Tuesday in February.

What should you test during Black Friday?

Test the offer layer. It's what the buyer is actually comparing, and you can switch any of it off in one click if it misfires.

Test during the sale	Freeze from mid November
Percent off vs dollar off vs gift with purchase	Page layout and section order
Free shipping threshold ($75 vs $100)	Checkout flow and payment options
Bundle tiers (buy 2 vs buy 3)	Navigation, menus and collection structure
Deadline copy ("Ends Monday midnight")	Theme code, new apps, new scripts
Promo banner wording and placement	Product photography swaps across the catalog
Email subject lines and send times	Anything you can't roll back in 60 seconds

Intelligems' holiday guide lands in the same place: test gift-with-purchase tiers, discount depth and volume discount structures, and stay away from new buttons, checkout flows and navigation that can confuse returning customers mid-sale.

Here's why the offer matters so much. Run the math on a store like this, a hypothetical: 40,000 visitors over the four-day sale.

Offer A is 25% off sitewide. Conversion rate 3.4%, average order value $64. That's 1,360 orders and $87,040. Revenue per visitor: $2.18.

Offer B is a free gift on orders over $100. Conversion rate drops to 3.0%, but average order value climbs to $96. That's 1,200 orders and $115,200. Revenue per visitor: $2.88.

Fewer orders. $28,160 more revenue. And the gift costs you far less margin than 25% off every single order.

This is why I tell founders to stop judging holiday tests by conversion rate alone. A discount that "wins" on orders can lose on revenue. If you want the no-discount version of that thinking, read how to grow average order value without discounting.

What should you freeze before Black Friday?

Everything structural. Freeze it early and don't touch it.

Shopify's own engineers run a code freeze before Black Friday, and their rule is blunt: only critical fixes can be deployed. If the platform itself locks down, your theme should too.

I'd set the freeze for November 13 this year, two weeks before Black Friday. Any layout test still running on that date gets called or killed. No new apps. No new scripts. No "quick" section reorder at 11 pm on Thanksgiving.

The risk is lopsided. A layout test on a normal week might win you a few tenths of a point of conversion rate. A layout test that breaks the mobile add-to-cart button on Black Friday, with more than half your buyers on a phone, can burn a whole day of your best traffic before anyone notices.

And the structural work still matters. It just belongs in October and January, on clean traffic. If your product page needs a real rebuild, do it before the freeze. We've seen what that kind of rebuild does on normal traffic: on one bedding brand's adjustable pillow page, conversion rate went from 1.0% to 4.4% and average order value from $125 to $152, taking revenue per visitor from $1.25 to $6.68 (see the full case study numbers). Real client numbers, not typical results, and not a promise of what your store will do. The point is timing. That kind of change needs a calm month to prove itself, not the loudest weekend of the year.

Test the offer during the sale. Rebuild the page before the freeze. Mixing the two is how a record weekend turns into a support inbox full of "the button won't work."

For the page itself, the pre-freeze rewrite is covered in how to write a Shopify product page for Black Friday traffic. If you're building a dedicated sale page, study these Black Friday landing page examples before the freeze.

Should you use automatic winner rollout or a fixed split?

This is where AI testing tools earn their keep in November.

A fixed-split test is the classic version. Half your visitors see A, half see B, for the whole test. At the end you read the result with statistical confidence. Clean, slow and built for long-term decisions. It's the default method in any serious ecommerce A/B testing program.

An automatic winner rollout, usually called a multi-armed bandit, starts at 50/50 and then keeps shifting traffic toward whichever version is converting better while the test is still running. The name comes from a gambler choosing between slot machines: keep pulling the one that's paying, but check the others now and then in case you guessed wrong.

The most common method behind it is Thompson sampling. In plain words: for every new visitor, the tool makes a quick guess at how good each version really is based on the results so far, picks the version whose guess comes out on top, and updates after every order. Versions that keep winning get picked more. Versions that lose still get a trickle, so a slow starter can recover.

Wingify (the company behind VWO) documents exactly this in its help center: its bandit uses Thompson sampling with an epsilon-greedy layer that keeps a slice of traffic exploring, and it says bandits suit optimizers who care more about maximizing a metric in a short time and can give up on statistical significance.

That's a four-day sale in one sentence. Here's the math on the same hypothetical 40,000 visitors:

Fixed 50/50 split $101,120

Bandit rollout $108,160

All traffic on the winner $115,200

Offer A: 3.4% conversion rate x $64. Offer B: 3.0% x $96. Bandit assumes 75% of traffic ends up on Offer B.

A fixed split sends 20,000 visitors to each offer: 680 orders at $64 plus 600 orders at $96, which is $101,120. A bandit that ends up sending 75% of traffic to Offer B gets 900 orders at $96 plus 340 at $64, which is $108,160. That's $7,040 more from the same visitors, because fewer of them were spent learning something you already knew by Saturday.

The trade: a bandit gives you revenue, not proof. You won't get a clean answer you can bet next year's strategy on. For the sale, that's fine. For permanent page changes, use a fixed split, the way we lay out in how to A/B test a Shopify product page.

Why do Black Friday winners lose in January?

Because the buyer who made them win went home.

Here's the trap. Your "Ends Monday midnight" banner crushed it on Black Friday. So in December someone leaves it up with a new date. Then January. Now it's a fake deadline on every page, and the full-price buyer who shows up on January 14 sees a store that's always ending something.

I've seen founders treat a holiday winner like a law of nature. It isn't one. It's a result about one crowd on one weekend.

Intelligems says the same thing in plain terms: holiday test results might not apply all year, so rerun the winners in January, when customers are less focused on deals. Real deadlines lift conversion. Fake ones erode trust, which is the whole argument in our breakdown of whether countdown timers really increase conversion rate.

A holiday winner is a hypothesis for January. It's not a decision.

So on the Tuesday after Cyber Monday, December 1 this year, roll every page back to its control. Write down what won and by how much. Then queue the best idea as a fresh fixed-split test on January traffic, with no sale running, and let it earn its spot.

What's your 2026 Black Friday testing calendar?

Here's the whole plan on one page.

1. Now to October 16: launch any structural test (layout, product page rewrite, checkout) so it finishes on normal traffic.
2. October 17 to November 12: pre-test your two strongest offers on email or a small slice of paid traffic.
3. November 13: freeze. Call or kill every structural test. No new apps or scripts.
4. November 26 to 30: run one offer test as a bandit. Watch revenue per visitor, not only conversion rate.
5. December 1: roll back to control and log the results.
6. January: retest the holiday winner as a fixed split on full-price traffic.

What to do next

Open your testing app today and write down every test that's live. Anything structural gets an end date before November 13. Anything that won't be done by then gets killed now, while it's cheap.

Then pick one offer question for the sale. Just one. Discount or gift. $75 threshold or $100. That's your Black Friday test.

---

Book Your Profit Audit

Black Friday rewards the store whose page was already built to close before the freeze, and the offer test only multiplies what the page can do. Grab a profit audit and we'll show you where your page leaks revenue per visitor right now, then how to rebuild a high-converting product sales page in less than 15 minutes.

Book Your Profit Audit →

Or go here to check it out → revenueflows.ai

P.S. Test the deal in November. Test the page in January. The store that mixes them up spends December cleaning up.