The winner is smaller than it looks

In 2023, the Journal of Consumer Research published a striking result: rounding the corners of a button raised clicks by 55%. It was statistically significant.

Then a group of researchers tested the same change again, three times, on millions of users. The lifts they measured were all under 1%.

Nobody lied. The first test had 919 users. A test that small can only "see" a huge effect, so the results that pass are the lucky overestimates. Statisticians call it the winner's curse, and it applies to every ad account that declares a winner after a handful of days.

So how long should an ad test run? At least a week on Meta and TikTok, two to three weeks on LinkedIn, and four to six weeks for the Google experiments its guidance covers, by each platform's own help pages.

A winner can be pure chance

In 2024 the Meta ads educator Jon Loomer ran an A/B test with three ad sets. One ended with 25% more conversions than another.

The three ad sets were identical. Same ads, same audience, same settings.

It is one test and it proves no rate. It shows what every budget owner should see once: a 25% gap can contain no information at all.

One significant result in three to five is not real

Results that pass a significance test are safer. Many are still false.

Figure 1

One significant result in three to five was not real

Share of significant results with no true effect

Default setting28% to 37%

Stricter setting18% to 25%

Two researchers studied 2,766 experiments on Optimizely. Among the results that reached significance at the platform's default setting, they estimated that 28% to 37% had no true effect behind them.

The cause connects straight to velocity. Most ideas do nothing. When most candidates are duds, even a good filter lets enough of them through to fill a winners' list.

And the real ones are smaller than they look

Back to the button.

Figure 2

A 55% winner, tested again

919 users the first time, millions after

First test55.5%

Re-test 10.16%

Re-test 20.29%

Re-test 30.73%

The re-tests were published in 2026 by a team that includes Ron Kohavi, Andrew Gelman and Guido Imbens. Their conclusion: the real effect was "approximately two orders of magnitude smaller."

For a budget this is the costlier error. A false winner wastes one decision. An inflated winner gets written into the forecast, and then the forecast is missed.

The scorekeeper is also a player

There is a second reason to doubt a win. The first report of a result usually comes from the platform that sold the ad.

Each platform counts the customers its own ads touched, by its own rules. Google, by default, counts a sale for up to 30 days after a click on its ad. A buyer who saw ads on three platforms can show up as a win on all three reports, while the business gained one customer. Every report can look good in a month when revenue did not move.

The only score that settles it is the advertiser's own: leads, customers and revenue, read across every channel at once.

The platforms already say how long to wait

None of this is hidden. It is in the help pages.

Figure 3

Platforms ask for one to six weeks

How long to run an ad test, minimum to recommended

Meta7 days

TikTok7 days

LinkedIn14 to 21 days

Google4 to 6 weeks

Meta asks for at least seven days, and adds a line that describes most day-to-day optimization: "We do not recommend testing informally, such as by turning ad sets or campaigns on and off manually."

The ceiling is how much a person can read

Everyone knows the rules. Wait for enough data. Test the winner again. Check it against revenue. Keep a record.

They get broken for the same reason. Each one takes hours, and the next decision is always due first. So the ad that is ahead on Wednesday gets the budget, and the ads that lost are switched off and forgotten.

That last habit throws away evidence the account already paid for. The losers say what the buyers did not want: the angle that never landed, the offer nobody took. That is the first line of the next brief.

What AI changes

A system that reads every result every day does not get impatient. It can hold a test to its end date, run the winner a second time before the money moves, score every platform against the same revenue, and carry what the losers taught into the next round.

None of that is cleverness. It is patience and bookkeeping, which is exactly what people run out of. That is how optimization compounds, and why it only works as part of a loop.

What to do on Monday

1. Set the end date before the test starts. Write down how long it runs and what result would count as a win. Then do not stop early, in either direction.
2. Run the winner again before scaling it. A second test on fresh traffic is the cheapest protection there is.
3. Cut the lift before it goes into a plan. If a short test says 30%, forecast far less.

That closes the formula: quality, velocity, optimization. The obvious next move is to hand each part to its own AI tool. The next article is about why that fails.