What is incrementality testing in advertising?

Incrementality testing uses a randomized holdout group (users who do not see your ads) to measure what conversions your advertising actually caused versus what would have happened organically. The difference between the exposed group and the holdout group is the incremental lift.

How does incrementality testing differ from attribution?

Attribution models measure correlation: which touchpoints appeared before a conversion. Incrementality testing measures causation: did the ad actually cause the conversion? A retargeting campaign can show high attributed ROAS (it touched many converters) while showing low incremental lift (those users would have converted anyway).

How big does the holdout group need to be?

Holdout groups typically range from 10 to 30 percent of the target audience. Smaller holdouts reduce the cost of the test (fewer missed conversions) but require larger audiences to reach statistical significance. Most tests require a minimum of 500 to 1,000 conversions across both groups to produce reliable lift estimates.

Incrementality Testing

Incrementality testing uses a control holdout group to measure what conversions advertising actually caused, separate from what would have happened without it.

Common questions

What is incrementality testing in advertising?: Incrementality testing uses a randomized holdout group (users who do not see your ads) to measure what conversions your advertising actually caused versus what would have happened organically. The difference between the exposed group and the holdout group is the incremental lift.
How does incrementality testing differ from attribution?: Attribution models measure correlation: which touchpoints appeared before a conversion. Incrementality testing measures causation: did the ad actually cause the conversion? A retargeting campaign can show high attributed ROAS (it touched many converters) while showing low incremental lift (those users would have converted anyway).
How big does the holdout group need to be?: Holdout groups typically range from 10 to 30 percent of the target audience. Smaller holdouts reduce the cost of the test (fewer missed conversions) but require larger audiences to reach statistical significance. Most tests require a minimum of 500 to 1,000 conversions across both groups to produce reliable lift estimates.

Most ad attribution models measure correlation, not causation. A user who saw your ad and then purchased may have been going to purchase anyway. Last-click, multi-touch, and even data-driven models cannot tell you whether the ad actually caused the conversion; they can only show that the ad was present before the conversion happened. Incrementality testing is the method that actually answers the causal question.

The standard design is a holdout test: a randomly selected subset of your target audience (10 to 30 percent, depending on the test) is excluded from seeing your ads for the test period. All other users see ads as normal. At the end of the test, you compare conversion rates between the two groups. The difference is the incremental lift: the conversions that your advertising actually caused, net of what would have happened without it.

Incrementality tests are not free. You are giving up conversions in the holdout group during the test period to get the data. They require sufficient audience size to reach statistical significance. They are point-in-time measurements; lift can change as campaigns evolve. But they are the most reliable method for calibrating whether your ad spend is actually driving business outcomes or just claiming credit for conversions that were going to happen anyway.

Example

A DTC brand runs a 90-day Meta incrementality holdout: 20 percent of the target audience is excluded from all Meta ads. At the end of the test, the holdout group shows a 6.2 percent lower conversion rate than the exposed group. The incremental ROAS for Meta is 3.1x: lower than platform-reported 4.8x but confirmed causal. Budget is reallocated accordingly.

Always on · Algorithm-led · Human-approved

Walk this operation against your account.

Thirty focused minutes with one of the founders. We map Incrementality Testing against your stack and tell you which entry point fits your stage.

Book a 30-minute call →Or write us