Creative and testing
Winner's curse
The tendency of the winning result in a test to look bigger than its true effect.
What it is
When many versions are tested, the one that comes out on top is often the one that got lucky as well as good. Its measured lift includes that luck.
Why it matters
For lead generation and e-commerceBudgets planned on the measured lift are planned on a number that will shrink.
In practice
Re-test big wins, and plan on a smaller effect than the test showed.
A common mistake
Scaling a winner on the strength of one test and calling the later drop creative fatigue.