Creative and testing
Statistical significance
A check on whether a difference in a test is larger than chance alone would often produce.
What it is
It is usually set as a threshold before the test starts. Significant means the result is unlikely to be pure chance, not that it is large or important.
Why it matters
For lead generation and e-commerceIt protects budgets from being moved on noise.
In practice
Fix the sample size and threshold in advance, and read the result once.
A common mistake
Ad tests often make many comparisons at once, and each one adds a chance of a false winner. Without correcting for that, some winners are not real.