The Evidence Ladder
Before you move budget, know how much weight your marketing numbers can carry.
Platforms don't grade their own homework. Channels often claim the same conversion, so platform totals can't settle a budget decision.
The evidence ladder ranks what each type of number can support, from platform reports up to a media mix model. Each step needs the one below it, which is why we find out where your evidence sits before recommending a test or a model.
5. Media mix model
What it isEvery channel valued on what it addsWhat it can tell youHow to split the budget4. Incrementality
What it isLift on the business result in test regions against comparison regions, with a noise checkWhat it can tell youWhat the channel actually added3. Controlled tests
What it isCreative and geo tests against comparison regionsWhat it can tell youWhich version or region did better, like with like2. Governed data
What it isBuying strategy, audiences, naming, tracking and spend, held to one standardWhat it can tell youNumbers you can trust and cut by tier and region. Nothing above this step can be read without it.1. Platform reports
What it isWhat the platforms say happenedWhat it can tell youActivity and delivery. The platform is marking its own work.
Last click stays on the page, for reference only. It over-credits search and under-credits anything upstream.
Testing starts with governed data
A test can only be read if media was bought the same way throughout. Buying strategy, audiences, naming, tracking and spend are held to one standard first, because a change in how media was bought can look like a change in results.
Region tests show what a channel added
Media runs in test regions, and the business result is compared with regions left out. The lift is measured on sales or leads, so a channel can show its value even without the click. Some channels can only be judged this way, and they stay parked until a lift test has run.
A lift has to clear the noise band
Before calling a lift, we run the same analysis on quiet weeks when nothing launched, several times over. The spread of "lift" that appears by chance is the noise band.
A result inside the band is reported as "not proven yet", and no lift is claimed. The channel is judged on its early signal for now. We re-run the check when the comparison regions, the market or the account structure change.
Every channel gets an evidence status
Reports mark each channel with one of four statuses, so anyone reading them can see what's proven and what isn't.
Channel evidence status
1 of 4 channels proven
| Channel | Evidence so far | Status | Next move |
|---|---|---|---|
| Online video | Region test, lift outside the noise band | Proven | Use it in budget decisions |
| Paid social | One region test, within normal variation | Not proven yet | Judge on early signal and re-test |
| Paid search | Platform reporting only | Tracked but not tested | Add to the test backlog |
| Radio | No tracking in place | Not measured | Set up tracking, or park it |
A media mix model comes last
A model can still fail if there's no governance across Insights, Strategy, Creative and Media, so we recommend one only once the steps below it are in place. From there, modelling extends beyond media into the 4 Ps, including price elasticity, and that's what makes forecasting possible.
What the ladder can't do
The ladder shows how much a number can support. It can't make a small test conclusive. Small regions and small budgets can leave a result inside the noise band, and a channel can stay "not proven yet" for some time.
Where we use it
Diagnostic
We check platform reports against your own sales and leads, then give each channel an evidence status.
Test and learn
We register tests against comparison regions, read them with a noise check, and say what's proven and what isn't.
Many of us don't have infinite budgets, so the incentive to test should be really strong. It's a case of accepting that not knowing is the beginning of knowing.
Where this comes up
When the board asks for the numbers, have the answer.
Tell us where it hurts: the budget conversation, the agency relationship, or numbers nobody trusts. We'll suggest where to start.
TALK TO NEIL →