Knowledge Paper 023 · Measurement
Can You Really Measure Advertising ROI?
Why marketing’s favourite question has no simple answer.
The short answer
Almost every client asks the same question.
“What is the ROI?”
It is a perfectly reasonable thing to ask.
Businesses should expect returns on investment.
But advertising does not behave like a vending machine.
You do not put in $100,000 and get back exactly $327,841.
The real world is much messier.
Advertising operates inside a noisy system involving millions of people, competitors, prices, seasons, distribution changes and random events.
That makes its precise financial contribution very difficult to isolate.
ROI is a business metric.
Return on investment belongs to the business.
Not to the advert.
An advertising campaign can be well planned, distinctive and effective, yet still sit inside a business producing poor returns.
The product may be weak.
The margin may be too small.
Distribution may be limited.
The website may be broken.
The sales team may fail to follow up.
A competitor may cut prices.
The economy may slow.
Advertising contributes to commercial performance.
It does not control the entire machine.
Everyone wants a clean number.
Senior managers like certainty.
Boards like certainty.
Procurement likes certainty.
Spreadsheets adore certainty.
So the demand goes out:
“Tell us exactly what the advertising returned.”
The problem is that a precise answer can be less truthful than an uncertain one.
A number with three decimal places still may be wrong.
Advertising does not happen in isolation.
Imagine sales increased by 8% after a campaign.
Was it the advertising?
Possibly.
But perhaps:
- a competitor increased its prices
- the weather changed
- consumer confidence improved
- a celebrity mentioned the product
- distribution expanded
- your sales team had a strong month
- the category entered its seasonal peak
- random chance produced an unusual run of purchases
Real businesses are noisy systems.
Advertising is attempting to move one part of that system.
Signal versus noise.
The problem is not necessarily that advertising has no effect. The problem is that the effect is might be small compared with everything else happening in the market.
Google tried to solve it.
Researchers Randall Lewis and Justin Rao analysed 25 large advertising field experiments involving major retailers and financial-services businesses.
The experiments collectively represented $2.8 million in advertising expenditure and reached millions of customers.
If anyone had enough data to produce clean ROI figures, this looked like a good place to start.
Instead, the researchers discovered that precise measurement remained seriously difficult.
The median confidence interval around estimated ROI was more than 100 percentage points wide.
Even the narrowest exceeded 50 percentage points.
In plain English, the experiments often could not tell businesses with useful precision whether a campaign was just breaking even or performing extremely well.
The mathematics is brutal.
One of the researchers’ representative examples involved trying to detect a $0.35 increase in sales per person.
That sounds manageable.
But those sales sat inside individual purchasing behaviour with a standard deviation of $75.
The effect was tiny.
The variation was enormous.
Even campaigns reaching hundreds of thousands or millions of people could remain statistically underpowered.
People are messy.
One person buys nothing.
Another spends $5.
Another spends $5,000.
Some people were always going to buy.
Some will never buy.
Some see the advertising and purchase six months later.
Some see it repeatedly and forget it.
Some do not consciously notice it but become slightly more familiar with the brand.
Human behaviour does not arrive in neat columns.
It arrives as chaos with receipts.
Evolution did not prepare us for statistical thinking.
Humans evolved in small groups.
Cause and effect were often immediate.
Eat the unfamiliar berry.
Become ill.
Hear movement in the grass.
Run.
Our brains became excellent pattern detectors.
That was useful.
It also means we often see simple causes inside complex systems.
Sales rose after the campaign.
Therefore, the campaign caused the rise.
Sales fell.
Therefore, the advertising failed.
Modern markets are not simple chains of cause and effect.
They are more like ecosystems.
Thousands or millions of people making decisions for different reasons at different times.
We confuse sequence with cause.
Something happened before something else.
Therefore, we assume the first thing caused the second.
This is one of humanity’s oldest cognitive habits.
Sometimes it is correct.
The rooster crowing before sunrise isn't what causes the sun to appear.
Advertising attribution systems often commit the same error with fancy spreadsheets or robots.
The false promise of attribution.
Digital advertising promised to solve the measurement problem.
Every click could be tracked.
Every conversion could be attributed.
Every customer journey could be mapped.
Or so the story went.
But the fact that something can be counted does not mean it caused the outcome.
People who click ads are often already more likely to buy.
People who search for a brand may already know it.
Retargeted customers may have purchased anyway.
Advertising is deliberately aimed at people considered more likely to respond.
That selection bias can make exposed audiences look more valuable even when the advertising caused little or none of the difference.
Randomised experiments are better.
Randomised control trials remain the best available way to estimate causal advertising effects.
Some people receive the advertising.
Others do not.
The commercial outcomes are compared.
This helps separate correlation from causation.
But even these experiments face the same basic problem.
The real effect may be very small relative to the volatility of buying behaviour.
Better method does not abolish uncertainty.
It reveals it more honestly.
The real problem is false precision.
When someone claims:
“This campaign produced a 247.3% ROI.”
The decimal point creates authority.
It suggests the answer has been weighed, calibrated and settled.
But precision and accuracy are not the same thing.
A measurement can be extremely precise and completely wrong.
Sometimes the scientifically responsible answer is a range.
Sometimes it is a probability.
Sometimes it is:
“We cannot know with that level of certainty.”
Then Goodhart’s Law arrives.
Goodhart’s Law says:
Demand a simple ROI figure and marketing activity begins bending towards whatever can produce one.
Clicks.
Last-click conversions.
Cheap leads.
Short evaluation windows.
Retargeting people who were already likely to buy.
Discounting that moves sales forward rather than creating new demand.
The campaign begins optimising for attribution rather than growth.
The dashboard improves.
The business might not.
Advertising can become too easy to measure.
Some activities generate immediate, visible responses.
Search advertising.
Direct-response promotions.
Affiliate links.
Retargeting.
These can be commercially useful.
But they often capture demand that already exists.
Long-term brand activity works differently.
It builds memory.
Familiarity.
Trust.
Mental availability.
Those effects may emerge months or years later.
They are real.
They are simply harder to assign to one advert or one reporting window.
The gym-session problem.
Nobody asks:
“What was the ROI of Tuesday’s gym session?”
They understand fitness is cumulative.
One session contributes.
It does not independently create the final result.
Advertising often works the same way.
Brands grow through repeated exposure.
Campaign after campaign.
Memory after memory.
Trying to isolate the exact return of one advert can be like asking which raindrop filled the reservoir.
So should we stop measuring?
No.
That would just be silly.
Measure everything that can genuinely help you make a better decision.
But choose measures according to the job the advertising is doing.
For short-term activation, that may include:
- incremental sales
- conversion rates
- customer acquisition cost
- gross margin
- profit contribution
For long-term brand building, it may include:
- category reach
- mental availability
- brand recognition
- distinctive-asset strength
- penetration
- share of search
- pricing power
- long-term profit growth
No single measure tells the whole story.
Measure decisions, not theatre.
A measurement system should help answer a decision.
Should we continue?
Should we increase reach?
Should we change the creative?
Should we invest more in brand?
Should we improve distribution?
Should we test a different audience or message?
The purpose of measurement is not to create the appearance of control.
It is to reduce uncertainty enough to act intelligently.
Common mistakes
Treating ROI as an advertising metric.
ROI belongs to the whole business.
Advertising is one contributor among many.
Believing every sale has one cause.
Customers are influenced by overlapping memories, prices, availability, recommendations and circumstances.
Mistaking attribution for causation.
The final measurable interaction is not necessarily what created the sale.
Optimising for what is easiest to count.
The most measurable activity is not automatically the most valuable.
Mistaking precision for accuracy.
Three decimal places do not rescue a bad model.
Ignoring long-term effects.
Advertising can influence future memory and demand long after the reporting window closes.
Using one number to judge every kind of advertising.
Brand building and activation do different jobs and require different forms of evidence.
TheSignalWorks View
We are not against measurement.
We are against measurement theatre.
Advertising should be accountable.
But accountability does not mean pretending uncertainty has disappeared.
Good strategy is not about demanding impossible precision.
It is about making better decisions with incomplete information.
Measure what matters.
Use experiments where possible.
Look for commercial patterns over time.
Accept ranges instead of fake certainty.
And never allow a seductive dashboard to replace judgement.
Key Takeaways
- ROI is ultimately a business metric, not an isolated advertising metric.
- Advertising effects are often small compared with the natural volatility of buying behaviour.
- Even very large randomised experiments can struggle to estimate advertising ROI precisely.
- Attribution does not automatically prove causation.
- False precision can encourage businesses to optimise for measurable activity rather than long-term growth.
- Better measurement reduces uncertainty; it does not abolish it.
Frequently Asked Questions
Can advertising ROI be measured?
It can be estimated, particularly through carefully designed experiments.
But the resulting estimate may have a very wide range of uncertainty.
Why is advertising ROI so difficult to measure?
Because advertising effects are often small relative to the enormous natural variation in customer behaviour and business performance.
Is digital advertising easier to measure?
It is easier to track.
That does not necessarily make it easier to establish true causation.
Are randomised control trials reliable?
They are generally more reliable than observational attribution methods, but even very large trials can remain statistically imprecise.
Does difficulty measuring ROI mean advertising does not work?
No.
It means the effect can be difficult to isolate from everything else affecting sales.
What should businesses measure instead?
Measure commercial outcomes alongside reach, mental availability, penetration, brand health, customer acquisition and other metrics relevant to the job the advertising is intended to do.
What is Goodhart’s Law?
Goodhart’s Law describes how a measure can become distorted once people are rewarded for achieving the measure itself.
Further Reading
- Randall A. Lewis and Justin M. Rao — The Unfavorable Economics of Measuring the Returns to Advertising
- Les Binet and Peter Field — The Long and the Short of It
- Byron Sharp — How Brands Grow
- Douglas W. Hubbard — How to Measure Anything
- Daniel Kahneman — Thinking, Fast and Slow
- Tim Harford — The Data Detective
Related Knowledge
We do training on all these topics and more. BOOK A SIGNAL SESSION.
About TheSignalWorks
At TheSignalWorks, we help organisations measure marketing without pretending the real world behaves like a spreadsheet.
Because the goal is not perfect certainty.
It is:
Better decisions, made with a clearer understanding of what can - and can't - be known.
GET THE NEXT PAPER
One email, one big idea
The important marketing ideas that your competitors probably don't know about.