Why a Media Mix Model Number Needs a Controlled Test Before It Moves Budget
On this page
In an invented example, an MMM attributes 18 percent of last quarter's revenue to paid social. A spend-cut experiment measures the incremental effect of a specified change over a specified window; it does not automatically reproduce that historical attribution. Define the proposed intervention before choosing its evidence.
What does a media mix model actually estimate?
A Bayesian media mix model (MMM) fits a regression across historical channel spend and outcomes such as revenue or conversions. It accounts for carryover effects (adstock, the fading influence of past spend) and diminishing returns (saturation, where extra spend on a channel earns less each additional dollar). The output is a contribution estimate per channel and, from that, a suggested reallocation of the next period's budget.
Publishing the assumption behind a contribution number is what lets a buyer check it before spend moves. The method's causal claim rests on an unconfoundedness assumption: that spend is not correlated with unobserved drivers of the outcome. That assumption is fragile because media spend is chosen by marketers, not randomized, and the model fits the pattern in historical spend and outcome data without running an experiment. When two channels moved together historically, for example a paid search campaign that always launched alongside a TV flight, the model can struggle to separate their individual contributions. Google's own applied research frames MMM as a causal inference method that is valid only under assumptions such as no unobserved confounding of the spend-outcome path, and treats controlled experimentation as the source of informative priors that make those assumptions more defensible (Google for Developers, "About MMM as a causal inference methodology").
What calibration methods close the gap between an MMM estimate and a measured effect?
Two approaches provide different kinds of evidence:
- Randomized geo lift. Assign eligible geographic regions to treatment and control, implement a specified spend contrast and measure outcomes. Matching can improve design but does not replace assignment. See Vaver and Koehler's 2011 geo-experiment paper.
- Difference in differences. Compare treated and untreated outcome changes under assumptions such as parallel untreated trends, no anticipation and limited spillovers. It is not randomized merely because it includes a control group; examine assignment, timing and trend evidence. See CausalPy documentation.
Match the analysis to the assignment and outcome. Randomized geo tests and observational DiD have different identifying arguments; neither validates every historical contribution or future response curve.
| Method | What it estimates | What it needs | Where it falls short |
|---|---|---|---|
| MMM | Channel response under the specification | Time-series data, identifying assumptions, relevant calibration | Confounding and unsupported spend extrapolation |
| Randomized geo lift | Effect of the tested spend contrast | Eligible regions, assignment, delivery and outcome measurement | Spillovers, limited regions and window-specific effects |
| Difference in differences | Effect under parallel untreated trends | Treated/control timing and comparable outcome data | Trend violations, anticipation and spillovers |
Where this decision sits before spend, not just after it
A launch, price change or message can also be studied before commitment. A randomized discrete choice task estimates the effect of tested attributes on stated choices under its assignment and response protocol. Hypothetical tasks and observed purchases differ in context and incentives; that difference does not imply that every choice share or effect is inflated. For a Subconscious study, specify the audience, alternatives, endpoint and uncertainty procedure. An independent human choice task can test agreement on that endpoint, while a purchase or spend intervention tests a different outcome. Confirm recruitment, fieldwork and reporting responsibility when planning either comparison. The research record provides aggregate choice-parameter evidence, rather than a guarantee for the proposed intervention.
Where this does not apply
Choose evidence for the outcome you intend to change. Historical MMM estimates, randomized geo contrasts and randomized choice tasks answer different questions. An effect on stated choice does not establish incremental channel revenue, and a geo test of one spend contrast does not validate every historical contribution. If the engagement needs time-series modeling, geo assignment or integration with a choice study, confirm those specific services and deliverables with the provider.
The practical rule
Specify the spend contrast, assignment, attribution window, adstock or washout assumptions and outcome uncertainty. Compare the measured effect with the allocation you intend to make and a prespecified practical effect or loss threshold. A new spend cut alone does not validate an 18 percent historical contribution or a whole response curve. Discuss the decision and evidence.
