When an MMM Overstates a Channel's Effect
On this page
A marketing mix model can overstate a channel's contribution when an omitted variable influences both spend and sales. Before shifting budget on its ROAS estimate, inspect the identifying assumptions and calibration evidence.
The failure mode: a hidden variable inflates one channel
An MMM estimates each channel's contribution from historical spend and outcome data. That works cleanly when nothing else is pulling on both the channel and the result at the same time; it breaks when something is.
In Juan Orduz's simulation, an unobserved variable (demand or seasonality) drove both one channel's spend and sales. The simulated data gave the second channel about 75% higher true ROAS than the first. The model that left the hidden variable out overestimated the first channel's ROAS by about 100% and underestimated the second channel's. The notebook reports those estimation errors. They show why an uncalibrated channel comparison can mislead a budget decision; the example does not establish the size of bias in a buyer's own model.
This is not a data-quality problem you fix by collecting more of the same kind of data. An unobserved confounder (a variable influencing both the channel and the outcome but absent from the model) stays invisible no matter how many additional weeks of spend and sales history get added.
How do experiment-informed priors calibrate the model?
Orduz's notebook reduces the omitted-variable bias by adding informative ROAS priors based on experiment information or domain knowledge. The calibrated model recovered the simulated channel effects much more closely, even with the hidden variable still missing. This is a controlled data-generating example, not a report that live field tests were run for each channel. Google Research describes the same technique in Media Mix Model Calibration With Bayesian Priors: a ROAS parameterization lets priors encode experiment results, and its simulations show reduced bias. Independent lift studies can supply that calibration information for a deployed model.
An observational model can make prospective counterfactual estimates under identifying assumptions. Experiments provide an additional source of information for checking and calibrating those estimates; they do not replace the model specification.
What does this mean before the next budget reallocation?
The decision a leader actually faces is not "trust the model or don't." It's narrower: does this specific ranking, the one about to move budget, rest on anything besides spend and sales history? If the answer is no, the honest move is to treat the ranking as provisional and test the channel the model says to cut before cutting it.
For media-spend ROAS, use a suitable incrementality experiment and outcome data to calibrate the MMM. A simulated message or offer test can develop a separate hypothesis about buyer choice; it does not directly measure sales caused by a channel’s spend.
Where does this example stop being evidence?
The worked example above is a third-party technical illustration of a calibration principle, not a Subconscious case result. No accuracy figure or customer outcome from Subconscious applies to it. What it is useful for is the pattern: a plausible-looking ranking, an invisible variable, and a specific fix that brought in experiment information. The leaderboard covers parameter-rank checks of choice studies, not media-lift calibration.