PyMC-Marketing vs. Meridian: What a Baseline-Modeling Benchmark Shows About MMM Attribution
On this page
A marketing mix model can hit a strong R² and still get channel attribution wrong. Those are two different questions: does the model predict total sales well, and does it correctly separate how much of that outcome came from each media channel versus the underlying trend. A team that checks only the first question can reallocate budget away from a channel that actually worked, because the model absorbed that channel's true effect into a flexible baseline term instead of assigning it to media.
PyMC Labs' follow-up benchmark compares PyMC-Marketing and Meridian on synthetic data with known channel contributions. PyMC Labs develops one of those libraries. The known generating truth enables recovery checks; the author's product interest and tested configurations should remain visible.
The two questions a fit metric can't answer
Goodness-of-fit metrics like R² and MAPE measure how closely predicted sales track observed sales, not whether that variation was assigned to the right source. A flexible baseline term can track short-term sales fluctuations closely, improving fit while quietly absorbing variation a media channel actually caused.
Contribution-recovery metrics (bias, standardized root-mean-squared error (SRMSE), and CRPS on the recovered channel effects) measure the second question directly, because the benchmark's synthetic data has a known ground truth for each channel's contribution.
What did the benchmark measure?
The follow-up evaluates PyMC-Marketing 0.17.0 and Meridian 1.2.1 on small, growing, and medium synthetic datasets, examining convergence, in-sample fit, and contribution recovery. These are historical version-specific results; automated spline-knot selection refers to the Meridian release tested in that run.
In the tested configurations, PyMC-Marketing uses explicit Fourier seasonality with optional longer-term trend modeling, while Meridian uses a spline baseline. These choices belong to the stated benchmark conditions, rather than a claim that current architecture and defaults are unchanged.
The table reports the benchmark's own published figures as historical results from that specific run of PyMC-Marketing v0.17.0 against Meridian v1.2.1. They describe one synthetic-data study, not a live measurement of either library's current performance.
| Business size (benchmark's own figures) | Library | R² (higher = better fit) | MAPE % (lower = better fit) | Durbin-Watson (closer to 2 = less autocorrelated residuals) | Bias on contribution recovery (lower = better) | SRMSE on contribution recovery (lower = better) |
|---|---|---|---|---|---|---|
| Small Business | PyMC-Marketing | 0.871 ± 0.020 | 7.10 ± 0.60 | 1.97 ± 0.17 | 82 ± 207 | 0.41 ± 0.23 |
| Small Business | Meridian | 0.930 ± 0.009 | 5.10 ± 0.40 | 1.89 ± 0.17 | 219 ± 259 | 0.66 ± 0.24 |
| Growing Business | PyMC-Marketing | 0.884 ± 0.015 | 6.70 ± 0.50 | 1.85 ± 0.15 | 56 ± 158 | 0.42 ± 0.34 |
| Growing Business | Meridian | 0.941 ± 0.004 | 4.70 ± 0.20 | 1.67 ± 0.09 | 49 ± 185 | 0.54 ± 0.49 |
| Medium Business | PyMC-Marketing | 0.954 ± 0.006 | 5.00 ± 0.30 | 1.88 ± 0.14 | 522 ± 2175 | 0.16 ± 0.09 |
| Medium Business | Meridian | 0.968 ± 0.002 | 4.20 ± 0.20 | 1.42 ± 0.06 | 1664 ± 3317 | 0.29 ± 0.24 |
In the reported runs, Meridian has higher in-sample R² and lower MAPE, but also higher contribution SRMSE across these dataset sizes. Bias point estimates are higher for the small and medium datasets; for the growing dataset, Meridian's 49 is lower than PyMC-Marketing's 56. The reported spreads are broad, so point estimates do not alone establish a stable ranking.
Why can a more flexible baseline look better and mislead at once?
A lower Durbin-Watson statistic indicates more positive residual autocorrelation. It does not alone diagnose baseline leakage or causality. Use contribution-recovery tests where truth is known, alongside residual checks. This benchmark round did not evaluate out-of-sample prediction.
What is the saturation problem that no library escapes?
The benchmark identifies a separate, structural limit that applies to both libraries: channels that saturate quickly are hard to recover accurately. Contribution recovery depends on spend varying within a channel's non-saturated response range; past that point, extra spend barely moves the outcome, so spend and sales stop moving together, and the model loses its ability to tell that channel's true effect apart from the baseline, even when the data still has plenty of spend fluctuation. A more flexible baseline makes this worse, because it can more easily absorb the variation that should have been assigned to the saturated channel.
Checking your own model before moving budget
This benchmark's warning applies past these two libraries: fit metrics and attribution metrics answer different questions, and passing one says nothing about the other. Before reallocating spend on a model's contribution numbers, the model's recovered contributions need their own check, separate from R² or MAPE.
For MMM calibration, use relevant incrementality evidence whose treatment, outcome, population, time window, and exposure match the modeled channel contribution. A generated buyer-choice effect does not supply that measurement contract. Subconscious's choice-study method and validation evidence concern defined pricing and messaging responses, a separate research task.
Limitations of this benchmark
This is a developer-authored synthetic-data study of specific library versions. It is not a real-company attribution validation or a general ranking of current MMM tools. Retain the tested versions, assumptions, diagnostics, and uncertainty when applying its findings.
Next step
Before moving budget, inspect recovery and calibration evidence relevant to the MMM's channel estimand. A pricing or messaging question requires a separate study of those alternatives; review the choice-study approach for that task.