PyMC-Marketing vs. Google Meridian: What to Check Before a Benchmark Moves Your Budget
On this page
A marketing analytics team choosing between PyMC-Marketing and Google Meridian can use published benchmarks to inspect sampling and parameter recovery. Before those findings inform ad-budget decisions, compare the tested versions, data-generating assumptions, diagnostics, and reproduction evidence with your own modeling requirements.
The decision this comparison actually supports
Library selection affects what an MMM can fit and diagnose. A synthetic benchmark can test recovery where the generating truth is known. It does not by itself validate channel contributions or budget recommendations on a company's observational data.
What are PyMC-Marketing and Meridian?
Both projects supply open-source Bayesian MMM tools. In the versions tested by the cited benchmark, PyMC-Marketing offered multiple sampling backends and Meridian used TensorFlow Probability. Inspect current project documentation for today's APIs and defaults; historical results apply to the tested configurations.
What the published benchmark claims
PyMC Labs' 8 September 2025 benchmark compared Meridian version 1.1.6 with PyMC-Marketing version 0.15.1 using synthetic datasets. It ran CPU tests on an n1-standard-32 Google Compute Engine machine and reported PyMC-Marketing results with Nutpie. The author develops one of the compared libraries; retain that disclosure alongside the conditions and code.
Developer authorship does not invalidate the measurements. Public code helps others inspect and rerun the experiment. Distinguish reproducing the same run from testing whether the result generalizes across datasets, hardware, configurations, and versions.
What to require before trusting the result
| Evidence to request | Original benchmark | Reproduction or extension |
|---|---|---|
| Who ran the test? | Name the author and product interest | Name the reproducer and any product interest |
| What conditions were used? | Pin versions, priors, data generator, sampler and hardware | State which conditions were retained or changed |
| Was an external rerun published? | Check for an actual dated record; authorship alone does not answer this | Link the results, protocol and runner |
| How are disagreements handled? | Inspect diagnostics, limitations and corrections | Report agreements, failures and explanations |
| Can the setup be inspected? | [Published benchmark code](https://github.com/pymc-labs/mmm-param-recovery) supports inspection | Provide code, environment and resulting measurements |
Where this pattern shows up beyond MMM libraries
Apply the same disclosures to any vendor's validation, including Subconscious. Subconscious's causal-fidelity paper compares generated-choice parameter estimates with published human studies. That is different from an independent party rerunning the platform, and different from MMM channel-contribution recovery. Neither comparison substitutes for the other.
A practical checklist for the reallocation decision
- Ask who ran the benchmark and whether they have a stake in the outcome.
- Ask whether anyone outside that team has reproduced the result on different data.
- Treat published code as an audit trail, not a substitute for replication.
- If external reproduction is unknown, record it as unknown and use the benchmark only within its demonstrated scope.
- Where the decision is large enough, run a smaller-scale internal test on your own data before committing spend to the recommendation.
What are the limitations of this comparison?
This comparison does not establish which library recovers channel contributions best on your data. Evaluate model assumptions, diagnostics, recovery on suitable synthetic cases, and relevant experimental calibration. Inspect actual reproduction records before claiming that a reported gap has or has not been independently reproduced. Subconscious's choice-study evidence concerns a different endpoint and should be judged with the same author, runner, task and validation disclosures.