PyMC-Marketing vs. Google Meridian: What to Check Before a Benchmark Moves Your Budget
A marketing analytics team choosing between PyMC-Marketing and Google's Meridian will find a published benchmark claiming one library is faster and more accurate than the other. Before that benchmark moves real ad budget, what matters is not which number is bigger, but who ran the test and whether anyone outside that team has reproduced it.
The decision this comparison actually supports
Reallocating spend across channels on the strength of a media mix model (MMM) is a real financial decision. Picking the wrong library on an unreplicated performance claim can mean trusting channel-contribution estimates that never held up outside the vendor's chosen test conditions.
What the two libraries are
Both PyMC-Marketing and Meridian are open-source Bayesian MMM libraries that handle multi-geo, hierarchical models. Meridian is Google's open-source MMM framework, which Google made generally available to advertisers. The architectural differences are documented: PyMC-Marketing supports multiple sampling backends (NumPyro, BlackJAX, Nutpie), while Meridian is built on a fixed TensorFlow Probability backend. The two libraries also differ in how they parametrize media response curves, model seasonality, and set default priors.
What the published benchmark claims
A benchmark authored by the team that builds PyMC-Marketing tested both libraries on synthetic datasets spanning four scales, from a single-market startup dataset (104 observations) to a global-enterprise dataset (10,400 observations), a 100x range in data complexity. The published results report large gaps in sampling speed, fit accuracy, and convergence, favoring the library built by the team that ran the test.
The benchmark's author and the benchmark's favored subject are the same organization. That does not make the underlying numbers wrong. It does mean the numbers have not cleared the bar that would let a buyer treat them as settled fact.
What to require before trusting the result
| Question | Self-reported vendor benchmark | Independently replicated evidence |
|---|---|---|
| Who ran the test? | The team that builds the winning library | A party with no stake in the outcome |
| Who chose the test conditions? | The benchmark author, using their own synthetic data | Conditions set or reviewed independently |
| Has anyone outside the vendor reproduced the result? | Not yet, by definition | Yes, on separate data or a separate run |
| What happens if the result doesn't replicate? | No mechanism forces a correction | Failure to replicate is itself the finding |
| Is the code available to check? | The benchmark code is published on [GitHub](https://github.com/pymc-labs/mmm-param-recovery), which lets a team inspect the setup, but code availability is not the same as an independent re-run | Independent re-run confirms or contradicts the original claim |
Where this pattern shows up beyond MMM libraries
The same structural problem, a vendor's own test deciding whether its own product looks good, applies to any tool evaluated on a benchmark its maker wrote. Subconscious does not build or benchmark media mix modeling software, and the parallel here is about evidentiary standard, not a competing product claim. Subconscious reports 93% replication accuracy against real human outcomes across 350 or more published human studies (go.subconscious.ai/paper), reported as a distinct validation metric from a different domain, not a substitute for or a rebuttal of either MMM library's reported numbers. The relevant point of comparison is process: a claim backed by independent replication carries a different evidentiary weight than a claim backed only by its author.
A practical checklist for the reallocation decision
- Ask who ran the benchmark and whether they have a stake in the outcome.
- Ask whether anyone outside that team has reproduced the result on different data.
- Treat published code as an audit trail, not a substitute for replication.
- If no independent replication exists yet, treat the claimed gap as a hypothesis, not a budget input.
- Where the decision is large enough, run a smaller-scale internal test on your own data before committing spend to the recommendation.
Limitations
This comparison does not settle whether PyMC-Marketing or Meridian produces better channel-contribution estimates on any specific company's data. Both libraries have documented architectural differences worth evaluating in areas like sampler flexibility, memory footprint, and seasonality modeling. What is not yet established is the size of the performance gap claimed by the vendor's own benchmark, because that gap has not been independently reproduced. See Subconscious's approach to causal evidence and replication and its live results for how the same evidentiary question, does a claim replicate outside the team that made it, applies to causal experiment design more broadly.