Should You Trust a Raw Tracking-Poll Average? A Hierarchical Model Answers That
On this page
A tracking average can mix trend movement with source changes and noise. A hierarchical model can estimate those components under assumptions; it does not automatically recover the true trend. Check source overlap, identification constraints, changing house effects, and predictive adequacy before acting on the modeled movement.
The decision this affects
Any team that watches a repeated measurement over time (approval, awareness, purchase intent, brand favorability) eventually has to decide whether a change is worth acting on. A raw average risks two mirror-image mistakes: treating noise as a real shift and moving a launch, price, or message in response to nothing, or dismissing a real shift as noise and missing the window to respond, because it never separates the latent trend from the biases and sampling error layered on top of it.
Why Does the Raw Average Mislead?
The PyMC Labs Markov-process example analyzes French presidential approval polls from 2002 to 2021, ten pollsters, and four methods. It illustrates conditional trend and house-effect estimation, rather than a validated deployment for a buyer’s tracking series.
A pollster-by-method breakdown of the same data shows the pattern: face-to-face polls report systematically lower approval than average; phone polls report slightly higher. Individual pollsters carry their own persistent lean independent of method. None of this is political bias. It is statistical bias: a sampling method or a house's weighting choices nudging a number in a consistent direction relative to the average across pollsters and methods, not relative to some independently known true value.
What Separates Signal From Noise?
The fix is to build the reporting biases explicitly into the estimate instead of averaging around them. Political scientists use the same hierarchical, dynamic estimation approach to recover latent public opinion from noisy survey series (Caughey and Warshaw, MIT/Political Analysis) and to pool polls with contextual information for dynamic forecasts (Political Analysis, Cambridge University Press).
- Treat the true trend as a hidden (latent) state. The model never observes the "real" approval level directly, only noisy polls that are a function of it.
- Let that hidden state move as a random walk. Approval this month depends on approval last month plus some innovation, not on some independent draw each period.
- Give every (pollster, method) pair its own bias term. House effects are estimated from the data rather than assumed away.
- Model overdispersion explicitly. A random-walk-plus-bias model evaluated with a plain binomial likelihood still underestimates how much polls actually vary. A beta-binomial likelihood, adding one parameter to separate variance from the mean, closes much of that gap.
- Partially pool across groups instead of fully pooling or fully separating them. Treating every president's trend as identical understates real differences between terms; treating each term as fully independent throws away shared information. A hierarchical structure, where each president's trend is drawn toward a common trend but free to deviate, does both jobs.
- Constrain the bias terms. A free intercept alongside free house effects does not resolve the ambiguity between the trend's level and the average reporting bias; a sum-to-zero constraint, a reference pollster, or an informative prior on the house effects does.
The published version of the model, correctly parameterized, tracks each president's approval trajectory through its natural cycles and recovers plausible pollster- and method-level bias estimates relative to the average across pollsters and methods: face-to-face confirmed low, phone confirmed slightly high, and individual pollster leans consistent with what a practitioner who collects these polls by hand already expected.
How Does This Differ From a Controlled Experiment?
The model above estimates a hidden state from repeated observational measurements. It never intervenes on anything. It answers "what is the true trend, net of measurement noise and reporting bias?" for an outcome that already happened.
A smoothing model does not identify the effect of a price, message, or launch action. For Subconscious experiments, inspect the randomized contrast, task endpoint, uncertainty, and matched human evidence. A simulator interval does not cover human transport error, and an aggregate research benchmark does not validate a new action.
The shared discipline is not the method. It is the refusal to read a single number at face value: both approaches insist on quantifying uncertainty and separating a real effect from an artifact of how the data was collected.
What this means for evaluating any tracked metric
| Question to ask about a tracked metric | What a raw average tells you | What a bias-and-noise-corrected estimate tells you |
|---|---|---|
| Did the trend move, or did the source mix change? | Components are mixed unless separately modeled | Conditional separation when sources overlap enough and house-effect assumptions are supported |
| How much noise affects the metric? | Can be quantified if sampling/design information is available | Modeled dispersion and uncertainty under the specified likelihood |
| Is one pollster or method systematically pulling the number in a direction? | Invisible unless checked by hand | Estimated directly, with uncertainty |
| Can this movement support action? | Check design, source mix, and uncertainty | Also check sensitivity and whether the decision needs intervention evidence |
Check whether sufficient sources overlap in time to identify house effects separately from trend. Compare fixed and changing house-effect specifications; stable offsets can misattribute a changing source effect to the latent trend.
Limitations
A number without its limits is a marketing claim. Noise correction like this answers a narrower question. It tells you what the underlying trend probably is, given the measurements you already have; it says nothing about what would happen under a different action, price, or message, because it never varies the conditions the data was collected under. A team that needs to know whether a specific intervention moves an outcome needs a controlled experiment, not a smoother trend line. Even a well-specified hierarchical model inherits the limits of its inputs: retrodictive checks in the source model show it still underestimates sharp swings from real-world events.
A matched human study tests agreement for a stated treatment and outcome and can fail. It does not turn an observational trend estimate into an intervention effect.
Next step
If the underlying question is which action actually changes the outcome, that calls for a controlled experiment, not a smoother read of an existing trend. See how a causal experiment differs from an observational read of the same market, or look at published replication results for a sense of how closely simulated experiments track real human behavior.