Can You Trust a Vendor's Uncertainty Intervals? A Test Case in Honest Uncertainty Modeling
On this page
An uncertainty interval that never changes shape, never widens near a data gap, and never reacts to a structural break in the underlying process deserves a second look. Constant width alone does not prove the interval is dishonest; the real test is calibration against actual outcomes, but non-responsive width is still a reason to ask how the interval was built. Before signing off on a pricing, product, or launch decision that rests on a vendor's error bars, the real question is whether those bars come from a model built to track uncertainty or from a model that reports one anyway.
What honest uncertainty modeling requires
The continued PyMC Mauna Loa example combines atmospheric readings and ice-core records. Its changepoint covariance represents the industrial revolution around 1800; distinct noise terms handle measurement-source differences. Uncertain input dates and GP hyperparameters add other uncertainty. The first notebook instead uses monthly atmospheric data and MAP fitting.
The continued example uses NUTS to propagate GP hyperparameter uncertainty. That is one approach to this model’s uncertainty, rather than a universal requirement that every honest interval come from a posterior.
What failure mode does this guard against?
A team that ships a pricing or launch decision on a confident-looking number, without checking whether it was ever uncertainty-tested, absorbs the cost of an action the underlying evidence did not support. The interval was cosmetic; the decision was not. One thing worth checking is structural: does the reported uncertainty change shape when the input data changes shape, or does it stay the same width regardless of what is happening underneath it? Non-responsive width is not proof of miscalibration by itself, but it is a reason to check calibration directly.
Where does this map to Subconscious, and where does it stop?
Subconscious.ai runs controlled causal experiments on the same premise: a result is only as trustworthy as the honesty of its uncertainty, though a well-calibrated interval around a poorly identified estimate can still mislead, so calibration is necessary but not sufficient. That is why experiments report confidence intervals and error bars where the study design supports them, rather than a single number standing in for a range. How that reporting works, and what it takes to design an experiment that can support it, is documented alongside the published method. Replication evidence against real-human baselines, which is a separate question from whether a model's internal uncertainty is well-calibrated, is tracked at /leaderboard.
The parallel stops at the technique. The Mauna Loa notebooks document an external GP forecasting workflow. The public company benchmark does not document its deployment by Subconscious; confirm the implementation and deliverables for a proposed engagement. It is a methodological reference point: what it looks like to take a structural break seriously instead of averaging through it. A buyer evaluating any causal or forecasting platform, including this one, should ask the same question: does the reported interval come from a model built to represent what is uncertain, or is it attached after the fact to a number that was never in question?
What to check before you rely on someone else's interval
- Ask whether the interval's width changes at known structural breaks in the data, or stays constant regardless of input when the estimated quantity varies over time; a static parameter interval needs checks suited to its own target.
- Ask how the interval is constructed: analytic standard errors, asymptotic approximations, bootstrap, conformal methods, or a Bayesian posterior have different assumptions and meanings.
- Ask whether inputs with uncertain timing or measurement error are modeled as uncertain, or treated as exact where that uncertainty can affect the target estimate, in forecasting or causal analysis.
- For a causal experiment specifically, ask whether the design supports a confidence interval at all, and whether that interval has been checked against real-human validation.
Treat these as diagnostic prompts. Request relevant empirical coverage or calibration, sensitivity analyses, and evidence for identification and human transport. Answering construction questions alone does not establish interval validity or a safe decision.