Can You Trust a Vendor's Confidence Intervals? A Test Case in Honest Uncertainty Modeling
A confidence interval that never changes shape, never widens near a data gap, and never reacts to a structural break in the underlying process is not measuring uncertainty. It is decoration on a single point estimate. Before signing off on a pricing, product, or launch decision that rests on a vendor's error bars, the real question is whether those bars come from a model built to track uncertainty or from a model that reports one anyway.
What honest uncertainty modeling requires
A worked example from the open-source PyMC ecosystem: build a time-series model of atmospheric carbon dioxide from Mauna Loa readings and ice-core records, two data sources with different precision, different sampling gaps, and a known structural break where direct atmospheric monitoring replaced indirect proxies. A generic forecasting default cannot represent any of that. The example instead uses a semiparametric Gaussian process, a custom mean and covariance function, and a dedicated changepoint kernel to represent the transition between data sources, and it treats the exact location of some inputs as itself uncertain rather than fixed (PyMC example gallery, "Gaussian Process for CO2 at Mauna Loa").
Each choice exists because ignoring it would have made the resulting interval wrong at exactly the point where a decision-maker needed it to be right: the changepoint. A full Bayesian fit with NUTS sampling, rather than a single maximum-likelihood estimate, is what lets the model report a range instead of a guess dressed up as a range (PyMC example gallery, "Example: Mauna Loa CO2 continued").
The failure mode this guards against
A team that ships a pricing or launch decision on a confident-looking number, without checking whether it was ever uncertainty-tested, absorbs the cost of an action the underlying evidence did not support. The interval was cosmetic; the decision was not. The tell is structural: does the reported uncertainty change shape when the input data changes shape, or does it stay the same width regardless of what is happening underneath it?
Where this maps to Subconscious, and where it stops
Subconscious.ai runs controlled causal experiments on the same premise: a result is only as trustworthy as the honesty of its uncertainty. That is why experiments report confidence intervals and error bars where the study design supports them, rather than a single number standing in for a range. How that reporting works, and what it takes to design an experiment that can support it, is documented at /research. Replication evidence against real-human baselines, which is a separate question from whether a model's internal uncertainty is well-calibrated, is tracked at /leaderboard.
The parallel stops at the technique. Subconscious does not run Gaussian process time-series forecasting, and the Mauna Loa example above is not evidence of a Subconscious product capability. It is a methodological reference point: what it looks like to take a structural break seriously instead of averaging through it. A buyer evaluating any causal or forecasting platform, including this one, should ask the same question: does the reported interval come from a model built to represent what is uncertain, or is it attached after the fact to a number that was never in question?
What to check before you rely on someone else's interval
- Ask whether the interval's width changes at known structural breaks in the data, or stays constant regardless of input.
- Ask whether the estimate comes from a full posterior or sampling procedure, or from a single fitted value with a bound bolted on.
- Ask whether inputs with uncertain timing or measurement error are modeled as uncertain, or treated as exact.
- For a causal experiment specifically, ask whether the design supports a confidence interval at all, and whether that interval has been checked against real-human validation.
A vendor that can answer all four concretely is reporting uncertainty. A vendor that cannot is reporting a number.