Subconscious

Why a Single Forecast Number Hides the Risk You're Actually Taking

Edited by Avi Yashchin, CEO and Co-Founder · · Updated

On this page

A point forecast leaves out the uncertainty and assumptions that determine a decision's downside. For a pricing or message change, report the expected outcome, relevant uncertainty, and the cost of acting or waiting. An interval is useful context; the investment rule also needs a business threshold and evidence that the estimate applies to the intended population.

Why a point estimate hides risk it doesn't report

The PyMC stochastic-volatility example models a latent volatility process using S&P 500 returns from May 2008 to November 2019. It illustrates time-varying return variance and posterior uncertainty. Buyer-response uncertainty can arise from different sources, including sampling, model misspecification, task behavior, and changing markets; the financial example does not prove that all of them share one volatility model.

For a new pricing decision, distinguish uncertainty about the estimated effect from variation in future outcomes and from assumptions about transport to the market. A single estimate conceals those distinctions. A narrow conditional interval can still coexist with substantial model or population mismatch.

What uncertainty does the decision need?

Choose the estimand and report the appropriate confidence, credible, or predictive interval with its assumptions. A randomized comparison may identify a treatment contrast under the design; a predictive forecast answers a different question. Human stated tasks, generated tasks, and actual transactions are different endpoints. An interval around one does not establish the others.

How to apply this before the next decision

  1. Ask what the estimate and interval describe. Record the endpoint, population, interval type, and assumptions.
  2. Check identification and validation. A treatment contrast needs a credible design or identification strategy. A forecast needs out-of-sample prediction and calibration checks.
  3. Set the business threshold. Compare expected contribution, implementation cost, downside tolerance, and the value of further information. Crossing zero is not a universal act-or-stop rule.
  4. Resolve consequential gaps. Obtain relevant human or live evidence when a modeled task does not support the endpoint required by the decision. Confirm scope and responsibilities rather than assume an automatic handoff.

A worked business threshold

In a hypothetical campaign, estimated incremental contribution is $1.20 per eligible customer with a conditional interval from −$0.30 to $2.70, while deployment costs $0.80 per customer. The central net estimate is $0.40, but the interval does not determine a rollout by itself. Assess the downside, uncertainty around costs, reversibility, and the cost of a further test. A confidence interval is not a probability distribution of possible effects; any expected-loss or probability calculation needs an appropriate model and explicit assumptions.

Decision sequence: define the endpoint, inspect uncertainty and validation, set the business threshold, and assess whether further evidence is valuable.
An interval and the business threshold inform a decision together.
Four checks for a forecast decision: estimate meaning, interval assumptions, evidence fit, and the cost of acting or waiting.
Conditional uncertainty does not include every source of decision risk.

Next step

Before committing budget, record the intervention or forecast, business endpoint, cost threshold, interval assumptions, and missing validation. Use the research process and aggregate validation evidence to assess scope, or book a review around the decision and its downside.