What a World Cup Forecasting Model Teaches About Trusting a Model at All

Edited by Avi Yashchin, CEO and Co-Founder · August 18, 2026

A model can hit its headline accuracy target and still be wrong on the exact numbers a decision depends on. That gap is the reason a calibration check belongs before every go/no-go call built on a probabilistic or simulation-based forecast, not after.

The decision this is really about

A team weighing a market-entry date, a launch window, or a resource allocation off a model's probabilities is not asking "is this model roughly right." It is asking whether the model is right on the specific cells the decision turns on. Those are two different questions; a model can pass the first while failing the second without it ever showing up in a top-line accuracy number.

A public forecasting project built to predict the 2026 World Cup group stage makes the gap concrete. It is not a Subconscious result, a customer study, or a causal-effect estimate: it is one modeler's independent forecasting exercise, published for its methodology. But the discipline it demonstrates is what separates a model worth acting on from one that only looks worth acting on.

Build the model, then interrogate it

The forecast represented each national team's attacking and defensive strength as an unobserved rating. Goals were count outcomes, and new matches changed the ratings instead of leaving them fixed. The result was a distribution over possible scores, group standings, and advancement outcomes rather than one supposedly certain answer.

Producing probabilities is only the beginning of validation. In a posterior predictive check, draws from the fitted model generate plausible new datasets. Those replications are then tested against observed patterns, especially the individual quantities that control the decision. The Stan project's documentation describes this method as a way to discover when a model fails to reproduce important features of the observed data (Stan Documentation, "Posterior and Prior Predictive Checks").

Where the check caught the model

The forecast's aggregate numbers looked fine: the share of teams scoring 0, 1, 2, or more goals in a match, and the 0-0 scoreline rate, all fell inside the model's predicted range.

Real matches ended level more often than the model's assumptions allowed. The model treated each team's goal count as statistically independent of the other's, a common simplification traceable to score-modeling work going back to Maher (1982), but a posterior predictive check flags which statistics the model fails to reproduce, not why; one plausible explanation is that two sides sitting at 1-1 late in a match play differently than the independence assumption implies. Whatever the mechanism, the result showed up as a small, specific miscalibration: too few predicted 1-1s and 2-2s.

The gap looks small in isolation, roughly one percentage point on the draw rate. But it sits exactly on the cells a group-stage table is built from: points, goal difference, and goals scored, all determined by exact scorelines. So a deficit that small sits on the exact cells a group-stage table is built from, and whether it shifts which teams the model says will advance is a question the model can answer by simulating advancement probabilities under both specifications, not something to assume. Work on visualizing Bayesian model checks makes the same point: predictive checks find where a fit is locally wrong, not just whether it is globally acceptable (Gabry, Simpson, Vehtari, Betancourt, and Gelman, "Visualization in Bayesian Workflow," Journal of the Royal Statistical Society: Series A, 2019).

Fix the flagged cells, not the whole model

Because the check pointed at specific cells rather than a vague "the model is off," the fix could be specific too: a small mixture term that adds extra probability to level scorelines at 1-1 and 2-2, tuned so it moved only the cells the check flagged and left the calibrated cells, including the 0-0 rate, untouched. Re-running the same predictive check afterward showed every flagged statistic back inside the model's interval.

That re-check confirms the correction moved the flagged cells as intended, but a fix tuned to those exact statistics will pass the same check by construction, so it is not independent evidence. A model is "fixed" only when held-out evidence, not the same diagnostic that caught the problem, confirms it. Before the forecast was trusted with the 2026 field at all, the corrected version and its predecessors were also backtested against the actual 2022 tournament: scoring group matches the model had never seen and comparing the result against baseline forecasts.

What this means for trusting a model before a decision

The lesson generalizes past football. A model built to inform a consequential call, such as where to launch, when to enter a market, or which segment to fund, should be checked against the specific quantities that call depends on, not against a single top-line accuracy figure.

The same discipline applies when a study moves from a simulated experiment to real-human validation without changing the causal question. Subconscious can validate studies with real human participants, comparing modeled responses against evidence from people while preserving the question under study. The replication leaderboard shows how that checking process holds up across published studies, and how Subconscious works explains where it fits into a team's existing decision process. For background on the research discipline behind that validation step, see the research program.

Before a team commits budget on the strength of any model's output, probabilistic, simulation-based, or otherwise, the standing question is the one this forecast had to answer for its own scorelines: not "is the aggregate accuracy good," but "is the model calibrated on the exact numbers this decision is riding on."