From a question
to a decision.
Define the alternatives. Choose the population. Run a controlled experiment and inspect what changes, for whom, and with what uncertainty.
01. Make the
decision concrete.
Imagine a grocery brand choosing between a lower price, a new formulation and a local-sourcing story. Start by specifying the options and the choice you want to understand.
What you define
The decision, its alternatives, the relevant population and the outcome. Customer research and product context can help make those choices specific.
A synthetic grocery example runs through this walkthrough.
What stays comparable
Use an experimental design that makes the contrast clear. Keep the question and measurement consistent as the conditions change.
02. Understand the
population and inputs.
Different sources measure different things. Geography, observation period, coverage and customer context determine what an input can tell you.
Source experiences load as you reach this section.
Loading evidence trace…
Loading population…
Loading outcome comparison…
Loading preference landscape…
Loading effect intervals…
Traits belong
to the same people.
Matching individual averages does not establish that combinations of traits are realistic. Geography, age, household and behavior need context.
Conceptual illustration of joint traits. These marks are not population estimates.
The number of combinations grows quickly.
With the same number of levels per trait, the number of possible combinations is levels raised to the number of traits.
100
10 levels × 2 traits. Possible combinations are not evidence of observed coverage.
Different checks
answer different questions.
Population description, experimental estimation and validation are related. None substitutes for the others.
Who does this describe?
Inspect the definition, geography, period, coverage and source of the population evidence. A matching average does not guarantee representative joint behavior.
Keep the evidence inspectable.
A source receipt explains the input. An experiment report explains the comparison. A validation study explains how a method was evaluated.
Read the validation evidence ↗Change the conditions.
Compare the response.
People do not all make the same trade-off. Inspect how groups respond, alongside the overall result.
Change the product story
Illustrative choice shares. Synthetic data, not a model result.
Illustrative report · synthetic results
Everyday grocery launch
The estimate is only part of the answer.
In this synthetic example, the price and ingredient contrasts are positive. The local-sourcing interval crosses zero, so its positive point estimate alone would be a poor basis for a confident recommendation.
03. Inspect the evidence.
Then make the decision.
A result should show the comparison, the estimated effect and its uncertainty. A simulation result is an input to a business decision, not a guarantee of its real-world outcome.
Read the result
Compare effect estimates, trade-offs and differences between populations. Keep the experimental conditions and source limitations alongside the interpretation.
Explore customer examples ↗Check the method
Inspect the human baseline and scoring rule used to evaluate the method. Separate a model’s evaluation from the evidence for your specific engagement.
Inspect the evidence ↗Bring the decision
on your desk.
We will work through the alternatives, the evidence and what an experiment can answer.
Test a decision