How to Evaluate Market-Simulation Evidence Before Committing Research Budget
On this page
Evaluate a market-simulation vendor by the evidence behind the decision, not the largest accuracy figure in its pitch. The decisive questions are whether the study isolates an action, reports uncertainty, replicates, and can be tested with real people without changing the causal question. A single correlation snapshot cannot answer those questions.
The cost of choosing poorly is not limited to the contract. A team can commit research budget, act on an unverified number, and discover after launch that the simulated result did not hold against customer behavior.
The budget decision is about evidence
Public case material shows that market simulation is being applied in wealth and asset management (EY). That is not, by itself, evidence that a result replicated or that a proposed action caused the measured outcome.
The useful comparison is not a category label or sales model. It is the type of proof a vendor can put behind the decision.
Compare the proof each method can provide
| Evidence offered | What it can support | What it cannot establish |
|---|---|---|
| Correlation against a benchmark | Agreement on a specified response, parameter or effect-estimate target | Effect-size calibration or validity for a new decision unless separately tested |
| Replicated randomized experiment with confidence intervals | The estimated effect of a defined action and the uncertainty around that estimate | Automatic performance across every market, buyer group, or launch |
| Real-human validation matched to the same causal question | Whether simulated and participant results answer the same decision question | A usability observation, clinical outcome, or guarantee of market performance |
First ask what is correlated: raw responses, estimated choice parameters, or treatment effects from randomized human studies. A correlation of effects can be relevant validation evidence, but its source design, scoring and holdout status still matter. Randomization identifies a contrast within the tested population; a simulator's contrast needs matched human evidence before it becomes a claim about real customers.
Questions that expose weak proof
Ask for answers that can be reviewed before procurement:
- What exact action and outcome did the study test?
- Was assignment randomized, and what comparison condition was used?
- How was uncertainty reported?
- Was the result replicated, and under what population and market conditions?
- Can the same causal question be tested with real people?
- Which decisions fall outside the method's stated limits?
If a deck repeats a benchmark without an inspectable source and scoring definition, exclude that performance claim from the procurement decision until the supplier provides them.
How does Subconscious test the action?
Subconscious runs randomized comparisons on a market simulation and reports intervals where the study design supports them. The estimate concerns the generated response the study defines, not observed behavior. When a decision warrants an additional check, scope a matched human study separately, with the same alternatives, population and endpoint. The simulated study does not guarantee it.
A simulated study is not an observed usability session, and it does not guarantee market performance. The Subconscious method is best suited to choices that can be expressed as a defined action, comparison, and measurable outcome.
For a simulated study, ask which data grounds the audience, which geographies and segments it covers, and where that representation has been validated. A configured audience is separate from recruited participants and fielded sample size. Published benchmark results provide another artifact to inspect, with a defined task and limits.
How do you set the procurement gate before reviewing proposals?
Require a pre-specified causal question, the experiment design, confidence intervals, replication evidence, written failure conditions, and a real-human validation plan when the stakes justify one. Apply the same standard to every vendor.
Simulation evidence has limits. It cannot support a price or delivery-speed comparison. It also cannot convert an unpublished method or a promotional number into proof. If a supplier cannot provide the required artifacts, include the risk of an unverified result in the budget decision.
Teams with a live market choice can discuss an appropriate study design.