Checking a Synthetic Experiment Result Before You Act on It
On this page
The decision: trust the number, or check it first
A synthetic discrete-choice experiment just told a research or insights leader which price, message, or feature wins. The next step commits budget: a launch, a pricing change, a positioning bet. Before that spend goes out, the question is whether this specific result is close enough to how real customers actually choose, or whether it needs an independent check first.
Getting this wrong is expensive in a specific way: the decision looks well-supported at the time, and the gap only surfaces after the budget is committed and the launch has shipped.
What does checking against published research look like?
A systematic review of health-related discrete choice experiments compares stated choices with revealed behavior using sensitivity, specificity and receiver-operating characteristics. Those classification measures assess a different task from rank correlation on estimated choice parameters. The review helps explain why a human choice instrument needs an external-validity check. It does not verify a synthetic population. Zhang et al., systematic review.
The July 2026 Subconscious working paper reports mean Spearman rank correlation on estimated choice parameters of 0.55 across roughly 300 replications and 0.73 across 43 design-filtered studies. That is a comparison of estimated parameter orderings, not sensitivity, specificity, sales lift or an accuracy guarantee. The paper is not peer reviewed. Working paper.
That figure is aggregate, not a per-run guarantee: it is measured across a body of replications, not certified fresh for every new experiment before a buyer sees results. A single synthetic run can still diverge from real behavior even when the reported average parameter-rank agreement is moderate or high.
When a corpus result cannot answer the buyer’s question
An aggregate parameter-rank result describes that validation corpus. A buyer still needs evidence for the specific audience, alternatives and decision outcome. A historical ranking match does not establish whether a current price or message changes purchase behavior.
For a consequential decision, separately commission a matched human comparison. Name the recruiter, inclusion criteria, assignment, outcome and analysis owner. A simulated result and a recruited-human choice result can then be compared under the same task.
A recruited human comparison is not a usability session, a clinical trial, or automatic proof that the finding will hold in market. It compares the simulated choice result with recruited participants answering the same question; it does not certify the launch, pricing, or messaging decision that follows.
What to check before you commit budget
- Does the result rest on a method with published replication evidence and stated limits, or only on internal claims.
- Is the decision high-stakes enough that the aggregate replication number is not sufficient, and a separately scoped human comparison is worth the added step.
- Does the validation step test the same causal question as the original experiment, not a different one.
Where does this fit before a launch decision?
Run the original synthetic experiment, check the method's published aggregate evidence, and scope a recruited human comparison for decisions where being wrong is expensive. The method evidence and its limits show what the public parameter-rank results do and do not cover. Use them to decide how much independent checking a given decision needs.
