Checking a Synthetic Experiment Result Before You Act on It
The decision: trust the number, or check it first
A synthetic discrete-choice experiment just told a research or insights leader which price, message, or feature wins. The next step commits budget: a launch, a pricing change, a positioning bet. Before that spend goes out, the question is whether this specific result is close enough to how real customers actually choose, or whether it needs an independent check first.
Getting this wrong is expensive in a specific way: the decision looks well-supported at the time, and the gap only surfaces after the budget is committed and the launch has shipped.
What checking against published research looks like
Subconscious designs synthetic experiments to be checked against independent, published choice research, not to stand on their own. A systematic review and meta-analysis of discrete choice experiments in health-related research found that prediction-accuracy comparison against real-world choice outcomes is an established, peer-reviewed practice (PMC, National Library of Medicine), not a bespoke internal metric invented for one vendor's dashboard.
Subconscious reports a 93% replication accuracy against independent studies as the standing evidence that its method matches real-world choice outcomes (go.subconscious.ai/paper).
That figure is aggregate, not a per-run guarantee: it is measured across a body of replicated studies, not certified fresh for every new experiment before a buyer sees results. A single synthetic run can still diverge from real behavior even when the underlying method replicates well on average.
When aggregate replication accuracy is not enough
Aggregate accuracy answers "does this method generally match real choices." It does not answer "does this specific result, for this specific audience and decision, match real choices." For a high-stakes call, the second question sometimes needs its own check.
Subconscious can test or validate studies with real human participants: a team can move a synthetic result into recruited real-human validation without changing the causal question being tested, so a high-stakes decision gets its own check instead of borrowing an aggregate number it was never measured against.
Real-human validation is not a usability session, a clinical trial, or automatic proof that the finding will hold in market. It confirms the choice result against recruited participants answering the same causal question; it does not certify the launch, pricing, or messaging decision that follows.
What to check before you commit budget
- Does the result rest on a method with published, independent replication evidence, or only on internal claims.
- Is the decision high-stakes enough that the aggregate replication number is not sufficient, and a dedicated real-human validation run is worth the added step.
- Does the validation step test the same causal question as the original experiment, not a different one.
Where this fits before a launch decision
Run the original synthetic experiment, check the method's aggregate replication track record, and add real-human validation for decisions where being wrong is expensive. See the research methodology, the case studies of past runs, and the leaderboard of replication results across studies before deciding how much independent checking a given decision needs.
