Validation
methodology
Synthetic respondents are useful only when their boundaries are public. This page explains how we validate the method, what 93% means, and where the evidence should not be stretched.
Read the paper↗︎// Protocol
How validation works
We run randomized controlled experiments on LLM-driven synthetic respondents, then reconcile those outputs against human holdouts wherever the decision context supports direct comparison.
The benchmark also replicates 350+ published human studies across 20+ domains. We reconstruct the original survey, match respondent profiles from the paper, estimate the same statistical model, and compare the simulated result back to human data.
That means the test is not whether a model writes a plausible answer. The test is whether it preserves the structure of human choice under an auditable protocol.
Randomized experiments first. Claims second.
Published methodology
Method paper
The paper documents how we validate synthetic respondents against published human studies and human holdouts.
It is the source of record for the public methodology while the benchmark and leaderboard continue to evolve.
350+ replicated studies. Human holdouts. Auditable scoring.
What 93% means
Reconciled against humans
The 93% figure summarizes agreement against real human behavioral outcomes across the validation set. It is a reconciliation benchmark, not a claim that every synthetic respondent is individually correct.
Read the paperObserved outcomes, not stated intent
The benchmark is grounded in human study results and holdout comparisons. We care whether the simulation preserves behavior under trade-offs, not whether it sounds like a survey respondent.
Multiple scoring lenses
Coverage, parameter sign agreement, rank-order correlation, and parameter-distance metrics compare simulated estimates with the human study from several angles.
View leaderboardContext-bound evidence
A strong score means the protocol recovered important properties of human choice in the tested domains. It does not mean synthetic respondents are universally reliable without local validation.
Living evidence
Leaderboard
The leaderboard is the public surface for ongoing benchmark evidence, showing model behavior across replicated studies and domains.
Study reconstruction
Each run starts from the original decision environment rather than a generic prompt, preserving the attributes and trade-offs that made the human study measurable.
Human model parity
We estimate the same statistical model used in the source study on AI-generated responses before comparing back to the human result.
Versioned protocol
Prompts, scoring logic, and assumptions are treated as part of the evidence so collaborators can audit, challenge, and extend the work.
Domain-level rollups
Public summaries help teams see where synthetic respondents are closer to human behavior and where the method needs more validation.
Decision relevance
The benchmark exists to decide when simulations can support pricing, messaging, product, and policy experiments, not to reward benchmark theater.
// Who we are
Leadership
Avi Yashchin
CEO and Co-Founder | Ex-Two Sigma quant. 2x exits to S&P 500. Guest lecturer at Wharton. Johns Hopkins CS, NYU Stern MBA
Dr. Subodh Dubey
Behavioral Economics | 20+ published papers. Pioneer in discrete choice theory
Tatevik Karapetyan
Research | Behavioral science researcher. Experimental design and causal inference
Connor Joyce
Behavior Change | Microsoft. Behavioral science product leader. Ex-Twilio
// What synthetic respondents cannot yet do
Limitations
Useful validation includes the boundary conditions.
Novel populations
Synthetic respondents should not be treated as validated for populations missing from the benchmark, absent from customer calibration data, or defined by experiences the model cannot observe.
Low-base-rate behaviors
Rare decisions, crisis behavior, fraud, and edge-case harms need human evidence or operational data. Synthetic samples can understate tails when the base rate is thin.
Non-US generalization
US-calibrated evidence does not automatically transfer to non-US markets. Language, institutions, regulation, and culture can move the choice structure enough to require local validation.
New contexts require holdouts
When the decision environment changes materially, we use holdouts or fresh validation before treating a simulation as decision-grade evidence.
Frequently asked questions
// Validation questions
For methodology review, contact partners@subconscious.ai.
Is 93% a universal accuracy guarantee?
No. It is a validation-set result against real human behavior. The page names the limits because new populations, rare behaviors, and non-US contexts need their own evidence.
Why use published studies?
Published studies give us known decision environments, documented respondent profiles, and human results that can be reconstructed and challenged.
Where does the leaderboard fit?
The leaderboard is living evidence. It summarizes how models perform as the replicated-study set, scoring logic, and model landscape evolve.
Can enterprise teams audit the method?
Yes. We share protocols, assumptions, and scoring detail with qualified research, legal, procurement, and executive stakeholders.
// Reconciliation metrics
Validation signals
| Signal | What it checks | Why it matters |
|---|---|---|
| Coverage probability | How often human-study parameters fall inside the simulated confidence interval | Shows whether uncertainty bounds are honest enough to use |
| Parameter sign agreement | Whether effect directions match the original human study | Prevents recommendations from flipping the causal story |
| Rank-order correlation | How well simulated preference order matches human preference order | Tests whether the model preserves relative trade-offs |
| Parameter distance | How closely simulated parameter values match human estimates | Keeps the benchmark tied to magnitude, not only direction |
// Deployment standard
Use the method where it has earned trust
Synthetic respondents are a way to run more experiments, faster. They are not a replacement for judgment, field evidence, or governance where stakes are high and the validation surface is thin.
Our deployment standard is simple: publish the method, show the reconciliation to humans, and name the conditions where more validation is required.
View the leaderboard to learn how behavioral simulation can transform your decision-making.