What to Demand Before You Trust a Synthetic-Respondent Vendor's Accuracy Claim
On this page
A vendor's self-reported accuracy number is a marketing claim until you can reproduce it. The question is not what number the vendor publishes, but what evidence backs it and whether you can replicate it against your own historical data. It has to survive test-retest checks, item-level scrutiny, and an independent replication before a decision leans on it.
Why the aggregate number alone is not proof
A single correlation figure between synthetic and real respondent distributions can hide a wide spread. A platform that reports one clean number might be averaging a long tail of low-agreement items against a smaller set of near-perfect ones, a different pattern than a methodology that holds steady across every item. Treat any headline accuracy figure as a starting point for questions, not the answer.
What to ask a vendor before you believe the number
| Evidence type | What it shows | Red flag if the vendor can't produce it |
|---|---|---|
| Test-retest reliability | Whether re-run dispersion matches human re-fielding dispersion, and results hold across prompt paraphrase and model version | Only reports one-time accuracy, or near-perfect stability with no comparison to human re-fielding variance |
| Item-level correlation | Whether accuracy holds question by question, not just averaged across a study | Reports only a single aggregate figure |
| Independent replication | Whether an independent evaluator reproduces prespecified results | Calls a vendor-run recreation of a third-party paper independent |
| Boundary conditions | Which question types, audiences, and decision contexts the accuracy claim does not cover | Presents the accuracy figure as universal, with no stated limits |
Reproducibility is useful, but stability alone can also reflect insufficient heterogeneity. Examine variation across prompts, seeds and model versions, compare it with relevant human re-fielding variation, and check whether subgroup and item-level distributions remain plausible.
What the published research actually supports
The academic literature on LLM-conditioned synthetic respondents is worth reading directly, not secondhand from a vendor's own summary of it. Argyle, Busby, Fulton, Gubler, Rytting, and Wingate pulled respondent demographics from the American National Election Studies, built those into backstories for a language model, and found the resulting response distributions correlated strongly with survey distributions across political-attitude batteries, including within demographic subgroups (Argyle et al., 2023, Political Analysis).
Horton's NBER working paper tests language-model agents in classic behavioral-economics scenarios, including social preferences, fairness and status quo bias. Qualitative agreement in these tasks does not establish product-specific effect magnitudes.
Bisbee, Clinton, Dorff, Kenkel and Larson's 2024 Political Analysis paper examines synthetic ANES feeling-thermometer responses. It reports understated response variance and sensitivity to prompts and model versions, among other discrepancies.
These papers test political survey responses and elicited economic choices, including hypothetical judgments. They do not directly validate the proposed randomized multi-attribute product conjoint in your category. Treat their results as task- and configuration-specific evidence, then ask what matches your intended outcome.
Where the accuracy gap is largest
The following are reasons to request additional evidence, rather than a universal ranking of where errors are largest:
- Novel behavior. Check performance on held-out outcomes and stimuli absent from the grounding material.
- Specialized audiences. Verify profile coverage, weighting and subgroup agreement rather than assuming a job-title prompt represents the audience.
- Audited claims. Determine the substantiation requirements for the specific claim and domain; an unrelated synthetic benchmark cannot meet them.
- High-stakes, real-consequence decisions. When both the synthetic and human tasks are hypothetical, agreement checks that task rather than consequential behavior. A purchase or live intervention comparator measures another endpoint and needs its own design.
How to run the validation yourself
A procurement team does not have to take any vendor's word for its accuracy number. The workflow is the same regardless of platform:
- Find a past study in your own archive with a known outcome distribution.
- Recreate the demographic, role, and segment specifications that defined the original sample.
- Run the equivalent question battery using the same stimuli and framing as the original study.
- Compare the new distribution to the original, both in aggregate and item by item.
- Decide from your own numbers, not the vendor's marketing page, whether the method is accurate enough for the decision.
Use a prespecified holdout unavailable to the vendor during tuning, document model and prompt versions, and compare matched stimuli and outcomes. Account for the human comparator's variability and the limits of transport beyond the archived study. Public availability creates a leakage risk; it does not prove that any particular study was used for training.
Where a causal-experiment approach fits
Ask Subconscious or any vendor to specify which evidence comes from a historical benchmark, which pilot data are held out and whether a new human comparison is included in the proposed scope. A historical benchmark does not guarantee that every current study has an independent human baseline or leakage check. Define the same estimand and stimuli for the human comparison where feasible; see the validation methodology.
Limitations
No headline score validates every audience or question. Match the evidence to the decision and cost of error, and state what the study cannot establish. The leaderboard describes published comparisons; request the actual protocol for the proposed pilot.
Next step
Before signing, ask for the exact configuration, scoring rule, item and subgroup results, stability checks and a prespecified independent comparison. Discuss your decision and the evidence needed to use a modeled result.