Synthetic Respondent Validation: How Simulated Results Compare With Human Studies

Edited by Avi Yashchin, CEO and Co-Founder · November 12, 2026

How accurate are synthetic respondents? On replications of published choice experiments, Subconscious's simulated respondents agree with human respondents on the ordering of attribute effects partially. The July 2026 working paper, summarized on fidelity.subconscious.ai, reports a mean Spearman rank correlation of estimated choice parameters of 0.55 across roughly 300 replicated studies and 0.73 across the 43 studies that pass design filters. Agreement on the size of an effect is not yet validated. Of the 308 detailed comparisons in the public study registry on 2026-11-12, 281 could not compare sizes, 20 were inconclusive and 7 were not equivalent; where the registry states a size ratio, synthetic effects ranged from 1 percent of the human size to 3.8 times it. A simulated study can screen which attribute changes move a choice where a matched published study held; a human study is needed for the size of the effect and for any revenue or share forecast.

This page grades Subconscious's simulated results against the published human studies they replicate, study by study, with the misses included. Subconscious runs controlled choice experiments on simulated buyers and reports the estimated effect of each change with a confidence interval (method described on fidelity.subconscious.ai); the evidence record and the methodology validation case carry the paper's aggregate figures with their limits. Every figure below comes from a public, dated source, linked beside it.

The answer, in one table

What the published paper reports

The paper replicated roughly 300 published stated-preference studies through one pipeline: a factorial design fixes the experiment, simulated respondents are drawn to match census statistics from the American Community Survey and the National Household Travel Survey, and a discrete choice model estimates the parameters. The score is the Spearman rank correlation between the estimated choice parameters of the synthetic sample and those reported by the human study. It is not raw-response agreement and it is not an accuracy percentage.

The two headline figures describe two cohorts. Across all replications the mean is 0.55. After order-invariance design filters, 43 studies remain and their mean is 0.73. The paper is a working paper by Avi Yashchin, dated July 2026, and has not been peer reviewed; fidelity.subconscious.ai labels both figures "working-paper claim" and notes that the per-study replication data is not yet public.

The same paper reviewed 381 papers, found on arXiv, that use language models as stand-ins for human subjects. Of those, 104 compare model output with real human data; of the 104, 24 test the effect of an intervention, and four of the 24 conclude that the model recovers the human effect well. Each paper was coded from its title and abstract by a language model, and each verdict is the paper's own self-report; the 381-paper funnel is published with its corpus. The question a pricing or messaging decision turns on is the interventional one: whether the synthetic population responds to a change like people.

Study by study: the registry on 2026-11-12

Subconscious publishes a study registry with one page per replicated human study. On 2026-11-12 it listed 581 selected experiments from 552 papers, frozen for audit on 9 November 2026, with 369 marked completed. 308 study pages carried a detailed comparison, from saved runs dated between 2026-09-27 and 2026-11-08. Each comparison states how many of the human study's effects (the significant ones, where the page says so) the synthetic run got in the right direction, the rank correlation across effects, and whether the effect sizes were comparable.

The table below lists every detailed comparison in which the registry states the human sample size: 17 of the 308, largest sample first. Each study name links to its registry page; studies are named by their registry key. The direction count and rank correlation come from each page's "Interpretation and limits" line. The downloadable file holds all 308, with the registry's own verdict wording and a link to each study page.

Across the 308 detailed comparisons, the registry's verdicts were: direction holds in 127 studies, mixed in 92, fails in 89; rank holds in 103, mixed in 93, fails in 112. Direction and rank both hold in 78 studies and both fail in 73. These are counts of the registry's own labels on the retrieval date. They are not a new accuracy figure and they will change as runs are added or verified.

Two cautions belong beside this table. First, verification is incomplete: the registry's index lists open blockers on most listed experiments, and its audit note says historical runs lack resolved-model receipts. Second, the paper's aggregate figures come from the pipeline the July 2026 paper describes, and the registry's comparisons come from later runs, so the two are not the same measurement. Subconscious has not published a verified aggregate of the registry runs. When that verification finishes, this section will carry it with the date.

Download the table: synthetic-respondent-validation-2026-11-12.csv. It holds 308 rows, one per registry study page with a detailed comparison: the human sample size where stated, the saved-run date, the number of paired effects, the share in the same direction, the rank correlation, the size ratio where stated, the registry's three verdicts, its interpretation sentence and the source URL.

Where the simulation fails

On the Hainmueller and Hopkins immigrant conjoint (single-study walkthrough), the registry's saved run of 2026-11-06 ranked 29 significant effects at 0.83 and got 26 in the right direction. Among its reversed effects, the human penalty on applicants from Iraq came out slightly positive (human -0.106, calibrated synthetic +0.006). The working paper reports an earlier run of the same study, and the fidelity page says its sign flips concentrated in origin and legal-history effects. A buyer whose decision hinges on an attribute like that should confirm it with people.

Of the 308 comparisons, 73 fail on direction and rank at once. Two are in the table above: Gracia 2014 (6 of 9 effects in the right direction, rank 0.53) and Wezemael 2013 (3 of 6, rank 0.57). Among those without a stated sample size, Unterberger 2021 got 4 of 10 effects in the right direction and ranked them at -0.30, the reverse of the human order. For each of these, the registry's advice is to use the result only to generate hypotheses to test with people.

On effect size, the registry multiplies synthetic effects by a per-study calibration factor before it compares them (the Hainmueller 2014 page shows one) and says that agreement after scaling does not establish agreement in raw sizes. The fidelity evidence page, last verified 2026-08-15 by its own note, states that no Subconscious result has yet shown synthetic estimates landing inside human confidence intervals experiment by experiment.

What Pew found, and how this evidence relates to it

Pew Research Center had Anthropic's Claude Opus 4.6 answer three 2026 survey waves as digital twins of its American Trends Panel members. Across nearly 300 questions the synthetic estimates differed from the human results by 12 percentage points on average, the gap exceeded 15 points on about 28 percent of questions, and Pew concluded that AI polling does not replace surveying real people.

This page does not dispute that finding. Pew measured the level of opinion on single questions. The evidence above measures something narrower: whether a simulated population orders and signs the effects of attributes in a designed choice experiment the way a human sample did. The claim this page supports is that a simulated choice experiment can screen what moves a decision where a matched study held.

What AAPOR recommends

The AAPOR Task Force report "Responsible AI Integration in Survey Research", released May 2026, calls synthetic responses "the most risky of the core tasks we consider", prefers the term synthetic responses to synthetic samples because the method is an estimate of what specified respondents would say rather than a sampling design, asks researchers to distinguish human from AI-generated data clearly, and notes that evaluation is shifting from single-item averages to whether synthetic responses preserve relationships and subgroup structure.

Against those points: every result on this page is labelled synthetic, every comparison names the human study and gives its sample size where the registry supplies one, the misses are listed with the hits, and the unit of evaluation is a treatment effect rather than a single-item average. The report also warns that models are retrained without notice; each detailed comparison in the registry is dated, and each study page lists the configured models behind its saved runs.

How Aaru's benchmark differs

Aaru published "Population Simulation at the Replication Floor" on 2026-09-24. It compared simulated answer shares with published survey results across 2,993 questions from 186 studies, reporting a mean total variation distance of 5.98 points after a sampling adjustment (7.62 before) and a mean absolute error of 2.78 percentage points after the adjustment (3.53 before). That is a distributional benchmark: it asks how close a simulated answer share lands to a published one. The same publication compares simulations with a credit card purchase panel and reports 92.4 percent of brand pairs ranked in the panel's order, against 77.8 percent for surveys; that is a ranking against observed purchases, not a test of a treatment effect. The evidence on this page is interventional: it asks whether the effect of changing an attribute matches the human effect. These two benchmarks measure different outcomes, so their headline figures do not support a vendor ranking.

What this means for a decision

The paper and the registry compare simulated estimates with human results; that comparison, not the interval on any single simulated effect, is the measure of distance from a human result. For what a synthetic respondent is and the questions to ask a vendor before trusting one, read Synthetic respondents: what they are and when to trust one. To scope a study with a human calibration step, book a decision review.

How this page was made

By: Avi Yashchin, CEO and Co-Founder. Drafted with AI assistance on 2026-11-07 and updated on 2026-11-12 from the public sources linked above. Every number was read at its source on those dates: the fidelity.subconscious.ai evidence page, the Subconscious study registry index and its study pages, Pew's 2026-09-30 report, the AAPOR task force PDF and Aaru's 2026-09-24 publication. No private run data was used.

Update log

• 2026-11-07: first version, from an earlier registry snapshot.
• 2026-11-12: new registry snapshot of 308 detailed comparisons; the study table now lists all 17 comparisons with a stated human sample size; Hainmueller 2014 updated to the saved run of 2026-11-06; Alphonce 2012, Mele 2021 and Bosseaux left the table (Alphonce's page no longer carries a detailed comparison; the other two remain in the download); paper figures now cited from fidelity.subconscious.ai, because the paper's PDF link no longer resolves.

Cite this page

Subconscious.ai (2026). Synthetic respondent validation: how simulated results compare with human studies. Retrieved from https://subconscious.ai/blog/methods-and-validation/synthetic-respondent-validation-against-human-studies. Data: https://subconscious.ai/data/synthetic-respondent-validation-2026-11-12.csv. Underlying paper: Yashchin, A. (2026). Grounding the Map of Causality. Working paper, not peer reviewed, summarized at https://fidelity.subconscious.ai/.