Can LLM-Generated Open-Ended Survey Responses Match Real Issue Distributions?
On this page
A consumer insights leader deciding whether to trust LLM-generated open-ended answers before fielding qualitative research needs one question answered: do those answers reflect the same aggregate distribution of issues real respondents raised, or only what sounds plausible? A published benchmark against a real survey gives a direct, if partial, answer.
How did researchers test LLMs against a real government survey?
The benchmark drew on ANES, the American National Election Studies, and specifically its 2020 Time Series Study: a survey run by the University of Michigan and Stanford University under National Science Foundation funding, which asked respondents, among other open-ended items, what they saw as the biggest problems facing the country, in their own words rather than from a fixed list. Researchers pulled a test set of 100 real respondents, took their demographic profiles (age, gender, race, education, income, occupation, religious affiliation, and self-placed political alignment), and prompted several large language models to generate an open-ended answer to that same question as if each model were that respondent, answering in 2020.
Allen Downey’s PyMC Labs benchmark, published June 3, 2025, constructed 18 issue categories using LLM-assisted coding, classified real and generated answers with the same procedure, and compared frequencies using MAE, Kendall’s tau, and Spearman correlation. Spot checks supported the coding, but shared automated measurement can introduce errors into both sides of the comparison; independent human coding would test that dependence.
What held up, and what didn't
| Dimension | Result pattern |
|---|---|
| Issue ranking (Kendall's tau, Spearman) | Positive rank correlations, without reported confidence intervals: several models preserved which issues real respondents raised most often relative to each other |
| Issue frequency (MAE, share) | Weaker match: after rescaling to a common average number of problems per response (real respondents averaged 1.7), models still showed higher error in the share of mentions assigned to each issue |
| Temporal accuracy | Most models failed the historical-time instruction. Almost every model overestimated healthcare-access mentions; the source also discusses inflation as a later-salient issue. |
No model exactly reproduced the reference distribution, and the reported ranking measures generally looked stronger than frequency agreement. The source reports no confidence intervals, and the 100-person reference sample adds sampling uncertainty. Temporal drift affected most models; the benchmark does not establish a universal failure across every model or task.
Why the ranking-versus-frequency gap matters for a research decision
If a team is using LLM-generated text to explore likely themes before committing budget to fielded qualitative research, this benchmark suggests treating relative signal (which issues come up more than others) as more trustworthy than any specific rate or count the model reports. Frequency-level output looks precise but was the least reliable part of this result.
For issue salience, validate generated themes and frequencies against a new human sample using a consistent coding scheme. That is a descriptive comparison. A randomized study of which message changes choice asks a different causal question; Subconscious’s experimental workflow should be evaluated against that separate endpoint.
What are the limitations of this benchmark?
The result covers one prompting method, one open-ended question, and one historical dataset. It does not validate other tasks, populations, or future time periods. A workflow requiring historical realism needs its own dated human reference and an explicit check for later information in generated answers.
Where does this fit in a research workflow?
Use this benchmark to inform theme exploration and survey preparation, then retain actual human verbatims for claims about what respondents said. For an experimental business decision, inspect the separate evidence and endpoint on the leaderboard, or discuss the study.
