Can LLM-Generated Open-Ended Survey Responses Stand In for Real Verbatims?
A consumer insights leader deciding whether to trust LLM-generated open-ended answers before fielding qualitative research needs one question answered: do those answers reflect what real respondents actually said, or only what sounds plausible? A published benchmark against a real survey gives a direct, if partial, answer.
The test: matching LLMs against a real government survey
The benchmark drew on ANES, the American National Election Studies, and specifically its 2020 Time Series Study: a government-run survey that put one open-ended question to respondents, asking what they saw as the biggest problems facing the country, in their own words rather than from a fixed list. Researchers pulled a test set of 100 real respondents, took their demographic profiles (age, gender, race, education, income, occupation, religious affiliation, and self-placed political alignment), and prompted several large language models to generate an open-ended answer to that same question as if each model were that respondent, answering in 2020.
Both the real and generated responses were then classified into a shared list of 18 recurring issues (healthcare, economy, immigration, and so on), and the resulting issue-frequency distributions were compared using three statistics: mean absolute error (MAE) between issue frequencies, Kendall's tau, and Spearman's rank correlation (the latter two measuring how well the relative ordering of issue importance matched, independent of exact frequency) (Scaling Open-Ended Survey Responses Using LLM-Paired Comparisons, Public Opinion Quarterly, academic.oup.com/poq/article/90/3/630/8551356).
What held up, and what didn't
| Dimension | Result pattern |
|---|---|
| Issue ranking (Kendall's tau, Spearman) | Moderate-to-strong agreement: several models preserved which issues real respondents raised most often relative to each other |
| Issue frequency (MAE, absolute rate) | Weaker match: models tended to name more problems per response than real respondents, who averaged 1.7 problems each; researchers rescaled results to a common average before comparing |
| Temporal accuracy | Consistent failure: nearly every model overestimated concern about issues that became salient *after* 2020 (healthcare access, inflation), even though the prompt explicitly instructed the model to reflect "the conditions and topics of discussion in 2020" |
The pattern is directional: no model reproduced the real issue distribution, relative ranking survived better than absolute frequency, and every tested model showed the same time-period drift. That drift documents an LLM defaulting to its training-data-shaped view of an issue rather than the view a real respondent actually held, sometimes called temporal or demographic flattening.
Why the ranking-versus-frequency gap matters for a research decision
If a team is using LLM-generated text to explore likely themes before committing budget to fielded qualitative research, this benchmark suggests treating relative signal (which issues come up more than others) as more trustworthy than any specific rate or count the model reports. Frequency-level output looks precise but was the least reliable part of this result.
That gap is why a single simulated pass isn't the last step for a consequential decision. LLM roleplay output isn't production-ready on its own; Subconscious runs the same behavioral question as a controlled study, then validates it with real participants, without changing the underlying causal question between the two steps (how we work).
Limitations of this benchmark
This result comes from one demographic-conditioned prompting method, tested against one open-ended question, on one historical dataset. It speaks to issue-salience recall under a time-travel instruction, not to how LLM-generated text performs on other open-ended tasks, other demographic compositions, or unprompted (non-time-shifted) questions. The temporal-drift failure is worth planning around: any workflow that asks a model to represent a past or hypothetical state should expect the model to lean on more recent or more salient information unless that's specifically tested for and corrected.
Where this fits in a research workflow
Use LLM-generated open-ended text the way this study's authors frame it: a way to get an early read on likely themes, or to pre-test how a question might land with different groups, before spending on fielded qualitative work. Teams that want that baseline check built into the workflow can see how Subconscious structures controlled experiments and human validation on the research and leaderboard pages, or book time to discuss a specific question.
