Can an LLM Stand In for a Human Survey Respondent? What One Benchmark Found
A large language model prompted to answer as a person with a given age, income, and education can predict that person's political party about as well as a supervised model trained for the task, though individual-level party identification is only modestly predictable from demographics to begin with. Ask the same model to predict how many hours that person watches television, and its accuracy is inconsistent. That gap is what a research team should check before trusting an LLM persona on a real go-to-market decision.
What was actually tested
An independent benchmark study gave several large language models a real respondent's traits (age, sex, race, education, income, occupation, religion) and prompted each to answer in that person's place (arXiv, "Large Language Models as Virtual Survey Respondents"). The source data came from the General Social Survey (GSS), a public dataset of adult US residents' attitudes and behaviors, and the models were scored on two questions:
- Party identification, self-reported on a seven-point spectrum running from Strong Democrat at one end to Strong Republican at the other.
- Hours of daily TV viewing, sorted into five buckets ranging from one hour or fewer up to six or more.
Each model's error was scored as the mean absolute distance between its answer and the real respondent's, averaged over a random sample of 100 held-out respondents, and compared against the baselines below. That comparison is intentionally asymmetric: the random forest is supervised on in-domain data, while the language models rely only on pretraining, so it contextualizes the LLM output rather than claiming the two methods are equivalent. GSS microdata and its published cross-tabs are public, so the language models' pretraining may itself include the demographic-attitude relationships being tested, a possible source of overlap the study does not rule out.
| Method | How it works | Training data |
|---|---|---|
| Naive baseline | Always predicts the median response | None |
| Random forest | Supervised model trained on real respondents | 3,000 respondents (party ID); 2,000 (TV hours) |
| LLM persona | Model prompted with demographic traits, no task-specific training | None (pretraining only) |
Where the model held up
On political party identification, most large models tested, including GPT-4o, Claude 3.7 Sonnet, and Gemini 2.0 Flash, landed within the range between the naive baseline and the random forest, several approaching its accuracy, though the study reports no standard errors, so these differences cannot be called equivalence. Claude 3 Opus came close to the random forest despite no task-specific training.
A control run removed all demographic information from the prompt and asked the same question generically. Performance collapsed below the naive baseline across every model tested. That comparison is suggestive but not conclusive: under mean absolute error a naive median predictor is hard to beat, so falling below it in the no-demographics arm is close to expected regardless of whether the model uses demographic signal, and the ablation changed the prompt's content and framing along with the demographic fields.
Where it broke down
Daily TV-viewing hours is a weaker-signal task: it correlates less tightly with standard demographic variables, and the random forest itself had less training data available. Model performance here was substantially more variable: runs typically fell between the naive baseline and the random forest, but plenty landed outside that band on either side, and no model consistently beat the rest in the reported results, a contrast with the more stable rankings for party identification.
That instability is the practical risk: a model can perform like a trained classifier on one question and a coin flip on the next, and the study did not measure model confidence or calibration, so there is no signal available to distinguish which case is which.
Why this matters before a GTM decision ships
This failure mode is exactly what a human-baseline check is designed to catch before a team acts on it. This benchmark scores individual-level point prediction of a marginal attitude, not recovery of the conditional response distribution or the treatment effects a choice experiment needs; an insights or research lead checking whether an LLM persona's answers are usable for a specific question is at most checking how demographically predictable that question is, and an ungrounded model roleplay does not surface that on its own.
Subconscious can test or validate studies with real human participants, so a team can move from a simulated experiment to a real-human baseline check without changing the underlying causal question. A result that looks strong in simulation can then be confirmed, or caught, before it becomes a pricing, messaging, or positioning decision.
What this study does not establish
None of the specific accuracy figures, model rankings, or benchmark results in this study are Subconscious findings, and Subconscious did not run or endorse this experiment. A separate industry review of synthetic survey samples has also documented significant limitations in AI-generated responses relative to real respondent panels (Verian Group, "Synthetic Sample in Social Research"). The lesson to take from both is procedural, not a number to reuse: test the demographic predictability of the question before trusting a model's answer to it, and check a weak-signal result against a real human baseline before it drives a decision.
Next step
Compare methods for testing a causal marketing question, including where a real-human baseline changes the answer, at Subconscious's research and leaderboard pages, or see how the validation workflow runs end to end.
