Synthetic Research: When to Trust It, and When to Validate With Real People
On this page
A VP of Insights choosing whether to add synthetic research needs to decide which questions can use modeled exploration and which require observed human evidence. An interval from a simulation describes uncertainty under that model; it does not establish that a pricing, launch, or public claim holds in the target human population.
What is synthetic research?
Synthetic research uses AI-generated personas, conditioned on demographic, psychographic, and behavioral data, to simulate how a target population responds to a stimulus: a concept, a message, a poll question. Researchers assemble personas into panels, run a stimulus against the panel, and get back a distribution of responses with natural-language reasoning for each one.
Three terms often get conflated:
- Synthetic respondent: the individual AI agent that answers a single study, conditioned to hold a specific set of beliefs and background.
- Synthetic persona: the reusable profile behind a respondent, covering demographics, psychographic traits, and decision-making frameworks. Saved once, queried across projects.
- Synthetic panel: a group of configured personas assembled for a study. In a hypothetical result, 60 percent accept a feature, 30 percent raise an objection, and 10 percent ask about pricing. Those are shares of generated responses, not estimates of human prevalence.
The premise traces to a 2023 finding: conditioning a language model on the detailed background of a real poll respondent produced opinion distributions that tracked actual human responses in benchmark national polls (Argyle et al., a study on prompting language models with individual respondent profiles to reproduce human sample distributions, published in Political Analysis by Cambridge University Press). That result moved the method from academic benchmarking into product, marketing, and insight teams.
Where the method stops being enough
Simulation is directionally useful and limited in predictable ways:
- Human-population claims need suitable evidence. Simulation can quantify variation under its model, but that uncertainty does not by itself establish a real-population estimate. A public figure may require a suitable survey, census, administrative record, or observed behavioral data, an auditable inference method, and the applicable disclosure or regulatory review.
- It lags on novel behavior. Personas are built on historical patterns. A category with no real-world analog, or a sudden macroeconomic shift, can outpace it.
- It inherits training-data skew. Models trained heavily on English-language, Western text default to generalized assumptions for audiences underrepresented in that data. A community outside the training distribution needs real members validating the finding.
- It doesn't touch the physical world. A simulated persona doesn't pull out a credit card, sit through a shipping delay, or churn after a bad support call. Longitudinal behavioral tracking still needs real behavioral data.
The decision: synthetic alone, recruited alone, or both in sequence
| Situation | Right method |
|---|---|
| Early exploration or instrument drafting, including hard-to-recruit audiences | Simulation may help generate hypotheses; access difficulty or sensitive data does not establish fidelity or privacy compliance |
| A capital-committing decision, external quantitative claim, regulatory submission, or legal evidence | Suitable human evidence and inference for the actual claim and applicable protocol |
| Narrowing many options down to the 1 to 3 that matter before a final call | Synthetic first, recruited second |
Simulation can help refine alternatives and the research instrument before a human study. Retain a baseline and credible competing options so that synthetic filtering does not remove a human winner. Cost and precision depend on the actual design, not just the number of finalists.
Where does Subconscious sit in that sequence?
A discrete-choice experiment is a design for comparing alternatives; mixed logit and ICLV are modeling approaches for analyzing choices. None supplies human-population validity merely by returning an interval. A synthetic comparison needs fidelity checks; a recruited study needs a defensible sample, endpoint, and analysis before its estimate supports an external claim.
Carry the decision question from the simulated comparison into an aligned human study. Match alternatives and outcome definitions, document fielding changes, and retain disagreements or an inconclusive result. Agree who will deliver the human study rather than assuming it uses an identical instrument.
For consequential claims, agree the evidential and legal requirements in advance. Human data, suitable inference, and an interval may be necessary, but an interval alone does not make a result ready for audit or a regulatory submission.
Running the sequence without wasting the human study
- Define the decision, not just the topic. Name the population, the alternatives being compared, and the outcome that decides the action, not just "test the messaging."
- Run the simulated pass first. Test the full set of variants, including the ones expected to lose, so the elimination is evidence-based rather than assumed.
- Choose alternatives for confirmation. Use the simulated read to propose finalists, retaining a baseline and plausible challengers when screening error could matter.
- Size the human study for the decision. Testing fewer alternatives may reduce fieldwork, but choose the sample from the effect worth detecting, precision, repeated-task design, and segment needs. Do not shrink it solely because the shortlist is shorter.
- Run the human study. Use a suitable sample and design to estimate the tested human response, reporting uncertainty and the difference from the simulated result.
- Match the claim to the evidence. Report the population, endpoint, uncertainty, fidelity limits, and outstanding review. A confidence interval alone does not authorize a pricing, public, or regulatory claim.
Skipping step 5 for a capital-committing decision is the mistake this framework exists to prevent: treating an unvalidated directional read as a statistically defended one.
Review Subconscious's published replication results and leaderboard, read how the causal experiments are built, or book time to scope a study before committing a budget line to either method.