What reviewers should ask about synthetic market research data
On this page
A reviewer needs a matched calibration study and the information needed to judge its scope: population, task, model version, prompt, exclusions, human reference, uncertainty, and possible training overlap. Calibration evidence belongs alongside provenance and data-handling checks when vetting synthetic respondents.
- Check demographic coverage and endpoint-specific agreement; neither substitutes for the other.
- Bisbee and colleagues found disagreements in survey distributions and regression relationships. Those comparisons do not identify price or feature intervention effects.
- A Qualtrics partner article reports one matched attitudinal benchmark; its commercial and task scope must remain attached.
- Silicon samples, Sarstedt and colleagues' term for LLM-generated survey respondents, are on firmer ground for upstream pretesting and pilot work than for confirmatory claims a decision will actually rest on (Psychology & Marketing).
Why representation and model agreement need separate checks
Bisbee and colleagues’ Political Analysis study compares demographic-conditioned political survey responses and regression relationships. Disagreement shows that similar demographic profiles need not reproduce distributions or associations. It does not establish causal price or feature effects. An intervention claim requires an identification design and a matched outcome comparison.
What does the vendor-run attitudinal benchmark establish?
In February 2, 2026 partner content, Qualtrics product scientist Derrick McLean reports a Google Search attitude comparison. Qualtrics Edge Audiences, ChatGPT-5, Gemini, and humans were matched on age, gender, and search-use frequency, leaving 518 cases per source. The vendor reports average Cohen’s D differences of 0.07 for its model and 0.87/0.88 for the general models across eight attitude questions. Exact Gemini revision is not specified. This is commercially interested descriptive evidence for one benchmark, not independent causal intervention validation.
What should you actually ask a synthetic research vendor?
Request evidence matching the actual task and population. A randomized choice comparison should specify identified contrasts, effect magnitudes, uncertainty, and predeclared human agreement criteria. A descriptive calibration should specify distributional errors. Rank correlation is not an error bound on market outcomes. Ask how prospective holdouts or training-overlap checks address contamination.
Two ways to vet a synthetic sample
| Representativeness checklist | Calibration study | |
|---|---|---|
| What it tests | Population coverage, sourcing, demographic composition, and data handling | Agreement on a specified distribution, association, or intervention endpoint |
| What it can miss | A sample can match census figures and still produce a coefficient with the wrong sign | Doesn't catch demographic gaps in the underlying panel used to build the calibration |
| Evidence to request | Frame, matching, exclusions, and underrepresented groups | Independent reference, error metric, uncertainty, and task scope |
| Best for | Screening a vendor's basic legitimacy and data handling | Deciding whether to trust a specific causal claim before you act on it |
Passing a benchmark is evidence for that benchmark. For the new decision, require matched population and task support, an independent holdout where feasible, calibration error and uncertainty, and a threshold for escalation when expected loss or unresolved magnitude makes action premature.
Where is it safe to trust a synthetic respondent today?
It's safest for early-stage, upstream work, screening concepts, drafting a questionnaire, exploring which messages are worth testing further, and riskiest for confirmatory claims a real budget decision depends on. Sarstedt and coauthors, writing in Psychology & Marketing, make this the center of their guidance: silicon samples are defensible for pretesting and pilot studies, and risky when treated as a substitute for a confirmatory study (Wiley). That maps onto the calibration question directly. A vendor with no calibration study might still be useful for a first pass at message testing. The same vendor without a calibration study has no business producing the number that decides which action a team takes with real money behind it.
What a calibration number needs to specify
A calibration number needs a stated population, task and error metric. A simulator interval is conditional on its response process, model and analysis; its coverage needs checking and it does not bound human transport error or future market demand. Standard multinomial logit imposes IIA: changing another option leaves the relative odds between the remaining alternatives fixed. Mixed logit can relax that aggregate restriction through taste heterogeneity, while ICLV does not automatically remove it. These are choice-model specifications. Request the actual estimation and uncertainty procedure separately from the randomized design and identification assumptions, then test substitution on relevant holdouts.
Checking the vendor's track record before you sign
Use the research record for published aggregate cohort results. The working paper's per-study replication data are not public. Request study-specific model and protocol revisions, unsuccessful cases and transport boundaries from the provider before relying on a result for the proposed decision.
The concrete next step: before your next synthetic research engagement, ask the vendor for their calibration study, the human baseline it was checked against, the error number, and whether that number covers the class of decision you're about to make. If they can't produce it, treat the output as a pretest, not a confirmatory result. More on how the underlying replication protocol works is on the methods and validation page. If you want to walk through a specific decision with someone who can show the calibration behind it, reach out.