Why Synthetic Consumer Ratings Need Better Elicitation
Synthetic consumers are easy to generate and hard to trust. Ask a language model for a score on a 1-5 purchase-intent scale and it may overuse the middle, avoid extreme answers, and produce a distribution unlike a human panel. The problem isn't just which model answers the question; the method used to elicit and score the answer can change the result.
A 2025 preprint tested a method called Semantic Similarity Rating, or SSR. Instead of demanding a number from the model, SSR lets the model answer in natural language. It then compares that answer with reference statements for each point on the rating scale and converts the similarities into a probability distribution.
The distinction matters for any team using simulated buyers to test a product, price, or message: a familiar survey format does not guarantee familiar response behavior.
Why direct numerical ratings break down
A prompt that asks a language model for a single rating forces it into a narrow response format, without checking whether the resulting distribution behaves like human survey data.
Direct ratings can cluster around 3 on a 1-5 scale and suppress disagreement, making concepts look more similar than they are. A team may get a clean table of scores while losing the variation needed to rank ideas or spot a weak concept.
SSR separates two tasks:
- The model explains its purchase intent in ordinary language.
- A semantic mapping step compares that answer with reference anchors and assigns probabilities across the rating scale.
What the evaluation found
The evaluation drew on 57 real consumer surveys (9,300 human responses) fielded by a large consumer products company; method-level comparisons are made at the survey level. Its reported results included:
- Product rankings that matched human survey rankings 90% of the time
- Response distributions that lined up with actual survey results by more than 85%
- Wider, more discriminating response patterns than direct numerical prompting
The evaluation also compared GPT-4o with Gemini 2.0 Flash and tested whether age, income, and product category changed the synthetic responses in plausible ways. The method did not require task-specific training data or fine-tuning.
Those results support a narrow conclusion: text elicitation followed by semantic mapping reproduces these aggregate survey patterns better than asking a language model for a rating directly. They do not establish person-level fidelity, universal validity across categories, or a replacement for human research. See Subconscious's research for how elicitation and mapping choices are validated before a study result is used to support a decision.
What a buyer should ask before using the result
An aggregate match can still hide errors that matter to a decision: before using a synthetic panel, inspect the validation at the level where the business choice will be made.
Does the method preserve distributions?
A matching average can conceal variance collapse, so compare the full distribution, not just the mean. Look for missing extremes, excessive neutral responses, and concepts that the synthetic sample fails to distinguish.
Does it preserve rankings?
Early product screening often depends on ordering concepts rather than predicting an exact score. Ranking agreement should be reported separately from distributional similarity because the two measures answer different questions.
Does it work by segment?
Aggregate prediction is usually easier than individual or subgroup simulation. A method can match the total sample while flattening differences by age, income, culture, or category experience; segment claims need their own evaluation.
Does the validation match the intended use?
Purchase intent is not the same as satisfaction, trust, relevance, or actual purchase behavior. A method validated on one response type should not be assumed to transfer to another: new questions, categories, scales, and populations need fresh checks.
A practical research sequence
Synthetic evidence is most useful when it narrows a decision before a more costly commitment. A disciplined sequence is:
- Define the exact product, pricing, messaging, or launch choice.
- Generate natural-language responses from a specified target population.
- Map those responses into a measure with stated reference anchors.
- Compare distributions, rankings, and segment behavior against a human baseline.
- Use the synthetic result to screen scenarios, not to claim certainty.
- Reserve human research for validation where the decision risk warrants it.
This workflow treats calibration and validation as part of the product, not as a footnote. The full preprint provides the mathematical details and experiments. Subconscious's how we work page describes how a synthetic study like this one can move to real-human validation without changing the underlying question.
The useful lesson is simple: better prompts alone do not make synthetic research reliable. The elicitation method, mapping procedure, and human baseline determine whether the output can support a decision.