Simulated vs Recruited Research: Choose the Evidence Your Decision Needs
On this page
Choose the method by the consequence of being wrong. Use a simulated experiment to compare actions and narrow a directional question. Use real-human research when a consequential decision needs human validation, documented sampling, or observed behavior. When the stakes justify both, carry the same causal question from simulation into a human study.
Match the method to the claim
The useful distinction is the claim each method can support, not simulated versus real.
| Research path | Best matched decision | What it can establish | What it cannot establish alone |
|---|---|---|---|
| Simulated experiment | Which concept, message, audience, or action deserves further investment | Directional differences between controlled alternatives | How real people will behave in market |
| Recruited human study | Whether a consequential launch, price, or public claim holds with real participants | Human responses with a documented sample | Revealed behavior unless the study observes a real action |
| Simulation followed by human validation | Which action to take when the decision is both exploratory and consequential | A narrowed causal question followed by a human check | Automatic proof of market performance |
A recruited survey records stated answers. A click, purchase, or abandonment endpoint requires a study that observes that action. A study of LLM alignment compares models' reported general principles with decisions in contextualized prompts. It documents an LLM consistency problem, rather than observing a human survey-to-purchase gap.
When should you use a simulated experiment?
A simulated experiment compares controlled alternatives before committing capital: concept rounds, message variants, audience exploration, competitive framing, pre-research scoping.
A randomized comparison holds the defined task steady, varies an action, and estimates the resulting difference in generated responses. That is stronger evidence for the study's contrast than an unstructured opinion, but it still needs a transfer check before a market claim.
Hard-to-reach buyer roles can still be represented in an early experiment, but representation is not recruitment: a modeled senior buyer helps form and test a hypothesis, not prove one participated.
Does recruitment guarantee clean data?
A method's failure modes belong published beside its results. Real participants matter, but the recruitment label does not guarantee clean evidence. Nonprobability surveys face documented risks from fraudulent respondents, bots, inattentive answers, and professional survey-taking. NORC's literature review summarizes these quality threats and the controls researchers use to detect them.
Judge a recruited study by its sample, screening, quality controls, instrument, and connection to the decision. Use it when:
- A single launch, pricing, or positioning call is expensive to reverse.
- The market context is novel or has changed enough that prior patterns may not hold.
- The result will become a public statistic and needs documented human sampling.
- The study can connect stated reasons to analytics, sales data, or an observed action.
A recruited study is not ground truth when its sample or instrument is weak.
Carry one causal question through both stages
The strongest sequence does not run two unrelated studies. A simulated experiment sharpens the alternatives, outcome, and target population, then hands that same causal question to a human validation study.
For a Subconscious study, specify the target population, alternatives, outcome, and analysis before running the synthetic comparison. Carry those elements into a separately scoped human study and record any instrument changes.
The causal-fidelity working paper evaluates agreement in estimated choice-parameter rankings across published study replications. Review design filters and study-level failures as well as aggregates. That endpoint does not remove the need for validation of the buyer's decision.
A modeled audience is not a standing pool of people who consented to participate. Request the study's population definition, data provenance, geographic coverage, and generator configuration before assessing relevance.
What should you do before choosing a research method?
Before choosing a method, write down the action, the outcome, and the evidence required to release capital.
- For a reversible directional choice, run the controlled simulated comparison and state its boundary.
- For a hard-to-reverse decision, require real-human validation of the same causal question.
- For a behavioral claim, observe behavior rather than treating stated preference as a proxy.
- For a public number, require documented sampling and methods.
- For either path, define failure conditions before seeing the result.
The leaderboard is useful when replication evidence and failure modes matter to the procurement decision. If the decision is already defined, bring the action, alternatives, and required evidence to a working session.