AI Research vs Real Users: A PM Decision Framework
On this page
A product manager chooses a research method for a specific decision: a feature bet, a price change, a message, or a positioning call. The method should match the evidence that decision requires.
Use a simulated behavioral experiment when the question is about causal choices across defined options and the result will be treated as directional evidence. Use real participants when the decision depends on actual product use, spending, lived context, or proof that a person understood something. Sequence both when a simulated finding will drive a consequential action and needs a study-specific check against human responses.
Choosing poorly creates two risks: shipping on a simulated signal that was never checked against real behavior, or committing to real-user fieldwork for a reversible, low-stakes call that did not require it. Recruiting and scheduling time varies by audience and method, so get a quote for your own brief before you commit to a date.
The core tradeoff
A controlled behavioral experiment on a simulated population suits breadth: testing many concepts, framings, or messages before committing budget to one. A simulated respondent does not carry a real budget, a real procurement process, or years of lived context behind a workflow.
Real-user research, including interviews, usability sessions, and surveys, is grounded in responses and behavior nobody had to model. It can capture physical interaction with a product and the weight of a real financial or emotional decision.
When simulated testing is the right call
Early-stage concept exploration. A team narrowing five feature ideas to two can use a controlled behavioral experiment to compare all five under one defined causal question. The result can reduce the option set, but it does not prove how people will use the final feature.
Reversible decisions with bounded consequences. A feature flag, message, or concept suits simulated testing when the team can reverse the action and will treat the result as directional, not conclusive.
Sprint-speed calls. A prioritization decision due before a fixed deadline, with no time to recruit and schedule participants, gets a directional read from a simulated session it would not otherwise have in time.
Copy and messaging testing. Which value proposition changes preference, or which feature name is clearer, can be framed as explicit choices for a simulated population. The output is comparative evidence, not proof of market performance.
Pre-validation before a human study. A simulated test can identify which concepts deserve a real-human check. The human study still carries the burden of validating the selected causal question.
Competitive positioning. Modeling how buyers choose between two product descriptions across several framings can reveal which causal contrasts deserve further validation.
When you need real users
High-stakes pricing decisions. Willingness to pay depends on a real budget and a real pain of paying. A simulated population can model price sensitivity directionally, but a revenue-affecting price call needs a check against people who are actually spending money, and a defensible quantitative number needs a properly sized sample, not a handful of respondents (NN/g: Quantitative User Research: Study Guide).
Usability testing with complex interactions. Observing someone click through a multi-step workflow and hit real edge cases requires a real person in front of a real prototype. Simulated testing can evaluate a described flow, not the physical and cognitive experience of using the software. The reverse holds for open-ended usability problems: a small qualitative sample, as few as five participants, can already surface most of a workflow's usability issues, even though that same small sample would not support a quantitative claim (NN/g: Why 5 Participants Are Okay in a Qualitative Study, but Not in a Quantitative One).
Emotional and behavioral nuance. Whether someone trusts a feature with sensitive data, or how they feel about a change to a workflow they have used for years, involves personal context a model approximates rather than replicates.
Regulatory or compliance validation. A modeled answer does not establish that actual users understood a consent flow or disclosure. Identify the applicable jurisdiction, filing or claim requirement, then agree the evidence and validation protocol with the responsible reviewer.
Checking a simulated finding before it drives a decision. Periodically running the same question through both methods and comparing results is how a team learns which questions its simulated signal answers reliably.
| Decision dimension | Simulated behavioral experiment | Real-user research |
|---|---|---|
| Who or what responds | A simulated population under controlled conditions | Recruited people responding from lived context |
| Best use | Comparing concepts, messages, choices, or framings | Observing use, probing context, or confirming a consequential decision |
| Financial stakes | Directional because no real spend or procurement occurs | Can examine decisions made with an actual budget |
| Interaction complexity | Evaluates defined choices and described experiences | Can observe live, multi-step product interaction |
| Compliance evidence | Does not prove that a person understood a disclosure | Can produce documented evidence from real participants |
How do you sequence both with a matched-question replication?
Subconscious runs a randomized behavioral experiment on a simulated population. A human check of the same causal question is then a matched-question replication: people see the same attributes and levels, with the task wording adapted for a human survey, and a respondent sample that matches the simulated audience. Each adaptation is recorded, because a different instrument or sample can change the result. The human step is a study-specific check, not a general endorsement of every simulated result. Who recruits and fields it, and the timeline, are scoped per engagement in a decision review.
A funnel that uses both stages this way, as a hypothetical example: run a simulated experiment across 10 concepts to narrow to 3. Then run 5 real-user interviews on those 3 as qualitative discovery, to hear why people react as they do and to find problems the simulation could not model. Interviews do not quantify an effect. If the finalist choice needs behavioral effect evidence, design a separate quantitative check, such as a randomized survey with enough respondents or a bounded live test, and commit a full usability study to the winner after that. Treat the simulated shortlist as provisional. Reserve human checks for uncertain or strategically important discarded concepts and overlooked segments, so the process can detect simulation false negatives. Validate the screening sensitivity before treating rejection by the simulation as grounds to drop a concept.
The public fidelity evidence is the July 2026 causal fidelity working paper, which is not peer reviewed. It reports a mean Spearman rank correlation of 0.73 on estimated choice parameters across the 43 studies that pass its design filters, and 0.55 across roughly 300 replications. The paper also reports misses in its limitations section. These figures measure how closely simulated estimates rank the parameters that published human studies found. They do not measure accuracy for a new market, effect-size agreement, or realized sales, and they do not cover every possible research question.
Four questions before choosing a method
- How reversible is this decision? Easily reversible, such as copy or a feature flag, favors simulated testing. Hard to reverse, such as pricing, core architecture, or brand positioning, needs a real-user check.
- What does the deadline allow without lowering the evidence standard? Time available before deciding is a planning constraint, not evidence that one method is valid. If the decision requires observed human behavior and that evidence is unavailable, narrow, postpone, or mark the decision unresolved.
- Does the decision touch money or emotion? If people are paying for something, or the change reaches a workflow they rely on personally, lean toward real users.
- Are you exploring or confirming? Exploring a wide option set favors simulated testing. Confirming a final call before it ships favors real users.
What can't the validation step settle?
A matched-question replication checks one causal question. It is not open-ended usability testing of a live prototype, and it does not walk a person through a multi-step UI. A simulated population is not a recruitable panel for interviews. These limits are stated here so a buyer knows what to reconfirm before acting. Uncertainty intervals, segment breakdowns, and decision-memo outputs are specific to each study and need confirming per engagement, not treated as a standing guarantee.
Make confidence question-specific
One practical example is to compare simulated findings with existing human research on the same question for a few sprints, recording where the two agree and where they diverge. Pick the length of the comparison by how many decisions you expect to make in that period. Trust in the simulated signal should be earned for the specific question, not assumed globally.
Put the evidence burden on the decision
Do not use simulated testing merely because a deadline is close. Use it when the causal question, action, and evidence standard fit the method. When actual spending, physical interaction, compliance, or lived emotional context determines the answer, collect human evidence or treat the decision as unresolved.
See how a study is designed, or book a decision review and bring a live decision to identify which evidence it requires.