Real Panel or Causal Experiment First: Deciding Before You Field a Survey
On this page
A consumer insights or growth lead should choose respondent source and study design separately. A real-consumer panel can support a controlled experiment from the start. A synthetic study can also compare alternatives, with human transfer remaining an evidence question.
What a consumer panel answers
Pollfish's concept-testing page describes monadic and sequential tests, A/B comparisons, and conjoint resources using real respondents. Inspect assignment, order effects, questionnaire, estimator, and outcome for the proposed study rather than assume a fielded survey is only descriptive.
The strength of a real panel is also its constraint. Once a survey is written and fielded, changing the question means re-fielding it. Each round trip carries real recruitment cost and real calendar time. That is fine when the question is already scoped. It is expensive when the question is still being worked out.
What does it cost to guess wrong before you field a survey?
There are two ways to lose money on this decision. Field a full panel survey on a poorly scoped question, and the budget produced an answer to the wrong question. Or skip human validation on a decision that needs verified respondent behavior, and the business ships an unconfirmed claim. Neither failure is about the panel vendor's quality. Both are about sequencing: what gets tested before a team commits real recruitment budget.
How do you narrow the question before you recruit anyone?
Subconscious runs randomized choice studies with generated responses to estimate effects of specified message, concept, or feature attributes within the simulation. Those effects can inform trade-offs; they do not alone identify psychological mechanisms or prove real-customer effects.
A separately scoped human study can check the same alternatives and causal question. Align the target population, instrument, outcome, and analysis, and document differences. Decide which options to field using practical thresholds and uncertainty, rather than assume every discarded synthetic option is ineffective.
Comparing the two starting points
| Human-panel experiment (e.g. Pollfish) | Synthetic choice experiment (Subconscious) | |
|---|---|---|
| Respondent source | Real consumers, recruited per study | Generated responses from a simulated market; any human study is scoped separately |
| What it answers | The specified contrast on responses from recruited people, if the design identifies it | The specified contrast on generated choices inside the simulation |
| Cost of a revised question | A new recruitment and fielding round | Re-running the comparison in the same simulation |
| Where it fits | The question requires direct human responses or an audience poorly represented in simulation | The question fits the modeled population and validated choice task |
| What to check | Recruitment, assignment, instrument and response quality | Generator fidelity, coverage, dependence and human transfer |
Where the two combine
A synthetic study can help screen candidates when its evidence fits the task. Specify a useful-effect threshold and preserve options where uncertainty or population gaps create a false-negative risk. A human-panel experiment can also come first. No fixed number of survivors is guaranteed.
Where does this approach break down?
A synthetic choice study supplies generated responses, not recruited respondents. Scope any human-validation study separately, and obtain its recruitment provider, eligibility, fielding, and quality-control details. Use the research evidence to assess fit and failure modes before deciding on simulation.
Making the call
Choose direct human fielding when the endpoint or audience requires it. Use exploratory research when the question is unclear. Use a controlled comparison once alternatives and outcome are defined, and choose its respondent source from the evidence and risk.
Check current leaderboard replication results before weighting a simulated comparison, or book time to talk through where a given question falls.