Which Validation Check Should a Product Manager Run Before Engineering Starts?
Pendo's 2024 product benchmark report reports that 6.4% of features drove 80% of click volume in its benchmark. This measures concentration of usage; it does not mean the remaining features received no engagement. Before engineering starts, a product manager still has to choose the right check: explore a concept, test usability with people, or compare defined variants in an experiment.
Three checks, three different questions
A simulated read on a concept, prototype, or feature description surfaces early objections before a final design exists. It tells a PM what stands out or confuses, not what a buyer will actually choose when the alternative is doing nothing.
An unrandomized usability session with recruited users shows whether participants complete a task and where they get stuck. It does not by itself isolate a variant effect. A randomized usability comparison can estimate effects on its recorded task outcomes, subject to the assignment and study design.
A controlled causal experiment puts two or more variants in front of the same target audience under the same conditions and estimates which one moves the outcome. It answers the question the other two leave open: which specific change caused the difference.
Matching the check to the decision
| Decision in front of the PM | Best-suited check | What it cannot tell you |
|---|---|---|
| Does this concept land, or confuse people, before a prototype exists? | Simulated read on a concept or PRD excerpt | Which variant a buyer would actually pick |
| Can a real user complete the flow without help? | Unrandomized usability session with recruited participants | A causal effect of a variant without a suitable comparison design |
| Which of two pricing tiers, feature variants, or messages changes the specified response? | Randomized comparison across variants | Purchases or other outcomes the study does not measure |
Concept exploration and unrandomized usability sessions help narrow questions and options. A designed comparison can test a specified contrast before engineering commits to one variant; confirmatory status depends on the protocol and analysis.
When does a controlled experiment change the launch decision?
New Age Floral used five iterative pricing experiments, with consumer behavior modeled across 125 participants. Its case study reports a $60 price to maintain modeled market share and a drop-off above $100. The record concerns an existing product and does not establish realized sales at either price. Finta changed its annual price from $348 to $950 and reintroduced a $99 monthly plan. Its CEO reported five new customers on the monthly plan; that observation does not isolate the effect of the research from other changes.
Subconscious runs randomized experiments on a simulated market. Its public validation is a parameter-rank comparison with human studies (a mean Spearman rank correlation on estimated choice parameters of 0.73 across 43 design-filtered studies, from the July 2026 working paper, not peer reviewed). That comparison does not validate an individual launch decision. The experiment addresses "which action" questions: which price, which feature variant, which message changes what a modeled buyer chooses. It fits once exploratory or qualitative work has narrowed the field to a short list of variants, when the remaining question is which one to ship.
What does a causal read not do?
A simulated experiment does not replace watching a real person try to use the product. Confirm human recruitment and delivery scope separately for the proposed study. When a launch decision is high-stakes, a team can run the simulated comparison first, then scope a recruited human study separately before committing, using the same variants (method evidence).
Before the ticket goes to engineering
A concept that has not been shaped yet needs an exploratory read. A flow that might confuse people needs real users watching it. A feature variant, price, or message that will only ship once needs a controlled comparison. Talk to Subconscious about setting up that comparison before the engineering ticket gets written.
