How to Validate a Product Idea Before You Commit Engineering Budget
Before committing engineering budget, identify the uncertainty that could make the product investment fail: problem relevance, feasible solution, understandable positioning, or actual paid demand. Choose evidence that addresses that uncertainty. An AI conversation, a structured concept choice, a customer interview, a usability test, and a live offer test answer different questions.
Why is "will people buy this" really four separate questions?
Use four questions to organize the evidence. They are a proposed checklist rather than a measured ranking of why products fail.
| Layer | Question | Evidence to seek |
|---|---|---|
| Problem | Is this pain frequent and severe in the intended population? | Verify with relevant customer records, interviews or a descriptive study |
| Solution | Can the proposed approach solve the task better than current practice? | Use a feasible prototype, human task observation and the relevant comparison |
| Positioning | Do eligible buyers understand the offer and distinguish it from alternatives? | Pretest comprehension and compare defined statements on a named response |
| Demand | Will buyers pay under actual offer conditions? | Check costs and feasible terms, then validate with relevant purchasing evidence |
The appropriate order depends on the unresolved risk and dependencies. An observed purchase answers a different question from a generated or hypothetical preference.
Define the segment before testing anything
Define the buyer segment whose reaction matters. A useful segment design mixes distinct viewpoints rather than one uniform "target customer":
- Target-buyer groups. Include relevant company sizes, decision roles and product-use contexts; decide coverage from the actual population.
- Adjacent buyers. Include them when expansion is a real decision, and report their evidence separately from the primary target.
- Non-adopters or prior abandoners. Investigate their objections and constraints alongside favorable responses.
- 1 competitor's customer. Someone already solving the problem with an alternative, who can point to what that alternative does well and where it falls short.
These are candidate viewpoints, not minimum respondent counts or evidence of representativeness. Define eligibility, coverage and the sample or generation method separately.
Why use a controlled experiment instead of an open-ended AI chat to compare concepts?
For an intervention question, specify baseline and candidate versions, randomly assign eligible units, and measure the named response. Holding a segment constant or changing one feature without controlled assignment does not alone establish attribution. More than one attribute can be varied in an identified design when the study needs it (ISPOR experimental-design guidance).
For a Subconscious comparison, agree on the generated-choice task, alternatives, population, assignment, estimator, uncertainty and independent validation. The response contrast remains within the configured task. Map the four layers to their own evidence:
- Problem: describe pain and current coping behavior from relevant sources; a difference between segments is not itself an intervention effect.
- Solution: use human usability or task evidence for operational performance; generated concept preference can supply a screening hypothesis.
- Positioning: check comprehension, then compare alternatives with a defined response and assignment when estimating an effect.
- Demand: treat generated or hypothetical offer selection as a limited response; validate actual paid behavior and viable economics for the investment decision.
What is a simulated comparison good for, and where does it stop working?
A generated screen can surface rejection hypotheses, but poor grounding can falsely reject a concept real customers would adopt. Retain borderline alternatives and the status quo. Before abandoning an expensive or strategically important idea, use an independent human check suited to the question. More draws, a rank, or a confidence interval does not establish demand.
Define validation before reading the screen: intended endpoint, relevant population, acceptance criteria, and action if evidence fails or remains inconclusive. Align generated and human task elements where appropriate; actual usage or purchasing may need a deliberately different behavioral measure.
Before greenlighting build work
Before greenlighting engineering, name the decision owner, unresolved risk, prerequisite evidence, feasible implementation cost, and stop or advance criterion. Use discovery, prototype tasks, controlled choices or live behavior according to that risk. Bring the current alternative and candidate concept to a study review, and review aggregate method evidence separately from validation of the new investment.
