Setting the Validation Line for Synthetic Panels
On this page
A useful validation standard lets an insights leader approve a specific use of evidence, correct a model, or withhold an unsupported claim. Subconscious makes that standard concrete through controlled choice experiments on simulated buyers and comparisons against human results. The useful output is an acceptance record: what the study supports, where the evidence differs, and which action the responsible owner can accept.
Set the line at the claim and the decision
The published Mintel study shows why a single pass label is too blunt. Subconscious modeled preferences for a 50 ml facial moisturizer using demographics alone, without human preference or choice data as inputs. Comparisons with two human survey waves of 1,000 respondents each reported 0.79–0.88 Jensen–Shannon distribution agreement. That metric describes agreement between preference distributions, not the percentage of individual answers that were correct.
The same case records a useful local exception. Human demand fell sharply between $34 and $48, while the model showed a smoother decline. A leader can retain the reported distribution finding and restrict the upper-price inference. The disagreement gives the team a specific correction to make, rather than requiring every result to be discarded or every model recommendation to be confirmed.
This is the validation line: accept the supported claim at its measured endpoint, then judge the proposed action against the evidence and economics the action needs. Distribution similarity alone does not approve a price, establish sales, or forecast revenue.
Write the acceptance rule before reading the result
Start with the sentence that will reach the decision owner. “The modeled preference distributions agreed with this human reference” requires different evidence from “raising the price will improve net receipts.” A price recommendation needs the exact offers, eligible audience, alternative, response endpoint and applicable costs. Recorded purchases require transaction evidence; a modeled choice remains labeled as modeled.
Next, inspect the available study. Existing human research can be sufficient when its population, alternatives, period and endpoint fit the intended use. Check the actual source and design, including assignment and uncertainty, rather than treating the word human as a quality certificate. A randomized comparison on simulated respondents identifies a response contrast within that modeled study; transfer to human behavior needs relevant supporting evidence.
Keep two rules separate:
- Validation tolerance: the metric, scale and local comparisons that must agree sufficiently for a stated model use. Define meaningful discrepancies in advance. Average agreement can conceal a consequential price band or audience mismatch.
- Action threshold: the result and uncertainty that justify the proposed business change, using actual costs and constraints. For a higher price, the owner may accept some choice loss. The largest choice estimate is not automatically the best commercial option.
Record respondent counts, analysis units and uncertainty appropriate to the design. Repeated answers from one respondent need dependence-aware analysis. A narrow interval describes precision under the stated analysis; it does not establish population coverage or transfer to a new task. See the methodology and supported study configurations.
An acceptance record that preserves the useful result
The following is an illustrative review of the published Mintel evidence, not a report of Mintel's internal approval process or a new study result.
| Record item | Evidence available | Permitted wording or next action |
|---|---|---|
| Preference-distribution claim | Reported 0.79–0.88 distribution agreement against the two human waves | Cite the metric, task, demographics-only inputs and reference populations together. |
| Upper-price recommendation | The published case records a sharper human decline than the model between $34 and $48 | Flag the local mismatch. Do not approve a $48 recommendation from overall distribution agreement. |
| Commercial decision | The public summary does not supply the complete price contrast, uncertainty and actual net-receipts inputs needed for this approval | Inspect the existing underlying evidence and economics. Use an adequate reference to correct the recommendation; specify only the remaining evidence gap if the record is insufficient. |
A completed record names the source versions, applicable alternatives and audience, prior tolerance, action rule, exceptions and accepting owner. The owner can approve a bounded claim while declining a broader recommendation. That is more useful than calling the entire project “validated” or “exploratory.”
Disagreement can support a better action
Agreement with a model is not a prerequisite for using sound human evidence. If a suitable human comparison contradicts the model, the team can correct the model's use and act on the stronger reference. Check that the apparent conflict compares the same endpoint and feasible alternatives before changing the recommendation.
When neither source resolves the decision under the prior rule, record an unresolved outcome. Name the missing contrast, population or measurement. A new study is one possible response; inspecting an existing result, correcting an input or narrowing the claim may answer the question.
Scientific review, business acceptance and permission to release a claim are separate responsibilities. The research lead accepts the design and inference. The business owner accepts the action and actual economic assumptions. Relevant legal, privacy, security or technical owners retain their responsibilities. Human review does not necessarily mean recruiting new respondents, and a choice experiment does not establish sensory performance, usability or legal clearance.
Bring one proposed claim, the available evidence and the accepting owner's rule to a decision review. Subconscious can help turn the research question into a controlled comparison and a recommendation with a clear evidence boundary. See how the work is scoped and the published cases.