What a Weak Replication Correlation Would Tell a Product Team
A weak correlation between simulated and human preference estimates calls for inspecting the matched estimates, uncertainty and decision risk before choosing a product. A summary coefficient can show imperfect agreement; it cannot identify the attributes that failed or establish sales demand on its own.
Which tomato attributes could a buyer test?
Skreli and colleagues' 2017 tomato-choice study examines production type, production system, origin and price in Albania. Its authors limit the sample to urban consumers and identify hypothetical willingness-to-pay bias. Those boundaries matter for a food-brand leader considering an organic product or a price premium in a different market.
| Attribute | Comparison to recover before replication |
|---|---|
| Production type | Organic versus conventional wording and certification context |
| Production system | Open-field versus greenhouse descriptions |
| Origin | The original geographic levels and reference category |
| Price | Currency, tested levels and coding |
A human study provides evidence about its own task. A later simulation needs a separate record linking the human and simulated estimates, model configuration, run date and analysis. The public Subconscious working paper reports aggregate parameter-rank results; it does not provide a tomato-specific run, so this page reports no coefficient or p-value.
What does a weak comparison actually tell a team?
First inspect what was correlated. Attribute importance, individual part-worth coefficients and treatment effects answer different questions. Confirm matching reference levels, signs and scales before calculating agreement. Report ties, missing estimates and the number of compared parameters.
Next inspect the misses. An aggregate correlation does not show whether the simulation overweights organic certification, reverses an origin preference or misestimates the price coefficient. A matched estimate table and appropriate uncertainty analysis let the buyer judge whether a disagreement changes the proposed decision.
Statistical significance is a separate question from commercial usefulness. A small number of compared parameters can produce imprecise inference. A non-significant result does not establish that the estimates agree or that they are unrelated. Report the calculation and uncertainty instead of treating a threshold as a validation certificate.
How should the result change the next study?
Suppose a tomato brand is considering an organic premium. A disagreement over the price coefficient makes a price recommendation particularly sensitive; agreement on attribute order alone does not resolve that uncertainty. Test the relevant prices with the intended customers and calibrate stated willingness to pay against evidence appropriate to the commercial decision.
Use simulation to explore alternatives where its limits fit the task. Require human or market evidence when the cost of a wrong launch or reformulation is high. Possible exposure to the published study also needs to be reported before a historical match is treated as evidence for an unseen task.
Review the public methodology for the aggregate benchmark's scope. Book a decision review to define the product alternatives, target customers and evidence needed before committing a launch budget.