Method validation
Methodology validation
Mean Spearman rank correlation on estimated choice parameters across 43 design-filtered studies: 0.73.
Decision
How closely do simulated choices reproduce the parameter rankings in published human studies? The comparison needs a named measurement task, a scoring method, and documented limits.
Interpretation
The July 2026 working paper reports mean Spearman rank correlation on estimated choice parameters of 0.55 across roughly 300 replicated studies and 0.73 across the 43 passing design filters. These are aggregate parameter-rank results. They do not give a buyer's study an accuracy percentage or establish agreement on the size of a real-world treatment effect.
Comparison
The comparison behind the decision.
| Comparison | What the source records |
|---|---|
| Human reference | Estimated choice parameters reported in the original published human studies. |
| Simulated comparison | Estimated parameters from reconstructed choice tasks on simulated respondents, scored by their rank ordering. |
| Broad replication cohort | Mean Spearman rank correlation of 0.55 across roughly 300 replications. |
| Design-filtered cohort | Mean Spearman rank correlation of 0.73 across 43 studies, after removing order-bias and low-information designs. |
Yashchin (July 2026), working paper, not peer reviewed
Numbers
Figures with provenance.
0.55
mean parameter-rank correlation across roughly 300 replicated studies
Inspect provenance
0.73
mean parameter-rank correlation across 43 studies passing design filters
Inspect provenance
43
studies passing design filters; a study count, not a respondent count
Inspect provenance
Model-predicted
Modeled and observed evidence.
Predicted
Estimated choice parameters from a reconstructed study on synthetic respondents.
Observed
The parameter estimates reported in the original human study.
Agreement
0.73 mean Spearman rank correlation across 43 studies passing design filters; 0.55 across roughly 300 replications.
Yashchin (July 2026), working paper, not peer reviewed
Limits
What this record establishes.
Spearman correlation runs from −1 to 1: 1 means the same rank ordering, 0 no monotonic rank agreement and −1 reversed ordering. These aggregate parameter-rank results do not establish effect-size agreement, realized business lift or accuracy for a buyer's study. The July 2026 working paper is not peer reviewed; 43 is a study count, not a respondent count.
Causality
Parameter-rank agreement does not establish effect-size agreement. A business decision needs validation matched to its intervention, population and outcome; this aggregate benchmark is a starting point.
- Intervention
- The attributes and levels varied in the original choice studies, reconstructed for simulated respondents.
- Outcome
- The rank order of estimated choice parameters.
- Design
- A discrete-choice replication pipeline compared simulated estimates with published human estimates. The working paper proposes a separate standard for held-out treatment effects and confidence-interval agreement.
Most AI generates language. Ours helps people make decisions.