Mean parameter-rank correlation is 0.55 across roughly 300 replications and 0.73 across 43 studies passing design filters. July 2026 working paper, not peer reviewed.
Spearman rank correlation compares the order of estimated choice parameters in a reconstructed study with the published human estimates. It does not measure the percentage of correct responses or realized sales lift.
The working paper proposes held-out treatment-effect and confidence-interval checks. The aggregate parameter-rank scores do not establish agreement on effect sizes or guarantee an individual study's result.
The public research compares the ordering of estimated choice parameters with published human estimates. It does not establish sales lift or guarantee the result of your study.
The paper’s 43-study cohort excludes order-bias and low-information designs. It is a selected cohort, not all replications. Rank correlation runs from −1 to 1: the same ordering is 1, no monotonic agreement is 0 and reversed ordering is −1. Rank agreement does not establish effect-size agreement.
The model snapshot below is an undated static CSV extract retained in the site repository. Its original run date and design-filter membership are not recorded. It is separate from the paper’s aggregate cohorts and is not a current availability or model-selection recommendation. Request the original run record before treating a row as validation for your study.
Similarity uses a 0 to 1 scale for parameter proximity. Coverage and direction match use 0 to 100 scales. Coverage describes how often human parameters fall within simulated confidence intervals; direction match describes agreement on the sign. Studies is the number in that model and domain row, not the number of human respondents.
Read the public working paperHistorical model snapshot: 32 rows, 8 models, 5 domains. The paper's 43 design-filtered studies are a separate cohort.
| Claude Sonnet (Databricks) | 56.78 | 40.07 | 82.63 | 19 |
| GPT-4 (Azure OpenAI) | 55.62 | 46.02 | 78.21 | 19 |
| Claude Sonnet (Databricks) | 54.63 | 45.79 | 70.95 | 23 |
| Gemini Flash (Google) | 54.63 | 23.84 | 81.59 | 19 |
| GPT-4 (Azure OpenAI) | 53.87 | 35.27 | 73.22 | 2 |
| Claude Haiku (Databricks) | 53.62 | 39.92 | 76.60 | 19 |
| GPT-3 Instruct (Azure OpenAI) | 52.79 | 41.50 | 75.97 | 18 |
| GPT-4 (Azure OpenAI) | 52.67 | 34.92 | 78.74 | 9 |
| Gemini (Google) | 51.09 | 33.46 | 76.83 | 9 |
| Claude Sonnet (Databricks) | 50.82 | 28.54 | 77.55 | 9 |
Five of 22 effects from one immigration conjoint. Each row puts the published human estimate beside our simulated estimate.
Score across all 22 matched effects: Spearman 0.64. Signs agree on 16 of 22.
One replication of Hainmueller et al. (2014), effects in probability of choice. Misses: unauthorized prior entry flips sign, and country of origin is scrambled. Working paper, not peer reviewed.
An average can hide the risk. Move the threshold to see how often each plan falls below it.
Loading outcome comparison…
Illustrative distributions, not customer data.
Change price and battery life. Each tile shows which option takes the largest choice share.
Loading preference landscape…
Illustrative choice shares, not fitted estimates.
Record its population, attributes, levels and choice task. A different outcome requires a different validation.
Run the specified task on simulated respondents and estimate the choice parameters.
Spearman correlation asks whether the estimates have a similar ordering. It does not establish agreement on their magnitude.
A buyer still needs checks matched to the decision, including population coverage and human outcome validation. Aggregate results do not substitute for those checks.