LLM Model Performance Leaderboard

Comparative analysis of various LLM models across different research domains, measuring their performance on key statistical metrics.

Filter by Research Domain

Scroll horizontally to see all columns →

LLM Model	Domain	Manhattan Similarity (%)↓	Coverage Probability (%)	Directional Effect Matching (%)	Spearman Correlation (%)
gpt4	Agricultural Sciences	51.45🥇	24.30	53.48🥇	43.50🥇
gemini flash	Public Health	50.51🥇	25.37	68.93🥈	53.09🥈
sonnet	Business Administration	50.35🥇	15.63	68.58🥈	44.00🥉
o1	Economics	50.19🥇	18.67	65.19	19.56
gemini flash	Consumer Research	49.45🥇	23.84	81.59🥈	63.63🥇
sonnet	Public Health	49.23🥈	45.79🥇	70.95🥇	52.57🥉
haiku	Business Administration	49.20🥈	18.51🥉	62.30🥉	46.00🥈
gemini	Business Administration	48.70🥉	29.16🥈	60.10	34.50
sonnet	Agricultural Sciences	48.60🥈	29.08🥇	47.40🥈	29.00🥈
gpt3-instruct	Consumer Research	48.43🥈	41.50🥈	75.97	45.28

🥇Top performer

🥈Second best

🥉Third best

Showing 1 - 10 of 32 entries

About the Metrics

Manhattan Similarity (%)

We recreate the conjoint survey from a published study and gather responses from an AI model. Next, we estimate a discrete choice model similar to the one reported in the study. By computing choice probabilities for each option using both the estimated and reported parameters, we measure their similarity using Manhattan distance. This metric quantifies how closely the AI replicates human decision patterns on a probability scale.

Coverage Probability (%)

We recreate the conjoint survey from a published study and collect responses from an AI model. Then, we estimate a discrete choice model similar to the one in the study. To assess similarity, we check whether the study's reported parameters fall within a confidence interval constructed based on estimated parameters. This metric evaluates Human–LLM Equivalence while accounting for sampling differences.

Directional Effect Matching (%)

We recreate the conjoint survey from a published study and collect responses from an AI model. After estimating a discrete choice model similar to the study's, we compare the signs (positive or negative) of the estimated parameters with those reported in the study. This metric measures Human–LLM Equivalence in terms of preference alignment.

Spearman Correlation (%)

We recreate the conjoint survey from a published study and collect responses from an AI model. After estimating a discrete choice model similar to the one in the study, we calculate the Spearman correlation between the estimated and reported parameters. This metric quantifies Human–LLM Equivalence based on preference ordering.

Domain-Specific Leaderboards

Detailed performance breakdowns for each research domain.

Public Health

Scroll horizontally to see all columns →

LLM Model	Manhattan Similarity (%)↓	Coverage Probability (%)	Directional Effect Matching (%)	Spearman Correlation (%)
gemini flash	50.51🥇	25.37	68.93🥈	53.09🥈
sonnet	49.23🥈	45.79🥇	70.95🥇	52.57🥉
haiku	47.97🥉	44.93🥈	65.83	42.48
o1	46.22	25.67	68.72🥉	53.35🥇
gpt3-instruct	44.50	41.12	52.27	17.86
gpt4	43.17	43.12🥉	67.59	47.04
gpt o3	41.22	21.05	55.21	32.55
gemini	40.19	40.47	55.42	22.78

Consumer Research

Scroll horizontally to see all columns →

LLM Model	Manhattan Similarity (%)↓	Coverage Probability (%)	Directional Effect Matching (%)	Spearman Correlation (%)
gemini flash	49.45🥇	23.84	81.59🥈	63.63🥇
gpt3-instruct	48.43🥈	41.50🥈	75.97	45.28
haiku	48.18🥉	39.92	76.60	49.79
sonnet	46.71	40.07🥉	82.63🥇	57.74🥈
o1	46.03	25.33	75.20	56.05🥉
gpt4	44.62	46.02🥇	78.21🥉	53.63
gemini	42.92	37.12	74.81	45.21
gpt o3	35.48	17.91	56.59	22.47

Economics

Scroll horizontally to see all columns →

LLM Model	Manhattan Similarity (%)↓	Coverage Probability (%)	Directional Effect Matching (%)	Spearman Correlation (%)
o1	50.19🥇	18.67	65.19	19.56
gpt3-instruct	47.97🥈	21.58	75.79	34.78
gemini	47.29🥉	33.46🥈	76.83	46.78🥉
sonnet	46.54	28.54	77.55🥉	50.67🥈
gpt4	45.46	34.92🥇	78.74🥇	51.56🥇
gemini flash	44.31	23.81	77.87🥈	43.89
haiku	39.17	31.17🥉	73.57	31.78
gpt o3	37.20	4.77	63.35	27.44