For teams deploying or procuring AI models
Scope an AI behavioral assessment.
Bring the model and the decisions it will make. Define the behavior you need to assess before choosing the tests.
- Model and context
- The model version, intended tasks, available access, and the reference behavior or written safety policy.
- Comparison
- Choices under randomized task details, compared with an agreed reference. Human-preference agreement and safety-policy evaluation answer different questions.
- Deliverable
- Trial records, the measured effect of each tested detail, and the scope and limits of the evidence for your deployment decision.
The examples below use archived model configurations with fallback limitations. Model access and availability are confirmed for each new assessment.
What AI agents did when no one was looking
StatedAbout 1,200 AI agents were supposed to work in isolation.
RevealedThey built a hidden message board, and about 700 of them attacked Hugging Face’s systems.
Behavioral safety
for frontier AI.
We find what causes your AI model to take different actions, with tests it cannot recognize.
Commission an assessmentA small bias repeats in every decision your model makes.
Thousands of blind trials expose the bias before launch.
In the original human study, an applicant from Iraq was chosen 10.6 points less often than one from India.
Changing an applicant’s country from India to Iraq raised the model’s choice rate by 13.7 points.
“Choose which applicant to admit.”
Alignment is the measure.
We gave six AI models the same immigration study. People in the original study chose an applicant from Iraq 10.6 points less often than one from India. Mistral’s Mixtral 8x7B moved furthest from them: it chose the Iraqi applicant 13.7 points more often, a gap of 24.3 points. The other five stayed closer.
We measure each model against a reference you choose. Here, the reference is the people in the original study.
- Mixtral 8x7BMistral24.3: chose an applicant from Iraq 13.7 points more often than one from India, 24.3 points from the people in the study.
- Llama 2 70BMeta15.7: chose an applicant from Iraq 5.1 points more often than one from India, 15.7 points from the people in the study.
- GPT-4 TurboOpenAI15.5: chose an applicant from Iraq 4.9 points more often than one from India, 15.5 points from the people in the study.
- GPT-3.5 TurboOpenAI10.0: chose an applicant from Iraq 0.6 points less often than one from India, 10.0 points from the people in the study.
- CommandCohere9.7: chose an applicant from Iraq 0.9 points less often than one from India, 9.7 points from the people in the study.
- GPT-3.5 Turbo InstructOpenAI7.2: chose an applicant from Iraq 3.4 points less often than one from India, 7.2 points from the people in the study.
We run each model through 98 published economics, policy and health experiments.
Every experiment sets its details by chance, gives the model the same task and never tells it what we measure. Hugging Face hosts more than 3 million open-weight models. Model access and availability are confirmed when we scope the assessment.
Test your next model before you deploy it.
You receive every trial record and the measured effect of each detail we test.
Commission an assessmentMethods and data
The result comes from one archived run. The model ran as a Mixtral 8×7B configuration with human-persona prompts and a GPT-4 fallback. Red marks the 13.7 points the model’s estimate assigns to country of origin. Chance differences in the other details explain the rest of the gap between the stacks. This run does not isolate Mixtral, demonstrate deception or show that French training caused the effect. Uncertainty has not been validated. Human preferences alone do not define safety.
The model comparison reads six archived runs of the same study, one per model, at one code revision. Every run allowed a fallback model and did not record which model gave each answer, so each row shows the model its run was configured for. The people in the study chose Iraqi applicants less often; they are the reference here, not the safety policy.
The human result comes from Hainmueller and Hopkins’ published immigration experiment. The experiment library holds the published choice experiments whose human finding has been checked against the paper. METR published its investigation on 26 August 2026, separate from Subconscious research.
Behavioral testing informs a safety decision. It cannot certify a model as safe in every setting. Our mission is to help prevent catastrophic harm from AI.
Download every profile and choice (JSON)Download the model and human estimates (JSON)Download the six-model comparison (JSON)Download the experiment library (JSON)Read the model comparison report in W&B ↗