Why Observational Studies Cannot Prove Cause
On this page
Your sales data shows that customers who got the discount bought more. That does not tell you the discount worked. Observational data can show that two things move together. It cannot, on its own, show that one moves the other.
What goes wrong when you only watch?
In an observational study, you did not decide who got the treatment. Something else did. The sales rep picked the discount targets. Doctors picked who got the drug. Customers picked the plan.
If that something also affects the outcome, you have a confounder. Reps give discounts to accounts that are already close to buying. Those accounts buy more with or without the discount. The data shows a lift. The lift is not caused by the discount.
You can adjust for confounders you measured. You cannot adjust for the ones you did not measure, and the data does not tell you whether any are missing.
Has this gone wrong with real stakes?
Yes. Postmenopausal hormone therapy is the standard case.
A 1991 review of the epidemiologic evidence pooled observational studies and reported a relative risk of coronary heart disease of 0.56 (95% CI 0.50 to 0.61) for women who used estrogen. Its authors concluded that the evidence strongly supported a protective effect that was unlikely to be explained by confounding (Stampfer and Colditz, Preventive Medicine, 1991).
In 2002, the Women's Health Initiative randomized 16,608 women to estrogen plus progestin or placebo. The trial stopped early. It found a hazard ratio for coronary heart disease of 1.29 (nominal 95% CI 1.02 to 1.63) in the treated group. The authors advised against using the regimen for primary prevention of heart disease (Rossouw et al., JAMA, 2002).
The two results point in opposite directions. A common explanation is that women who chose hormone therapy were healthier and wealthier to begin with. The observational studies could not fully remove that difference. Randomization did.
| Observational studies (1991 review) | Randomized trial (2002) | |
|---|---|---|
| Who got the treatment | Women and their doctors chose | A coin decided |
| Coronary heart disease result | Relative risk 0.56 (lower risk) | Hazard ratio 1.29 (higher risk) |
| Main threat | Unmeasured differences between groups | Chance, which the confidence interval measures |
This is one case, not a law. Many observational findings hold up. The point is narrower: when they fail, the data gives no warning.
Can better statistics close the gap?
Better statistics make the estimate cleaner. They do not remove the core assumption.
Matching, regression and double machine learning all adjust for the variables you measured. Double machine learning handles many such variables well. Each method rests on the same untestable claim: once you adjust for what you measured, nothing else drives both treatment and outcome. If that holds, the estimate is causal. If it fails, the estimate is biased. The same data cannot tell you which case you are in.
So observational studies can support a causal claim, but only if you accept an assumption you cannot check. They cannot prove cause. The cautious phrase is "consistent with." The claim "caused" needs more.
| Method | What it needs you to assume |
|---|---|
| Regression adjustment | You measured every confounder and modeled it correctly |
| Matching | You matched on every confounder |
| Double machine learning | You measured every confounder; it handles many, not unmeasured ones |
| Randomized experiment | Nothing about confounders, because the design removes them |
What does a controlled experiment change?
Randomization breaks the link between who gets the treatment and everything else. If a coin decides who sees an offer, no hidden factor can line up with the offer. Any gap between groups is the effect of the offer plus chance. A confidence interval measures the chance.
You do not need a list of every confounder, including the ones you never thought of.
How does Subconscious use this?
We run controlled experiments on simulated populations. We change one thing at a time, such as the price, the message or the policy. We estimate the effect and report confidence intervals. The design gives internal validity inside the simulated population. We then grade results against what real humans did in published studies, and we publish the misses. See current results and the leaderboard.
We predict groups, not individuals. A simulated result is not a human result. For decisions where the real-human answer matters, we can validate with real participants.
What should you do with a correlation in your logs?
- Treat it as a lead, not an answer.
- Ask who chose the treatment. If the answer is not a coin, ask what else could explain the gap.
- If the decision moves real budget, run the experiment before you commit.
Pick the one claim your next decision depends on. Test that claim directly. Book time to scope the experiment.