Compiling Code Is Not Validating a Model
A data science team asks an LLM agent to generate a PyMC model from a plain-language description, and the code compiles on the first or second try. Does that mean the model is right?
No. Compiling is a syntax check. It confirms the code runs without a Python traceback. It says nothing about whether the priors, likelihood, or causal structure match the decision the model is supposed to inform.
What a compile-and-revise loop actually checks
A recent hackathon prototype built by PyMC contributors demonstrates the pattern. The agent generates a candidate PyMC model from a natural-language prompt, sends it to a remote sandbox that attempts to compile it, and, if compilation fails, feeds the resulting Python traceback back to the agent so it can revise the model.
In one documented example, a capture-recapture model first failed with:
TypeError: HyperGeometric.dist() missing 1 required positional argument: 'k'Compiling and modeling correctness are different checks:
| A compile check catches | A compile check misses |
|---|---|
| Missing required arguments, like the traceback above | Whether the prior on population size is defensible |
| Type errors and shape mismatches | Whether the likelihood matches how the data were generated |
| Import and syntax errors | Whether the model's causal structure fits the decision it's meant to inform |
The agent used that traceback to produce a version that compiled and ran. The revised model sets up the same three inputs as before: 25 bears tagged on the first pass, 20 captured on the second, and 4 of those recaptures already tagged. It then bounds the unknown total population at 500 and gives it a DiscreteUniform prior no smaller than either capture count:
N = pm.DiscreteUniform("N", lower=max(n1, n2), upper=N_max)The likelihood swaps in the correctly ordered HyperGeometric call, pm.HyperGeometric("k", N, n1, n2, observed=k_observed), and the model samples 3,000 draws after 1,000 tuning steps before summarizing the posterior over N.
That fix is real. The prototype's own builders flag the population upper bound and the choice of likelihood as modeling decisions, not compiler outputs, and they're explicit that other formulations of the same problem remain open: a different, equally valid prior on population size would compile just as cleanly.
Why prompt wording changed the model's structure
A prompt-sensitivity test on a separate Think Bayes problem, the "lions and tigers and bears" classification task, shows this most clearly. A Dirichlet distribution gives this task a concise structure. After Python type annotations became a system-level requirement, the generated result shifted toward more convoluted categorical models with additional places to attach annotations. Removing that requirement restored the simpler, better-suited model.
Both versions compiled. Only one matched the structure of the actual problem.
The gap between "runs" and "correct"
Research on LLM-generated code shows execution success and correctness are separate properties: code can hallucinate a plausible-looking function or import that never existed, and it can pass an execution check while still encoding a wrong assumption or invalid dependency; see the analysis in Library Hallucinations in LLMs: Risk Analysis Grounded in Package Fabrication. A companion study on execution-based verification for generated code, CodeHalu: Investigating Code Hallucinations in LLMs via Execution-based Verification, reaches the same conclusion from a different angle: verifying that code executes catches a narrower class of errors than verifying that the code does what it's supposed to.
For a statistical or causal model, "supposed to do" means: the likelihood matches how the data were generated, the priors encode defensible assumptions, and the model's structure supports the actual decision at hand, not just the actual numbers in the training example.
Where independent review has to sit
A compile-and-revise loop catches outdated syntax, hallucinated function signatures, and shape mismatches. It is not a substitute for methodological review of whether the model's assumptions are the right ones for the question being asked.
Subconscious's approach to causal experimentation treats this as a hard line: execution success is necessary but not sufficient for causal validity. A model that compiles still needs to be checked against the decision it's meant to support. When that check matters, Subconscious can test or validate studies with real human participants, so a team can check a model's predictions against real behavior and catch miscalibration, though matching human data cannot certify the model's causal structure is correct. How we approach validation covers this in more depth, and current benchmark results are on the leaderboard.
The takeaway
Treat "it compiled" as the first gate, not the last one. Before a generated statistical or causal model informs a real decision, confirm someone with modeling expertise has reviewed its assumptions, independent of whether the code ran.
