What a Validation Gate Is, and Why It Should Decide What an AI Agent Ships

Edited by Avi Yashchin, CEO and Co-Founder · August 13, 2026

An AI agent's output looks finished the moment it stops generating text. Whether it matches something real is a separate question, and most AI-agent systems never force an answer before the output ships. A research team behind the open-source tool alchemize built a system that forces the answer, and the architecture is a useful model for any buyer deciding whether to trust an AI agent's output for a business decision.

The problem: code translation an AI agent could not simply generate once and ship

alchemize converts statistical models written in PyMC, a Python library for Bayesian modeling, into Rust code that runs faster. The naive approach, asking an AI agent to write the Rust translation once, fails for the same reason any one-shot AI generation fails: nothing checks whether the output is correct before it reaches production.

The team's fix was architectural, not a smarter prompt. They put the agent inside a loop with four tools:

generate or update the Rust implementation
compile it and surface the error messages
compare its numerical output against the original PyMC model's exact reference values
read code or error logs for diagnosis

The agent cannot stop until the compiled Rust output matches the source model's numbers. If a gradient is off, the agent reads the mismatch, finds the missing term, and rewrites: a hierarchical model with a zero-sum constraint took three to four iterations before its gradients matched. That validation step, not the code generation step, is what makes the system behave like a compiler instead of a generator: it produces output that's provably correct against a spec, and the agent only finishes once it clears that bar.

How much iteration a validation gate actually takes

The team logged convergence across five model types of increasing complexity. A two-parameter Normal model cleared the gate in 4 tool calls on 40K tokens, passing the first time it ran. A three-parameter linear regression model matched that: 4 tool calls, 54K tokens, no retries needed. A twelve-parameter hierarchical model took 8 tool calls and 153K tokens, and needed one retry before its gradients matched. A three-parameter GP model built on an ExpQuad kernel was the most expensive of the five, spending 11 tool calls and 467K tokens across three retries. A 142-parameter ZeroSumNormal model, despite its much larger parameter count, converged in 9 tool calls and 484K tokens after two retries.

Simple models converge on the first pass. Unusual constraints or larger parameter counts need more iterations, which shows the gate is doing real work, not rubber-stamping.

The team separately benchmarked the resulting Rust output's runtime against an already-optimized Numba backend across those same model types, with per-model speedups ranging roughly from 3x to 7x depending on model complexity. One maintainer noted part of that gap reflects headroom in the traditional backend's compilation pipeline, not a hard ceiling on non-AI approaches, worth reading as a model-specific figure, not a fixed multiplier.

Where the validation gate breaks down

The team was direct about the limits, and they matter for anyone evaluating a similar architecture:

Context limits on large models. A portfolio-optimization model with thousands of parameters exceeded the agent's context window; the full computation graph has to fit for the loop to work at all.
No formal convergence guarantee. Every tested model converged within a handful of iterations, but that is an empirical observation from testing, not a proof that every model will.
Spot-check validation has blind spots. The gate compares output at a single reference point. Subtler numerical issues, such as cancellation, underflow, or precision loss over long runs, can survive a spot check.
Toolchain dependency. The approach requires a working Rust build pipeline in whatever environment runs it.

None of this makes the architecture unsound; it marks where the guarantee stops, information a buyer needs before extending trust to the pattern.

Why this generalizes past one transpiler

The team's bigger claim isn't about Rust or PyMC. It's that the pattern (thin scaffolding, an agent that reasons about each case individually, and a validation step the agent cannot bypass) applies to any tool whose core job is transformation. The same team used the loop to convert the large majority of a public benchmark set of Stan models into PyMC and Rust, with both gradient and log-probability values validated at each conversion, and also applied it to translation between two other modeling frameworks.

That is the architectural takeaway for anyone evaluating an AI-agent system before trusting its output for a business decision: the question is never whether the agent looks confident. It's what reference value the output was checked against, and whether the agent was mechanically blocked from shipping until it matched.

The same discipline, applied to a decision instead of a code path

The transpiler validates a translated program against the exact number the original model produced. Subconscious validates a different kind of output, a causal experiment run on a simulated population, against a different kind of reference: real human behavior. Subconscious can test or validate studies with real human participants, moving from simulation to real-human validation without changing the causal question being asked.

Subconscious does not build compilers or transpile code; the comparison is architectural, not a claim about what the product does. Not every study is checked this way; the practice fits when the decision depends on it, the same way the transpiler's gate exists because a silently wrong translation is unacceptable. If you're deciding whether to trust an AI-generated study result the way you'd trust a compiled binary, see how a study moves through that process before it reaches a decision.

A cycle of four stages: generate code, compile it, compare its output to reference values, diagnose any mismatch. Diagnosis loops back to generation. Only a match exits the loop, into a final "ships" box. — The agent can only exit the loop through the comparison step, and only when its output matches the reference values.

Four limits: models too large for context fail outright; convergence is observed, not proven; the gate checks one reference point and can miss precision loss elsewhere; the approach needs a working build pipeline. — The gate proves correctness at a reference point, not everywhere, four gaps mark where that guarantee ends.

Sources

pymc-labs/alchemize: LLM-based, self-correcting transpiler, GitHub.
Agent SDK overview, Anthropic.