Self-Improving AI Agents: When Is an 'It Got Better' Claim Real?
A coding agent's instructions get rewritten by an optimizer, a benchmark score goes up, and someone proposes shipping the change. Before a data science or ML engineering lead signs off on that, the real question isn't whether the score moved. It's whether one run is strong enough evidence to act on.
What a self-improving agent actually does
A self-improving agent measures its own performance against a benchmark and edits its own guidance (usually a skill document, a block of instructions injected into a frozen model's context), keeping only edits that beat a held-out score. An optimizer model reads a batch of scored runs, proposes bounded add, delete, or replace edits, and accepts an edit only when it strictly improves that held-out result (SkillOpt: Executive Strategy for Self-Evolving Agent Skills).
Nobody hand-writes the improved instructions anymore. Somebody still has to write the benchmark that decides what "improved" means, and a benchmark with no held-out score gives the optimizer nothing to climb.
One experiment that behaved exactly as expected
In one documented run, an optimizer was pointed at a modeling skill and a synthetic benchmark of 63 tasks spanning linear regression, small hierarchical models, and time series. Scoring gated on hard checks (code executes, sampling completes, convergence diagnostics clear a threshold, zero divergences) before a continuous quality score was computed on top.
The optimized skill cleared all 18 held-out tasks, up from 14, with the entire gain concentrated in the hierarchical family, which moved from 2 of 6 tasks passing to 6 of 6. The graded quality score moved alongside it, from 0.596 to 0.754. The accepted edit was a 47-line diff: wider priors to match unstandardized data, explicit integer casting an underlying library required, and a firmer sampling budget for the small-group cases. Every one of those edits maps to a known, diagnosable failure (a divergence, a type error, a prior that was too tight), visible directly in the run's trace.
The pattern behind every genuine win in this kind of loop: a benchmark that returns a stable, repeatable score, and failures an optimizer can diagnose by reading output rather than recalling an exact library version.
Where the same loop went nowhere
A second attempt pointed the optimizer at a set of hand-built tasks drawn from a working practitioner's real modeling exercises rather than synthetic data. The accept gate took zero edits. The optimizer's diagnosis of the gap was correct: it identified a missing section on an unfamiliar library's API, but it wrote that section from memory, the details didn't match the installed library version, and the edit it proposed introduced a new error rather than fixing one.
Two properties killed the loop here, independent of the optimizer's skill. The task set was slow to score, on the order of thirty minutes per fit, which made the validation gate noisy enough that even a real improvement would have been hard to distinguish from chance. And real data has no known ground truth to grade recovery against, so the graded signal that drove the first experiment wasn't available at all.
The fix was not a smarter optimizer. It was rebuilding the same task shape as a synthetic, gradable benchmark with a stable score, which moved the same domain back into the working zone and produced measurable, trace-diagnosable wins.
Why a single run is the wrong unit of evidence
The clearest illustration of why one favorable score is not proof came from a benchmark run across six independent seeds sharing the same test split. Three of those seeds ended the run with zero edits accepted, and yet their scores still moved, a shift traceable only to sampler and agent randomness between otherwise identical runs. Against that background, two seeds produced real gains (+0.157 and +0.169), one produced a real regression (-0.286), and the rest sat inside the noise band.
Two independently-run seeds converged on the same fix, initializing an ordered model's cutpoints with a sorted starting value rather than letting the sampler choose, which is the kind of agreement that separates a transferable improvement from a lucky roll. Without seeing several seeds and the no-op baseline spread between them, a single favorable run and a single unfavorable run are indistinguishable from noise.
Hand-authored versus optimized: what changes and what doesn't
| Property | Hand-authored skill | Optimizer-edited skill |
|---|---|---|
| Who writes the edit | One person, working through it manually | An optimizer, drawing on batches of scored runs |
| How you know it helped | Nothing formal; someone reviews the output by eye | It has to clear a held-out validation score |
| Where the human effort goes | Producing the answer itself | Designing the benchmark that decides what counts as good |
| Regression safety | None; drift only surfaces once it reaches users in production | Automatic re-scoring every time the model or library changes |
| Where it's reliable | Anywhere, but unverifiable | Stable, graded gate paired with trace-diagnosable failures |
Neither column removes the need for judgment. The optimized column just moves the judgment earlier, into the benchmark design, where it can be checked once instead of argued about after every edit.
The decision this changes
For a team deciding whether to adopt an automated skill-rewrite, a coding-agent prompt swap, or any "it scored higher" claim as grounds for a production change: demand the held-out benchmark before the edit, demand more than one seed, and demand the no-op baseline spread that tells you what noise alone looks like. A change that clears that bar is evidence. A change that doesn't is a single lucky or unlucky draw, and treating it as proof risks reversing a real decision.
That is the same discipline behind testing whether a business action (a price change, a message, a positioning move) actually caused the outcome a team is claiming credit for. Subconscious's research and leaderboard methodology runs controlled, replicated experiments with confidence intervals for the same reason: one observed result isn't a causal, repeatable effect, in a benchmark or in a market test. Subconscious does not build, train, or optimize AI agent skills or prompts. The discipline shown here is the transferable part, not the specific tooling.
What this doesn't cover
This is a benchmark-engineering practice for coding-agent instructions, not a usability study, a clinical trial, or an automatic proof that an agent performs better in production. If a team wants to move past benchmark scores entirely and check how an actual claim performs with real people, that is a separate step: see how Subconscious moves from a simulated test to real-human validation without changing the underlying question being tested.