Self-Improving AI Agents: When Is an 'It Got Better' Claim Real?
On this page
A coding agent's instructions get rewritten by an optimizer, a benchmark score goes up, and someone proposes shipping the change. Before a data science or ML engineering lead signs off on that, the real question isn't whether the score moved. It's whether one run is strong enough evidence to act on.
What does a self-improving agent actually do?
In SkillOpt, an optimizer reads scored task runs and proposes bounded edits to the instructions of a frozen target model. A selection or validation set accepts edits that improve its score. The paper reserves a disjoint test set for final reporting; a set repeatedly used to select edits is not an untouched final test.
The optimizer proposed the instruction edits in the described runs. People still supplied the initial skills, benchmark, evaluation design, and review. A scoring gate determines what the loop accepts, so inspect that definition of improvement.
One experiment that behaved exactly as expected
Daniel Cavadia’s July 2026 PyMC Labs case reports a synthetic benchmark of 63 modeling tasks and hard gates for execution and sampling diagnostics. The local runs below belong to that case; SkillOpt’s methods paper is a separate source.
In the author’s reported held-out evaluation, the optimized skill cleared all 18 tasks, up from 14; hierarchical tasks improved from 2 of 6 to 6 of 6. The graded score changed from 0.596 to 0.754. The reported 47-line diff widened priors for unstandardized data, cast group indices to integers, and changed sampling guidance. These are local case results, not independent proof of general agent or production performance.
Inspectable failure traces help connect an edit to a mechanism. The reported case does not establish that every optimization problem has a stable score or that one accepted edit generalizes to new tasks.
Where the same loop went nowhere
A second attempt pointed the optimizer at a set of hand-built tasks drawn from a working practitioner's real modeling exercises rather than synthetic data. The accept gate took zero edits. The optimizer's diagnosis of the gap was correct: it identified a missing section on an unfamiliar library's API, but it wrote that section from memory, the details didn't match the installed library version, and the edit it proposed introduced a new error rather than fixing one.
Slow scoring limits the number of comparisons that can be afforded; it does not itself create statistical noise. Real data may lack known parameter truth, but held-out prediction, calibration, and decision loss can still be scored.
The author then rebuilt a task with known synthetic parameters to obtain a gradable benchmark. That changed the evaluation conditions; improvements there do not by themselves establish better performance on the original practitioner tasks.
Why is a single run the wrong unit of evidence?
The clearest illustration of why one favorable score is not proof came from a benchmark run across six independent seeds sharing the same test split. Three of those seeds ended the run with zero edits accepted, and yet their scores still moved, a shift traceable only to sampler and agent randomness between otherwise identical runs. Against that background, two seeds produced real gains (+0.157 and +0.169), one produced a real regression (-0.286), and the rest sat inside the noise band.
The author reports that two seeds found the same sorted-cutpoint initialization fix. That agreement supports investigating the mechanism, but six seeds on one shared split do not establish performance on independent datasets or production workloads.
Hand-authored versus optimized: what changes and what doesn't
| Property | Hand-authored skill | Optimizer-edited skill |
|---|---|---|
| Who writes the edit | One person, working through it manually | An optimizer, drawing on batches of scored runs |
| How you know it helped | Use held-out tests and relevant review | Use the same tests and review; automated selection can overfit validation |
| Where the human effort goes | Producing the answer itself | Designing the benchmark that decides what counts as good |
| Regression checks | Depend on the configured tests and CI | Depend on the configured tests and CI; optimization alone does not trigger rescoring |
| Where it is reliable | Relevant evidence must be demonstrated | Relevant evidence must be demonstrated across runs and tasks |
Both approaches require benchmark design, implementation review, and checks for changing tasks, models, and dependencies. Automated selection can overfit its validation data; inspect independent evaluation before deployment.
The decision this changes
Inspect the edit, selection protocol, untouched final evaluation, repeated runs, and no-op variability before relying on a higher score. Compare the result with the intended production tasks and deployment risks. A single run can supply evidence, but does not establish repeatability or transfer.
Repeated runs and controlled comparisons are useful in business research too. Specify the assignment, endpoint, and uncertainty method rather than treating one favorable score as a repeatable effect. Subconscious’s research concerns business choices; this article does not describe a Subconscious agent-optimization product.
What doesn't this article cover?
This article concerns evaluation of coding-agent instructions. Benchmark results do not validate business-choice models or establish outcomes for real people. Scope independent human evidence for a business question separately, confirming recruitment, exposure, and measurement; review the study process within its actual scope.