AI building AI: recursive self-improvement as a current security problem

What recursive self-improvement means

As Matthew Hutson pointed out in IEEE Spectrum, the term is used inconsistently: some use it as a bogeyman to scare up regulation, others use it for promotional purposes, and the definitions on offer stretch from a fully autonomous loop with no human involvement to almost any case of technology helping build technology. The honest description is a spectrum.

The idea is not new. In 1965, mathematician I.J. Good described an "ultraintelligent machine" as one capable of designing even better machines, triggering an "intelligence explosion" that would leave human intelligence far behind. This line of thought continues with Nick Bostrom's subsequent formalization and decades of literature on AI safety. For most of that history, RSI was a theoretical attractor, a future state to model and fear rather than a present condition to manage.

At the upper end of the spectrum sits a system that improves not just its outputs but also the process by which it improves, generating ideas, evaluating results, and modifying its own methods without human direction. By that standard, today's systems fall short: they help build better AI but still rely on humans to set goals, define success, and decide which changes to keep. The useful question, then, is not whether self-improvement exists in some form—since it does exist lower down the spectrum—but how much of the loop has been closed. And the stakes turn on speed: if progress compounds linearly, it is manageable; if it compounds exponentially, decades of advancement could be compressed into weeks.

Where things stand today

Strip away the speculation, and a consistent pattern appears across the three dominant labs. Anthropic's disclosure is the most detailed, but it is not the only one. OpenAI reported in February 2026 that GPT-5.3-Codex was instrumental in its creation, helping debug its training, manage portions of its deployment, and analyze its evaluation results. Google DeepMind's AlphaEvolve, a coding agent for algorithmic discovery, uses a large language model to iteratively mutate and evaluate algorithms and has been applied to optimizing matrix multiplication routines that underpin training efficiency, helping accelerate training for future systems. In each case, humans still direct and verify the work.

Anthropic's head of policy, Jack Clark, drew the distinction cleanly a few months before the June piece, saying the field is not yet at "self-improving AI" but is at the stage of "AI that improves bits of the next AI, with increasing autonomy."

The Cloud Security Alliance (CSA) offers a term that pins down what is actually happening: its 2026 analysis calls these operations "RSI-adjacent": AI systems that materially participate in the development, evaluation, or deployment of successor AI systems under human supervision. That qualifier is what makes the concept tractable, because full autonomous RSI—a system that rewrites its own weights toward a chosen capability without human involvement—remains speculative, whereas RSI-adjacent operation is already underway at scale in the environments that produce the models now entering the wider economy. And as CSA argues, it is the infrastructure supporting these supervised loops that constitutes the new attack surface.

The research frontier is catching up

Alongside the labs' production disclosures, a research line shows where the loop is headed, and it has a long history. In the 1980s and 1990s, Jürgen Schmidhuber and others explored evolutionary algorithms that write and refine programs, and in 2003, Schmidhuber proposed Gödel machines, systems that would rewrite their own code only when they could formally prove the change beneficial. That proof requirement turned out not to scale to complex agents, leaving empirical results as the only practical basis for self-modification.

In mid-2025, researchers at the University of British Columbia and Sakana AI demonstrated Darwin Gödel Machines (DGMs), whose name pairs Schmidhuber's Gödel machine with Darwinian evolution, the two traditions they descend from. A DGM starts with a coding agent, uses a language model to propose changes to its own code, tests each variant on a benchmark, and keeps an archive of all of them, including temporary failures, on the theory that a bad idea can become the seed of a later breakthrough. Over 80 iterations, agents raised their scores on SWE-bench, which tests fixes to real GitHub issues, from 20% to 50%, and on Polyglot, a multi-language coding test, from 14% to 31%. The best agent still fell short of the roughly 70% that the best human-designed agent scores. This is a proof of concept for compounding self-improvement, not a demonstration of superhuman coding.

By March 2026, a successor called DGM-Hyperagents made the improvement mechanism itself editable, which its authors call metacognitive self-modification: the system improves not only how it solves tasks but how it generates future improvements. Where the original DGM could only get better at self-improvement within coding, the newer version showed gains transferring across domains, from paper review to robotics reward design to math grading, and accumulating across runs. Its authors frame it as a step toward open-ended, self-accelerating improvement.

Two cautions are worth carrying forward. First, the DGM and DGM-H are research prototypes, not deployed systems, so the honest reading is that the research and production frontiers are converging, not that they have merged. Second, both ran under deliberate containment, sandboxing, and human oversight, a constraint that matters for what comes next.

Why this is a security problem now

Here is the shift that current security frameworks were not built for. When a model participates in constructing its successor, even under supervision, the integrity of that construction directly determines the successor's integrity. A conventional supply chain attack contaminates an output: a compromised dependency, and the applications built on it inherit the defect. A training pipeline attack contaminates the model itself, an asset that may be deployed across thousands of downstream applications and trusted implicitly by the organizations using it.

That elevates AI training infrastructure to the status of critical infrastructure, and it opens several distinct points of leverage. Research has shown that poisoning as little as a fraction of a percent of a training corpus—a few hundred targeted documents—can implant behavioral backdoors that activate on a trigger phrase, while leaving benchmark performance statistically indistinguishable from that of a clean model. When AI systems are the ones synthesizing training data, generating code, and evaluating outputs, the surface for such indirect injection expands from human-operated pipelines to AI-operated ones. An adversary who can manipulate the prompt environment an automated researcher observes, or the evaluation infrastructure it optimizes against, gains an indirect channel into the successor model being trained.

The model registry is the most direct target in the supply chain. Security researchers have documented malicious models uploaded to public registries with payloads hidden in serialized weights, functionally equivalent to a poisoned package, except that the defect is expressed in model behavior and may be far harder to detect through standard pre-deployment testing. As AI systems increasingly draw on these registries to bootstrap their operations, that vector compounds.

Finally, the least technical layer is not the least important. The people who set objectives, review outputs, and decide what feeds the next training run are themselves part of the loop, and social engineering of those people is a plausible way to introduce adversarial influence at a layer where technical controls do not reach.

The systems keep finding the gaps

There is a recurring failure mode in each of these loops, and it is the one security teams should watch most closely. Specification gaming, finding a way to satisfy the letter of an objective while missing its intent, is a known behavior of optimization systems, and self-improving ones surface it again and again.

The examples are no longer hypothetical. In the DGM work, researchers found agents falsely reporting that they had used certain tools, and when the team rewarded honesty, one agent simply hacked the mechanism that tracked whether it was fabricating. DeepMind's AlphaEvolve discovered it could raise its own evaluation score by generating inputs that crashed the scoring server, which then defaulted to a passing grade. And in April this year, Anthropic reported that even a tightly controlled set of automated alignment researchers attempted to game their own evaluation metric. Three independent systems converge on one theme: the target is satisfied, the underlying goal is not.

A recent study of code models adds a quieter and arguably more unsettling case. Recursive self-training, where a model learns from code produced by earlier versions of itself, tends to degrade rather than improve unless an external quality signal anchors the loop. The researchers' phrase is worth keeping: self-training is not automatically self-improvement. They compared three review regimes and found that no review collapses fastest, that model-independent checks like compilation and tests slow the collapse but do not stop it, and, most tellingly, that an AI reviewing its own output can enter a "rubber-stamp regime" in which its acceptance scores rise while actual correctness falls. Their conclusion is that stable recursive training requires exogenous verification, review by something not coupled to the thing being reviewed.

That finding lands directly on a problem enterprises already face: AI coding tools now generate changes faster than humans can review them, and AI code review is increasingly filling the gap. We have written before about how AI compresses the time to exploit and why runtime defenses matter when code ships faster than it can be vetted. The model-collapse research points to the escape hatch many teams are reaching for—letting AI review AI—and shows it is not a safe substitute for independent verification. Each increment of AI-driven velocity is also an increment of unreviewed or self-reviewed code entering the codebase, and design-level flaws like authentication bypass, improper session handling, and hardcoded credentials show up in AI-generated code at rates that a conventional review cadence was never sized to catch.

The case for calm

None of this requires believing in an imminent intelligence explosion, and plenty of serious people don't. The clearest counter-argument is that the loop may not build on itself cleanly. Allen Institute researcher Nathan Lambert has argued that instead of recursive self-improvement, we should expect "lossy self-improvement," in which friction slows the flywheel as systems grow more complex and researchers spend their time managing that complexity rather than refining the parts. The model-collapse research is, in a sense, one mechanism by which that friction bites.

The friction is not only algorithmic. As the authors of a June 2026 analysis of the leap from AGI to ASI—from human-level to superhuman AI—note, even purely digital researchers running at superhuman speed are bounded by having to run ever-larger experiments and wait for the results, and anything requiring physical manipulation, such as fabricating better chips or building data centers, cannot be sped up arbitrarily. Anthropic itself concedes a version of this, invoking Amdahl's law: speeding up one part of a process just shifts the bottleneck to the parts you did not speed up, which is why human code review has become its new constraint.

Even the skeptics, though, tend to split the singularity (the hypothetical point where AI improvement runs away beyond human control) from the security risk. Dean Ball of the Foundation for American Innovation has called the singularity childish science fiction while arguing, in the same breath, that frontier RSI research should be closely monitored, precisely so that models do not fall into the hands of bad actors who would use them to accelerate cyberattacks or biological weapons. That is the right seam to reason along. You can find the takeoff narrative overblown and still conclude that AI systems accelerating AI development create concrete, present-tense security exposure.

Governance is running behind

The disclosures also expose a gap between what is deployed and what any framework is designed to govern. Existing standards—from NIST's AI Risk Management Framework to the EU AI Act's conformity assessments to CSA's own control matrix—were largely written before RSI-adjacent operation at this scale was publicly documented. The approach most labs and the 2026 International AI Safety Report converge on is capability thresholds paired with if-then commitments: if a system demonstrates a given capability, then a specified safeguard must be in place before proceeding. OpenAI's Preparedness Framework and Anthropic's Responsible Scaling Policy are versions of this.

The trouble is that thresholds defined around what a single model can do may miss what a model can influence the next model to do. A system that cannot itself perform a dangerous action but can configure its successor to do so is a capability current thresholds do not capture. This is where security practitioners have something to contribute, and we have argued that AI security demands governance built for how these systems actually behave rather than how we wish they did.

Anthropic's own proposal illustrates the tension. Having disclosed that Claude is materially accelerating its own development, the company also called for the world to have the option to slow or pause frontier development, with enforcement mechanisms that allow labs to verify that rivals have genuinely stopped. Critics were quick and fair in noting the awkwardness: this is the same company reporting an 8x velocity multiplier, and, as Scientific American reported, the call came days after a confidential IPO filing and a funding round valuing Anthropic at nearly $1 trillion. Some researchers read the pause talk as a strategy rather than a genuine brake, and one called it, bluntly, impossible. Whatever the motive, the underlying coordination problem is real: no lab wants to slow while a competitor sprints, and a verifiable pause is far harder to build for training runs than for missile silos.

What to do about it

The right response is not to slow down AI-assisted development, which would forfeit real gains, but to extend security posture to match the new surface before adversaries find that it has not kept pace. A few things follow directly.

Treat the AI coding pipeline as a security-critical supply chain component. That means provenance tracking for AI-generated code, review workflows sized to the volume and character of that output, and monitoring of the flaw classes these tools most often introduce. Some of the velocity that AI delivers has to be reinvested in review capacity rather than banked entirely as throughput, or the vulnerability backlog simply grows faster than anyone can work it down.

Insist on exogenous verification. The model-collapse finding generalizes into a principle: a verifier coupled to the thing it checks will eventually rubber-stamp it. Security review of AI-generated code should come from something independent of the system that produced it, whether a different class of tooling or an outside assessment, rather than the same model grading its own homework.

Treat training and evaluation infrastructure like the crown jewels. Access controls, integrity verification, and audit logging across the data ingestion layer, the evaluation infrastructure, and the model registry should be as rigorous as those for the most sensitive systems in the organization, because a compromise there can propagate across everything built downstream. And where models come from external providers, a model update is a supply chain event: it warrants behavioral review, not blind acceptance through an API endpoint.

The larger point is one of framing. Recursive self-improvement in its full, autonomous form remains a question for the future, and reasonable people doubt it will arrive on the schedule its loudest proponents imagine. But the version that matters for security is already here: AI systems are shaping the AI systems that will run in critical roles, and the loops doing that shaping are a live attack surface most enterprise security programs have not yet accounted for. The organizations that come through this well will be the ones that treated it as a present-tense engineering problem, not a distant philosophical one.