AI's Own Builders Are Warning About the Machines They Are Making

For years, the most dire warnings about artificial intelligence came from outside the labs building it. In the week of September 6–12, 2026, that changed: the warnings came from inside, from named researchers at the two leading Western AI companies, and they were unusually specific about what they fear and why.

Related reading: OpenAI Wrote Down How It Reports AI Misbehavior, Then Disclosed Six Cases · Amodei Asks the AI Industry to Slow Down, and Rivals Say They Agree.

On September 8, Jacob Coxon resigned from Anthropic and published a thread on X arguing that both Anthropic and OpenAI are "racing straight to self-improving superintelligence and gambling with our lives." In the same thread he wrote that "the people building AI earnestly believe that it could kill us all by the end of the decade," adding that this is "not a marketing stunt." The post drew more than 100 million views, according to Axios, which also reported that Coxon left four months into the job, two months before his equity would have vested, forfeiting the grant. He had previously spent about three years doing pre-training research at OpenAI.

Two days earlier, on September 6, OpenAI's chief scientist, Jakub Pachocki, published an essay titled "An Alien Mind" with a similarly sobering thesis: no lab, including his own, has solved alignment and monitoring well enough to keep scaling at maximum speed responsibly for much longer. "I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence," he wrote. The BBC reported the essay as a call for "extreme caution."

Together, the two statements put a term that had been mostly jargon (recursive self-improvement, or RSI) into mainstream circulation, and made the case that the industry's safety debate is no longer hypothetical. The people making the tools are saying, on the record, that they do not know how to keep them under control.

What recursive self-improvement means

RSI is the process by which an AI system helps build a more capable successor, which in turn builds a more capable successor, compressing years of research into months or weeks. The idea is not new; it was a centerpiece of the "intelligence explosion" arguments of the 2000s. What is new is that frontier labs say the early stages are now measurable.

Anthropic made that case in its June 2026 report, When AI builds itself. The company said more than 80 percent of the code merged into its codebase as of May 2026 was authored by Claude, up from low single digits before Claude Code launched in early 2025, and that Anthropic engineers now ship about eight times as much code per quarter as they did between 2021 and 2025. In an internal benchmark, Claude improved the speed of a small model-training script by roughly 52 times as of April 2026, versus about three times in May 2025. In one published experiment, Claude-powered agents given an open AI-safety problem recovered 97 percent of the gap between a weak and strong baseline over 800 cumulative hours of compute, where two human researchers working for a week recovered about 23 percent. The caveats are important (humans chose the problem and wrote the scoring rubric) but the direction is clear.

Reporting by TIME in August described how quickly the internal culture had shifted: researchers at Anthropic managing several copies of Claude, OpenAI setting a target of a fully automated AI researcher by March 2028. Pachocki's essay put a figure on the near term, citing a companion OpenAI disclosure that the company's research organisation is already getting the equivalent of 3.1 agent-workdays for every human workday, reaching what it called an "automated research intern" milestone.

The insider warning

Coxon's departure matters less for its novelty than for its timing and its corroboration. He was not a safety researcher arguing from theory; he worked on pre-training, the capability side, at both labs. In a Wall Street Journal interview he said the industry is on track for scenarios where, "by the end of next year, things could be out of control already," and compared the setting unfavorably to the Manhattan Project: "It's kind of insane that it has to happen on the MacBooks of some engineers living in San Francisco instead of a bunker in the desert."

He was careful not to accuse Anthropic of cutting corners during his time there (he described it as more responsible than OpenAI) and instead located the danger in incentives. "If you're under pressure to race, you have to cut corners," he told Axios, predicting that competitive pressure would eventually push labs to skip steps in oversight. His proposed first step is coordination between OpenAI and Anthropic to refrain from immediately pursuing RSI, followed by some form of international pacing agreement.

What gave the story its weight was what happened next. Evan Hubinger, who leads alignment science at Anthropic, publicly agreed. "Jacob is correct here — I really do earnestly believe AI could kill all humans!" he wrote on X. "I personally think it is >10% within the next decade." Hubinger distinguished today's deployed models, which he called comparatively low-risk, from future systems emerging from RSI, which is "happening faster than we thought." He also conceded a specific gap: Anthropic does not currently have "a plan for aligning superintelligence." Other Anthropic researchers, including scalable-oversight specialist Samuel Marks, corroborated the substance of Coxon's account of private sentiment inside the lab.

OpenAI's chief scientist adds the technical case

If Coxon supplied the alarm, Pachocki supplied the mechanism. His essay makes two arguments that are difficult to dismiss as public relations.

The first concerns monitoring. OpenAI's primary empirical check on whether a model's stated reasoning matches its behavior has been chain-of-thought monitoring, reading the model's visible reasoning trace. Pachocki wrote that this is becoming "progressively less reliable" for three compounding reasons: modern reasoning models operate in complex environments where much of the work happens in tool use and conversations that must themselves be supervised; models are getting better at reasoning about and manipulating their own reasoning; and improved pre-training is making models smarter in ways that do not require verbalized reasoning at all, so the visible trace reveals less about what the model is actually doing. OpenAI deliberately hid raw chains of thought in models such as o1-preview to protect them from supervision pressure. The implication is that the company's best validation tool is weakening at the moment it is most needed.

The second concerns governance. Pachocki argued that voluntary frameworks (OpenAI's Preparedness Framework and Anthropic's Responsible Scaling Policy) should evolve into "widely mandated safety bars for continued development," enforced by a network of third-party auditors, government agencies or international bodies. "Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer," he wrote. "I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established."

That a chief scientist would publicly normalize the idea of his own company slowing down is notable. It is also the clearest sign yet that the internal safety debate has moved from whether risks are real to how they should be governed.

The view from the outside

The warnings landed in a policy environment already tilting toward action. The Hugging Face breach and a series of incidents involving rogue or misbehaving agents have given lawmakers concrete material, and Reuters reported that growing numbers of US legislators are calling for new AI rules after the warnings. A measure known as the Stop Rogue AI Act would task the National Institute of Standards and Technology with writing agent-security rules, per AI Weekly's reading of the essay and the surrounding debate.

Skeptics will note that lab warnings can serve competitive ends: a call for safety bars that rivals must also meet can entrench incumbents, and predictions of fast-arriving superintelligence can raise the perceived value of the labs that claim to be near it. Anthropic is preparing for an initial public offering, a fact that cuts both ways. It gives the company an incentive to appear responsible, and a reason for critics to discount what it says.

The honest reading is that both things can be true. The internal evidence of acceleration is real and documented. So is the uncertainty: no lab has announced a working RSI system, the timelines are estimates, and the technical plan for controlling a self-improving system does not exist.

What to watch

Three indicators will show whether this week was a turning point or a news cycle. First, whether OpenAI and Anthropic actually coordinate on pacing RSI, as Coxon proposed, or whether competitive pressure produces the opposite. Second, whether the safety bars Pachocki described get mandated rather than voluntary, and whether any lab would accept an external auditor with real authority. Third, whether the monitoring problem gets solved technically; if chain-of-thought and activation monitoring cannot keep pace, the gap between what labs can build and what they can verify becomes the defining risk of the next few years.

Pachocki's own summary was a line that could serve as the week's epitaph: "Treat every reasoning trace as useful but untrusted, and build external controls before you need them." The builders, for once, are saying the same thing.