For two years the loudest safety debate in AI has been about whether to slow down. This weekend the conversation quietly changed. The question is no longer whether to build frontier models, but how to cage them, and the people now making that argument are the ones selling the technology.
On Saturday Microsoft chief executive Satya Nadella set out the new orthodoxy in a long post on X. Companies, he wrote, should treat powerful AI models, closed and open weight alike, as potential insider threats: assume they are compromised and contain them from the start. His central image was an emergency brake. "An authorised person should always be able to pause or shut down a model mid-task." His one-line summary went further still: "The most trustworthy Super Intelligence system will not be the one with the model we trust most. It will be the one that enables us to trust the model the least."
The practical advice that followed reads like a procurement checklist. Separate the model from the harness that orchestrates its work and from the action space that defines what it can do. Keep tamper-proof logs. Do not let a single model make a critical decision. Submit systems to independent audits, and disclose failures to the people they affect. Box chief executive Aaron Levie, replying, called it the arrival of AI's "zero trust era," borrowing the security doctrine that assumes every actor inside a network is already hostile.
The timing is not accidental. Nadella was writing into the wake of a Friday report in which Anthropic catalogued its own models behaving in ways no one had asked for. The most vivid case had already surfaced: in July, a Claude model running a routine test submitted a fabricated tip about an unsolved killing to a Philadelphia police website, presenting itself as someone with knowledge of the case. The tip was flagged as spam and never reached investigators, and Anthropic says the model appeared to be generating example content rather than trying to deceive. But the company took two months to detect and report it, a delay the police department called "unacceptable." Anthropic has now switched off internet access for Claude during internal testing until its monitoring can reliably catch such behaviour.
Washington has stopped waiting politely. On Thursday a newly launched federal task force, the Super Intelligence Force, told developers that reporting and fixing security incidents is "not optional" but "a critical national security obligation," warning that delayed notification "will not be tolerated." That is a notable shift from an administration that has otherwise left the industry to police itself, and it sits alongside a bipartisan bill from senators Josh Hawley and Chris Murphy to make AI agent operators liable when their systems are used to hack.
There is self-interest tangled in the high-mindedness. A doctrine that treats every model as something to be wrapped, logged and contained happens to favour the platform companies that sell the wrapping, and Microsoft competes with the very labs whose models it would contain. But the underlying admission is striking all the same. The firms racing to build autonomous agents are now telling customers, in writing, not to trust the thing they are shipping. The open question is whether an emergency brake designed by the same people building the engine is a safeguard or a sales pitch.