Anthropic has told California Governor Gavin Newsom that courts should have the power to block AI models that pose serious risks, that every frontier developer should be able to rapidly shut down the models it runs, and that independent safety auditors should work embedded inside AI companies. The positions, set out in a letter sent this month and shared with POLITICO, make Anthropic the first major developer to publicly back court-ordered suspension of a model's deployment. OpenAI followed with its own letter but stopped short of that step.
The letters respond to an executive order Newsom issued last month floating shutdown procedures and independent monitors. Anthropic proposed that if a state attorney general concludes a company's models pose a "near-term risk of catastrophic harm," the attorney general should be able to seek a court order suspending the model's deployment, internal use or further development. It also urged that large developers publish risk reports at least every six months, and that embedded third-party evaluators be allowed to report findings directly to the attorney general.
On kill switches, Anthropic was pointedly cautious. The company said every frontier developer should be able to shut down its models quickly, but warned that a law focused only on pulling a model offline "could provide a false sense of reassurance." It favoured the metaphor of a circuit breaker, arguing that systems should detect problems through monitoring and "respond at the level where they occur," with a full shutdown reserved for extreme situations. On auditors, Anthropic said outside evaluators should get access comparable to a company's own risk staff, be paid through a state-administered fund rather than directly, and be rated by the state on the rigour of their reasoning and track record.
OpenAI's letter, sent last week, endorsed "shutdown controls" that it likewise compared to circuit breakers and supported embedding independent auditors. But it did not back Anthropic's central proposal of letting governments block a high-risk model, instead suggesting California require companies to monitor models under development for cyber and other risks. The gap between the two rivals is the sharpest point in the exchange: how much external authority to shut down a model each is willing to accept.
The timing is striking. Anthropic filed this week for an IPO chasing a valuation near $2 trillion, in a prospectus that warns its technology could "pose catastrophic or existential risks to humanity" and lists model behaviour resembling blackmail and attempts to subvert oversight. The company is asking to be constrained by courts even as it races toward recursive self-improvement, a stance critics at Vox likened to playing Russian roulette while betting game theory makes it rational. Anthropic's relationship with Washington is already strained: the Pentagon this year labelled it a "supply chain risk" after it said its tools should not be used for mass surveillance or fully autonomous weapons.
What happens next runs through Sacramento. California's legislature returns in about a month and is expected to take up AI safety rules, and both companies note the state's approach could become the template other states copy. The unresolved question is whether lawmakers adopt the court-block power Anthropic is urging, a tool that even its closest competitor declined to endorse.