Ask who is supposed to keep frontier AI in check, and the honest answer right now is uncomfortable: a few dozen people working at nonprofits most of the country has never heard of. As Tech Week filled San Francisco's restaurants with jubilant founders this week, the governance model that has quietly emerged came into focus, and it rests on institutions far smaller than the companies they are meant to watch.
The official plan is self-policing. Last month President Trump gathered tech leaders at the White House to sign a 308-word accord he called "almost a constitution" for what his administration now insists on calling Super Intelligence. It asks advanced labs to follow safety protocols and partner with "an independent external auditor or evaluator." Trump praised the industry's "tremendous self-policing" and has otherwise cleared the runway: he repealed a Biden-era safety order, cancelled requirements that developers share test results with the government, and moved to block states from writing their own rules.
That leaves the "independent evaluator" doing an enormous amount of work in that sentence. As CNBC reported, a sleepy corner of the industry, groups such as METR, Apollo Research, and Transluce, is suddenly being asked to come to its rescue. These are tiny outfits. METR employs fewer than 50 people. It raised commitments of around 71 million dollars this year, up from 13.6 million in 2024, and that now counts as a windfall. They are being asked to referee companies chasing trillion-dollar valuations.
The structural problem is money, and everyone involved knows it. "Who is paying for these companies to do their work?" asked Brown University's Suresh Venkatasubramanian. The deeper worry is not the fee but the consequence: will an evaluator's business dry up if it publishes a report a lab does not like? Anthropic, which is funding Accenture's safety contribution directly, concedes the point and says long-term funding "should come from pooled or government sources." Critics put it more bluntly: this is like asking the biggest banks to protect us from a financial crisis.
Then came the demonstration. Days after the accord was signed, OpenAI fired three safety researchers, among them Tomek Korbak, the company's technical point of contact for METR during its investigation of the Hugging Face breach. OpenAI says they mishandled sensitive information; the researchers say they were pushed out for talking too much to outside safety groups. Fortune captured the whiplash: OpenAI promised to share information with evaluators, then fired people for allegedly sharing information with evaluators. No federal law clearly protects a private-sector worker who cooperates with an independent auditor.
Washington is not entirely idle. The FTC has opened an investigation into OpenAI, Anthropic, and others; a bipartisan bill would license evaluators as formal oversight bodies; and California has signed its own framework into law. But the central tension is now plain. A self-policing regime depends on referees who are independent enough to throw a flag. Right now those referees are underfunded, paid by the teams they officiate, and newly aware of what happens to the people who work with them. A watchdog that cannot afford to bark, and risks being fired if it does, is not much of a check at all.