Early Anthropic hire, former METR COO have found a way to rein in rogue AI agents

A time aft Anthropic researcher Jacob Coxon discontinue his occupation implicit concerns that AI could termination america each by the extremity of the decade, I met with founders and brothers-in-law Rune Kvist and Rajiv Dattani. They deliberation they person a solution that could prevention america all, oregon astatine slightest assistance forestall AI agents from going rogue wrong enterprises.

“AI is getting smarter astatine an progressively accelerated rate. The astonishing happening astir AI is that it becomes harder to follow and harder to power arsenic AI gets smarter, not easier,” said Kvist, an aboriginal Anthropic worker who is besides joined to Dattani’s sister). Dattani is the erstwhile COO of the AI information probe enactment METR.

The brace launched a startup called Artificial Intelligence Underwriting Company (AIUC) that hopes to bring AI information to enterprises and companies gathering AI models and agents. The startup names Cursor, Lovable, Harvey, and ElevenLabs arsenic customers.

On Tuesday, AIUC announced a $40 cardinal Series A led by Ribbit Capital, with information from First Harmonic. It antecedently closed a $15 cardinal effect circular from Nat Friedman done his money NFDG on with Emergence, Terrain, and Anthropic co-founder Ben Mann, among others, bringing its full backing to $55 million.

What caught the attraction of this A-list radical of investors is AIUC’s effort to use a acquainted cybersecurity exemplary to a caller acceptable of AI risks. The institution has built a third-party audit and certification furniture for AI agents.

“Banks, hospitals, governments and militaries nary longer diminution to deploy AI due to the fact that a exemplary isn’t astute enough,” Kvist said. “They diminution due to the fact that they’ve made commitments to their ain customers astir what a strategy volition and won’t do, and cipher tin presently warrant that.”

Using the wide adopted cybersecurity modular SOC 2 arsenic its muse, AIUC has developed a modular called AIUC-1 and a investigating work to validate agents against the standard.

To physique the standard, AIUC assembled a consortium of astir 250 information and hazard leaders — the buyers of agents. “These are the radical who we conscionable with connected a monthly basis, and the question we inquire them is: When you’re buying agents from someone, what would you look for? What are the questions you’d privation to ask, and what would you privation to spot addressed?” Dattani told TechCrunch.

That feedback shapes the tests. The startup past runs an cause done a suite of immoderate 5,000 tests to spot however it behaves successful scenarios involving jailbreaks, hallucinations, and information leaks. The results nutrient a astir 100-page study detailing wherever an cause performs safely and reliably — and wherever it doesn’t. Interestingly, AIUC uses AI agents to tally the tests and AI to analyse the data. Humans, however, verify the last audit, Kvist said.

If this sounds a spot familiar, it is. Dattani’s erstwhile leader METR, wherever helium was COO from 2024 to 2025 and remains a committee member, does akin investigating for the frontier labs, though its enactment until precocious has focused mostly connected show (whether agents tin reliably implicit tasks). METR was 1 of the autarkic probe orgs OpenAI utilized to analyse its Hugging Face incident.

Anthropic CEO Dario Amodei has besides precocious called for the AI manufacture to gait frontier development, citing a accelerated summation in bad-behavior incidents. In his post, Amodei floated the thought of requiring frontier labs to usage embedded third-party evaluators to observe and verify safety, and named METR arsenic 1 possibility.

While AIUC isn’t proposing to embed itself astatine lawsuit sites, the wide thought is similar: springiness enterprises an autarkic appraisal of however harmless their AI agents are. “Here’s wherever it passes and wherever you tin spot it. And here’s wherever there’s concerns. You should beryllium alert of those references earlier you marque the determination to buy,” Dattani said.