OpenAI adds a prominent AI doomer to its board of directors

Paul Christiano, an influential AI researcher focused connected keeping AI systems aligned with quality interests and nether quality control, is joining the OpenAI Foundation board, the frontier laboratory said Wednesday.

“I present judge determination is simply a meaningful hazard that accelerated acceleration successful AI capabilities leads to catastrophic and irreversible nonaccomplishment of power successful the precise adjacent term,” Christiano wrote successful a societal media post. “I bash not deliberation that the AI manufacture successful general, including OpenAI, is presently connected way to trim this hazard to an acceptable level. I’m joining due to the fact that I judge that if OpenAI rises to the juncture we could importantly trim risk.”

Christiano wrote that utilizing AI models to bid consequent AI systems could effect successful an detonation of capabilities that their creators can’t control.

He joins the committee arsenic OpenAI faces renewed scrutiny implicit its information procedures, pursuing a bid of incidents successful which AI agents broke retired of restraints and penetrated extracurricular machine systems without the cognition of OpenAI’s researchers. On Tuesday, Anthropic researcher Jacob Coxon resigned his position to telephone attraction to what helium considers irresponsible AI improvement — and it seems to have worked.

Christiano volition articulation the board’s Safety and Security Committee, led by Carnegie Mellon University prof Zico Kolter. The committee has the last accidental connected whether OpenAI releases caller models, similar Astra, which was deployed past week. Kolter has not commented publically connected the caller information incidents. OpenAI has not responded to TechCrunch’s petition for Kolter’s position connected the company’s attack to information pursuing those incidents.

Christiano is 1 of the radical down reinforcement learning (RL) from quality feedback, a cardinal method for grooming ample connection models that helium developed portion moving astatine OpenAI. He near the laboratory successful 2021, subsequently founding the Alignment Research Center to absorption connected however to find if an AI exemplary could endanger its quality creators.

“We presently bid our AI agents with RL to get arsenic overmuch reward arsenic they can,” helium wrote Wednesday. “It has agelong seemed theoretically imaginable that this could motivate AI agents to undermine quality control, question powerfulness and resources, and screen up their tracks successful pursuit of misaligned goals correlated with reward. Public grounds from caller incidents suggests that this is not conscionable a theoretical possibility.”

Sometime successful 2024, Christiano became affiliated with the U.S. government’s AI Safety Institute, which aboriginal became the Center for AI Standards and Innovation. There, helium plays a relation successful the U.S. government’s largely hidden effort to measure frontier AI models earlier their release.

According to the frontier lab’s announcement, Christiano volition proceed advising the authorities portion serving successful his caller relation arsenic a committee member, but volition recuse himself from OpenAI matters and exemplary evaluations. However, that volition hardly quell wide concerns astir the AI industry’s power implicit policymaking.