The Week Everybody Started Talking About AI Doomsday

In this newsletter, we’ve been talking about the idea that this space would ask for regulation for years.

Until now, it was mostly a proxy war: lobbying state-level politicians to propose bills like SB-1047, which tried to impose rules that, back in 2024, would have made it almost impossible for open source to thrive.

But now, they are tired of doing proxy wars and are asking for it directly. Most of you know what my opinions are on this matter. However, today I’m going to focus strictly on the facts so you can decide for yourself.

I’ll review the current situation, revisit other events in history that were demonized as much as AI is today, and finally, I’ll provide an objective analysis of how likely efforts to regulate AI are to succeed independently of what each side thinks.

From woman-deforming bicycles to world-destroying machines, humans have had their fair share of extinction scares. Is AI a sham like the rest were, or the real deal?

Let’s dive in.

“Now I am Become Death, the Destroyer of Worlds”

That was what Oppenheimer allegedly said after the Trinity test, the first successful nuclear bomb test on July 16th 1945, three weeks before they dropped two nuclear bombs in Hiroshima and Nagasaki.

And now, based on the comments made by many AI researchers and echoed by Dario Amodei in his letter begging for tight regulation to prevent them and others in the AI race from destroying humanity, it seems these researchers are feeling the same way.

But before we discuss if they should, let’s first give a quick rundown in case you missed what happened.

Please stop me; I beg you

Days after the very controversial tweet from Jacob Coxon, who resigned from Anthropic because he felt the frontier labs were racing toward human extinction, with many inside Anthropic, including the alignment lead (foremost safety expert), Evan Hubinger, chiming in to support the young researcher, Dario has now spoken.

In this blog, he basically agreed and asked to pace the frontier through informal commitments and regulation. In other words, he’s asking to be forced to slow down.

Almost immediately, Sam Altman, his alleged sworn enemy, supported the idea, making it very clear this regulatory effort is a joint effort between both labs. Even Elon Musk, the latter's sworn enemy, agreed too.

The reasons are the usual ones: we are going too fast, and we know too little about these models to allow ourselves the privilege of moving too fast.

The fact that Dario is saying this isn’t surprising at all, but this is the first time he has tried to explain how we would do it. He divides his mental framework into three levels: embedded evaluators, democratic coordination, and global coordination.

The first is using third-party independent evaluators embedded in the organization, with a say in how safe a model is. The idea is to increase trust by having external evaluators chime in and, I assume, even preventing releases if they don’t think…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.