The AI Breakthrough We Might Regret

SAFETY

Why New Models Might Have Higher Risk than Reward.

The launch of OpenAI’s model Astra, the codename for GPT-6, is the first launch whose incredible results have been clearly obscured by the doom risks associated with it.

This model breaks records by a wide margin, and still, most of what you hear is fear about CoT monitorability, controllability, and existential risks.

I mostly avoid these discussions because they are full of extrapolations and esoteric arguments that lead nowhere, but this is the first time I feel like I can't avoid them. The evidence is simply too strong to dismiss as fearmongering, no matter how cynical I remain about the intentions behind some of the events we’re discussing today.

To do so, we are going to review the biggest AI-led cybersecurity incident ever witnessed, with thousands of AIs attacking an organization, to see what they are already capable of and how Astra-level models could only make this worse.

We’ll discuss how AI’s nature makes it dangerous, and how, for the first time in my time in this industry, I could see regulation as unavoidable. At the end, I’ll explain in detail the strategy I believe these labs will follow to achieve regulatory capture.

Let’s dive in.

AIs LOVE to Cheat

Here’s some news for you: the agents you use “like” to cheat when they are allowed. They “love” to misbehave. If you’re a regular of this newsletter, you already know and understand why, but if not, let me explain.

Training obsessive entities

AIs learn in two ways: imitation and reinforcement.

Imitation, as the name suggests, is having them imitate data, like teaching your kid to write by copying what you write. This is slow and requires an insurmountable amount of data.

Too-cool-to-avoid-jargon researchers call this ‘teacher-forcing’ because no matter what the AI predicts as the next word, the word that gets used for the second round is the ground-truth one.

The outcome is a model that has imitated so much data that it knows a lot about anything you ask. But you’ll agree that imitation is not the best way to make them “smart.” Giving them the answer to every problem isn’t the best way to teach them problem-solving skills.

For that, in the next training phase, we do the opposite. We hide the solution and give them a goal: “solve this problem.”

And thanks to a human equivalent of centuries of imitation learning, even if the model hasn’t solved a single problem by itself until now, it “knows” promising ways because it has seen them in the past.

It has not solved a single maths problem by itself before that, but it has seen countless solutions to maths problems, so there’s a higher-than-chance possibility that the model suggests the correct approach.

From here on, we run what’s basically trial-and-error problems at scale, and whenever the AI happens to use the correct approach (or not, as we’ll see later) to solve a problem and reaches the correct answer, we reinforce…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.