Models Misbehaving, Cerebras' Expansion, & Adoption Data

TLDR;

Today, we look at OpenAI’s “alarming” data about model misbehavior, China’s new bet on hybrid architectures, the Manus sensation, and Poe’s surprising results on model adoption.

We then move into markets to discuss Microsoft and OpenAI’s weird relationship and Cerebras’ infrastructure expansion and core value proposition.

Some of the things you’ll learn today include:

What reward hacking is.

What makes Cerebras’ hardware different from a GPU

The reality of modal adoption

What are state-space models and why do they matter

AI SAFETY

Reasoning Models Misbehave.

Apart from their just-released agents API (more on Thursday), OpenAI has published a blog post showcasing real examples of situations where the models misbehave, like hacking or modifying the problem at hand to achieve goals it’s consistently failing at.

This misbehavior has prompted OpenAI to develop internal tools to monitor the chains of thought—essentially a sequence of thoughts that breaks problems into multi-step problem-solving, which is crucial for complex tasks—that these models generate during their responses to detect this dangerous behavior.

In one example, the model implements a solution that it openly acknowledges as “unnatural, but might pass the test,” clarifying their inclination to do what’s necessary to meet the goal. This can make already hard-to-trust systems unusable if this trend continues.

TheWhiteBox’s takeaway:

This might feel like a unique new problem, where AIs might be developing consciousness or something along those lines. Bullshit. If you know a thing or two about Reinforcement Learning (RL), this behavior, also known as ‘reward hacking,’ isn’t new, let alone unexpected; it’s a decades-old problem.

Reasoning models are large language models that have undergone RL training. In this process, the model is given a goal and a set of constraints, and it must explore and exploit its way to solving the problem. In this training phase, the model learns not by correctly predicting the next word in a sequence but by producing a correct final response.

It isn’t about the process but the result.

To help it navigate the environment, we reward and punish good and bad actions accordingly, framing this as a problem of reward maximization. This means that these models may become obsessed with achieving high rewards and the final goal by any means possible, even luring them into finding hacks that take them to the goal even if the solution is not plausible.

In this reward maximization situation, if goals become impossible to attain, these models can even develop deceiving or cheating methods to achieve them, which is precisely what OpenAI is outlining.

For these reasons, OpenAI makes three arguments to justify its approach to dealing with this misbehavior:

Chains of thought should remain unrestricted and closely monitored

They should remain hidden from the user in unrestricted settings

And they should stay in natural language to make them understandable to humans.

The third point is a clear shot at companies like Meta, which are exploring latent thoughts via the COCONUT architecture, which we covered in detail in this newsletter a while ago. In that paradigm,…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.