An Impossible Result, R1, & More

TLDR;

This week’s big revelation? Models trained on incorrect data showed significant mathematical gains. We’ll explain how this is even possible and what it reveals about the future of modern AI.

Meanwhile, Berkeley’s Intuitor project flips the script by training models using their own certainty as feedback, pointing to a new path for learning in fuzzy, non-verifiable domains like writing.

Also, DeepSeek R1 emerges as a serious contender to OpenAI’s o3 in performance (and at a fraction of the cost), while Anthropic open-sources a powerful interpretability tool that might be the key to defusing dangerous behaviors before they emerge.

And finally, as AI token demand explodes, we once again ask: where’s the revenue?

FRONTIER RESEARCH

Training on Incorrect Data… Works?

A group of researchers has obtained an 'impossible' result in AI: a model has seen massive gains in maths accuracy (+28%) by training on... incorrect data. But how? This finding reveals a great deal about the reality of frontier AI. Modern AI models have reached new heights in "intelligence" by being trained on verifiable data, where we can check if the model's answer is correct or wrong, serving as a good learning signal and giving us models like Gemini 2.5, o3, or Claude 4 that, in some instances, match or outperform the best of the best among us (in constrained settings, though). But, unlike incumbents, will tell you, are these models actually learning new reasoning capabilities? Well, the answer is no, as this 'new' training method, which is generating so much hype in the industry, is just amplifying certain behaviors that only emerge if the model already knows them (we have much more evidence of this, by the way). To prove this, researchers trained models on incorrect rewards but realized that only some of them benefited. Alibaba's Qwen models did improve performance, but Llama (Meta) and Olmo (AI2) didn't. But how is that possible? The reason is that Qwen models are more inclined to generate math and code data by default (due to Alibaba's extra focus on maths and coding during training). So, even if the training signal is weak (wrong or incomplete), it is still sufficient to "bias the model's response distribution more toward maths and code," meaning it still incentivizes models to generate "math/code-esque" outputs that lead to improved performance. Think of this using this stupid example: If I tell you that 1+1 = 5, obviously wrong, it still forces you to recall how addition works, helping you improve at it, yet you're still smart enough to know 1+1 does not equal 5 not to learn 'wrong addition'; the key is that weak math training still helps you, but only if you know how to add in the first place! What this means for you and me is that this training method isn't teaching the model anything new; it's just making pre-learned behaviors more likely. Put another way, reasoning models (o3) are not better mathematicians than their non-reasoning counterparts (GPT-4o); they are simply much more likely to generate…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.