The Anatomy of the AI Mind

In partnership with

THEWHITEBOX

TLDR;

Today, we analyze the most exciting research I’ve ever read in AI, giving amazing insights into deciphering the mysterious mind of AI models, thanks to the leading AI Lab Anthropic.

You’ll not be disappointed by what you read.

We also examine new investment rounds, Google's striking new SOTA model, new computer agents, and more.

Enjoy!

THE I IN AI

Google is the New King

On Tuesday, I covered some eye-opening new research that exposed the intelligence issues of frontier AI models. The research tested the capacity of our frontier reasoning models to solve complex mathematical questions that required a perfect proof to be counted as correct; it wasn’t about getting to a final correct answer, but the reasoning process should be perfect, too.

This exposed the models’ limitations, as none surpassed the 2% mark, and with models like o1-pro costing $200 to run with the API, sound results were quite embarrassing.

Well, the same day the researchers published the results, Google launched Gemini 2.5 Pro, and yesterday the researchers published the results for Google. And well, they speak volumes: 24%, twelve times better than the best of the rest.

Insane.

Importantly, as mentioned, the model couldn’t have been trained on these tests as it was released the same day, so the results are legit.

Fascinatingly, it seems that the model did not rely on code to solve the tasks (most reasoning models right there reasoning on code in order to test it and iterate over the mistakes), making it even more impressive.

TheWhiteBox’s takeaway:

All signs are there. Google is the new SOTA, and this is hardly debatable. Not only because of this insane result, but also because:

They are 34 ELO points ahead of everyone else in LMArena, implying a win-rate over the next best model, GPT-4o, or 55% (beating it soundly in most tasks)

They are 8% ahead in the Aider polyglot benchmark, the most demanding and well-respected coding benchmark.

They are first or second in every SEAL ScaleAI benchmark, notably doubling the next best model in Humanity’s Last Exam.

Furthermore, it has the largest context window by far with 1 million tokens (~750,000 words), crucial for tasks like coding where repository context can be massive.

I use it with Cursor, and I have to say I prefer it over Claude 3.7 Sonnet because it doesn’t overthink that much and commits very strictly to solving my task. The former tends to change many things I didn’t ask for.

Ah! And we are talking about the Experimental version; the actual 2.5 Pro model has yet to be released. Google is so back.

But will that translate into real traction (aka money)?

AGENTS

AI Research Agents, Closer than we Think?

OpenAI has published new research. In it, they present a new benchmark for agents, PaperBench, which evaluates agents' capability to replicate top AI research from ICML 2024 (International Conference on Machine Learning).

The models are evaluated using rubrics co-designed by the researchers behind these papers…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.