AI's Elephant in the Room: Can We Run AI?

TLDR;

Welcome to TheWhiteBox, the place to get the recap of the most important things taking place in AI recently.

Today, we have a newsletter packed with insights: new data on o3’s real intelligence, Google’s secret to AGI, OpenAI buying Chrome?, SK Hynix’s quarterly earnings, progress on home-cleaning robots, and sleep-time compute - a solution to the elephant in the room of AI: compute affordability.

Enjoy!

FRONTIER INGELLIGENCE

ARC-AGI publishes o3 results

As you probably know, all AI Labs, from OpenAI to Anthropic, have one thing in mind: Artificial General Intelligence, or AGI, an AI that will be capable of performing most economically valuable work.

And the benchmark to test whether we are making progress in that direction is the ARC-AGI

The researchers behind the ARC-AGI challenge have published the results for o3 and o4-mini, both in the original benchmark (which is an earlier version of OpenAI’s o3 saturated—solved) and in the new one, (born as a result of that saturation to test new capabilities).

The o3 model OpenAI released last week performs worse (53%) than its December version, which had access to a much larger amount of test-time compute (allowing the model to think for longer on the task).

Unsurprisingly, both o3 and o4-mini were unable to produce meaningful results in the second benchmark (ARC-AGI-2), achieving below 3%.

In a later blog, researchers explain why they did not disclose the results obtained by o3-high and o4-mini-high, the thresholds with the highest compute capacity, which had better preliminary results across the board. The issue was that these models were unable to answer most of the tasks. As it seems, they enter a rabbit hole of compute they can’t get out of.

This is AGI, said Tyler Cowen. Really?

TheWhiteBox’s takeaway:

Don’t get me wrong, as soon as you interact with o3, you realize the model is the best AI you’ve ever talked to; it just is. The comments are incredibly nuanced; it feels like you are talking to an expert in almost any field you can fathom.

That said, I’m adamant that this is just a very well-executed illusion of intelligence, a well-executed “speaking database,” if I may. But in the key areas that define human intelligence, the precise ones being tested here, like adaptability to new problems and key pattern abstraction (the type of stuff you would be evaluated in if you took an IQ test), you easily start to see the cracks, and performance comes crashing down.

But why does this happen?

How can a ‘PhD-level’ intelligence model (as proclaimed by some), which can give you a top-level report on quantum computing, can’t solve a simple task such as the one below, which a young kid or early teenager could solve easily?

The answer is that we make a mistake by evaluating intelligence as a task complexity problem rather than a task familiarity one. In layman’s terms, we evaluate a model’s intelligence for how complex the hardest problem they can solve instead of how hard the most challenging…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.