Is OpenAI o3 Really AGI?

FUTURE

The world may have changed, and we might not have realized it yet.

Yesterday, OpenAI shocked (and this is not hyperbole) everyone with the announcement of OpenAI o3 and o3-mini, the brand new models of the ‘o’ family (they skipped ‘o2’ due to trademark reasons).

o3 results are so astonishing that some people are actually convinced that it is AGI, as it destroys some of the so-called ‘impossible’ benchmarks for current models.

The importance of this announcement cannot be understated. If I’m being honest, it quite frankly outdates many of the AI breakthroughs we commented on… just weeks ago. And I promise this is not hype; even the biggest LLM skeptics, like François Chollet, are speechless.

Today, we are discussing everything (published and, more importantly, unpublished) on the historic announcement:

The incredible results obtained in some of the most challenging benchmarks (including those assembled by Fields Medal awardees, the Nobel Prize of mathematics), focusing primarily on what OpenAI has publicly shared,

We will then move on to the things OpenAI is less keen on talking about as, luckily, researchers with insider information have spilled the beans, especially regarding what makes o3 different from o1,

We will also explain what is OpenAI’s real moat and how this superhuman model represents a victory over one of AI’s hardest problems,

Of course, we will answer the real question: Is this AGI? Always in a non-hyped way,

And finally, I will force you to completely reframe your future around AI to ensure you aren’t left behind.

o3 opens a new chapter in our lives, so let’s ensure you are ready.

Let’s dive in!

Results That Simply Don’t Make Any Sense

For months, we all feared that the era of step function improvements between generations was over and that models were plateauing (nonetheless, since GPT-4, trained in 2022 and released in March 2023, new models only represented marginal improvements over former generations).

But these fears are definitely over with o3.

A New Coding Frontier

Some published results (the model has yet to be released) are so impressive that you wouldn’t believe them unless they were from a company like OpenAI.

For starters, the model appears to be on a completely new level regarding coding.

As seen below, the model surpasses o1’s might in both SWE-bench Verified, a benchmark that analyzes how good a model is at autonomously solving GitHub issues, and Codeforces, a platform that provides a space for programmers to solve challenging algorithmic problems and participate in contests. Top-tier programming challenges, so to speak.

Source: OpenAI

To ground the results, the model considered state-of-the-art for coding until today, Anthropic’s Claude 3.5 Sonnet, achieves 49% in the former.

For the latter, Mark Chen, OpenAI’s research lead, claims he has an ELO of around 2,500, smaller than O3. This is despite Mark being considered one of the best competitive programmers in the world. It's also higher than Jakub Pachoki’s score, OpenAI's Chief AI Scientist since Ilya Sutskever's departure.

These results have caused many software developers…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.