The Whale Strikes Again

In partnership with

THEWHITEBOX

TLDR;

This week, we discuss the latest AI drama, OpenAI’s cheating accusations on xAI, the extremely creepy advent of the first musculoskeletal robot, and an amazing example of agents working together to create a web app from scratch.

Moving on, we discuss the birth of a new AI super lab and Crunchbase’s AI startup success AI predictor. We then move to discuss China’s latest AI model by Kimi AI and two breakthroughs in AI biology and research, the largest foundation model for biology ever, and Google’s AI Co-scientist.

Finally, for this week’s trend of the week, we discuss DeepSeek’s latest research and how they intend to turn the world of AI upside down.

FRONTIER RESEARCH

The OpenAI vs. xAI Drama

Model performance on Math-olympian level maths problems. Source: xAI

On Monday, xAI released their Grok-3 and Grok-3 Reasoning model families, which we covered in detail in Tuesday’s Premium-only news segment. Allegedly, these models were superior to OpenAI’s entire offering, making Elon Musk’s claim to have built the “smartest AI on Earth” valid.

But it seems that we might have been misled, leading to the latest drama in the AI industry, after the former accused the latter of dishonesty or, in the words of OpenAI’s Head of Applied Research, just straight cheating.

The reason is the numerous charts, like the one shown above, that xAI used to prove Grok-3’s frontier-level capabilities. At first glance, if you look at the graph, you simply assume that the darker blue represents Grok-3’s performance without reasoning capabilities, and the lighter blue represents the results when reasoning capabilities are added. Everyone, including OpenAI employees and me, interprets it this way because it seems evident without further information and assumes xAI is being honest.

Under this assumption, Grok-3 Reasoning appears to be way ahead of o1 and also ahead of o3-mini. However, in the release blog, xAI clarifies that the lighter blue part is the result after ‘cons@64.’ And that, my dear reader, changes things a bit.

What you’re looking at above is the following:

The darker blue already refers to Grok-3 Reasoning, but on a pass@1 metric. In other words, it shows Grok-3 Reasoning’s performance when measuring whether it correctly solved the problem in one try.

The lighter blue refers to Grok-3 Reasoning’s performance on ‘cons@64,’ which translates to ‘consensus at 64 tries.’ In layman’s terms, it shows the performance when the consensus response (the most common) across 64 tries is correct. For instance, if the accuracy is 93%, it means that 93 out of 100 problems in the benchmark, the consensus response in every problem’s 64 tries happens to be the correct answer.

This wouldn’t be a significant issue if it weren’t for the fact that the results xAI shows in the graph for o3-mini are not ‘cons@64,’ but pass@1. Therefore, in an apples-to-apples comparison, Grok-3 Reasoning gets less than 80% while o3-mini gets 87%, clearly proving that, benchmark-wise, o3-mini is better. However, looking at the benchmark makes you believe that…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.