Proof or Bluff, The Question Is...

TLDR;

This week, we see new proof of the limitations of current frontier models and OpenAI’s U-turn back to open-source. We also discuss Coreweave’s disappointing IPO, how companies continue to fail at adopting AI, and Elon Musk’s self-acquisition move.

Finally, we examine the best model available today and discuss LumaAI’s new and exciting motion control video generation AI.

Enjoy!

INTELLIGENCE

Proof or Bluff? Reality Check for LLMs

If my last newsletter was full of excitement and optimistic views (which I do stand by), it’s always lovely to ground ourselves back into reality every once in a while.

LLMs are becoming more natively multimodal, as we saw with GPT-4o image generation, increasing their “cross-modal intelligence” when evaluated as being able to “reason” across many modalities, but that doesn’t mean they are getting smarter in the areas where they are most limited.

In fact, we are pretty much stuck in that regard. A group of researchers wanted to test whether LLMs are actually becoming powerful math reasoners. And based on results, the answer is a hard no, we are still very limited in some parts of the “intelligence stack”.

The models were tasked with solving USAMO mathematical tests, which, unlike the more commonly tested AIME, require correctness across the entire reasoning process, not just the final correct answer.

And when you do so, the results are the ones you can observe above, a measly 2% as the top score, and with some models, like o1-pro, having outrageous costs over $203.44 to get just a score of 1.2%.

I can’t insist enough on the importance that costs are to intelligence progress. Nobody will use a “super smart” AI no one can afford. Additionally, being smarter per unit of power is by definition being more intelligent, so it’s also a great way to see which models are really smarter rather than simply more compute-intensive.

Among the many reasons behind these pyrrhic results, researchers point to several concerning issues:

Complete lack of exploration (lack of creativity). Remember last week we talked about intelligence as a combination of intuition + exploration, in other words, defining intelligence as having good suggestions (ideas) on solving a problem and having the capacity to explore different resolution paths? Well, reasoning models seem to be extremely bad explorers (meaning they don’t actually reason, or mildly), banging their head against the wall using the same approach every single time (only Gemini Flash-Thinking showed a higher degree of exploration).

They consistently made wrong assumptions.

Worst of all, they seem to be convinced that a problem was solved when, in fact, it wasn’t. This is a stark difference compared to human experts, who may or may not solve a task but know when it has been solved.

Can we call this a reality check? I think we can.

TheWhiteBox’s takeaway:

With AI, we must always strive to balance hype and optimism with a healthy dose of reality. Yes, models are still progressing, but no, they are by no means ‘as smart as PhDs’ or…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.