Gambling, Small Models, & False Alarm

In partnership with

THEWHITEBOX

TLDR;

I’ve been on a short holiday for a few days, so I haven’t had the time to write a thoroughly researched Leaders article for this week. Instead, as a one-time thing, I’m giving you another Premium news rundown. It’s packed with insight!

Today, we take a good look at the always-forgotten small models, which, once the industry realizes, will play a vital role in adoption. We also take a look at Google’s insane new image models, Anthropic’s free academy, and interesting market data on Intel, Cohere, and prediction markets, among other relevant news.

Enjoy!

HRMs

It was, in fact, too good to be true

A couple of weeks ago, a group of Chinese researchers presented a ground-breaking result. With just 27 million parameters, 24k times smaller than DeepSeek R1, a model managed to beat this model and other frontier models on the ARC-AGI benchmarks, a highly popular choice to compare top models in reasoning tasks.

The breakthrough was a new hierarchical architecture combining two modules:

High-level module, in charge of things like planning and high-level actions (i.e., pick that cup)

Low-level module, in charge of the low-level actions (i.e., move arm in this specific set of angles to reach the cup)

The idea was inspired by the brain, drawing it closer to ‘human intelligence’, at least on paper. The news went so viral that it forced the ARC AGI team to reproduce the results and check whether the claims were factual, and, well, they were indeed too good to be true.

→ READ THE OFFICIAL BLOG FOR MORE DETAILS

Although the results were legit and reproducible (the reproductions gave slightly lower results, but still validated the claims), the biggest letdown (a great disappointment actually) is that the architecture is not the reason the model worked so well, as comparing it to a normal Transformer (a ChatGPT-style model the size of the original HRM) yields almost identical results.

Instead, the model's surprising results can be attributed to the use of three elements known for years to be good reasoning priors: refinement, memorization, and data augmentation.

The HRM includes a prediction refinement loop, but a Transformer equipped with the same loop reaches similar results.

Through data ablations, the validating team showed that a great deal of the performance could be explained via training only on the samples most similar to the ones being tested on, suggesting strong memorization dependency.

Data augmentations also worked well, but this is nothing new; it has been a textbook method known for decades in this industry.

In a nutshell, we just found a “new” way of getting the exact same conclusions as always.

TheWhiteBox’s takeaway:

This news has a good thing about it, in that it might reignite the interest in small models, which are often overlooked by the AI hype train just for not “being that smart.”

But as I explain in the Liquid AI news in the Product section below, they are extremely important for the future of an industry…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.