Is AI Progress Really Slowing?

In partnership with

FUTURE

Is AI Really Slowing Down?

The entire industry seems to be in the ‘acknowledgment phase’ that AI progress is slowing. However… is it though?

On the one hand, some are continuing the pompous claims, fearing that investors might call it a day. Conversely, LLM skeptics are taking self-awarded victory laps, boasting how they were right all along.

But who is actually correct?

The key to that question is none other than the emergence of a new paradigm, one that not even OpenAI predicted or dared to implement yet, which has seen an explosion in interest and hype over the last months. The success or failure of this new proposal feels like the very last bullet AI has to prevent all hype around it from crumbling down.

Today, we will introduce ourselves to the most significant shift AI has made in years, describing this ‘next big thing’ with a level of detail and intuition you will hardly find elsewhere.

Damning Evidence

First and foremost, it’s not only AI that seems to be in a state of poor progress; it’s the scientific community overall.

A lot of Research is Bullshit.

As illustrated by Sabine Hossenfelder, most academic papers today are bullshit.

This is extremely weird, considering that being a researcher has never been more glamorous. Concerning AI, the trend is faster than anywhere else, as shown below:

Source

But is this exponential increase in research publications being met by an exponential increase in breakthroughs?

It's not even close. Most of this new research does not entail breakthroughs but marginal improvements over decades-old discoveries. Nonetheless, most of the research that led to our frontier AIs is old.

For instance, ChatGPT’s prowess is based on the following:

MultiLayer Perceptrons, invented back in 1958

Self-attention, invented back in 2014

Residual connections, another crucial element, were first presented in 2015

The optimization algorithm used to adapt the weights of the model during learning, AdamW, was published by now Anthropic researcher Durk Kingma in 2014

And the Transformer, the paper Google published that pieced all together, was published 7 years ago.

See what I mean? We have been riding the wave of breakthroughs that are generally around ten years old or more and, in some cases, older than humankind’s first moon landing.

For instance, the Llama 3.1 models are almost identical to the research published by Google in 2017, except for minor adaptations.

But if you think OpenAI’s o1 models are any different, think again, as the pattern continues.

Monte Carlo Tree Search (MCTS), a fundamental piece behind not only o1 models but also AlphaGo and AlphaZero, two of the most important AI models in history, is based on the Monte Carlo method discovered in 1946 in Los Alamos by Stanislaw Ulam during the process of building the Hydrogen Bomb, and first used combined with Neural networks to create the MCTS method we still use today in 2006, already 18 years ago.

In other words, despite the boom in interest in AI, no…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.