Anti-Hype Guide, Part 1. The Era of Experience

LEADERS

The Anti-Hype Guide for 2025

In the last few weeks, two of the most influential people of our time, Rich Sutton, one of AI’s Godfathers, and Andrej Karpathy, considered by many (yours truly included) as the clearest thinker in the AI space, have come out with quite the anti-hype takes regarding modern AI.

Coupled with the general sentiment of AI being in a bubble, this has caused serious turmoil, even if both are generally optimistic about AI and its value to society.

Take today’s article as a way to:

Intuitively understand what AI experts who aren’t looking to sell you something think AI’s limitations are

Learn about recent research that they point out as a potential way out of these issues.

And ironically, how their somewhat bearish views about AI could in fact justify the lavish spending we are seeing even more, growing the AI bubble larger.

In an industry where every article, every blog, and every announcement is designed to sell you something, take today’s piece as a way to see the AI industry in a more straightforward, unbiased way, a cure from this ‘hype illness’ all of us, me included, suffer from.

Let’s dive in.

AI’s Big Limitations: The Sutton View

Rich Sutton's interview in the Dwarkesh Podcast is a very, very esoteric conversation; truly not meant for someone who isn’t deeply in the weeds of AI.

But we can make it easy to understand. And we must, because there’s a lot that can be understood about this industry based solely on Sutton’s views.

Where he’s coming from

First and foremost, Rich Sutton is a legendary figure; he’s the author of what is possibly the most critical essay in the history of this industry, at least in terms of money thrown into the ideas shared. In it, he argues that the best AI algorithms are those that unlock more compute and data at scale.

It’s called the bitter lesson because it proves that human intervention (human data, human heuristics, etc.) that we use to train models actually works worse compared to a simpler algorithm that simply allows more compute thrown into the problem.

He also happens to be one of the fathers of Reinforcement Learning (RL), a way of training AIs extremely popular these days that is similar to how you would train your dog: reward them whenever they make something desirable, “punish” them when they make mistakes.

Over time, this training system allows the dog to reinforce—hence, the name—actions more likely to yield the treat they most desperately want, and will avoid the actions that make their owner mad. Interestingly, this is literally the same approach we use with AIs today.

But what is the ‘dog’ in AI’s case?

To use RL effectively, you need a prior, something from which we can sample actions that aren’t purely random.

For example, no matter how hard you try to reward good actions, you can’t train a six-month-old baby to calculate a second-order derivative; you need your prior (i.e.,…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.