What's Next in AI?

In partnership with

FUTURE

What’s Next in AI?

Although this newsletter has been transitioning to more hands-on information, we mustn’t forget that theory and research are the foundation of AI.

So, realizing I haven’t given you guys an updated of what’s new in AI, I’ve spent two weeks going deep into research (ah, the old times) to answer:

What are the most exciting pieces of research right now? What’s up next?

Thus, today, we are examining the latest research from Google, Cohere, Apple, MIT, Mistral, NVIDIA, and more to determine what the incumbents are most excited about and what breakthroughs will matter in the coming months.

It’s an honest, hard look at what AI is (and isn’t) currently, and, by the end, you will be fully up-to-date with the industry on a deeper level hardly attained by almost anyone.

Specifically, we will:

understand current limitations,

learn how models work in first principles,

list the challenges faced by AI Labs and what they are doing about,

and, notably, describe what I believe the next generation of AI models will be.

Let’s dive in.

First, the Real Issues

To understand what’s troubling AI Labs and what they are doing about it, we must first acknowledge the issues models currently exhibit.

The Big Constraint

Models need context to work well. In fact, just today, an executive at Google claimed AGI would be built by whoever solves context and memory, not intelligence.

Put simply, if we could feed a model all the relevant context it needs to execute a task, it would likely do it well. With foundation models like ChatGPT, it isn’t about whether the model can do something, but instead whether it has the context to do so correctly.

So, why just not send them infinite context? The big problem with that is that they suffer from a compute and cache explosion due to the way they process data.

The reason models like ChatGPT work so well is that they have an inductive bias (an architecture choice) known as ‘global attention;’ when given the context, they pay attention to every single part of this context, no matter how large it is, which implies an enormous “cognitive effort” and, worse, a tendency to lose itself amongst the noise.

The reason we do this is to ensure the model has full context to predict the next word.

For example, if you want to talk with ChatGPT about a Sherlock Holmes book you’ve just sent to it, and you ask it, ‘What was the butler’s name?’ and that name is only mentioned once at the beginning of Chapter 2, way back in the context window, the only way the model can respond is if it scrutinizes every single word in the entire book.

More formally, they are stateless, meaning they don’t have a persistent, compressed memory; they simply reread everything all the time.

Put another way, if you give ChatGPT a million words of context, the model will scrutinize every single word, giving all of…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.