Mamba, the ChatGPT Killer & The Great Deflation

🏝 TheTechOasis 🏝

Breaking down the most advanced AI systems in the world to prepare you for your future.

5-minute weekly reads.

TLDR:

AI Research of the Week: Mamba, the ChatGPT Killer

Leaders: The Great Deflation

🎅 AI Research of the week 🎅

Two researchers have made the boldest move in years: throwing the biggest algorithmic breakthrough of the 21st century out the window.

Named Mamba, it achieves what was once thought impossible: matching or beating the Transformer, the architecture behind ChatGPT, while being faster and a lot cheaper.

Everyone seems to be talking about it, so let’s uncover what Mamba is.

The Gift that Keeps on Giving

Since its release in 2017, the Transformer architecture has become the ‘de facto’ choice for natural language modeling (models that generate text).

ChatGPT, Gemini, and Claude are based on this seminal architecture as its parallelizing capabilities, ideal for GPUs, sent recurrent networks, the previous state-of-the-art, into oblivion.

The intrusiveness of this architecture is such that the ‘T’ in ChatGPT stands for ‘Transformer’.

But what is the Transformer?

In very simple terms, it makes words ‘talk’ to uncover the relationships between them. This is known as the attention mechanism.

For instance, the attention mechanism over a pronoun will lead it to pay attention to its corresponding noun.

Works wonderfully, but at a huge cost.

Mastering inefficiency

Put simply, the Transformer yields impressive performance at the cost of being extremely inefficient.

To understand how models like ChatGPT process language, imagine you’re halfway reading a Harry Potter book.

However, to successfully read the next page, you have to store and review all the pages read previously in your mind.

Thus, to read the next chapter, you also have to remember every other detail read until now, be it relevant or not.

That’s how Transformers like ChatGPT process language.

Seems very inefficient, right? Yes, because it is.

Humans do remember many things read until that point, but we forget irrelevant data like Hermione’s summer activities. They are not relevant to the storyline, so our mind forgets about it.

What I am trying to tell you is that Transformers don’t compress context, they store it as a whole.

Naturally, as context isn’t compressed, the computation and, above all, memory requirements of Transformers grow considerably as the text sequence grows larger.

Specifically, quadratic and linear complexity for training and inference. In other words, in training, doubling the sequence quadruples the cost, and in inference, doubles it.

So what do research labs do?

To avoid costs spiraling out of control, they limit the ‘workspace’ of the model to a context window, which is why ChatGPT has a limited size of text they can process.

We also limit the context window because Transformer performance decreases over sequences longer the training context window.

This is called the extrapolation problem.

Over the years, many different architectures with more ‘efficient’ attention mechanisms have been proposed, but the loss of performance has prevented them from really substituting the vanilla Transformer.

So what did Mamba’s…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.