JAMBA, RAG 2.0, DeepSeek Experts, and The Key to GPT-5

🏝 TheTechOasis 🏝

part of the:

Breaking down the most advanced AI systems in the world to prepare you for the future.

5-minute weekly reads.

TLDR:

AI Research of the Week:JAMBA, The First Production-Grade Hybrid

RAG 2.0, The Death of Standard RAG?

DeepSeek, Hyperspecialization of Experts

Leaders: Are Agentic Workflows the Path to GPT-5?

🔋 Jamba & The Hybrid Forthcoming 🔋

For almost six years, nothing has beaten the Transformer, the heart of all Generative AI models.

However, due to its excessive costs, many have tried to dethrone it but failed.

But we can finally hear the winds of change. Not to substitute the Transformer, but to create hybrids. And we finally have our very first production-grade one, Jamba.

A Quadratic Nightmare

In technology, there’s always a trade-off. And in the case of the Transformer, it’s a big one.

And although we aren’t going into the technical details of the Transformer for the sake of the length and because we have described them many times, here’s the gist.

Models like ChatGPT, Gemini, or Claude are all based on a concatenation of Transformer blocks.

Transformer block. Source

Each of these blocks contains two things:

An attention operator

A Feedforward layer (MLP in the image)

The former enforces the renowned attention mechanism, a mixing operation that processes the input sequence and helps each word in the sequence pay attention to the words that matter (verbs into nouns, adverbs to verbs, pronouns to nouns, and so on).

For a full review of attention, read my blog here.

The latter is a feedforward layer, which helps in extracting key features and relationships from the data.

But as attention has a problem, it has quadratic cost complexity. In layman’s terms, if the input sequence is doubled, the cost of processing it quadruples.

In practice, for short sequences, this isn’t much of a problem, but for longer sequences, the computation and memory requirements both skyrocket.

For reference, if we look at popular models like LLaMa-2 7B or Mistral-7B, despite their small size, for a sequence of 256k tokens (192k words) they require 128GB and 32GB respectively, despite the latter using Grouped-Query Attention.

And this is just one single input.

For this reason, for years, the Transformer has been incapable of scaling to larger sequence lengths due to its prohibitive costs.

And despite some newsworthy innovations like Ring Attention facilitating things by reducing communication overhead between GPUs, it still doesn’t solve the ‘quadratic barrier’.

And this takes us to Jamba.

Mamba, Attention, and MoE

Jamba, an open-source model created by AI21 labs, is an LLM that looks too good to be true.

Instead of just using Transformer blocks, it includes another piece in the puzzle, Mamba blocks.

But why?

Mamba is stateful. While the Transformer has to consider the entire context every single time, Mamba has memory.

Imagine you are writing a book, and you are writing page 101. But for every single new word you write, you have to reread all the previous pages every time…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.