[Premium] Liquid LLMs, Chip Crash, & More

TLDR;

Technology:

🤯 Linearizing LLMs To Make Inference Viable Works Amazingly Well

😎 Unleashing System 2 Thinking in o1 Models

Markets:

🫣 The Signal ASML’s Crash Sends to Markets

🤨 Hyperscalers are Eating the World

Product:

🫡 Liquid Foundation Models Are Finally Here

🥵 xAI Launches Grok API. It’s Pricey

FRONTIER RESEARCH

Linearizing LLMs To Make Inference Viable Works Amazingly Well

Hazy Research is a group of Stanford researchers, including some of the most prominent researchers in AI optimization. These include, among others, Tri Dao, co-founder of TogetherAI and creator of Flash Attention, or Albert Gu, co-founder of CartesianAI and, jointly with the former, creators of Mamba, an architecture that many consider Transformer’s biggest rival (most large language models today run on an architecture named ‘Transformer’).

But if Transformers are so incredible, why would one want to change them? Simple: They are notoriously inefficient to run. Their biggest issue is non-compressed memory, which means they require the entire past to be available to work correctly.

In layman’s terms, unlike humans, who compress memory (we recall only our most relevant past experiences and forget the rest), Transformers act as if humans could remember every single thing that happened in the past.

That sounds awesome, but in practice, it means that the compute requirements (how much compute is required to run the LLMs) are not only enormous but grow quadratic to sequence length (i.e. if the text input sequence doubles, compute quadruples, if it triples, compute increases nine-fold), but also grow linearly with sequence length memory-wise (doubling a sequence doubles the memory requirements, as we saw in our internal Notion piece on how to calculate your GPU requirements to run LLMs).

As discussed in detail recently in this newsletter, this is the most significant engineering problem in the AI industry.

Now, the group above has presented their work on linearized attention. And it's amazing.

TheWhiteBox’s take:

Although I will publish much more detail in a Notion article, the idea is that we can match the performance of quadratic LLMs with subquadratic counterparts by taking a standard Transformer and ‘linearizing’ it.

In other words, substitute the quadratic attention layers with linear attention ones, train the latter to match the output of the former, and swap them. In other words, we train layers that do not require quadratic compute requirements to ‘act’ like the standard ones, a best-of-both-worlds approach.

It’s still unclear whether this will hold on at scale, but the first results look amazing. They train linearized Llama 7B, 70B, and 405B that match the performance of the original models in popular benchmarks while having a much-desired inference optimization, as depicted in the above image (the linearized models are the ‘LoLCAT’ reference).

The only issue is that, as we are teaching an optimized model to match a sub-optimized yet more expressive model, we first need to have the pre-trained quadratic Transformer nonetheless, as linearization is a post-training method, which means this is just to improve inference, not training.

And the best thing of…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.