AI Washing Crisis, Semis Ripping, Anthropic's Dreaming, & More

TLDR;

Hey there! This week, we have another claim from a new lab that has, allegedly, found AI’s “Holy Grail”, Anthropic’s super interesting and meaningful dreaming feature, why AI Washing is no longer in vogue, why semiconductor stocks are all doubling in value this past month, a new string of very powerful OpenAI releases, and a textbook example of what I believe is the future of enterprise AI.

Enjoy!

RESEARCH

Have we finally found the Holy Grail?

If you ask AI researchers what the holy grail of Large Language Models (LLMs) is, it’s probably subquadratic attention that actually scales.

I know, if you aren’t an AI geek like me, that sounded like gibberish, but that’s precisely why you’re reading this newsletter, because I can explain this stuff in simple terms.

I went into excruciating detail here recently, but to simplify, at the heart of every AI model you’ve used lately sits a component known as the Transformer, which does two things: processes the sequence (how words in the input interact with each other, like picking to which noun an adjective refers), and adds internal model knowledge to enrich the context.

That is, a prediction is made by considering both the input context and the model's prior knowledge of the topic. The issue is that the sequence-level processing, which we call ‘attention’, is extremely computationally intensive.

As every word has to talk to every other word, it requires a lot of computation and grows quadratically with sequence length (i.e., if the sequence doubles in length, computational requirements quadruple, if the sequence triples, computation nine-folds).

Say you have the sequence “Everyone feared the Ottoman red-bearded pirate”. The word “pirate” will attend to all previous words and decide which ones carry more valuable information, and enrich its meaning with those.

For example, it may attend to “red-bearded” and build the concept ‘red pirate’, which ‘evolves’ the meaning of pirate from a standard one to one that had a red beard.

With that context, the LLM might induce that we’re talking about Barbarossa, which it has read a lot about, and start adding information from its internal knowledge about that particular pirate to enrich the prediction.

That is why LLMs can talk about things that aren’t in the context; they can retrieve them from their own memory.

To build such intuitions, every single word in the sequence goes through this “What should I attend to?” process, explaining why this gets very expensive very fast as sequences grow.

Therefore, finding a way to minimize the computational requirements of this procedure has long been the most coveted algorithmic breakthrough in AI.

Much like humans, we would expect there to be a way for AIs to conveniently decide which parts to attend to, thereby considerably compressing computational requirements. Now, a new company thinks they got it.

They are promising numbers that feel absurdly good to be true: a 12-million-context window (meaning their models can handle between 12 and 72 times more context at any given time than…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.