Microsoft just changed the game with LongNet

🏝 TheTechOasis 🏝

According to famous AI researcher and YouTuber David Shapiro, a human will read between 1 and 2 billion tokens in his/her lifetime.

And now Microsoft’s newest architecture, LongNet, ingests 1 billion tokens, a human’s lifetime read count, in half a second.

In contrast, the most advanced AI chatbot in the world in terms of length, Claude, ‘only’ reaches 100,000 tokens, or around 75,000 words, a complete Harry Potter book.

That’s still a huge number, but it’s 10,000 times less than LongNet.

With LongNet, we enter the realm of the possibility of models that could potentially ingest the complete Internet at once, taking us closer to humanity’s greatest achievement, Artificial General Intelligence, or AGI.

But how on Earth has Microsoft built such a crazy model?

Attention is all you need… and hate

At the core of every AI chatbot in existence today, from ChatGPT to last week’s newsletter protagonist Pi, we have the Transformer architecture, what’s probably the greatest discovery in the history of AI.

And at the center of the Transformer we have the attention mechanism, the key to unlocking a machine’s capacity to understand context in text.

In simple terms, attention works by making all words in the given sequence of text ‘talk’ with each other, allowing the model to understand the relationships between them and, thus, understand the meaning of the text.

This works wonders but has a problem, it’s super computationally expensive.

Specifically, the attention mechanism used by today’s model has a quadratic relationship between text sequence length and computational cost.

In other words, if we double the length of the input text sequence, the costs to run the chatbot quadruple.

Consequently, AI companies building LLMs have no option but to limit the input sequence maximum size.

For instance, in ChatGPT’s case is 32k tokens, or 26k words.

This is a huge limitation, as humans are better suited to answer the more context they’re given.

And machines are no different, as naturally, it will be easier for a chatbot to summarize a book if you provide the complete transcript than if you provide only a chapter.

But more importantly, having a vast input sequence capacity is critical to another feature of LLMs, in-context learning.

The richer the input, the better the outcome

When answering any question they are asked, unless provided in the prompt, chatbots will trust the knowledge embedded in their weights, achieved during training.

This is not ideal, as this corpus will include very valuable data, but also false data.

Luckily, one of the best features of LLMs is their few-shot capacity, or as I like to call it, “learning on the go”.

In layman’s terms, you can provide the information you need the chatbot to use in the actual prompt, and the model will successfully use that data to respond.

Needless to say, a great number of situations where AI chatbots become particularly useful involve this provision of information to the model in real-time.

Up until LongNet, providing very long texts…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.