Meta might have created a new AI monster

🏝 TheTechOasis 🏝

🤖 This Week’s AI Insight 🤖

Nothing in this world is perfect.

ChatGPT, Claude, Bard… all have one thing in common.

They can’t frickin’ scale.

However, Meta (Facebook) is determined to end this with MegaByte, a new Transformer model that could put all the aforementioned models to shame by being capable of handling word counts not in the low thousands like ChatGPT…

But in the millions.

It’s All Fun and Games Until You See the Bill

If you’re an avid user of Generative AI technology like I am, ChatGPT has become a fundamental part of your tech stack.

I speedrun my study of AI research papers with dozens of papers produced every week thanks to the AskYourPDF plugin, I quickly generate awesome headlines that convey the message I want, and I plan my travels to the smallest detail with the Expedia and GetYourGuide plugins.

To me, this is what peak consumer-end technology feels like.

But there’s a problem.

GPT-4 not only sits behind a paywall of $20 but it’s also hard-capped to 25 messages every three hours.

This undeniably affects the user experience and OpenAI knows this, but it is what it is… as OpenAI is spending millions per day in running GPT-4 for us.

This has severely crippled the experience in two ways:

Inference (model execution) limits like the one I’ve just mentioned

Severely limited sequence lengths

And while one can live with limiting the number of inferences you can run on premium models, the second limitation is a huge problem.

And that’s precisely what Meta’s MegaByte is solving.

The Great Computation Problem

Transformers are some of the most expensive technology to run in the world.

For reference, LLaMa, a 62 billion-parameter model by Meta had a cost of $5 million just in training, alone.

Imagine the costs of training GPT-4, a model that could easily be 10 times larger than that, or more.

And the reason these models are so expensive is due to their sheer size.

Also, the fact that the entire model is run for every token (groups of 3 to 4 letters) in the input sequence, means that the computational costs of running the model are quadratic to the input length of the sequence.

In other words, the longer the sequence of words you send to the model, the increasingly expensive it becomes to run it.

Naturally, they’ve resolved to limit the amount of text you can send to these models.

Long books, podcasts, or videos are simply out of the question due to their sheer size.

Well… that was before MegaByte.

A Genius “Compromise”

As you may imagine by now, MegaByte was designed with one sole objective, to allow long-sequence modeling, prohibitive to models like ChatGPT.

Incredibly, Meta claims it can handle 1 million tokens, or around 750,000 words.

For reference, ChatGPT allows, at the time of writing, approximately 6,000 words… at most.

This means that MegaByte proposes a x125 improvement over ChatGPT’s best model.

But how does MegaByte pull it…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.