LEGO, the Future of TikTok & The Inevitable Death of the App

Sponsored by

🏝 TheTechOasis 🏝

Breaking down the most advanced AI systems in the world to prepare you for your future.

5-minute weekly reads.

TLDR:

AI Research of the Week: LEGO, the Future of TikTok & Your Kids

Leaders: The Inevitable Death of Apps

🤩 AI Research of the week 🤩

ByteDance has announced a new model named LEGO that not only has impressive capabilities, but also gives us a clear idea of the future of its products, including the app your kids are glued to right now.

The world-famous TikTok. And you might not like what this future looks like.

At all.

Pay close attention, AI

Even though Multimodal Large Language Models like ChatGPT are all the rage right now, they are surprisingly limited.

Therefore, since the beginning of 2024, researchers have put a lot of focus on making these models better.

The promises of success are so many that some of the largest corporations in the world, like Apple with its recently covered by this newsletter model Ferret, are working on it.

But why are MLLMs so limited?

The attention problem

Most MLLMs today, except some notable exceptions like Fuyu, are based on the combination of two AI components:

A Large Language Model (LLM), aka the brain of the model. It processes all inputs and generates the response the MLLM will give back to you.

One or more modality encoders. An encoder is an AI model that takes an input, like text, audio, images, or video and outputs a representation of that input.

But what is a representation?

You will see this term everywhere in AI literature. It’s what an AI model ‘sees and understands’ from a certain input. Tangibly speaking, it’s the portrayal of the input, like the word ‘table’ or a picture of one, as a vector of numbers, as that’s the only way machines can process data.

This transformation is done with models like encoders. The key is that they capture the semantics of the input, so the numbers in the vector aren’t random. Thus, if you calculate the representation of two different tables and a cat, the vectors of the tables will be very similar, but very different to the vector representation of ‘cat’, because they are semantically different concepts.

Next, researchers, to save money, do what we call ‘grafting’.

They take a pre-trained encoder, like OpenAI’s CLIP model to process images, and an LLM, and connect them using an ‘adapter’.

This adapter is required because, as the encoder and LLM have been trained separately, their respective representations of the same thing, like ‘table’, may be different.

As the generative part of the MLLM always comes through the LLM, the usual practice is to use the adapter to transform the representation of the encoder into the ‘embedding space’—the vector space of representations—of the LLM.

Sounds complicated, but the intuition is simple.

Imagine a Japanese woman and a blind American man facing the sunset.

Even though the Japanese woman can explain what she’s seeing perfectly…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.