Deep Fusion and the New Age of Software Development

🏝 TheTechOasis 🏝

Breaking down the most advanced AI systems in the world to prepare you for your future.

5-minute weekly reads.

TLDR:

AI Research of the Week: CogVLM, a Revolutionary Multimodal Solution To the MLLM Problem

Leaders: OpenAI Shows Us the Future of Software Applications… and Much More

🤯 AI Research of the week 🤯

A group of researchers has presented a new model that revolutionizes the current multimodal AI design standards while blowing out of the water almost all competition.

The introduces an innovative concept, Deep Fusion, a new design that mitigates the biggest problem faced by Multimodal Large Language Models (MLLMs) today, the “shallow alignment problem”.

Were to deliver on its potential, the CogVLM model could become a seminal research paper that will draw the attention of researchers around the world to create a new family of MLLMs, Deep Fusion models.

The actual results? Impressive capabilities like coding math problems from images, and many more we’ll see shortly.

But first and foremost, what is the shallow alignment problem?

It couldn’t be that easy

Building an LLM is a cumbersome task.

You need a huge dataset of text documents, a team of world-class researchers, and a powerful GPU cluster. In other words, you need a ‘whole lotta monei’ and talent.

And if you want to make your model commercially available, you need to make it aware of what to say or not say.

For that alone, you will also need an army of human annotators to perform Reinforcement Learning from Human Feedback (RLHF), which in turn needs even more capital.

The result?

GPT-4’s initial model took $100 million to build end-to-end, and the first version of LLaMa’s 65-billion-parameter model burned $5 million in just 21 days.

But building an MLLM… that is a totally different beast.

Not only do you have to do the above process, but you also have to train an image encoder, the piece in the architecture that processes images, and make them work together.

Hence, unless you are part of the magnificent seven, or one of such companies is fiercely funding your efforts, you simply don’t have the means to train these models.

Alphabet, Amazon, Apple, Meta, Microsoft, Nvidia and Tesla are considered the ‘magnificent seven’.

Consequently, open-source researchers, driving their efforts mainly through University funding, have to be smart about their designs.

Grafting, an elegant answer

To avoid having to build the complete model by scratch, the most common method of creating MLLMs is grafting, the process of connecting an already-trained image encoder and an LLM through an adapter, usually a Q-former (like the one we saw last week in SALMONN), or an MLP layer.

The rationale is clear. If you want to use components that are already trained, you need to align them, i.e. make them talk the same language.

To do so, once the image encoder processes the image, the resulting representation is projected into the embedding space of the LLM.

Frontier AI models represent the world (images, text, audio,…) in…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.