Microsoft Leads Us Into the World's of LMMs and MultiAgents

🏝 TheTechOasis 🏝

Breaking down the most advanced AI systems in the world to prepare you for your future.

5-minute weekly reads.

TLDR:

AI Research of the Week: Microsoft presents LLaVa 1.5, a new open-source paradigm

Leaders: Everyone will Manage Teams in the future, only most of the team members won’t be human; The MultiAgent Paradigm is now a reality

🤯 AI Research of the week 🤯

I’ve talked about the AI war between open-source and private models many times, and always the outcome seems to be the same.

Open-source seems great and full of promises, but is often simply wishful thinking and impracticality.

But now things might have changed.

Microsoft, alongside the Universities of Wisconsin-Madison and Columbia, has presented the newest version of the LLaVa model, LLaVa 1.5.

LLaVa, one of the first truly performant Large Multimodal Models (LMMs), has been upgraded and the results are highly impressive, considering it is orders of magnitude, and I mean hundreds of times, smaller than models like GPT-4 Vision, OpenAI’s newest release that is taking the world by storm.

The recently released paper not only gives us huge insight on how state-of-the-art multimodal models are built, but also manages to prove the entire industry wrong.

Yes, everybody was wrong about open-source, including myself.

Oh my Sweet, Sweet Grafting

First and foremost, we must clarify what multimodality is, as this word gets thrown around aimlessly.

What really is multimodality

In a strict and concise form, multimodality represents the capacity of a model to process at least two different modalities, a modality being a type of input data (words, images, sounds, et al).

However, this definition can easily be fooled. If I get a speech-to-text model and an LLM, I can easily create a solution that processes both speech and text.

But does that make my solution multimodal?

Yes, but the models used certainly aren’t multimodal.

A great example of this is ChatGPT. In its new GPT-4V version, ChatGPT is image & text multimodal, but it also allows you to process and synthesize speech. Thus, the solution processes three types of input (text, images, and speech) but the model in reality is only multimodal for text and images.

Don’t get fooled by User Interfaces!!

In its purest form, true multimodality occurs when a model’s different modality components share a common embedding space.

In other words, taking a visual and text model like LLaVa, it happens when the visual and text components encode the input into the same high-dimensional vectorial space.

Embedding space: As machines only understand numbers, all input, be that text, images, or spectrograms, is transformed into a dense vector, dense because it incorporates key semantic information about the underlying data.

In this space, similar concepts have similar vectors, signaling the model that the closer two vectors are, the similar the underlying data they represent is.

This is how machines understand our world. It’s all based on relatedness, as OpenAI describes it.

Therefore, if an image of the Taj Mahal and its Wikipedia…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.