SALMONN, the First Model that Hears like Humans do

🏝 TheTechOasis 🏝

Breaking down the most advanced AI systems in the world to prepare you for your future.

5-minute weekly reads.

TLDR:

AI Research of the Week: SALMONN, the First-Ever Audio-Language Multimodal LLM

Leaders: Why Biden’s AI Executive Order May Change the World Forever

🤯 AI Research of the week 🤯

People often underestimate the importance of hearing to function correctly in our world and, more importantly, as an essential tool for learning.

As the famed Helen Keller once said, “Blindness cuts us off from things, but deafness cuts us off from people” and let’s not forget that this woman was blind and deaf.

Therefore, it's only natural to see hearing as an indispensable requirement for AI to become the sought-after superior ‘being’ that some people predict it will become.

Sadly, current AI systems suck at hearing.

Yes, OpenAI’s Whisper understands speech pretty well, and other models capture audio events very efficiently.

But hearing is much more than that. Humans understand speech, encode random noises, and enjoy music, making ‘generic hearing’ one of the last features that AI couldn’t replicate from humans.

Now, a new model created by the company behind TikTok, ByteDance, challenges this vision.

SALMONN is the first-ever multimodal audio-language AI system for generic hearing, a model that can process random audio signals from the three main sound types: speech, audio events, and music.

What’s more, as we’ll see shortly, it showcases truly unique, never-seen-before capabilities like audio storytelling or audio-speech co-reasoning.

And today, we are understanding how it works.

Building True Multimodality

To build models that process data in more ways than text, we usually connect an encoder and a Large Language Model (LLM) through some sort of adaptor or cross-attention mechanism, with cases like the recently-covered LLaVa.

Image-language multimodality is very common, but audio, on the other hand, is much less visible today.

But in both cases, a latent problem persists.

Most of these models are great at standalone vision, audio, or language tasks, but fail miserably when facing cross-modal tasks, especially when talking about audio models.

For instance, if we send the model a sound file with the sentence “Why am I shouting?” it should give completely different answers depending on the situation of the person.

To the left, the audio would probably include gunshots or explosions, and the one on the right would include high music, singing, and so on.

Thus, both situations require different answers, but current audio models can’t handle such complex tasks.

Amazingly, SALMONN can.

A Tale of Three Steps

Although at first glance SALMONN looks similar to how other Multimodal LLMs are trained, it has some important distinctions.

Bridging data types

LLMs like ChatGPT take a text sequence and complete it by predicting the ‘next word’.

To do so, they have been trained by seeing millions upon millions of text passages.

But if we send them an image or audio, it won’t work because they won’t understand them, just like you won’t get an answer back if you ask…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.