Breaking the language barriers with Meta's SeamlessM4T

🏝 TheTechOasis 🏝

Welcome to TTO, the future of AI and the world made simple.

🤯 AI Research of the week 🤯

Because it’s not what you say, but how you say it.

For a long time, developing high-quality speech AI models has been an obsession for research teams all around the world.

The potential for AI to bridge the ‘language gap’ and create universal machine translators is huge, as there are more than 7,200 known languages in the world.

Sadly, as AI is overly reliant on data, most AI solutions today focus primarily on the world's 10-12 most spoken languages, leaving around half of the world’s population behind.

Thus, what if we could create an AI model that allows us to communicate with anyone, anytime, and in real-time?

We aren’t quite there yet, but we have just made substantial progress thanks to Meta, with their SeamlessM4T models that support speech-to-speech translation, speech-to-text translation, text-to-speech translation, text-to-text translation, and automatic speech recognition for up to 100 languages.

All in one.

The Holy Grail

Building a high-performant speech model has long been considered a utopia in the world of AI. Especially if we’re talking about translating text or speech between languages.

For reference, for a text-only solution, English-Italian labeled data can be easily sourced for amounts above 100 million text pairs, but you will sweat quite a bit to get over 2 million pairs for speech training.

And we’re talking about two fairly popular languages, imagine the minority ones.

Speech data is, put simply, very scarce.

However, new types of AI models trained in unsupervised or self-supervised approaches are yielding great results. This is great because it allows researchers to use audio data that hasn’t been curated to train great models.

Unsupervised and self-supervised solutions do not require extensive human labeling of the data, allowing models to be trained with huge corpora, impossible if we had to label every single example manually.

Consequently, we’ve seen a proliferation of cascaded models, models that are specialized in specific use cases (speech-to-text, or text-to-speech, for instance) combined to create new models that are capable of doing a complete speech-to-speech translation exercise, the holy grail of AI speech modeling.

But now Meta has pushed the boundaries of AI one step further, to create the first model that can perform all text and speech-related tasks for translation for up to 100 different languages.

The everything model.

But how does it work?

1 + 1 = 100 - UnitY

As I always say, true multimodality is achieved when a unique set of weights is optimized for several modalities, which is precisely what Meta has done.

And, in the process, they also allow for these weights to not only handle multiple modalities, but also multiple tasks.

To achieve this, they have built the UnitY architecture, as per the image below:

This architecture is comprised of two parts:

The X2T model for text generation

Complete UnitY solution (X2T + T2U + HiFi-GAN) to synthesize speech

The X2T model takes…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.