Music ControlNet, democratizing music, & the power of SLMs

🏝 TheTechOasis 🏝

Breaking down the most advanced AI systems in the world to prepare you for your future.

5-minute weekly reads.

TLDR:

AI Research of the Week: Music ControlNet, Democratizing Music Creation

Leaders: SLMs, The Key to Deploying AI Everywhere in Our Lives

🤯 AI Research of the week 🤯

If there’s one ubiquitous element across almost all recent AI solutions, is that almost all of them are based on text-prompting.

ChatGPT for Text-to-text

Stable Diffusion for Text-to-image

Runway for Text-to-video

And so on.

Basically, AI right now is all about requesting something through text and getting whatever that model specializes in. And that includes music.

In fact, Models like MusicGen have become extremely popular, as you can literally create music using prompts like “Classic reggae track with an electronic guitar solo” and literally get something back that sounds good.

However, the fun ends pretty much there. You have no control over the generated melody, meaning that the practicality is nonexistent.

It’s a fun toy and nothing more.

But now, researchers from Adobe Research and Carnegie Mellon University, have changed the game by applying the amazing discovery that revolutionized image models to music synthesis to music, creating the first model that hands you the power to create under your own accord.

For the first time, the AI musician is now under control, and the potential is unbound.

It’s all a Transformation

If you asked me to define AI in one word, that would be ‘transformation’.

In a nutshell, what we describe as AI is nothing but a black box with a statistical algorithm inside that takes data in some form as an input and outputs data in another form, usually in the form of a prediction.

In its purest form, especially if we think about neural networks, these models are simply a huge function that models a certain task.

For instance:

ChatGPT is simply a huge function that has learned that, upon receiving a set of words, to predict the next one.

DALL-E, upon receiving a text description, predicts the noise in an image and takes it out to uncover the new image behind the noise.

And what makes neural networks so great is that, with enough data and some clever tweaking in the structure of this function, they can learn to perform any data transformation.

From words to music, and everything in between.

Heck, in last week’s newsletter, we saw how they turned brain signals from imagined movements into real robotic movements!

This capacity to model ‘anything’ is what makes neural networks be described as ‘universal function approximators’.

But why am I telling you this? Simple, because this is the intuition we need to understand how incredible Music ControlNet is.

Applying Control

Current text-to-music models like MusicGen by Meta or MusicLM by Google are already quite impressive, but lack any sense of control.

A common problem in the space

In short, they have learned very powerful global attributes of a multitude of genres and instruments, the appropriate moods…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.