AudioSep, separating audio on command

🏝 TheTechOasis 🏝

When the nuclear bombs were dropped in Japan, everybody knew the world had changed forever.

And after the Soviet Union rejected the idea of stopping the creation of nuclear bombs, the atomic race officially started.

Eventually, after the disaster at Bikini Atoll, all leading participants signed a document stating that they would cease nuclear tests in all forms except one:

Subterranean nuclear detonations.

Why?

Simple, at that time it was impossible to differentiate a subterranean nuclear detonation from an Earthquake with a seismograph, which meant that all countries continued nuclear testing anyway.

To solve this, one of the most important algorithms in history, if not the most, was created, the Fast Fourier Transform (FFT).

This algorithm broke down the vibrations gathered by the seismograph into its frequencies, allowing it to differentiate earthquakes from nuclear bombs, a discovery that, had it been done a few years earlier, would have changed the course of time forever.

If you’re curious, this video by Veritasium explains it beautifully.

But why am I telling you all this?

Simple, because Fourier transforms are an essential component of AudioSep, the first foundation model for language-queried audio separation, or the model with the power to separate almost any sound by simply asking it.

It’s all frequencies

Have you ever wondered why the same note played by a piano and by a trumpet sound different?

Even though you can clearly hear both playing the same note, pianos and trumpets sound completely different to the ear.

And the reason for this is what we call frequencies.

When we represent an audio wave, most sounds we hear in the world, such as having a conversation with your friend in a busy street, are complex, meaning that the sound of your friend’s voice is bundled with the sound of a car honking behind or a random baby crying.

Therefore, if we represent that audio clip as a waveform, it could look something like this:

However, this purely random wave can be, in fact, broken down into canonical sinus and cosine waves at different amplitudes and phases, that when summed up give you this random wave.

These waves have different frequencies, and to obtain them you have to apply the Fourier transform. By doing this transformation, we gain huge insight into the audio, and we can even reject unwanted frequencies if needed, critical to elements like music production.

This may seem super complicated, but it’s a very similar concept to how the last generation of AirPods cancel noise by creating antinoise, eliminating the undesirable frequencies.

Consequently, for any noisy mixture of audio like the one we were discussing before, we can break the mixture into different audios:

The voice of your friend

The honking car

The crying baby

etc.

But why is this so interesting?

A plethora of use cases

Noise separation has incredibly useful applications, such as:

Hearing Aids and Assistive Devices: Audio separation allows users to focus on a single speaker amidst background noise.

Forensic Audio Analysis: Extracting…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.