Bringing Images Back to Life, by Google

🏝 TheTechOasis 🏝

Breaking down the most advanced AI systems in the world to prepare you for your future.

5-minute weekly reads.

🤯 AI Research of the week 🤯

There are times when AI feels like magic.

And this is one of those times.

The gif you’re seeing below was actually an image that was brought “back to life” by Google Research.

And that dragging movement you see? That’s actually me, as you can try the model for free by inducing the motion you desire.

In short, Google has presented a new method that predicts the movement of oscillating objects from still images, turning them into videos of that object moving.

Welcome to 2023, when AI meets magic.

The Universal Approximator

When humans try to predict movements around us, we leverage complex physics laws, mathematical equations, and so on.

But do we really need to provide those laws to AI to help it learn to predict them?

Firstly, that would be unfeasible, as when studying nature’s oscillations you have to take into account too many variables: wind, water currents, respiration, or other natural rhythms.

Luckily, as neural networks are universal function approximators, they do not require learning those laws and principles directly, they can induce them indirectly through observation, just like a human would.

In layman’s terms, you can teach a model to predict oscillations by simply looking at them.

Therefore, this model wasn’t trained with the laws of physics and mathematics, but by simply observing videos.

That’s what makes Deep Learning unique and so powerful, given the right data, it can learn anything.

The importance of frequency

Next, another insightful tweak the researchers made to train the model was to do so in the frequency domain.

Frequency measures the amount of times an event occurs in a second. Why? Well, working in the frequency domain allows to uncover “hidden data”.

As a fun fact, the Fast Fourier Transform used in this study was invented back in 1965 to detect subterrenean nuclear bomb detonations, which were impossible to separate from earthquakes in the time domain.

In other words, instead of the model predicting the movement of the object in the time domain (in second 3 the object will be here, in second 6 here) the model did so in the frequency domain (the movement of the object at 0.2Hz is this, at 0.6Hz this).

But why?

If we want to bring an image back to life with a moving object, we need to predict the position of that object at consequent time frames, generate a new image for every time step, and concatenate it, thus creating a video where the object moves.

That means that, in the time domain, for every pixel in the first image, we need to predict the position of that pixel 2T times, ‘T’ being the number of new frames we are generating (which is a lot if we want the video to be smooth) and 2 for requiring ‘x’ and ‘y’ coordinates for every pixel.

Enjoyed the introduction? Subscribe or sign in to continue with this edition.