NVIDIA shocks the world with insane AI-generated videos

🏝 TheTechOasis 🏝

🤖 This Week’s AI Insight 🤖

NVIDIA’s VideoLDM is the most impressive AI in 2023

If an image is worth 1000 words, what’s a video worth?

I’ll let you decide by checking this fully AI-generated video below:

Yes, this is 100% AI.

NVIDIA and the creators of Stable Diffusion have presented VideoLDM, a new state-of-the-art video synthesis AI model that proves, once again, that the world will never be the same.

But how does this amazing new AI marvel work and what other amazing stuff can it do?

Hard problems require smart answers

When synthesizing (generating) video from AI, you run into two huge issues:

Cost: Generating videos in standard ways is computationally intensive.

Lack of training data: AI video needs video data. And while text and images are accessible, video data no.

But how on Earth did they pull it off?

Work smart, not hard

Videos are simply frames (images) sequenced in a specific timeframe.

The more frames, the longer the video

The bigger the pixel size, the better quality the video has

And the more frames you can fit per second, the smoother the video.

In other words, it’s all images.

So why not use our powerful image generators like Stable Diffusion (SD) as the basis of our video synthesis model?

This way, we can use our widely accessible image data sets to train our model, instead of relying on costly and inaccessible video data.

Sounds awesome, but there’s an obvious issue.

If you ask SD to generate a panda two times in a row, both pandas will be observably different.

It doesn’t give a damn, or remember, how the previous panda looked, it’s just trained to give you what you ask at the point in time.

Zero consistency.

Models like ControlNet allow you to apply consistency, but you’re conditioning (forcing) the model to output data in a certain way; here, we want fully-creative video synthesis with no human interference.

So what can we do to solve this?

Simple, give it temporal awareness.

And so, the NVIDIA researchers propose here an innovative architecture that interleaves temporal layers with the SD spatial ones.

But hold on, I know that sounds way too overcomplicated.

However, it’s quite simple to understand with the following intuition:

What learns what and where

A neural network is simply a set of parameters that learn “something”, usually defined as a training objective.

For instance:

ChatGPT’s parameters learn to predict the next word in a sequence.

SD’s parameters learn to generate an image from a text prompt.

But these models, if we look inside, are packed into blocks of neural networks, which are then packed into layers formed by neurons, which in turn are defined by certain parameters (the elements that eventually learn that “something”).

This means that, while some blocks and layers learn to do some stuff, others will learn to do other stuff.

For instance, in our case:

The spatial layers are the Stable Diffusion original layers, already capable of impressive…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.