Lumiere's Text-to-Video & Rabbit vs Apple

🏝 TheTechOasis 🏝

Breaking down the most advanced AI systems in the world to prepare you for your future.

5-minute weekly reads.

TLDR:

AI Research of the Week: Google’s Lumiere Model Looks Amazing

Leaders: Will The Rabbit Eat The Apple?

🤩 AI Research of the week 🤩

Scott Galloway, the famous business influencer, bet that 2024 would be Google’s AI year.

And it’s starting to look like it.

Now, they have launched Lumiere, a huge breakthrough in text-to-video.

Their approach to AI video synthesis is not only revolutionary, but it also showcases incredible video quality and a wide range of amazing downstream skills like video in/outpainting, image animation, and video styling, making it the new reference in the field.

But how does it work? Let’s get into the weeds.

The Everlasting Problem

Out of all data modalities, video is, without a doubt, the hardest.

Considering videos are simply a concatenation of images, called frames, displayed at a certain frame rate per second (the higher the fps the smoother the video) the logical path to building text-to-video (T2V) is to depart from a text-to-image model (T2I), like DALL-e or Stable Diffusion.

However, T2V adds an extra layer of complexity: Time.

In other words, it’s not enough to generate multiple frames (you can generate as many as you need with the T2I model) but they must be consistent over time.

And this has proven to be a huge problem, as this forces AI videos to be very short, and they nevertheless tend to showcase artifacts, visual imperfections like the orange blob that suddenly appears in this AI-generated video below.

Source: NVIDIA

And the reasons for these inconsistencies reside in how these models are built, and that Lumiere revolutionizes.

Source: Google

Originally, the video synthesis process involves three steps:

A T2I prior generates a set of key frames set across the complete duration of the video.

Next, several TSR (Temporal Super Resolution) models “fill in” the gap between the keyframes with a set of new frames.

If the two key frames are a serious person and that same person smiling, the TSR models generate the complete set of frames that generate the smiling gesture.

Then, a set of SSRs (Spatial Super Resolution) will take the low-resolution frames and upscale them to enhance the quality of the video, as most T2V models work in low-resolution pixel space for a more efficient process.

Finally, the SSR outputs are “stitched” together and that gives you the video.

Bottom line, AI videos are simply taking an image generator and training it to generate consistent images through time in batches and piece them together.

And these works… kind of.

Just like if you are filming an actor and midway through the sketch he breaks character and you try to finish the rest of the sketch by forcing him into that specific position to continue and avoid losing the first part, chances are that no matter how well you edit it, the cut is going to be visible.

Considering…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.