Demystifying OpenAI's STUNNING New Model Sora & Meta's World Model V-JEPA

Sponsored by

🏝 TheTechOasis 🏝

Breaking down the most advanced AI systems in the world to prepare you for the future.

5-minute weekly reads.

TLDR:

AI Research of the Week: Sora, OpenAI’s Stunning New Video Model

Leaders: V-JEPA and World Models

🤩 AI Research of the week 🤩

The feeling that things have changed. That’s what almost everyone who has seen Sora felt.

Last week, we talked about Google’s Lumiere model and how it was better than anything we had seen. Well, a week later, OpenAI’s Sora blows Lumiere completely out of the water.

No words can describe it, so here’s an example:

Source: OpenAI

Thus, today, we are unpacking all that is known about this life-changing model, and what it means for the future of many industries.

Best of both worlds

Sora is OpenAI’s first text-to-video, image-to-video, video-to-video model, a world simulator that generates the most impressive videos we have ever seen, literally miles ahead of anything else.

Similarly to ChatGPT, it has seen Internet-scale data, but of videos and their descriptions, to learn to create up to 1-minute-long high-resolution videos.

However, the similarities with ChatGPT don’t end there.

Just like LLMs treat text as tokens (groups of 3/4 letters on average), Sora treats frame patches as tokens too, calling them “visual spacetime patches”.

But what does that even mean?

In a nutshell, Sora treats videos by decomposing them into two elements:

Frames. Just like any other video model, the easiest way to build videos is to consider them as concatenations of frames (static images), as Google’s Lumiere does.

Visual patches. Each frame is then decomposed into groups of pixels. However, these patches are 3D, as they capture more than one frame so that we can account for time too.

Then, depending on the size ratio you want the output to be, those patches are assembled into that shape to generate the video.

But how does it actually generate videos?

As we have described many times, diffusion is the standard approach to synthesizing images and videos.

During training, noise is progressively added into an image, and then the model has to predict how much noise was added, take it out, and reconstruct the original image based on the given text condition that describes the original image, like “a portrait of a relaxed cat”.

That way, during inference, the model is given a noise canvas and it’s capable of creating new images based on the given text description.

Think of diffusion as a similar exercise to what an artist like Michelangelo or Bernini would do with a marble block. They took out the “excess” marble to uncover the statue.

So, what’s the end-to-end process then?

Sora’s generation process

During training, the model is given two things, a video, and a text description of that video.

As they required Internet-scale data, they trained a separate video captioning model that took millions of videos and described them through text, instead of doing it manually.

Then, the process follows four steps: compression, patching,…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.