Stable Diffusion 3 & The Sinister Future of AI-Based Ads

In partnership with

🏝 TheTechOasis 🏝

Breaking down the most advanced AI systems in the world to prepare you for the future.

5-minute weekly reads.

TLDR:

AI Research of the Week: Understanding the magic behind Stable Diffusion 3

Leaders: Video, The Next AI Interface

🤩 AI Research of the week 🤩

This week Stability AI announced Stable Diffusion 3 (SD3), the next evolution of the most famous open-source model for image generation.

It displays amazing results in fidelity and resolution, making it both visually and quantitatively speaking, the best text-to-image (T2I) model in the industry today.

Also, it showcases the most impressive AI-generated image I have ever seen, where the line-rendering of a zebra is accurately represented:

More importantly, it finally solves one of the hardest problems in AI images, text-image alignment, by introducing several innovative design approaches (some probably inspired by Sora from OpenAI) that will most probably set the standard for what’s to come.

Mastering Flows

If we think about a generative model, be that ChatGPT or SD3, there is a common definition; they all learn the underlying distribution of the training data to sample from it.

But what does that mean?

It’s all learning patterns

If we think about ChatGPT, when we say that it learns the training data distribution, we are indirectly saying that the model learns the patterns and structures of the training text.

In layman’s terms, it learns how words follow each other. T2I models like SD3 are no different in that sense, but learning the distribution of the training images.

Simply put, they learn things like “if I see a group of pixels depicting a dog face, the rest of the body must be near”.

However, here’s the caveat, T2I are mainly flow diffusion models, meaning they learn to perform a transformation between the original distribution and the target one (the patterns and structures of the training data).

Again, what does that mean?

Flow Generative Models

Flow models, unlike those like ChatGPT that sample new words to follow a sequence from the data distribution, learn a ‘mapping’ that will take a noisy, random ‘canvas’ and turn it into an image that is indeed similar to other images it has seen during training and conditioned on the user’s request.

That way, it can draw things like “an animal playing basketball”:

Source: OpenAI

But why do we need this flow process, and instead sample from the learned distribution of images like we do with ChatGPT for text?

Well, you want your model to have some sort of randomness, or stochasticity, so that the newly generated samples are new images, not mere copies of ones that it saw during training.

This allows the model to generate new images from one single prompt like “a baboon close-up”.

Source: OpenAI

In SD3’s case, this flow is a diffusion, where the model learns to predict the noise in an image and take it out.

However, here SD3 innovates quite a bit by applying Rectified flows, where this flow between…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.