Genie, converting images into videogames & the State of Hardware

Sponsored by

🏝 TheTechOasis 🏝

Breaking down the most advanced AI systems in the world to prepare you for the future.

5-minute weekly reads.

TLDR:

AI Research of the Week: Genie, Turning Images into Videogames

Leaders: Understanding NVIDIA’s Boom & The State of Hardware

🤩 AI Research of the week 🤩

Amid its current controversies with its misaligned LLM, Gemini, Google continues to innovate in other areas of AI.

Now, it has created an entirely new modality by announcing Genie, the first-ever model to create action-controllable environments.

Put simply, this could be the precursor to bespoke AI-generated videogames from one single provides image.

But it can be much more, as it has been framed as a potentially viable path to generalist agents, the milestone all AI labs in the world seem to be working toward.

So, what is Genie?

Genie in an AI Bottle

Genie is a spatiotemporal transformer that takes in a single image or sketch and action and generates a new frame where that action is executed.

As a video is nothing but a stacked set of images (known as frames) shown to you at a given frequency (number of frames shown by second), the key element here is that Genie allows you to ‘play the video’ by giving you the power, through actions, to decide what the next frame will be.

In other words, the world is created dependent on the actions you take.

For example, if we look at the image below, if you provide an image and the ‘jump’ action, the model generates a new frame where the now playable character is jumping while it adapts the entire background to that action.

Consequently, as every demanded movement is met with a new frame, you are essentially building a videogame in real-time.

But with examples like Genie, I think it’s best if you click on this link, see it for yourself, and come back to understand how it works.

First and foremost, the first key element is the concept of spatiotemporal transformer (ST), but what on Earth is that?

An ST transformer is a model that performs the attention mechanism, the technique that underpins products like ChatGPT, but applied on two dimensions: space and time.

Attention allows differents parts of a sequence to share information with each other. In the case of ChatGPT, it helps the different words in a sequence to ‘talk’ to each other to figure out the meaning of the overall sentence.

Here, we apply it to video.

To comprehend how this works, let’s say we have a video of a panda eating bamboo.

Space. By breaking each image into different patches, attention allows these patches to share information and figure out their role in the image. In the case of the panda, spatial attention will tell Genie what the panda is doing in that specific frame, like holding a bamboo stick with the mouth open.

Time. By performing attention over several frames, the time variable of the attention mechanism will tell Genie…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.