RT-2, The World's Most Intelligent Robot

🏝 TheTechOasis 🏝

One of the good things about ChatGPT becoming superintelligent is that, at least, it doesn’t have a physical form.

That way, at least it can’t come after humanity if it goes rogue, right?

But now, a group of researchers in Google DeepMind have combined the power of visual-language models (VLMs) with robotics to create an armed robot that understands language and executes actions based on it.

Dubbed RT-2, it has set a new state-of-the-art for robotics in terms of generalization and emergent capabilities, a step-function increase over everything we had seen until now.

And to pull this off, they’ve created a new class of AI models.

A new model class

VLAs are visual-language-action models that are able to output actions a robot can then use to perform the desired movement.

Of these, RT-2 is the first of its kind, capable of ingesting images and text, and outputting not only text but robotic actions too.

In other words, RT-2 is multimodal, the new paradigm for Generative AI that aims to create models that combine multiple modalities (text, images, video, etc.) into one unique model, a necessary step toward AGI and, ultimately, superintelligence.

Of course, due to the impressive nature of current language models like GPT, researchers decided there was great potential in using these web-scale models for robotic applications.

The reason for this was simple, creating a robotic model from scratch using only robotic data was impossible given the scarcity of such data.

However, when trying to use web-scale LLMs or VLMs for robotic applications, the results were, to put simply, mild.

Specifically, the capacity of these models to generalize and perform well in unseen environments or with unseen objects was really bad.

But then researchers at Google came to a realization.

What if we use an {image + text => text} VLM and we co-fine-tune it with both visual-language tasks (like answering questions about images) and robotic data, to teach the model to output robot actions besides text?

And that my friends is what we can now call VLAs.

But what are they really?

An action predictor

Just like LLMs like ChatGPT, visual-language models do the same thing, they learn to predict the best next token based on the provided input.

For LLMs, you give them a text sequence, and they end it with more text.

For VLMs, you give them an image and text and they give you text back too.

Now, DeepMind has trained RT-2 to be capable of not only outputting text, but also outputting robotic actions that would then be used by a robot as input, as you can see below:

In layman’s terms, it’s ChatGPT equipped with a camera that provides the observations, and predicts the action tokens the robot has to perform based on the camera observations.

In case wondering, we’re talking about an 8DoF robot, with 6 degrees of freedom for movement, 1 for gripper extension, and another for action termination.

But did they succeed?

Generalization and emergence

With VLAs,…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.