OpenAI and Google Present your Future Coworkers

🏝 TheTechOasis 🏝

Breaking down the most advanced AI systems in the world to prepare you for your future.

5-minute weekly reads.

TLDR:

AI Research of the Week: Google Deepmind presents a new robotics paradigm

Leaders: Unveiling the Game-changing Potential of GPT-4Vision

🤯 AI Research of the week 🤯

One can safely say that Google Deepmind is to robotics what OpenAI is to Large Language Models.

And even though AI-based robotics generates as much fear as hype due to the intrusive notion of humanity creating highly intelligent AI models embodied into the physical realm, this week we have seen proof that the field is steadily approaching its ‘ChatGPT’ moment.

Whether we like it or not.

And their new models, the RT-X family, are a statement of fact that AI has reached a point of no return.

A multiple-body motion predictor

Some months ago in this newsletter, we saw the case of RT-2, Google Deepmind’s state-of-the-art robotic arm.

RT-2, the universal arm

This model, a first of its kind, is a VLA (Vision-Language-Action) model that, given a video frame and an instruction, predicts the movements required from its actuator to perform the instruction based on the observations of its camera.

Source: Google Deepmind

The model is comprised of a Vision Transformer and an LLM.

The former processes the images, and the latter text, both encoding their respective inputs into a common embedding space so that the LLM could then decode the actions the arm had to execute.

Here, robotic actions are treated as LLM tokens. In other words, this LLM was fine-tuned to output actuator motions instead of words.

This was a first in the industry, in the sense that we had managed to positively transfer the knowledge a model learned from text and images into a robotic arm that would then be much more capable of processing natural language and vision instructions.

In summary, RT-2 is a model that, given a complex instruction in natural language, will understand the task and successfully predict the required actuator moves to move the arm and complete it.

Source: Deepmind

As you can see in the image above, RT-2 performed very semantically complex tasks such as ‘pick land animal’, correctly choosing between a horse figure and an octopus fluffy toy.

Much more than a robotic arm

For all these features, RT-2 was never meant to be a simple new robotic arm model, but a paradigm change.

It represented the effort of the industry in doing for robotics what has been achieved for natural language and vision… a general-purpose model.

And, in a way, RT-2 was the first stone in building the pavement in that direction.

Now, RT-X is a new iteration of this vision. And what an iteration that is.

A multibody approach

When announcing RT-X, Google also announced the release of the Open X-Embodiment repository, a cross effort of up to 21 universities and institutions in the world to assemble the biggest robotics dataset known to date, with many different types of robots,…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.