Robotics' ChatGPT moment, Dumpster GPUs, & Bond Tragedies

TLDR;

Welcome back! Today we have a quite heterogeneous lineup. From what could be Robotics’ ‘ChatGPT moment’ to the tragedy we are witnessing on bond markets, which could have a tremendous impact on AI’s ability to finance itself. We also include Cerebras’ new product, a DumpsterCluster, and more interesting stuff.

Enjoy!

ROBOTICS

Robots are now zero-shot learners

As published by Generalist on August 19, the robotics startup has introduced GEN-1.5, a multimodal robot foundation model that can learn simple physical tasks from a single 3–12 second demonstration, without fine-tuning or gradient updates.

Generalist calls this approach “physical prompting”: it places a demonstration in the model’s context window, and the robot immediately attempts the inferred task with no training, succeeding in some very impressive cases.

Across 10 manipulation tasks (including opening jars, handling zippers and retrieving money from wallets), GEN-1.5 achieved an average 59% success rate from one demonstration. With around five minutes of task-specific data and 10 gradient updates, success increased to 83%. The company emphasizes that the tasks remain simple and short-horizon and that one-shot performance is still relatively brittle.

The model can also combine multiple demonstrated behaviors, imitate some demonstrations performed directly by humans, transfer demonstrations from simulation to real robots, and improvise with previously unseen tools. Generalist says it didn't explicitly train for these capabilities, but they emerged as it scaled pretraining on physical interaction data. GEN-1.5 has been training continuously for more than eight months.

TheWhiteBox’s takeaway:

This feels a lot like how Large Language Models (LLMs) were in 2020. Why?

The key is this physical prompting idea, where the model learns to do a task it hasn't seen before simply by showing a demonstration in the prompt.

This may sound familiar to those who have been in the space for longer, because that’s exactly what we achieved with LLMs back in 2020 in the seminal paper ‘Language Models are zero-shot learners’, published by OpenAI back when it had both OpenAI and Anthropic’s founders on the same team.

In that paper, they showed that, by scaling model sizes, you really didn’t have to retrain your model for every new task; just show the LLM a few examples, and the model would use those to learn how to do the task. Somehow, the model became capable of doing stuff it had never seen until that moment.

Now, robots are starting to do the same.

Six years later, LLMs are incredibly powerful AIs, and we’re now starting to see the same emerging behaviors in robotics.

The remaining question is reliability, which just isn’t there, as most tasks barely reach even 90% accuracy. As I’ve said multiple times, this isn't acceptable; you should aim for 99.9% or even 99.99% (three and four nines of reliability).

However, there’s hope: this is the same team that showed 99%+ success rates after task-specific post-training, which proves my earlier point: we’ll need task-specific training to reach the necessary production-grade reliability.

In layman’s terms, this is how I see things moving on:

We are…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.