Moving Robots with Your Mind and the AI Data Opportunity

🏝 TheTechOasis 🏝

Breaking down the most advanced AI systems in the world to prepare you for your future.

5-minute weekly reads.

TLDR:

AI Research of the Week: NOIR, Moving Robots with Brain Signals

Leaders: Envisioning a Future of AI Empowerment and Economic Opportunity

🤯 AI Research of the week 🤯

At this point, my mom thinks some AI applications have become nothing far from magic.

And today we are going to talk about one of those cases, NOIR.

Developed by Stanford University, this is a model that allows humans to move robots with their minds, something not short of extraordinary.

NOIR is a general-purpose model with the capacity to decode imagined motions through brain signals into interpretable robotic motions that execute the action imagined by the person.

Thanks to advancements like NOIR, one can envision a world where physically-impaired humans see their quality of life massively improved.

But first, how does it work?

A Tale of Two Components

To decode the brain signals, the researchers decided to use electroencephalography (EEG).

But EEG is hard to decode, especially considering complex behaviors like:

Choosing an object visually

Deciding how you’re going to pick it up

Deciding what to do with it

Perform the action

To fix this, NOIR is comprised of two components:

A modular goal decoder

A robot to perform actions

The modular decoder will break down the human goal into three parts: What object, How to interact with it, and Where to interact.

Focusing on the what, NOIR takes in SSVEP signals, a type of brain signal that matches the flickering frequency of certain stimuli.

The concept of frequency describes the number of times a certain thing happen in a second. Measured in Hertzs (Hz) if I am capable of jumping three times per second, my jumping frequency is 3 Hz.

In layman’s terms, if you watch an object flickering at a certain frequency, your brain will emit SSVEP signals at that exact frequency.

Consequently, if you have objects emitting light at different frequencies, the one with the highest correlation to the frequency of the SSVEP signal will be the object being observed by the human.

Traditionally, researchers incorporated LED lights into objects to make them flicker, but here the Stanford researchers leverage Meta’s DINO computer vision foundation model to simulate this.

When observing a certain scene, DINO segments all objects appearing in the image.

Then, the researchers simply add a virtual flickering mask (every mask at a slightly different flicker frequency to be able to discriminate) to every segmented object.

This way, we know what object the humans are looking at.

Object segmentation with SAM. Source: Meta

Next, to understand what the human wants to do with that object, the how and the where… things get a little bit trickier.

From Imagination to Motion

When we humans think about performing a movement, this thought generates brain signals too. This is what we describe as Motion Imagery (MI).

The challenge here is that we want to be able to…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.