MindEye, a new mass surveillance tool or the key to understanding the brain?

🏝 TheTechOasis 🏝

It’s becoming really hard these days to find AI news not directly related to ChatGPT or Bard.

But the truth is that AI is poised to disrupt almost every aspect of our lives, including neuroscience.

Now, a recent breakthrough is giving us a glimpse of a future where minds can be read and the mysteries of some of humanity’s most unknown illnesses are unraveled.

The language of machines

At the end of the day, every AI application involves some sort of transformation.

That is, it’s all about ingesting data and transforming it into useful stuff, be that a new text sequence, or a new image.

Particularly, each model has its own particular way of understanding our world.

Its own “latent space”.

For this, models encapsulate their gained knowledge into vectors, forming an abstract, numerical representation of what our world means to them, grouping vectors that are similar, and separating those that aren’t.

If we were capable of drawing this (we can’t, these vectors have +500 dimensions), it would look something like this:

The problem is that each model has its own “latent space” and most times isn’t comparable (a “dog” in ChatGPT’s world is different from Bard’s).

Consequently, a great deal of AI research these days is focused on bridging these worlds between models to achieve “multimodality”, models that can process data in different forms like text or image, and map them into the same latent space.

In layman’s terms, multimodal models are capable of understanding that a picture of a dog and an image of a dog refer to the same thing.

Now, a stacked team of researchers from Princeton, Stability AI, among others have achieved an incredible milestone, mapping fMRI scans into image vector spaces.

Put simply, AI can now read your mind, and you’re about to understand why.

Bridging brain scans and images

In very simple terms, Mind’s Eye is a new AI model that it’s capable of two things:

Image reconstruction

Image retrieval

And the impressive thing is that the model can do both by simply looking at fMRI scans… without actually seeing the original image.

For instance, MindEye is capable of recreating images that highly resemble the original image seen by the human subject.

Regarding image retrieval, the results are equally impressive:

Seems like alien technology, but how does Mind’s Eye work?

Mapping the worlds of fMRI and Images

By looking at the image below, Mind’s Eye looks hella more complicated than it actually is.

It’s composed of two different pipelines:

A high-level pipeline optimized to achieve highly semantic image retrievals and recreations

A low-level pipeline optimized to achieve accuracy in low-level metrics like color, texture, or spatial position

In layperson terms, if the original image was a “man surfing a wave”, the former ensures that the outputted image represents that, and the latter ensures that the structure of the image, like the position of the surfer, is respected.

To create this remarkable solution, MindEye is formed by six components:

CLIP encoder

Enjoyed the introduction? Subscribe or sign in to continue with this edition.