FAn, the Model that Tracks Anything

🏝 TheTechOasis 🏝

Imagine it’s 2030.

You’re walking your dog late at night. The atmosphere is dead quiet, but you hear a buzzing sound somewhere around you.

You feel weird… as if you were being watched. As something was following you.

You try to hide from open spaces, but every time you come into the open, you hear it again. Eventually, you can even see it.

It’s a drone, and it’s following you. And there’s nothing you can do about it.

Source: Author with MidJourney

Seems like yet another CyberPunk story, right?

But, to your surprise, that technology already exists, and you’re going to see it for yourself.

The ultimate tracker

Two of the most prestigious universities in the world, MIT and Harvard, have collaborated to create FAn, the “Follow Anything model”.

FAn is an open-vocabulary-any-object tracker.

In other words, it’s capable of tracking any object you desire, as long as it’s in its camera view.

It’s also multimodal, meaning that it can track objects you define with words, like “a ball”, by clicking on the object in the frame, or even using bounding boxes.

That’s all you need, and FAn will track it with insane accuracy.

Even in cases where the object temporally disappears, FAn is capable of redetecting the object and continuing to track it.

The applications are as life-changing for some as extremely dangerous for others.

But what’s the intuition behind it?

The world of feature spaces

One of AI's most important, yet misunderstood, discussions is the idea of AI representations.

You see, to make AI work, data needs to be in numerical format, as computers only understand numbers. For such, we use vector embeddings.

These are numerical representations of the data you give the model, be that text, images, spectrograms, you name it.

For instance, the word ‘dog’ will take a form such as:

In order to understand the patterns and abstract meaning behind data, AI models create this compressed, n-dimensional numerical space (called a feature space or embedding space in AI literature) that has one principal law:

Similar concepts in real life should have similar vectors in this space. Dissimilar concepts should have different vectors.

For instance:

This is how machines understand meaning and context. ChatGPT has such representations, as so does MidJourney.

Additionally, one of the most dynamic fields of AI research is making these spaces ‘multimodal’, or capable of placing together similar concepts in different formats:

In case wondering, to evaluate how similar two vectors are, we calculate vector similarity using several techniques such as cosine similarity or Euclidean distance

Now, FAn takes these concepts and builds them into the best zero-shot autonomous tracker.

But how does it work?

Modular architectures are here to stay

As I’ve said time and time again, AI is top of mind today thanks to the concept of foundation models like GPT powering solutions like ChatGPT.

Previously, models were trained to very domain-specific use cases and classes. For instance, FAn would have been built to detect a particular…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.