We've Hit AI Hardware 2.0

We’ve Hit AI Hardware 2.0

Is heterogeneous hardware the new norm? I profound believe so, but that isn’t good news for neither of us.

Over the last few days, something has changed in the AI industry: hardware. For years, hardware dictated how AI software was built.

Now, it’s the other way around.

This leads us to hardware 2.0, heterogeneous hardware built purposefully not only for AI, but for AI inference in particular. After reading this article in full, you will:

Intuitively understand how AI workloads work to a level of depth even most so-called experts don’t reach

Deeply understand ‘where AI hardware is going’ by understanding NVIDIA’s new product, the POD, the most powerful AI server the world has ever seen, in striking detail most miss.

The coveted inference technique that heterogeneous hardware finally unlocks,

And winners and losers of all this, which is particularly relevant, considering most stock market action in AI is hardware-based, as AI software companies are mostly private.

This is a little bit longer content than usual. If the AI industry feels alien to you, it won’t be as much after this read. And if you are already an expert, you’ll be an even bigger one.

This week is my personal agent week; I’m going to go deep into the trenches to test the limits of what can be done with agents, and I’ll share my updated AI stack, or how I use AI in my daily routine, next weekend. Stay tuned!

Inference is taking over the world

Inference, the act of serving AI models to users, is the most important single word in the industry today.

However, it’s also AI’s biggest concern.

GPUs were not built for it

When a handful of AI researchers at Google presented the Transformer architecture back in 2017, the algorithm that has underpinned almost every inch of progress over what is now nearing a decade, they built it to the image and likeness of NVIDIA GPUs.

As researchers realized that AIs were going to get bigger and require more computations per second, they turned to the hardware that could deliver the highest compute per second: the GPU, and created an algorithm built to leverage GPUs' strengths.

That ‘strength’ is parallelization, being capable of doing a lot of computations in parallel so as to increase the number of computations one can do per second. Data is stored in memory, then fed to the processors, which perform many parallel computations and return the results to memory.

Simply put, the more parallel a workload is, the better suited for GPUs. Conversely, if a workload is highly sequential, you’re leaving out a lot of ‘GPU potential’, just like buying a Ferrari that can reach 190 miles per hour to ride through the city at 30.

Okay, I think we all knew GPUs were important in AI. Nevertheless, today, we spend more than $500 billion per year on this hardware, making the efficient use of it of utmost importance.

But how do we measure this?

Enjoyed the introduction? Subscribe or sign in to continue with this edition.