Why Optics is the New Memory

In AI markets, memory chip companies, or the ‘memory trade’ in general, have absorbed most of the attention as the world places memory as the mother of all bottlenecks.

Well… not quite.

Instead, eventually, the real answer will be networking, which explains why optics is the next big thing in AI. And to prove my point, I’m taking action today by announcing my next stock investment.

By the end of this article, you’ll have learned the instrumental value of optics for AI by learning the ‘ins and outs’ of AI hardware scaling and the crucial role optics plays, understanding why it will displace copper, as well as the intricacies of the technology (switches, CPO, etc.) that will give you nearly unparalleled intuition of what to expect from AI in the next decade.

In the age of AI, everyone is talking about compute while everyone should be talking about networking.

Let’s uncover why.

This is not investment advice. I’m just sharing my investments in the spirit of transparency and because many of you have shown great interest in knowing what I invest in. If anything, this is just an explanation with just as much value for those that want to understand the technology as for those that want to make better investment decisions.

However, I can’t guarantee profits. Please do your own research before making any decisions.

Stop Looking at Compute, Look at Networking!

Kim Jeung-ho is the father of High Bandwidth Memory, or HBM, the memory technology used in AI data centers today and the most coveted product on the entire planet right now, and I’m not even kidding.

And while today’s article is about optics, the point I’m trying to make is that we are in the midst of a profound narrative shift regarding ‘what matters in AI’.

And what will matter the most soon won’t be compute, but two other things: first memory, then optics.

The Importance of Communication

I wouldn’t blame you for thinking that, whenever you interact with an AI model, what’s “on the other side” is a GPU that stores the model, runs your query, and responds.

But that can’t be further from the truth.

On the other side, you have an entire server worth millions of dollars, storing around 72 NVIDIA Graphics Processing Units (GPUs) tightly connected to one another, responding to you.

NVIDIA’s CEO, next to one of these racks

This means the actual AI is distributed across several accelerators, communicating over kilometers (about 3.4 km, or ~2 miles) of copper. This wiring, central to today’s discussion, is called ‘scale-up’ networking gear.

Ok, so? Simple:

When a computation workload is distributed, that usually means that the real bottleneck is not how powerful your computer is, but how fast your distributed hardware communicates.

For example, if your GPU can process 100 operations per second but, due to communication constraints, you can only feed it 10, it will do so very quickly, but the limitation is palpable: you are leaving a lot of performance on…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.