The Five Horses of Bandwidth

Over the last two weeks, a series of events like Hot Chips have laid out the roadmap for the AI industry for the remainder of the decade.

And the conclusion is pretty straightforward: Bandwidth is king.

Investors love magic solutions, so here’s one: success in AI progress over the next five years at least will be proportional to how much bandwidth-pilled your bets are. Bandwidth constrains progress, and most hardware decisions today are moving toward increasing bandwidth.

This means that, for the foreseeable future, alongside power density, bandwidth will define who wins and who loses, as those that position themselves in the critical path to unlocking more bandwidth will win.

This leads to seven big trends, the Five Horses of Bandwidth: five mega trends that are emerging in AI that will shape the remainder of the decade.

Let’s dive in. But first, why is bandwidth at the center?

All You Need is Bandwidth

If you’re a regular, you know I’ve been saying for a while that we’ve moved into a memory-centric paradigm. But that definition was kind of incomplete, which is what we’re fixing today.

But first, why?

Von Neumann didn’t expect this

Since its emergence in 1945, most computers have followed the Von Neumann architecture, which breaks a computer into three parts: the compute, the memory, and the communication bus between them.

The memory feeds instructions and data, the compute executes them, and outputs that need to be stored go back to memory.

Since then, there was really only one darling: compute. Most progress focused on maximizing compute capacity growth. Memory followed, but lagged.

And with the advent of AI models, both have lagged behind model growth; neither can keep up with AI's ever-growing machine needs.

Source: SK Hynix

But I digress.

The reason was that most computer workloads were compute-bound, meaning that what defined how fast work was done was how much compute power you had available. Conversely, being memory-bound tells you that what defines performance is memory bandwidth, the ability to move vast amounts of data per second between compute and memory.

Source

For example, if a workload required your computer to do 1,000 operations for every byte of data you moved (remember, data has to be moved from memory to compute to be processed), if your computer could only handle 500 operations/byte (that limit is called the ridge point, which can be seen above), you were compute bound, because even in the event your memory fed work to compute perfectly, your compute could still only deliver 500 operations for every byte, so even under perfect conditions, your computer would take 2 seconds.

On the other hand, if you had a computer with a ridge point of 2,000 operations/byte, for that same workload, you’re now memory-bound, because no matter how much stronger you make your compute (say, you increase its capacity to 3,000), it won’t change a thing because what stalls your performance is your inability to feed enough work to your overqualified compute cores.

If anything,…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.