AI's Elephant in the Room: Energy Constraints

🏝 TheTechOasis 🏝

part of the:

In AI, learning is winning.

This post is part of the Premium Subscription, in which we analyze technology trends, companies, products, and markets to answer AI's most pressing questions.

You will also find this deep-dive in TheWhiteBox ‘Actionable Insights’, for future reference if necessary and to ask me any questions you consider.

Although all my sources are cited, I want to highlight one specifically: Semianaysis; it has really helped me go as deep as I needed into the matter.

With NVIDIA finally claiming the most important throne in capitalism as the most valuable company in the world for a few days, we seriously need to discuss AI’s elephant in the room.

It’s not NVIDIA’s insane run;

it’s not the Hyperscalers’ pathetic AI revenues;

and it’s not a potential Chinese blockade in the Taiwan Strait.

Despite all these being obvious issues that must be addressed, the biggest looming threat in AI, even acknowledged by Sam Altman, Bill Gates, and Mark Zuckerberg, is energy constraints.

Today, we are exploring this topic nose-deep.

By the end of this piece, not only will you be much more aware of the biggest threat in AI, but you will also have a much richer understanding of the functioning of some of the biggest corporations in the world (and prominent players in the space) and some numbers and facts that will become a central topic of discussion soon and a key driver in your future investments.

AI Unit Economics

First, I want to give a high-level overview of AI's unit economics to set the stage.

As you probably know by now (please read my deep dive on NVIDIA otherwise), the world’s AI runs on GPUs.

GPUs excel at matrix multiplication, the crucial operator required to run Large Language Models (LLMs). This allows the entire process to be parallelized, ensuring great performance and competitive latency.

However, GPUs are extremely energy-hungry. But how much and how unprepared are we for what’s to come?

A Worrying Comparison

Today, standard web and file storage server nodes require an average of 1 KiloWatt of energy. On the other hand, a state-of-the-art GPU node packed with 8 NVIDIA H100s requires upwards of 11 KW, a ten KW increase.

Although NVIDIA claims the consumption of a node is 10.2 KW maximum, it grows larger when you factor inefficiencies.

These numbers probably don’t say much right now, but they will in a minute.

Scarily, unlike traditional IT equipment, running costs are a measly part of the overall picture due to the insane capital costs of owning GPUs, at an average beyond $20k for NVIDIA H100s, the state-of-the-art.

Naturally, this tenfold increase translates into a much higher average energy consumption per request, with a Google query consuming around 0.3 Wh, and a ChatGPT request around 2.9, according to the Electric Power Research Institute (EPRI).

Concerningly, according to estimates, Google’s AI-powered search, called AI overviews, may increase the average consumption to almost 9 Wh, 30 times that of a standard query.

Enjoyed the introduction? Subscribe or sign in to continue with this edition.