How Agents Are Shaping AI, Part 2, Hardware

Welcome to the second part of how agents are shaping AI, which is now a three-part series because I’ve realized there’s much to say here.

In this issue, we’ll focus on the hardware side, explaining how agents’ computational requirements push us to rethink not only accelerator design, but the data center as a whole.

We’ll do this by:

Reviewing how agents work and how they actually run in hardware. You'll visualize exactly what happens, giving you unique intuition for why elements like memory are so necessary. In other words, you probably know memory is a bottleneck, but you’ll see how true that is today.

Using that to explain the key remaining feature in the most innovative architecture ever released: DeepSeek v4.1 Flash

A detailed explanation of sandboxed agents like Muse, and why they are the first non-coding agent product I’m actually using.

So without further ado, let’s dive in.

How inference is ‘felt’ by hardware

Last week, I explained how agents change the computational workload considerably, moving AIs away from a decode-heavy workload to a prefill one, which are the two phases of inference.

Recall that prefill measures the time to obtain the first token, a computationally heavy workload because the agent has to process the entire input all at once. After that first token is produced, we cache much of the computation and use it to generate the following tokens more quickly, known as decode.

As explained last week, if with standard chatbots the average sequence had a smaller amount of input tokens and very long output responses, with agents the tide reverses; agents have to process enormous amounts of input tokens received not only from the user but from tools, databases, and the virtual machine and simply reason about what to ask for next.

In this paradigm, output tokens are much smaller by comparison, which means agents are mostly prefill-heavy.

And this has a profound impact on hardware. But to see why, we need to dive a little deeper into what is really going on.

AIs are databases… sort of.

Years ago, detractors of large language models (LLMs) defined them as ‘databases’ to argue they weren’t really intelligent, but simply algorithms that had compressed the world’s information into a smaller package. They couldn’t reason; it was just a ‘bag of words’.

Sadly for those people, the reinforcement learning age that started in September 2024 quickly eradicated that belief. However, the database analogy is still useful for understanding how the models are used for inference.

In a way, it’s still true and LLMs do behave like databases; general-purpose ones.

At heart, an LLM is a sequence-to-sequence model. It’s literally a huge mathematical function that takes in a sequence and produces the next part of the sequence. That’s all they are. You give them context and, in the case of agents, a state (i.e., the environment and progress of what they are doing), and they produce the ‘next thing’.

Thus, an agent is simply an LLM that can not only…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.