The Ultimate Guide for AI, Part II

Last week I wrote my longest newsletter ever, 8,500 words long, which covered the essentials of AI software and hardware, from the very basic key intuitions about AI models (what they are, how they learn), to the key intuitions in AI hardware that helped readers understand why GPUs and other accelerators are used, why memory is so important, and other key ideas the industry holds dear.

And finally, we discussed the “interesting” world of AI finance, particularly some numbers to understand how this entire industry aims to make money.

Today, I bring you the second part of the guide. In this one, I’ve focused more on the product, detailed AI inference maths, and future trends.

Let’s dive in.

What is an agent really?

Definitions of an agent are like opinions; everyone has one. Interestingly, despite everyone talking about it, most people don’t even understand what an agent is, when, in fact, it’s quite simple.

Loops and tools

I’ve talked about agents in detail in the past, so I won’t dwell too much on definitions, only the essentials. An agent has three components:

The AI model. Almost always a Large Language Model (LLM). Receives tokens, outputs tokens. Send new tokens; return new output tokens. It’s a continuous back-and-forth. Sometimes, the AI returns a tool call, a request to use a certain tool.

Memory. Also known as the context system, it’s responsible for handling the input tokens the AI receives. It determines what the AI receives and, based on the AI’s outputs, what must be remembered in future interactions.

Tools. These enable the AI to execute tools or actions.

At this point, it’s mandatory that we ask: why do agents need external memory in the first place?

We can understand the need for external tools (AIs can’t use tools directly; they are simply next-word predictions), but they do have internal memory and knowledge (known as parametric memory), so why the need for external systems providing extra context?

Doesn’t the AI know it all?

Statelessness and the continual learning problem

When you think about it, it’s pretty ironic that AIs, which have “seen it all,” need support from an external memory component; much like a human has Google search a smartphone unlock away, AIs need external memory too.

But why? And the answer is twofold: AIs are stateless and have a continual learning problem.

The first one is crucial to understanding the dynamics of what is going on under the hood when you interact with Gemini, Claude, or any LLM for that matter.

LLMs don’t have state. Every single interaction is like a new blank page in a book. Unless they’ve been trained on you, if the ChatGPT app doesn’t provide some context about you, when you click ‘New chat’ and you send something to the model, it’s like the first interaction it’s ever had with you. And if you click ‘New chat’ again, the model loses context from the previous conversation and starts anew with you.

What this means is that it…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.