Harnesses Are the Stars, Crypto risks, & a market that doesn't get it

TLDR;

Welcome back! This week, we have several market-related news items, as it was a big week for public company earnings, as well as considerable excitement around agentic harnesses, which are proving to be instrumental to driving AI progress. We’ll also cover how OpenAI has started the next chapter of the AI price wars.

Enjoy!

AGENTS

Daily Reminder that Harnesses Are As Important As Models

I’ve already talked multiple times, even very recently, about how important harnesses are to AI performance these days.

The harness is basically the system that governs the AI’s behavior. It decides what the AI sees (context engineering), how it sees it, the available tools the model can use, and so on.

It’s not AI; it’s the system built around it.

In reality, the ‘model’ is just a piece of software that receives a sequence of text and returns text back. But the magic is largely about the quality of what you give them—to this day, AIs remain perfect ‘GIGOs’, Garbage In Garbage Out systems, so handling context is vital.

The latest evidence of this comes from OpenAI’s scores on ARC-AGI 3, what’s possibly the hardest AI benchmark in existence. With two simple harness changes, the model went from just over 10% score to almost 40%. To be clear, it’s the exact same AI, just with a slightly modified harness.

But how is this possible?

The uniqueness of this benchmark is that it’s meant to test an AI’s ability to solve problems it has not seen before on the spot; it has to interact with an environment, figure it out, and learn how to solve the puzzle. No instructions, no rules, just environment feedback.

Doing this, we’re testing whether the AI can actually reason, not just plainly memorize and solve tasks not because it can think through them, but because it simply remembers them.

The problem was that the AI model failed miserably, which is surprising considering it’s the same model that is pushing maths discovery forward. How is it possible that an AI capable of such impressive discoveries fails at a puzzle your young cousin can solve?

And while it’s known that current AI models have “spikey intelligence,” meaning they exhibit incredible performance in given areas and very modest performance in others, the real issue here was the harness. Let me explain.

AI models have what we call a ‘context window,’ a limited amount of context they can handle at any given time, like the working memory (think of this as similar to you being unable to hold too many thoughts at one go).

These windows have grown pretty large to one million tokens (approximately 750,000 words), but they are still exhausted pretty easily on hard tasks. This has led to one of the most common harness patterns: compaction.

The idea, illustrated in the thumbnail, is to, upon reaching the end of the context window, create a compressed summary of all that happened, and create a new context window that starts with this summary. This…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.