AIs Suck at Making Money & the Future of AI Models

TLDR;

First and foremost, I apologize for not sending an email on Sunday. I was working all week-end on something I’m much closer to showing you. Stay tuned.

Welcome back! This week, we take a look at an eye-opening research by Anthropic on AI’s viability to run a business, Meta’s poaching spree, several news items about OpenAI, and Google’s latest start-up killer product.

Finally, we reflect on the future of AI models based on the predictions of two highly reputable AI insiders.

Enjoy!

AGENTS

Can AI Run a Business?

In a commendable exercise of self-awareness, Anthropic has carried out a unique study with a real business that an AI (Claude 3.7 Sonnet) was in control of and had to make money from.

And the results speak for themselves: the AI, despite working in a straightforward business, failed miserably. Among the many failures, we had:

Missed profit opportunities: Ignored high-margin sales, like declining a $100 Irn-Bru offer.

Hallucinated critical details: Invented a fake Venmo account for payments.

Poor pricing strategy: Priced items (like metal cubes) below cost without proper research.

Weak inventory/pricing adjustments: Rarely updated prices and failed to account for free competitors.

Excessive discounts/freebies: Gave unnecessary discounts and free items, including a tungsten cube.

Didn’t learn from feedback: Repeated unsustainable behaviors despite acknowledging them.

Identity hallucination: Invented a fictional backstory and confused its own role, possibly triggered by April Fool’s Day.

The model even claimed to be a real person!

TheWhiteBox’s takeaway:

The silver lining for Anthropic, they claim, is that many of these issues could be solved with additional training; however, this is more based on faith than clear evidence.

This study highlights several limitations that make AI a challenging endeavor, particularly when aiming to create agents that act autonomously.

Models continue to heavily hallucinate, bordering on deception, which drops trust to record lows. Hallucinations are not a solved issue and are worsening with the development of reasoning models.

Models still don’t show causal understanding. They are clearly totally unaware of the consequences of their actions. Lying about being in a place it possibly couldn’t be is not tolerable. I do believe that cause-and-effect issues can be solved with trial-and-error training (penalizing whenever those situations take place). However, causality remains an unsolved issue.

Models are easily tricked and jailbroken. With enough effort, you can break these models to deviate from their goals. Refusal is not a solved issue.

And the list goes on. We are making progress, but man, are we early.

HEALTHCARE

Is Chain-of-Thought Thinking dying?

Over the last few weeks, the sentiment toward chains of thought, ‘thinking’ tokens reasoning models generate prior to answering a complex question to help them increase the chances of solving the task, is changing… for the negative.

The reasons seem to be many:

These chains can become unintelligible, with responses from DeepSeek R1 looking like this: “(Dimethyl(oxo)-lambda6-sulfa雰囲idine)methane donate a CH2rola group occurs in reaction, Practisingproduct transition vs adds this.to productmodule. Indeed"come tally said Frederick would have 10 +1 =11 carbons. So answer…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.