Premium News Recap, Third Week of August

TLDR;

Technology:

😍 Forget Devin, Meet Genie

👩‍🔬Sakana Release AI Scientist

😣 Apple Puts Open-Source and Agents to Shame

Markets:

💸 The State of Venture Capital, Q2

📈 GenAI Efforts Boost Revenues by 6%

Product:

💕 Replika CEO says people will marry AIs

☢️ OpenAI Voice Mode Accidentally Clones User’s Voice

New Notion Content:

💾 Why LLMs are not intelligent, but Glorified Databases

AGENTS

Forget Devin, Meet Genie

Remember Devin? The LLM agent that amazed the world back in March, making people claim that software engineers had their days numbered?

Well, they aren’t the best anymore. Cosine, a company out of the legendary YCombinator accelerator program, has announced a new state-of-the-art for AI software engineers (video included in the link).

It achieved a 30% success rate on SWE-bench, a benchmark that measures how well agents can solve GitHub issues, almost triple what Devin claimed back in March.

TheWhiteBox’s take

After reading the founder’s tweets, I have a nice idea of what they built. This isn’t a new algorithmic breakthrough. This is an OpenAI model with an increased context window (a model not released to the public but provided by OpenAI to Cosine).

To achieve such impressive results, they did one of the steps to improve the reasoning we covered recently: data augmentation. In other words, they have built a dataset based on real tasks, actions, and procedures that real software engineers do, and the model is essentially replicating that behavior.

Importantly, the CEO confirmed that they did not use any ‘fancy’ stuff like Monte Carlo Tree Search (MCTS) or Reinforcement Learning, which we have covered multiple times here as ‘long-inference’ techniques:

That’s exactly it, good data is all you need! — Alistair (@AlistairPullen) 6:27 PM • Aug 12, 2024

Just better data. However, they did wrap the model on an agentic workflow (planning, retrieval, code writing, code execution) as mentioned in their “technical report.”

As for the model’s limitations, the CEO mentioned that they were limited by the ‘base model intelligence.’ They are referring to the fact that current LLMs are still very dumb, as we saw on Sunday. Also, let’s not forget that this benchmark only tackles GitHub issues, so no, this is not taking us anywhere closer to AGI.

AGENTS

Sakana Releases the AI Scientist

If we didn’t have enough with Genie, the AI Software Engineer, we now also have the AI Scientist, proposed by Sakana. This agent could be even more impressive, as it’s allegedly capable of handling the entire research process.

The company even released several papers (including one on Grokking, the trend we covered on Thursday) entirely written by the model (the link includes several examples you can read).

The model generated the idea, explored it, wrote down the experiments and the code, tested, peer-reviewed its findings and released the paper using the style and format common to such documents.

BENCHMARKS

Apple Puts Open-Source and Agents to Shame

In case you haven’t realized from the two previous news, AI is moving on to the next…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.