Meta's Self-Rewarding Models & The AI Global Tax

Sponsored by

🏝 TheTechOasis 🏝

Breaking down the most advanced AI systems in the world to prepare you for your future.

5-minute weekly reads.

TLDR:

AI Research of the Week: Meta’s Bet on SuperHuman Models

Leaders: The Threat of a Global AI Tax

🤩 AI Research of the week 🤩

Meta, the company behind Facebook, Whatsapp, and Rayban’s Meta glasses, has announced a recent, highly promising AI breakthrough, Self-Rewarding Language Models.

Their results have allowed their LLaMa-2 70B model to surpass models like Claude 2, Gemini Pro, and GPT-4 0613, despite being at least an order of magnitude smaller.

However, that is not the true breakthrough, as these new models also show signs of being a reasonable path to achieving superhuman abilities by eliminating the human from the equation.

But what does that mean? And is that even a good thing?

Let’s find out.

The Rise of a New Alignment Method

To this day, in all frontier models like ChatGPT, or Claude, humans play a crucial role in their creation.

As explained in my newsletter from two weeks ago, the later stages of the training process of our best language models include human preference training.

In a nutshell, we make our models achieve higher utility and reduce the risk of harmful responses by teaching them to respond how a human expert would have.

The previous link goes into much more detail, but the gist is that we have to build a very expensive human preferences dataset, which essentially are two responses to any given prompt where a human expert has decided which one is better.

Source: Anthropic

This is very expensive, requiring extensive human—expert—labor.

At this point, you have to take two possible directions:

Source: DPO research paper (Rafailov et al)

Tangible Rewarding through Reinforcement Learning from Human Feedback (RLHF), where you build a reinforcement learning pipeline that requires a reward model (of the same size and quality as the model being trained) and use a policy optimization process where the trained model learns to maximize the reward, aka you measure the reward model’s score on your model’s responses and you train it to achieve higher scores.

Intrinsic Rewarding through Direct Preference Optimization (DPO), where the model optimizes against the optimal policy instead of materializing the reward. In other words, the model implicitly maximizes the rewards by explicitly optimizing the optimal policy. In simpler terms, you simply perform an algebra trick to avoid having to materialize a reward (which would require a reward model) by directly learning the policy that maximizes that reward.

Although very new, DPO is already being cited as a major breakthrough, as it shows equal or even better results than RLHF, while being dramatically cheaper and easier to build.

But Meta has taken DPO and gone a step further with the question… do we actually need humans?

The SuperAlignment Problem

Albeit their undeniable credentials, both RLHF and DPO are still bottlenecked by us, humans.

The reason is that they require the human preferences dataset, which means…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.