An Alignment Revolution & Our Future According to Experts

🏝 TheTechOasis 🏝

Breaking down the most advanced AI systems in the world to prepare you for your future.

5-minute weekly reads.

TLDR:

AI Research of the Week: DPO, An Alignment Revolution

Leaders: What the World’s Best Think About Our Future

🤩 AI Research of the week 🤩

“It is only rarely that, after reading a research paper, I feel like giving the authors a standing ovation.“

If this is how one of the most prominent researchers in the world, Andrew Ng, refers to a recent research paper, you know it’s awesome.

A group of researchers from Stanford and CZ Biohub have presented DPO, a new alignment breakthrough that could give back to the open-source community the capacity to challenge the big tech companies, something thought as impossible… until now.

The Rich Game

When one looks at the numbers, it’s easy to realize that building the best Large Language Models (LLMs) like ChatGPT is a rich people’s game.

Building ChatGPT

The current gold standard is as follows:

Source: Chip Huyen

You first assemble billions of documents with trillions of words and, in a self-supervised manner, you ask the model to predict the next token (word or sub-word) to a given sequence.

Next, you want to teach the model to behave in a certain way, like being able to answer questions when prompted.

That takes us to step 2, where we assemble a dataset where the model performs the same exercise, but on a curated, dialogue-based form, becoming what we define as assistants.

But we not only want the model to behave in a certain way, we also want it to maximize the quality and safety of its responses.

Reinforcement Learning from Human Feedback, or RLHF, steps 3 and 4, does exactly that by teaching the model to not only answer as expected but to give the best and safest possible answer aligned with human preferences.

However, there’s a problem.

Money goes brrrr

Put simply, RLHF is very, very expensive.

It involves three steps:

Building a preference dataset. By prompting the model in step 2, they build a dataset of {prompt, response 1, response 2}, where a group of humans decides which response is better, 1 or 2.

Training a reward model. Step 3 in the previous diagram, you build a model that learns to predict, for every new response, how good is it. As you may imagine, we use the preference dataset to train this model, by training it to assign a higher score to the preferred response out of every pair.

Maximize reward. Finally, in step 4 of the diagram, we train the model to maximize the reward. In other words, we train our model on a policy that learns to obtain the highest rewards from the reward model. As this model represents human preferences, we are implicitly aligning our model to those human preferences.

In layman’s terms, you are training a model to find the ideal policy that learns to take the best action, in this case word…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.