LLaMa 2's release changes the GenAI landscape

🏝 TheTechOasis 🏝

In the biggest news of recent months, Meta has launched the second version of its world-famous LLM, LLaMa, and in the process has also released its first chatbot, LLaMa-2-Chat.

But this isn’t your ordinary “look how cool our new LLM is” type of release, Meta is actively trying to change the AI narrative.

Forever.

In fact, I would say that this release could indeed change AI permanently, and kickstart an era where AI access and knowledge are, finally, democratized.

The Open-Source era.

A new state-of-the-art for open-source

First things first, if something becomes clear after reading the 70+ page paper is that LLaMa 2 is incredibly good.

Trained with a 40% larger dataset, it also doubles its context window to 4k tokens (3,000 words approximately).

Quality-wise, as shown in the image below, the LLaMa 2-Chat 70-billion-parameter model beats basically everyone it competes against, being slightly better than ChatGPT despite being much smaller, and is unequivocally superior to any other open-source model.

Naturally, it’s still inferior to GPT-4 (not shown in the image), but we’re talking about a model that could be easily 20 times bigger, so that isn’t surprising.

But the main protagonist of this research isn’t how good this model is, but how much detail they’ve put into explaining how it was trained.

Meta has dropped several gems that are too good not to talk about.

Safety first

The first “different” thing Meta did here is to optimize for helpfulness and harmlessness separately, creating what’s possibly the safest chatbot there is right now.

To do this, they trained LLaMa-2-Chat using two reward models instead of one.

Let’s go over this beautifully drawn diagram:

Training a GenAI chatbot involves four steps:

First, we train a base model by optimizing it to predict the next token in a sequence of text. This is the most expensive part because involves ingesting the “entire” Internet text into the model.

This pretrained model is then fine-tuned with a curated dataset of {prompt, desired answer} pairs. Dubbed ‘behavior cloning’ by OpenAI, the model here learns to behave in a desired way. This is the first version of LLaMa-2-Chat.

Next, we want to optimize the model against human preferences, like following instructions, while becoming much less prone to harmful responses. Using a copy of the previous model, we ‘chop off’ its prediction head to, instead of predicting the next word in a sequence, output a scalar value to how good a response was to a certain prompt, according to human preferences. This is called the Reward Model (RM).

Finally, we train LLaMa-2-Chat against this reward model with the objective of maximizing the reward. In other words, the chatbot learns to write responses to prompts that yield the highest value possible according to the RM.

And that gets you the final LLaMa-2-Chat model.

But if you watch the image carefully, you’ll realize that Meta created two RMs, one optimized for helpfulness, the other for safety.

This is a first in AI.

The reason for…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.