OpenAI has Shown Us the Next Design Paradigm for ChatGPT

🏝 TheTechOasis 🏝

Although Generative AI is all the buzz and something thought of as new thanks to the release of ChatGPT back in November, it’s actually not that new.

In fact, Large Language Models (LLMs) have been around since June 2018, when OpenAI released GPT-1.

However, it took them more than four years to launch ChatGPT, the first commercial release for GenAI.

But why?

Simple, because another innovation was required to bring GenAI into the masses, and OpenAI was the first to perfect it.

Now, they have just released the evolution of that breakthrough, and the results it shows turn ChatGPT, the world’s most advanced AI chatbot that sucks big time with maths, into a world-class mathematician.

The Half-True Definition of LLMs

When people refer to ChatGPT, they describe it as an AI model that predicts the next word in a sequence.

And I’m one of those guys when I want to keep the definition short.

However, that’s a nuanced statement, because if we want to be purists, that’s the definition of a base LLM.

ChatGPT is much more than that, as when you just train a model to predict the next word in a sequence, it’s basically useless.

For instance, let’s say you ask it “What’s the capital of France?”

ChatGPT automatically responds “Paris”, but this base model could respond with “What’s the capital of France riddle”.

Wait, what? How does that make sense?

It’s actually pretty darn simple:

A base model has been trained with one simple task, predicting the next word.

That’s a simple probabilistic calculation; the model is simply giving you back the most probable answer.

And as these models are trained by handing them a huge part of the Internet’s text, the response “What’s the capital of France riddle” will probably be just as common as “Paris”, because the Internet is far from being a perfectly-written novel all along.

And what’s even more worrying, base models have no filter whatsoever, and they easily became racist, homophobic, and hate-speech machines, making the interacting experience a very unpleasant one.

So what did OpenAI do to turn this unusable monster into the ChatGPT the world marvels upon?

Humans are the Answer

Leveraging a discovery by Paul Christiano et al, they decided to train the model to output its responses in the best way possible.

The process was simple.

They would train another model to score the outputs given by the base model.

To do so, they assembled a team of humans that sampled the base model’s responses (when ChatGPT was first released, this base model was GPT-3.5) multiple times for a specific question/instruction.

Then, those human engineers would score the responses, so that it was clear which answer was most useful, or the least racist, for instance.

With that, they trained a reward model that learned to predict, for a set of responses to a particular instruction, which one would have received a higher score by the human engineers.

That way, this model became an artificial representation…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.