Orca, an open-source chatbot that rivals the king, ChatGPT

🏝 TheTechOasis 🏝

Everyone loves the idea of community-built AI chatbots that match or exceed the performance of privately-owned models like ChatGPT.

The thought is almost romantic; the power of the people defeating the tech giants back at Silicon Valley.

However, as leaderboards like Chatbot Arena prove time and time again, the billions-backed chatbots remain undisputed kings.

But now, ironically one of the tech giants, Microsoft, has presented Orca, an open-source, much-smaller-than ChatGPT model that, using an innovative training method, is the first model ever to look in the eyes of the private models to say, “We’re in the same league now”, as proven in the table below:

Despite being dozens of times (most probably hundreds in the case of GPT-4) smaller than the models it’s competing against, Orca even defeats them in some cases, while completely obliterating what was until now considered the best open-source model, Vicuna.

This raises the question:

“Will open-source actually win in the end?”

A new way of teaching

In an effort to try and match the performance of private, much larger models like ChatGPT, researchers have focused on applying the concept of distillation, the teaching method.

In simple terms, it involves using the responses from one of the big guys (it’s usually GPT-4), to teach a much smaller model to imitate the teacher.

The reason for this is simple.

Although the larger the model the better it’s capable to learn a larger corpus of data and, thus, achieving greater performance, larger models are often considered heavily underfitted.

In other words, you can achieve equally-good results with much smaller models.

Consequently, the ideal AI chatbot creation process among the open-source community tends to be as follows:

A very large model that has learned complex representations of huge amounts of data is used as the ‘teacher’. This model has been trained by private companies, like ChatGPT or Bard.

A much smaller, around 5 to 15 billion-parameter model is chosen as the ‘student’

Then, the student is tasked with minimizing the difference between its outputs and the outputs of the teacher, thereby learning to imitate it

This way, a much smaller model is capable of learning the style of the teacher and achieving ‘similar’ results, while being vastly cheaper to run

However, this default procedure has a problem.

While these models learn the style and language coherence of the teacher effectively (with examples like Vicuna, or Alpaca) they fail miserably to capture its reasoning capabilities.

In other words, when evaluated against complex tasks, they greatly underperform the teacher.

But Orca did things a little differently and ended that problem.

An inexplicable result

One of the main points of the research is that most open-source models overstate their capabilities.

For instance, while Vicuna reaches around 85% quality of GPT-4 outputs for style and language coherence, when measured against complex tasks, the gap widens to an almost embarrassing 5400% in cases like seven-step logical deduction problems, as shown in the previous image.

In other words, GPT-4 performs 55 times…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.