Evolutionary Model Merging, New Models without Training

🏝 TheTechOasis 🏝

part of the:

In AI, learning is winning. While Thursday’s newsletter talks about the present, the Leaders segment on Sundays will make you aware of the future trends of AI with actionable insights to set you apart from the rest.

10-minute weekly reads.

🥇 This week on Leaders… 🥇

Leaders: Evolutionary Model Merging, New Models without Training

Did you know about… Google’s New Discovery, many-shot learning?

Welcome back to Leaders, the premium segment that brings light and prepares you for the future of AI.

For today’s segment, what if I told you that you will create the AI models of the future?

Sounds hard to believe, but this is the promise Evolutionary Model Merging (EMM) intends to deliver.

A group of Japanese researchers has found a way to automate model merging (creating new models without actual training), opening the door to a world where everyone gets to create their models without relying on the ever-more powerful incumbents.

It allows the creation of unique models, often bearing knowledge from highly unrelated domains and boosting the creation of models in underrepresented areas like non-English languages and minority cultures.

Consolidation And a Scary Sight

A quick overview of the scale increase that data and compute have seen over the past few years provides a stark image of how demanding AI is becoming and how embarrassingly far we are from meeting future requirements.

A Great Explosion

Back in 2022, Google Deepmind released a paper on measuring the optimal compute requirements for LLM training.

For an 8-billion parameter model, they recommended around 200 billion tokens of data, which amounts to around 150 billion training words.

Now, just two years later, Meta’s Llama 3-8B was recently trained for up to 15 trillion tokens, or 11,25 trillion words.

That’s 75 times more training data.

Compute-wise, according to Andrej Karpathy, the model consumed around 2e24 FLOPs (that’s 24 zeros), which is one hundred times more FLOPs than what Chinchilla (the aforementioned paper) recommended.

FLOPs are floating-point operations per second, the total amount of simple calculations the GPUs training Llama did. It’s the most straightforward way of measuring LLM training throughput and consumption.

While it’s important to note that the number is slowly approaching the embarrassingly random yet still mentioned 1×1026 threshold that Joe Biden’s AI Executive Order set for training AI models without government overview, it’s safe to say these numbers will be dwarfed shortly.

But why? Well, for two main reasons: the data effect and the compute paradigm shift.

Data and Thinking Modes

All Deep Learning today is based on imitation learning. Specifically, generative AI models are trained to replicate the training data. Thus, the higher quality, or ‘smarter’, the data is, the better the results.

And with the exponential increase of synthetic data, data quality will increase over time, as proven by models like Microsoft Phi-3, models that are already at the ChatGPT-3.5 level (the original ChatGPT) despite being in the 10-billion parameter range, as we discussed on Thursday.

Yet, scale also continues…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.