Mixtral of Experts, the New Open-Source King, and why 2024 will be the year of AI Robotics

🏝 TheTechOasis 🏝

Breaking down the most advanced AI systems in the world to prepare you for your future.

5-minute weekly reads.

TLDR:

AI Research of the Week: Mixtral of Experts, Europe’s AI Racehorse

Leaders: 2024, the Year of Robots

🤯 AI Research of the week 🤯

Just like any other week these days, a new open-source model has come out.

But this time, things are different.

Not only the model has been released in a very geeky fashion through a peer-to-peer torrent network.

The model itself is, well, different.

Emulating one of the core features that turned OpenAI’s GPT-4 into the world’s most advanced model (with the excuse of Gemini Ultra), Mistral’s new model, Mixtral 8×7B, is the first open-source Sparse Mixture-of-Experts foundation model that is as impressive as it is highly performant, making it the best open-source model to date.

But it isn’t stopping there, as it is up to six times faster than models of its size, making it the best model in the world in terms of performance relative to cost and speed.

Europe seems to have found its AI champion, and today we are going to make sense of this engineering marvel.

The Feedforward Layer Problem

When you are at the frontlines implementing Generative AI, you know how to cut through the bullshit.

If you just look at what journalists and bloggers alike say, you will think that ChatGPT et al are already vastly used around the globe.

Far from it.

Just do the numbers

Yes, the users can be counted in the millions, but if you look at what corporations are doing, to this date, the numbers are far more modest.

Sure there’s interest, but most Generative AI projects bump into the same obstacle:

Costs.

Costs can be divided into two:

Hosting costs: How expensive is it to have your model in a GPU cluster

Inference costs: How much does it cost to run the model

Regarding the former, if we take LLaMa 2 70-billion-parameter model, it has a float16 precision. In other words, each parameter occupies 16 bits in memory, or 2 bytes.

Thus, if we have 70 billion parameters, that means that the weight file occupies 140 GB of memory.

Taking the new NVIDIA GPU, the H200, it has a capacity of 141 GB, which means LLaMa 2 almost doesn’t fit in a state-of-the-art GPU. However, you need to account for inference costs, so you will need at least another one.

Best practices suggest up to 4 times your nominal capacity, so you would need 4, which in H200 that means at least $100K of CAPEX investment, and that’s in the low-end of the spectrum for just one instance, without counting the actual hosting costs.

Regarding inference costs, the issue comes when you realize that in Transformer-based models (basically all NLP foundation models today), for every token prediction, the entire model is queried.

For. Every. Token.

This is why we need to store the entire model in memory in the first place.

Enjoyed the introduction? Subscribe or sign in to continue with this edition.