OpenAI Changes the Game & The Arrival of the King: LLaMa 3.1

TLDR;

🗞️ News from Numina, OpenAI, Mistral, & SuperMicro.

📚 New short courses, instant LLMs, & more.

💰 OpenAI Changes the Economics

🦙 Meta Releases the New King, Llama 3.1

THEWHITEBOX

Weekly News 🚨🚨

Welcome back!

To start the newsreel strong, it’s not common to have the greatest mathematician alive, Terence Tao, be ‘surprised’ by the results of an AI. But this is what the Numina + HuggingFace team has achieved in the AIMO competition, an Olympic-level problems maths competition.

The main method used was fine-tuning the model with reasoning-augmented data. In other words, just like we covered in our piece ‘Is AI really intelligent?’, one of the most promising avenues of reasoning research is creating synthetic data that helps models learn better reasoning thought processes.

Casually, this method has also been used in training the new supermodel LLaMa 3.1 (more on that below).

Moving to hardware, Supermicro, an AI hardware company that produces racks, liquid cooling, and other essential components that GPUs need to work, claims that its new cooling system ensures up to 40% less power consumption for an equal budget.

Nonetheless, the company has grown 200% year-over-year and is expected to grow an additional 152%.

You would be surprised how important these systems are and how much energy they require. For reference, an NVIDIA H100 GPU has a Thermal Design Power (TDP) of 700W (the power it needs to work).

Yet, the total per-GPU consumption of H100 clusters is on average double that value, as all the additional components (like networking or cooling), add almost another 700W to your bill.

With the new Blackwell GPUs intented primarily for liquid cooling (you could even say required, as Blackwells will reportedly increase the per-rack KW demand to 40KW), SuperMicro is having a lovely time these days.

Finally, if Meta’s blowout release below wasn’t enough, Mistral announced the release of Mistral 2 Large yesterday.

At just 123 billion parameters, Mistral claims it’s on par with Llama 3.1 405B, GPT-4o, and Claude 3.5 Sonnet on math and code while being much more efficient (or pareto-optimal, the corny way researchers describe this).

This release must not be ignored, as the model efficiently runs on a single H100 node (8 GPUs) despite having frontier-level capabilities.

This means the model’s entire GPU foundation communicates through NVIDIA’s 900 GB/s NVLink bandwidth, crucial to reduce latency (node-to-node communication is done through NVIDIA Infiniband, dropping communication bandwidth to 25 GB/s, or 36 times slower).

In other words, staying under 8 GPUs is essential. While at first Llama 3.1 405B might need two nodes (or 16 GPUs) at FP16 precision (810GB), Meta released a quantized FP8 version that occupies 405B, which means Llama 3.1 405B can also be run on a single node.

The era of efficient frontier models has arrived.

LEARN

The Insights Corner

🧐 The Creator of Mamba and how it could change the AI paradigm

😍 Want to learn how to pre-train an LLM? You can do so with Deeplearning.ai’s newest short course.

🤩 Witness…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.