Silicon Tears, A Preventable Crash, New Rounds, & More

TLDR;

This week, you’ll understand why DeepSeek’s cost efficiencies are massively overblown {🥵 OpenAI Accuses DeepSeek of Stealing} but also how SV incumbents are nevertheless panicking and openly asking the US Gov to save them {🌏 Anthropic CEO Calls For Cold War Escalation}.

We will expose the hypocrisy some tech tycoons have regarding AI safety while actively helping the DoD make war with AI {🫣 Pentagon Claims AI Speeding its ‘Kill Chain’}, and I’ll also give you my reasons as to why OpenAI’s latest valuation feels very hard to justify {🌚 OpenAI Eyeing a $300 Billion Valuation}.

Finally, we will see how the open-source community, both Chinese and from the US, continue to push the entire space forward {📦 Alibaba Ups the Ante} and how, with just $30, you can prove that DeepSeek’s recipe just works {🤩 Building Zero at Home for $30}.

Enjoy!

CHINA

OpenAI Accuses DeepSeek of Stealing

The word is out. OpenAI claims to have proof that DeepSeek used distillation to train v3, which would explain the extremely low training budgets. David Sachs, the Trump Administration’s AI and Crypto Tzar, made this very explicit accusation on live TV based on the SV start-up's alleged proof.

TheWhiteBox’s takeaway:

How does one word, distillation, change the entire picture? NVIDIA’s crash can be summarized as 'China has built a model as good as Western ones for 20x less money, aka GPU overspending is finally proven. Sell NVIDIA!' First and foremost, even if distillation weren't in the picture, that statement would still be false due to inference-time compute, but I digress. But this turn of events takes the opinions of AI influencers and investors who compared both models to a new level of public embarrassment because it's an apples-to-oranges comparison. But why? And what is distillation? Also known as the teacher-student method, it introduces a new term to the loss formula (KL Divergence) that 'forces' the model (the student) to imitate the teacher's behavior by cloning its output distribution (accessible through the OpenAI API top_logprobs parameter) while still learning to predict the next word in a sequence (the standard cross-entropy objective).

And what does all that even mean?

As you can see below, we can access the ‘top k most likely words’ at every sequence step. Thus, what the distillation method does is to ensure that (using the example below) not only ‘measuring’ is chosen by v3 if ChatGPT is likely to choose it, but also to ensure that ‘standing’ comes second, ‘with’ third, and so on, and with as similar probabilities as possible.

In other words, it teaches the student model to handle tasks like the teacher would, to generate sequences 'in the style' of the teacher model. And why is this a training shortcut? Think of it this way: Comparing the training of v3 with GPT-4 is like comparing the learning curve of a student learning by itself (GPT-4) with one that is using a teacher (ironically, GPT-4) to teach the student (v3). Thus, while DeepSeek's results are remarkable…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.