NVIDIA's Self-Sabotage? Mapping Sound to Sight, & More

TLDR;

Technology:

🧐 HuggingFace’s Test-Time Compute Analysis

🤩 Translating Sounds into Sights

😩 Pre-training is Finally Dead

Markets:

🤨 Frontier AI Labs Fail Hard on Safety

😟 NVIDIA Self-Sabotage

Product:

🤩 We Need to Talk About Google’s Veo2

🫡 Pika2.0, a New Super Rival to Sora

TEST-TIME COMPUTE

HuggingFace’s Test-Time Compute Analysis

HuggingFace, the most famous open-source AI company that holds the biggest repository of AI models for download and even lets you run the models on dedicated resources, has launched an analysis on test-time compute, the idea we’ve talked about for a while where we give Large Language Models (LLMs) “more time to think.”

In their analysis, they find that this technique unequivocally improves a model’s performance, even taking a Llama 3.2 1B and 3B models, considered ‘minute’ in current standards, to beat their larger brothers, Llama 3.1 8B and 70B respectively, in the MATH-500 benchmark.

Of course, we need to be careful about this statement, as Llama 3.2 3B required 256 generations per benchmark question to beat the overall performance of Llama 3.1 70B. However, the point still stands: test-time computing makes your models better in “reasoning” tasks like math or coding.

The blog post then comments on the different test-time compute methods used and how they compare.

TheWhiteBox’s takeaway:

The biggest thing to highlight here is that to build the test-time compute factor into models, HuggingFace discusses the very same points we detailed in our discussion on OpenAI’s strategy:

The use of a Process-Supervised Reward (PRM) model, the model that scores not only the final model response but also the intermediate steps, which we described as crucial for o1’s performance.

The idea of self-consistency in one of two ways: by majority vote or best-of-N. When generating multiple samples to solve a problem, a majority vote picks the most frequently generated answer. On the other hand, Best-of-N uses a reward model to choose the best response out of the ‘N’ responses that the model generated to answer, the latter of which, as we mentioned on Thursday, is what OpenAI’s o1 models use.

The fact that HuggingFace mentioned the exact same methods I did on Thursday doesn’t make me a genius. Instead, it proves my point I always mention to argue why open-source will win: you can’t hide technology behind closed doors, and no matter how much you try to protect your technology moat, people will always manage to reverse engineer your models as HuggingFace did.

MULTIMODALITY

Translating Sounds into Sights

A research team at the University of Texas at Austin (which happened to be my alma mater for six months back in 2018), has published a fascinating AI model that turns street sounds into images that reasonably match the sound.

The model was fed a huge dataset of street sounds and matching street images, and the AI model learned the mapping between the sound and the image that most reasonably represents that sound.

As seen above, the model impressively generates images fairly similar to the ones with precise…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.