Elon's Master Plan, China's Retaliation, & More

TLDR;

Technology:

😍 Democratizing Private Evals

🛢️ Using AI to Find Old Oil Wells

🤯 Finally Making Quantization Work

Markets:

😳 Elon Wants to 10X the World’s Largest Supercomputer

😱 China Opens Anti-Trust Case on NVIDIA

🥸 What Companies Does OpenAI Rely On

Product:

🥳 Finally, OpenAI Releases Sora

👾 Gemini Rules

VIDEO GENERATION

Democratizing Private Evals

Databricks has introduced new synthetic data capabilities to enhance the evaluation of AI agents within its Mosaic AI platform.

These features enable developers to generate synthetic datasets from existing document corpora, facilitating the assessment of AI applications’ quality, cost, and latency.

By integrating synthetic data generation—and evaluation—directly into the Databricks environment, the process becomes more streamlined, eliminating the need for external tools and complex data transfers.

TheWhiteBox’s takeaway:

One of the biggest performance boosters an enterprise can leverage when adopting Generative AI is developing its own private evaluation datasets. I can’t emphasize this enough.

Assuming you know by now that fine-tuning an open-source model on your data is the most secure and best quality/cost deployment you can aim for, it’s of little use to test your fine-tuned model on public benchmarks.

Instead, you want to build an internal dataset tailored to that model and task and check whether its performance (especially in matters of robustness, so you should aim for ‘pass@k’ datasets) is good enough.

However, anyone who has worked in a company and deployed a technological solution knows how painful it is to get your subject matter experts (SMEs) to help you build the dataset and execute the testing phase seamlessly while coordinating the inputs of potentially hundreds of people, and then iterate over the results.

A ‘pass@k’ dataset is one where each test needs to be executed correctly ‘k’ consecutive times to count as a valid pass. That way, we test the model’s robustness.

Here, leveraging synthetic data, Databricks proposes creating the datasets using synthetic data based on your proprietary data, turning this weeks-long process into “minutes.” Of course, you might want to run this dataset by several SMEs before actual testing, but building the dataset is the longest part, and this Databricks API does just that.

Private evaluation dataset generation APIs, whether through Databricks or other vendors, are among the things most companies will adopt in 2025 (unless they are clueless).

DISCOVER WITH AI

Using AI to Find Old Oil Wells

In the 3 million square miles of contiguous US territory, 170 years of oil digging have produced hundreds of thousands of oil wells. However, many of them do not appear in formal records, leading to a plethora of oil wells that we don’t currently know exist.

If not plugged correctly, these wells can contaminate subterranean waters and release methane. Methane is 28 times more powerful at trapping heat in the atmosphere than carbon dioxide, negatively contributing to climate change.

Now, a group of researchers at Berkeley Lab have found a way to use AI to find these hidden wells by scouring thousands of old maps uploaded by the US…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.