OpenAI's Secret Leaked? DevDay, Pika 1.5, & More

In partnership with

THEWHITEBOX

TLDR;

🫡 Here’s All You Need to Know from OpenAI’s DevDay

⚡️ Pika 1.5 Shocks the World

🍎 Apple Announces MM1.5

😍 NVIDIA Crushes it with the NVLM Family

💽 The State of Semiconductors

[TREND OF THE WEEK] Self-Correction LLMs. OpenAI’s Secrete Leaked?

OPENAI DEVDAY

Here’s All You Need to Know of OpenAI’s DevDay

News from yesterday: OpenAI finally announced its new funding, $6.6 billion, at a $157 billion post-money valuation (meaning that investors invested at a $150.4 billion valuation).

OpenAI held its Developer’s Day two days ago, announcing many new features. They announced a ‘Real-time API,’ a new API (the interface you use to interact with OpenAI’s proprietary models, which can’t be accessed directly) that allows you to create voice products similar to Advanced Voice Mode (AVM), where you can talk with an AI that feels human thanks to its real-time latency.

If you haven’t heard or seen how incredible AVM is, I recommend this hilarious clip where the model imitates an Indian scammer (before you get offended, the user who published it is also Indian, it’s just a fun joke).

OpenAI also showcased a real-time demo of this API during DevDay.

They also introduced vision fine-tuning. Now, you can fine-tune your image models with your images, tailoring the experience. Furthermore, they introduced prompt caching between different API calls for an undisclosed amount of time.

But what does that mean? As we have explained several times, during inference, LLMs generate the KV Cache; they store part of the computations that are repeated between each new predicted word to avoid recomputation.

This has two effects: cost savings, as the model has to perform fewer calculations to obtain the new predictions, which in turn results in less latency. However, the KV Cache is usually particular to a given interaction. This is an issue because multiple API calls might refer to the same context or might be very similar among them.

Thus, now OpenAI stores this cache in a separate disk array and, upon receiving a new request, first checks if there’s a cache hit (if the question is related to content already cached). If so, it retrieves the cache, saving considerable costs and providing a better user experience.

Finally, they also released the most interesting new feature, model distillation. We’ve often discussed distillation as the tool that created powerful ‘medium-sized’ models that are now state-of-the-art, like GPT-4o, Claude 3.5 Sonnet, or Llama 3.1 70B. Now, OpenAI allows you to use GPT-4o responses (and I assume o1 models, too, in the future) to train a new GPT-4o mini model.

The following thread collapses all demos into one post.

TheWhiteBox’s take:

OpenAI is steadily becoming a super Generative AI platform. Having left their ‘God AGI’ nonsense behind, they have now become a full product company and are shipping valuable features.

The real-time API is clearly a differentiated product. For now, the category of use cases that require speech-to-speech interactions is entirely owned by OpenAI, as the user experience feels…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.