Apple Against its Own Demons

🏝 TheTechOasis 🏝

Breaking down the most advanced AI systems in the world to prepare you for the future.

5-minute weekly reads.

TLDR:

AI Research of the Week: Apple Might Have a Real AI Problem

Leaders: Is The Future of AI Becoming a Deadlier Ad Weapon?

🤩 AI Research of the week 🤩

For months, everyone in the AI industry has been asking the same question: ‘What is Apple doing?’

We now know, and it’s not looking good. Or, put bluntly, it’s outright embarrassing.

But while some are, in pure sensationalist fashion, proclaiming this is the beginning of the end for Apple, I argue there’s room for hope, as the company of the bitten apple has just released what’s probably the biggest contribution to open-source AI in months, their MM1 model family.

Let’s start with the positive stuff before moving on to the very ugly part.

The MM1 Family, Apple’s Commitment to Open-Source

In its usual low-key way of publishing Generative AI research, Apple has silently presented the MM1 family, a set of very capable Multimodal Large Language Models (MLLMs) that achieve state-of-the-art results in many tasks.

But more importantly, they are the greatest contribution to open-source we have ever seen from a big-tech company.

Committed to openness

Training MLLM pipelines are some of the most well-guarded secrets in the world today.

Indeed, the coarse guidelines are pretty much known all around:

the use of the Transformer and the attention mechanism as the backbone for data distribution learning,

the global training steps (pre-training, and fine-tuning),

the alignment methods (both Reinforcement Learning-based like RLHF, or non-RL like DPO),

sparsity techniques like Mixture-of-Experts,

And so on.

Not only all previous methods are known but, more importantly, they are common across all LLMs.

But with such large models, the problem is much less of a theoretical one but an engineering and cost one.

In those situations, the devil lies in the details, and simple mistakes might imply losses in the millions of dollars.

Therefore, the small tweaks that some of the brightest minds of our time figure out behind closed doors in Silicon Valley are what set apart some LLMs from others.

Yet, the moat is not only based on human talent, but also on compute. Some tweaks can only be figured out by rote experimentation, something only the richest companies in the world can provide.

Consequently, some claim these companies have a moral duty to share their discoveries.

But, you guessed it, most don’t, which is why Apple’s recent research is so important.

The essence of MLLMs

In this case, Apple has taken the opposite approach, and they have released a paper where not only do they explain all the different ablations they perform (testing and retrying different alternatives to see what performs best), but they are totally open on the data used — both in pre-training and fine-tuning — as well as several conclusions they have reached during testing that could help open-source researchers enhance their work.

But before moving into…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.