[Premium] An AI Tongue, the AI Inventor, Google's Nuclear Plans, & More

TLDR;

Technology:

😱 Intellect-1, The First 10-Billion-Parameter Decentralized AI

🚨 Zyphra Releases Zamba2-7B, SOTA for Small Models

🫢 OpenAI Shares o1 model’s Meta-Prompt

🤔 Backtrack Samplers

Markets:

☢️ Google Buys 500 MW of Nuclear SMRs

🤯 NVIDIA Is Living the Dream

⚡️ State of AI Report

Product:

👅 An AI Tongue to Identify Liquids

🤑 Using AI to Speed Up Inventions

DECENTRALIZED AI

Intellect-1, The First 10-Billion Decentralized LLM

Is decentralized AI training becoming a reality? That’s what some are striving to achieve with Intellect-1, the first 10-billion-parameter LLM being trained in a decentralized manner, currently underway as you read this newsletter.

After achieving the first milestone at 1 billion parameter model, the team is ready to make history again. But what is decentralized training?

Not to be confused with distributed training (training distributed between at least 2 GPUs, but usually done over thousands for frontier AI training), decentralized training refers to techniques in which a group of independent players get together to train an AI model instead of one player, like Microsoft or Google, administering the entire training.

This could lead to a future where independent researchers, or even laymen, get together to train open-source AI models that rival the size of the Hyperscaler training runs and even get paid for it.

But how does it work? This training leverages the Open DiLoCo distributed training algorithm and the Prime decentralized training framework, both heavily influenced by Google’s DiLoCo.

I’ll cover this fascinating research in full detail on Thursday’s trend of the week.

TheWhiteBox’s take:

If you believe in open-source, this type of news is significant because it paves the way for an AI future where everyone can participate and reap the benefits of this technology.

Of course, after hearing the term decentralized, you may wonder what blockchains' role is here, if any. While the Crypto industry is very scammy, blockchain has a clear and valuable raison d’être: it economically incentivizes decentralization, which offers attributes that centralized workloads can’t offer.

In distributed settings, the more decentralized a network is, the safer it is. Thus, through compute power (proof-of-work, like Bitcoin) or through asset staking (proof-of-stake, like Ethereum) the distributed nodes in the network secure it by actively validating transactions occurring in it, with the transactions added to the chain in groups called blocks. This builds a ledger (the blockchain) that, if truly decentralized (which is rarely the case in Crypto), is almost impossible to tamper.

And why do cryptocurrencies exist?

Simple: to economically incentivize the network nodes to participate in the security and get paid for it. In a nutshell, if decentralized training becomes a thing with efforts like Intellect-1, blockchains will play a crucial role in building economies around these training workloads, so independent people creating AI models are rewarded for their data or compute, thereby preventing the value accrual of AI being given to a handful of players concentrated on the US West Coast.

OPEN-SOURCE

Zyphra Releases Zamba2-7B

Zyphra has released Zamba2-7B, a new open-source model. At its…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.