KVASVECTOR/BESTWEB/Shutterstock and Adobe modified by Blockworks
This is a segment from The Breakdown newsletter. To read full editions, subscribe.
In a 2003 research paper, Google engineers shared their new approach of building “reliable computing infrastructure from clusters of unreliable commodity PCs.”
This was a radical change. Instead of paying up for super high-end Sun servers engineered for near-perfect reliability, Google was embracing cheap, commodity hardware running free Linux software. They then developed additional software to work around the hardware’s frequent failures.
Redundancy through volume, they found, is cheaper than precision through engineering.
This is the model for modern AI data centers, too: You can’t have a three-month training run crash just because one GPU dies out of 20,000.
It’s not, however, the model for manufacturing the GPUs themselves.
Instead, semiconductor fabrication plants are built to produce GPUs with an ultra-low level of defects, which demands near-perfection at every step of an incredibly complex process — the opposite of Google’s recommendation for data centers.
But this is not engineering perfectionism. It’s an economic necessity.
“Why do we need to have 99.9999% reliability at every step of the process?” Ben Thompson asks. “It’s all the cost of capital equipment.”
Mostly that’s because the $200 million ASML machines every fabrication plant uses to make cutting-edge semiconductors “are the most overengineered things on earth,” Thompson explains.
This creates a feedback loop of spiraling cost and perfection: Fabrication plants have to be near-perfect to be profitable because the ASML machines are so expensive — and the ASML machines are so expensive because they have to be perfect.
As a result, it now costs as much as $20 billion to build a single semiconductor fab.
Thompson compares the process of making semiconductors with hyper-complex ASML machines with the process of making rockets at SpaceX, where Elon Musk chooses to “blow up a bunch of Starships because you don’t want to be wasting anything on complexity that actually isn’t necessary.”
Repeatedly blowing up rockets helps SpaceX find the “exact line where you want to be,” as Thompson puts it — the line between complexity and usefulness that allows them to avoid the astronomical upfront costs required by perfection.
In other words, failure can be a feature, not a bug, of complex systems.
Failure is a core feature of crypto systems, too: Every blockchain is designed around the assumption that some percentage of its nodes will be malicious or faulty.
This makes blockchains resilient — like Google’s data centers, they continue to produce valid blocks despite individual failures. But that fault tolerance doesn’t extend to the code running on the blockchain.
Smart contract code, written by fallible humans, will inevitably have bugs or design flaws. This can be costly, because unlike traditional finance, there’s no reversing a crypto transaction once a flawed contract or protocol is exploited.
Code eliminates intermediaries, but at the cost of zero fault tolerance.
In traditional finance, where systems are permissioned and transactions are reversible, banks can aim for near-perfection by implementing ever-stricter KYC/AML rules and hiring ever more compliance officers.
In crypto finance, where systems are permissionless and transactions are irreversible, we can only aim to learn from our mistakes.
There have been a lot of them: from The DAO exploit that nearly sank Ethereum in 2016 to the record-setting $1.5 billion Bybit hack in 2025.
With luck, though, these could come to look in hindsight like Elon Musk’s exploding rockets. Each exploit reveals an attack vector — reentrancy, oracle manipulation, protocol design flaws — that can be fixed in future iterations.
Like Starship explosions, the failures show what actually needs hardening vs. what was unnecessary caution — each an opportunity to find the right balance of complexity and usefulness in a new kind of financial system.
Crypto is often judged by its failures, which is fair enough, given their frequency and size.
But the failures aren’t bugs in the process of building a new financial system. They are the process.
To enjoy crypto, then, you’ll have to enjoy failure, too.
Happy new year.
