Chapter 11 · 2006–now

The second lunch

The free lunch came back in different silicon: a stubborn bet, a bedroom experiment, and the trade replaying at datacenter scale.

Chapter 8 ended with the industry told to go parallel and hating it. But one kind of chip had been parallel all along, hiding in plain sight inside gaming PCs. A GPU — a graphics chip — paints millions of pixels per frame, and the pixels don’t wait on each other, so the chip is built as thousands of small workers doing arithmetic side by side. Gamers bought them to make explosions prettier. A few scientists looked at all that idle arithmetic and started sneaking real math through the graphics pipeline, disguised as pictures.

Jensen Huang, NVIDIA’s leather-jacketed co-founder, made the bet that turned the trick into a platform. In 2006, NVIDIA shipped CUDA — a way for ordinary programmers to use the gaming chip for any computation, no disguise required. The cost was enormous: years of engineering, silicon on every chip reserved for a market that didn’t exist. Wall Street called it waste, and for half a decade Wall Street looked right. CUDA was a door standing open in a field, waiting for somebody to need it.

The somebody was a grad student with two gaming cards. In 2012, Alex Krizhevsky — Ukrainian-born, at the University of Toronto, in Geoffrey Hinton’s lab, where neural networks had been kept alive through two decades of academic winter — trained a network called AlexNet for the ImageNet competition, a contest to recognize objects in photographs. The training ran for days in a bedroom on two consumer GPUs. AlexNet didn’t just win. It obliterated the field by a margin the competition had never seen, beating laboratories with budgets a thousand times larger. Every serious lab on earth read the result the same way overnight: neural networks work — you were just running them on the wrong chip.

Hold that against chapter 5’s pattern, because it’s the same trade in new silicon. Neural networks are like Lisp — an idea invented decades before the hardware could afford it, warehoused by two AI winters, waiting. The doubling never died; after the wall it kept paying out in parallel form, and GPUs collected the payments. When the hardware finally caught up to the idea, the idea ate the world in eighteen months.

And so the free lunch returned. The old lunch: write it slow, next year’s clock will save you. The second lunch: don’t make the worker faster — hire ten thousand workers. The slider below is the whole economics lesson: one painter versus an army painting a wall, with the catch made visible — coordination cost. Some jobs split perfectly. Some don’t split at all. And the army itself is never free, which is why the second lunch, like the first, is a trade and not a gift: burn silicon, and staggering amounts of electricity, to save human time. The GPU datacenters filling with racks today are chapter 5’s ladder rebuilt at industrial scale — and their power bills are this era’s version of the bloated download, the cost everyone tolerates because the curve is paying.

Run the counter-current too, because it never stopped. Even in the second lunch’s boom, efficiency keeps its revenge: Rust exists because C’s fifty-year-old memory bugs finally cost too much; phone chips took over laptops because power efficiency beat raw speed; the biggest companies now design their own silicon rather than accept the waste of general-purpose parts. Nobody, at any scale, ever really gets to stop caring about the rung below.

So end where the map ends — at the GPU datacenter, where every thread of this book physically converges. Bell’s metered dial tone is the business model. Linux, the escaped tape’s grandchild, runs every rack. Containers, the 1979 footnote grown up, are the unit of work. And the racks themselves burn parallel silicon, eating the second lunch. One deal, in 1913, to fund a lab. One price, in 1956, that let everything escape. Everything since is compound interest.

A closing note, in the author’s own voice. I work in this industry, and I never forget that I’m standing on the shoulders of these nerds — the unicycle rider, the man who wanted to play his space game, the student who apologized for how small his kernel would stay. They’re my heroes. This site is my thank-you note.