Bet on compute
Every clever trick in AI lost to more compute. The lesson for anyone building now is uncomfortable.
In 2019, Rich Sutton wrote a short essay called The Bitter Lesson. Seventy years of AI research, one conclusion: general methods that use more computation beat clever methods built on human knowledge. Not sometimes. Every time, eventually.
Look at the list.
Chess. People spent decades putting grandmaster knowledge into programs. Deep Blue beat Kasparov in 1997 mostly by searching more positions than anything before it.
Go. AlphaGo learned from human games and beat Lee Sedol in 2016. A year later, AlphaGo Zero learned only by playing against itself, with no human games at all, and beat the version that beat Lee Sedol 100 games to 0.
Vision. Years of hand-designed code for finding edges and corners. In 2012 a neural network called AlexNet, trained on two gaming graphics cards, beat all of it.
Language. Grammars, parsers, rules. Then one objective, predict the next word, and a lot of GPUs.
Every time, the clever people were right for a while. Every time, compute ran them over.
Why it's bitter
Because the clever thing feels like progress. You understand the problem, you put that understanding into code, and you get a result this year. The general method looks dumb and loses, until the hardware catches up. Then it wins, and all your understanding turns out to have been a speed bump.
Compute gets cheaper every year. Your cleverness doesn't.
What it means if you're building
Ask one question about anything you build: does it get better when compute gets cheaper?
A prompt with twenty hand-written rules for edge cases doesn't. The next model makes half of them wrong, and you spend a week finding out which half.
A set of real examples and an eval does. So does a pipeline that can swap in a bigger model the day one comes out. Those get better while you sleep.
Don't compete with the next model. Build the thing the next model makes better.
It's humbling. A lot of what feels like engineering is scaffolding for a model that isn't good enough yet. When the model gets good enough, the scaffolding gets thrown away.
Bet on compute. It hasn't lost yet.