It just predicts the next word
People say it to make language models sound small. Think about what predicting the next word actually takes.
"It just predicts the next word."
People say this to make language models sound small. Fancy autocomplete. A parrot with a GPU.
Think about what predicting the next word actually takes.
Ilya Sutskever has a good example. Take a murder mystery. On the last page, the detective gathers everyone in a room and says: "The killer is..." To predict the next word, you have to have understood the whole book. The clues, the lies, who was where. There's no shortcut. Everything you understood has to come out in that one word.
Now do that for the whole internet. Physics, contracts, code, arguments, jokes. To predict the next word across all of it, a model has to learn how all of it works. It can't memorise it. There isn't room. It has to learn it.
Prediction is compression
Here's the part people miss. If you can predict the next word well, you can compress text. Guess the next word, and if you're right, you store almost nothing. Only the surprises cost space. That's how the best compressors already work.
Better prediction, smaller file. They're the same problem.
And compression is understanding. A list of every position of every planet is a lot of numbers. Newton's laws are a few lines. That's the best compression of the sky anyone has found, and it's also what we call understanding it.
People have bet on this for a long time. Since 2006 there's been a prize for compressing Wikipedia, on the idea that compressing it better means understanding it better.
In 2023, DeepMind took a language model trained on text and used it as a compressor. It compressed images better than PNG and audio better than FLAC. Formats that were built by hand for exactly those jobs, beaten by a model that learned to predict text.
That's not autocomplete. That's something that found patterns deep enough to work on things it was never trained on.
So do you
You're a prediction machine too. You predict the next step on the stairs, and when it isn't there, your whole body notices. You probably predicted the end of this sentence before you got to the last word.
"It just predicts the next word." So do you. The word "just" is doing all the work in that sentence.
Whether today's models are close to the limit is a fair argument. Whether the goal is small isn't. Predicting the next word isn't a trick that looks like intelligence. Push it far enough and it's the definition.