This series opened with a dismissal you have probably heard: "LLMs are just predicting the next token." Ten parts later, you are in a position to see what that sentence actually describes, and why "just" is doing some very heavy lifting.
A language model is a prediction machine
Everything in this series has been about one shape of problem: given an input, predict an output. Hours studied in, exam score out. Parent height in, child height out.
A language model is the same shape of problem with a different pair of columns. The feature is the text so far; the target is whichever token comes next.
| Text so far (feature) | Next token (target) |
|---|---|
| It was a dark and | stormy |
| The cat sat on the | mat |
| To be or not to | be |
Train a model on billions of rows like these, harvested from books and the web, and you get something that can continue any text you hand it. Generate one token, append it, feed the longer text back in, and repeat: that loop, prediction after prediction, is a chatbot writing you a paragraph.
But models eat numbers, not words
Every model in this series took a number in and pushed a number out. Text needs a conversion step, and that step is called tokenisation: chop the text into pieces (tokens, roughly words or chunks of words), and give every distinct piece an ID number from a big fixed dictionary.
"It was a dark and stormy"
→ ["It", " was", " a", " dark", " and", " stormy"]
→ [1122, 572, 264, 6319, 323, 13458]
Those IDs are what the network actually receives, and what it produces is not a single number but a score for every token in its dictionary, converted into probabilities: " night" 62%, " evening" 9%, " sea" 3%, and so on down the list. Predicting the next token really means ranking the whole dictionary by likelihood.
Try it: a next-word predictor you can steer
The toy below has been "trained" on a few dozen sentences by simply counting which word follows which. Given the current word, it looks up every word that ever followed it and how often, exactly the probability ranking described above, just with counts instead of a neural network. Click a candidate to accept it (or let the model pick), and watch a sentence assemble itself one prediction at a time.
the
The model's predictions for the next word (click one to accept it):
Two things are worth noticing. First, even this comically simple model produces mostly grammatical text, because "which word tends to follow which" already encodes a surprising amount of structure. Second, notice its limitation: it only ever looks at one word of context, so it happily wanders ("the cat chased the ball was red"). A real LLM differs in exactly that respect: it weighs the entire text so far, thousands of tokens, when ranking the dictionary, and it uses a trained neural network rather than a lookup table to do it. But the game being played, rank the possible next tokens and pick one, is the same one you just played.
Everything you learned, scaled up
Here is the payoff. Every concept in this series maps directly onto what happens inside a large language model:
| In this series | In an LLM |
|---|---|
| Feature → target (hours → score) | Text so far → next token |
| Weight and bias, 2 knobs | Weights and biases, hundreds of billions of knobs |
| Neurons with activation functions | The same, arranged in many wide layers |
| Loss function (MSE) | A loss for probabilities (cross-entropy): "how surprised was the model by the true next token?" |
| Gradient descent + backpropagation | Exactly the same loop, run on thousands of GPUs |
| Learning rate | Still there, still fiddly, still tuned with care |
| Overfitting and held-back data | Still there: a model can memorise its training text, and evaluation still relies on data it has not seen |
| Local minima and "good enough" | Still there: nobody finds the global minimum of a trillion-dimensional loss surface, and nobody needs to |
There is genuinely new machinery in modern LLMs that this series has not covered, the transformer architecture and its attention mechanism (the trick for weighing all the earlier context at once), and embeddings (the learned number-lists that represent each token's meaning). Those are stories for another series. But they are refinements on top of the engine you now understand, not replacements for it. Strip any LLM to its skeleton and you find neurons doing w·x + b, activation functions bending the results, a loss measuring the surprise, and gradient descent turning billions of knobs a tiny step at a time.
One last secret: base models and alignment
Everything described above, the giant network trained by gradient descent to predict the next token, produces what is called a base model (you will also hear pretrained model). And here is the thing: a base model is exactly what this article says it is, a next-token predictor, and nothing more. Hand it "It was a dark and stormy" and it will continue the story. But ask it "What is the capital of France?" and it may just as happily reply with more questions, because in its training data, one question is very often followed by another. It has no notion that it is supposed to be answering you. Left to roam, base models drift, repeat themselves, and sometimes produce output that reads as eerily incoherent, not because they lack knowledge, but because "continue this text plausibly" is genuinely all they were trained to do, much like our toy predictor wandering into "the cat chased the ball was red."
The step that turns that raw text-continuer into the helpful assistant you actually talk to is called alignment (or post-training). After pretraining, the model is additionally trained on examples of good conversations, question in, helpful answer out, and refined with human feedback on which of its responses people prefer. The machinery is the same as everything in this series, gradient descent nudging weights to reduce a loss, but the goal shifts: no longer "predict the next token of the internet," now "respond the way a helpful assistant would." That is what teaches the model to answer questions rather than continue them, to follow instructions, to talk in a particular manner. The raw capability comes from pretraining; the conversational character you experience is taught afterwards.
So the next time someone says a language model is "just predicting the next token," you can agree cheerfully, and then explain what it takes to predict one well, and what it took to make the predictor talk to you like an assistant.
Congratulations: you now understand the core building blocks of machine learning well enough to be the most dangerous person in the room at any party involving software engineers.