Key takeaways
If you've used a chatbot, you've watched this happen. The first few words land almost before you've finished reading your own prompt, and then the rest dribble out one at a time, as if the model were thinking harder about word forty than it did about word four. It isn't. The reason is duller than that: to produce each new word, the model has to weigh every word that came before it. The more you've typed, the more weighing gets stacked onto each word it hands back.
It shows up in other ways too. Long prompts cost more than the word count alone would suggest. And the models that brag about swallowing a whole novel tend to get worse with one, not better, once you actually feed it in.
All of it traces back to one mechanism: attention. Attention is how the model decides which parts of your input matter for the word it's about to produce. There's nothing clever about it, really. It's a single idea with one expensive side effect, and once you've seen both, the behaviour above stops looking strange.
There's almost no math ahead, and where a little is unavoidable, a diagram does the work instead. By the time you're through, you should be able to look at your own latency graph, or your own bill, and point at the part of the model responsible for it.
Through the guide again. This section is even cleaner than the last, so most of the value here is in two cross-section patterns that only show up when you read the post as a whole, which is exactly what §5 of the reference is for.
Within-section audit
Paragraph 1 — clean. "That is attention." is a single punchy fragment (§3.2), fine at that count. "Everything else in this post is the bill that one sentence runs up" is technically a preview line (§2.2), but it's a short, vivid, thematically-tied metaphor rather than a formulaic "what follows is a tour" restatement, so I'd keep it. It earns its place.
Paragraph 2 — clean. The "A quick note on vocabulary" aside and the words-vs-tokens explanation are genuinely useful and well-shaped. Nothing to fix.
Paragraph 3 — one real flag. "Cat carries a lot. It is who did the sitting. Mat carries some. The two thes carry almost nothing." is four hard stops in a row, the staccato version of the punchy cadence (§3.2). The repeated "carries" is borderline anaphora (§2.4), but here it's a coherent metaphor doing pedagogical work, so I'd keep the word and just reduce the number of full stops so it flows as a weighting rather than a drumroll.
Two cross-section patterns (the important part)
No triads, no roadmap scaffolding, no signature vocabulary, no dash overuse.
Fixed section (non-contracted register preserved)
Here is the whole idea: to make sense of a word, the model looks at the words around it and decides how much each one matters. That is attention. Everything else in this post is the bill that one sentence runs up.
A quick note on vocabulary. The model does not work in words; it works in tokens, which are usually chunks smaller than a word. I will say "words" while explaining the idea, because it reads better, and switch to "tokens" the moment real numbers turn up, since that is the unit they are counted in.
Take the word sat in the cat sat on the mat. To place it, the model glances at every other word and weighs how much it matters. Cat carries a lot, since it is who did the sitting; mat carries some; the two thes carry almost nothing. The model runs that weighing for every word at once, each one sizing up all the others. Underneath the metaphor, it is multiplication across large tables of numbers.
So how does the model decide that cat matters to sat and the does not?
Every word puts out three things. It asks a question: what am I looking for? It holds up a label: here is what I am. And it carries a payload, the actual content it will hand over if asked. When one word's question matches another word's label, the second word passes its payload to the first, scaled by how good the match was.
The question and the label are small, cheap advertisements; the payload is the cargo. Hold on to that asymmetry, because it decides what has to be stored later.
Think about it in the cat sat because it was tired. The word it sends up a question: which thing am I? Every other word shows its label. The one on cat reads, roughly, noun, animal, the one doing things, and it fits. So cat hands over its payload, and from there the model treats it as the cat.
None of this is understanding. Questions meet labels, cargo changes hands, and something that behaves like meaning falls out the far side. The question-and-label image is a crutch, by the way. The model is not really querying a catalogue, it is taking dot products between vectors. But the crutch holds weight all the way to the end of this post, so I am going to keep leaning on it.
If every word weighs every other word, the weighing piles up fast.
Four words, sixteen pairs. Double it to eight and you do not get thirty-two, you get sixty-four. Double again to sixteen and you are at two hundred and fifty-six. The work climbs with the square of the length, which is a polite way of saying it gets out of hand.
This is the first place attention reaches into your life directly. Double the prompt and the work roughly quadruples, with the cost and the wait climbing alongside it. So "just put more in the context" does not scale the way the flat per-token rate on most API dashboards quietly implies. A prompt that looks only a little longer can cost a great deal more, and the pricing page rarely warns you.
A model writes one word at a time, and to pick the next one it has to look back over everything so far: your input, plus every word it has already written.
The naive version of this is appalling. At each new word, redo the question-label-payload work for every earlier word from scratch, again and again, for the length of the whole answer. Nobody does that. The first time the model meets a word it works out that word's label and payload once and writes them down. Every later step reads the note instead of rebuilding it. These saved notes are the key-value cache: keys are the labels, values are the payloads. They are the only reason generation is fast enough to ship.
The catch is memory. The notes have to sit in GPU memory, and they grow with every word, for every request running at the same instant.
Put numbers on it. Take Llama-3.1-70B serving a single user. A 32K-token context holds roughly 10 GB of notes; let the conversation run on to 128K and you are closer to 40 GB. The model's weights, by contrast, are a one-time cost: loaded once, shared by everybody on the box. The notes are not. They are counted again for every user and they swell with every word exchanged. On a busy server, or inside one long rambling conversation, the notes can eat more memory than the model itself. That, not parameter count, is usually what caps how many people you can serve at once, and it is the number that never makes it onto the marketing page.
Reading the prompt, then writing the answer
This is what sits behind the slow, word-by-word crawl.
Generation runs in two stages, and they barely resemble each other. First the model reads your prompt. It can do this in a single parallel pass, because the whole thing is already in front of it and every word can weigh every other word at once. This stage is quick, and it ends the instant the first word of the answer appears. That is why the first word feels immediate.
Then it writes, and writing is a different animal. Word ten cannot be chosen until word nine exists, so the words come out in a line, one at a time, with no parallel trick to rescue you. The model runs once per word, in a straight march to the end.
The surprising part is where the slowness comes from. The arithmetic at each step is trivial; what eats the time is hauling all of those notes back out of memory before the step can run. The chip is not thinking, it is moving data. Generation speed is really a memory-bandwidth problem wearing a compute costume.
Consider the giant context window.
A model that accepts a million tokens sounds like a model that remembers a million tokens. It is not, and everything above is why. I would treat most very large advertised windows as a ceiling you are allowed to approach, not a workspace you are meant to fill.
Two things go wrong as the prompt gets long. The first is the cost you have already met: a million-token prompt means a mountain of weighing and a mountain of notes, so it is slow and expensive at the top end. On the biggest models the wait before the first word can stretch into seconds. The second is sneakier.
The model handles the beginning and the end of a long prompt well and lets the middle slide. Bury a crucial fact halfway through a hundred-page dump and there is a real chance it never surfaces in the answer. Researchers named this getting lost in the middle, and it gets worse as the window grows, for the dull reason that a bigger window is mostly more middle.
So treat the big number as a budget with terrible interest rates. A few thousand tokens placed where the model actually looks will beat a million shovelled in and hoped over.
If the notes are the bottleneck, the obvious move is smaller notes. That is most of what recent models have done, and it is why something released this year often runs far cheaper than a two-year-old model of the same size.
The fix starts with a detail I skipped. The model does not read with one set of eyes. It has many attention heads, each looking at the text in its own way. The early designs gave every head its own private notes, which duplicated an enormous amount. The repair, called grouped-query attention, lets a group of heads share a single set of notes. Same model, far fewer notes.
And the saving is not marginal. Back to the 70B model: with every head hoarding its own notes, a 32K context wants something like 80 GB of cache, more than a single H100 has room for once the weights are loaded, which is the wall that used to push you onto a second GPU. Share the notes and the same context drops to roughly 10 GB. That eightfold cut is, more or less single-handedly, why the cache no longer forces a 70B onto a second card.
Newer designs push harder. DeepSeek's approach compresses the notes down to a small latent and only expands them when they are needed, shaving the cache again. The specifics differ from model to model, but they are all aimed at the same thing: shrink the pile of notes, because the pile is the cost. When a release claims to be cheaper to run, this is almost always what changed behind the scenes.
There is a second, separate family of tricks worth knowing about: FlashAttention and its descendants. They do not make the notes any smaller; they just move them through the chip's memory more cleverly. Same answer, same cache size, less time wasted shuffling data around. Modern systems run both kinds at once: one to make the notes smaller, the other to move them faster. It is worth not confusing the two, because they scale and fail in different ways.
None of this was theory for its own sake; each piece should change something you actually do.
When the bill climbs faster than the prompt grew, that is the square-law, not a billing error. Long context is priced steeply because the work underneath it grows with the square of the length, whatever the flat per-token rate suggests.
Deployment is where this bites hardest. The question is never just whether the model fits in memory. It is whether the model plus all those notes, for every concurrent request, fits. The notes are almost always what runs out first. A model with shared or compressed notes will serve far more users on the same card, which is why the attention design buried in the spec sheet deserves as much of your attention as the parameter count on the front of it. I would argue it deserves more.
When the output streams slowly, throwing compute at it will not help. You are waiting on memory. Smaller notes and faster memory will; a bigger GPU with the same bandwidth mostly will not.
And if a long-context model lets you down, go and fix the middle yourself. Pull the passages that matter and put them near the start and the end, where the model reads best. Do not outsource that to the window.
The idea was simple the whole way through: every word weighs every other word. The cost is that the weighing grows with the square of the length, and the storage piles up in memory. Hold those two facts and the model stops being a black box: the slow output, the steep bill, and the fact that it went missing in the middle all have plain causes, and most of them have workarounds.
One honest caveat. The numbers here, things like cache sizes and the exact point where long context falls apart, move quickly as models change. The shapes are stable; the figures are not. Check them against your own model's docs before you go sizing hardware on the strength of a blog post, this one included.