What is LLM inference?
LLM inference is the process of using an already-trained large language model to generate a response to a given input, running the model’s learned computation forward on new text rather than adjusting the model’s own parameters the way training does. It happens one token at a time, the model predicts the most likely next piece of text given everything that came before, adds that prediction to the sequence, and repeats the process until it produces a complete response, and understanding this basic mechanism is the foundation for everything else this collection covers about serving, optimizing, and deploying language models in production.
Why inference and training are fundamentally different operations
Training a language model means adjusting its internal parameters repeatedly over a massive amount of example text until the model gets progressively better at predicting what comes next in a sequence, and this process happens once, or at least happens rarely relative to how long the finished model stays in use afterward. Inference happens every single time someone uses the finished model, and it involves no adjustment to the model’s parameters at all, the model’s learned knowledge stays completely fixed, and inference simply runs that fixed knowledge forward against a new input to produce a new output.
This distinction matters because it explains why a model’s behavior stays consistent between separate conversations even though it seems to learn within a single conversation, what looks like learning during a conversation is really the model using everything already written in that conversation, the earlier messages, as part of its input each time it generates a new response, not any actual change to the model’s underlying parameters. Once a conversation ends, none of that context persists into the model’s next unrelated use, the parameters themselves are exactly what they were before, unaffected by what happened during inference.
How next-token prediction produces a complete response
At its core, a language model performing inference does one specific thing repeatedly: given a sequence of tokens, it produces a probability distribution over what the next token is likely to be, and the system generating a response picks one token from that distribution, adds it to the sequence, and feeds the whole thing back into the model to predict the next token after that. This process repeats, one token at a time, until the model produces a special token signaling it’s finished, or until some other stopping condition is reached, at which point the accumulated sequence of generated tokens becomes the complete response.
How a token gets picked from that probability distribution is itself a meaningful design choice, always picking the single most likely token produces consistent but often bland, repetitive output, while introducing some controlled randomness into that choice, sampling from the distribution rather than always taking the top prediction, tends to produce more varied and often more natural-feeling responses at the cost of some consistency between repeated runs of the same input. This sampling behavior is why the same prompt can produce a somewhat different response each time it’s run, even though the underlying model and its parameters haven’t changed at all between those runs.
The two distinct phases every inference request goes through
A single inference request involves two phases with meaningfully different performance characteristics, covered in more depth in this collection’s discussion of inference serving. The first phase processes the entire input prompt in one pass, computing the model’s internal representations for every token in that prompt simultaneously, since the whole prompt is already available upfront and doesn’t need to be processed sequentially. The second phase generates the response itself, one token at a time, where each new token depends on everything generated before it and has to be produced sequentially rather than all at once.
This split is why the time before a response starts appearing often feels different from how quickly each subsequent piece of text appears afterward, the first phase does a burst of work across the whole prompt at once, while the second phase does a smaller amount of sequential work at each individual step, and a model’s overall response time reflects both of these phases combined rather than either one in isolation. Recognizing this distinction matters for understanding where a slow response comes from, a long prompt affects primarily the first phase, while a long generated response affects primarily the second.
Why generating text sequentially is fundamentally expensive
The sequential nature of the second phase, where each new token depends on everything generated before it, is the core reason language model inference is computationally demanding in a way that’s genuinely different from many other kinds of computation. A system can’t simply generate the tenth token of a response before generating the ninth, since the tenth token’s prediction depends directly on that ninth token already being part of the sequence, which rules out the kind of straightforward parallelization that speeds up many other computational tasks.
This inherent sequential dependency is exactly the problem the KV cache, covered in this collection’s dedicated article on the topic, exists to make less costly, avoiding the need to recompute earlier tokens’ internal representations from scratch at every single step, and it’s also why techniques like speculative decoding, covered in the discussion of inference optimization, exist specifically to find ways around this sequential bottleneck, having a smaller, faster model draft ahead so the larger model can verify several tokens in a single pass rather than generating each one individually.
How the size of the model shapes what inference costs
A larger model, one with more parameters, does more computation at every single step of this process, for both the initial prompt-processing phase and for generating each individual token afterward, which is why a larger, more capable model is also a slower and more computationally expensive one to run inference against. This tradeoff between capability and inference cost runs through nearly every practical decision covered elsewhere in this collection, whether to use a smaller, faster model for a given task, whether to apply quantization or other compression techniques to reduce that cost, whether a use case needs the extra capability a larger model provides or would be served just as well by a smaller, cheaper one.
This is why understanding the basic mechanics of inference matters beyond pure technical curiosity, nearly every decision about deploying a language model in a real product ultimately traces back to this same underlying tradeoff between a model’s capability and the computational cost of running its inference process, and a team that understands where that cost comes from is better equipped to make deliberate, informed choices about it rather than treating inference cost as an opaque, unavoidable fact about whichever model they happened to choose.
What determines how good a given inference result is
The quality of a specific inference result depends on more than just the underlying model’s own learned capability, it also depends on the input given to the model, since a model can only predict a good continuation of whatever sequence it’s been given, and it depends on the sampling settings used to pick tokens from the model’s predicted distribution, since the same underlying prediction can produce a more conservative, consistent output or a more varied, exploratory one depending on how that final token selection is configured. This is part of why the same underlying model can produce very different practical results depending on how it’s being used, the prompt it’s given, the sampling settings applied to it, and the specific task it’s being asked to perform.
Recognizing that inference quality is a function of the model plus the way it’s being used, not the model in isolation, is an important corrective to treating model choice as the only variable that matters when building something on top of a language model. Two teams using the exact same underlying model can get meaningfully different practical results depending on how carefully they’ve thought through the rest of the inference process surrounding that shared model.
Common mistakes people make around LLM inference
1. Assuming a model learns or changes during a conversation, when what’s actually happening is the model using the accumulated conversation as input each time, with its underlying parameters staying completely fixed.
2. Treating a slow response as purely the model’s fault without distinguishing whether the delay comes from a long input prompt or from generating a long response, since these two phases have genuinely different performance characteristics.
3. Assuming the exact same input will always produce the exact same output, missing how sampling settings introduce deliberate, controlled variability into the token selection process.
4. Focusing purely on model choice as the determinant of output quality, missing how much the prompt and the sampling configuration actually shape a specific inference result.
5. Treating inference cost as a fixed, unavoidable fact about a given model rather than something that traces back to specific, understandable mechanics, like the sequential nature of token generation, that can be actively optimized against.
What connects these mistakes is a surface-level understanding of inference as a mysterious black box rather than as a specific, understandable process, one that predicts a next token, samples from that prediction, and repeats, and once that basic mechanism is clear, the reasoning behind serving decisions, optimization techniques, and deployment tradeoffs covered throughout the rest of this collection follows directly from it rather than needing to be taken on faith.
The deeper point about LLM inference is that everything a language model actually does in production, every response a user sees, every cost a team pays, every latency a user experiences, comes down to this same repeated, sequential process of predicting and selecting one token at a time, and the entire surrounding discipline of serving, optimizing, and deploying these models is ultimately about making that one core mechanism run as efficiently and as reliably as the situation actually demands.