How LLMs Work for Engineers: Tokens, Context Windows and Sampling

You don't need to train a large language model to design systems around one, but you do need a working model of how an LLM turns input into output, because it explains latency, cost, context limits and many failure modes. This guide covers tokens, the transformer at the level a system designer needs, context windows, sampling (temperature and top-p), how generation happens one token at a time, and how all of that becomes per-token pricing.

Tokens

LLMs read and write tokens: chunks of text produced by a tokenizer, often a whole common word, part of a rarer word, or punctuation. In English a token averages roughly three-quarters of a word, so 1,000 tokens is about 750 words. Code, numbers and non-English text often use more tokens per word.

• Limits (context window, max output) are in tokens.
• Price is per token, usually per million, with output tokens costing several times more than input tokens.
• Latency grows with output tokens, because they're generated one at a time.

What happens inside, briefly

Each token becomes an embedding vector. A stack of transformer layers updates every token's vector using attention, which lets each position look at all earlier positions to decide what's relevant. The final layer produces a score for every token in the vocabulary; those scores become a probability distribution over the next token. That's all an LLM does: predict the next token, repeatedly.

Two phases matter for serving: prefill reads the whole prompt in one parallel pass (fast per token, sets the time to first token), and decode generates output tokens one by one (sets the streaming speed). Serving systems keep each token's attention keys and values in a KV cache so they aren't recomputed every step. See LLM serving.

Context windows

The context window is the maximum number of tokens (prompt plus output) the model can attend to in one call. Everything the model knows about this request has to fit: system instructions, conversation history, retrieved documents, tool results. The model remembers nothing between calls; your application resends what it needs each time.

• Long contexts cost more (you pay for every input token) and are slower to prefill.
• Models use information in very long contexts unevenly; facts buried in the middle are recalled less reliably than those at the start or end.
• So even with large windows, selecting the right context (RAG, memory) usually beats stuffing everything in.

Sampling: temperature and top-p

Even at temperature 0, outputs aren't guaranteed identical across calls or model versions. Design for variation: validate outputs, retry, and pin model versions in production.

From base model to assistant

1. Pretraining on a huge text corpus teaches next-token prediction and broad knowledge.
2. Instruction tuning (supervised fine-tuning on examples of instructions and good answers) makes it follow requests.
3. Preference tuning (from human or AI feedback) makes answers more helpful and safer.

The knowledge has a cutoff date and isn't your private data, which is why products add retrieval and tools. Adapting models yourself is covered in fine-tuning vs RAG.

Why LLMs fail the way they do

• Hallucination: the model produces fluent, plausible text whether or not it knows the answer. Grounding with retrieval and citations, and asking it to say when it doesn't know, reduce it.
• Instruction conflicts: text in the prompt from untrusted sources can be read as instructions. See prompt injection.
• Format drift: free-text output doesn't reliably match a schema. See structured outputs.
• Arithmetic and exact recall: better handled by tools (a calculator, a database query) than by the model.

Turning tokens into cost and latency

Request: 2,000 input tokens, 400 output tokens
Cost   = 2,000 × input_price + 400 × output_price
         e.g. at $3 / $15 per 1M → $0.006 + $0.006 = $0.012 per request
Latency ≈ time_to_first_token + output_tokens ÷ decode_speed
         e.g. 0.5 s + 400 ÷ 60 tok/s ≈ 7.2 s total, first text at 0.5 s (stream it)