Tokens and Context Windows: A Working Engineer's Guide

Most production bugs involving context windows are not exotic. A support transcript grows a little longer than anyone tested, a retrieval step returns one chunk too many, and the request either fails outright or silently drops the part of the prompt that mattered. The fix is not a bigger model. It is treating the context window as a budget that your code owns, measures and enforces.

What a token actually is

A tokenizer splits text into units drawn from a fixed vocabulary, usually learned with a byte-pair or similar subword algorithm. Common words often map to a single token, while rare words, identifiers and long numbers are broken into several pieces. Whitespace and punctuation are frequently folded into neighboring tokens, so " hello" and "hello" can be different tokens.

The practical consequences are easy to miss. Token counts vary by language, so the same message translated into another language may cost noticeably more. Source code, JSON and base64 blobs tend to be token-dense because of symbols and unusual character sequences. And because each model family ships its own vocabulary, a count from one tokenizer is only an estimate for another.

• Never estimate from characters in production. Rules of thumb like "about four characters per token" are fine for napkin math and wrong often enough to break a hard limit.
• Use the tokenizer that matches the model you call. If the provider exposes a token-counting endpoint or library, wrap it behind one function in your codebase.
• Count the whole request. Chat formats add per-message overhead for roles and separators, and tool definitions or JSON schemas sent with the request also consume input tokens.

The anatomy of a context window

The context window is the maximum number of tokens the model can attend to in one request, and in most APIs it covers both input and output. If a model has a limit of N tokens and your prompt uses nearly all of it, there is very little room left for the answer. That is why the output reserve is the first line item in any budget, not an afterthought.

A typical application request has four competing consumers:

1. System prompt and tool definitions. Mostly static, versioned with your code, and the easiest part to measure ahead of time.
2. Conversation history. Grows with every turn and is the most common cause of creeping overflow.
3. Retrieved or injected context. Documents, search results or records fetched for this request; variable in size and in usefulness.
4. Output reserve. Tokens you hold back for the response, typically matched to the max_tokens (or equivalent) parameter you send.

Budgeting the window in code

The budget should be computed per request, right before the call, using real token counts. Subtract the output reserve and the system prompt from the model's limit, then fill the remainder in priority order. A common policy is to give retrieved context a capped share and spend whatever is left on the most recent conversation turns.

1from dataclasses import dataclass2from typing import Callable3 4# count_tokens is a stand-in: wrap the tokenizer that matches your model.5CountFn = Callable[[str], int]6 7 8@dataclass(frozen=True)9class Budget:10    context_limit: int11    output_reserve: int12    system_tokens: int13 14    @property15    def available(self) -> int:16        return self.context_limit - self.output_reserve - self.system_tokens17 18 19def fit_context(20    budget: Budget,21    chunks: list[str],  # sorted by relevance, best first22    history: list[str],  # oldest first23    count_tokens: CountFn,24    retrieval_share: float = 0.6,25) -> tuple[list[str], list[str]]:26    remaining = budget.available27    retrieval_cap = int(remaining * retrieval_share)28    kept_chunks: list[str] = []29    for chunk in chunks:30        cost = count_tokens(chunk)31        if cost > retrieval_cap:32            break33        kept_chunks.append(chunk)34        retrieval_cap -= cost35        remaining -= cost36 37    kept_history: list[str] = []38    for turn in reversed(history):  # newest turns are kept first39        cost = count_tokens(turn)40        if cost > remaining:41            break42        kept_history.append(turn)43        remaining -= cost44    return kept_chunks, kept_history[::-1]

A few details in that sketch are deliberate. The loops break rather than skip, so the model never sees a lower-ranked chunk while a higher-ranked one was dropped, and never sees a turn without the turns that followed it. The retrieval_share is a tunable policy, not a law; a document Q&A product might favor retrieval, while a coding assistant in a long session might favor history. Whatever you choose, log the counts so you can see how often each consumer is being squeezed.

When content does not fit: truncation and summarization

Eventually something has to give. There are a handful of standard strategies, and the right one depends on what kind of information you are willing to lose.

Truncation

• Drop oldest turns. Simple and predictable, but the user's original goal often lives in the first message. Many teams pin the first user turn and drop from the middle instead.
• Drop lowest-ranked chunks. Natural for retrieval, as long as your ranking is trustworthy. Check it with an evaluation set rather than assuming.
• Truncate inside a document. Cutting a single long document at a token boundary can split a sentence or a table row. If you must, cut at paragraph or section boundaries and mark the cut explicitly, for example with an [omitted] marker the model can see.

Summarization

Rolling summarization replaces older turns with a compact summary, usually produced by a smaller model once history crosses a threshold. It preserves gist across long sessions, but it is lossy in ways that are hard to predict: exact numbers, identifiers and user constraints are the details most likely to be paraphrased away. A useful pattern is to extract structured facts (names, IDs, decisions, open questions) alongside the prose summary, and to keep both in the prompt.

Failure modes to watch in production

• Silent provider-side truncation. Some APIs and frameworks trim input without raising an error. Enforce your own limit before the call so the behavior is yours, not a default you did not choose.
• Output cut off mid-answer. A finish reason that indicates the length limit was hit means your output reserve was too small. Alert on it; a truncated JSON object is a parse error waiting to happen.
• Budget drift after prompt changes. Adding a few examples to the system prompt quietly steals space from everything else. Measure the static parts in CI and fail the build when they cross a threshold you set.
• Tokenizer mismatch after a model switch. If you route between models, compute budgets with the tokenizer and limits of the model actually selected for the request.

None of this requires sophisticated infrastructure. One function that counts tokens, one place that computes the budget, and a handful of logged numbers per request will catch most context-window problems long before a user does.