Deployment · Guide
Controlling Latency and Cost in LLM Applications
Latency and cost in LLM systems are driven by a short list of levers: tokens in, tokens out, which model runs, and how often you call it. This guide walks through each lever and the trade-offs that come with pulling it.
- Author
- Sam Okafor (sample)
- Published
- Reading time
- 10 min read
- POST /answer2380 ms
- embed_query
- vector_search
- rerank
- llm.generate1880 ms
- validate_output
A prototype that feels fast with one user and a short prompt can become slow and expensive once real traffic, real documents and multi-step pipelines arrive. Before optimizing anything, it helps to know where time and money actually go, and that starts with a simple model of a single call.
Where time and money go
For most hosted models, a request has two phases. The model first processes the input (often called prefill), then generates output one token at a time (decode). Input processing is comparatively fast per token, while output generation is sequential, so output length usually dominates end-to-end latency. Billing typically counts input and output tokens separately, and output tokens are often priced higher.
That gives you two numbers worth tracking for every call: time to first token (TTFT), which reflects queueing, network and input processing, and total duration, which grows with output length. Measure both at p50 and p95. Averages hide the slow tail, and the tail is what users complain about.
In multi-step pipelines, also look at the shape of the request rather than individual calls. Three sequential model calls each taking a couple of seconds will feel slow no matter how well each one is tuned. Ask whether steps can run in parallel, whether a step can be skipped for simple inputs, and whether a cheaper non-LLM component, such as a keyword filter or a rules engine, can handle part of the work before a model is involved at all.
| Lever | Latency effect | Cost effect | Main risk |
|---|---|---|---|
| Streaming | Lower perceived latency | None | Harder validation of partial output |
| Exact-match caching | Large on hits | Large on hits | Stale answers, low hit rate |
| Semantic caching | Large on hits | Large on hits | Wrong answer for a similar-looking query |
| Model routing | Lower for routed traffic | Lower for routed traffic | Quality drop on misrouted requests |
| Smaller prompts | Lower TTFT | Lower input cost | Losing context the model needed |
| Output limits | Lower total duration | Lower output cost | Truncated answers |
| Batching | Higher per request | Often lower | Not suitable for interactive paths |
Streaming: fix perceived latency first
Streaming does not make the model faster, but it changes what the user experiences. Showing the first words quickly turns a long wait into visible progress. For chat and drafting interfaces, it is usually the cheapest improvement available.
Streaming complicates anything that needs the whole response before acting. If you validate structured output against a schema, apply a moderation check, or post-process citations, decide explicitly whether to buffer and validate before display or to stream and retract on failure. For machine-consumed outputs, such as a JSON payload passed to the next pipeline step, streaming rarely helps.
Caching: exact and semantic
Exact-match caching keys a stored response on everything that influences the output: the model identifier, the full rendered prompt, sampling parameters, tool definitions and the prompt template version. It is safe and simple, but in conversational products the hit rate is often low because inputs rarely repeat verbatim. It shines on deterministic, repeated work like classifying the same product titles or re-running a pipeline over unchanged documents.
Semantic caching embeds the query and returns a stored answer when a previous query is close enough in vector space. It can raise hit rates substantially, and it can also return a confidently wrong answer. "How do I cancel my order?" and "How do I cancel my subscription?" may sit close together in embedding space while needing different answers.
Also check whether your provider offers prompt or prefix caching, where a repeated prompt prefix (such as a long system prompt) is processed more cheaply on later calls. It usually rewards keeping static content at the start of the prompt and variable content at the end, which is a good habit regardless.
Model routing and prompt size
Not every request needs your most capable model. Classification, extraction and short reformatting tasks often run well on a smaller, faster model, while open-ended reasoning may not. A router can be as simple as a lookup by task type, or a cheap classifier that estimates difficulty. Start simple: routing on task type is easy to reason about and easy to test.
import asynciofrom dataclasses import dataclass # llm is a hypothetical thin wrapper around your provider's client.from app.llm import llm @dataclass(frozen=True)class Route: model: str max_output_tokens: int timeout_s: float SMALL = Route("small-model", max_output_tokens=300, timeout_s=4.0)LARGE = Route("large-model", max_output_tokens=800, timeout_s=12.0) def choose_route(task: str, prompt_tokens: int) -> Route: if task in {"classify", "extract"} and prompt_tokens < 4_000: return SMALL return LARGE async def complete(task: str, prompt: str, prompt_tokens: int) -> str: route = choose_route(task, prompt_tokens) call = llm.complete(model=route.model, prompt=prompt, max_tokens=route.max_output_tokens) try: return await asyncio.wait_for(call, timeout=route.timeout_s) except asyncio.TimeoutError: if route is SMALL: raise # One bounded fallback, never an open-ended retry loop. fallback = llm.complete(model=SMALL.model, prompt=prompt, max_tokens=SMALL.max_output_tokens) return await asyncio.wait_for(fallback, timeout=SMALL.timeout_s)Every routing rule is a quality decision, so put routed traffic through the same evaluation set as the default path. Suppose 70% of requests are classification: routing them to a smaller model could cut cost meaningfully, but only if accuracy on your own labeled data holds up.
Prompt size is the other half. Audit the system prompt for instructions that no longer change behavior, trim few-shot examples to the ones that measurably help, and retrieve fewer, better chunks rather than many marginal ones. Set max_tokens (or its equivalent) per task; a classification call that can only produce a label should not be allowed to produce a paragraph.
Batching offline work
Work that nobody is waiting on, such as nightly enrichment, backfills or evaluation runs, should not compete with interactive traffic. Many providers offer asynchronous batch interfaces with relaxed turnaround times and different pricing; check your provider's current terms. Even without one, a queue with controlled concurrency keeps offline jobs from exhausting the rate limits your live users depend on.
Timeouts, retries and budgets
- Set timeouts per step, derived from an end-to-end deadline. If a user request has a deadline, each downstream call should get what remains of it, not a fixed generous default.
- Retry only what is retryable. Rate-limit and transient server errors justify a retry with jittered backoff; validation failures usually need a different prompt, not the same one again.
- Cap agent and pipeline loops. Any loop that calls a model needs a maximum number of iterations and a token budget, enforced in code.
- Budget per tenant and per feature. Track token spend by feature and customer so a single runaway workflow is visible within minutes, not at the end of the billing cycle.
The common thread is measurement. Without per-call records of tokens, model, latency and cache outcome, every optimization is a guess. With them, you can rank the levers by actual impact and pull the ones that matter for your traffic.
Examples and figures in this article are illustrative. Adapt thresholds and tooling to your own system and data.