Logging and Tracing LLM Applications

A user reports that the assistant gave a wrong answer yesterday afternoon. Without good telemetry, you cannot tell whether retrieval returned the wrong documents, the prompt template changed, the model was swapped, or the answer was cut off by a length limit. With good telemetry, you can usually answer that in minutes. These notes describe the setup we recommend learners build into every project from the first deployment.

What to log for every model call

Think of each model call as a function invocation you may need to replay. The record should let you reconstruct the request, explain the response and attribute its cost.

• Model identity: the exact model name or version string returned by the provider, not just the alias you requested.
• Parameters: temperature, max output tokens, stop sequences, tool choice and any response-format settings.
• Prompt version: a template identifier and version, plus a hash of the fully rendered prompt.
• Token counts: input and output tokens as reported by the provider, and cached-input tokens if applicable.
• Timing: time to first token, total duration and time spent queued or retrying.
• Outcome: finish reason, error class, retry count and whether output passed validation.
• Retrieval context: document and chunk IDs with their scores, rather than the chunk text itself.
• Tool calls: tool name, argument hash or redacted arguments, duration and success or failure.

Raw text versus hashes

The most consequential decision is whether to store prompts and completions verbatim. Raw text makes debugging dramatically easier, and it also turns your logging pipeline into a store of whatever users typed: names, account numbers, health details, pasted credentials. That data then flows to every system your logs reach, often with broader access and longer retention than your primary database.

A workable middle ground is tiered storage. Default telemetry carries hashes, IDs, counts and a short redacted preview. Full payloads, if you keep them at all, go to a separate store with restricted access, a short retention period and an audit trail. A stable hash of the rendered prompt still lets you group identical requests and detect when a template change altered every prompt, without exposing content.

Structuring spans across a pipeline

A single user request in an LLM application often fans out into retrieval, reranking, one or more model calls and tool executions. Distributed tracing models this as a tree of spans that share a trace ID. Each span has a name, a start and end time, a status and a set of attributes.

1POST /chat                         trace_id=4bf9...  1,840 ms2├── retrieval.search               k=20              120 ms3├── retrieval.rerank               kept=5             85 ms4├── llm.generate                   model=large-model 1,410 ms5│   └── tool.lookup_order          status=ok          210 ms6└── output.validate                schema=answer_v2     6 ms

Keep span names low-cardinality and stable, such as llm.generate or retrieval.search, and put variable details in attributes. That keeps dashboards and queries usable. Propagate the trace context into background jobs and queues so asynchronous steps join the same trace instead of appearing as orphans.

Instrumenting with OpenTelemetry

OpenTelemetry provides a vendor-neutral API for traces, metrics and logs, and most observability backends can ingest its data. The project has also been developing semantic conventions for generative AI attributes; check the current conventions and prefer them over invented names where they fit your needs. The example below uses the tracing API directly around a hypothetical model wrapper.

1import hashlib2 3from opentelemetry import trace4 5from app.llm import llm  # hypothetical thin wrapper around your provider client6from app.redact import redact_pii  # your redaction function (see below)7 8tracer = trace.get_tracer("app.llm")9 10 11def digest(text: str) -> str:12    return hashlib.sha256(text.encode("utf-8")).hexdigest()[:16]13 14 15def generate(prompt: str, question: str, chunk_ids: list[str], *,16             model: str, temperature: float, prompt_version: str) -> str:17    with tracer.start_as_current_span("llm.generate") as span:18        span.set_attribute("llm.request.model", model)19        span.set_attribute("llm.request.temperature", temperature)20        span.set_attribute("llm.prompt.version", prompt_version)21        span.set_attribute("llm.prompt.hash", digest(prompt))22        span.set_attribute("llm.input.preview", redact_pii(question)[:200])23        span.set_attribute("retrieval.chunk_ids", chunk_ids)24 25        result = llm.complete(model=model, prompt=prompt, temperature=temperature)26 27        span.set_attribute("llm.response.model", result.model)28        span.set_attribute("llm.usage.input_tokens", result.input_tokens)29        span.set_attribute("llm.usage.output_tokens", result.output_tokens)30        span.set_attribute("llm.response.finish_reason", result.finish_reason)31        return result.text

Two habits make this pay off. First, record the response model separately from the requested one, because aliases can resolve to different versions over time. Second, attach the trace ID to user-facing feedback, such as a thumbs-down, so a complaint links directly to the exact trace.

Traces are also the raw material for metrics. Token counts, durations and finish reasons recorded on spans can be aggregated into dashboards for cost per feature, p95 latency per model and the rate of length-limited responses. At higher volumes you may need to sample traces to control storage costs; if you do, prefer tail-based sampling that keeps every errored or slow trace, so the requests you most need to debug are never the ones that were dropped.

Redacting PII before it leaves the process

Redaction belongs as close to the source as possible, before data reaches an exporter, a log shipper or a third-party backend. Collector-side processors are a useful second layer, but they should not be the only one.

1. Pattern-based redaction for structured identifiers such as emails, phone numbers and card-like digit sequences. Fast and predictable, but it misses free-form personal details.
2. Allowlisting attributes so only known-safe fields are exported. This is stronger than trying to blocklist everything sensitive.
3. Entity detection with an NER model for names and addresses, accepting that it will have false negatives.
4. Truncation of previews so that even a missed detail is limited in scope.

Tool calls deserve extra care

Tool arguments are generated by the model and often contain user data verbatim, such as an order number or an email address passed to a lookup. Apply the same redaction to tool arguments and results as to prompts, and record the tool name and status unconditionally, since those are rarely sensitive and are essential for debugging agent behavior.

Good LLM telemetry is mostly ordinary observability practice applied with discipline: stable names, structured attributes, correlated IDs and a clear policy for sensitive data. Build it before you need it, because the incident that needs it will not wait.