Loading page
P.06Deployment
Instrument an LLM application with tracing, then build dashboards and alerts for cost, latency, errors and output quality signals.
01
Once an LLM feature is running, questions arrive quickly: why was this response slow, which prompt version produced that answer, and why did spend increase last week? Without tracing and structured logs, these questions are answered by guesswork.
In this project you take an existing LLM service — such as one of your earlier projects — and add end-to-end observability. You trace every request through retrieval, model calls and tool use, record token usage and cost, and set up dashboards and alerts that point to the cause of a problem, not just its symptoms.
02
The application emits OpenTelemetry traces with spans for each pipeline stage and model call. An OpenTelemetry Collector forwards data to a tracing backend and a metrics store, and Grafana dashboards and alert rules sit on top.
Instrument
Add spans for each request stage and attach attributes such as prompt version, model, token counts and cache hits.
Log safely
Record inputs and outputs with redaction rules and sampling so sensitive data is not stored by default.
Collect
Route traces and metrics through an OpenTelemetry Collector to the tracing backend and metrics store.
Visualize
Build dashboards for latency percentiles, cost per request, error rates and usage by feature.
Alert
Define alert rules for latency, error and spend thresholds, with links back to example traces.
03
Suggested stack. Equivalent tools are fine; document the choice in your decision record.
04
Stage 01
Choose the application to instrument and list the questions you want observability to answer.
Stage 02
Add spans and attributes across the request path and confirm traces are complete and correctly nested.
Stage 03
Derive token usage, cost estimates and latency metrics, then build the core dashboards.
Stage 04
Define alert rules, simulate incidents such as slow responses and error spikes, and check that alerts fire.
Stage 05
Write a short runbook explaining how to move from an alert to the trace and likely cause.
05
06
Build the foundations, projects, and habits needed to design more dependable AI applications.