LLM Observability and LLMOps: A Production Architecture for Tracing, Evals, and Drift

A model that scored well in your eval suite last month can quietly degrade in production this week, and your dashboards will show nothing. Latency is flat. Error rates are zero. Yet support tickets climb because the assistant started hedging, hallucinating a deprecated API, or refusing valid requests. This is the failure mode that llm observability llmops practices exist to catch — the silent quality regression that never trips a traditional alert. The problem is that quality in a language system is semantic, not a status code, and the cause is often a one-line prompt edit, a model version bump, or a slow drift in the questions users ask.

Treating a large language model deployment like a stateless web service guarantees blind spots. You need to see inside each request: the prompts, the retrieved context, the tool calls, the token economics, and a continuous judgment of whether the output was actually good. That requires a purpose-built stack.

What this covers: a concrete, vendor-neutral architecture for instrumenting LLM and agent systems with OpenTelemetry GenAI conventions, building offline and online evaluations, detecting drift, and gating regressions in CI — plus the trade-offs that bite teams in production.

Context and Background

Application performance monitoring grew up around deterministic services. A request maps to a code path, the code path either works or throws, and latency plus error rate plus a few business metrics tell you almost everything. Language models break every assumption in that model. The same prompt yields different completions across calls because sampling is stochastic. A response can return in 800 milliseconds with a clean HTTP 200 and still be wrong, biased, or unsafe.

Three properties make LLM systems uniquely hard to observe. First, non-determinism means you cannot reproduce a bad output by replaying inputs alone; you need the captured prompt, parameters, and seed. Second, cost is per-token and highly variable, so a runaway agent loop can ten-times your bill without raising a latency alarm. Third, quality is semantic — correctness, faithfulness to context, tone, and safety — none of which a status code expresses.

This gap birthed a category. The OpenTelemetry GenAI semantic conventions now define a standard vocabulary for model spans, so a trace records gen_ai.request.model, token counts, and the operation as portable attributes rather than vendor-specific blobs. On top of that base, tools such as Langfuse, Arize Phoenix, Traceloop’s OpenLLMetry, and Braintrust add prompt management, evaluation runners, and dataset curation. The common thread is the trace: a structured record of one request’s full journey through your prompts, retrievers, and tools.

Adopting these conventions early matters because it keeps you portable. If you instrument with OTel GenAI spans rather than a proprietary SDK, you can change backends without re-instrumenting. That portability is the same discipline we argued for in our LLM gateway architecture guide: standardize the seam between your application and the model layer once, and you keep optionality as the ecosystem churns. The rest of this post builds the observability layer that sits behind that seam.

The word “LLMOps” is doing real work here, not just rebranding MLOps. Classical machine-learning operations centered on training pipelines, feature stores, and model registries — the lifecycle of a model you trained. Most LLM applications consume a model someone else trained and differentiate through prompts, retrieval, tools, and orchestration. The operational surface shifts accordingly: you version prompts instead of weights, you evaluate semantic quality instead of validation loss, and you monitor a request that fans out across retrievers and tool calls rather than a single inference. Observability is the foundation that makes that shifted lifecycle manageable, because none of the new control points — prompt changes, retrieval quality, agent loops — are governable if you cannot see them.

The Observability Architecture

An LLM observability architecture has five stages: instrument the application with GenAI spans, export them through a collector that samples and redacts, store traces and metrics in queryable backends, render them in trace-aware dashboards, and feed sampled traces into evaluation and drift workers that loop scores back to alerting. The trace is the spine — everything else hangs off it.

Figure 1: End-to-end LLM observability pipeline — GenAI spans flow through an OpenTelemetry collector that samples and redacts, into trace and metrics stores, then fan out to dashboards, online eval workers, and drift-aware alerting.

The diagram traces one path: your application and agents emit spans through the OpenTelemetry SDK; the collector applies sampling and PII redaction; processors split the stream into a trace store and a metrics store for token and cost aggregates; dashboards read both; and online eval workers pull sampled traces, produce quality scores and drift signals, and route anomalies to alerting tied to your service-level objectives. Each stage is independently scalable and swappable.

Instrumentation with GenAI spans

Instrumentation is where signal is born or lost. Under the OpenTelemetry GenAI conventions, every model interaction becomes a span carrying gen_ai.* attributes: the system name, request model, response model, token usage, and operation. Prompts and completions are captured as span events or attributes, gated behind a content-capture flag because they often contain sensitive data. Token counts and a derived cost field let you aggregate spend per route, per user, or per feature without log scraping.

Most teams should not hand-roll this. Libraries like OpenLLMetry and the Phoenix and Langfuse SDKs auto-instrument popular client libraries, emitting conformant spans for you. Where you do write spans manually — custom agent loops, bespoke retrievers — keep the attribute names conformant so backends parse them natively. A minimal manual span looks like this.

from opentelemetry import trace

tracer = trace.get_tracer("llm.app")

def call_model(client, model, messages):
    with tracer.start_as_current_span("chat") as span:
        span.set_attribute("gen_ai.operation.name", "chat")
        span.set_attribute("gen_ai.request.model", model)
        resp = client.chat.completions.create(model=model, messages=messages)
        usage = resp.usage
        span.set_attribute("gen_ai.usage.input_tokens", usage.prompt_tokens)
        span.set_attribute("gen_ai.usage.output_tokens", usage.completion_tokens)
        span.set_attribute("gen_ai.response.model", resp.model)
        # content capture gated behind a flag for PII safety
        if CAPTURE_CONTENT:
            span.add_event("gen_ai.content.prompt", {"messages": str(messages)})
        return resp

That span is portable. Any OTel-compatible backend ingests it, and the token attributes feed cost dashboards directly. The content event is opt-in, which keeps you from logging customer data by default.

A few attributes carry outsized value and are worth capturing on every span. Record gen_ai.request.temperature and top_p so you can correlate output variance with sampling settings when a route suddenly turns flaky. Capture gen_ai.response.finish_reasons to distinguish a clean completion from one truncated by a length cap — truncation is a silent quality killer that looks fine until you read the cut-off answer. Add a stable gen_ai.request.id or your own correlation ID so a user-reported bad answer maps back to the exact trace. These cost almost nothing to set and save hours of guesswork later.

The trace tree for agents, tools, and RAG

A single chat completion is one span. An agent request is a tree. The root span represents the user’s request; child spans represent each retrieval, each model call in the planning loop, and each tool invocation. Span context propagation links them, so you can open one trace and see exactly why an agent took nine model calls and three tool runs to answer a simple question.

Figure 2: A single agent request as a span tree — the root agent span parents a RAG retrieval, multiple LLM planning calls, and a tool execution, with results flowing back up to the user.

This tree is the single most valuable artifact in LLM debugging. When an agent misbehaves, the trace shows the precise prompt that produced a bad tool decision, the context chunks the retriever returned, and where latency or cost concentrated. Without it, you are reconstructing a multi-step reasoning failure from disconnected logs. With it, root-causing a bad answer is a matter of reading one tree top to bottom.

For retrieval-augmented generation specifically, capture the retrieved chunk IDs and similarity scores as span attributes. That lets you answer the recurring question — was this a generation failure or a retrieval failure? — without guessing. If the right chunk was never retrieved, no amount of prompt tuning fixes it; the problem lives upstream in your index or embedding model.

Span timing on the tree is its own diagnostic. Because each child span carries its own start and end, you can decompose total latency into retrieval time, model time, and tool time at a glance. A common surprise is that the slow part is not the model at all — it is a synchronous tool call to a third-party API, or a retriever doing an expensive reranking pass. Without the tree you would blame the model and tune the wrong knob. With it, the bottleneck is visually obvious: the widest bar in the waterfall is where your latency budget goes.

Cost decomposition works the same way. Sum the token-derived cost across a trace’s child spans and you learn that one verbose intermediate planning prompt, repeated every loop iteration, accounts for most of an agent’s spend. That insight is invisible in an aggregate dashboard but jumps out of a single annotated trace. The discipline of reading individual traces, not just rollups, is what turns observability from a reporting exercise into an optimization tool.

Collector pipeline, storage, and dashboards

The collector is your control plane. It receives spans over OTLP, applies tail-based sampling so you keep all error and high-latency traces while sampling the rest, and runs redaction processors that strip emails, keys, and other PII before anything is persisted. Centralizing this logic means you change sampling and redaction policy in one config rather than redeploying every service.

Storage splits by access pattern. Traces with full prompt and completion payloads go to a trace store optimized for span queries and tree reconstruction. Aggregate metrics — tokens, cost, latency percentiles, eval scores — go to a time-series store for cheap dashboards and alerting. Keeping bulky trace payloads out of your metrics path keeps dashboards fast and your retention bill sane. Token and cost metrics here also feed capacity planning, a theme we develop in our piece on AI inference cost optimization.

Dashboards must be trace-aware, not just metric-aware. A good LLM dashboard lets you pivot from a spend spike or a quality dip straight into the underlying traces. You see a drop in faithfulness score on one route, click through, and read the actual traces that scored poorly. That pivot — aggregate signal to concrete example — is what separates LLM observability from generic metrics dashboards bolted onto a model API.

Cardinality is the quiet trap in this layer. It is tempting to tag every metric with the user ID, the full prompt hash, and the model version, but high-cardinality labels explode the cost of a time-series store and slow queries to a crawl. Keep aggregate metrics low-cardinality — route, model, operation, status — and push the high-cardinality detail into the trace store where it belongs. The split between a lean metrics path and a rich trace path is precisely what lets you keep both fast dashboards and deep drill-down without paying for one to get the other.

Retention policy deserves explicit thought rather than a default. Traces carrying full prompt and completion payloads are large and sensitive, so a short retention window — days, not months — keeps both storage cost and compliance exposure bounded. Aggregate metrics and eval scores are small and non-sensitive, so you can retain them far longer to support trend analysis across quarters. Tiering retention by sensitivity and size, rather than applying one window to everything, is how mature teams keep observability affordable as traffic grows.

Evals, Drift, and the Feedback Loop

Tracing tells you what happened; evaluation tells you whether it was good. The two operate on different clocks. Offline evals run against a fixed golden dataset before you ship, gating changes in CI. Online evals run continuously against sampled production traffic, catching the regressions that only appear with real user inputs. Together they form a feedback loop: production traces seed datasets, datasets gate changes, changes deploy, and online scores confirm or refute the improvement.

Figure 3: The eval and drift feedback loop — sampled production traces and user feedback feed an LLM judge and heuristics; scores plus embedding-based drift signals curate a golden set that drives prompt and model fixes back into production.

Offline evals are deterministic in structure: a curated set of inputs with expected properties, scored the same way every run, so a number moving means a real change. Online evals trade that control for realism. They score whatever production actually sent, which surfaces failure modes your golden set never imagined. You need both, and you need them sharing a scoring vocabulary so an offline regression and an online drop are comparable quantities.

The two clocks create a healthy tension. Offline evals answer “did this change make things worse on the cases we already care about?” before any user sees it. Online evals answer “are things actually good right now, on the traffic we actually have?” after deployment. A change can pass offline and still fail online because the golden set lagged reality — users started asking about a product you shipped last week, and your dataset has no examples of it. The fix is mechanical: online evals flag the new low-scoring cluster, you curate examples from it into the golden set, and the next offline run guards against it. The datasets are not static documents; they are a moving record of what production looks like.

Scoring methods and their pitfalls

Three scoring families dominate. Heuristic checks — regex, JSON-schema validation, length and format rules — are cheap, deterministic, and catch structural failures fast. Model-graded evals, the “LLM-as-judge” pattern, use a strong model to rate faithfulness, relevance, or toxicity against a rubric. Human review is the gold standard you can only afford in samples.

LLM-as-judge is powerful and treacherous. Judge models exhibit known biases: they favor longer answers, prefer their own family’s outputs, and are sensitive to option ordering in pairwise comparisons. The literature documents these effects clearly; the MT-Bench and Chatbot Arena study quantifies position and verbosity bias and shows where judge agreement with humans holds and where it breaks. Mitigations include swapping option order and averaging, anchoring scores to a reference answer, and calibrating the judge against a human-labeled subset before trusting it. Never deploy a judge you have not calibrated.

Eval type	What it scores	Cost	Determinism
Heuristic	Format, schema, length, keywords	Very low	High
Embedding similarity	Semantic closeness to reference	Low	High
LLM-as-judge	Faithfulness, relevance, toxicity	Medium-high	Low
Human review	Ground-truth quality, nuance	Very high	Medium

Drift detection and user feedback

Quality erodes for two reasons: your system changed, or the world did. Output drift — your responses shifting distribution while inputs stay stable — usually points to a model version bump or a prompt edit. Input drift — users asking new kinds of questions — points to a changing world your golden set no longer represents. Both matter; both are detectable.

Embedding-based detection is the workhorse. Embed inputs and outputs, then watch the distribution of those vectors over time using a distance measure such as population stability index or a divergence metric against a reference window. A statistically significant shift fires a signal worth investigating, even before quality scores move, because drift often precedes measurable degradation. Pair this with explicit user feedback — thumbs, corrections, regenerations, abandonment — which is noisy but high-signal, especially when a feedback dip correlates with an embedding shift on the same route.

Implicit feedback is often richer than explicit feedback, and your traces already contain it. A user who immediately rephrases the same question signals dissatisfaction without ever clicking a thumb. A copy-to-clipboard event on a code answer is weak positive signal. An abandoned session right after a response is weak negative signal. Capturing these interaction events as span attributes turns ordinary product telemetry into a continuous, no-cost quality proxy that covers far more traffic than the small fraction of users who bother to rate anything explicitly.

Set drift thresholds against a stable reference window, not a rolling one, or you risk boiling-frog blindness — a slow daily shift never trips a same-day comparison but adds up to a large quarter-over-quarter change. Anchor to a known-good period and measure against it. When the signal fires, the right first move is to open the drifted traces and read them: drift detection tells you something changed and where, but only the traces tell you what changed and whether it actually hurts quality.

The loop closes by curation. Traces that score poorly or sit in a drifted region become candidate additions to your golden dataset. You label them, fold them in, and your offline suite now guards against the exact failure production just surfaced. That is the mechanism that makes the suite improve over time instead of rotting. Choices about whether to fix a regression with a prompt change, a retrieval change, or a model swap connect directly to the trade-offs in our fine-tuning versus RAG versus long-context analysis.

Regression gating in CI

The discipline that ties evals to engineering is treating prompts and model configs as versioned artifacts that pass evals before merge. A pull request that edits a prompt triggers the golden dataset, scores the result, and blocks if any metric falls below threshold. This stops the most common production regression — a well-intentioned prompt tweak that helps one case and breaks five others — at the door.

Figure 4: A CI eval-gating pipeline — a prompt or model pull request runs the golden dataset, scores against thresholds, blocks on failure, and on pass proceeds through canary and online-eval checks to full rollout or rollback.

Gating must be statistically honest. A single eval run is noisy because the judge and the model are both stochastic, so compare distributions across multiple runs, not single scores, and set thresholds with that variance in mind. A gate that fails on noise trains your team to override it, which is worse than no gate. After the gate passes, a canary stage with online evals confirms the change behaves on real traffic before full rollout, with automatic rollback if online scores drop.

Trade-offs, Gotchas, and What Goes Wrong

Capturing prompts and completions is where observability collides with privacy. Those payloads frequently contain personally identifiable information, secrets, or regulated data, and persisting them turns your trace store into a compliance liability. Redact at the collector before storage, gate content capture behind explicit flags, and set aggressive retention limits on payload-bearing spans. The convenience of full prompt capture is real, and so is the breach surface it creates.

Sampling is the cost lever and the coverage risk. Full capture gives perfect visibility at storage and processing cost that scales with traffic; aggressive sampling cuts cost but can miss the rare failure that matters most. Tail-based sampling is the pragmatic middle: keep every error and slow trace, sample the boring successful ones. Get the policy wrong and you either bankrupt the observability budget or lose the one trace you needed.

LLM-as-judge introduces its own failure modes. Judges cost real tokens, so evaluating every production trace can rival inference spend; sample deliberately. Judges are biased, as covered above, and they can be gamed — optimizing your system to please a judge rather than users produces eval scores that rise while real quality stalls. Periodically recalibrate against human labels and treat judge scores as a proxy, never as ground truth.

Two operational traps remain. Alert fatigue kills observability programs: drift signals are inherently noisy, and a stream of false alarms teaches the team to ignore the channel, so tune thresholds and require corroborating signals before paging. And instrumentation adds latency — synchronous evals or verbose span exports can slow the request path, so run evals asynchronously off sampled traces and export spans in batches rather than blocking the user’s response.

A subtler failure is the golden dataset that silently rots. A static eval set captures the world as it was the day you built it. As your product and users move on, that set keeps testing yesterday’s distribution, so your evals stay green while real quality slides on the questions people actually ask now. The only defense is the curation loop: continuously fold drifted and low-scoring production cases back into the set, and periodically prune examples that no longer reflect reality. An eval suite is a living asset with maintenance cost, not a one-time deliverable, and teams that forget this discover it the hard way when a “fully passing” suite ships a visible regression.

Finally, beware over-trusting any single number. A composite quality score that aggregates faithfulness, relevance, and toxicity can stay flat while one component collapses and another improves, masking a real problem. Keep component scores visible alongside the composite, and treat every automated metric as a proxy that points you toward tra