LLM Observability and LLMOps: Tracing, Evals, Drift

Last Updated: November 11, 2026

A model that passed your eval suite last month can quietly degrade this week while every dashboard stays green. Latency is flat, error rates are zero, and HTTP 200s keep flowing. Meanwhile support tickets climb because the assistant started hedging, citing a deprecated API, or refusing valid requests. LLM observability exists to catch exactly this: a semantic regression that never trips a classic alert.

The stakes rose in 2026. Agents now chain a dozen model and tool calls per request, the OpenTelemetry GenAI conventions moved into their own repository and renamed core attributes, and the open-source tooling market consolidated through a string of acquisitions. Teams that wired observability by hand in 2024 are now carrying broken dashboards and unportable traces.

This guide treats LLM observability as a measurement system rather than a logging feature. You will leave with a reference architecture, current attribute names, a collector configuration, the statistics behind honest eval gates, a drift method you can run in fifty lines, and a sizing model for the observability stack itself.

What this covers: the four control loops of LLMOps, OpenTelemetry GenAI instrumentation, agent span trees, calibrated LLM-as-judge evals, drift detection, CI eval gating, cost and capacity, tooling choices, failure modes and a checklist.

Context and Background

Application performance monitoring grew up around deterministic services. A request maps to a code path, the path either works or throws, and latency, error rate and a few business counters tell you almost everything. Language models break that contract in three ways.

First, outputs are stochastic. The same prompt yields different completions, so you cannot reproduce a bad answer by replaying the input alone; you need the captured prompt, parameters, model version and retrieved context. Second, cost is per token and highly variable. A runaway agent loop can multiply a bill without moving latency percentiles. Third, quality is semantic. Correctness, faithfulness to retrieved context, tone and safety are not expressible as a status code.

LLMOps is the operational discipline that responds to those properties. It is not MLOps with a new label. Classical MLOps centred on training pipelines, feature stores and model registries for a model you trained. Most LLM applications consume a model someone else trained and differentiate through prompts, retrieval, tools and orchestration. You version prompts instead of weights, evaluate semantic quality instead of validation loss, and debug a request that fans out across retrievers and tools. Our guide to prompt and model version management with registries and eval gates covers the release side of that lifecycle in depth.

The plumbing has standardised around OpenTelemetry. Its GenAI semantic conventions define spans, metrics and events for model calls, agents, tools and the Model Context Protocol (MCP). Two 2026 facts matter for anyone instrumenting now. Semantic conventions v1.42.0, released on 12 June 2026, moved the GenAI and MCP conventions into a dedicated semantic-conventions-genai repository, leaving redirect stubs on the old documentation pages. And every gen_ai.* attribute is still at Development stability, so names can change. The most visible renames already happened: gen_ai.system became gen_ai.provider.name, and gen_ai.usage.prompt_tokens and gen_ai.usage.completion_tokens became gen_ai.usage.input_tokens and gen_ai.usage.output_tokens.

The vendor layer shifted too. ClickHouse announced its acquisition of Langfuse on 16 January 2026 and stated that Langfuse remains open source under its existing MIT license for core features. Other deals were reported through the spring, including ServiceNow and Traceloop, the company behind the Apache-2.0 OpenLLMetry instrumentation. Arize Phoenix remains self-hostable under the Elastic License 2.0. The lesson is the same one our LLM gateway comparison draws for routing: standardise the seam once, and keep the backend swappable.

LLM Observability Architecture: A Measurement System, Not a Logging System

LLM observability is the practice of capturing every model, retrieval and tool step of a request as a structured trace, then scoring sampled traces for quality, tracking how inputs and outputs drift over time, and feeding both signals into release gates. Tracing explains what happened; evals and drift explain whether it was good.

Figure 1: LLM observability reference architecture. GenAI spans pass through a collector that samples and redacts, land in separate trace and metrics stores, and feed dashboards, online eval workers, SLO alerting and golden-set curation.

Read the diagram left to right. The application and its agents emit spans through the OpenTelemetry SDK, which batches them to a collector. The collector makes the keep-or-drop decision with tail sampling and strips sensitive values before anything is persisted. Payload-bearing traces go to a trace store with a short retention window. Low-cardinality aggregates such as tokens, cost and latency go to a metrics store. Eval workers read sampled traces, write scores back as metrics, raise SLO alerts and nominate traces for the golden dataset.

Four control loops, four clocks

The single most useful mental model for LLM observability is that LLMOps runs four control loops at different speeds, and observability is the sensor layer for all of them.

The request loop runs in milliseconds to seconds. Its sensor is the trace, and its actuators are guardrails, timeouts, retries and fallbacks. The quality loop runs in minutes to hours. Its sensor is online evaluation of sampled traffic, and its actuator is an alert or an automatic rollback. The distribution loop runs over days to weeks. Its sensor is drift detection on inputs and outputs, and its actuator is dataset curation. The change loop runs per pull request. Its sensor is the offline eval suite, and its actuator is the merge gate.

Most failed LLMOps programmes have one loop wired and the others missing. A team with beautiful traces but no online evals can see every request yet cannot say whether quality fell last Tuesday. A team with a CI eval suite but no drift monitoring ships safely against a golden set that slowly stops resembling production. Naming the loops makes the gaps visible during design reviews.

The trace is the unit of evaluation

Classic monitoring aggregates first and drills down later. LLM observability inverts that order. The trace is the unit you score, label, cluster and copy into datasets, so every downstream system depends on traces being complete and joinable.

Completeness means the trace carries enough to replay and judge the request: the prompt template ID and version, rendered variables or a reference to them, model and provider, sampling parameters, retrieved chunk IDs with scores, tool inputs and outputs, finish reasons and token counts. Joinability means a stable correlation ID that links the trace to user feedback events, support tickets and later human labels. Without that ID, a thumbs-down in your product analytics can never be tied to the exact prompt version that caused it.

This is also why the trace store and the metrics store must stay separate. Metrics answer “how much and how often” cheaply. Traces answer “why” expensively. Mixing them, for example by tagging metrics with user IDs or prompt hashes, explodes time-series cardinality and slows every dashboard.

Instrument against the GenAI conventions, not a vendor SDK

Portable LLM observability starts with standard attribute names. The current conventions require gen_ai.operation.name and gen_ai.provider.name on inference spans. gen_ai.request.model is conditionally required, and gen_ai.response.model plus input and output token usage are recommended. Inference span names follow the pattern {gen_ai.operation.name} {gen_ai.request.model}, for example chat gpt-4. Defined operation values include chat, text_completion, generate_content, embeddings, retrieval, execute_tool, create_agent and invoke_agent, with newer workflow and memory operations under development.

Prompt and completion content is deliberately not a default span attribute. The conventions carry it as structured gen_ai.input.messages and gen_ai.output.messages, emitted only when content capture is switched on. The old flat gen_ai.prompt and gen_ai.completion attributes were removed without a direct replacement. Instrumentation libraries expose content capture through environment variables, but the exact variable values differ by library, so check each one rather than assuming.

A minimal manual span for a chat call, using current attribute names, looks like this:

from opentelemetry import trace

tracer = trace.get_tracer("support-assistant")

def call_model(client, model, messages, prompt_version, capture=False):
    with tracer.start_as_current_span(f"chat {model}") as span:
        span.set_attribute("gen_ai.operation.name", "chat")
        span.set_attribute("gen_ai.provider.name", "openai")
        span.set_attribute("gen_ai.request.model", model)
        span.set_attribute("gen_ai.request.temperature", 0.2)
        span.set_attribute("app.prompt.version", prompt_version)  # your own namespace
        resp = client.chat.completions.create(
            model=model, messages=messages, temperature=0.2)
        span.set_attribute("gen_ai.response.model", resp.model)
        span.set_attribute("gen_ai.usage.input_tokens", resp.usage.prompt_tokens)
        span.set_attribute("gen_ai.usage.output_tokens", resp.usage.completion_tokens)
        span.set_attribute("gen_ai.response.finish_reasons",
                           [c.finish_reason for c in resp.choices])
        if capture:  # opt-in only, redact downstream
            span.add_event("gen_ai.client.inference.operation.details",
                           {"gen_ai.input.messages": str(messages)})
        return resp

Two details deserve emphasis. Your own attributes, such as prompt version or tenant tier, belong in an application namespace rather than gen_ai.*, so a future convention change never collides with them. And finish reasons are cheap insurance: a length finish reason marks a truncated answer that looks healthy in every latency chart. In practice, most teams should let an auto-instrumentation library such as OpenLLMetry or OpenInference emit these spans and hand-write spans only for custom agent loops. Our deep dive on OpenTelemetry GenAI semantic conventions and collector pipelines walks through attribute coverage library by library.

The collector is the control plane

In an LLM observability pipeline, the OpenTelemetry Collector is where sampling, redaction and routing policy live. Centralising that policy means you change it in one configuration instead of redeploying every service. A practical baseline keeps every error and every slow trace, samples a fraction of healthy traffic, and redacts secrets and personal data before export:

processors:
  tail_sampling:
    decision_wait: 15s
    policies:
      - name: keep-errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: keep-slow
        type: latency
        latency: {threshold_ms: 12000}
      - name: baseline
        type: probabilistic
        probabilistic: {sampling_percentage: 10}
  redaction:
    allow_all_keys: true
    blocked_values:
      - "[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}"   # email
      - "sk-[A-Za-z0-9]{20,}"                               # API-key-like
    summary: info
  batch: {}
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling, redaction, batch]
      exporters: [otlphttp/traces]

Tail sampling must see a whole trace before deciding, which is why decision_wait matters for agents. If an agent request routinely runs for 40 seconds and the wait is 15 seconds, late spans arrive after the decision and you get fragmentary traces. Set the wait above your p99 agent duration, and if you run several collector replicas, route by trace ID so all spans of one trace reach the same instance. Regex redaction is a floor, not a ceiling; it catches structured secrets but not a customer describing a medical condition in free text. Pair it with retention limits and access controls on the trace store.

Tracing Agents and RAG: Reading the Span Tree

Agents are where LLM observability earns its keep. A single chat completion is one span. An agent request is a tree. The root invoke_agent span represents the user’s request, and its children represent each retrieval, each planning call, each tool execution and each guardrail check. Context propagation stitches them together, so one trace shows exactly why an agent took five model calls and two tool runs to answer a simple question.

Figure 2: One agent request as a span tree. The root invoke_agent span parents guardrail checks, a retrieval span, two planning chat spans with their tool executions, and the final answer span. Dotted edges show data dependencies rather than parent-child links.

The solid edges in Figure 2 are parent-child relationships in the trace. The dotted edges show which outputs fed which later steps: retrieved chunks feed the final answer, and the first tool result shapes the second plan. Your trace viewer will not draw the dotted edges for you, which is precisely why recording chunk IDs and tool outputs as attributes matters. They let you reconstruct data flow, not just timing.

Generation failure or retrieval failure?

The most common debugging question in retrieval-augmented generation (RAG) is whether the model failed or the retriever did. Record the retrieved chunk IDs, their similarity scores and their rank positions on the retrieval span. When a user reports a wrong answer, you check whether the chunk containing the correct fact was retrieved at all. If it was not, no prompt change will fix it; the problem lives in chunking, embeddings or the index. If it was retrieved but ranked eighth behind seven distractors, you have a reranking problem. If it was ranked first and the model still ignored it, you have a generation or prompt problem.

That three-way split is the highest-leverage diagnostic that LLM observability gives RAG teams. It requires nothing more than consistent attributes on the retrieval span plus a judge or human who can say which chunk was relevant. Our post on RAG retrieval evaluation with recall at k, nDCG and golden sets covers the offline metrics that complement this per-trace triage.

Latency and cost decomposition

Every child span carries its own start and end time, so total latency decomposes into retrieval, model and tool time at a glance. Consider an illustrative agent trace with a 9.4-second total. Retrieval takes 0.3 seconds, the two planning calls take 1.1 and 1.4 seconds, the final answer takes 2.2 seconds, and the two tools take 0.4 seconds and 3.6 seconds. The slowest component is not a model at all; it is a synchronous SQL tool scanning an unindexed table. An aggregate latency chart would have pushed the team toward a faster model, which would have saved perhaps a second while the real three-and-a-half-second problem sat untouched.

Cost decomposes the same way. Suppose the system prompt for the planner is 2,800 tokens and the agent loops four times on a hard query. That one template accounts for 11,200 input tokens before any user content, and it repeats on every iteration. Summing token usage per span name across a day of traces usually reveals that one or two templates dominate spend. Prompt caching, a shorter planner prompt or a cheaper planning model then becomes an evidence-based change rather than a guess. The economics behind those choices are worked through in our guide to LLM token economics and the gross margin of AI features.

Agent-specific signals worth recording

Agents fail in ways single calls cannot. Record the iteration count per request, because a rising mean iteration count is an early warning of a planner that has started looping. Record tool error rates per tool, because a third-party API that starts returning empty results often shows up as the agent retrying with paraphrased queries rather than as an error. Record which guardrail fired and why, because a sudden jump in refusals after a model upgrade is a quality regression that looks like a safety improvement.

The conventions now define agent-level metrics such as gen_ai.invoke_agent.duration, gen_ai.invoke_agent.tool_calls and gen_ai.invoke_agent.inference_calls, alongside the long-standing client metrics gen_ai.client.token.usage and gen_ai.client.operation.duration. Because these are still at Development stability, treat the names as a target and keep a mapping layer in your collector so a rename does not break dashboards.

For serving-side latency, the conventions also describe server metrics such as gen_ai.server.time_to_first_token and gen_ai.server.time_per_output_token. If you self-host inference, these connect your observability stack to capacity planning; our guide to LLM inference SLOs for TTFT, TPOT and goodput explains how to turn them into service-level objectives.

Evals That Hold Up: Offline, Online and Calibrated Judges

Tracing tells you what happened. Evaluation tells you whether it was good. The two eval modes run on different clocks and answer different questions, and LLM observability is incomplete without both.

Offline evals run a fixed golden dataset before a change ships. Because the inputs and scoring are fixed, a moving number signals a real change, which makes offline evals the right tool for merge gates. Online evals score a sample of live production traffic continuously. They trade control for realism and surface failure modes your golden set never imagined, such as questions about a product feature that launched last week.

Figure 3: The quality and distribution loops. Sampled production traces and user feedback flow through heuristic checks, a calibrated LLM judge and a human review queue. Scores and drift signals drive triage, which curates a versioned golden set that gates the next release.

The loop in Figure 3 is what keeps an eval suite alive. Online scores and drift signals flag low-quality clusters, triage reads the actual traces, and confirmed failures become new golden-set examples. The next offline run then guards against the exact failure production just surfaced. Note the calibration edge: human labels flow into a kappa check that decides whether the judge can still be trusted.

Three scoring families

Heuristic checks are cheap, deterministic and fast: JSON-schema validation, regex rules, length limits, citation presence, forbidden-phrase lists. They catch structural failures and should run on every response, not a sample. Model-graded evals, the LLM-as-a-judge pattern, use a strong model with a rubric to rate faithfulness, relevance, helpfulness or policy compliance. Human review is the ground truth you can only afford in samples.

Eval type	What it scores	Relative cost	Repeatability	Best use
Heuristic	Schema, format, length, keywords	Very low	Deterministic	Every response, hard gates
Reference similarity	Closeness to a reference answer	Low	High	Golden sets with known answers
LLM-as-a-judge	Faithfulness, relevance, tone	Medium to high	Moderate	Sampled online traffic, CI suites
Pairwise judge	Which of two outputs is better	Medium to high	Moderate	Prompt and model A/B comparisons
Human review	Nuanced correctness, harm	Very high	Varies by rater	Calibration and disputed cases

LLM-as-a-judge works only after calibration

The foundational study here is Zheng and colleagues’ “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” (arXiv:2306.05685). It reported that strong judges such as GPT-4 reached over 80% agreement with human preferences, the same level humans reach with each other. The same paper documented position bias, verbosity bias and self-enhancement bias, along with limited reasoning on some tasks. Both halves of that finding matter: judges are useful, and they are biased in predictable directions.

Calibration turns a judge from an opinion into an instrument. Have two or more humans label 200 to 400 traces against the same rubric, measure human-human agreement, then measure judge-human agreement with Cohen’s kappa, which corrects for chance agreement. If the judge’s kappa against humans is close to the humans’ kappa against each other, the judge is fit for that rubric. If it falls well short, tighten the rubric, add reference answers, or switch to pairwise comparison, which is usually more reliable than absolute scoring.

Mitigations for the documented biases are mechanical. Run pairwise comparisons twice with the order swapped and count only consistent verdicts. Ask for a short rationale before the score, and anchor scoring to a reference answer where one exists. Use a judge from a different model family than the system under test, which reduces self-preference. Recalibrate whenever you change the judge model, the rubric or the product domain, because a judge calibrated on billing questions has not been calibrated on legal ones.

The statistics of an honest eval gate

This is where many LLM observability programmes quietly fail. Most eval gates are statistically naive. A team sees a pass rate fall from 86% to 83% on a 200-example golden set and blocks the merge, or worse, sees it rise and celebrates. Both reactions ignore sampling error.

For a pass rate near 85% on 200 examples, the standard error is the square root of 0.85 times 0.15 divided by 200, which is about 0.025. The 95% confidence interval is therefore roughly plus or minus 4.9 percentage points. A three-point drop is inside the noise. To detect a three-point drop between two versions with 95% confidence and 80% power, a standard two-proportion calculation needs roughly 2,200 examples per arm. Few teams have a golden set that large.

Three practical responses follow. Use paired comparisons on the same examples, which removes between-example variance and is far more sensitive than comparing two independent pass rates; McNemar’s test on the discordant pairs is the classic choice. Run each version with several seeds or samples and average, because both the system and the judge are stochastic. And split the gate: a hard gate on deterministic checks and safety categories where any regression blocks, and a soft gate on judged quality that blocks only when the paired difference is significant.

from math import comb

def mcnemar_exact_p(b: int, c: int) -> float:
    """b = old pass / new fail, c = old fail / new pass, on the same examples."""
    n, k = b + c, min(b, c)
    if n == 0:
        return 1.0
    tail = sum(comb(n, i) for i in range(k + 1)) / 2 ** n
    return min(1.0, 2 * tail)

# 400 paired examples: 19 regressions, 7 improvements
print(round(mcnemar_exact_p(19, 7), 4))  # ~0.029 -> block the merge

In that example, the overall pass rate fell by only three points, which an unpaired comparison would dismiss as noise. The paired test sees 19 regressions against 7 improvements on identical inputs and flags it as significant at the 5% level. That is the difference between a gate that catches real regressions and one that trains engineers to click override.

Drift Detection Without Boiling Frogs

Quality erodes for two reasons: your system changed, or the world did. Output drift, where responses shift while inputs stay stable, usually points to a model version change, a provider-side update or a prompt edit. Input drift, where users ask new kinds of questions, points to a world your golden set no longer represents. Both are detectable, and drift often shows up before judged quality moves, which makes it an early-warning layer for LLM observability.

Measure distributions, not averages

Averages hide drift. A mean response length can stay constant while the distribution splits into very short refusals and very long rambles. Track distributions of a few cheap scalar signals: response length, refusal rate, citation count, tool-call count, judge score and the share of finish reasons. For each, compare the current window against a fixed reference window.

The population stability index (PSI) is a simple, interpretable comparison. Bin the reference distribution, compute the share of observations in each bin for both windows, and sum the difference in shares multiplied by the log of their ratio. As a worked example, take four equal reference bins of 25% each. If the current window shifts to 15%, 25%, 25% and 35%, the PSI is about 0.085. Practitioners in credit risk commonly treat values below 0.1 as stable and above 0.25 as a major shift. Those cut-offs are a convention, not a standard, so tune them on your own history.

Embedding drift catches topic shift

Scalar signals miss topic shift. Users can start asking about a new product line with identical lengths and refusal rates. For that, embed inputs with a fixed embedding model, cluster the reference window with k-means, then assign each new input to its nearest reference centroid. Track the share of traffic per cluster with PSI, and track the share of inputs whose distance to every centroid exceeds a threshold. That second number is your novelty rate, and a rising novelty rate is the clearest signal that the golden set needs new examples.

import numpy as np
from sklearn.cluster import KMeans

def psi(ref_share, cur_share, eps=1e-6):
    r = np.clip(ref_share, eps, None); c = np.clip(cur_share, eps, None)
    return float(np.sum((c - r) * np.log(c / r)))

def embedding_drift(ref_emb, cur_emb, k=24, novelty_q=0.99, seed=7):
    km = KMeans(n_clusters=k, n_init=10, random_state=seed).fit(ref_emb)
    ref_d = km.transform(ref_emb).min(axis=1)
    threshold = np.quantile(ref_d, novelty_q)          # 1% novelty in reference
    cur_d = km.transform(cur_emb).min(axis=1)
    ref_share = np.bincount(km.labels_, minlength=k) / len(ref_emb)
    cur_share = np.bincount(km.predict(cur_emb), minlength=k) / len(cur_emb)
    return {"cluster_psi": psi(ref_share, cur_share),
            "novelty_rate": float((cur_d > threshold).mean())}

The embedding model must stay pinned. If you upgrade it, the vector space changes and every comparison against the old reference becomes meaningless. Re-embed the reference window whenever the embedding model changes, and record the embedding model version on the drift job’s own output.

Anchor to a fixed reference window

Rolling comparisons, such as today against yesterday, suffer from boiling-frog blindness. A slow daily shift never trips a day-over-day alert yet accumulates into a large quarter-over-quarter change. Anchor to a known-good period, such as the two weeks after your last validated release, and refresh the anchor deliberately when you ship a validated change. When drift fires, the first move is to open the drifted traces and read them. Drift tells you something changed and where; only the traces tell you what changed and whether it hurts.

User feedback as a drift and quality signal

Explicit feedback is sparse and biased toward strong reactions. Implicit feedback is richer and already sits in your product telemetry. A user who immediately rephrases the same question is signalling dissatisfaction. A regenerate click is a weak negative signal, a copy-to-clipboard on a code answer is a weak positive one, and an abandoned session right after a response is a weak negative. Attach these events to the trace through the correlation ID and you get a quality proxy covering far more traffic than any judge budget allows. A dip in implicit signals on one route that coincides with an embedding shift on the same route is a strong reason to page someone.

Regression Gating in CI and Release

The change loop is where LLM observability meets the release process. It turns evals into engineering discipline. Prompts, model identifiers, retrieval settings and tool schemas are versioned artifacts that must pass evals before they reach users. The most common production regression is a well-intentioned prompt tweak that fixes one case and breaks five others, and a gate stops it at the door.

Figure 4: The change loop as a sequence. A pull request triggers the golden set across several seeds, the gate blocks or publishes a new prompt version, and a canary slice is scored by online evals before promotion or rollback.

The sequence in Figure 4 has two gates, not one. The offline gate catches regressions on known cases before merge. The canary gate catches regressions that only appear on real traffic, by routing a small slice to the new version and scoring it with the same online evaluators used in production. Automatic rollback should trigger on hard signals such as schema-failure rate, refusal rate or safety-category hits, while judged-quality differences should open a review rather than roll back unattended, because small canary samples are noisy.

What belongs in the golden set

A useful golden set is stratified, not random. Include common intents in proportion to traffic, a deliberate over-sample of high-risk intents such as payments or medical questions, adversarial and jailbreak attempts, known past failures with their fixes, and a slice of recent novel inputs from drift detection. Tag every example with its source and stratum so you can report pass rates per stratum. An aggregate pass rate that hides a collapse in one small, high-risk stratum is the classic way a gate passes a dangerous change.

Version the golden set like code. Each eval run records the dataset version, the judge model and rubric version, the system version under test and the seeds used. Without that record, a pass-rate change between two runs could be caused by any of four things, and you cannot tell which.

Provider-side changes are changes too

Not every regression arrives through your repository. Hosted model aliases can point to new snapshots, providers adjust safety systems, and default parameters shift. Pin explicit model snapshots where the provider offers them, record gen_ai.response.model on every span so you see what actually served the request, and run a scheduled nightly eval against production configuration even when nothing in your code changed. A nightly run that suddenly fails with no corresponding commit is the cleanest possible evidence of an upstream change.

Sizing and Costing the Observability Stack

LLM observability is not free, and its two largest costs are usually trace storage and judge tokens. A short sizing model prevents surprises. All figures below are illustrative assumptions for a mid-size product, not measurements of any specific system.

Assume 1 million agent requests per day, eight spans per trace, content capture on, an average of 6,000 input tokens and 500 output tokens of payload per request, and about four characters per token. Raw payload is roughly 26 KB per trace, plus attributes, so call it 30 KB, or about 30 GB per day before compression. Columnar trace stores typically compress repetitive text well; at an assumed 5:1 to 8:1 ratio, that is 4 to 6 GB per day. With a 14-day payload retention window you hold roughly 50 to 85 GB, which is modest. With 13 months of retention, it becomes a multi-terabyte, high-sensitivity archive that your security team will rightly question.

The biggest storage lever is deduplicating templates. In most systems the rendered system prompt is identical across thousands of requests. Store the template ID and version once in a registry and record only the variables per trace, and payload volume often falls by more than half. The same trick makes traces easier to read, because a reviewer sees which template ran instead of scrolling past 3,000 tokens of boilerplate.

Judge cost scales with sampling rate. Judging 5% of 1 million requests is 50,000 judge calls per day. If each judge call reads 3,000 input tokens and writes 200 output tokens, that is 150 million input and 10 million output tokens per day. At hypothetical prices of $1 per million input tokens and $4 per million output tokens, the judge bill is about $190 per day, or roughly $5,700 per month. Substitute your provider’s real prices; the structure of the calculation is the point. Stratified sampling, a smaller calibrated judge for routine rubrics, batch APIs for non-urgent scoring and prompt caching on the fixed rubric prefix can each cut that figure substantially.

Finally, instrumentation must stay off the critical path. Export spans asynchronously through a batch processor, run evals in workers that consume sampled traces, and never call a judge synchronously inside a user request unless it is acting as a guardrail with its own latency budget.

Choosing LLM Observability Tooling in 2026

The LLM observability tooling market consolidated during 2026, which is the strongest argument yet for instrumenting against OpenTelemetry rather than a proprietary SDK. The table compares architectural properties of common choices; it is a starting point, not a ranking, and you should confirm current licensing and features against each project’s documentation before committing.

Option	License model	Instrumentation path	Strengths	Watch for
Langfuse	MIT for core features, per ClickHouse’s announcement	Native SDKs plus OpenTelemetry ingestion	Tracing, prompt management, datasets and evals in one self-hostable product	Which features sit outside the MIT core
Arize Phoenix	Elastic License 2.0	OpenInference, built on OpenTelemetry	Strong eval and experiment tooling, free self-hosting	ELv2 limits offering it as a managed service
OpenLLMetry	Apache 2.0	OpenTelemetry auto-instrumentation	Vendor-neutral spans into any OTLP backend	It is instrumentation only; pair it with a backend
General APM with GenAI support	Commercial	OTLP ingestion	One pane with infrastructure telemetry	Depth of eval and dataset features varies
Build on OTel plus your warehouse	Your choice	Collector to columnar store	Full control, joins with product data	You build the eval and review UI yourself

A sensible default for most teams is OpenTelemetry instrumentation, a collector you control, and a backend chosen for its eval workflow rather than its tracing, because tracing is now a commodity while calibrated evaluation and dataset curation are not. Keep the collector configuration in version control, and you can dual-export to a second backend for a migration without touching application code.

Trade-offs, Gotchas, and What Goes Wrong

Content capture versus privacy. Prompts and completions often contain personal data, secrets or regulated information. Persisting them turns the trace store into a compliance liability. Redact at the collector, keep content capture opt-in per route, restrict access to payload-bearing traces, and keep their retention short. Regex redaction does not catch free-text disclosures, so treat payload traces as sensitive even after redaction.

Sampling versus coverage. Full capture gives perfect visibility at a cost that scales with traffic. Aggressive probabilistic sampling cuts cost but can drop the rare failure that matters most. Tail sampling that keeps all errors and slow traces is the pragmatic middle, but it has its own trap: semantic failures return HTTP 200 and look fast, so a status- and latency-based policy samples them at the baseline rate only. Add a policy that keeps traces flagged by cheap inline heuristics, such as a length finish reason, a guardrail hit or a schema failure.

Judges can be gamed. Optimising a system to please a judge rather than users produces scores that rise while real quality stalls. This is Goodhart’s law with a language model in the loop. Keep a human-labelled holdout the team never optimises against, and investigate whenever judged quality and implicit user feedback diverge.

The golden set rots. A static eval set captures the world on the day you built it. As the product changes, the suite keeps testing yesterday’s distribution and stays green while real quality slides. Budget ongoing curation time, prune examples that no longer reflect reality, and track the novelty rate from drift detection as the trigger for new examples.

Alert fatigue. Drift and judged-quality signals are noisy. A stream of false pages teaches the team to mute the channel. Require two corroborating signals before paging, such as a judged-quality drop plus an implicit-feedback dip on the same route, and send single-signal anomalies to a daily triage queue instead.

Unstable conventions. This one is specific to LLM observability in 2026. Every gen_ai.* name is still at Development stability, and the 2026 repository move came with deprecation notices on attributes that were themselves recent replacements. Insulate dashboards with a translation step in the collector or the query layer, pin your instrumentation library versions, and read the changelog before upgrading.

Fragmented agent traces. Agents that hand work to queues, background jobs or other agents lose trace context unless you propagate it explicitly through message headers. The symptom is a root span that ends early and a set of orphan spans with no parent. Make context propagation part of the definition of done for any new tool or queue integration.

Practical Recommendations

For LLM observability, start with the sensor you are missing, not the one that is easiest to buy. If you have no traces, instrument with an OpenTelemetry-native library and route everything through a collector you own; that single step makes every later choice reversible. If you have traces but no quality signal, add cheap heuristics on all traffic and a judge on a small stratified sample, and spend a week calibrating the judge against human labels before you trust a single score.

Then wire the loops together. Give every trace a correlation ID that joins it to feedback events. Turn the golden set into a versioned artifact with strata, and make the CI gate paired and statistically honest. Add drift detection on a fixed reference window, and use its novelty rate to decide when the golden set needs new cases. Finally, write down the retention, redaction and access policy for payload traces before an auditor asks.

A short checklist to run this quarter:

Instrument with current gen_ai.* names, including gen_ai.provider.name and gen_ai.usage.input_tokens.
Route all spans through a collector with tail sampling, redaction and a decision wait above p99 agent duration.
Record prompt template ID and version, retrieved chunk IDs and finish reasons on every request.
Run heuristics on all traffic and a calibrated judge on a stratified sample.
Gate merges with paired tests across several seeds, with hard gates for safety strata.
Score canary traffic with the same online evaluators before full rollout.
Monitor drift against a fixed reference window and track the novelty rate.
Keep payload retention short and template payloads deduplicated.

Frequently Asked Questions

What is LLM observability?

LLM observability is the practice of capturing each step of a language-model request, including prompts, retrieval, tool calls, token usage and latency, as structured traces, then scoring sampled traces for quality and tracking drift over time. It extends classic monitoring, which only sees status codes and latency, with semantic signals such as faithfulness, relevance and refusal rate. The goal is to detect quality regressions that return a healthy HTTP 200 and to explain their root cause from the trace.

How is LLM observability different from traditional APM?

Traditional application performance monitoring assumes deterministic code paths where failures throw errors. LLM systems fail semantically: a fast, successful response can still be wrong or unsafe. LLM observability therefore adds content-aware traces, per-token cost tracking, model-graded and human evaluation, and distribution drift detection. It also treats the individual trace as the unit of analysis, because traces are scored, labelled and copied into eval datasets, whereas APM mostly aggregates metrics and drills down only when something breaks.

What are the OpenTelemetry GenAI semantic conventions?

They are OpenTelemetry’s standard vocabulary for generative AI telemetry, covering spans, metrics and events for model calls, agents, tools and MCP. Required inference attributes include gen_ai.operation.name and gen_ai.provider.name, with model and token-usage attributes recommended. In June 2026, semantic conventions v1.42.0 moved them into a dedicated GenAI repository. All gen_ai.* attributes remain at Development stability, so pin library versions and expect occasional renames such as the earlier move from gen_ai.system to gen_ai.provider.name.

Is LLM-as-a-judge reliable enough for production evals?

It is reliable when calibrated and unreliable when assumed. The MT-Bench and Chatbot Arena study found strong judges agreed with human preferences over 80% of the time, matching human-to-human agreement, but also documented position, verbosity and self-enhancement biases. Calibrate each rubric against a few hundred human-labelled examples using Cohen’s kappa, swap answer order in pairwise comparisons, use a judge from a different model family, and recalibrate when the judge, rubric or domain changes.

How do you detect drift in an LLM application?

Compare current distributions against a fixed, known-good reference window. For scalar signals such as response length, refusal rate and judge score, use a measure like the population stability index. For topic shift, embed inputs with a pinned embedding model, cluster the reference window, and track cluster shares plus a novelty rate for inputs far from every cluster. When drift fires, read the affected traces before acting, because drift shows that something changed, not whether it hurts.

What is the difference between LLMOps and MLOps?

MLOps manages the lifecycle of models you train: data pipelines, feature stores, training runs, registries and deployment. LLMOps manages applications built mostly on models someone else trained, so the versioned artifacts are prompts, retrieval settings, tool schemas and model identifiers rather than weights. Evaluation shifts from validation loss to semantic quality, and observability must follow requests through retrievers, tools and agent loops. Teams that also fine-tune models need both disciplines working together.

References

OpenTelemetry, semantic-conventions-genai repository: GenAI spans, metrics, events and MCP conventions.
OpenTelemetry, Semantic conventions for generative AI metrics: legacy documentation location, now pointing to the GenAI repository.
Zheng, L. et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, arXiv:2306.05685.
ClickHouse, ClickHouse welcomes Langfuse: the future of open-source LLM observability, 16 January 2026.
Arize AI, Phoenix self-hosting license: Elastic License 2.0.

By Riju — about