OpenTelemetry GenAI Semantic Conventions: Tracing LLM Calls, Agents, Tokens and Cost (2026)

An LLM request that “worked” can still be a production incident. It can return HTTP 200 after eleven seconds, burn forty thousand input tokens because a retrieval step stuffed the context window, call the same tool four times in a loop, and quietly cost ten times what the pricing spreadsheet predicted. Classic application monitoring sees one slow POST. It cannot tell you which model answered, how many tokens were billed, or which step of the agent loop misbehaved.

The OpenTelemetry GenAI semantic conventions are the vendor-neutral answer: a shared vocabulary of span names, gen_ai.* attributes and metrics so that any instrumentation library, Collector and backend describes an LLM call the same way. They are also still moving, which changes how you should adopt them.

You will leave with a precise map of the span and metric model, the agent and tool span tree, a runnable Python setup, a policy for capturing prompts without leaking personal data, and a Collector pipeline that redacts, samples and derives cost signals.

What this covers: current status and where the spec lives, inference spans and attributes, agent and tool spans, metrics, content capture and PII, instrumentation libraries, the Collector pipeline, cardinality and cost, and failure modes.

Context and Background

Observability for LLM applications grew up in two waves. The first wave was product-specific: each LLM observability vendor and each open-source tracer invented its own schema, so the same call might be recorded as llm.request.model, ai.model.name or openai.model. Dashboards, alerts and cost reports written against one schema broke when the team switched libraries or backends. The second wave is convergence on OpenTelemetry (OTel), the Cloud Native Computing Foundation project that standardises traces, metrics and logs and the OTLP wire protocol.

Semantic conventions are the part of OTel that standardises names. HTTP has http.request.method; databases have db.system.name; generative AI has the gen_ai.* namespace. A convention does not instrument anything by itself. It tells instrumentation authors which attribute to set, so a backend can render token usage without per-vendor glue code.

Two status facts matter before you write any code, and both were verified against current sources for this post. First, the GenAI attributes, spans and metrics are in the Development stability level, not Stable. A secondary source that tracks the spec reports that the only stable attributes appearing on GenAI spans are inherited ones such as error.type, server.address and server.port. Development means names can change without a long deprecation window. Second, the GenAI conventions have moved out of the main semantic-conventions repository into their own repository, open-telemetry/semantic-conventions-genai. The old pages on opentelemetry.io now show a “moved” notice, and the new repository’s README listed its schema URL as “TODO” when I read it. I could not independently confirm the exact release in which the split happened, so treat any date for it as unverified.

The practical consequence is a discipline: pin your instrumentation versions, isolate attribute names behind a thin layer you control, and plan for a rename. If you already run OTel, the foundations in our post on W3C Trace Context, baggage and context propagation apply unchanged, because a GenAI span is an ordinary span that happens to carry well-known attributes. For the broader operating model around evals, drift and cost, see our overview of LLM observability and LLMOps architecture.

Why a standard matters more for LLMs than for HTTP

HTTP latency is roughly comparable across services. LLM calls are not: the same prompt can cost ten times more on one model than another, latency scales with output length, and “success” is a judgement the transport layer cannot make. A shared vocabulary lets you ask portable questions: which model has the highest p95 time-to-answer, which route is the largest consumer of input tokens, which tool fails most often. Without it, every question becomes a bespoke query against a vendor-specific schema.

What the standard deliberately does not cover

Conventions describe the shape of telemetry, not its quality. Whether an answer was correct, grounded or safe is an evaluation problem. You can attach evaluation results as attributes or events, but the spec’s job ends at “what was called, with what, how long, and how many tokens”. Keeping that boundary clear prevents the common mistake of expecting a trace viewer to replace an eval harness.

The GenAI Span Model: One Vocabulary, Four Layers

Direct answer: the GenAI conventions define spans named {gen_ai.operation.name} {gen_ai.request.model}, such as chat gpt-4o-mini. Required attributes are the operation name and the provider name. Usage attributes carry input and output token counts, and agent runs add invoke_agent and execute_tool spans so a whole agent loop becomes one readable trace tree.

Figure 1: Where GenAI telemetry is produced and processed. The instrumentation library sets gen_ai.* attributes, the Collector redacts and samples, and large content can travel by reference instead of inline.

The figure shows the four layers you control. The application and agent framework produce the work. An instrumentation library wraps the provider SDK or framework and translates each call into spans and metrics. The OTel SDK batches and exports over OTLP. The Collector is the policy point where you map legacy names, delete sensitive keys, sample and route. Content that is too large or too sensitive for a span can bypass the span path through a separate store, referenced from the span.

Operations and span names

The attribute gen_ai.operation.name identifies what kind of call this is. The values reported by the sources I checked include chat, text_completion, embeddings, generate_content, retrieval, execute_tool, create_agent and invoke_agent. The span name combines the operation and the model, for example chat gpt-4o-mini or embeddings text-embedding-3-small, so a trace list is scannable without opening spans. When there is no model (a tool execution, say), the span name falls back to the operation plus the tool or agent name in the spec’s rules; check the current agent-span document for the exact format because this is one of the details still under revision.

The span kind for a call to a remote model provider is CLIENT, consistent with other outbound calls. That matters for service maps: the model provider appears as a downstream dependency, and the attributes server.address and server.port (both stable, inherited from core conventions) tell you which endpoint answered. If you proxy through a gateway, the address is the gateway, and gen_ai.provider.name still says which logical provider you meant.

Core inference attributes

The attributes below are the working set for a chat span. Names come from the spec as summarised by the sources cited at the end; requirement levels are noted where a source stated them.

Attribute	Meaning	Notes
`gen_ai.operation.name`	Kind of operation	Required
`gen_ai.provider.name`	Provider such as `openai`, `anthropic`, `aws.bedrock`	Required; replaces the older `gen_ai.system`
`gen_ai.request.model`	Model you asked for	Conditionally required when available
`gen_ai.response.model`	Model that actually answered	Recommended; can differ from the request
`gen_ai.response.id`	Provider response identifier	Useful for support tickets
`gen_ai.response.finish_reasons`	List such as `stop`, `length`, `tool_calls`	Key for diagnosing truncation
`gen_ai.usage.input_tokens`	Prompt tokens billed	Recommended
`gen_ai.usage.output_tokens`	Completion tokens billed	Recommended
`gen_ai.request.temperature`, `.top_p`, `.max_tokens`, `.seed`	Sampling parameters	Recommended; also `.top_k`, penalties and stop sequences
`error.type`	Failure class	Conditionally required on error
`gen_ai.conversation.id`	Groups turns across requests	Conditionally required when available

Two details deserve emphasis. The request and response model fields exist because aliases and routing are common: you ask for a moving alias such as a “latest” tag and the provider answers with a dated snapshot. If you only record the request model, a silent provider-side model change looks like an unexplained quality regression. And finish_reasons is a list because some APIs return multiple choices; a value of length on a span with an unusually high output token count is the signature of a truncated answer that downstream code then tries to parse as JSON.

Token usage, caching and reasoning

Input and output tokens are the base of any cost model. Two further attributes, reported by a source I read for this post, make the model more accurate: gen_ai.usage.cache_read.input_tokens and gen_ai.usage.cache_creation.input_tokens for prompt caching, and gen_ai.usage.reasoning.output_tokens for hidden reasoning tokens. Providers bill cached reads, cache writes and reasoning tokens at different rates from plain tokens, so a cost estimate built only from input_tokens and output_tokens can be wrong by a wide margin on workloads that use caching. Whether a given instrumentation library populates these depends on the provider SDK; check before you build a dashboard on them.

A worked illustration, with made-up prices to show the arithmetic, not real rates: suppose a model charges 3 currency units per million uncached input tokens, 0.30 per million cache-read tokens and 15 per million output tokens. A call with 20,000 input tokens of which 18,000 are cache reads, plus 800 output tokens, costs 2,000 × 3/1e6 + 18,000 × 0.30/1e6 + 800 × 15/1e6 = 0.006 + 0.0054 + 0.012 = 0.0234. If your pipeline treated all 20,000 as uncached it would report 0.06 + 0.012 = 0.072, more than three times too high. The lesson is that cost per request is a function of at least four counters, and the spec gives you names for them.

Agent Observability: Spans for Agents, Tools and Retrieval

Direct answer: an agent run is modelled as one invoke_agent parent span whose children alternate between chat spans (the model deciding) and execute_tool spans (the code acting). Tool spans carry gen_ai.tool.name and a call identifier that links back to the model turn that requested it, so you can see loops, retries and wasted tokens in one tree.

Figure 2: The span tree of a three-turn agent run. The parent is the agent invocation; model turns and tool executions are siblings in order, and tools can have their own downstream client spans.

A single user question to a support agent might produce the tree in Figure 2. The agent invocation starts, the model is called and asks for a tool, the tool executes (and makes an HTTP call to an order API that your normal HTTP instrumentation records as a child), the model is called again with the result, a second tool runs against a vector store, and a final model call produces the answer. Each box is a span with a duration, so the critical path is visible at a glance. In a trace list the time-axis view of this tree answers questions a log search never could: was the latency spent in the model, in a slow tool, or in sequential calls that could have run in parallel?

Agent and tool attributes

The agent-level operations are create_agent (building or registering an agent, relevant for remote agent services) and invoke_agent (running it). The attributes reported for these spans are gen_ai.agent.id, gen_ai.agent.name, gen_ai.agent.description and gen_ai.agent.version, with gen_ai.tool.definitions describing the tools offered to the model. For an execute_tool span the working set is gen_ai.tool.name, gen_ai.tool.type (values reported: function, retrieval, extension) and gen_ai.tool.call.id. That last attribute is the join key: the model’s response contains a tool call with an identifier, and the tool span records the same identifier, so you can connect “the model asked for X” to “X ran and took 900 ms” without guessing from timestamps.

The attribute gen_ai.conversation.id solves a different problem. A chat session spans many HTTP requests, and each request is its own trace. Putting the same conversation identifier on every invoke_agent span lets you query “show me everything in this session” even though the traces are separate. Be careful what you use as the value: a raw user identifier or an email turns an operational attribute into personal data. Use an opaque, rotating session token.

Frameworks versus the spec

Agent frameworks do not all emit this tree. Some emit it natively, some through an adapter, and some emit their own span names. The Dash0 explainer I used for status checks notes, for example, that the OpenLLMetry project from Traceloop shipped its own span shape before the GenAI conventions existed and has not fully converged, and that several agent tools and coding assistants now export telemetry in the gen_ai.* shape. I could not verify the per-framework state for every library, so the honest guidance is: run your framework in a staging environment, export to a local Collector with a debug exporter, and read the actual span names before you build dashboards.

When the framework’s output does not match, you have three options. Wrap the agent entry point and each tool function with manual spans (shown in the code section below). Use a Collector transform processor to rename attributes into the gen_ai.* names. Or accept the library’s shape and build views per library, which defeats the purpose of a standard. The first two are better; the transform approach is attractive because it keeps application code free of observability glue.

Retrieval and the rest of the trace

Retrieval-augmented generation adds spans that are not model calls: an embedding request, a vector store query, a reranker. The embeddings operation covers the first, and retrieval appears in the reported operation list for the second. Vector databases frequently have their own database client conventions, which means one trace can contain both gen_ai.* spans and db.* spans. That is a feature: a slow answer caused by an unindexed vector query shows up as a slow database span under a retrieval step, not as a mysterious model delay.

For agents that call other agents over a network, correct context propagation is what keeps the tree connected. The traceparent header must travel with the outgoing call, which is exactly the mechanism described in our W3C trace-context guide. A broken propagation hop shows up as two disconnected traces, each missing half the story. The same applies to Model Context Protocol (MCP) servers: the new GenAI repository’s README mentions MCP conventions, but I did not review their contents, so verify them before relying on them.

Metrics: Duration, Token Usage and Cost Signals

Traces explain one request; metrics describe the fleet. The GenAI metric conventions define two client-side histograms that I verified in two sources: gen_ai.client.operation.duration and gen_ai.client.token.usage. From my reading of the spec (not re-verified in this run, because the official page had moved), the duration unit is seconds and token usage is a histogram of token counts with a gen_ai.token.type attribute separating input from output. The metrics share dimensions with spans: operation name, provider name, request model and response model.

There is also a family of server-side metrics for model-serving systems (for example request duration, time to first token and time per output token). I recall these from the spec but did not verify them this run, so check the repository before adopting names. They matter if you operate the model server yourself, for instance a vLLM or similar deployment, because the client cannot see queueing inside the server.

Why histograms, and why the bucket choice matters

Latency of an LLM call is heavy-tailed: a short classification may take 300 milliseconds while a long generation takes 40 seconds. Default histogram buckets designed for web requests, which top out around ten seconds, would squash every long generation into the last bucket and make your p95 meaningless. The spec offers advisory bucket boundaries for both histograms (from memory, a geometric sequence for token counts that spans single tokens up to tens of millions); use them or define your own and apply them consistently through an OTel SDK View. If your histogram buckets are wrong, your percentile alerts are wrong, and no amount of tracing fixes that.

Deriving cost without storing cost

A tempting design computes dollar cost inside the application and records it as an attribute. Resist it. Prices change, discounts differ by contract, and cached-token rates differ by provider, so a cost number baked into old telemetry cannot be recomputed. Record the raw counters (input, output, cache read, cache creation, reasoning, model name) and compute cost at query time or in a recording rule against a price table you version. The token-usage histogram gives you the sum per model and operation; multiply by the current table and you have cost per route, per team or per tenant, provided you carry a low-cardinality attribute such as a route name or a team label.

That last condition is where deployments go wrong. Per-tenant cost attribution is valuable, but putting a raw tenant identifier on a metric multiplies time series by tenant count. A common compromise is to keep tenant on spans (high cardinality is fine for traces, within reason) and aggregate to a bounded set of tiers or teams on metrics. A cross-reference worth reading is how MLflow models the same idea inside an experiment tracker: see our guide to MLflow 3 GenAI tracing and the model registry, which shows a trace-centred workflow that can also ingest OTel data.

Span metrics as a fallback

If your instrumentation emits spans but not metrics, the Collector’s spanmetrics connector can derive request counts and duration histograms from spans. Figure 4 includes this branch. It is a good safety net, but note the cost: it computes on sampled data if you sample before it, so place it before tail sampling or you will undercount.

Content Capture, PII and the Opt-In Model

Prompts and completions are the most useful and the most dangerous data in a trace. They explain why an answer was wrong, and they contain whatever users typed: names, account numbers, medical details, secrets pasted by accident. The conventions treat this as opt-in, and for good reason.

Figure 3: Lifecycle of one instrumented call. Usage and finish reasons are known only after the response, so the span is completed late; content, if enabled, is captured at this point and then redacted at the Collector.

Attributes, events or references

The content fields reported by sources are gen_ai.input.messages, gen_ai.output.messages and gen_ai.system_instructions. The current design, as described by the Dash0 explainer, puts prompts and completions in an event named gen_ai.client.inference.operation.details rather than in ordinary span attributes, and its stated reasoning is that content often contains personal data and that span attributes are indexed and size-limited. The article also notes that the event conventions are still settling, and that the Python contrib instrumentation exposes modes called span_only, event_only and span_and_event once you opt into the newer shapes.

A third path exists in the Python OpenAI instrumentation documentation: a completion hook that uploads prompts and completions to an fsspec-compatible destination (a local path, a gs:// or an s3:// location) and records reference URIs in gen_ai.input.messages.ref and gen_ai.output.messages.ref. By-reference capture is the strongest design for regulated workloads, because the span carries a pointer and access to the content is governed by the object store’s own permissions, retention and encryption, not by whoever can open the trace UI.

The environment variables

Two variables control most Python behaviour. OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental opts into the newest GenAI shape; without it, the OpenAI instrumentation documentation states that the package follows an older semantic convention version (v1.30.0) and does not capture content. OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT turns content capture on: true on the legacy path, or one of the three modes above with the experimental opt-in. The specification itself, per the explainer, mentions the content variable only as an example and leaves the mechanism to instrumentation authors, so names and accepted values can differ between libraries. Read each library’s documentation rather than assuming.

A pragmatic policy

A policy that survives audit has four parts. Default production capture to metadata only: model, tokens, finish reason, latency, tool names. Allow full content in development and staging, where inputs are synthetic. For production debugging, enable content capture for a sampled slice or for an explicit allow-list, with a short retention period. And redact at the Collector as a second line of defence, because developers will eventually enable capture in the wrong environment.

Do not mistake redaction for anonymisation. Regex-based removal catches email addresses and card-like numbers but misses names, addresses in free text and context that identifies a person indirectly. If your data classification says the prompt is sensitive, the safe design is not to put it in the telemetry stream at all, and to store it by reference with access controls. Tool arguments and return values deserve the same treatment as prompts; they often contain the most specific identifiers in the whole interaction, such as an order number or a customer record.

Hands-On: Instrumenting a Tool-Calling Agent in Python

The goal of this walk-through is a minimal, vendor-neutral setup: automatic instrumentation for the model client where a library exists, plus manual spans for the agent and tools, exporting over OTLP to a local Collector. The code below was written for this post from the documented attribute names and has not been run in this environment; treat it as a starting sketch and pin versions yourself.

Step 1: Provider and exporter

The SDK needs a TracerProvider with a batch processor and an OTLP exporter. The package names are the standard OTel Python distribution: opentelemetry-sdk, opentelemetry-exporter-otlp and, for OpenAI clients, the contrib package opentelemetry-instrumentation-openai-v2 (the version string 2.3b0 appeared in one tutorial; the b marks a beta release, which is consistent with the Development status of the conventions). Pin an exact version in your lockfile and upgrade deliberately.

# UNTESTED sketch. pip install opentelemetry-sdk opentelemetry-exporter-otlp \
#   opentelemetry-instrumentation-openai-v2 openai
import os
os.environ.setdefault("OTEL_SEMCONV_STABILITY_OPT_IN", "gen_ai_latest_experimental")
# Content capture stays OFF by default. Enable only in dev or staging:
# os.environ["OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT"] = "true"

from opentelemetry import trace, metrics
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
from opentelemetry.instrumentation.openai_v2 import OpenAIInstrumentor

resource = Resource.create({
    "service.name": "support-agent",
    "deployment.environment.name": os.getenv("ENV", "dev"),
})
tp = TracerProvider(resource=resource)
tp.add_span_processor(BatchSpanProcessor(
    OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces")))
trace.set_tracer_provider(tp)

mp = MeterProvider(resource=resource, metric_readers=[
    PeriodicExportingMetricReader(
        OTLPMetricExporter(endpoint="http://localhost:4318/v1/metrics"))])
metrics.set_meter_provider(mp)

OpenAIInstrumentor().instrument()   # chat spans and GenAI metrics appear automatically

Step 2: Manual spans for the agent and each tool

Libraries instrument the model call; they usually cannot know where your agent loop begins. Wrap the loop in an invoke_agent span and each tool in an execute_tool span, using the attribute names from the conventions. Keep the span name aligned with the documented pattern, and record failures with the standard exception API so error.type is populated.

# UNTESTED sketch. Uses documented gen_ai attribute names; verify against the
# current spec before relying on them, because they are in Development status.
import json
from opentelemetry import trace
from opentelemetry.trace import SpanKind, Status, StatusCode
from openai import OpenAI

tracer = trace.get_tracer("support_agent", "0.1.0")
client = OpenAI()

TOOLS = [{"type": "function", "function": {
    "name": "search_orders",
    "description": "Find orders for the current customer",
    "parameters": {"type": "object",
                   "properties": {"query": {"type": "string"}},
                   "required": ["query"]}}}]

def search_orders(query: str) -> dict:
    return {"orders": []}          # replace with a real lookup

def run_tool(call) -> str:
    with tracer.start_as_current_span(
            f"execute_tool {call.function.name}", kind=SpanKind.INTERNAL) as span:
        span.set_attribute("gen_ai.operation.name", "execute_tool")
        span.set_attribute("gen_ai.tool.name", call.function.name)
        span.set_attribute("gen_ai.tool.type", "function")
        span.set_attribute("gen_ai.tool.call.id", call.id)
        try:
            args = json.loads(call.function.arguments)
            result = search_orders(**args)
            return json.dumps(result)
        except Exception as exc:
            span.record_exception(exc)
            span.set_attribute("error.type", type(exc).__name__)
            span.set_status(Status(StatusCode.ERROR))
            return json.dumps({"error": "tool_failed"})

def invoke_agent(user_text: str, session_token: str, max_turns: int = 6) -> str:
    with tracer.start_as_current_span("invoke_agent support_bot") as span:
        span.set_attribute("gen_ai.operation.name", "invoke_agent")
        span.set_attribute("gen_ai.provider.name", "openai")
        span.set_attribute("gen_ai.agent.name", "support_bot")
        span.set_attribute("gen_ai.conversation.id", session_token)  # opaque, not a user id
        messages = [{"role": "user", "content": user_text}]
        for turn in range(max_turns):
            resp = client.chat.completions.create(
                model="gpt-4o-mini", messages=messages, tools=TOOLS)
            msg = resp.choices[0].message
            if not msg.tool_calls:
                span.set_attribute("app.agent.turns", turn + 1)
                return msg.content or ""
            messages.append(msg)
            for call in msg.tool_calls:
                messages.append({"role": "tool", "tool_call_id": call.id,
                                 "content": run_tool(call)})
        span.set_attribute("app.agent.turns", max_turns)
        span.set_attribute("error.type", "max_turns_exceeded")
        span.set_status(Status(StatusCode.ERROR))
        return "Sorry, I could not finish that request."

Three design choices in this sketch are deliberate. The tool span is created in the same process as the model call, so context propagation is automatic through the active span. The conversation identifier is an opaque token, not a user identifier. And the loop has an explicit turn limit that is recorded as an attribute and an error type, because an agent that loops until a budget runs out is one of the most expensive failures you can have, and it should be queryable. The attribute app.agent.turns is a custom name in an application namespace; keep your own attributes out of gen_ai.* so a spec rename never collides with them.

Step 3: Run a Collector and read the output

Before sending anything to a vendor backend, send it to a Collector with a debug exporter and read raw spans. This one check catches most problems: missing gen_ai.provider.name, token attributes absent because the provider did not return usage in streaming mode, or tool spans orphaned in their own trace. The next section turns that Collector into a policy point.

The Collector Pipeline: Mapping, Redaction, Sampling and Cost

The OpenTelemetry Collector is where you enforce the rules that application developers will not remember. It receives OTLP, applies processors in order and fans out to exporters. For GenAI telemetry, four jobs belong here.

Figure 4: A Collector pipeline for GenAI telemetry. Order matters: bound memory first, normalise names, delete sensitive keys, then sample, then batch for export. Span metrics branch off before sampling.

Job 1: Protect the Collector itself

Place memory_limiter first in every pipeline so a burst of large LLM payloads cannot exhaust memory and crash the Collector. Large prompts make spans much heavier than typical web spans, which is why this deserves explicit attention. Pair it with a batch processor at the end to reduce export calls.

Job 2: Normalise attribute names

Because the conventions are in Development, instrumentation libraries on different versions emit different names. The Dash0 explainer recommends a transform processor that maps legacy names to their gen_ai.* equivalents until the repository publishes a schema URL. A typical case is the older gen_ai.system attribute, which became gen_ai.provider.name. Doing this centrally means an upgrade of one service does not break dashboards for the rest.

Job 3: Redact content

An attributes processor with the delete action removes named keys from spans. A transform processor can do conditional work such as hashing a value or deleting a key only when the environment is production. Content that arrives as span events needs OTTL statements aimed at span events rather than spans; confirm the exact context name for your Collector version in the OTTL documentation.

Job 4: Sample intelligently

Head sampling (decide at the root) is cheap but blind: it cannot know that a trace will contain an error or a three-minute tool call. Tail sampling buffers a whole trace and decides after it ends, which suits LLM workloads where the interesting traces are the failed, slow and expensive ones. The cost is memory and a routing requirement: all spans of a trace must reach the same Collector instance, usually via a load-balancing exporter keyed on trace ID.

The configuration below is a sketch. It is untested here; processor settings and OTTL syntax change between Collector releases, so validate it with otelcol validate against your distribution.

# UNTESTED sketch. Validate against your Collector version.
receivers:
  otlp:
    protocols:
      http: {}
      grpc: {}

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 20
  transform/genai_names:
    error_mode: ignore
    trace_statements:
      - context: span
        statements:
          - set(attributes["gen_ai.provider.name"], attributes["gen_ai.system"]) where attributes["gen_ai.provider.name"] == nil and attributes["gen_ai.system"] != nil
          - delete_key(attributes, "gen_ai.system")
  attributes/redact:
    actions:
      - key: gen_ai.input.messages
        action: delete
      - key: gen_ai.output.messages
        action: delete
      - key: gen_ai.system_instructions
        action: delete
  tail_sampling:
    decision_wait: 15s
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow
        type: latency
        latency: { threshold_ms: 10000 }
      - name: sample-rest
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }
  batch: {}

connectors:
  spanmetrics: {}
  forward: {}

exporters:
  otlphttp/backend:
    endpoint: https://otel-backend.example.com
  debug: {}

service:
  pipelines:
    traces/in:
      receivers: [otlp]
      processors: [memory_limiter, transform/genai_names, attributes/redact]
      exporters: [spanmetrics, forward]
    traces/sampled:
      receivers: [forward]
      processors: [tail_sampling, batch]
      exporters: [otlphttp/backend]
    metrics:
      receivers: [otlp, spanmetrics]
      processors: [memory_limiter, batch]
      exporters: [otlphttp/backend]

In that sketch the forward connector hands the full, redacted stream from the first traces pipeline to the sampling pipeline, while spanmetrics receives the same unsampled stream so request counts and duration histograms are not undercounted. Connector availability depends on your Collector distribution, which is another reason to validate before use.

Sampling arithmetic, with illustrative numbers

Suppose an assistant handles 2 million model calls a day and each trace averages 8 spans at roughly 3 KB each when content is excluded (an assumption for illustration, not a measurement). That is about 48 GB per day of raw span data. Keeping all errors and slow traces plus a 5 percent random sample might retain on the order of 8 to 12 percent of traces, depending on your failure rate, cutting storage roughly tenfold. If you then enable content capture inline, span size can grow by one to two orders of magnitude because prompts with retrieved context run to tens of thousands of tokens. That is the quantitative argument for by-reference storage and for a short retention window on content.

Cardinality: the quiet cost driver

Span attributes can be high-cardinality; metric labels cannot. The attributes that are safe as metric dimensions are bounded sets: operation name, provider, request model, response model, finish reason. Unsafe ones are session identifiers, user identifiers, response identifiers, tool call identifiers, free-text error messages and raw prompt fragments. If your metrics backend bills by active series, one careless label can multiply the bill. Review every custom attribute against the question “how many distinct values can this take in a month?” before it reaches the metrics pipeline. Be especially wary of gen_ai.response.model if a provider returns dated snapshots: the set is bounded but grows slowly over time, so use a recording rule or relabel step to collapse it when you only need the model family.

Trade-offs, Gotchas, and What Goes Wrong

Spec churn. Everything in the gen_ai.* namespace is Development-level, and the move to a separate repository with an unpublished schema URL means there is no formal migration channel yet. A rename can land in a minor release of an instrumentation package and silently change what your dashboards see. Mitigate with pinned versions, a Collector mapping layer, and an alert on “spans with operation name but no provider name”, which catches schema drift early.

Streaming responses. Token usage is known only when the provider reports it, and for streaming calls that may happen in the final chunk or only if you request it in the API options. A span can end with no usage attributes, and your cost dashboard under-reports without any error. Test streaming and non-streaming paths separately and compare billed tokens with recorded tokens for a day.

Span duration is not user-perceived latency. For streaming, users care about time to first token, not time to last. The client duration histogram measures the whole call. Record first-token timing as a span event or custom attribute if the user experience depends on it, and check whether your library already emits the relevant metric.

Retries hide cost. Many SDKs retry on rate limits or server errors. Depending on the library, retries may appear as separate spans or be hidden inside one. Hidden retries inflate duration without inflating recorded tokens, and billed retries may not be billed at all, depending on the provider. Know which your stack does.

Asynchronous and parallel tool calls. When a model requests several tools in one turn and you run them concurrently, the sibling spans overlap. Context must be passed explicitly into worker threads or tasks, or the tool spans attach to the wrong parent or start new traces. This is the most common cause of a “broken” agent tree.

Content capture creep. A debugging session enables capture, nobody turns it off, and weeks later the trace store holds every customer message. Defend with environment checks in the Collector, short retention and, ideally, by-reference storage.

Sampling that destroys the signal. Head sampling at 1 percent will drop most of the rare, expensive failures you built tracing to find. Conversely, tail sampling without trace-ID-aware routing produces partial traces and wrong decisions. Neither failure produces an error message; both just give you a worse picture.

False precision in cost. Token counts reported by providers are authoritative for billing, while counts estimated locally by a tokenizer can differ. Use provider-reported usage whenever it exists, and treat any derived cost as an estimate to reconcile against the invoice.

Practical Recommendations

Start narrow. Pick one service, turn on the model-client instrumentation, point it at a Collector with a debug exporter, and read the spans before touching dashboards. Then add the manual agent and tool spans, because that is where the diagnostic value concentrates. Only after the shape is stable should you build token and latency dashboards and alert rules.

Treat the conventions as a moving target and design for it. Keep an inventory of the exact attribute names your dashboards use, so a rename is a one-hour fix instead of an investigation. Keep price tables outside telemetry and versioned. Keep personal data out of metrics entirely and out of spans unless there is a clear, reviewed reason.

Finally, align with the rest of your observability stack rather than building a parallel one. A GenAI span is a normal span, so the same sampling, retention and access-control policies, the same trace context propagation and the same backends should serve it. The value of standardisation is that the model call shows up in the same trace as the HTTP request, the database query and the queue message that surround it.

Adoption checklist

Pin instrumentation package versions and record them in the build.
Opt in to the newest conventions deliberately and test in staging first.
Default content capture to off in production; use by-reference storage if you need content.
Wrap each agent run in invoke_agent and each tool in execute_tool, with the call identifier set.
Record request and response model, finish reasons and all token counters, including cache and reasoning tokens where available.
Put memory_limiter first and tail sampling after redaction in the Collector.
Keep high-cardinality identifiers off metrics; compute cost from counters, not stored dollars.
Add an alert for spans missing required attributes to detect schema drift.

Frequently Asked Questions

Are the OpenTelemetry GenAI semantic conventions stable?

No. Sources checked for this post describe the gen_ai.* spans, attributes and metrics as Development status, with only inherited attributes such as error.type, server.address and server.port being stable. In practice that means names can change, so pin versions, use the opt-in environment variable deliberately, and centralise attribute mapping in the Collector. Check the semantic-conventions-genai repository for the current status before a major rollout, because it was still pre-release when this was written.

What is the difference between gen_ai.system and gen_ai.provider.name?

gen_ai.provider.name is the current required attribute identifying the provider, with values such as openai, anthropic and aws.bedrock. The older gen_ai.system attribute is its predecessor. Older instrumentation versions may still emit the old name, which is why a Collector transform processor that copies the old value into the new attribute is a common migration aid. Dashboards should query the new name once your fleet is upgraded.

Should I record prompts and completions in production traces?

Usually not by default. Prompts contain whatever users type, including personal and confidential data, and span attributes are indexed and size-limited. Capture metadata in production (model, tokens, finish reason, latency), use full content in staging, and if production content is needed, enable it for a sampled slice with short retention or store it by reference in an access-controlled object store. Redact at the Collector as a second safeguard, knowing pattern-based redaction is imperfect.

How do I calculate LLM cost from OpenTelemetry data?

Record raw counters rather than dollars: input tokens, output tokens, cache read and creation tokens where available, reasoning tokens, plus the response model. Join them to a versioned price table at query time. Cached and reasoning tokens are billed differently from plain tokens, so ignoring them can misstate cost substantially. Use provider-reported usage, and reconcile your estimate against the monthly invoice periodically.

How do I trace an agent that calls tools?

Wrap the agent run in an invoke_agent span and each tool call in an execute_tool span with gen_ai.tool.name and the call identifier from the model’s response. Model calls become chat children automatically if the client library is instrumented. Pass context explicitly into any thread or async task so the tool spans attach to the right parent, and keep outbound traceparent headers intact for remote agents.

Do I need the Collector for LLM tracing?

Not strictly, since SDKs can export directly to a backend, but the Collector is where redaction, attribute mapping, tail sampling and span-metric generation are enforced independently of application code. For LLM workloads with sensitive content and bursty payloads, running one is the safer design. Start with a single instance and a debug exporter, then add load-balanced tail sampling when volume demands it.