Tail-Based Sampling with the OpenTelemetry Collector: Distributed Tracing Cost Control (2026)

Most tracing bills are paid for traces nobody will ever open. A healthy checkout request that returns in 80 milliseconds produces the same thirty spans as a failing one, and in a typical production system more than 95 percent of traces look like the healthy one. Keeping all of them is expensive. Keeping a random few percent is cheap, but it throws away the exact traces you will want at three in the morning: the errors, the slow outliers and the requests from the one customer who is complaining.

Tail-based sampling resolves that tension by deferring the keep-or-drop decision until the trace is complete, so the decision can look at what actually happened. The price is a stateful, memory-hungry tier in your telemetry pipeline that must be sized, routed and monitored like any other distributed system. Done well, it cuts stored trace volume by an order of magnitude while keeping every error and slow trace. Done badly, it silently produces broken traces and biased metrics.

What this covers: how head, tail and adaptive sampling differ, the trace-ID routing that makes tail sampling possible, the tail_sampling processor and its policies, a two-tier OpenTelemetry Collector configuration, memory and decision_wait sizing math, a worked cost model, the span metrics connector that keeps RED metrics honest, failure modes, and where Refinery and Jaeger fit.

Context and Background

Distributed tracing records the path of one request across services as a tree of spans. The trouble is volume. Span count scales with request rate multiplied by call-graph depth, and every hop adds spans for the server handler, the client call, database queries and sometimes in-process work. A system serving a few thousand requests per second can emit tens of thousands of spans per second, and backends charge by ingested gigabytes, indexed spans or retained days. That is the core of distributed tracing cost.

The OpenTelemetry project frames sampling as the main cost lever. Its sampling concepts page describes sampling as one of the most effective ways to reduce the cost of observability without losing visibility. It distinguishes two families. Head sampling makes the decision as early as possible, before the trace is complete, and cannot depend on data elsewhere in the trace. Tail sampling decides after considering all or most of the spans, so it can keep traces that contain an error or exceed a latency threshold.

That same page is candid about the cost of the second approach. Tail sampling is hard to implement, because the rules must evolve with the system. It is hard to operate, because it needs stateful components that hold large amounts of data and may need many nodes. And it is often vendor-specific, because paid backends may offer only what the vendor builds. The page also notes that the two are commonly combined: head sampling first to cut volume, tail sampling before export.

In the open-source ecosystem the reference implementation is the tail_sampling processor in the OpenTelemetry Collector contrib distribution. Honeycomb’s Refinery is the best-known standalone alternative, and Jaeger ships its own remote and adaptive sampling strategies. We cover all three, but the Collector is the center of gravity because it is vendor-neutral and because OTLP is now the common wire format across backends. If you are still deciding how telemetry should leave your services, our comparison of OTLP versus Prometheus remote write explains why the Collector sits at the center of most modern pipelines.

Two framing points before the mechanics. First, sampling is not the only cost lever: dropping unused attributes, shortening span names, removing noisy instrumentation and shortening retention all help, and they compose with sampling. Second, sampling changes what your data means. Any statistic computed from sampled traces needs a weight, and any alert built on trace counts breaks unless you generate metrics before sampling. Section by section, this post keeps returning to that second point, because it is the one teams discover late.

Head-Based vs Tail-Based Sampling: The Decision Problem

Every sampler answers one question: given a trace, should we keep it? The difference between approaches is how much information the sampler has when it answers, and where the answer is computed.

Head-based sampling decides before the story is known

In head-based sampling the decision is made at the root of the trace, typically in the SDK. The standard SDK sampler is ParentBased wrapping TraceIdRatioBased. The root span hashes its trace ID against the configured ratio. If it wins, the trace is marked sampled; if not, it is not. Child spans, including those in other services, honour the parent’s decision, which is propagated in the W3C Trace Context traceparent header as the sampled bit of the trace-flags field.

This is cheap. An unsampled span can be a non-recording span, so the unsampled majority costs almost nothing in CPU, memory or network. It is also consistent: because the whole trace shares one decision, you never see half a trace. The limitation is structural. At the moment the root span starts, nobody knows whether the request will fail, whether the database will stall, or whether this user is a high-value account. A 5 percent head sample keeps 5 percent of errors, exactly as it keeps 5 percent of everything.

You can raise the odds for known-important requests, for example by sampling 100 percent of a specific route or of requests carrying a debug header. That works when importance is visible at the start. It does not work for the failures that matter most, which are by definition the ones nobody anticipated.

Tail-based sampling decides with the whole story

In tail-based sampling every span is exported, usually to a collector, which groups spans by trace ID and holds them for a short window. When the trace looks complete, or a timer fires, a set of policies inspects it. Does any span have an error status? Is the end-to-end duration above two seconds? Did it touch the new payment service? Only traces that satisfy a keep policy are forwarded.

That makes tail sampling the only practical way to guarantee “keep every error trace”. It also enables latency-aware retention, per-service rates and anomaly-focused debugging. The cost moves from the backend bill to the collection tier: all spans must reach the sampler, so network egress and collector CPU scale with total volume, not kept volume, and the sampler must buffer spans in memory.

Adaptive and dynamic sampling sit in between

A third family adjusts rates automatically. Dynamic samplers watch the frequency of values in a chosen key, such as route plus status code, and lower the rate for common combinations while keeping rare ones at higher rates. Honeycomb’s Refinery documents dynamic sampling in exactly these terms, alongside rules-based sampling, throughput-based sampling that holds spans per second under a limit, and deterministic probability sampling keyed on trace ID. Jaeger offers an adaptive mode as well, though you should check your Jaeger version’s documentation before relying on its exact behaviour.

The strength of adaptive sampling is that it removes manual tuning: a flood of identical health-check traces gets automatically squeezed. The weakness is explainability. When rates float, you must record the rate applied to each kept trace so downstream math can re-weight it, and operators must trust a control loop they did not write.

The comparison below summarizes the trade space. Treat the cost and complexity cells as qualitative.

Dimension	Head-based	Tail-based	Adaptive or dynamic
Decision point	Trace start, in SDK or first collector	After trace completes, in collector tier	Either, driven by observed key frequency
Can guarantee all errors kept	No	Yes, within buffer limits	Partly, if error is part of the key
Collector memory	None	High, buffers traces for decision_wait	Moderate to high
Network cost to the sampler	Low, only kept traces leave SDK	High, all spans leave SDK	Depends on placement
Routing requirement	None	All spans of a trace to one node	Often same
Operational complexity	Low	High	High
Statistical weighting	Simple, one ratio	Per policy, needs care	Per key, must be recorded

Core Architecture: The Two-Tier Tail Sampling Pipeline

The direct answer: tail-based sampling needs every span of a trace to arrive at the same collector instance. In practice you run two collector layers. The first layer receives spans from services and uses the load-balancing exporter to route by trace ID. The second layer runs the tail_sampling processor, buffers each trace, applies policies after a wait window, and exports only the kept traces to the backend.

Figure 1: Two-tier tail-based sampling. Tier 1 routes by trace ID and derives metrics from 100 percent of spans; tier 2 buffers whole traces and applies sampling policies.

The diagram shows two outputs from the first tier. The traces path goes through the load-balancing exporter to the second tier, where sampling happens. A second path feeds the span metrics connector, which turns every span into RED metrics before any dropping occurs. That side channel is what keeps your dashboards accurate, and we return to it in its own section.

Why a single collector is not enough

A lone collector with the tail_sampling processor works, up to a point. It receives all spans, so it trivially sees whole traces. The moment you scale to two replicas behind an ordinary load balancer, spans of one trace land on different instances. Each instance sees a fragment, applies the policies to that fragment, and reaches a different verdict. The error span may reach instance A, which keeps its fragment, while the root span reaches instance B, which drops it. The backend then holds a trace with a missing root.

The processor README states the requirement plainly: all spans for a given trace must be sent to and received by the same collector instance for effective sampling decisions. It recommends two layers of collectors, with the load-balancing exporter in the first and the tail sampling processor in the second. A single layer with two pipelines per instance is technically possible, but the README recommends separating the layers for better failure isolation. We agree. A tier that routes should not share a failure domain with a tier that holds gigabytes of state.

The load-balancing exporter: consistent routing by trace ID

The contrib load_balancing exporter distributes telemetry across a set of backend collectors. Its routing_key setting decides what is hashed. For traces the default is traceID, which sends every span of a given trace to the same destination. Other keys exist, such as service, which hashes service.name, but for tail sampling you want trace ID.

Backends are discovered through a resolver. The README documents a static list of hostnames, a DNS resolver pointed at something like a Kubernetes headless service, a k8s resolver that watches EndpointSlices and needs RBAC permission to get, list and watch them, and an aws_cloud_map resolver. The DNS resolver defaults to a five-second refresh interval; the Kubernetes resolver reacts faster because it watches the API rather than polling DNS. Only one resolver can be configured.

A subtle but important property: when the backend list changes, the exporter remaps only a fraction of keys, roughly R over N according to the README, where R is the number of routes and N the number of backends. More backends mean less disruption per change. But “less” is not “none”, and the traces in flight during a remap are exactly the ones that get split. We examine that failure mode in the trade-offs section.

The README also lists the exporter’s maturity: traces and logs are beta, metrics alpha. Its own queue, retry and timeout settings under load_balancing are off by default, which means that by default a failed delivery to one endpoint is not rerouted to a healthy one. During topology changes you will want to enable them; the otlp sub-settings, which are on by default, handle transient backend errors.

Tier 1: the stateless router

Tier 1 holds almost no state, so it can be small, autoscaled and restarted freely. It terminates OTLP from services, adds resource attributes if needed, optionally runs the span metrics connector, and exports through load_balancing. Here is a minimal configuration.

# tier1-router.yaml  (stateless, scale on CPU)
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  batch:
    send_batch_size: 8192
    timeout: 200ms

connectors:
  span_metrics:
    histogram:
      explicit:
        buckets: [5ms, 25ms, 100ms, 250ms, 1s, 2.5s, 10s]
    dimensions:
      - name: http.request.method
      - name: http.response.status_code
    metrics_flush_interval: 30s

exporters:
  load_balancing:
    routing_key: traceID
    protocol:
      otlp:
        tls:
          insecure: true
        timeout: 5s
    resolver:
      k8s:
        service: otel-sampler.observability
    # queue and retry help during tier-2 topology changes
    sending_queue:
      enabled: true
      queue_size: 20000
    retry_on_failure:
      enabled: true
      max_elapsed_time: 30s
  otlp/metrics:
    endpoint: metrics-backend:4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [load_balancing, span_metrics]
    metrics:
      receivers: [span_metrics]
      exporters: [otlp/metrics]

Treat the exact exporter field names as version-dependent. The structure is what matters: a traces pipeline that fans out to both the load balancer and the connector, and a metrics pipeline that receives from the connector. Always validate your config with the collector’s validate command against the version you deploy.

Tier 2: the stateful sampler

Tier 2 is where memory goes. Each replica receives a stable slice of trace IDs, buffers spans per trace, and runs the policy set. Because each replica is a self-contained shard, you scale tier 2 horizontally and the ring spreads the load roughly evenly. Spread is statistical rather than load-aware; the README says routing ignores backend load, and gives the expectation that distribution should be similar across backends. A few enormous traces can still make one replica hotter than another, which is why maximum_trace_size_bytes exists.

In the next section we open up the processor itself.

Inside the tail_sampling Processor: Buffers, Timers and Trace Sampling Policies

The tail_sampling processor is conceptually simple. It maintains a map from trace ID to the spans seen so far. When the first span of a trace arrives, a timer starts. When the timer expires, the processor evaluates the policies against whatever it has and either forwards the spans or discards them. Figure 2 shows the life of one trace through both tiers.

Figure 2: Lifecycle of a trace. All spans, including late ones inside the window, reach one tier-2 node; the verdict is computed once, when the timer fires.

The settings that control memory and correctness

The README documents a compact set of top-level options. decision_wait is how long the processor waits after the first span before deciding; the documented default is 30 seconds. num_traces is how many traces are held in memory, default 50,000, and when the limit is reached the oldest trace is evicted. expected_new_traces_per_sec helps the processor pre-allocate data structures. maximum_trace_size_bytes drops traces larger than the limit immediately, which protects a replica from a runaway trace.

Two newer options matter in practice. decision_wait_after_root_received lets the decision happen earlier once the root span has arrived, which shortens residency for well-behaved traces; it is disabled by default. And decision_cache keeps the verdicts, by trace ID, in two LRU caches, one for sampled and one for non-sampled, so that spans arriving after the trace data has been released can still be handled consistently. Both cache sizes default to zero, meaning inactive, and the README advises setting them larger than num_traces. There is also sample_on_first_match, which stops evaluating once a policy matches, and a sampling_strategy option choosing between trace-complete and span-ingest behaviours. Check the README for the version you run, because this processor has evolved quickly.

Here is the key mental model: decision_wait is both your correctness window and your memory multiplier. A longer wait catches slower traces and late spans, and costs proportionally more RAM.

The policy catalogue

Policies are the heart of trace sampling policies. The README lists these types, each with its own config block.

Policy type	What it matches	Typical use
`always_sample`	Everything	Test, or last rule in a composite
`status_code`	Any span with status OK, ERROR or UNSET	Keep all error traces
`latency`	Trace duration above `threshold_ms`, optional upper bound	Keep slow traces
`string_attribute`	Attribute key matches listed values, optional regex	Keep a customer, route or service
`numeric_attribute`	Attribute within min and max	Keep large payloads, high retry counts
`boolean_attribute`	Attribute equals true or false	Honour a force-sample flag
`probabilistic`	Hash of trace ID below `sampling_percentage`	Baseline sample of healthy traffic
`rate_limiting`	Spans per second with burst capacity	Cap volume
`bytes_limiting`	Bytes per second with burst capacity	Cap storage growth
`span_count`	Trace has between `min_spans` and `max_spans`	Keep unusually deep traces
`trace_state`	W3C tracestate key matches values	Honour upstream sampling hints
`ottl_condition`	OpenTelemetry Transformation Language expressions	Complex custom matches
`and`, `not`, `drop`	Combine or invert sub-policies	Compound rules, exclude noise
`composite`	Ordered sub-policies with rate allocation	Budgeted multi-rule sampling

Policies are evaluated independently, and a trace is kept if any policy says sample, unless a drop policy matches. The README also notes the older invert_match option on string_attribute is deprecated in favour of drop or not. One consequence of “any policy keeps” semantics: adding a policy can only increase the kept volume, with the sole exception of drop. That is why the baseline probabilistic policy and the error policy do not need to be exclusive.

A realistic policy set

Figure 3 illustrates the decision order we recommend as a starting point. Drop obvious noise first, keep what must be kept, then take a thin random baseline of the remainder.

Figure 3: Policy evaluation order. Noise is dropped first, error and latency traces are always kept, and a thin probabilistic baseline preserves a representative sample of healthy traffic.

The corresponding Collector YAML for tier 2 looks like this.

# tier2-sampler.yaml  (stateful, scale on memory)
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 85
    spike_limit_percentage: 15
  tail_sampling:
    decision_wait: 20s
    decision_wait_after_root_received: 5s
    num_traces: 60000
    expected_new_traces_per_sec: 400
    maximum_trace_size_bytes: 4194304
    decision_cache:
      sampled_cache_size: 200000
      non_sampled_cache_size: 400000
    policies:
      - name: drop-health-checks
        type: drop
        drop:
          drop_sub_policy:
            - name: health
              type: string_attribute
              string_attribute:
                key: url.path
                values: ["/healthz", "/readyz", "/metrics"]
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: keep-slow
        type: latency
        latency:
          threshold_ms: 2000
      - name: keep-vip-and-canary
        type: string_attribute
        string_attribute:
          key: deployment.channel
          values: [canary, vip]
      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 5
  batch:
    send_batch_size: 8192
    timeout: 1s

exporters:
  otlp/backend:
    endpoint: tracing-backend:4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp/backend]

Processor order matters. memory_limiter goes first so back-pressure applies before buffering; tail_sampling should come before batch so batching operates on the already-reduced stream. The numbers here are placeholders sized for the worked example later; do not copy them blindly.

Composite policies for budgets

The composite policy is for when you want a hard spans-per-second ceiling shared among several rules. You give it max_total_spans_per_second, an ordered list of sub-policies, and a rate_allocation that assigns each a percentage of the budget. During a storm of errors the errors policy cannot consume the entire budget, so slow-trace and baseline rules still get a share. It is a blunt but useful tool for protecting a backend from your own policy set during an incident, which is precisely when error traces spike.

The caveat is that a budget turns your “keep all errors” guarantee into “keep errors up to their allocation”. That is a legitimate engineering trade, but it should be a conscious one, written into the runbook.

Probabilistic policy versus the probabilistic sampler processor

Do not confuse the tail_sampling probabilistic policy with the separate probabilistic_sampler processor. The policy hashes the trace ID with an optional hash_salt using FNV-1a and keeps a percentage. It is a decision input inside tail_sampling.

The standalone probabilistic_sampler processor is a head-style tool you can run anywhere in a pipeline. Its README describes three modes, all consistent: hash_seed, the default, which hashes the trace ID using 14 bits of information and requires collectors in the same tier to share the seed; proportional, which uses 56 bits of randomness from the trace ID or an explicit randomness value and applies the percentage regardless of earlier sampling; and equalizing, which accounts for earlier sampling and lowers items to a uniform minimum probability. The processor is beta for traces, and the fail_closed option defaults to true so malformed IDs are rejected rather than passed through. A common cost pattern is to run it in tier 1 to cut truly high-volume, low-value services before the expensive buffering tier.

Consistent Probability Sampling: tracestate, Thresholds and Adjusted Count

Anyone doing sampling at several points in a pipeline eventually meets the problem of arithmetic. If tier 1 keeps 50 percent and tier 2 keeps 10 percent of what it sees, is the effective rate 5 percent? Only if the two decisions are independent, and independent decisions break traces. The cure is consistency.

One random value, many thresholds

OpenTelemetry’s probability sampling specification defines a scheme where all participants use the same 56-bit randomness value, called R. R is either carried explicitly in the rv sub-key of the ot tracestate entry, or taken from the least significant 56 bits of the trace ID when the SDK knows the trace ID is random, as in W3C Trace Context Level 2. Each sampler then has a rejection threshold T, computed as the probability of rejection multiplied by two to the 56th power. A span is kept if R is greater than or equal to T.

Because everyone compares the same R, decisions nest: a trace kept at 10 percent is automatically kept by any sampler configured for 20 percent or more. There are no contradictory decisions and no broken traces. The threshold is recorded in the th sub-key, with trailing zeros removed. The spec’s own example is a 1 percent sample encoded as ot=th:fd70a4. Downstream samplers may raise the threshold, lowering probability, but must not lower it, since that would introduce statistical error. As of this writing the specification page is marked Development status, so treat the details as evolving.

Why adjusted count matters to your dashboards

The same record gives you the adjusted count: the reciprocal of the sampling probability, roughly how many original traces each kept trace represents. A trace kept at 5 percent represents 20. If every kept span carries its threshold, you can estimate true request counts by summing adjusted counts. The spec gives the formula as 2^56 divided by (2^56 minus the threshold), and says it is undefined for spans without a threshold, which is what you get from non-probabilistic decisions.

That last sentence is the important one for tail sampling. A policy that keeps a trace because it contained an error is not a probabilistic decision, so the kept trace has no meaningful adjusted count. The kept set is a stratified, biased sample: all errors, all slow traces, and a 5 percent slice of the rest. You can still reason about it, but only by stratum, and only if you know the strata. The README mentions a usetracestate feature gate for the processor, and notes that sample_on_first_match should not be combined with it. Verify the current status of that gate in the release you deploy before building on it.

The practical rule: use sampled traces for looking at individual requests and for comparing populations you can define, and use metrics generated before sampling for counts, rates and percentiles. The next sections turn that rule into architecture.

Sizing the Sampler: decision_wait, Memory and Replica Math

Capacity planning for tier 2 is a Little’s Law exercise. The number of traces resident in memory at any moment equals the arrival rate of new traces multiplied by how long each stays. Each stays for roughly decision_wait, or less if decision_wait_after_root_received fires earlier. Everything else follows.

A worked example (illustrative numbers)

Assume a platform that handles 2,000 new traces per second at peak. Each trace averages 30 spans. Assume an average of 0.8 KB per span on the wire, which is plausible for spans with a dozen attributes but must be measured in your environment. Those are illustrative inputs, not benchmarks.

Span rate: 2,000 x 30 = 60,000 spans per second.
Wire throughput: 60,000 x 0.8 KB = 48 MB per second, about 4.1 TB per day.
Residency with decision_wait of 20 seconds: 2,000 x 20 = 40,000 traces in memory fleet-wide.
Add 50 percent headroom for burst and for slow traces that sit until the timer: 60,000 traces.

In-memory size is larger than wire size. Spans decoded into the collector’s internal pdata structures carry pointers, string headers and maps. A multiplier of two to three times wire size is a reasonable planning assumption, but measure it by watching RSS against num_traces under a replayed workload. Using 2.5x: a trace is about 24 KB on the wire and 60 KB in memory. 60,000 traces x 60 KB is 3.6 GB fleet-wide.

Spread across six replicas, each holds about 10,000 traces and 600 MB for trace data, plus Go runtime overhead, caches and batch buffers. A 2 GB memory limit per replica leaves comfortable headroom. In YAML terms, per-replica num_traces should be at least the per-replica arrival rate times decision_wait, plus margin: (2,000 / 6) x 20 = about 6,700, so 10,000 is a reasonable setting. The config shown earlier used 60,000 for illustration of a smaller fleet; always derive the number for your own replica count.

The sensitivity is linear in decision_wait. Doubling it to 40 seconds doubles resident traces and memory. That is why you should not leave the 30-second default untouched without reason, and why measuring your real trace duration distribution matters more than any rule of thumb.

Choosing decision_wait from trace duration data

decision_wait should be longer than the duration of nearly all of your traces, since spans arriving after the decision are late. Pull the distribution of root span durations from your span metrics. If p99.9 of trace duration is 6 seconds, a 15 to 20 second window covers it with room for exporter batching delays and clock skew between services. If you have long-running traces, such as batch jobs or workflows that last minutes, tail sampling is a poor fit for them. Handle those with a separate pipeline: head-sample them, or give them specific rules, because no sensible decision_wait holds a ten-minute trace in memory.

Asynchronous systems deserve special attention. A message published to a queue and consumed an hour later is, depending on instrumentation, either linked as a separate trace or continued in the same one. If continued, the first half of the trace will be decided long before the second half arrives. Using span links rather than parent-child relationships across the queue boundary keeps each side a bounded trace, and the sampling decision for each is self-contained.

Deciding how many replicas

Throughput also bounds replica count. Tier 2 must decode, buffer and evaluate every span. As a planning anchor, measure spans per second per vCPU on your own policies, because OTTL conditions and regex string matching cost far more than status_code checks. Run a replay at 1.5x your peak and watch CPU, memory and the processor’s own metrics. If the sampler cannot keep up, the symptom is not an error message: it is sampling_trace_dropped_too_early, meaning traces evicted from the buffer before a decision, because num_traces was reached.

Self-observability: the metrics that tell you it is healthy

The processor exports internal telemetry. The README lists, among others, a counter for traces dropped before a decision, a histogram of how long traces stay in the buffer (removal age), a histogram of late-arriving span age, a timer for decision latency, and per-policy counts of sampled and not-sampled traces labelled by policy name. The exact Prometheus-style names depend on your exporter and version, so inspect your collector’s own metrics endpoint. Alert on three conditions at minimum:

Any sustained non-zero rate of traces dropped too early, which means the buffer is too small or the arrival rate has outgrown the fleet.
Late-span age concentrated above your decision_wait, which means the window is too short for real trace durations.
A policy whose sampled share changes abruptly after a deploy, which usually means attribute names or semantic conventions changed and the policy silently stopped matching.

For general guidance on keeping long-term metrics from this telemetry affordable, see our comparison of Prometheus long-term storage with Thanos, Mimir and VictoriaMetrics.

The Cost Model: What Tail Sampling Actually Saves

The reason to run all this machinery is the bill. Here is a worked model using the same illustrative traffic as above, so the numbers connect. All prices are placeholders. Replace the per-gigabyte rate with your vendor’s, and the error and slow fractions with your measured ones.

Baseline: keep everything

Volume: 48 MB per second is roughly 4,147 GB per day.
At an assumed ingest price of 0.10 USD per GB, that is about 415 USD per day, or roughly 12,400 USD per month, before storage and query charges.

Option A: head-based 5 percent

Keeping 5 percent of traces cuts the figure to about 207 GB per day and 20.7 USD per day. It is simple and cheap. But with an assumed error rate of 1.5 percent of traces, you now retain 5 percent of 1.5 percent: of 2,000 traces per second, 30 per second are errors and only 1.5 survive. Over a day that is about 130,000 error traces retained out of roughly 2.6 million. For a short, narrow incident you may retain a handful, which may be none of the ones you need.

Option B: tail-based with the policy set above

Assume 1.5 percent of traces are errors and 1 percent are slow but not erroring, and that the two sets are roughly disjoint (an upper bound on overlap-adjusted volume). All of those are kept. The remaining 97.5 percent are sampled at 5 percent.

Kept fraction: 0.015 + 0.010 + 0.975 x 0.05 = 0.07375, about 7.4 percent.
Stored volume: 4,147 GB x 0.07375 is roughly 306 GB per day.
At the assumed rate: about 30.6 USD per day, versus 415 for everything and 20.7 for head 5 percent.

So tail sampling costs about ten dollars a day more than head 5 percent in this model, and in exchange retains 100 percent of errors instead of 5 percent. That is the real trade: you pay roughly half again as much for a categorically different guarantee.

The hidden costs on the other side of the ledger

Backend savings are not the whole picture. Tail sampling moves cost into your own infrastructure.

Compute and memory. In the sizing example, six replicas with 2 GB limits and a few vCPUs each. Whether that is cheap depends on your cloud, but it is real money and real on-call burden.
Network. All spans cross the network at least twice: service to tier 1, and tier 1 to tier 2. Cross-zone transfer charges can rival the saved ingest fees. Keep tier 1 as an agent or zonal gateway, and consider compression on the OTLP link.
SDK overhead. Services must now record and export every span, so the SDK sampler becomes AlwaysOn. CPU and allocation cost per request rises compared with a 5 percent head sample, where unsampled spans are cheap non-recording objects.
Engineering time. Writing, testing and re-validating policies is an ongoing job.

A fair accounting subtracts those from the savings. For teams below roughly a few hundred spans per second the net is often negative; a straightforward head sample is the right answer. As volume or the cost of missed incidents grows, tail sampling’s economics flip.

A hybrid that often wins

A common compromise is to head-sample aggressively only the services that are both extremely high volume and low value, such as a health-checked sidecar mesh or a cache layer, using the probabilistic_sampler in tier 1. Everything else flows through unmodified to tail sampling. This cuts the network and memory cost of the sampler by the share of spans you removed, while preserving the all-errors guarantee for the services that matter. Document the exemption explicitly: an unsampled trace has no spans in the backend, and the first question in any postmortem will be why nothing is there.

Keeping RED Metrics Honest with the Span Metrics Connector

The dangerous failure of sampling is not a missing trace, which is visible. It is a wrong number. If your request-rate chart is computed from stored spans, then a policy change from 5 percent to 4 percent makes traffic appear to fall by 20 percent. If your latency histogram is computed from kept traces, it is dominated by the slow and failed ones you deliberately over-kept, and your p50 looks terrible when users are fine.

The fix is to generate metrics from 100 percent of spans before any dropping happens, and treat sampled traces purely as exemplars for investigating those metrics.

How the connector works

The contrib spanmetrics connector acts as an exporter in a traces pipeline and a receiver in a metrics pipeline. It aggregates spans into two core families, per the README: a calls counter (traces.span.metrics.calls by default naming, counting spans per dimension set, with error counts derived from the status code) and a duration histogram (traces.span.metrics.duration). Histograms can be explicit-bucket, with default buckets from 2 ms to 15 s, or exponential, with a default maximum of 160 buckets per range. The dimensions setting adds attributes such as HTTP method, route or status code to the series. metrics_flush_interval defaults to 60 seconds, and aggregation_temporality defaults to cumulative, with delta as the alternative.

The connector can also attach exemplars: exemplars.enabled, default false, attaches spans to data points, up to five per point by default. That makes a latency spike on a dashboard clickable into a representative trace. Be aware that the exemplar may point at a trace which tail sampling later drops. If your exemplars matter, keep the connector’s candidates and the sampler’s policies aligned, for example by always keeping slow traces, which are the ones exemplars on a latency histogram tend to point at.

Where to put it

Place the connector where it sees all spans. In the architecture of Figure 1 that is tier 1, as a second exporter alongside the load balancer, so metrics are generated before routing. Alternatively place it in tier 2 as a separate pipeline fed by the same receiver, upstream of the sampler. What must not happen is to put it after tail_sampling in the same pipeline, because then it counts only the survivors.

Multiple replicas raise a series-identity question. Each tier-1 replica produces its own cumulative counters for the same dimension set, and a backend that sees identical label sets from different replicas will treat the writes as one series and interleave them. Make series unique per replica, for example with a resource attribute for the collector instance, or aggregate downstream. Verify this against your pipeline; the README for the connector is the reference for supported options in your version.

Cardinality is your next bill

Every dimension multiplies series count. Route, method, status code and service are fine. User ID, request ID and full URL are not. Prefer templated routes (/orders/{id}) over raw paths, and cap dimension combinations. The metrics you save by not storing trace analytics can be quickly spent on a cardinality explosion in the metrics store, so review any new dimension the same way you review a new index.

For AI workloads the same pattern applies. Agent and LLM traces tend to be long, with big attribute payloads and slow spans, which interact badly with decision_wait and maximum_trace_size_bytes. Our guide to OpenTelemetry GenAI semantic conventions for LLM and agent observability covers what those spans contain and why they need their own sampling rules.

Trade-offs, Gotchas, and What Goes Wrong

Tail sampling fails in quiet ways. The pipeline keeps running, dashboards stay green, and the only symptom is a trace that is missing the span you wanted. Know the failure modes before you meet them in production.

Figure 4: A rebalancing event splits a trace between two samplers. Each makes a partial decision, so the backend ends up with fragments or nothing.

Late spans

A span that arrives after the decision has nowhere to go. If the trace was kept, the span arrives and the processor must decide what to do with it; if the trace was dropped, the span is orphaned. The decision_cache addresses this by remembering verdicts after the data is released, so a late span for a kept trace is forwarded and a late span for a dropped trace is dropped. It is off by default, and the README advises sizing the caches above num_traces. Even with a cache, a kept trace’s late spans are exported after the rest, so the backend must tolerate spans arriving out of order. Most do.

Clock skew compounds lateness. Spans are timestamped by the host that created them, and a skewed host can make a trace look longer or shorter than it is, nudging it across a latency threshold. Run NTP or equivalent and treat latency policies near the threshold as soft.

Collector restarts and the state you lose

Tier 2 holds seconds of unsampled data in memory. A crash or rolling restart loses every in-flight trace on that replica. The README documents a drop_pending_traces_on_shutdown option, which determines whether a graceful shutdown flushes decisions or drops pending traces. In either case a hard kill loses the buffer. Expect a small loss with each deployment and plan rollouts accordingly: one replica at a time, with a pod disruption budget and a readiness gate that waits for the buffer to turn over.

Do not use a persistent queue to paper over this. The load-balancing exporter’s sub-exporters do not support persistent queues according to its README, and persisting unsampled spans for a sampler is the wrong shape of durability anyway.

Rebalancing splits traces

Figure 4 shows the characteristic hazard. When tier 2 scales out or in, or a replica is replaced, the ring changes. By the exporter README’s reckoning roughly R over N of the routes move. Traces already being buffered on the old owner continue to receive their remaining spans at the new owner. Each node sees a fragment. The old owner may keep a fragment because the error span arrived there; the new owner may drop the rest because its fragment looks healthy.

Mitigations are layered. Use a resolver that reacts quickly and consistently, such as the Kubernetes resolver, rather than slow DNS refresh. Enable queue and retry on the load_balancing exporter so a brief unavailability does not drop data. Scale tier 2 by large, infrequent steps instead of fine-grained autoscaling on memory, and never autoscale on a short window. Use the decision cache so that a replica which has already decided a trace can handle stragglers coherently. Finally, accept that every topology change loses a little fidelity, and avoid causing them during business-critical windows.

The exporter README also mentions the groupbytrace processor as a way to make the routing decision once for a whole trace when backends change often. It buffers spans for a trace and releases them together, which reduces split-trace risk but adds another stateful stage. Consider it if your tier 2 is genuinely volatile; if it is, fix the volatility first.

Skewed hash and giant traces

Hashing by trace ID spreads traces evenly by count, not by size. One batch job that emits a 100,000-span trace lands entirely on a single replica, which can blow its memory limit and trigger the memory_limiter, which then refuses data for every other trace on that node. Set maximum_trace_size_bytes so oversize traces are dropped on arrival rather than allowed to hurt neighbours, and send known-giant workloads to a separate pipeline with a different policy. Also watch the per-replica spread of resident traces; a persistently hot replica is a sign of a pathological tenant.

Policies that silently stop matching

A string_attribute policy keyed on http.status_code stops matching the day an instrumentation upgrade renames it to http.response.status_code. Nothing errors; the policy simply never fires and your kept volume drops. Attribute names also differ between resource and span scope. Add a canary: a synthetic request that always errors and should always be kept, and alert if that trace is not found in the backend within a minute.

Orphan traces and partial trace views

Even correct sampling can produce traces that look broken: a kept trace missing spans from services that were exempt from tracing, a head-sampled segment that never reached the sampler, or a span dropped by a size limit. Users interpret “missing root span” as a bug in your tracing. A short explanation in the trace UI or runbook saves hours of confusion.

SDK-side requirements

Tail sampling cannot keep what the SDK never sent. If services still run ParentBased(TraceIdRatioBased(0.05)), the tail sampler sees only that 5 percent, and its guarantee collapses to “all errors among the traces head sampling chose”. Configure services to export all spans (AlwaysOn) when the tail tier exists, or use the head sampler only for the exempt, high-volume services described earlier. The trace-flags sampled bit also matters: backends and collector components may treat unsampled-flagged spans differently, so test the exact combination you run.

Vendors and Alternatives: Refinery, Jaeger and the Backends

The Collector is not the only way to run a tail sampler, and it is not always the best.

Honeycomb Refinery

Refinery is Honeycomb’s tail-sampling proxy. Per Honeycomb’s documentation, it works at the level of an entire trace and can keep or discard the whole trace based on what happened in it. It supports dynamic sampling that adapts rates to the frequency of key values, rules-based sampling with fixed rates for known conditions, throughput-based sampling that holds spans per second under a limit, and deterministic probability sampling keyed on trace ID. It also handles OpenTelemetry logs, sampling those tied to a trace along with it. Refinery clusters route spans of a trace to a single peer, which is the same routing requirement the Collector solves with the load-balancing exporter. I did not verify Refinery’s clustering and rules-file details for this post; read its scaling documentation before depending on any specific behaviour. If you already use Honeycomb, it is the path of least resistance, and it can sit in front of a Collector pipeline.

Jaeger sampling strategies

Jaeger, a CNCF project, historically supports remote sampling, where SDKs poll the Jaeger collector for per-service sampling strategies, as well as an adaptive mode that adjusts rates to hit a target per-operation throughput. Both are head-based in the sense that the decision is made at or near the SDK; they are not trace-completion samplers. Jaeger v2 is built on the OpenTelemetry Collector, so the tail_sampling processor is available there as well. I did not re-verify the exact Jaeger version behaviours for this post, so confirm against the documentation of the version you run.

Backends with built-in sampling

Several commercial vendors offer ingestion-time or tail-style sampling inside their pipelines. These save you the operational burden of tier 2 at the price of lock-in and per-GB ingest charges for data you will drop. A practical test: if the vendor charges on ingestion rather than retention, sampling inside the vendor does not reduce your bill, so do it before the data leaves your network.

Where continuous profiles fit

Sampling decides which requests to remember; it does not tell you why a code path is slow. Pairing kept slow traces with continuous profiling gives you the “which line” answer without keeping more traces. We cover that signal in OpenTelemetry profiling and continuous profiling with eBPF and OTLP.

Practical Recommendations

Start by deciding whether you need tail sampling at all. If your tracing bill is small, your trace volume is low, or your debugging does not depend on finding every error trace, a 10 percent head sample with a few always-on rules is enough, and you should stop there. Tail sampling pays when the volume is high and the cost of a missed incident is higher than the cost of a stateful tier.

If you do adopt it, roll it out in stages. Begin with metrics: deploy the span metrics connector first and move your RED dashboards and alerts onto it, so that nothing depends on stored traces for counts. Then add the load-balancing tier with the sampler in “keep everything” mode (an always_sample policy), and watch memory, late spans and dropped-too-early metrics under real load. Only then tighten the policies, one at a time, comparing the kept volume with your model.

Write policies as an explicit contract with the teams that use traces. “We keep all errors, all traces over two seconds, all canary traffic and five percent of the rest” is a statement people can plan around. Version the config, test it with synthetic traces in CI, and treat attribute-name changes as breaking.

A checklist for the first production rollout:

[ ] Services export all spans (AlwaysOn), except documented high-volume exemptions.
[ ] Tier 1 is stateless, autoscaled on CPU, and uses routing_key: traceID.
[ ] Tier 2 is a fixed-size fleet scaled in large steps, with memory_limiter ahead of tail_sampling.
[ ] num_traces is derived from arrival rate x decision_wait per replica, plus 50 percent headroom.
[ ] decision_wait is set from the measured duration distribution, not the default.
[ ] decision_cache is enabled and larger than num_traces.
[ ] maximum_trace_size_bytes is set; known giant workloads are routed elsewhere.
[ ] Span metrics are generated before sampling and dashboards read them, not traces.
[ ] Alerts exist for traces dropped too early, late-span age and per-policy share changes.
[ ] A synthetic always-error canary trace verifies the keep path end to end.
[ ] Rollouts are one replica at a time with a disruption budget.

Frequently Asked Questions

What is tail-based sampling in OpenTelemetry?

Tail-based sampling is a method where the keep-or-drop decision for a trace is made after most or all of its spans have been collected, instead of at the start. In the OpenTelemetry Collector it is implemented by the tail_sampling processor, which buffers spans per trace for decision_wait, then applies policies such as error status, latency threshold or attribute match. It enables guarantees like keeping every error trace, which head sampling cannot offer.

What is the difference between head-based and tail-based sampling?

Head-based sampling decides at the start of a trace, usually in the SDK, using only information available then, such as the trace ID. It is cheap and stateless but cannot know whether the trace will fail. Tail-based sampling decides at the end, using the full trace, so it can keep errors and slow traces. The cost is a stateful collector tier, higher network traffic, and operational complexity.

Why do I need two collector tiers for tail sampling?

All spans of a trace must reach the same collector for the processor to see a complete trace. A normal load balancer spreads spans across replicas, fragmenting traces. The first tier uses the load-balancing exporter with routing_key: traceID to send every span of a trace to one tier-2 instance, where tail_sampling runs. Separating the layers also isolates the routing failure domain from the memory-heavy sampling tier.

How much memory does the tail sampling processor need?

Resident traces equal new traces per second times decision_wait. Multiply by average trace size in memory, which is typically a few times its wire size. For example, 2,000 traces per second with a 20-second wait holds 40,000 traces; at an assumed 60 KB each that is about 2.4 GB fleet-wide before headroom. Those numbers are illustrative, so measure your own span sizes and watch dropped-too-early metrics.

Does sampling break my RED metrics and latency percentiles?

It does if you compute metrics from stored traces. Sampling distorts counts and biases percentiles when you keep errors and slow traces on purpose. Generate metrics from 100 percent of spans before sampling, using the span metrics connector, and use sampled traces only as exemplars for investigation. Then rate, error and duration dashboards remain accurate regardless of how aggressively you change sampling policies.

Can I combine head and tail sampling?

Yes, and the OpenTelemetry documentation notes it is a common arrangement: head sampling removes the highest-volume, lowest-value traffic first, then tail sampling decides on the remainder. The caveat is that tail sampling can only choose among traces that survived the head. Limit head sampling to explicitly exempt services, and keep the rest at full export so the error guarantee holds.