OpenTelemetry Logs Pipeline: A Unified Telemetry Architecture for 2026
For a decade, logs lived in a separate universe from metrics and traces. You shipped them through Fluentd or Logstash, indexed them in something expensive, and grepped when an incident forced you to. Traces went to Jaeger. Metrics went to Prometheus. Three agents, three wire formats, three mental models, and no reliable way to jump from a slow span to the log line that explained it. The opentelemetry logs pipeline collapses that fragmentation into one signal path: a single instrumentation model, a single wire protocol (OTLP), and a single programmable routing tier — the Collector — that fans out to whatever backends you choose. This is the architectural shift that made 2026 the year most platform teams finally retired their bespoke logging stack.
This article is a reference-architecture walk-through, not a marketing tour. It assumes you already run Prometheus and some tracing backend and you want to fold logs into the same fabric without doubling your bill.
What this covers: the OTel logs data model and how it reached stability, the Collector’s receiver-processor-exporter internals, agent-versus-gateway topology, log-trace correlation through trace_id, OTTL transformation, tail-based sampling constraints, connectors, and a worked cost-control example.
Context and Background
OpenTelemetry began as a merger of OpenTracing and OpenCensus in 2019, and for its first years it was a tracing project with a metrics ambition bolted on. Logs came last, and deliberately so. The maintainers recognised that logging already had thirty years of entrenched practice — syslog, log4j, structured JSON, the whole menagerie — and that a greenfield log format nobody adopted would be worse than useless. So the logs specification took a different design stance: rather than ask the world to re-instrument, it defined a data model rich enough to represent any existing log record, plus a bridge API so established logging libraries could feed OTel without applications rewriting a single log statement.
The logs data model reached stable status in the specification, and the OTLP protocol carries logs alongside traces and metrics as a first-class signal. That stability matters for a reference architecture: stable means the wire format and semantic conventions will not break under you, and backends can commit to supporting them. The OpenTelemetry logs specification documents the LogRecord fields precisely, and the model is intentionally a superset of what syslog and structured loggers already emit.
Why does this belong in a platform team’s roadmap rather than an application team’s backlog? Because the value is in correlation and consolidation, both of which are infrastructure concerns. When your service mesh, your Kubernetes control plane, and your application code all speak OTLP, you can enforce redaction, sampling, and cost policy in one place instead of in a dozen agents. If you are already investing in observability at the network layer — for example with eBPF-based service mesh observability — a unified telemetry pipeline is the natural collection tier that sits above it. The pipeline is where infrastructure telemetry and application telemetry finally converge on one schema.
The Unified Telemetry Pipeline Reference Architecture
A production OpenTelemetry deployment is almost never a single component. It is a two-tier collection topology feeding a set of specialised backends, with the Collector doing the heavy lifting of transformation and routing in between. Understanding the shape of that topology is the difference between a pipeline that scales and one that falls over at the first traffic spike.
Figure 1: End-to-end unified telemetry pipeline — SDKs export OTLP to per-node agent Collectors, which forward to a stateful gateway pool that samples, transforms, and fans out to metrics, trace, log, and cold-archive backends.
Figure 1 shows the canonical layout. Applications and services emit all three signals through the OpenTelemetry SDK, which exports over OTLP to a local agent Collector running as a DaemonSet — one per node. The agent’s job is cheap and local: receive, enrich with node and pod metadata, apply memory limits, batch, and forward. It does not make global decisions because it cannot; it only sees the traffic on its own node. The agents forward to a gateway Collector pool — a horizontally scaled, stateful set of Collectors that see aggregated traffic and therefore can make decisions that require a global view, chiefly tail-based sampling and cross-service routing. From the gateway, telemetry fans out to backend-specific exporters: metrics to Prometheus, traces to Tempo or Jaeger, logs to Loki or OpenSearch, and a copy of high-value or compliance-relevant records to cheap object storage for cold archival.
Direct answer: A unified OpenTelemetry pipeline uses two Collector tiers — lightweight agents on every node for local enrichment and batching, and a stateful gateway pool for global operations like tail sampling and routing — connected by OTLP, so that logs, metrics, and traces share one collection fabric and one policy enforcement point before fanning out to purpose-built backends.
Agent tier versus gateway tier
The two-tier split is not architectural fashion; it is forced by the physics of certain operations. Batching, compression, and metadata enrichment are embarrassingly parallel and belong close to the source, which is why agents run as a DaemonSet: each handles only its node’s volume, memory pressure is bounded, and a node failure loses only that node’s in-flight buffer. Tail-based sampling, by contrast, requires that every span of a given trace arrive at the same Collector instance so the sampler can see the complete trace before deciding to keep or drop it. A per-node agent cannot guarantee that — spans from one trace are scattered across nodes. The gateway tier solves this by load-balancing on trace_id so all spans of a trace converge on one gateway replica. This is why you cannot simply run tail sampling on your agents; the coordination constraint dictates a stateful, trace-id-aware tier.
OTLP as the single wire protocol
OTLP (the OpenTelemetry Protocol) is the connective tissue. It is a Protobuf-over-gRPC (or HTTP) format that carries logs, metrics, and traces with a shared notion of Resource and Scope attributes. Resource attributes describe where telemetry came from — service.name, k8s.pod.name, cloud.region — and are attached once per batch rather than per record, which keeps the wire efficient. Scope attributes describe the instrumentation library. Because all three signals share this envelope, a single OTLP connection multiplexes your entire telemetry stream, and a single set of resource attributes makes correlation across signals mechanical rather than heuristic. The OTLP specification defines the delivery semantics, including the partial-success responses and retryable status codes that a well-behaved gateway must honour under backpressure.
Backend-agnostic export
The strategic payoff of this architecture is that backends become swappable. Because applications export OTLP and the Collector owns the exporter configuration, moving from a self-hosted Loki to a commercial log backend — or splitting traffic across both during a migration — is a Collector config change, not a fleet-wide re-instrumentation. Your instrumentation is decoupled from your vendor. This is the single most valuable property of the pipeline for teams that have been burned by proprietary agents, and it is why even organisations that keep a commercial observability vendor increasingly put an OTel Collector in front of it as an insurance policy.
Inside the Collector: Receivers, Processors, Exporters, and Connectors
The Collector is a pipeline engine. Every unit of work is a component of one of four kinds, wired together into named pipelines in YAML. Getting the ordering right inside a pipeline is where most of the operational subtlety lives, because processors execute strictly in the order you declare them and a misordered pipeline can drop data, blow up memory, or silently fail to redact a secret.
Figure 2: Internal flow of a Collector logs pipeline — OTLP and filelog receivers feed a memory limiter, then resource detection and OTTL transformation, then batching, before exporting; a spanmetrics connector derives metrics from trace data.
Figure 2 traces a single log record through a Collector. Receivers ingest data: the otlp receiver accepts native OTLP from SDKs and downstream agents, while the filelog receiver tails log files on disk and parses them into the OTel data model — the workhorse for migrating hosts that still write to files. Processors transform the stream in order: the memory_limiter sits first as a safety valve, the resource detection processor stamps Kubernetes and cloud metadata, an OTTL transform processor redacts and enriches, and the batch processor groups records by size and time window to make exports efficient. Exporters send data onward, typically over OTLP again or in a backend-native format. Connectors are the newest and most interesting kind: they act as an exporter on one pipeline and a receiver on another, letting you derive one signal from another inside the Collector.
The connector concept
Connectors unlock patterns that used to require a separate service. The spanmetrics connector consumes a trace pipeline and emits RED metrics — request rate, error rate, duration — into a metrics pipeline, so you get service-level dashboards without instrumenting metrics separately. The count connector emits metrics counting log records or spans matching a condition, which is how you turn “how many ERROR logs per service” into a cheap time series instead of an expensive log query. Because connectors run inside the Collector, the derived signal inherits the same resource attributes as its source, so a metric produced by spanmetrics correlates cleanly with the traces that produced it. This is the mechanism behind Figure 2’s branch from the span connector into a metric exporter.
A runnable Collector configuration
The following is a real, runnable gateway Collector configuration for a logs-and-traces pipeline with OTTL redaction, tail sampling, and a spanmetrics connector. It is deliberately complete enough to adapt rather than a fragment.
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
memory_limiter:
check_interval: 1s
limit_percentage: 80
spike_limit_percentage: 20
k8sattributes:
passthrough: false
extract:
metadata:
- k8s.pod.name
- k8s.namespace.name
- k8s.node.name
transform/redact:
log_statements:
- context: log
statements:
- replace_pattern(body, "(?i)(authorization: )[^ ]+", "$$1REDACTED")
- delete_key(attributes, "http.request.header.cookie")
tail_sampling:
decision_wait: 10s
num_traces: 50000
policies:
- name: keep-errors
type: status_code
status_code:
status_codes: [ERROR]
- name: keep-slow
type: latency
latency:
threshold_ms: 500
- name: baseline-probabilistic
type: probabilistic
probabilistic:
sampling_percentage: 5
batch:
send_batch_size: 8192
timeout: 5s
connectors:
spanmetrics:
histogram:
explicit:
buckets: [100ms, 250ms, 500ms, 1s, 2s]
dimensions:
- name: http.route
- name: service.name
exporters:
otlp/logs:
endpoint: loki-gateway:4317
tls:
insecure: false
otlp/traces:
endpoint: tempo-distributor:4317
prometheus:
endpoint: 0.0.0.0:8889
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, k8sattributes, tail_sampling, batch]
exporters: [otlp/traces, spanmetrics]
metrics:
receivers: [spanmetrics]
processors: [memory_limiter, batch]
exporters: [prometheus]
logs:
receivers: [otlp]
processors: [memory_limiter, k8sattributes, transform/redact, batch]
exporters: [otlp/logs]
Read the service.pipelines block carefully, because it is where the architecture becomes concrete. The traces pipeline exports to Tempo and into the spanmetrics connector; the metrics pipeline then receives from that connector and exports to Prometheus. The logs pipeline runs redaction before batching so no unredacted record ever reaches an exporter. The memory_limiter is first in every pipeline — this is not optional, and the next section explains why the ordering is a correctness property, not a style choice.
Choosing a deployment mode
The decision of what runs where is worth making explicitly rather than by accident. The matrix below summarises the trade-offs.
| Concern | Agent (DaemonSet) | Gateway (Deployment) | Sidecar (per-pod) |
|---|---|---|---|
| Node-local enrichment | Excellent | Not possible | Excellent |
| Tail-based sampling | Impossible (partial view) | Correct home | Impossible |
| Blast radius on failure | One node | Pool-wide, needs HA | One pod |
| Resource overhead | Low, shared per node | Moderate, scales with volume | High, multiplied by pods |
| Config change frequency | Rare, stable | Frequent, policy lives here | Rare |
| Best for | Collection and batching | Sampling, routing, cost policy | Strict multi-tenant isolation |
Most teams run agents plus a gateway. Sidecars are a niche choice for hard tenant isolation where a shared agent is unacceptable, and they carry a real overhead penalty because you pay the Collector’s baseline memory for every pod. For the deeper Kubernetes topology questions this raises — how gateways scale across clusters and regions — the patterns in multi-cluster Kubernetes management with Cluster API map directly onto where you place gateway pools.
Log-Trace Correlation: The Third Pillar Earns Its Name
The phrase “three pillars of observability” is over-used to the point of cliché, but the reason logs are called the third pillar is precise and worth stating exactly: logs became a peer of traces and metrics only when they gained a reliable, standardised way to point at the other two. That mechanism is trace_id and span_id carried inside the LogRecord.
Figure 3: A single request propagates W3C trace context across two services; every log line each service emits carries the same trace_id and its own span_id, so the backend can pivot from any span to the exact logs that request produced.
Figure 3 shows the flow. A request enters Service A, which starts a span and receives a trace_id. When Service A logs, the OTel logging bridge automatically stamps the active trace_id and span_id onto the log record — the application does not add these fields manually. Service A propagates context to Service B using the W3C traceparent header, so Service B’s spans and logs carry the same trace_id. Both services export to the Collector, and the backend indexes the shared trace_id. The result is the capability that justifies the entire pillar metaphor: from a slow span in the trace UI, one click surfaces every log line — across every service — that the request produced, in order. No timestamp guessing, no grepping by request ID you hopefully remembered to log.
How the correlation actually gets populated
The correlation is not magic; it depends on two things being true. First, the application must have an active span in context when it logs — the trace context is read from the same context propagation mechanism the SDK uses for spans. Second, you must use the OTel logging bridge (or an SDK-native appender) rather than writing logs to a file that a separate agent scrapes with no knowledge of trace context. This is the subtle reason file-scraping migrations are only a first step: a filelog receiver tailing a plain text file recovers the log body but cannot reconstruct a trace_id the application never wrote into the line. Full correlation requires the application to emit structured logs that already carry the IDs, which is why mature deployments move from file-scraping to the logging bridge over time.
Resource and scope attributes as the correlation glue
Beyond trace_id, correlation across metrics and logs relies on shared Resource attributes. Because OTLP attaches the same service.name, k8s.pod.name, and deployment.environment to all three signals, a spike in an error-rate metric and the ERROR logs behind it share identifying attributes, so a dashboard can link them without a bespoke join. This is why enforcing consistent resource attributes at the Collector — via the resource detection and k8sattributes processors — is not housekeeping but the foundation of cross-signal navigation. Sloppy or inconsistent resource attributes are the single most common reason correlation “doesn’t work” in practice, and the fix lives in the pipeline, not the application.
OTTL: Programmable Transformation Inside the Pipeline
The OpenTelemetry Transformation Language (OTTL) is the Collector’s built-in language for reading and mutating telemetry as it flows through a processor. It is what turns the Collector from a dumb pipe into a policy enforcement point. OTTL statements operate on a context — log, span, metric, datapoint, resource — and each statement is a function call, optionally guarded by a where condition, that transforms fields in place.
Three use cases dominate in production. Redaction removes or masks sensitive data before it ever reaches a backend: the replace_pattern and delete_key calls in the config above strip an Authorization header and a cookie attribute, which is how you keep PII and secrets out of your log store and out of scope for a compliance audit. Enrichment adds derived context — setting a tenant.tier attribute from a namespace label, or normalising a status field — so downstream queries are cheaper and dashboards are consistent. Routing uses conditions to send different records down different pipelines: paired with the routing connector, an OTTL condition like where severity_number >= SEVERITY_NUMBER_ERROR can direct error logs to a hot, indexed backend while info logs go to cheap cold storage.
The reason OTTL matters architecturally is that it moves these decisions out of application code and out of per-agent scripting and into one declarative, version-controlled place. Redaction logic that used to be scattered across Logstash grok filters, Fluentd Ruby snippets, and application log wrappers now lives in one Collector config that you can review, test, and audit. That consolidation is a security win as much as an operational one — there is exactly one place where a reviewer confirms that no path exists for a raw credential to reach the log backend.
OTTL execution model and ordering
One subtlety trips up newcomers: OTTL statements within a processor run top to bottom, and later statements see the mutations of earlier ones. If you enrich an attribute and then route on it, the enrichment must come first; reverse them and the routing sees the un-enriched value. Statements are also context-scoped, so a log statement cannot directly mutate a resource field — you switch context explicitly. This ordering discipline is the same reasoning that puts the memory_limiter first in a pipeline: in a stream processor, position encodes causality. Treat an OTTL block like a small program with side effects, test it against representative records, and pin it in version control so a reviewer can reason about the exact sequence of transformations any record undergoes.
Migrating From Fluent Bit, Fluentd, and Logstash
Almost no team adopts an OpenTelemetry logs pipeline greenfield; you migrate an existing logging estate, and the migration order determines how much risk you take. The pragmatic path is incremental co-existence rather than a flag-day cutover, because logs are the signal your on-call engineers reach for first and you cannot afford a gap.
The lowest-risk first step keeps your existing agent — Fluent Bit, Fluentd, or a Logstash shipper — in place and simply adds an OTLP output to it, pointing at a Collector that re-exports to your current backend unchanged. Fluent Bit and Fluentd both ship OpenTelemetry outputs, so this is a configuration change on the agent, not a redeployment of your fleet. At this stage nothing about the stored data changes; you have merely inserted the Collector as a future policy point. Once that path is proven, you begin moving transformation logic — the grok patterns, the mutate filters, the Ruby snippets — out of the legacy agent and into OTTL on the Collector, deleting each legacy filter as its OTTL equivalent goes live. This is where most of the operational simplification is banked: one language, one config, one review surface replacing three tools’ worth of bespoke scripting.
The next step retires the legacy agent entirely on hosts that write to files by switching to the Collector’s filelog receiver, which tails and parses the same files. At this point Fluent Bit or Fluentd is gone from those nodes and the Collector agent owns collection end to end. The final and most valuable step — and the one that is genuinely a re-instrumentation rather than a config change — is adopting the OTel logging bridge in the application so logs are emitted already carrying trace_id and span_id, unlocking the correlation that file-scraping cannot reconstruct. Sequencing it last means every earlier step delivers value on its own, so the migration is never all-or-nothing and can pause safely at any stage.
Trade-offs, Gotchas, and What Goes Wrong
A unified pipeline concentrates power and therefore concentrates risk. Four failure modes account for most production pain, and each has a specific structural cause and remedy.
Figure 4: The four dominant pipeline risks — metric cardinality blowup, tail-sampling gaps from misrouted spans, backpressure when a backend slows, and cost overrun from full-fidelity ingest — each with its structural remedy.
Cardinality blowup is the most expensive mistake. When you promote a high-cardinality log attribute — a user ID, a request ID, a full URL with query parameters — to a metric label via a connector, you can create millions of unique time series, and metrics backends price and perform on series count. A single careless dimension can multiply your Prometheus memory footprint overnight. The remedy is disciplined attribute pruning in OTTL before any metric-producing connector, and never adding unbounded fields as spanmetrics dimensions. Treat every new metric label as a cardinality liability until proven bounded.
Sampling gaps occur when spans of one trace reach different gateway replicas because load balancing is not keyed on trace_id. The tail sampler on each replica then sees a partial trace and makes inconsistent keep/drop decisions, producing broken traces where some spans survive and others vanish. The fix is a loadbalancing exporter in front of the gateway pool that routes by trace_id, guaranteeing whole-trace locality — the coordination constraint that dictated the two-tier topology in the first place.
Backpressure is what happens when a backend slows or fails and telemetry keeps arriving. Without a bound, the Collector’s queues grow until it is OOM-killed, and you lose everything in flight — often exactly when you most need the data, during an incident. This is why the memory_limiter is first in every pipeline: it begins refusing new data before memory is exhausted, applying backpressure to senders that then retry, rather than letting the Collector die. Pair it with a persistent sending queue so a brief backend outage drains from disk instead of dropping.
Cost overrun is the quiet failure. Exporting every log and every span at full fidelity to an indexed backend is technically fine and financially ruinous. The remedy is the whole cost-control toolkit — filtering, sampling, attribute pruning, aggregation, and tiering — covered next.
Practical Recommendations
Start by putting a Collector in front of whatever you have today, even if it just re-exports to your existing backend unchanged. That single move decouples instrumentation from vendor and gives you a place to enforce policy later, with no application changes and near-zero risk. From there, migrate collection incrementally: point Fluent Bit or Fluentd at an OTLP output, or replace them outright with the filelog receiver, so hosts that write to files join the pipeline before applications are re-instrumented. Only then invest in the logging bridge to earn full trace_id correlation, which is the payoff that justifies the whole exercise.
On cost, treat it as a design constraint from day one rather than a surprise on the invoice. Here is an illustrative worked example (numbers are illustrative, not vendor benchmarks): suppose you ingest 10 TB of logs per month and an indexed backend charges roughly $2 per GB ingested — about $20,000/month. Apply an OTTL filter that drops health-check and debug logs (assume 40% of volume), prune five verbose attributes to shrink each record (assume a further 15%), and tier info-level logs to object storage at a fraction of the cost while keeping only warnings and errors hot. The hot path drops to roughly 3.5 TB, cutting indexed spend to about $7,000/month, with the archived remainder costing a small fraction of the difference. The savings come entirely from Collector configuration, not from losing signal you actually query.
It is worth being precise about where in the pipeline each cost lever acts, because they compound multiplicatively rather than additively. Filtering with an OTTL where condition removes whole records at the earliest processor, so everything downstream — batching, export, indexing, storage — never pays for them; this is the highest-leverage cut and belongs as early as the pipeline allows. Attribute pruning shrinks the records that survive, reducing both the bytes on the wire and the indexed field count, which matters because many backends price on indexed fields as well as raw volume. Aggregation via the count connector replaces a flood of near-identical log events with a single time series — turning “one million debug lines saying the same thing” into one counter — which is dramatic for chatty subsystems. Sampling reduces trace volume while tail policies preserve the traces you actually care about. Tiering is the catch-all: anything that must be retained for compliance but is rarely queried goes to object storage, where it costs a small fraction of hot indexed storage and can still be rehydrated on demand.
The discipline that makes this stick is measuring cost per signal, per service, continuously, so a team that suddenly triples its log volume is visible before the invoice arrives rather than after. Because every lever above is a Collector configuration change, cost governance lives in the same version-controlled pipeline as everything else — a pull request, not a procurement conversation.
A concrete rollout checklist:
- [ ] Deploy a gateway Collector in front of the existing backend; change nothing else first.
- [ ] Add
memory_limiterfirst and a persistent sending queue in every pipeline. - [ ] Enforce consistent resource attributes with
k8sattributesand resource detection. - [ ] Move redaction into one OTTL processor; delete scattered pe
