eBPF Observability with Pixie + Cilium: Tutorial (2026)

For most of the last decade, “Kubernetes observability” was a polite word for a punishing instrumentation tax — wire OpenTelemetry SDKs into every service, redeploy on every schema change, and then hope someone remembered to do it for the Python service that the data team owns. By mid-2026 the bill has finally come due, and the alternative that has crossed the practitioner threshold is eBPF observability with Pixie and Cilium. The pitch is unsexy and irresistible: install one DaemonSet per node, get full HTTP, gRPC, MySQL, Postgres, DNS, Redis, and Kafka visibility across every pod in the cluster, without touching application code. This tutorial walks the install, the wiring, and the gotchas end to end so an SRE who has never run eBPF in production can have signal flowing in an afternoon.

We will set up a real cluster — kind for the laptop reader, GKE/EKS/AKS commands for the team reader — install Pixie for in-cluster L7 traces and Cilium Hubble for L3-L7 flow visibility, write a working PXL script that surfaces slow endpoints, route the resulting telemetry into an OpenTelemetry Collector, and explain exactly where this stack stops being magic (TLS, kernel versions, cardinality, and a few sharper edges around encrypted gRPC). By the end you should be able to defend the architecture in a review and operate it past the demo phase.

Why eBPF Observability Took Over in 2026

Answer-first summary: eBPF observability won the 2026 Kubernetes argument because it broke the three habits that made traditional APM painful — language-specific SDKs, redeploy-to-instrument, and per-service config sprawl. With a single privileged DaemonSet attaching to kernel hooks, you get protocol-aware visibility across every pod regardless of language, with zero code changes and predictable per-node overhead. Pixie covers application-layer traces; Cilium Hubble covers network and identity flows; together they replace four or five traditional agents.

The architectural shift is not really about eBPF as a technology. eBPF — extended Berkeley Packet Filter — has been in the kernel since 2014. What changed in 2024 and 2025 was the maturity of the user-space ecosystem around it. Cilium graduated from CNCF in October 2023 and by 2026 is the default or recommended CNI on EKS, GKE Dataplane V2, AKS Azure CNI Powered by Cilium, and most on-prem distributions. Pixie graduated to CNCF sandbox in 2021, was donated by New Relic, and by 2026 is the most cited zero-instrumentation tool in production SRE writeups. Both are stable, both have multi-vendor support, and both finally have the L7 protocol coverage that traditional APM users expect.

The cost story matters too. A traditional APM agent typically allocates 50-150 MB per pod and adds 1-5 percent CPU overhead per service. Multiply that by 800 microservices in a mid-size cluster and the bill is real. eBPF agents are per-node — one PEM (Pixie Edge Module) per node, one Cilium agent per node — and their cost is dominated by the eBPF programs themselves and the data they aggregate, not by how many pods you run. On a typical 32-core node we see PEM hold steady at 200-400 MB with the cluster doing 50k requests per second, and Cilium agent at 100-250 MB. The shape of the cost curve flattens.

The third reason this pattern took over is auditability. eBPF events are anchored at the kernel, before user-space libraries can lie. For latency, for connection state, for cgroup attribution, for socket failures, the eBPF view is more faithful than what an in-process SDK reports. SREs running incident reviews have stopped accepting “the trace said it was 12 ms” when the kernel saw a 280 ms TCP retransmit. The kernel does not have a marketing budget.

Two caveats up front, because we will hit them in the install: eBPF needs a recent kernel (5.4 is a baseline; 5.10+ is what you actually want, and 6.x kernels unlock the most modern features like BPF LSM and fexit probes), and TLS visibility is non-trivial. Pixie handles TLS for OpenSSL, BoringSSL, GnuTLS, and Go’s crypto/tls through uprobes, which gives surprisingly broad coverage in practice, but statically-linked exotic TLS stacks remain a blind spot. We come back to this in the gotchas section.

Reference Stack: Pixie + Cilium + OTel

Answer-first summary: The 2026 eBPF observability reference stack has four layers — kernel hooks (syscalls, tc, XDP, tracepoints), eBPF programs attached to those hooks, user-space agents that aggregate the data (Pixie PEM, Cilium agent, Hubble Relay), and backends that consume it (Pixie’s own Live UI plus Tempo, Loki, Mimir, Prometheus, Jaeger, fronted by Grafana). An OpenTelemetry Collector sits between the eBPF agents and the long-term backends so the layout stays vendor-neutral.

The diagram above is the mental model worth memorizing. Read it bottom-up.

At the kernel layer are the hook points. Syscalls (read, write, sendmsg, recvmsg, connect, accept) carry the request and response payload bytes that L7 parsers need. The traffic control (tc) and XDP hooks see every packet entering and leaving the node, which is what Cilium uses for its datapath and what Hubble taps for flows. Tracepoints — stable, kernel-defined event points — expose scheduler, networking, and filesystem events that are useful for performance debugging. kprobes and kretprobes attach dynamically anywhere in the kernel and are the workhorse for “I need to see this specific kernel function.” uprobes do the same trick in user space, which is how Pixie reads TLS data — it hooks into SSL_read and SSL_write after the bytes have been decrypted but before they leave the library.

At the eBPF program layer, small verifier-checked bytecode programs run in response to events at those hooks. They cannot loop unboundedly, cannot dereference arbitrary memory, cannot block. They are constrained by design — that constraint is what keeps eBPF safe enough to run in production kernels. The programs emit data into BPF maps and perf/ring buffers, which the user-space agents drain.

The user-space agent layer is where Pixie’s PEM and Cilium’s agent live. PEM is a per-node DaemonSet that loads Pixie’s eBPF programs, drains the ring buffers, parses the L7 protocols (HTTP/1.1, HTTP/2, gRPC, MySQL, PostgreSQL, Cassandra, Redis, Kafka, DNS, Mongo, NATS in current builds), assembles them into structured rows, and stores those rows in an in-memory columnar table per node. Cilium’s agent does the network side — it programs the datapath, resolves pod identities via Kubernetes labels, runs L7 parsers for HTTP, gRPC, Kafka, and DNS where policy demands it, and ships flow records to Hubble Relay.

The backend layer is deliberately split. Pixie ships with its own UI (“Pixie Live”) that runs PXL scripts and surfaces the columnar data directly — this is the killer feature, and you should use it. But the same eBPF data is too valuable to lock inside Pixie’s UI, so the modern pattern is to also export it through an OpenTelemetry Collector to long-term storage (Tempo for traces, Loki for logs, Mimir or Prometheus for metrics, Jaeger if you already standardized on it), with Grafana on top for the dashboards your incident management process already trusts. Cilium Hubble’s flow events flow into the same OTel Collector and onward to Loki and Mimir, or directly into Hubble’s UI for the live service map.

A small but real point about how this reference stack composes with existing instrumentation. eBPF observability does not replace application-emitted spans, because spans carry semantic context that the kernel cannot reconstruct — business identifiers, user IDs, queue names, retry counters. The mature 2026 pattern is hybrid: eBPF for breadth (every service, every protocol, every node), SDK spans for depth (the half-dozen services where the team has invested in business-meaningful traces). Both end up in the same backend, joined on trace IDs that the SDK still emits, and the resulting service map covers the cluster while the deep traces cover what matters.

If you have wired up LLM agent telemetry recently — covered in our piece on LLM agent observability with OpenTelemetry GenAI conventions — the same OTel Collector layout slots in here unchanged. eBPF telemetry follows the same OTLP envelope as any other source once it leaves the agent.

Cluster Setup and Pixie Install

Answer-first summary: A working Pixie install needs a Kubernetes cluster with kernel 5.4 or newer, an account on Pixie Cloud (or a self-hosted Pixie control plane), Helm 3.10+, and roughly 1 GB of memory headroom per node for the PEM. The install is a two-step px deploy or helm install, and within 60 seconds the Live UI is populated with HTTP traces from any workload already running.

We will start with a local kind cluster so the reader can follow along without a cloud bill, then note the differences for managed clusters. Use a Linux host — Mac and Windows kind clusters run Kubernetes inside a Linux VM, which works but means the eBPF view is of the VM, not your host applications. That is usually fine for tutorial purposes.

# Confirm a recent kernel on the host
uname -r
# Want 5.10+; eBPF features used by Pixie need 5.4 minimum.

# Install kind and create a multi-node cluster
go install sigs.k8s.io/kind/cmd/kind@v0.23.0

cat <<EOF > kind-pixie.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
  - role: worker
networking:
  disableDefaultCNI: true     # we will install Cilium next
  kubeProxyMode: none         # let Cilium replace kube-proxy
EOF

kind create cluster --name ebpf-obs --config kind-pixie.yaml

Note that we are disabling the default CNI and kube-proxy in the kind config because Cilium will replace both. Skip this if you are joining an existing cluster — Cilium can coexist with most CNIs in a migration phase.

Now Pixie. Install the px CLI and authenticate.

# Install the px CLI
bash -c "$(curl -fsSL https://withpixie.ai/install.sh)"

# Authenticate against Pixie Cloud (or your self-hosted plane)
px auth login

# Deploy Pixie into the current kube-context
px deploy --cluster_name ebpf-obs

px deploy is a convenience wrapper around a Helm install. Under the hood it creates the pl namespace, installs the Vizier operator, and the operator reconciles a Vizier CRD that materializes the cluster-side components: the Vizier controller (which talks to Pixie Cloud over an mTLS gRPC channel), one or more Kelvin pods (the distributed query coordinators), and a PEM DaemonSet (one PEM per node).

If you want to drive it through Helm directly — for GitOps reasons or to pin versions — the equivalent is:

helm repo add pixie-operator https://pixie-operator-charts.storage.googleapis.com
helm repo update

helm install pixie pixie-operator/pixie-operator-chart \
  --namespace pl --create-namespace \
  --set deployKey=$(px deploy-key create -q) \
  --set clusterName=ebpf-obs \
  --set pemMemoryLimit=2Gi \
  --set dataAccess=Full

Once the install settles — kubectl -n pl get pods should show vizier-cloud-connector, vizier-metadata, vizier-query-broker, kelvin, and a vizier-pem pod per node, all Running — the Live UI at work.withpixie.ai will list your cluster. Click in, pick px/service_map, and you should see a generated service graph for whatever workloads are running. If your cluster is empty, deploy the canonical demo:

px demo deploy px-sock-shop -n px-sock-shop

The Pixie data path is worth a careful look before we move on. It is the part of the architecture that surprises engineers the most.

Each PEM loads Pixie’s eBPF programs against the node’s kernel and drains events into its in-memory columnar table store — think of it as a per-node Apache Arrow buffer that holds the last few minutes of http_events, mysql_events, conn_stats, and similar tables. When you run a PXL script in the Live UI, the query flows: UI → Pixie Cloud API → mTLS to Vizier → Vizier asks Kelvin to plan → Kelvin fans out predicate-pushed-down queries to every PEM → PEMs run the query in place against their local tables and stream partial aggregates back → Kelvin merges → Vizier streams to the API → API to UI.

The point that catches people is that data never leaves the cluster by default. The columnar tables live on the nodes; only the query results — the rows the script actually asks for — cross the mTLS channel out to Pixie Cloud. This is what makes Pixie acceptable to security teams that would normally object to a SaaS observability tool, and it is what makes self-hosted Pixie a meaningful option for regulated industries. The data plane is in-cluster; the control plane can be SaaS or self-hosted.

A few install knobs you will want to know about. pemMemoryLimit defaults to 2 GiB per node; on small kind clusters drop it to 1 GiB; on busy production nodes raise it to 4 GiB so the retention window is useful. dataAccess: Full allows PXL scripts to read raw request/response bodies; set it to Restricted in environments where redaction is mandatory. clockConverter and clusterUID matter if you have multiple clusters in one Pixie account — name them clearly. Finally, on hardened nodes you may need to allow bpf() and perf_event_open() syscalls explicitly, and the PEM DaemonSet needs hostPID: true to attach uprobes to processes outside its own namespace.

Cilium Hubble Setup and Service Graphs

Answer-first summary: Installing Cilium 1.17+ with Hubble Relay gives you a cluster-wide service flow graph, identity-aware L3-L7 visibility, and a hubble CLI that filters flows by namespace, label, or protocol. The Helm install takes about three minutes; turning on Hubble UI and the metrics exporter is two extra flags. With kube-proxy replacement enabled, Cilium also gives you the data path that produced the flows you are looking at.

We disabled the default CNI when we created the kind cluster. Install Cilium now.

# Install the cilium CLI
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
curl -L --fail --remote-name-all \
  https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz
sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin

# Helm install with Hubble enabled and kube-proxy replaced
helm repo add cilium https://helm.cilium.io/
helm repo update

helm install cilium cilium/cilium \
  --version 1.17.3 \
  --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=ebpf-obs-control-plane \
  --set k8sServicePort=6443 \
  --set hubble.enabled=true \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true \
  --set hubble.metrics.enabled="{dns,drop,tcp,flow,icmp,http}" \
  --set hubble.tls.auto.enabled=true \
  --set hubble.tls.auto.method=helm \
  --set bpf.lbExternalClusterIP=true \
  --set ipam.mode=kubernetes

A few of these flags matter. kubeProxyReplacement=true swaps out kube-proxy for Cilium’s eBPF-based service load balancer — faster, sees pod identities natively, and removes a moving part. hubble.relay.enabled=true puts Hubble Relay in place so you get cluster-wide flows instead of per-node ones. hubble.ui.enabled=true ships the service-map web UI. hubble.metrics.enabled turns on the Prometheus exporter for flow-derived metrics — these are the metrics you will scrape into Mimir.

Verify health:

cilium status --wait
hubble status

You should see Cilium reporting all nodes healthy and Hubble Relay reporting the relay alive with per-node flow counts. Port-forward the Hubble UI to see the live service graph:

cilium hubble ui

The browser tab that opens shows pods grouped by namespace and connected by edges weighted on flow volume. Click an edge and you get the underlying flows — source identity, destination identity, protocol, HTTP method and path for HTTP flows, gRPC service and method for gRPC flows, and status. This is the part that makes engineers stop in their tracks the first time. You did not deploy a single SDK and you can see every HTTP call between every pod in the cluster.

For the CLI lovers, hubble observe is the equivalent of tcpdump for L3-L7 events:

# All HTTP flows in the default namespace
hubble observe --namespace default --protocol http

# Just drops, to find policy violations
hubble observe --type drop --last 5m

# A specific service pair, with JSON output for scripting
hubble observe --from-pod default/frontend --to-pod default/api -o json

The architecture under that simple CLI is layered. Each node’s Cilium agent runs the eBPF programs at the tc, XDP, socket, and cgroup hooks. The agent’s L7 parser plugs in only for protocols you have asked Hubble to observe, so HTTP and gRPC visibility costs a little more than pure L3-L4. The per-node agent ships flow records to Hubble Relay, which aggregates across the cluster and exposes the API the CLI and UI both consume.

A few production knobs you will need eventually. Set hubble.relay.replicas to 2 or more in production. Set hubble.eventBufferCapacity to something larger than the default (4095) on busy nodes, otherwise you will drop events under load and hubble status will report a non-zero drop rate. If you want long-term flow archive, point Hubble Relay’s export at an OTel Collector with the cilium receiver or use the JSON file exporter and ship those files via Vector or Fluent Bit. Finally, for clusters with thousands of pods, watch the Hubble cardinality — every flow carries source and destination labels, and unbounded label sets will blow up your metrics backend. We come back to this in the gotchas section.

On managed clusters the install differs only slightly. EKS clusters created with the AWS VPC CNI need a migration — see Isovalent’s documentation for the supported procedure; on GKE you choose Dataplane V2 at cluster create time and Cilium is the data plane out of the box; on AKS you can ask for Azure CNI Powered by Cilium. In each case Hubble has to be enabled explicitly, but the agent and datapath come with the distribution.

If you have been weighing this against a service-mesh approach to service-to-service observability, our Kubernetes Gateway API ingress migration ADR discusses the boundary — Gateway API plus Cilium covers most of what teams used to reach for Istio for, and Hubble fills the observability slot without needing a sidecar mesh.

Capturing HTTP/gRPC/DB Traces

Answer-first summary: Pixie’s PXL is the lever that turns raw eBPF events into actionable telemetry. A PXL script is Python-like, runs distributed across all PEMs in the cluster, and reads from typed tables (http_events, mysql_events, conn_stats, stack_traces, etc.). A 15-line script gives you P99 latency by service, top slow endpoints, error breakdown, and database query latency — without touching application code.

PXL is intentionally narrow. It is a dataframe DSL, executed in PEMs against their local columnar stores, with predicate pushdown. The runtime evaluates partials in parallel, Kelvin merges, and you get a single result in the UI or CLI. This means PXL scripts are cheap — they are not Splunk-style log scans, they are columnar aggregations against bounded recent windows.

Here is a working PXL script for slow HTTP endpoints by service. Save as slow_endpoints.pxl:

import px

# Pull recent HTTP events (PEM retains a rolling window)
df = px.DataFrame(table='http_events', start_time='-5m')

# Attribute every event to its source pod and service
df.service = df.ctx['service']
df.pod = df.ctx['pod']
df.namespace = df.ctx['namespace']

# Normalize the request path so /users/123 and /users/456 group together
df.endpoint = px.uri_recompose(
    px.pluck(df.req_path, '/'),
    px.regex_match('^[0-9a-f-]+$', df.req_path),
    keep_query=False,
)

# Compute latency in ms from the kernel-measured request/response delta
df.latency_ms = df.latency / 1.0e6

# Group and aggregate
slow = df.groupby(['service', 'endpoint']).agg(
    count=('latency_ms', px.count),
    p50_ms=('latency_ms', px.quantiles),
    p99_ms=('latency_ms', px.quantiles),
    error_rate=('resp_status', lambda s: px.mean(s >= 500)),
)

# Filter and sort
slow = slow[slow.count > 20]
slow = slow.sort_values('p99_ms', ascending=False)
px.display(slow.head(25), 'slow_endpoints')

Run it from the CLI:

px run -f slow_endpoints.pxl -c ebpf-obs

The first time you run this against a real workload, it is unsettling how complete the picture is. You will see latencies for endpoints you forgot you had. The http_events table includes the full request and response, including headers and body samples — the column names you will reach for most are req_method, req_path, req_headers, req_body, resp_status, resp_headers, resp_body, latency (nanoseconds, kernel-measured), and trace_role (client/server). Equivalent tables exist for mysql_events, pgsql_events, redis_events, kafka_events, dns_events, mongodb_events, and cql_events (Cassandra). The schema is documented at the Pixie docs site.

For gRPC, the trick is that Pixie sees HTTP/2 frames and reconstructs the gRPC method from the :path pseudo-header. Filter by req_headers["content-type"] == "application/grpc" and parse the path. Pixie also includes req_body and resp_body columns that contain the protobuf payload as base64 — you can either decode in PXL with px.pb_decode(...) against a registered protobuf schema, or just keep the raw bytes and decode downstream.

For database tracing, the tables are pre-parsed. A working query for slow MySQL queries:

import px

df = px.DataFrame(table='mysql_events', start_time='-10m')
df.service = df.ctx['service']
df.latency_ms = df.latency / 1.0e6

# Group by normalized query text
df.query = px.normalize_mysql(df.req_body)
slow = df.groupby(['service', 'query']).agg(
    n=('latency_ms', px.count),
    p99_ms=('latency_ms', px.quantiles),
    error=('resp_status', lambda s: px.sum(s != 0)),
)
slow = slow.sort_values('p99_ms', ascending=False)
px.display(slow.head(20), 'slow_mysql')

The data path under these scripts is the part that surprises people who have only used trace backends. The query is executed in the PEMs themselves against the kernel-sourced rows, and only the aggregated results cross the network. The cluster does not have to ship raw events to a central indexer; it ships answers.

A few PXL patterns worth knowing. px.add_otel_span_ids(df) injects trace and span IDs derived from connection metadata, which lets you correlate PXL output with SDK-emitted spans downstream. px.equals_pii_safe(df.req_body, "credit_card") is the redaction primitive you actually want — it gives you visibility into request bodies without splatting PII into your tables. And the px/ directory in the Live UI ships dozens of curated scripts (px/service_map, px/pgsql_data, px/http_data, px/cpu_flamegraph) that are excellent reading for learning the schema.

The other half of “capturing traces” is CPU profiles. Pixie’s stack_traces table is populated by an eBPF-based sampling profiler that fires every 10 ms by default. A single PXL script can produce a flame graph for any pod, language, runtime — JVM, Go, Rust, Python, Node — and the output is the same kind of folded-stack format that pprof and Speedscope already understand. The script is documented in px/cpu_flamegraph. For a Rust service in production, it took us a single PXL run to identify a hot serialization path that an SDK-based profiler had been quietly truncating because the symbols lived in a stripped binary.

Routing eBPF Telemetry to OTel + Backends

Answer-first summary: The OpenTelemetry Collector is the seam that keeps your eBPF stack from being locked into any vendor. Both Pixie (via the OTel exporter plugin and Pixie’s OTLP push) and Cilium Hubble (via the OTel cilium or hubblexporter paths) can emit OTLP into a Collector DaemonSet, which then enriches with Kubernetes attributes, samples, batches, and fans out to Tempo, Loki, Mimir, Prometheus, and Jaeger.

The collector layout we recommend for a 2026 install is a two-tier topology — a DaemonSet collector on every node for receiving local agent traffic and doing the cheap, fast enrichment work, plus a small Deployment of gateway collectors that handle tail sampling, fan-out, and durable buffering. The OpenTelemetry community has been converging on this split for two years; it is what the Cilium Hubble OTel exporter assumes and what the Pixie OTel reference architecture publishes.

A minimal node-local Collector config that takes Hubble flows on OTLP and forwards to a gateway:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  k8sattributes:
    auth_type: serviceAccount
    extract:
      metadata: [k8s.namespace.name, k8s.pod.name, k8s.node.name]
  batch:
    send_batch_size: 8192
    timeout: 2s
  resourcedetection:
    detectors: [env, k8snode, system]

exporters:
  otlp/gateway:
    endpoint: otel-gateway:4317
    tls:
      insecure: true

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [k8sattributes, resourcedetection, batch]
      exporters: [otlp/gateway]
    traces:
      receivers: [otlp]
      processors: [k8sattributes, resourcedetection, batch]
      exporters: [otlp/gateway]

On the Pixie side, the OTel export uses a small bridge that runs OTelExportSinkOperator PXL scripts on a schedule and pushes spans to your collector. The reference script (px/otel_export) ships out of the box; the only thing you need is the OTLP endpoint of your node-local collector and an API key if your gateway requires one. On the Cilium side, point Hubble Relay at the same OTLP endpoint by setting hubble.export.dynamic.enabled=true and configuring the file-based or OTLP-based exporter in the Helm values.

Once the wiring is in place, the cluster produces a useful unified picture in Grafana. The Tempo data source shows service-level traces drawn from Pixie’s HTTP/gRPC parsing; the Loki data source shows Hubble flow logs filtered by namespace and label; Mimir has the L4 flow metrics, the HTTP request rate by status class, and the per-namespace egress byte counts. Build the dashboards once, get them right, and they will serve incident reviews for years because the underlying data sources do not change as you redeploy services.

For trace storage costs, tail sampling at the gateway collector is the only sustainable approach. Pixie’s HTTP table grows with traffic; if you forward every request as a span you will blow up Tempo’s ingest. The Collector’s tail_sampling processor lets you keep all error spans, all spans over a latency threshold, and a probabilistic sample of the rest — typically 1-5 percent of healthy traffic. Configure it once, save several thousand dollars a month.

Trade-offs, Gotchas, and What Goes Wrong

Answer-first summary: eBPF observability is not free magic. The four traps that bite real teams are kernel version mismatches (older nodes silently lose features), TLS visibility gaps for statically-linked or exotic crypto stacks, cardinality explosions in Hubble metrics if labels are not pruned, and per-node memory consumption when retention windows are tuned too aggressively. None are dealbreakers, but each needs an explicit decision.

The kernel version trap is the most common. Pixie’s HTTP/2 and TLS uprobes lean on features that landed cleanly only in 5.10. Run an older node and you will see silent gaps — some protocols simply will not be parsed. Cilium has a feature matrix on its site that maps kernel version to capability; consult it before assuming “eBPF” means “the same eBPF” across your fleet. On AWS, Bottlerocket and Amazon Linux 2023 ship modern kernels; on Azure, AKS Mariner is up to date; on GKE, COS is current. The risk is mostly on bare-metal clusters running long-lived RHEL or Ubuntu LTS hosts.

TLS is the second trap. Pixie’s uprobe approach works for OpenSSL, BoringSSL, GnuTLS, and Go’s standard crypto/tls — that covers the vast majority of services. It does not work for statically-linked Rust binaries using rustls without symbols, for Envoy’s BoringSSL fork in some hardened builds, or for any process that opts into kTLS where decryption happens in the kernel. For those, the only options are SDK-emitted spans or an eBPF kTLS hook (still experimental in 2026). Audit your fleet for these cases before promising “full L7 visibility” to leadership.

Cardinality is the third. Hubble’s flow exporter, with its rich Kubernetes labels, can produce metrics with thousands of unique label combinations per service. If you ship those raw into Mimir or Prometheus, ingest will fall over. The mitigations are: prune labels at the Hubble side using hubble.metrics.enabled with explicit context filters, drop high-cardinality dimensions in the OTel Collector with metricstransform, and tag namespaces as either “high-fidelity” (full labels) or “low-fidelity” (aggregated). Cluster cardinality budgets are a real planning artifact; make one.

Finally, memory. PEM holds its columnar tables in RAM. The default 2 GiB per node will give you a few minutes of retention on a busy node and tens of minutes on a quiet one. If your incident workflow needs a longer scrollback, raise the limit — but that memory is gone for application workloads, so it is a real trade-off. The same applies to Hubble’s event buffer. Watch for Dropped counts in hubble status and pl/vizier-pem OOMKills as the canary signals. Adjacent debate: how this stack composes with industrial-grade data lakehouses, which we cover in our Iceberg vs Delta vs Hudi industrial lakehouse ADR — the same telemetry can feed both real-time SRE views and long-term analytics tables.

Practical Recommendations

Answer-first summary: Start with Cilium (you probably need a CNI anyway), turn on Hubble for the free service map, then add Pixie for L7 trace depth. Keep your existing OpenTelemetry SDKs for the services where business context matters, and bridge eBPF telemetry into the same OTel Collector you already run. Treat kernel version, TLS coverage, and cardinality as planning artifacts before the rollout, not surprises after.

A concrete checklist that has held up across the eBPF rollouts we have seen reach production:

Adopt Cilium first if you are already in a CNI decision window; Hubble is the cheapest observability win you will get this year.
Pilot Pixie in a non-production cluster for a week before going wide; the Live UI alone is a sufficient case for the budget conversation.
Lock down dataAccess: Restricted for any cluster that touches regulated workloads; raw body inspection is a feature, not a default.
Standardize on an OTel Collector DaemonSet and a small gateway Deployment; resist the temptation to wire each agent directly to a backend.
Inventory kernel versions and TLS stacks before the rollout; document the gaps in a runbook so on-callers know where blind spots live.
Set explicit cardinality budgets per namespace; review them quarterly.
Keep application SDK spans for the half-dozen services where business semantics matter; do not try to replace them with eBPF.
Bake PXL scripts into your incident process — write a px/incident_quick_look script that captures the data you would want during a sev2.

FAQ

What is eBPF observability and why does it matter for Kubernetes?
eBPF observability uses kernel-level instrumentation to capture network, system, and application-layer events without modifying application code. For Kubernetes, this means a single per-node DaemonSet (Pixie PEM or Cilium agent) can see HTTP, gRPC, database queries, DNS lookups, and TCP flows for every pod on that node, regardless of programming language or framework. Compared to traditional APM, eBPF observability removes the redeploy-to-instrument step, eliminates per-pod agents, and gives kernel-faithful latency measurements. It does not replace SDK-emitted spans for business-context tracing, but it covers the breadth that SDKs cannot.

How does Pixie compare to Cilium Hubble?
Pixie and Cilium Hubble are complementary, not competitive. Cilium Hubble is a network-flow observability tool tied to the Cilium CNI — it gives you per-flow visibility, identity-aware service maps, and L7 metrics for HTTP, gRPC, Kafka, and DNS. Pixie is an application-observability tool — it gives you full HTTP and gRPC request/response inspection, database query latency, CPU profiles, and a scriptable query language (PXL). In production, teams typically run both: Cilium for the CNI and L3-L7 network view, Pixie for the deeper application-protocol view and ad-hoc debugging.

Is Pixie production-ready in 2026?
Yes. Pixie is a CNCF sandbox project (originally donated by New Relic in 2021) and is deployed in production at hundreds of organizations as of 2026. It runs on every major managed Kubernetes (EKS, GKE, AKS, OpenShift) and on most on-prem distributions. The Pixie Cloud control plane is offered as SaaS, and a self-hosted control plane is fully supported for regulated environments. The data plane stays inside the cluster — only query results cross the mTLS boundary to the control plane — which is what makes Pixie acceptable to most security teams.

What kernel version do I need for Pixie and Cilium?
Pixie officially supports Linux kernel 4.14 and above, but in practice you want 5.4 minimum and 5.10 or newer to get full HTTP/2 and TLS uprobe coverage. Cilium supports 4.19 minimum, but advanced features (kube-proxy replacement, full L7 policy, WireGuard transparent encryption, BPF LSM) need 5.10 or 5.15. Run uname -r on every node; if you have a heterogeneous fleet, plan a kernel upgrade as part of the rollout.

Can eBPF see encrypted HTTPS and gRPC traffic?
Yes, with caveats. Pixie attaches uprobes to common TLS libraries (OpenSSL, BoringSSL, GnuTLS, Go’s crypto/tls) at the points where data is already decrypted in user space, so you see the plaintext request and response. This covers most fleets — anything written in Go, most Python and Node services, anything linked against OpenSSL. It does not cover statically-linked Rust binaries using rustls without symbols, kTLS-enabled processes where decryption happens in the kernel, or some custom or exotic crypto stacks. For those, you fall back to application SDK spans or live with the gap.

eBPF Observability with Pixie + Cilium: Tutorial (2026)

eBPF Observability with Pixie + Cilium: Tutorial (2026)

Why eBPF Observability Took Over in 2026

Reference Stack: Pixie + Cilium + OTel

Cluster Setup and Pixie Install

Cilium Hubble Setup and Service Graphs

Capturing HTTP/gRPC/DB Traces

Routing eBPF Telemetry to OTel + Backends

Trade-offs, Gotchas, and What Goes Wrong

Practical Recommendations

FAQ

Further Reading

Related

Comments

Leave a Reply Cancel reply

Tag Cloud

Categories