Continuous Profiling in Production: Parca vs Pyroscope vs eBPF Profilers (2026)

Your dashboards say p99 latency doubled after Tuesday’s deploy. Metrics tell you which service. Traces tell you which span. Neither tells you which function, on which line, burning which CPU cycles. By the time an engineer SSHs into a node to run perf record, the regression may have rolled off, the noisy neighbour may be gone, and the flame graph you capture is a sample of a different problem. Continuous profiling fixes this by treating stack samples as a fourth telemetry signal: always on, stored cheaply, and queryable by time, label and version after the incident has already happened.

In 2026 the category has three viable shapes. Pyroscope, now at 2.0 after a ground-up storage rewrite, is the Grafana-native choice. Parca, from Polar Signals, is the eBPF-first, whole-host option. And the eBPF profiler lineage that Elastic donated to OpenTelemetry is becoming the shared collection layer under several backends. This post compares them at the mechanism level, with worked overhead and storage arithmetic, so you can choose and roll out without guessing.

What this covers: how sampling and stack unwinding actually work, the architectural differences in the Parca vs Pyroscope debate, how to size overhead and storage, failure modes, and a staged rollout plan.

Context and Background

Profilers are old. perf, gprof and DTrace have been around for decades. What changed is the operating model. Google described “Google-Wide Profiling” in a 2010 IEEE Micro paper: sample a small slice of machines continuously, store the stacks centrally, and query them fleet-wide. The insight was economic, not technical. If a fleet spends a few percent of its CPU in one hot function, shaving that function pays for the profiler many times over, and you only find it by looking at production, not at a benchmark.

Three developments made the idea mainstream outside hyperscalers. First, Go’s pprof format, a gzipped protobuf of samples, locations, functions and labels, became a de facto interchange format. It is the wire format Parca and Pyroscope both accept, and it is the conceptual ancestor of the OpenTelemetry profiles data model. Second, eBPF let a kernel-side program capture user and kernel stacks without modifying the target process. Third, object storage became cheap enough that storing profiles for weeks is a rounding error next to your metrics bill.

The standards picture is moving. OpenTelemetry announced profiling support in March 2024, with Elastic pledging to donate its eBPF-based profiling agent, and the goal of linking profiles bidirectionally with logs, metrics and traces (see the OpenTelemetry profiling announcement). Reporting on Pyroscope 2.0 notes the Profiles signal has since reached alpha and that Pyroscope 2.0 has native OTLP profiling ingest. Grafana’s own pyroscope.ebpf component in Alloy embeds a fork of that OpenTelemetry eBPF profiler, which is why the “Parca vs Pyroscope” framing is less binary than it was in 2022.

If you already run the surrounding stack, the integration story matters. Profiles correlate naturally with traces, so the context propagation covered in our W3C Trace Context and Baggage guide is the join key you will want. If your metrics are in a long-term store, the cardinality and retention trade-offs in Thanos vs Mimir vs VictoriaMetrics apply to profile labels in almost identical form.

Why not just run perf on demand?

On-demand profiling answers “what is this process doing now”. Incident work asks “what was it doing at 02:14 last night, and how does that differ from last Thursday”. The second question needs retention, labels and diffing. It also needs the capture to be safe enough to leave on, which is a property of the collection mechanism, not of the UI. That is where the three families diverge.

Core Argument: Two Collection Models, One Storage Problem

The core claim of this post: the real choice is not “Parca or Pyroscope”. It is how you collect (in-process SDK versus host-wide eBPF) and where you store (a purpose-built profile database versus something bolted onto your existing stack). Collection determines your language coverage, symbol quality and security footprint. Storage determines your cost curve and query latency. The products bundle those choices differently, and since 2025 they have started to unbundle them.

Direct answer: Use an eBPF profiler (Parca Agent, or Alloy’s pyroscope.ebpf built on the OpenTelemetry eBPF profiler) when you want zero code changes and whole-node coverage. Use language SDKs in Pyroscope when you need runtime-specific detail such as goroutines, allocation or lock profiles. Most mature fleets end up running both, feeding one backend.

Figure 1: Reference architecture for continuous profiling. Two collection paths (in-process SDK and node-level eBPF agent) converge on a profile store backed by object storage, queried through a flame graph UI.

The diagram shows the shape that every deployment shares. Collectors sample stacks at a fixed frequency, attach labels (service, pod, version, region), batch and upload pprof-like payloads, and a store indexes by labels and time while deduplicating symbols. The query layer merges samples across the selected label set and time range into a single flame graph, or diffs two such selections.

Collection model 1: in-process SDK and pull scraping

The Pyroscope lineage began here. For Go, the runtime already exposes net/http/pprof endpoints. A scraper, historically the Pyroscope agent and now Grafana Alloy’s pyroscope.scrape, polls those endpoints on an interval and ships the profiles. For Java, Python, Ruby and others, an SDK or an agent (async-profiler for the JVM, for instance) samples inside the runtime.

The benefit is semantic richness. Inside the runtime you can see goroutines, heap allocations by call site, mutex contention and wall-clock time, not just on-CPU samples. The cost is deployment friction: every service needs an SDK or an exposed endpoint, every language needs separate treatment, and a team that does not opt in is invisible.

Collection model 2: eBPF whole-host sampling

An eBPF profiler attaches a program to a perf event, typically a CPU clock event, firing on every logical CPU at a set frequency. When it fires, the BPF program walks the stack of whatever was running, user and kernel, and increments a counter keyed by the stack and process. User space reads the aggregated map periodically, so the per-sample work stays inside the kernel and the data crossing to user space is a compact histogram, not a stream of events.

Two published defaults are worth anchoring on. Parca’s documentation states its agent observes stack traces 19 times per second per logical CPU and requires Linux 4.18 or later. Grafana Alloy’s pyroscope.ebpf documents a default sample_rate of 19 samples per second as well. The odd number is deliberate: a prime-ish frequency avoids lock-step aliasing with periodic work such as 10 ms or 100 ms timers.

Collection model 3: the shared OpenTelemetry eBPF profiler

Alloy’s component page says it embeds a fork of OpenTelemetry’s eBPF profiler. That profiler brings dedicated unwinders for interpreted runtimes, and the docs list HotSpot JVM, Python, Ruby, PHP, Perl, V8 and .NET alongside native code (C, C++, Rust, Zig, Go). This matters strategically: when the collection layer is common, the backend becomes the differentiator, and switching costs drop because the payload is converging on OTLP profiles.

Deeper Analysis: Unwinding, Overhead and Storage Math

How stack unwinding actually works, and why it breaks

A sample is only useful if you can reconstruct the call chain. There are three techniques, and the failure modes differ.

Frame pointers. If every function keeps the caller’s frame address in the base pointer register (RBP on x86-64), unwinding is a linked-list walk: read RBP, read the return address above it, follow the saved RBP. It is a handful of memory reads per frame, which is cheap enough for BPF. The catch: compilers often omit frame pointers to free a register. The Alloy docs state this directly: binaries compiled without frame pointers produce shallow stacks, and they recommend -fno-omit-frame-pointer. Go on amd64 keeps frame pointers by default, which is one reason Go profiling has been easy for years. For distribution packages, check your distro: I believe several recent distributions changed their defaults to keep frame pointers, but verify against your own base images rather than trusting that.

DWARF-derived unwind tables. When frame pointers are absent, the compiler’s .eh_frame data describes, for each instruction address range, how to recover the caller’s stack pointer and return address. A user-space component preprocesses those rules into compact tables, loads them into BPF maps, and the in-kernel unwinder does a binary search per frame. This is the approach associated with Parca Agent; the agent’s design documents describe it, though the overview page I verified does not spell it out, so treat the exact internals as something to confirm in the Parca docs for your version. The costs are real: table generation per binary, map memory, and a per-frame lookup that is slower than a pointer chase.

Runtime-aware unwinders. For Python, Ruby, Java and Node, the native stack shows only the interpreter loop. You need an unwinder that walks the interpreter’s own frame structures and maps them to function names. The OpenTelemetry eBPF profiler ships these, per the Alloy documentation’s language list. They are version-sensitive: an interpreter minor release that changes an internal struct layout can silently degrade stack quality until the profiler is updated.

Figure 2: Three unwinding paths from a perf event to a symbolised stack. The decision depends on how the binary was compiled and which runtime it hosts.

The practical consequence: before you trust any flame graph, check its depth distribution. A profile where most stacks are two or three frames deep and end in [unknown] is an unwinding failure, not a performance finding.

Worked overhead arithmetic

Sampling cost scales with frequency, core count and stack depth. A simple model helps you reason before measuring. Let f be samples per second per CPU, N logical CPUs, c the CPU cost of one sample (BPF stack walk plus map update).

Total samples per second = f x N. For f = 19 and N = 64, that is 1,216 samples per second per node.

Suppose c is 5 microseconds for a frame-pointer walk of about 30 frames, and 20 microseconds for a DWARF-table walk of the same depth. These per-sample costs are illustrative assumptions for the arithmetic, not measured benchmarks. Then CPU burned per second:

• Frame pointers: 1,216 x 5 us = 6.08 ms per second of one core, or about 0.0095% of the node’s 64 cores.
• DWARF tables: 1,216 x 20 us = 24.3 ms per second, about 0.038% of the node.

Even if the real cost were ten times higher, you would still be under half a percent of node capacity. The sampling itself is rarely the problem. The overhead that actually bites sits elsewhere: user-space processing of the aggregated maps, symbolisation of new binaries, uploading profiles, and the memory held by unwind tables. Vendor documents describe eBPF agent overhead qualitatively as very low; Parca’s page gives no number. Measure on your own hardware, and compare node CPU with and without the agent over a week, not an hour.

Storage arithmetic

Profiles are sampled stacks, so their size depends on distinct stacks, not on request volume. A rough model: a process at 19 Hz over a 10-second upload window on 4 busy cores yields at most 760 samples. Many repeat the same stack, so a compressed pprof for one process might land in the tens of kilobytes. The InfoQ report on Pyroscope 2.0 notes that profiles can reach tens of megabytes in some cases, such as large symbol tables or heavy heap profiles.

Take an illustrative fleet of 500 nodes, each with 40 profiled processes, uploading every 10 seconds, at 50 KB per profile (assumed, not measured):

• Uploads per day per process: 8,640. Per fleet: 500 x 40 x 8,640 = 172.8 million profiles.
• Raw volume: 172.8 million x 50 KB = 8.64 TB per day, before any dedup or compression across profiles.

That is the number that sinks naive designs, and it is why storage architecture is the real differentiator. Two levers dominate: write amplification and symbol dedup.

Pyroscope 2.0: what changed

The InfoQ report (13 May 2026) on Grafana’s April 2026 announcement describes the shift concretely. In Pyroscope v1, every profile was written three times for durability. In 2.0 each profile is written once to object storage, which becomes the single source of truth. Profiles from the same service are co-located so symbol data (function names, source locations, stack traces) can be deduplicated; Grafana reports up to a 95 percent reduction in symbol storage in its own production environment, a vendor figure for their workload that you should not assume generalises. The read path is stateless, so any querier can serve any query and querier count follows demand. The segment writer is diskless and the store-gateway component was removed. Grafana says deployments that took 8 to 12 hours in v1 now take minutes, and Grafana Cloud Profiles processed 19.5 PB of profiling data on this architecture (the article’s date for the start of production use looks inconsistent with the announcement, so I am not repeating it).

For self-managed distributed deployments, object storage is now required. If you run on-premise with no S3-compatible store, that is a prerequisite to budget for.

Applying the three-times write multiplier to our illustrative 8.64 TB per day: v1-style replication implies roughly 25.9 TB of writes per day, versus 8.64 TB once. Dedup then reduces the stored footprint further. This is arithmetic on assumed inputs, but the ratio, 3x on the write path, comes from the published description.

Parca’s storage approach

Parca is built by Polar Signals around a columnar store. The InfoQ report mentions Polar Signals co-founder Frederic Branczyk discussing eBPF and the FrostDB time-series database. FrostDB is an embedded columnar database in Go, and Parca stores profile samples in column form so that label filtering and aggregation scan only needed columns. Parca can run as a single binary with local storage for small deployments, which is a genuine advantage for evaluation and for edge or single-cluster use. For multi-cluster retention, you will want to read Polar Signals’ current guidance on object storage and the managed offering, since the open-source project’s scale-out story is less documented than Pyroscope’s.

Figure 3: Write and read paths in an object-storage-first profile store. Writers batch and co-locate by service, deduplicate symbols and write once; stateless queriers fetch and merge on demand.

Querying: merge, diff and labels

A flame graph in production is not one process; it is a merge. You pick a label selector, say service=checkout, version=v142, and a time range. The store fetches matching samples and sums counts per stack. The diff view subtracts one merge from another, so a red frame means “more time here than in the baseline”. Diffing across deploy versions is the highest-value workflow, and it only works if you label profiles with a version or commit.

Keep label cardinality low. Pod name is useful but churns; if you retain it, retention cost grows with every rollout. A common compromise is to label by service, version, region and node pool, and keep pod as an ephemeral label with short retention.

pprof and the data model

pprof is a protobuf with string table, sample types (for example cpu in nanoseconds, alloc_space in bytes), samples referencing location IDs, and locations referencing function IDs with line numbers. Because the string table and function entries repeat across uploads of the same binary, cross-profile dedup is possible, which is exactly what service co-location exploits. If you can read a pprof file with go tool pprof -top, you can debug an ingestion problem without any vendor tooling.

Here is a minimal Go setup that exposes the standard endpoints for a pull-based scraper. It is the standard library; the scraper side is configuration you should check against your Alloy version.

package main

import (
    "log"
    "net/http"
    _ "net/http/pprof" // registers /debug/pprof/* handlers
    "runtime"
)

func main() {
    // Mutex and block profiles are off by default; enable at low rates.
    runtime.SetMutexProfileFraction(10)
    runtime.SetBlockProfileRate(10000) // nanoseconds
    go func() {
        // Bind to localhost or a pod-internal port, never the public listener.
        log.Println(http.ListenAndServe("127.0.0.1:6060", nil))
    }()
    // ... application work ...
    select {}
}

(Standard library usage, not run in this environment. Verify scraper configuration against your Alloy or Pyroscope docs.)

And a conceptual Kubernetes fragment for a node agent using Alloy’s eBPF component. The capability list below is taken from the Alloy documentation; field names for your chart may differ.

# Conceptual DaemonSet security context for an eBPF profiler (verify against current docs)
securityContext:
  runAsUser: 0
  capabilities:
    add: [BPF, PERFMON, SYS_PTRACE, CHECKPOINT_RESTORE, SYS_RESOURCE, DAC_READ_SEARCH, SYSLOG]
hostPID: true
# plus read-only mount of /sys/kernel/tracing and writable /tmp/symb-cache

Parca vs Pyroscope vs eBPF Profilers: Decision Matrix

The table compares the options on the dimensions that decide real adoption. Where a cell says “verify”, I could not confirm the detail from primary documentation this run.

Dimension	Pyroscope (2.0)	Parca	OTel eBPF profiler (via Alloy or other backends)
Primary collection	SDKs, pprof scrape, plus eBPF via Alloy	eBPF Parca Agent	eBPF, common collection layer
Code changes needed	Optional (SDK) or none (eBPF)	None	None
Runtime-specific profiles (goroutines, allocations, locks)	Yes via SDK or pprof endpoints	Needs manual instrumentation per docs	Mostly CPU samples
Interpreted runtimes	SDKs, plus eBPF unwinders in Alloy	Docs emphasise compiled languages (C, C++, Go, Rust)	HotSpot, Python, Ruby, PHP, Perl, V8, .NET per Alloy docs
Storage	Object storage required for distributed mode, write once	Columnar store (FrostDB lineage), single binary option	Backend dependent
OTLP profiling ingest	Native in 2.0 per reporting	Verify for your version	Native by design
Privilege needed	Low for SDK; high for eBPF	Root or broad capabilities on every node	Root or broad capabilities
Best fit	Teams already on Grafana, LGTM stack	Whole-host, no-instrumentation profiling, smaller footprints	Standardised collection feeding any backend

Three conclusions follow.

First, if you want runtime detail for a handful of critical Go or JVM services, SDK or pprof scraping gives you data an eBPF sampler cannot, because allocation sites and lock contention are properties of the runtime, not of the CPU’s program counter. Parca’s own documentation says runtime-specific information such as goroutines requires manual instrumentation.

Second, if your problem is breadth, an eBPF profiler wins. A new service shows up in profiles the day it is deployed, with no pull request to its owners. Breadth is what finds the surprise: a sidecar spending 8 percent of node CPU on JSON logging, a cron job that nobody owns.

Third, the privilege trade-off is unavoidable. Alloy’s eBPF component needs root in the host PID namespace, or a specific capability set including BPF, PERFMON, SYS_PTRACE and DAC_READ_SEARCH, plus access to kernel tracing. A security team will ask what happens if the profiler is compromised. Answer with the capability list, an image you build and sign, and a network policy that only permits egress to the profile store.

Worked example: finding a 6 percent regression

Imagine a payments service on 200 pods averaging 2 cores each, 400 cores total. After deploy v142, the CPU graph rises from 41 percent to 47 percent utilisation with no change in request rate. That is 6 percentage points of 400 cores, 24 cores, roughly a dozen on-demand instances’ worth of capacity depending on instance size (illustrative figures).

You select service=payments, compare version v141 against v142 in the diff view, and sort by self-time delta. A frame, say a JSON canonicalisation function, grew from 3 percent to 9 percent of samples: exactly the 6 points. You open the source line, find a new call inside a loop, and ship a fix. Without profiling you would have bisected commits and load-tested each. The mechanism here is arithmetic on sample proportions: if the process total is stable, share-of-samples deltas map straight onto CPU deltas.

Note the key precondition: you must have both versions’ data stored with a version label. Continuous profiling is a time machine only for the period you retained.

Sampling statistics: how much data is enough

A flame graph is a statistical estimate. If a function truly accounts for fraction p of CPU and you collect n samples, the standard error of the estimated share is about the square root of p(1-p)/n. At 19 Hz, one 4-core container over 10 minutes yields 19 x 4 x 600 = 45,600 samples. For p = 3 percent the standard error is sqrt(0.03 x 0.97 / 45,600), about 0.08 percentage points. Narrow. But for a function at p = 0.3 percent seen only in one pod for one minute (4,560 samples), the error is roughly 0.08 percentage points on a 0.3 base, a 27 percent relative error. Aggregation across replicas fixes this, which is why fleet merges are more reliable than single-process views for small hot spots. (Assumes independent samples, which is an approximation; correlated bursts violate it.)

Figure 4: Rollout decision flow. Start with node-level eBPF for coverage, add SDK profiles only for services where runtime-level detail pays back.

Trade-offs, Gotchas, and What Goes Wrong

Missing symbols. Stripped binaries produce hex addresses in flame graphs. Alloy’s component needs writable storage for a symbol cache, and for native code it relies on debug information being available; ship debuginfo to a symbol server or keep unstripped copies in your build artifacts. Go binaries carry their own symbol tables, which is why Go often works out of the box.

Shallow or broken stacks. The classic failure: C or C++ services compiled with -fomit-frame-pointer appear to have two-frame stacks. Fix the build flags or use a DWARF-capable unwinder. Alloy’s docs call this out explicitly. Treat a spike of [unknown] frames after a base-image update as a canary that the toolchain changed.

On-CPU only misleads. CPU sampling shows where cycles go, not where requests wait. A service blocked on a database lock may show a flat profile while latency soars. Use wall-clock or off-CPU profiling for tail latency, and pair profiles with traces for the waiting story.

Sampling blind spots. At 19 Hz, a 5 ms burst that repeats every second is seen about 9.5 percent of the time per CPU on average if it occupies the CPU for that window; a very short event repeated rarely may never appear. Do not use sampling profiles as proof of absence.

Label cardinality explosions. Labelling by pod, container ID or request ID multiplies series. Profile stores index labels like any time-series system, so the same discipline applies as in metrics.

Upgrade coupling. Pyroscope 2.0 requires object storage for distributed self-managed installs. If your platform is air-gapped without S3-compatible storage, plan the dependency before upgrading from v1.

Kernel and runtime churn. eBPF programs depend on kernel features; Parca documents a 4.18 minimum. Interpreter unwinders depend on runtime internals. Pin and test the profiler version alongside kernel and runtime upgrades, and watch the stack depth histogram after every change.

Security surface. A privileged DaemonSet that reads other processes’ memory structures is powerful. Restrict who can query profiles too: stacks and symbol names can leak internal service names and, with heap or argument-like labels, sensitive details.

Overhead is not zero. Even if the sampling math is tiny, symbolisation, upload bursts and memory for unwind tables are real. Set resource limits on the agent and alert on it like any workload.

Practical Recommendations

Start with the question you need answered. If it is “what is burning CPU across my fleet and what changed in the last deploy”, start with a node-level eBPF profiler, label with service and version, and send to one backend. If you already run Grafana, Pyroscope 2.0 with Alloy’s pyroscope.ebpf is the lowest-friction path; if you want a self-contained single-binary start or are not on Grafana, evaluate Parca. Because the collection layer is converging on the OpenTelemetry profiler and OTLP, keep the agent configuration portable and avoid hard-coding vendor-specific labels.

Roll out in stages. Pick one non-critical cluster, deploy at the default 19 Hz, and run for a week while you compare node CPU and memory against a control group. Check stack depth and the share of unknown frames. Then expand to production, with resource limits on the agent. Add SDK or pprof profiles only for services where allocation or contention profiles justify per-service work.

Make profiles part of the release process. Store the git SHA as a label, and add a link to the diff view in your deploy notification. The 2026 InfoQ report cites Monzo presenting this pattern, using Pyroscope to catch performance regressions during deployments.

Checklist:

• Confirm kernel version (4.18 or later for Parca) and the container runtime on every node pool.
• Verify frame pointers or debug info for your top five CPU-consuming binaries.
• Define a label schema: service, version, region, node pool; keep pod short-lived.
• Provide object storage if you self-host Pyroscope 2.0 distributed mode.
• Set retention (for example 14 days full, longer for aggregates) and estimate storage using your own profile sizes.
• Review the privileged capability set with security before rollout.
• Add a stack-depth and unknown-frame alert.
• Wire the diff view into deploy pipelines.

Frequently Asked Questions

What is continuous profiling?

Continuous profiling samples the call stacks of running production processes at a low, fixed frequency and stores them centrally with labels such as service and version. You can then merge samples into flame graphs, compare releases, and look back at past incidents. It complements metrics, logs and traces by showing which code consumes CPU or memory, not only which service is slow.

Is Parca or Pyroscope better?

Neither wins universally. Parca emphasises eBPF whole-host profiling without code changes and can run as a single binary. Pyroscope offers SDK and pprof scraping for runtime-specific profiles, plus eBPF through Grafana Alloy, and its 2.0 release moved storage to write-once object storage. Choose Pyroscope if you already use Grafana; evaluate Parca for no-instrumentation, whole-host coverage.

How much overhead does an eBPF profiler add?

Sampling cost is small because stack walks run in the kernel and only aggregated counts reach user space. At 19 samples per second per CPU, my illustrative arithmetic puts sampling well under one percent of a node. Vendors describe overhead as very low but publish few hard numbers, so measure your own nodes over a full week before committing.

Why are my flame graphs full of unknown frames?

Usually missing symbols or broken unwinding. Binaries built without frame pointers produce shallow stacks unless the profiler can use DWARF-derived tables, and stripped binaries lack function names. Rebuild with -fno-omit-frame-pointer, ship debug info to a symbol store, and confirm the interpreter version is supported by your profiler’s runtime unwinder.

Does OpenTelemetry support profiling?

Yes, as an emerging signal. OpenTelemetry announced profiling support in 2024, with Elastic pledging its eBPF profiling agent, and reporting says the Profiles signal has reached alpha status. Pyroscope 2.0 has native OTLP profiling support. Because the spec is still maturing, expect field and format changes, and pin versions in production.

Can continuous profiling replace tracing?

No. Traces show the path and latency of a request across services; profiles show where code spends resources inside a process. Together they let you move from a slow span to the function responsible. Profiles also miss waiting time unless you collect wall-clock or off-CPU data, a gap traces fill well.

Further Reading

• XDP and eBPF Layer 4 load balancing with Katran, Cilium and Maglev: the same eBPF safety and verifier constraints apply to profilers.
• W3C Trace Context and Baggage for distributed tracing: join profiles to traces.
• Prometheus long-term storage: Thanos vs Mimir vs VictoriaMetrics: label and retention trade-offs mirror profile stores.
• Model predictive control and digital twin optimization: a case of tight control loops where profiling the solver’s hot path matters.
• External: Grafana Alloy pyroscope.ebpf reference, Parca Agent documentation, OpenTelemetry profiling announcement, and InfoQ on Pyroscope 2.0.

By Riju — about