Bazel Remote Caching and Monorepo Build Systems: Hermetic Builds and Remote Execution (2026)

A monorepo build gets slow in a predictable way. Nobody makes a bad decision. The repository grows, every pull request touches a few files, and yet the pipeline recompiles, relinks and retests far more than the change justifies. Engineers respond by sharding CI, buying bigger runners and adding timestamp heuristics to scripts, and the bill climbs while the feedback loop stretches from minutes to the better part of an hour.

Bazel remote caching attacks the root cause rather than the symptom. If every build step is a pure function of declared inputs, then a result computed once by anyone, on any machine, can be reused by everyone. That single idea, enforced by hermetic builds and backed by a content-addressed store, is what lets Bazel scale to repositories that defeat conventional tools. Remote execution then pushes the same logic one step further by running the actions on a shared cluster.

This post is a mechanism-level guide for engineers who already run CI at scale. You will leave with a precise model of how the action cache and content-addressable store work, a way to reason about hit rates with worked arithmetic, an honest map of the alternatives, and a migration plan with the failure modes that hurt in practice.

What this covers: why monorepo builds slow down, the Bazel execution model, hermeticity and sandboxing, the Remote Execution API, hit-rate math, the 2026 version landscape (Bzlmod and Bazel 9), alternatives such as Buck2, Pants, Nx, Gradle and Turborepo, CI integration, migration cost, and failure modes.

Context and Background

Large repositories defeat naive build tools for three compounding reasons. First, dependency graphs are wide: a change to a low-level library invalidates thousands of downstream artifacts. Second, most tools decide staleness with file timestamps or coarse heuristics, which forces either over-building (safe, slow) or under-building (fast, wrong). Third, build state is local. A laptop, a CI runner and a colleague’s workstation each rebuild the same artifacts from scratch because nothing connects them.

The industry answer has two halves. The first half is a build system that knows the true dependency graph, so it rebuilds only what a change can affect. The second half is a shared cache so that work done once is never repeated. Google’s internal system, Blaze, pioneered both, and Bazel is its open-source descendant. Meta open-sourced Buck and later Buck2, and Twitter and Foursquare produced Pants. Each took the same core stance: explicit dependencies, deterministic actions, and a cache keyed by content.

It helps to be precise about what a build system is optimising. In a single-repo monorepo, the quantity that matters is the fraction of actions you do not execute. Suppose a pull request changes two files in a repository with 40,000 actions. A correct incremental build might execute a few hundred of them. A cold CI runner, however, executes all 40,000 unless an external cache supplies the other 39,000-odd results. That is why the remote cache, not the local incremental graph, dominates CI economics.

The tooling landscape in 2026 is mature. Bazel 9.0 shipped on 20 January 2026 as a long-term support (LTS) release, and according to the official Bazel 9 announcement it removes the code behind the legacy WORKSPACE system, leaving Bzlmod as the way to manage external dependencies. The wider ecosystem includes managed remote cache and execution vendors, self-hosted servers, and several credible alternatives. We cover all of them below.

This post connects to several others on the site. Hermetic, reproducible builds are the foundation for provenance attestations, which we examine in SLSA and Sigstore software supply chain security architecture. Cache design itself has a rich failure literature, and the stampede problem we discuss later has a direct analogue in cache stampede and thundering herd request coalescing. For the deployment side of the same discipline, see Argo CD versus Flux as a GitOps decision record.

Why monorepo builds actually get slow

Four mechanisms account for most of the slowdown, and they are worth separating because each has a different fix.

Graph fan-out. In a monorepo, a shared utility library might have ten thousand transitive dependents. Touching it legitimately invalidates all of them, and no tool can avoid that. What a good tool can do is notice when the invalidated target produces byte-identical output, so that dependents can stop early. Bazel does this implicitly: if a recompiled library emits the same bytes, the downstream action keys are unchanged and hit the cache.

Coarse change detection. Tools that detect change by timestamp rebuild after a git checkout even when file contents match. Tools that detect change by content hash do not. Content addressing converts “did anything change” from a guess into a lookup.

Environment leakage. If a compile step quietly reads /usr/include, an environment variable, or the current date, then two machines produce different outputs from identical source. The result is unusable as a shared cache entry. This is the hermeticity problem and it is the largest source of pain in real migrations.

Serialised critical paths. Even with perfect caching, a graph has a longest chain of dependent actions. Remote execution attacks this by running the many independent branches in parallel across hundreds of cores, so wall-clock time approaches the critical path rather than the sum of all work.

The Bazel Model: Targets, Actions, and a Content-Addressed Cache

Bazel remote caching works because a build is modelled as a directed acyclic graph of actions, each with fully declared inputs and outputs. Bazel computes a digest of an action’s command line, environment variables and input file contents. If that digest exists in the shared action cache, Bazel downloads the recorded outputs from the content-addressable store instead of executing the action.

Figure 1: Reference architecture for Bazel remote caching and remote execution. Clients check the action cache by digest, fetch outputs from the CAS on a hit, and either execute locally or send the action to workers on a miss.

Figure 1 shows the three roles. The Bazel client analyses the build and produces actions. A cache service answers two kinds of question: “do you have a result for this action digest” (the action cache) and “do you have a blob with this content digest” (the content-addressable store, or CAS). On a miss, the action runs either locally or on a remote worker, and its outputs are uploaded to the CAS with the result recorded in the action cache.

Workspace, packages, targets and actions

A Bazel repository is divided into packages, each a directory containing a BUILD or BUILD.bazel file. A package declares targets: rule instances such as cc_library, java_test or go_binary, each with a label like //services/auth:server. Labels are the stable names of the dependency graph.

Bazel works in phases. The loading phase evaluates BUILD files (written in Starlark, a deterministic Python-like language) to create targets. The analysis phase applies rule implementations to produce the action graph: a set of concrete commands, each listing its input artifacts and output artifacts. The execution phase walks that graph, consulting caches and running whatever is missing.

The separation matters. Because analysis is deterministic and side-effect free, Bazel can cache its results too, and the action graph is a precise statement of what depends on what. You cannot forget a dependency and have it silently work, because an undeclared input is simply not present in the sandbox.

The action key and what goes into it

According to the official Bazel remote caching documentation, each action declares its inputs, output names, command line and environment variables, and only variables passed through --action_env are included in the key. Conceptually the key is a digest over four things:

The command line (executable path and arguments).
The environment variables visible to the action.
The digests of every input file, including tool binaries.
The declared output paths and platform properties used to select an executor.

Change any of them and the digest changes, so the lookup misses. Keep them all identical and the lookup hits regardless of who computed the original. That is the whole contract, and almost every cache-related bug is a violation of it: something influenced the output but was not part of the key (a stale hit that is wrong), or something unrelated was part of the key (a miss that should have been a hit).

Action cache versus content-addressable store

The two stores solve different problems. The action cache (AC) maps an action digest to an ActionResult message: a small record listing output file digests, exit code, and references to captured stdout and stderr. The CAS maps a content digest to bytes. Over the HTTP protocol Bazel supports, the documentation describes these as the /ac/ and /cas/ path namespaces.

This separation yields two useful properties. Deduplication is automatic: two different actions that produce the same file store it once. And an action-cache entry is only meaningful if the CAS still holds every blob it references. If the CAS evicts a blob that an AC entry points to, the client must treat the entry as a miss and re-run. Production servers therefore tie eviction policy to reference integrity, or they accept that some AC entries become dangling and are lazily discarded.

Bazel also caches stdout and stderr for each action, a detail the documentation highlights, which means log output is not a reliable signal for estimating hit rates. Use the build event stream and execution summaries instead.

Bzlmod, WORKSPACE, and the Bazel 7, 8 and 9 arc

External dependencies used to be declared in a WORKSPACE file with imperative repository rules. That system suffered from order-dependent evaluation and no real version resolution. Bzlmod replaced it with a MODULE.bazel file per module, a minimal version selection algorithm similar in spirit to Go modules, and a central index called the Bazel Central Registry.

The timeline, as stated in the official sources I fetched: the Bazel roadmap notes that Bzlmod has been the standard since Bazel 7. The Bazel 9 announcement says Bazel 8.0 disabled the legacy WORKSPACE system by default and Bazel 9.0 removes the code that supports it. Bazel 8 also moved most built-in rules into separate modules, and Bazel 9 finishes that effort by moving the built-in C++ rules into rules_cc. The flag --incompatible_autoload_externally now defaults to empty, so you must explicitly load rule sets from external modules; the announcement says the flag will be removed in Bazel 10.0.

For this topic the practical consequence is twofold. First, a 2026 migration starts on Bzlmod. Do not invest in new WORKSPACE code. Second, dependency resolution is now part of the reproducibility story: a MODULE.bazel.lock file pins resolved versions so two machines resolve the same module graph. I could not verify from official pages whether any specific later point release (for example a 9.1 line) changes remote cache behaviour, so treat anything newer than 9.0 as something to check against release notes before adopting.

A runnable minimal example

The snippets below are a complete tiny module. They are illustrative scaffolding, and version numbers in bazel_dep lines should be checked against the Bazel Central Registry before use.

# MODULE.bazel
module(name = "acme_monorepo", version = "0.0.0")

bazel_dep(name = "rules_cc", version = "0.1.1")      # verify current version in the BCR
bazel_dep(name = "rules_go", version = "0.50.1")     # verify current version in the BCR
bazel_dep(name = "googletest", version = "1.15.2")   # verify current version in the BCR

# libs/hash/BUILD.bazel
load("@rules_cc//cc:defs.bzl", "cc_library", "cc_test")

cc_library(
    name = "hash",
    srcs = ["hash.cc"],
    hdrs = ["hash.h"],
    visibility = ["//visibility:public"],
)

cc_test(
    name = "hash_test",
    srcs = ["hash_test.cc"],
    deps = [":hash", "@googletest//:gtest_main"],
    size = "small",
)

# .bazelrc
common --enable_bzlmod
build --remote_cache=grpcs://cache.example.internal
build --remote_upload_local_results=false   # CI-only writers, see the poisoning section
build --remote_timeout=60
build --disk_cache=~/.cache/bazel-disk
build --experimental_disk_cache_gc_max_size=50G

The last line uses a flag the documentation says is available from Bazel 7.4 onward. A local --disk_cache is worth keeping even with a remote cache because it absorbs repeat work on a developer machine without a network round trip.

Hermetic Builds, Sandboxing and the Remote Execution API

Hermetic builds are builds whose outputs depend only on declared inputs: source files, tools, and explicitly passed flags and environment. A hermetic action produces the same bytes on any machine, which is the precondition for safely sharing cache entries. Without hermeticity, remote caching trades speed for silent incorrectness, because a cache hit may return an output that the local environment would not have produced.

What hermetic actually means in practice

Hermeticity is a spectrum, and it helps to name the levels. At the lowest level, the build declares source dependencies but still reads the host compiler and system headers. It is fast to adopt and gives correct incremental builds on one machine, but cache sharing across machines with different toolchains is unsafe. At the next level, toolchains are fetched and pinned: the C++ compiler, the JDK, the Go SDK and the Python interpreter all arrive as versioned, hashed downloads, so every machine runs the same tool binaries. At the highest level, actions execute in an isolated environment with no network, a controlled filesystem view, a fixed time and locale, and no inherited environment.

Most teams should aim for the middle level first and the top level where it pays off. Pinning toolchains eliminates the biggest class of cross-machine drift. Isolation then catches the remaining leaks, such as a script that reads $HOME, a test that opens a network socket, or a code generator that embeds a timestamp.

Sandboxing: how Bazel catches undeclared inputs

Bazel enforces declared inputs by executing each action in a sandbox directory that contains only the declared input files, typically as symlinks or hardlinks into the real source tree. On Linux it can additionally use namespaces to isolate the process view of the filesystem, network and process table. If a command touches a file it did not declare, the file is simply absent and the action fails with a clear error. That converts a latent correctness bug into an immediate build failure on the first run, which is far cheaper to debug than a wrong cache hit six weeks later.

Figure 2: The hermetic action boundary. Only declared sources, pinned tools and allowed environment variables enter the sandbox; the output digest becomes the cache value.

Sandboxing has a cost. Creating a sandbox with thousands of symlinks per action adds overhead, which is noticeable for very small actions and on filesystems with slow metadata operations. Bazel offers several strategies, from a lightweight process wrapper to stricter namespace-based isolation, and teams commonly choose stricter sandboxing in CI and a lighter one for local iteration. I am describing this from general knowledge of Bazel’s strategies; confirm the exact strategy names and defaults for your Bazel and OS combination in the official sandboxing documentation before standardising flags.

Determinism beyond the sandbox

A sandbox guards inputs. It does not make a tool deterministic. Compilers that embed absolute paths, archives that record file modification times, and code generators that iterate hash maps in random order all produce different bytes from identical input. Two different outputs for the same action key are not only a cache-efficiency problem. They break downstream cache hits, because consumers see a new input digest, and they undermine reproducible-build verification for supply chain provenance.

The standard mitigations are well known. Use -ffile-prefix-map style flags (or the equivalent for your compiler) to strip absolute paths. Normalise timestamps to a fixed value when creating archives. Sort any collection a tool serialises. Bazel’s own rules already apply many of these for built-in languages; custom rules and shell-based genrule actions are where drift enters.

The Remote Execution API

Bazel’s remote cache and remote execution share one wire protocol: the Remote Execution API (REAPI), defined in the open bazelbuild/remote-apis repository as gRPC services over protocol buffers. The documentation page I fetched describes the protocol definition (remote_execution.proto) and lists server implementations including Buildbarn, Buildfarm, BuildGrid and NativeLink, and notes that bazel-remote supports the gRPC API.

From my understanding of the API, which I did not re-verify line by line for this post, the main services are:

ActionCache: GetActionResult and UpdateActionResult, keyed by an action digest.
ContentAddressableStorage: batch check-for-missing-blobs, batch read, and batch update calls, keyed by content digest (a hash plus size).
ByteStream: streaming upload and download of large blobs.
Execution: Execute takes an action digest and returns a long-running operation, and workers perform the work.
Capabilities: lets the client discover supported digest functions and API versions.

A digest is the pair of hash and byte size, and the hash function is negotiated through the capabilities service. Because everything is addressed by digest, uploading is naturally incremental: the client asks which blobs are missing and sends only those.

How a remote-cached build proceeds

Figure 3: Cache hit versus miss. On a hit the client downloads outputs by digest; on a miss an executor runs the action and publishes outputs and the action result.

Follow one action through Figure 3. The client computes the action digest and calls GetActionResult. On a hit, the response lists output digests and the client fetches them from the CAS, or, with a feature commonly called “build without the bytes”, skips downloading intermediate outputs it does not need locally. On a miss, the client uploads missing inputs to the CAS, then either runs the command locally or sends an Execute request. The worker materialises the input tree from the CAS, runs the command, uploads outputs, and records the action result.

A subtle but important consequence: with remote execution, inputs travel as a Merkle tree of directories keyed by digest. Workers keep a local blob cache, so a second action sharing most of its input tree uploads and downloads almost nothing. This is why remote execution scales: data movement is proportional to what is new, not to what exists.

Remote caching versus remote execution

The two features are separable and most teams should adopt them in order. Remote caching only needs a storage service, is cheap to run, and removes duplicated work. Remote execution needs a scheduler and a worker fleet, introduces new failure modes, and is justified when the critical path or CPU demand outgrows what CI runners provide.

Capability	Remote cache only	Remote cache plus remote execution
Infrastructure	Storage service (HTTP or gRPC)	Scheduler, workers, storage, autoscaling
Speeds up	Repeated work across machines	Repeated work and first-time parallel work
Hermeticity required	Strongly recommended	Mandatory, workers share no local state
Failure blast radius	Slow builds if cache is down	Builds fail if the cluster is unavailable, unless local fallback is enabled
Typical cost driver	Storage and egress	Worker compute and idle capacity
Best first step for	Teams under about a few hundred engineers	Large C++ and Java codebases with long critical paths

The rough team-size guidance in the last row is my own heuristic, not a measured threshold. Evaluate by looking at your longest action chain and your CI queue times.

Cache Hit-Rate Math: A Worked Example

Hit rate is the headline metric, but the arithmetic is more nuanced than “higher is better”. The useful model is that build time is a mixture of cached and executed work, plus a fixed overhead for cache lookups and transfers. Let me build it step by step with explicitly illustrative numbers.

Assume a CI build graph with 40,000 actions. Suppose 30,000 are cheap (average 0.5 seconds), 9,500 are medium (average 8 seconds) and 500 are heavy (average 120 seconds: large links, test suites, code generation). Total sequential work is:

Cheap: 30,000 × 0.5 s = 15,000 s
Medium: 9,500 × 8 s = 76,000 s
Heavy: 500 × 120 s = 60,000 s
Total: 151,000 CPU-seconds, about 42 CPU-hours.

On a 16-core runner with perfect parallelism that is about 9,440 seconds, around 2.6 hours. Real graphs have dependencies, so it is worse. Now introduce a cache with a count hit rate of 95%. The naive reading says only 5% of work remains. But hit rate by count and hit rate by cost differ. Suppose the misses are concentrated: a typical pull request invalidates a few modules, and misses are biased towards heavy actions because heavy actions sit high in the graph, downstream of the change.

If the 2,000 missed actions are 1,500 cheap, 450 medium and 50 heavy, the executed cost is 750 + 3,600 + 6,000 = 10,350 CPU-seconds. That is 6.9% of the original work, close to the count-based 5%, so here the naive estimate roughly holds. Now change the distribution so that only 500 actions miss (a 1.25% miss rate by count), but they are 100 cheap, 300 medium and 100 heavy; the cost becomes 50 + 2,400 + 12,000 = 14,450 CPU-seconds, 9.6% of the work, despite the far lower miss rate by count. The lesson is to track cost-weighted hit rate (saved execution seconds divided by total execution seconds), not only the count ratio.

Adding transfer cost

Hits are not free. Each hit requires a lookup and, unless the build skips downloading intermediate outputs, a download. If the average cached output is 200 KB and the network moves 100 MB per second to a runner, downloading 38,000 outputs is about 7.6 GB, or 76 seconds of pure transfer, plus per-request latency. With 5 ms of round-trip overhead per request at a parallelism of 100, lookups add roughly 38,000 × 5 ms / 100 = 1.9 seconds. These figures are illustrative arithmetic, not measurements. The point: cache transfer is small next to compilation but not negligible, and it grows with action count and with fat artifacts such as container layers or large test fixtures.

Where hit rate is lost

Hit rate leaks in recognisable ways, and each has a different fix:

Volatile inputs: a generated version.h containing a git SHA or timestamp invalidates everything that includes it. Fix by isolating volatile data in a “stamping” mechanism that only affects the final linked artifact.
Non-portable action environments: --action_env=PATH or developer-specific variables put machine-specific data into every key.
Toolchain drift: unpinned compilers make every runner a separate cache partition.
Absolute paths in outputs: they change the output digest per checkout directory and cascade to dependents.
Wide inputs: a target that globs a huge directory depends on every file in it, so any change misses.

A good workflow is to run two identical builds on different machines and compare execution logs to find actions whose keys or outputs differ. Bazel provides an execution log and an explain-style facility for this; consult the current command-line reference for exact flags, as these have changed across releases.

Figure 4: A diagnosis loop for raising cost-weighted hit rate: build twice on different machines, diff action keys and outputs, fix the leak, and repeat.

Ecosystem: Rules, Cache Backends, Alternatives and CI Integration

A build system is only as useful as its rules and its infrastructure. Bazel’s strength is a broad rules ecosystem and a standard protocol; its weakness is the effort needed to adopt it. This section maps the terrain and is deliberately explicit about what I verified and what I am describing from general knowledge.

Rules ecosystem

Language support lives in rule sets such as rules_cc, rules_java, rules_python, rules_go, rules_rust, rules_jvm_external, rules_oci and rules_proto-family modules, distributed through the Bazel Central Registry. The Bazel 9 announcement confirms the direction of travel: built-in rules moved out of the core into separate modules, with C++ the last major one to move in 9.0. The benefit is that rule sets can ship fixes independently of Bazel releases. The cost is that upgrading becomes a matrix problem: Bazel version, rule set versions and toolchain versions must be compatible. Pin them in MODULE.bazel, commit the lockfile, and upgrade deliberately.

Many teams adopt Bazel for some languages and keep native tools for others. A common pragmatic pattern uses Bazel for C++, Go, Java and protobuf code generation where the payoff from hermetic caching is highest, and wraps frontend package managers such as npm or pnpm with rules that treat node_modules as a pinned external repository.

Remote cache and execution backends

The official documentation lists several cache options: nginx with the WebDAV module, bazel-remote (a self-hosted server with disk storage and garbage collection that supports both REST and gRPC, though the Bazel project does not provide technical support for it), Google Cloud Storage, and any HTTP/1.1 server supporting GET and PUT, with Amazon S3 among the examples the page mentions. For remote execution, the documentation names Buildbarn, Buildfarm, BuildGrid and NativeLink as related projects.

Commercial services such as BuildBuddy and EngFlow also provide managed cache, execution, and build analytics. I am naming them from general industry knowledge and from search results that surfaced BuildBuddy’s own technical explainer; I did not verify current pricing, SLAs or feature sets, so evaluate them directly.

Option	Type	Protocol	Operational burden	Notes
nginx WebDAV	Self-hosted cache	HTTP	Low	Auth must be added separately per the docs
bazel-remote	Self-hosted cache	HTTP and gRPC	Low to medium	Disk-backed with garbage collection; no official Bazel support
GCS or S3 bucket	Managed object store	HTTP	Low	Egress and request costs matter; latency higher than a local disk server
Buildbarn, Buildfarm, NativeLink, BuildGrid	Self-hosted cache and execution	gRPC (REAPI)	High	You run schedulers, workers, autoscaling
Managed vendors	Hosted cache and execution	gRPC (REAPI)	Low	Verify pricing, data residency and compliance

For cache-only deployments, a regional bucket fronted by a cache proxy is a reasonable starting point. Latency to the cache matters more than people expect, because every action performs at least one lookup. Co-locating the cache with CI runners in the same region or availability zone usually beats a cheaper but distant store.

Alternatives and where each fits

Buck2 is Meta’s Rust-based successor to Buck. Its documentation includes a remote execution guide, and it targets the same REAPI-style model. It is attractive if you want a fast, single-language-core system, but its public rule ecosystem is smaller than Bazel’s. I am describing this from general knowledge and the surfaced documentation title; check the current state before betting a migration on it.

Pants focuses on Python, Go, Java and Scala monorepos with a gentler adoption path: it infers many dependencies from imports rather than requiring hand-written BUILD dependencies. It supports remote caching and execution in the same ecosystem of protocols. The trade-off is a narrower language reach and less ecosystem gravity than Bazel.

Nx and Turborepo target JavaScript and TypeScript monorepos. They cache task outputs by hashing task inputs, which gives much of the benefit for npm-style workspaces with modest setup. They are not hermetic in Bazel’s sense, since tasks run in your normal environment, so cache correctness depends on discipline and on declaring inputs well. They fit teams whose repository is overwhelmingly JavaScript. I did not re-verify their current remote-cache hosting options for this post.

Gradle has a build cache with local and remote tiers for JVM and Android builds, keyed by task input hashes. Gradle caches at task granularity, whereas Bazel caches at the finer action granularity and sandboxes inputs. This frames the common Bazel vs Gradle comparison:

Dimension	Bazel	Gradle
Cache granularity	Per action, content-addressed	Per task, input-hashed
Hermeticity	Enforced by sandbox and declared inputs	Convention and plugin discipline
Multi-language	First-class, one graph	Strongest for JVM and Android
Learning curve	Steep, explicit BUILD files	Gentler, richer IDE integration
Remote execution	Native via REAPI	Not a core feature, per my understanding
Typical fit	Polyglot or very large repos	JVM-centred teams, Android apps

The honest decision rule: if your repository is mostly JVM and your cache hit rate is already high with Gradle, the migration cost to Bazel may not pay back. If you are polyglot, need remote execution, or require provable hermeticity for compliance, Bazel earns its complexity.

CI integration patterns

The most reliable CI pattern separates cache readers from cache writers. Pull request builds read the remote cache but do not write it (--remote_upload_local_results=false, which the documentation describes as making the build read-only against the cache). Trusted builds on the main branch, running in a controlled environment, write results. This keeps untrusted code from poisoning shared entries, and it keeps writes predictable.

Other practical points:

Use --bes_backend style build event streaming to a viewer so engineers can see which actions missed and why, rather than parsing logs.
Warm the cache on main so that pull requests start from a hot cache; a build triggered at merge time pays the cost once for everyone.
Exclude genuinely non-cacheable targets, such as tests that talk to external services, with the no-remote-cache tag the documentation describes.
Treat the cache like a service: set SLOs for lookup latency and availability, and configure Bazel to fall back to local execution if the cache is slow, using the remote timeout setting.
Run bazel test with explicit sharding and timeouts so that heavy suites parallelise on remote workers.

Reproducible builds and supply chain provenance

Hermeticity is also the cheapest route to verifiable builds. If two independent builders produce identical output digests from the same source and lockfile, you have strong evidence that the artifact corresponds to the source. That evidence is the substance behind provenance attestations: SLSA levels require a hardened, scripted build with generated provenance, and Bazel’s explicit inputs make provenance generation straightforward because the action graph is a machine-readable bill of materials. See our companion post on SLSA, Sigstore and software supply chain security architecture for how attestations and signing fit on top, and the SLSA specification for the official level definitions.

The caution runs the other way too. A shared remote cache is a supply chain surface: anyone who can write a malicious blob under a legitimate action key can influence every later build that hits it. We return to this in the failure modes.

Migration Strategy and Cost

Migrating a mature repository to Bazel is a multi-quarter programme, not a sprint. The teams that succeed treat it as incremental adoption with measurable milestones, not a big-bang rewrite. A sequencing that works for many organisations follows.

Phase 0: Measure. Before touching anything, record baseline CI duration, queue time, cost per build, and developer wait time at p50 and p90. You cannot defend the project without a before-and-after comparison, and these numbers decide whether the investment is justified at all.

Phase 1: Pilot a slice. Pick a self-contained, high-churn, slow-to-build subtree, such as a core library plus its tests, and build it with Bazel alongside the existing system. Prefer a language with strong rule support. Success criterion: the Bazel build is correct and a second machine achieves a high cost-weighted hit rate.

Phase 2: Pin toolchains and enforce hermeticity. Fetch compilers and interpreters through Bazel, enable sandboxing, and fix the leaks it exposes. This is where most of the effort goes, and it also benefits developers independent of Bazel because it ends “works on my machine”.

Phase 3: Turn on the remote cache for CI. Start with read-only for pull requests and writes from main. Monitor miss reasons and fix the top leaks using the diagnosis loop in Figure 4.

Phase 4: Expand coverage and retire the old build. Move subtree by subtree, keep both build systems green for a transition window, and make the Bazel path the required check before deleting the legacy one.

Phase 5: Consider remote execution. Adopt it only if the critical path or CPU demand justifies running a cluster. Many teams stop at phase 3 and are satisfied.

What it costs

Cost has three buckets. People: expect a small dedicated build team, often two to four engineers for a mid-sized organisation, during migration, tapering to a smaller steady-state team; this is a rule of thumb, not a measured figure. Infrastructure: cache storage plus egress, and for remote execution a worker fleet whose idle capacity is the dominant waste. Developer friction: BUILD file maintenance, a new mental model, and slower onboarding. Tools like Gazelle (for Go and others) generate BUILD files from source to reduce the maintenance load, and the Bazel team has released a Bzlmod migration helper according to the Bazel 9 announcement, which says the tooling will not complete a migration automatically.

The payback comes from three effects: lower CI compute because most work is cached, lower developer wait time, and fewer flaky or irreproducible builds. Quantify them with your Phase 0 baseline. If the numbers do not justify the programme, a lighter tool may be the right answer.

Trade-offs, Gotchas, and What Goes Wrong

Bazel remote caching fails in a small number of well-understood ways. Knowing them in advance is the difference between a smooth adoption and a loss of trust in the build.

Non-hermetic actions produce wrong hits. This is the worst failure, because it is silent. A genrule that reads an undeclared file works locally, populates the cache, and then returns a stale or machine-specific result elsewhere. Defences: sandbox everywhere, run periodic clean builds on a different machine and compare outputs, and review custom rules and shell commands as carefully as security-sensitive code.

Cache poisoning. The documentation notes that outputs are stored only as names and hashes, so you cannot tell which build produced a given output, and recommends clearing the entire cache and starting clean if it is poisoned. An attacker or a bad build agent with write access can insert a malicious blob under a legitimate action digest. Mitigate with write access limited to trusted CI, authenticated and authorised cache endpoints, network isolation, and the read-only default for developer machines and pull request builds. Remote execution adds a second concern: workers must not share state between tenants.

Flaky tests under caching. A passing test result is cacheable, so a flaky test that passed once can be “reused” and hide real instability, while a flaky failure wastes reruns. Use --flaky_test_attempts deliberately, tag known flakes, and track them. Caching also makes tests that depend on time, randomness or ordering look better than they are; make seeds explicit.

Cache stampedes on cold starts. When a popular dependency changes, hundreds of CI jobs miss simultaneously and all attempt to build and upload the same artifact. The same structural problem appears in every cache design, and the remedies, such as request coalescing and staggered warming, are discussed in our post on cache stampede and thundering herd patterns. Remote execution services often deduplicate identical in-flight actions, which mitigates this at the executor.

Dangling and evicted entries. If the CAS evicts blobs referenced by action cache entries, hits degrade into misses plus wasted lookups. Size storage for the working set of the last N weeks of main-branch builds, not for a single build.

Network and tail latency. A distant or overloaded cache can make builds slower than no cache. Set sensible timeouts, prefer regional co-location, and measure p99 lookup latency.

Migration fatigue and bus factor. If only one person understands the rules, the build becomes a bottleneck. Invest in documentation, shared macros, and guardrails like linters for BUILD files.

Anti-pattern: stamping everywhere. Embedding commit SHAs or timestamps into every artifact defeats caching. Stamp only at the final packaging step.

Practical Recommendations

Start with measurement and a narrow pilot, then add infrastructure only as the numbers demand it. Remote caching delivers most of the value at a fraction of the operational cost of remote execution, so treat execution as a later optimisation rather than a launch requirement.

Adopt Bazel on Bzlmod from day one. Per the official announcement, WORKSPACE support is removed in Bazel 9, so any investment in it is debt. Pin Bazel with a version file, commit MODULE.bazel.lock, and upgrade on a schedule that tracks the LTS line. Choose your cache backend by operational capacity: a managed service if you lack platform engineers, a self-hosted bazel-remote or object store if you have them and need control over data location.

Make cache writes a privilege. Pull requests and laptops read, trusted main-branch CI writes. Monitor cost-weighted hit rate, lookup latency and the top miss reasons weekly. Finally, decide against Bazel when it does not fit: a small JavaScript repository will be better served by Nx or Turborepo, and a JVM-only organisation with healthy Gradle caching may not recover the migration cost.

A short checklist you can paste into a planning document:

[ ] Baseline CI duration, queue time and cost recorded at p50 and p90.
[ ] Pilot subtree builds correctly in Bazel under sandboxing.
[ ] Toolchains pinned and fetched by Bazel; no host compilers.
[ ] Bzlmod only; lockfile committed; Bazel version pinned.
[ ] Remote cache co-located with CI; read-only for pull requests.
[ ] Cost-weighted hit rate dashboard and miss-reason review in place.
[ ] Poisoning response runbook: how to purge and rebuild the cache.
[ ] Decision gate for remote execution tied to critical-path data.

Frequently Asked Questions

What is Bazel remote caching?

Bazel remote caching is a mechanism that stores the results of build actions in a shared service so any machine can reuse them. Bazel hashes each action’s command, environment and inputs, looks the hash up in an action cache, and downloads the recorded outputs from a content-addressable store on a hit. Because the key reflects every declared input, a result computed once on one machine is safe to reuse elsewhere, provided the build is hermetic.

What is the difference between remote caching and remote execution in Bazel?

Remote caching reuses results that already exist; it needs only a storage service. Remote execution goes further and runs missing actions on a shared worker cluster, so first-time work also parallelises beyond the capacity of one machine. Both use the same Remote Execution API and the same digests. Caching is cheap and low risk to adopt; execution adds a scheduler, workers and new failure modes, and is justified for large builds with long critical paths.

Do I need hermetic builds to use a remote cache?

You can enable a remote cache without perfect hermeticity, but you risk wrong hits: an action that reads an undeclared input can store a result that is invalid on another machine. For safe sharing, pin toolchains, run actions in a sandbox, and control environment variables. Remote execution makes hermeticity effectively mandatory because workers share no local state with the developer’s machine, so missing declarations fail immediately.

Is Bazel better than Gradle for a monorepo?

It depends on the repository. Bazel caches per action in a content-addressed store, enforces hermeticity with a sandbox, supports many languages in one graph, and has native remote execution. Gradle has a build cache that works per task and is strongest for JVM and Android projects with an easier learning curve. Choose Bazel for polyglot or very large repositories and strict reproducibility; Gradle often suffices for JVM-centred teams already seeing good cache hits.

Does Bazel 9 still support the WORKSPACE file?

No. According to the official Bazel 9 announcement, Bazel 8.0 disabled WORKSPACE by default and Bazel 9.0, released as an LTS on 20 January 2026, removes the code that supports it. External dependencies are managed with Bzlmod and a MODULE.bazel file. The announcement also says a migration tool exists but will not complete migrations automatically, so plan manual work for repository rules and custom dependencies.

How do I prevent cache poisoning in a shared Bazel cache?

Restrict write access to trusted, controlled CI jobs and make everything else read-only, for example with the remote upload flag disabled on pull requests and laptops. Authenticate and authorise cache endpoints, isolate the network, and keep audit logs. Because outputs are stored by hash with no record of the producing build, the documentation advises clearing the whole cache if poisoning is suspected, so keep a tested purge-and-rebuild runbook.