Envoy xDS Control Plane Architecture: LDS, CDS, EDS, RDS, Delta xDS and Building One (2026)

Most teams meet Envoy through a YAML file, then discover that the file is the least interesting part of the system. The real product is the stream of typed, versioned protobuf resources that arrives over a long-lived gRPC connection, and the server on the other end of that stream decides what every proxy in your fleet believes about the world. An Envoy xDS control plane is that server. Get it wrong and you do not get an error page; you get a route that points at a cluster that does not exist yet, a listener that warms forever, or five thousand proxies reconnecting in the same second and flattening your management server.

This matters more in 2026 than it did when xDS was an Envoy-only curiosity. Istio, Envoy Gateway, Contour, and Consul all speak it, gRPC clients speak it without any proxy at all, and Kubernetes Gateway API implementations compile down to it. Understanding the protocol is now the difference between operating a mesh and merely installing one.

By the end you will know the resource model, the exact ACK/NACK and nonce mechanics, why ordering is a correctness problem and not an optimisation, how delta xDS changes the cost curve, and how to build a minimal control plane with go-control-plane.

What this covers: the data-plane and control-plane split, the resource types, state-of-the-world versus delta, ADS and make-before-break ordering, warming, scaling and reconnect storms, real implementations, proxyless gRPC, and a debugging toolkit built on the admin interface.

Context and Background

Envoy was designed around the idea that a proxy should be reconfigured without restarting. The static bootstrap YAML only needs to say where the management server is and who the node is. Everything else, listeners, routes, clusters, endpoints, TLS secrets, arrives dynamically through the discovery services collectively called xDS. The “x” is a placeholder: LDS for listeners, RDS for routes, CDS for clusters, EDS for endpoints, SDS for secrets, and so on. The protocol is specified in the official Envoy xDS REST and gRPC protocol documentation, which is the primary source for nearly every mechanism in this article.

The split matters architecturally. The data plane is the fleet of proxies moving packets and requests. The control plane never touches a request; it computes configuration and distributes it. That separation lets the data plane keep serving when the control plane is down, because Envoy retains the last configuration it received and reconnects in the background. The management server documentation states exactly this and recommends TCP or HTTP/2 keepalives so a half-dead connection is detected. Availability of the control plane therefore governs change, not traffic, which is a very different SLO from the one an API gateway carries. If you are comparing proxies at the gateway layer, the API gateway comparison of Kong, Envoy Gateway and Apache APISIX shows how each one models configuration distribution.

There are three ways to deliver xDS resources. The filesystem source watches a path with inotify and has no ACK or NACK; a bad file simply leaves the last valid configuration active. REST-JSON polling uses one outstanding request at a time, so nonces are optional, and it cannot do ADS. The gRPC streaming variant is what every production control plane uses, and it is the only one that supports delta and aggregated delivery. This article concentrates on gRPC streaming.

The incumbents are worth naming now, since each makes a different trade-off. Istiod translates Kubernetes and Istio custom resources into xDS and also acts as a certificate authority. Envoy Gateway is the Envoy project’s own Gateway API implementation, written in Go on top of go-control-plane. Contour is a Kubernetes ingress controller also built in Go. go-control-plane and java-control-plane are the reference libraries that implement the protocol machinery, not the translation of any platform’s resources. The go-control-plane README is explicit that the library provides infrastructure shared by many control planes and does not translate platform-specific resources into Envoy configuration. Keep that boundary in mind: the protocol layer is solved; the hard, interesting part of your control plane is your translator and your consistency model.

For multi-cluster and ambient architectures built on the same machinery, see the discussion of Istio ambient multicluster and the Gateway API inference extension.

The xDS Reference Architecture

A control plane has four moving parts: a source of truth, a translator, a versioned cache, and a protocol server. Everything else is detail.

The direct answer. An Envoy xDS control plane is a gRPC server that holds desired proxy configuration as versioned, typed protobuf resources and streams them to Envoy over the xDS protocol. Each proxy subscribes by resource type, acknowledges every response with an ACK or NACK carrying a version and nonce, and keeps serving the last accepted configuration if the server disappears.

Figure 1: Reference architecture of an Envoy xDS control plane, from source of truth to snapshot cache to per-node ADS streams, with ACK and NACK flowing back.

Figure 1 shows the data path. The source of truth might be the Kubernetes API, Consul’s catalog, a database, or a Git repository. The translator turns that into the typed resources Envoy understands. The cache stores a snapshot, a complete, internally consistent set of resources, keyed by a node group. The server answers stream requests from the cache and tracks what each client last acknowledged. Envoy and gRPC clients sit at the edge. Notice the dashed return path: ACK and NACK are the control plane’s only feedback about whether its output was usable, so a control plane that ignores them is flying blind.

Resource types and how they reference each other

The protocol defines distinct resource types, each with a type URL of the form type.googleapis.com/<proto name>; the Cluster type is type.googleapis.com/envoy.config.cluster.v3.Cluster. The supported v3 types in the protocol documentation are Listener, RouteConfiguration, ScopedRouteConfiguration, VirtualHost, Cluster, ClusterLoadAssignment, Secret, and Runtime. Resource type versioning is independent of the transport version, so a v3 Cluster can travel over any transport that supports it.

Figure 2: The xDS resource dependency graph. Each arrow is a name reference that the data plane resolves by issuing a new subscription.

The graph in Figure 2 is the key to everything that follows. A Listener (LDS) describes a socket and its filter chains; its HTTP connection manager names a RouteConfiguration (RDS) either inline or by reference. A route action names a Cluster (CDS). An EDS-type cluster names a ClusterLoadAssignment (EDS) holding the actual endpoints, grouped by locality and priority. TLS material in a Listener or Cluster is referenced as a Secret (SDS). Large route tables can be split with VirtualHost discovery (VHDS), and filter configuration can be delivered separately with extension config discovery (ECDS).

Every arrow is a name, not a pointer. The data plane resolves each name by subscribing, and that subscription is asynchronous. This is why consistency is a protocol-level concern: at any instant a proxy may hold a route that names a cluster it has not yet received.

Why Envoy always wildcard-subscribes LDS and CDS

Subscriptions use resource_names in state-of-the-world mode, or resource_names_subscribe and resource_names_unsubscribe in delta mode. The special wildcard name * applies only to Listener and Cluster. Envoy always uses wildcards for LDS and CDS: it wants everything the server decides it should have, and the server typically resolves “everything” by looking at the node identity in the first request. Other clients, gRPC for example, subscribe to specific listener names.

This is the hook for multi-tenancy. The node message, with its id, cluster, metadata and locality, is how a control plane partitions one fleet into many configuration views. In go-control-plane this is the NodeHash function that maps a node to a snapshot key. Choose it carefully: one snapshot per node means maximum precision and maximum memory; one snapshot per node group means shared work but coarser targeting.

The two orthogonal protocol variants

The protocol documentation describes two independent choices. First, state-of-the-world (SotW) versus incremental (delta). In SotW, each request carries the full list of subscribed names and, for LDS and CDS, every response must contain every resource. Delta sends only changes and supports lazy loading. Second, per-type streams versus Aggregated Discovery Service (ADS). Per-type streams give eventual consistency; ADS multiplexes all types on one stream so the server can sequence them.

The RPC names make the matrix concrete. SotW uses methods like StreamClusters and StreamAggregatedResources, with DiscoveryRequest and DiscoveryResponse. Delta uses DeltaClusters and DeltaAggregatedResources, with DeltaDiscoveryRequest and DeltaDiscoveryResponse. VHDS exists only in delta form. The transport version is encoded in the method name, so a server can tell what a client speaks without parsing its payload.

A bootstrap that selects ADS looks like this. The ads_config block names the management cluster, and both cds_config and lds_config point at the aggregated source:

node:
  id: edge-proxy-7
  cluster: edge
dynamic_resources:
  ads_config:
    api_type: GRPC          # use DELTA_GRPC for incremental xDS
    transport_api_version: V3
    grpc_services:
      - envoy_grpc: { cluster_name: xds_cluster }
  cds_config: { ads: {}, resource_api_version: V3 }
  lds_config: { ads: {}, resource_api_version: V3 }
static_resources:
  clusters:
    - name: xds_cluster
      type: STRICT_DNS
      connect_timeout: 1s
      typed_extension_protocol_options:
        envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
          "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
          explicit_http_config:
            http2_protocol_options:
              connection_keepalive: { interval: 30s, timeout: 5s }
      load_assignment:
        cluster_name: xds_cluster
        endpoints:
          - lb_endpoints:
              - endpoint: { address: { socket_address: { address: xds.internal, port_value: 18000 } } }

Two details matter. The connection_keepalive block is the HTTP/2 keepalive the documentation recommends so a dead connection is noticed. And the xDS cluster itself is static: the protocol documentation notes a known limitation that xDS clusters should appear first in static_resources, otherwise initialisation is slower. The api_type field is a verified Envoy option; treat the exact keepalive field names as something to confirm against your Envoy version’s reference.

Deeper Analysis: The Protocol Mechanics

Versions, nonces, ACK and NACK

Every DiscoveryResponse carries a version_info string and a nonce. The next DiscoveryRequest on the stream echoes the nonce in response_nonce. The version is a property of the resources for a given type and a given xDS server, not of the stream. The nonce, by contrast, is valid only for one stream and, in ADS, is tracked per resource type.

Figure 3: The ACK and NACK exchange on a CDS stream. A NACK echoes the previous good version with error detail attached, and the server then pushes a corrected version.

Figure 3 shows the lifecycle. The first request carries the node identifier and an empty version. The server answers with version v1 and nonce n1. Envoy validates and applies, then sends a request with version_info of v1 and response_nonce of n1: an ACK. When the server pushes v2 with nonce n2 and Envoy rejects it, the reply carries the previous version v1, the new nonce n2, and an error_detail message: a NACK.

Three subtleties trip people up.

First, the documentation says servers should detect a NACK by the presence of error_detail, not by comparing versions. A server that infers rejection from “the version did not advance” will misclassify the extra “resource hint” requests Envoy sometimes sends at the current version.

Second, an ACK means every resource in the response was valid in isolation. It does not guarantee the configuration was successfully applied end to end. A listener can be ACKed and still fail to bind a port.

Third, a NACK does not mean every resource was rejected. In SotW CDS, one malformed cluster among two hundred causes the response to be NACKed, and the practical consequence is that the previous configuration stays in force for that type. Control planes therefore need a blast-radius strategy: validate resources with the same schema Envoy uses before publishing, and keep a last-known-good snapshot.

The nonce solves a real race. Because a new request supersedes earlier requests of the same type on a stream, the server only needs to answer the latest. A stale nonce signals “do not respond to this request”: it was sent before the client saw your newest response. Without nonces, a delayed ACK for v1 could be misread as a rejection of v2.

State-of-the-world versus delta: where the cost goes

In SotW, LDS and CDS must send the complete state every time. If you have 8,000 clusters and one endpoint-health flag changes, the CDS response is still 8,000 clusters. Deletion is by omission: a Listener or Cluster missing from a SotW response is deleted, and an empty LDS or CDS response deletes everything of that type. For other types, deletion is implicit when the parent stops referencing the child, and an empty response is a no-op.

Delta xDS removes that tax. Responses carry only changed resources plus an explicit removed_resources list, and each resource has its own version. The request side changes too. A DeltaDiscoveryRequest is sent first on a stream, as an ACK or NACK with response_nonce set, or spontaneously to change subscriptions with no nonce. The nonce is required in delta mode, and a server must honour subscription changes even when the nonce is stale.

The biggest operational win is reconnect. A client can send initial_resource_versions listing what it already holds, so the server skips resources whose versions match. The protocol states clients must list every resource they still need, because no state carries over between streams.

Property	SotW	Delta
LDS/CDS payload on any change	Entire set	Changed resources only
Deletion signal	Omission from response	`removed_resources`
Missing-resource signal	None, client times out	Named in `removed_resources`
Reconnect cost	Full resend	Skips matching versions
Server state per client	Last version per type	Per-resource versions
Implementation complexity	Lower	Higher
VHDS support	No	Yes

The cost moves rather than vanishes. Delta shifts work from the wire to the server, which must track per-client, per-resource versions. For a fleet of 5,000 proxies each holding thousands of resources, that bookkeeping is memory. Istio ships delta xDS as a switchable option; the pilot-discovery reference lists ISTIO_DELTA_XDS as a boolean defaulting to true, described as sending only delta configuration changes rather than the full state, with the caveat that it may still occasionally resend unchanged configuration. I verified that entry against the reference; the exact behaviour per resource type in a given Istio release should be checked against that release’s notes.

Nonexistent resources and the 15-second rule

A subtle gap: how does a client learn that a name it asked for does not exist? In SotW there is no explicit signal, so the documentation recommends a client-side timeout of 15 seconds, and says Envoy applies this for RouteConfiguration and ClusterLoadAssignment during warming. In delta mode the server names the missing resource in removed_resources, though clients should still time out because the server may be slow. The corollary for control-plane authors is that the server must remember requested names and send an update if a requested resource appears later. A control plane that drops unmatched subscriptions silently leaves a cluster warming until the timeout fires.

TTLs and heartbeats

Resources can carry a TTL, advertised by the client feature xds.config.supports-resource-ttl. Re-sending a resource with a new TTL refreshes it; re-sending without a TTL removes it. A heartbeat is a resource message with the resource field unset and a version matching the last sent, which extends the TTL without counting as an update. TTLs are useful for ephemeral data such as runtime overrides, because a crashed control plane lets them expire rather than pinning stale values forever. In SotW mode, TTL requires resources to be wrapped in a Resource message, gated by xds.config.supports-resource-in-sotw.

Ordering, Warming and the Consistency Problem

Why eventual consistency drops traffic

With independent per-type streams, updates for different types race. Suppose you add cluster payments-v2 and a route that sends 10% of traffic to it. If the RDS update lands before the CDS update, the route references a cluster that does not exist. Envoy does not warm routes, so a request that matches that route has nowhere to go and fails. The protocol documentation says plainly that traffic can be dropped briefly during updates, for example when a route moves to a cluster that does not exist yet.

The fix is sequencing, and the documentation prescribes the order for a make-before-break update: CDS first, then EDS for those clusters, then LDS, then RDS, then VHDS, and finally removal of stale clusters and endpoints last.

Figure 4: Make-before-break ordering for an envoy xds control plane. Dependencies are created before anything references them, and removed only after nothing references them.

The logic is a topological sort of Figure 2 run in two directions. Additions flow from leaves to roots, so by the time a route names a cluster, the cluster and its endpoints are already present. Removals flow from roots to leaves, so a route stops pointing at a cluster before the cluster is deleted. A single mutation, “shift traffic from payments-v1 to payments-v2 and retire v1″, becomes four ordered pushes: add v2 cluster, add v2 endpoints, update the route, delete v1.

Only ADS gives you this control. On a single multiplexed stream the server decides which type to send next, and because each type still follows its own request and response rules, you can hold back RDS until the client has acknowledged CDS and EDS. On separate streams, you can only hope.

Warming: what “ready” means

Envoy does not activate a new cluster or listener until it has what it needs. Per the protocol documentation, a cluster warms until a ClusterLoadAssignment response arrives, and a listener warms until the RouteConfiguration it references arrives. Two Envoy-specific details are worth knowing. A cluster finishes warming only upon a new CLA response, even if the endpoints are unchanged, and after the timeout Envoy uses the cached CLA. A listener, on the other hand, can finish warming without an RDS response by using the last RouteConfiguration it received.

The operational reading: a control plane that withholds an expected EDS response stalls cluster activation, and a control plane that sends a CDS update twice with unchanged endpoints must still answer EDS. The documentation also tells servers to send updates only when resources actually change, because re-sending identical content wastes work on both sides. That tension, “answer when asked, stay quiet otherwise”, is why a hand-written server tends to go wrong and a library that models subscription state tends to go right.

Worked example: the partial rollout that fails

Take a mesh where a deploy controller does three things in one reconciliation: create cluster search-canary, create its endpoints, and edit the search route to add a 5% weighted split. A naive translator writes all three into the cache as separate updates on three independent streams.

On a healthy network the three responses arrive within milliseconds and nobody notices. Now add a slow EDS path, say the EDS stream is queued behind a large update for another cluster. The CDS response lands at t=0, the RDS response at t=20 ms, the EDS response at t=900 ms. For roughly 880 ms, 5% of search requests match a weighted entry whose cluster is still warming with no endpoints. At 2,000 requests per second, that is about 88 failed requests, from one deploy, with no error in any log of the control plane. The same change pushed through ADS in the documented order would have produced zero.

The lesson is that atomicity is a property you build, not one you inherit. Snapshot caches build it by publishing a whole consistent set and letting the server stage the delivery.

Building a Control Plane with go-control-plane

The snapshot cache model

go-control-plane’s central abstraction is the snapshot. A control plane builds a Snapshot, a version string plus a map from resource type to a list of resources, and stores it in a cache keyed by node. The library’s README describes its Simple cache as snapshot-based, maintaining a consistent configuration view per group of proxies, usable as an ADS server or as separate xDS servers. In ADS mode, it can hold responses until every referenced resource has been requested, which enables atomic updates of xDS collections.

The README also documents a Linear cache, an eventually consistent cache for a single type URL collection that keeps one linear version history and a version vector per resource. Use it when you have a large, flat, independently-updated collection, for example thousands of secrets or endpoints, and snapshot-per-change would be wasteful.

From the source, NewSnapshot(version string, resources map[resource.Type][]types.Resource) (*Snapshot, error) builds a snapshot, and (*Snapshot).Consistent() error checks it for referential integrity. The consistency check is the in-process guard against the race described above: it verifies that routes and endpoints referenced by other resources are present in the snapshot.

A minimal server

The following is a compact control plane. The two signatures above were checked against the repository source. The remainder, including constructor names, the callbacks interface, package import paths and the registration function, are written from memory of the library’s examples and should be treated as illustrative until compiled against the version you pin; the library has changed its package layout over time.

package main

import (
    "context"
    "fmt"
    "net"
    "time"

    cluster "github.com/envoyproxy/go-control-plane/envoy/config/cluster/v3"
    core "github.com/envoyproxy/go-control-plane/envoy/config/core/v3"
    endpoint "github.com/envoyproxy/go-control-plane/envoy/config/endpoint/v3"
    discovery "github.com/envoyproxy/go-control-plane/envoy/service/discovery/v3"
    "github.com/envoyproxy/go-control-plane/pkg/cache/types"
    "github.com/envoyproxy/go-control-plane/pkg/cache/v3"
    "github.com/envoyproxy/go-control-plane/pkg/resource/v3"
    xds "github.com/envoyproxy/go-control-plane/pkg/server/v3"
    "google.golang.org/grpc"
    "google.golang.org/protobuf/types/known/durationpb"
)

func makeCluster(name string) *cluster.Cluster {
    return &cluster.Cluster{
        Name:                 name,
        ConnectTimeout:       durationpb.New(time.Second),
        ClusterDiscoveryType: &cluster.Cluster_Type{Type: cluster.Cluster_EDS},
        EdsClusterConfig: &cluster.Cluster_EdsClusterConfig{
            EdsConfig: &core.ConfigSource{
                ResourceApiVersion:    core.ApiVersion_V3,
                ConfigSourceSpecifier: &core.ConfigSource_Ads{Ads: &core.AggregatedConfigSource{}},
            },
        },
    }
}

func makeCLA(name, host string, port uint32) *endpoint.ClusterLoadAssignment {
    addr := &core.Address{Address: &core.Address_SocketAddress{SocketAddress: &core.SocketAddress{
        Address: host, PortSpecifier: &core.SocketAddress_PortValue{PortValue: port}}}}
    return &endpoint.ClusterLoadAssignment{
        ClusterName: name,
        Endpoints: []*endpoint.LocalityLbEndpoints{{
            LbEndpoints: []*endpoint.LbEndpoint{{
                HostIdentifier: &endpoint.LbEndpoint_Endpoint{Endpoint: &endpoint.Endpoint{Address: addr}},
            }},
        }},
    }
}

func main() {
    ctx := context.Background()
    // ads=true: one aggregated stream, ordered delivery. IDHash keys by node id.
    snapCache := cache.NewSnapshotCache(true, cache.IDHash{}, nil)

    snap, err := cache.NewSnapshot("v1", map[resource.Type][]types.Resource{
        resource.ClusterType:  {makeCluster("search")},
        resource.EndpointType: {makeCLA("search", "10.0.0.12", 8080)},
    })
    if err != nil { panic(err) }
    if err := snap.Consistent(); err != nil { panic(err) } // reject dangling references
    if err := snapCache.SetSnapshot(ctx, "edge-proxy-7", snap); err != nil { panic(err) }

    srv := xds.NewServer(ctx, snapCache, nil)
    g := grpc.NewServer()
    discovery.RegisterAggregatedDiscoveryServiceServer(g, srv)
    lis, _ := net.Listen("tcp", ":18000")
    fmt.Println("xDS listening on :18000")
    _ = g.Serve(lis)
}

Three design choices are embedded in those few lines. Passing true for the ADS flag opts into aggregated delivery and the ordering that comes with it. IDHash keys each snapshot by node id, which means one snapshot per proxy; a production system usually hashes to a node group such as the cluster field or a metadata label to share work. And the call to Consistent() before SetSnapshot is the guard that prevents a half-built configuration from ever reaching a client.

Versions need a policy. A monotonically increasing integer is fine, but a content hash is better: if the new snapshot hashes to the existing version, skip SetSnapshot and send nothing, which honours the documentation’s rule against re-sending identical content. Remember that Envoy treats the version string as opaque; the server only needs it to differ when content differs.

Inspecting what the server sent

Verify the stream from the Envoy side with the admin interface. The /config_dump endpoint accepts resource= to select a repeated field such as dynamic_active_clusters, mask= to return a protobuf field-mask subset, and include_eds to include endpoint configuration:

# Which clusters did the control plane give me?
curl -s 'localhost:9901/config_dump?resource=dynamic_active_clusters&mask=cluster.name'

# Include endpoints (EDS) in the dump
curl -s 'localhost:9901/config_dump?include_eds' | head -n 60

# xDS subscription health: attempts, successes, rejections, connectivity
curl -s 'localhost:9901/stats?filter=control_plane|update_rejected|update_success'

# Cluster membership and per-host health
curl -s 'localhost:9901/clusters?format=json&filter=^search$'

The admin documentation notes that the config_dump proto is marked v2alpha, so its JSON form is not guaranteed stable, and that Envoy tries to redact private_key and password fields, but only for strongly typed protos. Never paste an unreviewed dump into a ticket.

Scaling the Control Plane

What actually costs CPU

Two operations dominate. Translation converts platform state into protobuf, and serialisation marshals those protobufs per client. Naively, every change multiplies by fleet size. The standard remedy is to translate once per node group and serialise once per distinct resource, caching the marshalled bytes. Istio’s control plane documentation and tuning guidance describe this as an xDS cache plus debouncing of change events; I could not retrieve the exact defaults for the debounce settings this run, so I do not quote them here. Check your release’s reference.

Debouncing is the cheapest big win. If a rollout touches 300 pods in 20 seconds, you do not want 300 full pushes; you want a quiet-period timer that batches events and a maximum-wait ceiling so a continuous storm cannot postpone pushes forever. The trade-off is latency: the debounce interval is added to every config change.

Reconnect storms and the thundering herd

The scariest event is not a change, it is a restart. When a management server instance dies, every client attached to it reconnects at once. Under SotW, each reconnecting client demands the full state.

Here is a worked, clearly illustrative calculation. Assume 5,000 proxies, a 3 MB serialised snapshot per proxy, and 10 Gbps of egress available to the control plane tier. The total is 5,000 × 3 MB = 15,000 MB, about 14.6 GiB, which takes roughly 12 seconds of pure wire time at line rate, before any CPU cost for serialisation, TLS, or the translation work needed if caches are cold. Those numbers are my own assumptions, not a measurement. The point is the shape: cost scales with fleet size times snapshot size.

With delta xDS and initial_resource_versions, a reconnecting proxy that already holds almost everything receives almost nothing. If each reconnect transfers only 40 KB of changed resources, the same fleet pulls roughly 195 MB. That is two orders of magnitude less, but only because the proxies kept their state; a freshly started proxy still needs the whole configuration.

I ran the arithmetic in a short script to be sure the figures above are self-consistent:

N, size_mb, gbps = 5000, 3.0, 10
total_gb = N * size_mb / 1024
wire_s = N * size_mb * 8 / (gbps * 1000)
delta_mb = N * 40 / 1024
print(f"{total_gb:.1f} GiB, {wire_s:.1f}s wire time, delta {delta_mb:.0f} MB")
# 14.6 GiB, 12.0s wire time, delta 195 MB

Mitigations stack. Client-side exponential backoff with jitter spreads reconnects across a window; spreading 5,000 connects over about 8 seconds cuts the peak to roughly 625 per second. Server-side admission control caps concurrent full-state sends. A rolling restart of management servers, with connection draining via gRPC’s GOAWAY, avoids synchronised reconnects. Envoy exposes rate_limit_enforced and pending_requests counters under the management server stats, indicating that it rate-limits its own requests to the management server, although the documentation page does not describe the mechanism.

Sharding

When one instance cannot hold every node’s state, shard. The usual approaches are by cluster or environment, by consistent hashing of node id onto control plane replicas, or by region. Each shard needs the full source-of-truth view for the nodes it serves but not for the whole fleet. Istio’s multicluster designs are a variant of this, with per-cluster discovery and remote secrets. The failure mode of sharding is skew: a hot tenant can overload one replica while others idle, so shard on a dimension that correlates with resource count, not just proxy count.

Real Implementations and xDS Beyond Envoy

Istio, Envoy Gateway, Contour and the library layer

The same protocol, four different philosophies. Istiod is a monolithic control plane: it watches Kubernetes and its own custom resources, merges them into an internal model, and pushes xDS to sidecars, gateways, and in ambient mode to node-level components. Its centre of gravity is push management: debouncing, throttling concurrent pushes, caching, and the delta option described earlier. Envoy Gateway is narrower: it implements the Kubernetes Gateway API, translating Gateway, HTTPRoute and related objects through an intermediate representation and then into xDS served via go-control-plane. Contour follows a similar pattern for ingress and HTTPProxy resources. java-control-plane serves the JVM ecosystem with the same snapshot-cache idea.

The practical difference for you as a builder is the translator, not the transport. If your source of truth is a service registry with a custom policy model, you will write a translator and borrow everything below it. If it is Kubernetes Gateway API, do not write a control plane; adopt one.

Choice	Best fit	Main cost
Istiod	Full mesh with mTLS, policy, telemetry	Large surface, tuning needed at scale
Envoy Gateway	Gateway API ingress and north-south	Not a sidecar mesh
Contour	Ingress with HTTPProxy semantics	Narrower than a mesh
Own control plane on go-control-plane	Custom registry, bespoke policy, edge fleets	You own correctness and upgrades

Proxyless gRPC: xDS without Envoy

xDS is no longer Envoy-only. The gRPC project adopted it so that a gRPC client can do service discovery, load balancing and routing straight from a management server, with no sidecar. The design proposal A27 describes an xds resolver and CDS and EDS load-balancing policies sharing one internal XdsClient. A channel opts in with an xds: target such as xds:///example.com:123. The client reads a JSON bootstrap whose location is given by the GRPC_XDS_BOOTSTRAP environment variable, containing xds_servers, a node block, and credential types.

Everything flows over a single ADS stream in dependency order: the client requests a Listener by the target’s server name, follows it to a RouteConfiguration, selects a Cluster, then requests its endpoints. This matches the protocol documentation’s note that a non-proxy client begins from the specific Listeners it needs and follows references downward, and that gRPC clients support only ADS. The original proposal also lists limits, such as no endpoint-level weights at that time. Those limits are from the proposal and gRPC has since expanded features through later proposals, so check the current gRPC xDS feature list before depending on a capability.

For control-plane authors the implication is significant. Your server now serves heterogeneous clients that do not share Envoy’s warming behaviour or its validation rules. The documentation warns that protoc-gen-validate constraints are not applied by all clients (gRPC does not apply them) and that control planes cannot assume clients validate identically to Envoy.

Gateway-style resource subscription at scale

One more pattern deserves a mention: scoped and on-demand delivery. Scoped routes (SRDS) and VHDS let a proxy fetch only the route slices it needs, which pays off when a gateway holds tens of thousands of virtual hosts. Lazy loading is a delta-only capability, and it trades a small first-request latency for a large reduction in steady-state memory and push volume. If your gateway multiplexes many tenants, design the control plane around on-demand subscription from the beginning, since retrofitting it onto a SotW server is painful.

Trade-offs, Gotchas, and What Goes Wrong

A NACK loop can freeze a fleet silently. If a bad resource is published, every proxy NACKs and keeps its old configuration. Traffic continues, dashboards stay green, and your intended change simply never lands. Alert on update_rejected rising, not just on server errors, and surface NACK error_detail text in control-plane logs. Pair it with last-known-good snapshots so you can roll back by republishing.

ACK is not “applied”. The documentation is clear that an ACK means every resource was valid in isolation. A listener can bind-fail, a secret can be unusable, an extension can load late. Do not drive rollout progress bars off ACK alone; sample actual behaviour through the admin interface or telemetry.

Duplicates and wildcards bite. Duplicate resource names in one response are an error that clients should NACK. And a client cannot unsubscribe from individual resources covered by a wildcard; it can only drop the wildcard. If you design per-resource opt-out on top of a wildcard subscription, it will not work.

Nonce bugs masquerade as flakiness. A server that ignores nonces will answer stale requests, double-send, or treat a hint request as an ACK. The symptom is intermittent warming delays that disappear under low load. Treat the nonce rule, “stale nonce means do not respond”, as part of correctness, not as an optimisation.

Delta is not free. It reduces bytes but increases server memory and implementation risk. A bug in per-resource version tracking can leave a client permanently stale in a way SotW, which resends everything, would self-heal. For small fleets with small configs, SotW over ADS is simpler and perfectly adequate.

Control plane down is a slow disaster. Envoy keeps serving with its last configuration, which is exactly why you may not notice an outage for days. Then a pod reschedules, EDS cannot update, and traffic goes to dead endpoints. Monitor control_plane.connected_state (1 connected, 0 disconnected) and alert on it. Also test cold start: a newly launched proxy with no management server has no listeners at all.

Eventual consistency without ADS. Using separate per-type streams to save complexity reintroduces the route-before-cluster race. If you have weighted routing, blue-green shifts or frequent cluster churn, use ADS or accept the dropped requests.

Validation drift. Because clients apply different validation, a resource that Envoy accepts may be rejected by a gRPC client or a different Envoy version. Pin and test against each client class you serve.

Practical Recommendations

Choose by team profile rather than by fashion. If you run Kubernetes and want a mesh or a Gateway API ingress, adopt Istio or Envoy Gateway and spend your effort on policy and observability. If you manage a bespoke fleet, such as an edge network with its own registry, build on go-control-plane with the snapshot cache and ADS; the library gives you the protocol and you supply the translator.

Start with SotW over ADS. It gives ordering, which is the property that prevents outages, with the least state. Move to delta when snapshot size times fleet size makes reconnects or pushes painful, which in practice means thousands of proxies or tens of thousands of resources per proxy. Treat that threshold as a measurement you take, not a number to copy.

Make every publish pass a consistency gate. Build the snapshot, call the consistency check, validate against Envoy’s own schema (for example by running a throwaway Envoy with --mode validate against the generated bootstrap), and only then store it. Keep the previous snapshot addressable for instant rollback.

Instrument the loop end to end: snapshot version published, per-node last ACKed version, NACK count with error text, push latency from source change to ACK, and stream count. A single dashboard showing “nodes not on the latest version” catches most incidents early.

Use ADS and the documented order: CDS, EDS, LDS, RDS, VHDS, removals last.
Hash node identity to a group, not to every proxy, unless you need per-proxy targeting.
Skip publishing when content is unchanged; version by content hash.
Add client-side jittered backoff and server-side admission control for reconnects.
Alert on update_rejected and control_plane.connected_state.
Treat config_dump output as sensitive and as unstable JSON.
Test with a proxyless gRPC client if you intend to serve one.

For the request-limiting layer that often rides on the same Envoy fleet, see the write-up on distributed rate limiting with token bucket, GCRA and Redis in Envoy.

Frequently Asked Questions

What is an Envoy xDS control plane?

It is a gRPC management server that stores desired proxy configuration as versioned protobuf resources and streams them to Envoy. Envoy subscribes by resource type (listeners, routes, clusters, endpoints, secrets), acknowledges each response with a version and nonce, and keeps its last accepted configuration if the server becomes unreachable. Istiod, Envoy Gateway and Contour are well-known examples, and go-control-plane is the usual library for building your own.

What is the difference between SotW and delta xDS?

State-of-the-world sends the complete set of resources for a type on each LDS or CDS response, and deletion is signalled by omission. Delta xDS sends only changed resources plus an explicit removed_resources list and lets a reconnecting client declare what it already holds through initial_resource_versions. Delta reduces bytes and supports lazy loading, but costs more server-side state and implementation complexity.

Why is ADS needed, and what order should updates follow?

ADS multiplexes all resource types on one stream so the server can sequence them. Without it, updates for different types race and a route can reference a cluster that has not arrived. The documented make-before-break order is CDS, then EDS, then LDS, then RDS, then VHDS, with stale clusters and endpoints removed last, so dependencies always exist before they are referenced.

What happens when Envoy NACKs a response?

Envoy replies with the previous good version_info, the new nonce, and an error_detail message. It keeps running the last accepted configuration for that resource type. Servers should detect a NACK by the presence of error_detail, not by comparing versions. A NACK does not mean every resource failed; it means the response as a whole was rejected, so one bad resource can block a whole update.

Can gRPC use xDS without Envoy?

Yes. gRPC clients can use an xds:/// target and a bootstrap file located through the GRPC_XDS_BOOTSTRAP environment variable to obtain listeners, routes, clusters and endpoints from an xDS server over ADS, with no sidecar proxy. gRPC supports only ADS and a subset of Envoy features, and it does not apply every validation rule Envoy does, so check its supported feature list.

How do I debug what a control plane actually sent?

Query the Envoy admin interface. /config_dump with resource= and mask= narrows the output, include_eds adds endpoints, /clusters shows discovered hosts and health, and /stats exposes update_success, update_rejected, version and connected_state under the control plane tree. Compare the version Envoy reports with the version your server believes it published for that node.