Kubernetes Operator Design: Controller-Runtime, Reconciliation Loops, CRDs and Finalizers

Most operators that fail in production do not fail because the author misunderstood Kubernetes APIs. They fail because the author wrote an event handler when the platform expected a convergence function. A Kubernetes operator that reacts to “pod created” or “field changed” will drift the first time an event is dropped, reordered or replayed after a restart. One that computes “what should exist, given everything I can see right now” survives all three.

This matters more in 2026 than it did when operators were a novelty. Databases, GPU schedulers, certificate issuers, GitOps engines and entire cloud control planes now ship as controllers, and platform teams increasingly write their own. The cost of a subtle reconcile bug is a hot loop that melts the API server, a leaked cloud resource, or a namespace stuck in Terminating.

You will leave with a working mental model of how controller-runtime moves an event from the API server to your code, a Reconcile skeleton in Go you can adapt, rules for designing the custom resource definition, and the finalizer, status and testing patterns that separate a demo from a dependable controller.

What this covers: control-theory framing, informers and workqueues, an idempotent Reconcile, owner references and finalizers, status and server-side apply, CRD schema and versioning, leader election, RBAC, envtest, scaling pitfalls, and when not to write an operator at all.

Context and Background

The operator idea was introduced by CoreOS in 2016: encode the knowledge of a human operator for a stateful application as software that runs inside the cluster and uses Kubernetes as its API. Technically, an operator is nothing more than a custom resource plus a controller. The custom resource is a new object type registered with the API server. The controller is a long-running process that watches those objects and drives the world toward what they describe.

Kubernetes itself is built from the same parts. The Deployment controller, the ReplicaSet controller and the Job controller are all reconciliation loops inside kube-controller-manager. An operator is simply the same pattern, delivered by you instead of by the project. That framing is useful because it tells you what is allowed to be hard: you inherit the semantics of the built-in controllers, including their failure modes.

The ecosystem has consolidated around a few building blocks. The Kubernetes project documents the pattern in its operator concept page. In Go, nearly everything is built on sigs.k8s.io/controller-runtime, the library that wraps client-go’s informers, caches and workqueues in a smaller API. Scaffolding comes from kubebuilder, which generates the project layout, the CRD manifests via controller-gen, and the manager wiring. The Operator SDK builds on the same foundation and adds Helm and Ansible-based operators.

Version facts, checked against the GitHub release pages while writing this post: controller-runtime v0.25.0 shipped in early September 2026 and bumps the Kubernetes libraries to the v1.37 line, with v0.25.1 following in mid-September to fix a PriorityQueue data race. The latest kubebuilder release I could confirm is v4.16.0, from September 2026, which ships controller-tools v0.22.0. Pin your own versions and read the release notes; controller-runtime is still a v0.x library and minor releases do contain breaking changes, such as the new EventRecorder interface in v0.25.0.

If your operator mainly provisions infrastructure, check whether you need custom code at all. Our post on Crossplane composition functions covers the declarative alternative, and I return to the decision at the end of this article.

The Reconciliation Loop: How a Kubernetes Operator Actually Works

A reconciliation loop is a function that takes the name of one object, reads the current state of the world, compares it to the desired state recorded in that object’s spec, and issues the smallest set of writes that moves reality closer to the spec. It is level-triggered, idempotent and safe to run at any time, any number of times.

Figure 1: Event path of a Kubernetes operator built on controller-runtime, from the API server watch to the Reconcile call and back.

Figure 1 shows the path. A reflector opens a list-then-watch stream against the API server and fills a shared informer cache. Event handlers translate cache events into keys of the form namespace/name and push them onto a workqueue. Reconcile workers pop keys, read from the cache, and write back to the API server. The key carries no payload: it says which object to look at, not what happened to it. That one design choice is the root of every other property in this section.

Level-triggered versus edge-triggered

Control theory distinguishes systems that respond to changes (edges) from systems that respond to the difference between a setpoint and a measurement (levels). A thermostat does not wait to hear that the room got colder. It samples the temperature, compares it with the target and acts on the error. Kubernetes controllers work like thermostats.

The practical consequence is robustness against lost information. Watches break, controllers restart, and the API server compacts history. An edge-triggered design that missed the “deleted” event has no way to find out later. A level-triggered design re-lists on restart, sees that the object is gone or present, and computes the right action from the current state alone. The Kubernetes documentation on controllers describes the same control-loop model and the “desired versus current state” contract.

The rule that follows is strict: never put logic in Reconcile that depends on which event triggered it. controller-runtime deliberately gives you only a reconcile.Request containing a NamespacedName. If you find yourself wanting the old object to compute a diff, you are asking for edge information the platform intentionally withholds. Store whatever you need in status or an annotation instead.

Informers, shared caches and the workqueue

Informers are the part of client-go that makes the rest affordable. Without them, every Reconcile would issue GET and LIST calls and a busy cluster would drown its own API server. With them, there is one watch per resource type per process, shared by all controllers inside the manager, and reads are served from memory.

Three details matter in practice. First, the cache is eventually consistent: a Get immediately after your own Create can return NotFound, because the watch event has not arrived yet. controller-runtime v0.25.0 added an experimental opt-in, Client.EnableReadYourWritesConsistency, aimed at exactly this, but it is new and labelled experimental, so design as though reads can be stale. Second, objects returned from the cache are shared pointers; mutating one without a deep copy corrupts every other reader. Third, the cache holds full objects for every type you watch, so memory scales with cluster size, which I come back to under scaling.

The workqueue provides the properties that make a naive loop safe. It deduplicates: if an object changes fifty times while its key is waiting, the worker runs once and sees the latest state. It guarantees that a given key is never processed by two workers at once, so you get per-object serialisation without writing locks. And it carries the retry policy. When Reconcile returns an error, the key is re-added with rate limiting. The default limiter in client-go combines a per-item exponential backoff, starting at 5 milliseconds and capped at 1,000 seconds, with an overall token bucket of 10 queries per second and a burst of 100. These are the library defaults as I know them from the client-go source; confirm against the version you pin before depending on the exact numbers.

controller-runtime v0.23.0 switched the controller queue to a priority queue by default, according to its release notes, and added an optional Priority field on reconcile.Result so that, for example, a freshly created object can be served before a resync of an old one. That helps on controller restart, when a cache fill queues thousands of keys at once and you would prefer new user requests to jump the line.

Results, errors and requeues

Reconcile returns (ctrl.Result, error), and the combinations have precise meanings. Returning an error requeues with backoff and logs the failure. Returning Result{RequeueAfter: d} with a nil error schedules a re-check after a fixed delay and resets the backoff history. Returning an empty Result{} means “done until something changes”. Bare Requeue: true is retained for compatibility, and its use is discouraged in current documentation, so prefer RequeueAfter or return an error.

Choose deliberately. Use an error for transient failure you want retried with growing delay. Use RequeueAfter for polling an external system that has no event source, such as a cloud resource that takes ninety seconds to become ready. Use neither when waiting on another Kubernetes object, because a watch on that object, wired with Owns or Watches, will wake you at the right moment for free.

Idempotency in practice

Idempotent means that running Reconcile ten times in a row with no external change produces the same cluster as running it once. Three habits deliver this.

Derive names deterministically from the parent, so a retry after a crash finds the child it already created instead of creating a duplicate. Use create-or-update semantics (controllerutil.CreateOrUpdate, CreateOrPatch, or server-side apply, covered below) rather than “create, and if that fails, give up”. And treat AlreadyExists and NotFound as normal outcomes, not errors, wherever the desired state is satisfied either way.

The classic anti-pattern is the counter or timestamp written into a child on every pass. Each write generates a watch event for the child, which wakes the parent through its owner reference, which writes again. That is a hot loop, and I return to it in the scaling section.

A Runnable Reconcile Skeleton in Go

The following skeleton targets a hypothetical Database custom resource. It is written against the controller-runtime API as of the v0.2x series; method signatures shift between minor releases, so compile it against the version you pin. It demonstrates the shape of a correct reconciler: fetch, handle deletion, ensure finalizer, converge children, publish status.

package controller

import (
    "context"
    "fmt"
    "time"

    appsv1 "k8s.io/api/apps/v1"
    apierrors "k8s.io/apimachinery/pkg/api/errors"
    "k8s.io/apimachinery/pkg/api/meta"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/runtime"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
    "sigs.k8s.io/controller-runtime/pkg/log"

    dbv1 "example.com/db-operator/api/v1"
)

const finalizerName = "db.example.com/cleanup"

type DatabaseReconciler struct {
    client.Client
    Scheme *runtime.Scheme
}

// +kubebuilder:rbac:groups=db.example.com,resources=databases,verbs=get;list;watch;update;patch
// +kubebuilder:rbac:groups=db.example.com,resources=databases/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=db.example.com,resources=databases/finalizers,verbs=update
// +kubebuilder:rbac:groups=apps,resources=statefulsets,verbs=get;list;watch;create;update;patch;delete
func (r *DatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    logger := log.FromContext(ctx)

    var db dbv1.Database
    if err := r.Get(ctx, req.NamespacedName, &db); err != nil {
        // Deleted before we got here: nothing to do.
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // Deletion path: run cleanup, then release the object.
    if !db.DeletionTimestamp.IsZero() {
        if controllerutil.ContainsFinalizer(&db, finalizerName) {
            if err := r.cleanupExternal(ctx, &db); err != nil {
                return ctrl.Result{}, err // retried with backoff
            }
            controllerutil.RemoveFinalizer(&db, finalizerName)
            if err := r.Update(ctx, &db); err != nil {
                return ctrl.Result{}, client.IgnoreNotFound(err)
            }
        }
        return ctrl.Result{}, nil
    }

    // Ensure the finalizer exists before creating anything external.
    if controllerutil.AddFinalizer(&db, finalizerName) {
        if err := r.Update(ctx, &db); err != nil {
            return ctrl.Result{}, err
        }
    }

    // Converge the owned StatefulSet.
    sts := &appsv1.StatefulSet{
        ObjectMeta: metav1.ObjectMeta{Name: db.Name, Namespace: db.Namespace},
    }
    op, err := controllerutil.CreateOrUpdate(ctx, r.Client, sts, func() error {
        sts.Spec.Replicas = &db.Spec.Replicas
        // ...populate selector, template, volumeClaimTemplates here...
        return controllerutil.SetControllerReference(&db, sts, r.Scheme)
    })
    if err != nil {
        r.setCondition(&db, "Ready", metav1.ConditionFalse, "ApplyFailed", err.Error())
        _ = r.Status().Update(ctx, &db)
        return ctrl.Result{}, err
    }
    logger.V(1).Info("statefulset reconciled", "operation", op)

    // Publish status from observed state, never from intent.
    ready := sts.Status.ReadyReplicas == db.Spec.Replicas
    status := metav1.ConditionFalse
    reason := "Progressing"
    if ready {
        status, reason = metav1.ConditionTrue, "AllReplicasReady"
    }
    db.Status.ObservedGeneration = db.Generation
    db.Status.ReadyReplicas = sts.Status.ReadyReplicas
    r.setCondition(&db, "Ready", status, reason,
        fmt.Sprintf("%d/%d replicas ready", sts.Status.ReadyReplicas, db.Spec.Replicas))
    if err := r.Status().Update(ctx, &db); err != nil {
        return ctrl.Result{}, err
    }

    if !ready {
        return ctrl.Result{RequeueAfter: 30 * time.Second}, nil
    }
    return ctrl.Result{}, nil
}

func (r *DatabaseReconciler) setCondition(db *dbv1.Database, t string,
    s metav1.ConditionStatus, reason, msg string) {
    meta.SetStatusCondition(&db.Status.Conditions, metav1.Condition{
        Type: t, Status: s, Reason: reason, Message: msg,
        ObservedGeneration: db.Generation,
    })
}

func (r *DatabaseReconciler) cleanupExternal(ctx context.Context, db *dbv1.Database) error {
    // Delete cloud snapshots, DNS records, etc. Must be safe to call repeatedly.
    return nil
}

func (r *DatabaseReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&dbv1.Database{}).
        Owns(&appsv1.StatefulSet{}).
        Complete(r)
}

var _ = apierrors.IsNotFound // retained for readers extending error handling

Figure 2: Decision flow of an idempotent Reconcile function, with the deletion branch handled before any creation.

Figure 2 maps the skeleton to a flow: every pass starts from a cache read, branches early on deletion, and ends by publishing status. The ordering is the point, because it guarantees that a crash at any line leaves a state the next pass can recover from.

Read the skeleton for what it does not do. It never inspects which event fired. It writes status through the status subresource client, not through Update. It sets ObservedGeneration on both the status and each condition so consumers can tell whether a “Ready” belongs to the current spec. And Owns(&appsv1.StatefulSet{}) means a change to the child enqueues the parent through its controller owner reference, which is why the reconciler never polls Kubernetes objects.

Ownership, Garbage Collection and Finalizers

Every object a controller creates should say who owns it, and every external side effect a controller causes should have a cleanup path. Kubernetes gives you two mechanisms for this, owner references for in-cluster children and finalizers for everything else. They are easy to confuse and they fail in opposite directions.

Owner references and garbage collection

An owner reference is a field in a child’s metadata that points at its parent by kind, name and UID. When the parent is deleted, the garbage collector in kube-controller-manager deletes the dependents. In controller-runtime you set it with controllerutil.SetControllerReference, which also marks the reference as the controller reference (controller: true) and sets blockOwnerDeletion. The controller flag is what lets Owns() map a child event back to the right parent, and only one owner per object may be the controller.

Two constraints catch people. Owner references cannot cross namespaces: a namespaced owner can own children only in its own namespace, and cluster-scoped objects cannot be owned by namespaced ones. If your design needs a namespaced custom resource to own cluster-scoped objects such as a ClusterRole or a PersistentVolume, owner references will not do it and you need a finalizer to clean up. Second, deletion propagation matters. With background deletion (the default for most clients), the parent disappears immediately and children are collected afterwards. With foreground deletion, the parent gets a foregroundDeletion finalizer and stays visible until children carrying blockOwnerDeletion are gone.

Finalizers as pre-delete hooks

A finalizer is a string in metadata.finalizers. While the list is non-empty, a delete request does not remove the object. The API server instead sets metadata.deletionTimestamp and leaves the object in place. Controllers that own a finalizer see the update, perform cleanup, then remove their string. When the list empties, the API server completes the deletion.

Figure 3: Finalizer-driven deletion. The object lingers with a deletionTimestamp until the controller confirms external teardown and removes its finalizer.

The sequence in Figure 3 is the entire contract, and it has sharp consequences. Add the finalizer before you create the external resource, not after, or a crash between the two leaks the resource. Make cleanup idempotent, because it will run more than once if removing the finalizer fails after the teardown succeeded. Never block forever: if cleanup depends on an external system that is permanently gone, the object sticks in Terminating, and a namespace deletion that includes it sticks too.

Design an escape hatch. A common pattern is an annotation such as example.com/skip-cleanup: "true" that an administrator can set to make the controller skip teardown and release the object. The alternative is an administrator manually patching the finalizer list to null, which works but bypasses your logic silently. Also keep the finalizer name domain-qualified, as in the skeleton, so that two controllers never collide.

Finalizers also interact with RBAC in one surprising place. Setting blockOwnerDeletion on an owner reference requires permission to update the owner’s finalizers subresource when the OwnerReferencesPermissionEnforcement admission plugin is enabled, which is why the skeleton’s markers include a databases/finalizers rule. Omit it and child creation can be denied on some clusters even though it works on your laptop.

Status, Conditions and Server-Side Apply

Status is an output, spec is an input

The spec is what the user wants, and only the user (or their pipeline) writes it. Status is what the controller observes, and only the controller writes it. Enable the status subresource on the CRD with // +kubebuilder:subresource:status, and the two are separated at the API level: writes to the main resource ignore status changes, and writes to /status ignore spec changes. This prevents a user’s kubectl apply from clobbering the controller’s observations, and it means status updates do not bump metadata.generation.

The generation field is the key to knowing whether your status is current. The API server increments metadata.generation on every spec change when the status subresource is enabled. Copy it into status.observedGeneration at the end of a successful pass. A consumer who sees generation: 7 and observedGeneration: 6 knows the controller has not yet processed the latest spec, and that “Ready=True” refers to old intent. Kubernetes’ own Deployment status follows the same convention.

Conditions over phases

Avoid a single phase string such as Pending, Running, Failed. It forces a state machine onto something that is really a set of independent facts. The API conventions favour a list of conditions, each with type, status (True, False, Unknown), reason (a CamelCase machine-readable token), message (human text), lastTransitionTime and observedGeneration. metav1.Condition and meta.SetStatusCondition implement this and update lastTransitionTime only when the status actually flips.

Pick a small, stable vocabulary. Ready as the summary, plus perhaps Progressing and Degraded, covers most cases, and tooling such as kstatus and many GitOps health checks understands Ready and the abnormal-true pattern. Make reasons stable enough that alert rules can match on them.

Server-side apply and field ownership

Server-side apply (SSA), generally available since Kubernetes 1.22, tracks which manager owns which field in metadata.managedFields. When you apply, you declare the fields you care about, and the server merges them, flagging a conflict if another manager owns a field you are trying to set to a different value. For controllers, this solves a problem that read-modify-write cycles handle poorly: sharing an object with other actors, such as an HPA that owns replicas on your Deployment.

With CreateOrUpdate, your controller fetches the whole child, mutates it and sends the whole thing back, so you implicitly claim every field and race with any other writer. With SSA you send a partial object listing only what you own, under a stable field manager name, and the server preserves the rest. Newer controller-runtime releases include typed apply support in the client, and the v0.23.0 release notes mention subresource Apply; check the API of your pinned version, since this surface has been moving.

The failure mode to know: SSA with Force: true takes ownership of conflicting fields, which silently overrides humans or other controllers. Use force for fields that are unambiguously yours, and let conflicts surface for the rest.

Designing the Custom Resource Definition

The API you publish is the most durable decision in the project. Go code can be refactored next week, but a CRD with users has the same compatibility obligations as any other Kubernetes API. A good custom resource definition is small, declarative and validated at the API server, not in your controller.

Structural schemas and what they buy you

apiextensions.k8s.io/v1 requires a structural schema: every field has a declared type, and the schema is fully specified, with no free-form objects unless explicitly marked with x-kubernetes-preserve-unknown-fields. In return you get pruning, where unknown fields are dropped instead of stored, defaulting from the schema, and server-side validation with useful error messages. kubebuilder generates this from Go struct tags and markers, such as // +kubebuilder:validation:Minimum=1, Enum, MaxLength and Pattern.

Prefer closed, typed fields over map[string]string and raw JSON blobs. Prefer enums to free strings for modes. Set sensible defaults in the schema (// +kubebuilder:default=3) so the stored object is explicit. And keep the spec free of anything the controller computes; that belongs in status.

CEL validation rules

CEL validation rules, written as x-kubernetes-validations in the schema, let you express cross-field constraints without running a validating webhook: “if storage.mode is persistent then storage.size must be set”, or “spec.replicas may not decrease” using oldSelf with a transition rule. The rules run inside the API server, so there is no network hop and no webhook to keep available. CEL validation for CRDs has been generally available since Kubernetes 1.29, to the best of my knowledge from the release notes; verify against your minimum supported cluster version.

The tradeoff is expressiveness. CEL is a deliberately limited expression language with cost limits enforced when the CRD is admitted; unbounded lists and strings make rule cost estimates exceed the budget, so add maxItems and maxLength bounds. Anything that needs a lookup of another object, an external call, or mutation still needs a webhook. For the broader picture of replacing webhooks with in-process policy, see our guide to ValidatingAdmissionPolicy and CEL.

Versioning and conversion

A CRD can serve several versions at once, but exactly one is the storage version in etcd. Moving from v1alpha1 to v1beta1 to v1 follows the usual Kubernetes deprecation discipline: add the new version, keep serving the old, flip the storage version, migrate stored objects, then stop serving the old one. If the schemas are identical, the default None conversion strategy only rewrites apiVersion. If they differ, you need a conversion webhook, which becomes a hard dependency of every read and write of that resource.

Treat the conversion webhook as critical infrastructure. If it is down, the API server cannot list or watch the resource, which can take your own controller’s cache down with it. The pattern that limits the blast radius is hub-and-spoke conversion in controller-runtime: designate one version as the hub, and implement conversion from each other version to the hub and back, so the number of conversion functions grows linearly rather than quadratically. Try hard to make changes additive and avoid needing conversion at all; adding an optional field is free.

Also remember that the etcd storage cost of large custom resources is real. The API server’s request size limit is bounded by etcd’s default maximum request size of about 1.5 MiB, so never put unbounded data into a spec or status. Our write-up on the etcd 3.7 control plane explains the storage layer that these objects live in.

Running It in Production: Leader Election, RBAC and Testing

Leader election and availability

Reconcilers assume one active writer per object. If you run two replicas of the manager for availability, enable leader election so only one runs controllers at a time. controller-runtime implements this with a Lease object in the coordination.k8s.io group: replicas race to acquire and renew it, and the loser stays warm, with its caches synced, ready to take over. Set LeaderElection: true and a stable LeaderElectionID in the manager options, and add the Lease permissions to the role.

Leader election gives you failover, not horizontal scale. A standby replica adds resilience but no throughput. To scale work, increase MaxConcurrentReconciles per controller, which allows that many different keys to be processed in parallel while still serialising each individual key. If you need real sharding across processes, you must partition by label or namespace yourself, since a single-leader manager owns the whole resource type.

Be careful with failover timing. Lease duration, renew deadline and retry period defaults trade off failover speed against API load, and a leader that loses connectivity must stop reconciling before the standby takes over. If your reconcile does long-running external calls, honour context cancellation, or two instances may briefly act at once.

RBAC by least privilege

An operator is a privileged workload: it holds credentials that can create and delete resources cluster-wide. Generate RBAC from +kubebuilder:rbac markers, as in the skeleton, so the role tracks the code, then review the generated ClusterRole. Prefer namespaced Roles when the controller watches a limited set of namespaces, and restrict secrets access to named resources where you can.

Cache configuration is also a security boundary. By default the cache lists and watches a type cluster-wide, which requires cluster-wide list permission. Restricting the cache to specific namespaces, or using label selectors on the cache for types like Secrets and ConfigMaps, reduces both permission scope and memory.

Testing with envtest

Unit tests with a fake client are fast, and I use them for pure logic, but the fake client does not implement real API server behaviour: validation, defaulting, CEL rules, finalizer-gated deletion, garbage collection and resourceVersion conflicts. The controller-runtime release notes list a steady stream of fake-client fixes, which is a reminder of how much behaviour it has to emulate.

envtest closes that gap by starting a real kube-apiserver and etcd locally, with no kubelet and no scheduler. It loads your generated CRD manifests, so schema validation and CEL rules are exercised for real. The setup-envtest tool downloads matching binaries for a given Kubernetes version. Kubebuilder scaffolds a suite using Ginkgo and Gomega with a suite_test.go that starts the environment and registers your reconciler with a test manager.

Know what envtest cannot do. Without controller-manager, no built-in controllers run: a Deployment you create will never get Pods, a StatefulSet never reports ready replicas, and garbage collection of owned children does not happen. Your test must set those statuses itself or assert on the objects you created. For end-to-end behaviour with real workloads, add a smaller layer of tests in a kind cluster. Note too the kubebuilder v4.16.0 change in which the scaffolded AfterSuite checks testEnv.Stop() once instead of retrying, so a slow shutdown now fails the suite.

A useful testing sequence for finalizers: create the object, wait for the finalizer to appear, delete it, assert the object still exists with a deletionTimestamp, simulate external teardown, then assert it disappears. That single test catches most ordering bugs.

Trade-offs, Gotchas, and What Goes Wrong

Hot loops. The most common production incident is a reconciler that writes on every pass. Each write emits an event, the event enqueues the key, and the key runs again. Causes include non-deterministic serialisation (maps iterated in random order into an annotation), timestamps in spec or status, and updating status with no real change. Fix by comparing before writing, using equality.Semantic.DeepEqual or patch helpers that skip no-op patches, and by filtering events with predicates. GenerationChangedPredicate drops status-only updates for the primary resource, but it also drops metadata changes such as annotations and deletion timestamps in some setups, so test the deletion path after adding it.

Cache memory. Informers hold full objects. An operator that watches every Pod, Secret or ConfigMap in a large cluster can use gigabytes. Limit it with namespace and label selectors, use metadata-only watches via PartialObjectMetadata for types where you only need names and labels, and use the transform option to strip large fields such as managedFields before caching.

Event storms and restart thundering herds. On startup the cache fills and every existing object is enqueued. Ten thousand custom resources each making several external API calls will hit rate limits. Bound concurrency, add jitter to periodic resync intervals, and rely on the priority queue to favour new work. Cloud API quotas are usually the first real limit you hit, not CPU.

Stale reads. Because the cache lags, a reconcile that creates a child and immediately lists children may not see it, then create a duplicate. Deterministic names turn this into an AlreadyExists error instead of a duplicate. For the rare case that needs strong reads, use the uncached API reader, and treat the new read-your-writes option as experimental until it has more field time.

Webhook availability. Conversion and validating webhooks sit in the request path. A failing webhook with failurePolicy: Fail can block all writes to your resource, including the deletes needed to repair it, and in the worst case deadlock your own controller’s cache. Prefer CEL for validation, and scope any remaining webhooks tightly.

Stuck deletion. Finalizers without escape hatches, owner references pointing at a missing owner UID, and cleanup that errors forever all lead to Terminating objects. Alert on objects whose deletionTimestamp is older than a threshold.

The big one: operators as a liability. Every operator you write is a distributed system you must upgrade, test and be paged for. CRDs are cluster-scoped and shared, so two teams installing different versions of the same CRD conflict. Do not write one if a Helm chart or a Job satisfies the requirement.

Practical Recommendations

Figure 4: Choosing a build approach for a Kubernetes operator, with envtest as the shared testing step.

Start with the choice in Figure 4. If the problem is composing existing resources into a higher-level API, a composition engine such as Crossplane, or a function-based approach with KCL, gives you the same custom resource experience without a Go codebase to maintain. If you need real logic, such as backup scheduling, leader failover or multi-step upgrades with health checks, write a controller. Kubebuilder is the default choice for Go, and operator-sdk is reasonable when you want Helm or Ansible-based operators or its integration with Operator Lifecycle Manager packaging. Other languages work, but the Go ecosystem has by far the deepest tooling.

Then keep the design simple. One kind per concern, a small spec, conditions in status, and as few cross-resource dependencies as you can manage. Prefer watching over polling, prefer CEL over webhooks, and prefer additive API changes over conversion.

A pre-launch checklist:

Reconcile is idempotent and keyed only on the object name, with no dependence on event type.
Deterministic child names, controller owner references on all in-namespace children.
Finalizer added before external resources are created, with idempotent cleanup and a documented escape hatch.
Status written only through the status subresource, with observedGeneration and a Ready condition.
CRD has a structural schema, bounds on every list and string, and CEL rules for cross-field checks.
Leader election on, MaxConcurrentReconciles set deliberately, RBAC generated and reviewed.
Cache scoped by namespace or label, large fields stripped.
envtest suite covering the create, update and finalizer-delete paths.
Metrics and alerts on queue depth, reconcile error rate and long-Terminating objects.

Frequently Asked Questions

What is a Kubernetes operator?

A Kubernetes operator is a custom resource definition paired with a controller that watches those resources and acts to make the cluster match what they describe. It packages operational knowledge, such as how to deploy, back up and upgrade a stateful application, as software that runs in the cluster. The controller follows the same reconciliation pattern as built-in controllers like the Deployment controller.

What is the difference between a controller and an operator?

A controller is any process that runs a reconciliation loop against Kubernetes objects. An operator is a controller that manages an application using its own custom resources, adding domain-specific knowledge. The terms overlap heavily, and in practice the distinction is mostly about scope: the Deployment controller is not usually called an operator, while a PostgreSQL controller with a Database CRD is.

What does controller-runtime do for me?

Controller-runtime is the Go library that builds on client-go to provide a manager, shared caches, a cache-backed client, workqueues with rate limiting, leader election, webhook serving and metrics. You write a Reconcile function and a builder call that says which types to watch. It removes the boilerplate of raw informers, which is why kubebuilder scaffolds projects around it.

Why does my object get stuck in Terminating?

A finalizer is almost always the cause. The API server marks the object with a deletionTimestamp and waits for every finalizer to be removed. If the owning controller is down, crashing, lacks permission, or its cleanup keeps failing, the list never empties. Inspect metadata.finalizers, check the controller’s logs, and fix the cleanup rather than patching the finalizer away, unless the external resource is truly gone.

Should I use kubebuilder or operator-sdk?

For a Go operator, either works, and both rest on controller-runtime. Kubebuilder is the upstream Kubernetes SIGs project with the tightest alignment to controller-runtime releases. Operator-sdk adds Helm and Ansible operator types and Operator Lifecycle Manager packaging helpers. If you only need Go and plain manifests, kubebuilder is the simpler path.

How do I stop an infinite reconcile loop?

Find what your reconciler writes on every pass. Compare desired and existing state before updating, avoid non-deterministic data like timestamps or map-ordered output, and skip status updates when nothing changed. Add predicates to ignore events you do not care about, and look at the controller’s reconcile rate metric to confirm the loop has stopped after the fix.

Kubernetes Operators in Depth: Controller-Runtime, Reconciliation Loops, CRD Design and Finalizers (2026)