Raft Consensus vs Paxos: Leader Election, Log Replication and Failure Modes (2026)

Every Kubernetes cluster you have ever run depends on a handful of small processes agreeing, thousands of times a minute, about what happened and in what order. When that agreement stalls, the symptoms are bizarre: deployments that hang, a control plane that answers reads but refuses writes, a “leader changed” alert that fires every ninety seconds for no visible reason. Raft consensus is the protocol behind etcd, Consul, CockroachDB, TiKV and a long list of other systems that carry production state, and understanding it at the mechanism level is the difference between guessing and diagnosing.

Paxos is its older and more famous cousin, and the usual framing, “Raft is Paxos made easy”, hides more than it reveals. The two protocols make different structural choices about who may lead, how logs may diverge, and how membership changes. Those choices determine how each fails in the field. This guide compares them at the level of terms, votes, log matching, quorum arithmetic, clocks and disks, then walks through the failure modes that actually page people.

What this covers: the consensus problem and the FLP result, Raft terms, election and commit rules, how Paxos and Multi-Paxos differ, quorum math, membership change, pre-vote and leader leases, how etcd, Consul and CockroachDB apply them, tuning election timeouts, failure modes, and how to test with Jepsen-style methods.

Context and Background

State machine replication is the idea underneath all of this. You have a deterministic program, a key-value store for instance, and you want several machines to hold identical copies so that the service survives the loss of some of them. If every replica applies the same commands in the same order, they stay identical. Consensus is the machinery that makes a group of unreliable machines agree on that single ordered log of commands, even while some crash, restart, or become unreachable.

The theoretical boundary was drawn in 1985 by Fischer, Lynch and Paterson. The FLP impossibility result says that in a fully asynchronous system, where message delay is unbounded and you cannot distinguish a slow node from a dead one, no deterministic protocol can guarantee both safety and termination if even one node may crash. Practical protocols respond by choosing safety unconditionally and liveness only under partial synchrony, the model formalised by Dwork, Lynch and Stockmeyer: eventually the network and processes behave well enough, for long enough, that progress can be made. Raft and Paxos both make this deal. They never return a wrong answer because of timing, but they may stall if timing is bad enough. That is why timeouts, clocks and disk latency matter so much in operations, and why most “consensus bugs” in production are actually liveness failures.

Paxos came first. Lamport described it in “The Part-Time Parliament”, written in the late 1980s and published in 1998, and later restated it plainly in “Paxos Made Simple” (2001). It became the foundation of Google’s Chubby lock service and, in Multi-Paxos form, of many internal replicated stores. Its reputation for difficulty is partly earned: the original paper specifies single-value agreement, and the engineering gap to a replicated log with leader election, reconfiguration and snapshots is left to the implementer. Chandra, Griesemer and Redstone’s account of building Paxos for Chubby is a famous catalogue of that gap.

Raft was designed by Diego Ongaro and John Ousterhout and published as “In Search of an Understandable Consensus Algorithm” at USENIX ATC 2014, with Ongaro’s PhD thesis filling in details such as membership change, log compaction and client interaction. Its stated goal was understandability, achieved by decomposing the problem into leader election, log replication and safety, and by restricting the design space: a strong leader, a log that never has holes, and one well-defined mechanism per concern. Today it is the default choice for new systems. etcd, which Kubernetes uses as its datastore, is a Raft implementation; for how replicated state interacts with geography see our guide to multi-region active-active database architecture, which builds on exactly these primitives.

One more distinction frames the rest of the article. Consensus protocols of this family are crash-fault tolerant: they assume nodes fail by stopping, not by lying. Byzantine fault tolerance (PBFT, HotStuff and friends) needs 3f+1 replicas rather than 2f+1 and is a different tool for a different threat model. Everything below assumes crash faults plus an unreliable network that can drop, delay, duplicate and reorder messages.

How Raft Works: Terms, Elections and the Replicated Log

Direct answer: Raft elects one leader per numbered term, and only that leader accepts client writes. The leader appends each command to its log, replicates it to followers, and marks it committed once a majority has stored it. Safety comes from the election rule: a candidate can only win if its log is at least as up to date as a majority of voters.

Figure 1: Raft node states. Followers become candidates when their election timer fires; a candidate that gathers a majority becomes leader, and any node that sees a higher term reverts to follower.

The diagram shows the three roles and every legal transition. A node starts as a follower. It stays a follower as long as it hears valid AppendEntries messages (heartbeats or log replication) from a current leader. Any node that sees a message carrying a higher term immediately adopts that term and steps down to follower, which is what makes stale leaders harmless.

Terms: a logical clock for leadership

Time in Raft is divided into terms, consecutive integers. Each term begins with an election and has at most one leader. Terms act as a logical clock: every RPC carries the sender’s term, a receiver with a smaller term updates itself, and a request from a sender with a stale term is rejected. This single rule, “higher term wins and everyone obeys it”, is how Raft detects that a deposed leader has been replaced without any wall-clock comparison.

Each server persists three things before answering any RPC that depends on them: its current term, the identity of the candidate it voted for in that term (votedFor), and its log. Persisting the vote is what enforces “at most one vote per term”, which in turn enforces “at most one leader per term”. If a node forgot its vote after a crash and voted twice in the same term, two candidates could each collect a majority. That is why a disk that lies about fsync can, in principle, break safety rather than merely liveness.

Leader election in detail

Each follower runs an election timer with a randomised timeout. The Raft paper suggests a range of 150 to 300 milliseconds as a workable example for a LAN, and the randomisation is the point: it spreads out candidacies so that usually one node times out first, wins, and starts sending heartbeats before the others expire. When the timer fires, the follower increments its term, votes for itself, resets the timer, and sends RequestVote RPCs to the others.

A server grants its vote if it has not already voted in that term and if the candidate’s log is at least as up to date as its own. “Up to date” compares the term of the last log entry first and the log length second. This is the election restriction, and it carries the whole safety argument: because committed entries exist on a majority, and any winning candidate needs votes from a majority, at least one voter holds every committed entry, and the candidate’s log must be at least as current as that voter’s. A leader therefore never needs to fetch missing committed entries from anyone; it already has them all.

If nobody gets a majority, which happens when two candidates split the vote, the term ends without a leader and timers expire again with fresh random values. Split votes are normal and self-healing, but with a poorly chosen timeout range (too narrow relative to network round trip) they can repeat many times, which is one route to a long unavailability window.

Log replication and the commit rule

Figure 2: One write through Raft log replication. The leader fsyncs locally, sends AppendEntries, advances commitIndex on majority acknowledgement and tells followers the new commit index in later messages.

Once elected, the leader handles all client commands. For each command it appends an entry (term, index, command) to its own log, writes it to stable storage, and sends AppendEntries to every follower in parallel. Each AppendEntries carries prevLogIndex and prevLogTerm: the index and term of the entry immediately preceding the new ones. A follower accepts only if its own log contains an entry at prevLogIndex with term prevLogTerm; otherwise it rejects, and the leader decrements that follower’s nextIndex and retries. This consistency check, applied inductively, yields the Log Matching property: if two logs contain an entry with the same index and term, then the logs are identical in all entries up to that index.

When a majority of servers, counting the leader, have stored an entry, the leader advances commitIndex and applies the entry to its state machine, then responds to the client. Followers learn the new commitIndex from subsequent AppendEntries and apply entries in order. There is one subtle rule worth memorising: a leader commits entries from earlier terms only indirectly, by committing an entry from its own current term that follows them. The Raft paper illustrates the reason with its Figure 8 scenario, in which an old-term entry replicated to a majority can still be overwritten by a later leader. Implementations typically have a new leader append a no-op entry on election so that earlier entries become committed promptly.

Conflicting entries on a follower, left over from a deposed leader that never reached a majority, are simply overwritten by the new leader’s log. Because such entries were never committed, no client was ever told they succeeded, and discarding them is safe. The leader’s log is only ever appended to; this “leader append-only” property is part of what makes Raft easier to reason about than designs where several proposers can write to a slot.

Worked quorum arithmetic

A cluster of n voting members needs a quorum of floor(n/2)+1 to elect or commit, and so tolerates f = n minus quorum failures. Three nodes: quorum 2, tolerates 1. Four nodes: quorum 3, tolerates 1, so the fourth node adds cost and an extra replication target without adding fault tolerance. Five nodes: quorum 3, tolerates 2. Seven nodes: quorum 4, tolerates 3. This is why consensus clusters are almost always sized 3 or 5.

Larger is not better. Commit latency is governed by the quorum-th fastest acknowledgement, and the leader must send every entry to every follower, so write fan-out and leader network cost grow with n. Illustratively, a five-node cluster in three availability zones laid out 2-2-1 survives the loss of any single zone (the worst case removes two nodes and leaves three, which is still a quorum), whereas a 3-node cluster stretched over two zones does not, because losing the zone with two nodes leaves one. These are illustrative layouts, not a recommendation for your topology.

Paxos, Multi-Paxos and Where Raft Actually Differs

Paxos is often presented as a rival to Raft; a more accurate description is that Raft is a specific, opinionated way to build a Multi-Paxos-style replicated log. The differences lie in the details, and the details are where operational behaviour comes from.

Figure 3: Single-decree Paxos runs two phases for every value. Multi-Paxos and Raft amortise phase one by electing a stable leader and then running only the accept phase per log slot.

Single-decree Paxos in one paragraph

Basic Paxos decides one value. A proposer picks a unique, increasing proposal number n and sends Prepare(n) to a majority of acceptors. Each acceptor promises not to accept anything numbered below n and replies with the highest-numbered proposal it has already accepted, if any. If the proposer collects a majority of promises, it must propose the value of the highest-numbered accepted proposal reported, or its own value if none was reported, in an Accept(n, v) message. A value is chosen once a majority of acceptors accept the same proposal. The constraint “adopt the highest previously accepted value” is the heart of the safety argument, and it is why two competing proposers cannot choose different values.

Two proposers can also livelock by repeatedly pre-empting each other with ever higher proposal numbers, the dueling proposers problem. The standard remedy is to elect a distinguished proposer, which is already a step toward Multi-Paxos and Raft.

Multi-Paxos is a family, not an algorithm

Multi-Paxos runs one Paxos instance per log slot and lets a stable leader skip the prepare phase for subsequent slots, so steady state costs one round trip to a majority per command, the same as Raft. But “Multi-Paxos” was never specified in complete detail by the original papers, and real implementations differ: whether slots can be filled out of order, how gaps are repaired, how leaders are chosen, how reconfiguration is sequenced. Systems built on it, such as Spanner, Chubby and many internal Google stores, each embody their own variant. Raft’s contribution was to pin these choices down in a single specification.

The structural differences that matter

Dimension	Raft	Multi-Paxos (typical)
Log holes	Never; log is contiguous and prefix-consistent	Allowed; slots may be accepted out of order and gaps filled with no-ops
Who can become leader	Only a node whose log is at least as up to date as a majority	Any node; it learns missing entries from acceptors during phase one
Commit of old-term entries	Only indirectly via a current-term entry	Handled by re-proposing during phase one
Log divergence	Followers overwritten to match the leader	Value chosen per slot by highest accepted number
Specification completeness	Election, replication, membership, snapshots specified	Core agreement specified; the rest left to implementers
Concurrent proposals	Single leader per term	Multiple proposers possible per slot

The first two rows are the real trade. Raft pays a restriction, only up-to-date nodes may lead, to get simplicity: the leader never has to recover missing data, and log repair flows in one direction. Multi-Paxos permits any node to lead and pays complexity during recovery, because a new leader must run phase one across the uncommitted suffix and reconcile what acceptors report. In exchange, Multi-Paxos can allow more parallelism in pipelining and out-of-order commit, which some systems exploit for throughput. Variants such as Flexible Paxos (Howard, Malkhi and Spiegelman) observed that the phase-one and phase-two quorums only need to intersect with each other, not each be a majority, which permits trading election quorum size against replication quorum size in carefully designed systems.

Zab and the wider family

ZooKeeper uses Zab, a primary-backup atomic broadcast protocol that shares Raft’s strong-leader shape: epochs play the role of terms, and a recovery phase synchronises followers to the leader. Viewstamped Replication, the 1988 protocol from Oki and Liskov, with its 2012 revision, is another close relative and arguably the closest ancestor of Raft’s structure. The family resemblance is strong enough that, in practice, a competent engineer who understands Raft’s invariants can reason about the others.

What this means for engineers

For almost every team the right decision is not Raft or Paxos in the abstract but which library or system to build on. If you are writing a new replicated component, use a battle-tested Raft library (etcd’s raft package, hashicorp/raft, raft-rs from TiKV, Dragonboat, openraft) rather than implementing the protocol yourself. If you are consuming a system, learn the protocol it implements well enough to read its metrics. The next section covers the production machinery, membership change, pre-vote and leases, that separates the textbook from the product.

Production Machinery: Membership Change, Pre-Vote, Leases and Reads

The Raft paper’s core algorithm assumes a fixed cluster. Real clusters grow, shrink, lose disks and get replaced, and real networks fail in odd shapes. Ongaro’s thesis and the implementations that followed add the mechanisms below.

Membership change: joint consensus and single-server changes

Switching directly from one configuration to another is dangerous, because different servers adopt the new configuration at different times. During the window, two disjoint majorities can exist, one of the old configuration and one of the new, and two leaders could be elected in the same term. Raft’s original answer is joint consensus: the cluster first commits a transitional configuration, C_old,new, in which agreement requires separate majorities from both the old and the new sets. Once C_old,new is committed, the leader commits C_new alone. At no point can two independent decisions be made.

The thesis also describes a simpler alternative: add or remove one server at a time. Any majority of the old configuration overlaps with any majority of a configuration that differs by a single node, so single-step changes are safe without a joint phase. Most implementations take this route, and etcd works this way, though the thesis route and implementation details have had subtle bugs over the years. Notably, the Raft community discovered and documented a bug in the single-server change algorithm involving a leader that changes configuration in successive terms before committing a no-op, which implementations fixed by requiring a current-term commit before a new change. Treat membership change as a place where reading the library’s documented constraints is mandatory.

A related operational feature is the non-voting learner, which etcd introduced as a supported member type in 3.4. A learner receives the log but does not count toward quorum, so a new replica can catch up (potentially transferring a large snapshot) without risking quorum availability, and is promoted to voter only once it is in sync. This directly fixes a classic failure: adding a new voter to a three-node cluster temporarily raises the quorum to three of four, so a slow or misconfigured newcomer can leave the cluster one failure from losing writes.

Pre-vote and check-quorum

Consider a node partitioned from the leader but still able to run its timer. It repeatedly times out, increments its term and campaigns, getting no votes. Its term climbs. When the partition heals, its high term reaches the healthy cluster, and the healthy leader, obeying the higher-term rule, steps down. A single flaky node has forced an unnecessary election and a window of unavailability.

Pre-vote, described in Ongaro’s thesis, closes this hole. Before incrementing its term, a would-be candidate sends a pre-vote request asking whether peers would vote for it; peers grant it only if they have not recently heard from a leader and the candidate’s log is up to date. Only after a majority says yes does the candidate increment its term and run the real election. An isolated node never gets pre-votes, so its term stays put. etcd added pre-vote in the 3.4 line (the feature is controlled by a configuration flag in the raft library and server), and many other implementations offer it.

Check-quorum is the complementary leader-side guard: a leader that cannot reach a majority within an election timeout steps down on its own. This matters for clients because a leader stranded in a minority partition would otherwise continue to believe it is leader, accept writes it can never commit and, depending on the read path, serve stale reads. Together the two mechanisms mean that leadership is only held while quorum contact exists and only challenged when a real majority has lost the leader.

Reads: log reads, ReadIndex and leader leases

Writes go through the log. Reads need not, but they must be linearisable if you want the guarantee that a read reflects every acknowledged write. Three strategies exist:

Log reads: treat every read as a command and replicate it. Correct but slow.
ReadIndex: the leader records its current commitIndex, confirms leadership with a round of heartbeats to a majority, waits until its state machine has applied through that index, then serves the read locally. This costs one round trip to a quorum but no disk writes, and it needs no clock assumptions. etcd’s default linearizable reads use this approach.
Lease reads: the leader assumes that for a duration after its last successful quorum heartbeat no other leader can exist, and serves reads locally without any network round trip. This is faster and is where clock assumptions enter: if the leader’s clock runs fast relative to followers, or a long pause lets the lease expire invisibly while the process believes it still holds it, a new leader can be elected and accept writes while the old one serves a stale read.

The safety of lease reads therefore rests on bounded clock drift and bounded pauses, not on the protocol itself. That is the exact same reasoning that makes time synchronisation relevant to this topic; the trade-offs between PTP, NTP and chrony are covered in our guide to PTP, NTP and chrony time synchronization. Systems such as CockroachDB use leader leases tied to clock bounds, and its documentation spells out the max clock offset assumption; the specific lease mechanism has evolved across versions, so consult the docs for the version you run rather than assuming.

Snapshots and log compaction

A log that grows forever is unusable. Each server periodically snapshots its state machine and discards log entries up to the snapshot’s last included index. A follower so far behind that the leader no longer has the entries it needs receives the snapshot through an InstallSnapshot RPC. Snapshot cadence is a trade: frequent snapshots keep the log and restart time small but cost I/O and CPU; infrequent ones shift cost to restart and to lagging followers. In etcd the backend database is compacted separately from the Raft log, which is why operators must run compaction and defragmentation of the key-value history in addition to snapshots, and why the default backend size quota (2 GiB by default in etcd documentation, adjustable) produces a cluster-wide write-refusing alarm when exceeded.

How the Big Systems Use It

Knowing the protocol is half the job; the other half is knowing which knobs your system exposes.

etcd consensus and Kubernetes

etcd is the reference Raft deployment for most cloud-native teams. Its core raft library is separated from the server and reused by other projects, which gives the implementation unusually broad exposure. The two timing parameters are the heartbeat interval (default 100 ms in the documentation) and the election timeout (default 1000 ms). The etcd tuning guide advises setting the heartbeat near the maximum round-trip time between members and the election timeout to at least ten times that round trip, and warns that election timeout must also be several times the heartbeat. The Kubernetes API server is effectively a stateless layer over etcd; when etcd loses its leader, the API server’s writes fail, even if reads served from its watch cache keep working for a short while. For GPU scheduling workloads whose control-plane churn leans heavily on etcd writes, see our walk-through of Kubernetes Dynamic Resource Allocation for GPU scheduling, where object volume per claim makes the write path worth watching.

Consul and the hashicorp/raft library

Consul servers use HashiCorp’s Raft implementation for catalog and key-value state. The documented guidance is to run three or five servers, never an even number, and to treat the server count as a fault-tolerance decision rather than a scale decision, since extra servers do not add write throughput. Consul exposes Raft-related timing through a performance multiplier setting (raft_multiplier) that scales the underlying timeouts; the documentation lists defaults for production and recommends keeping the value low on healthy hardware.

CockroachDB, TiKV and Raft groups at scale

Systems that shard data run many Raft groups. CockroachDB replicates each range (a contiguous slice of key space) as its own Raft group, and TiKV does the same with regions using the raft-rs library. This changes the failure model: instead of one leader for the cluster there are thousands of leaders, spread across nodes, so a node failure triggers many small elections at once, and heartbeats have to be coalesced or batched or the per-group traffic would swamp the network. These systems add leader leases or equivalents for fast reads, and a rebalancing layer that moves leaders and replicas. The operational lesson is that elections become a load event, not just a correctness event, and that disk and network saturation on a single node can cause a wave of lease transfers across thousands of ranges.

Trade-offs, Gotchas, and What Goes Wrong

Consensus protocols are proved safe under their model, and most production incidents come from the environment violating the model’s liveness assumptions or from implementation details. The following failure modes recur across Raft and Paxos systems.

Figure 4: A triage tree for flapping leaders. Slow disks, jittery networks and asymmetric partitions each produce the same symptom through different mechanisms, and each has a different fix.

Disk fsync stalls

The leader must make an entry durable before it counts toward the commit, and each follower must persist before acknowledging. So commit latency includes at least one fsync per replica on the critical path. If fsync latency spikes, because of a noisy neighbour on shared storage, a burst of compaction writes, or a cloud volume hitting its IOPS or throughput cap, heartbeats are delayed too, since the same thread or I/O path may be busy. When the leader’s heartbeat misses the followers’ election timeout, they campaign, and you see a leader change caused entirely by storage. etcd exposes wal_fsync_duration histograms for exactly this reason, and its hardware guidance recommends a low-latency dedicated disk (SSD-class) for the data directory; its documentation cites a 99th percentile fsync target in the low tens of milliseconds at most. Alert on the fsync p99 before you alert on leader changes. Never put a Raft log on a volume shared with a database doing bulk loads.

Split votes and mistuned timeouts

If the randomised election window is narrow relative to the network round trip, two candidates frequently start at nearly the same time and split the vote. The cure is a wider randomisation range, not simply a longer base timeout. The reverse mistake also exists: setting the election timeout too low on a cross-region cluster causes spurious elections on ordinary jitter, and setting it too high makes real failures take that long to detect. Failover time is bounded below by roughly the election timeout plus the election round trip, so the timeout directly sets your recovery-time floor.

Asymmetric partitions and flapping leaders

Real partitions are rarely clean. A common shape is a node that can send but not receive (or vice versa). A node that receives no heartbeats starts campaigning while the leader can still reach everyone else, disrupting the stable leader each time its higher term arrives. This is the exact scenario that pre-vote and check-quorum address; without them, a single bad link or an over-subscribed virtual NIC can produce repeated disruptive elections. The Raft thesis discusses disruptive servers of this kind directly. If your Raft library or version lacks pre-vote, upgrading is among the highest-value fixes available.

Process pauses and CPU starvation

A garbage-collection pause, a VM live-migration, a container CPU throttle or a noisy CPU-steal period can freeze a node for longer than the election timeout. A paused leader resumes believing it is still leader; its first RPC reveals a higher term and it steps down, which is safe. But a pause is indistinguishable from a crash to everyone else, so election churn follows. On Kubernetes, CPU limits that throttle an etcd pod below what it needs are a classic root cause. Pin or reserve CPU for consensus members and keep their processes on a runtime with predictable pauses.

Learner, quorum and snapshot traps

Adding a voter before it has caught up can leave you one failure from losing quorum, as noted earlier; use learners. A large snapshot transfer to a new member can saturate the leader’s network and slow commits for everyone; throttle or schedule it. Quorum loss, meaning a majority of members permanently gone, cannot be recovered by the protocol: you must restore from a backup or force a new cluster from surviving data, accepting possible loss of acknowledged writes. Restoring one member from an old snapshot while the others progress is a safety hazard in itself, because that member has “forgotten” votes and entries it previously acknowledged.

Clocks, leases and read anomalies

If a system uses leader leases for reads, correctness silently depends on bounded clock drift. A step in system time, an NTP misconfiguration or a paused VM can cause a stale read without any visible protocol error. If you cannot bound clocks, prefer ReadIndex-style reads. And do not confuse the consensus layer with application semantics: a client that times out on a write does not know whether the write committed, so retries need idempotency tokens or a session mechanism, which Raft’s client interaction section recommends.

When consensus is the wrong tool

Consensus buys linearizability at the price of a majority round trip per write, so a single leader’s throughput and the cross-site latency cap your write rate. For data that merges naturally, such as counters, sets or collaborative state on edge devices that go offline, conflict-free replicated data types avoid coordination entirely; our piece on CRDTs for digital twin synchronization shows where eventually consistent merge beats a leader, and a hybrid design with a small Raft core for metadata and CRDTs for bulk state is often the better architecture.

Tuning Election Timeouts and Testing with Jepsen-Style Methods

A worked tuning example (illustrative numbers)

Suppose a three-node etcd-style cluster spans three availability zones in one region, with a measured intra-region round-trip time of 2 ms at median and 15 ms at p99.9 during a noisy period, and an fsync p99 of 8 ms. A sensible starting point follows the documented rule of thumb: heartbeat interval near the worst expected round trip, say 50 to 100 ms, and election timeout at least ten times the round trip, say 1,000 ms. At that setting a leader that stops responding is replaced after roughly one to two seconds, and a transient 200 ms stall does not trigger a change. If the same cluster were stretched across regions with a 70 ms round trip, a heartbeat around 100 to 150 ms and an election timeout of several seconds would be needed, and write latency would now include that 70 ms hop for the nearest follower on every commit. The numbers are illustrative; measure your own round trips and fsync distribution and set the knobs from the data.

A short checklist for any change: raise heartbeat and election timeout together and keep the ratio at least five to ten; change one variable at a time; observe the leader-change counter and the proposal-failed and proposal-pending metrics for a day; and rehearse the failure you tuned for by killing a leader on purpose.

Observability signals worth alerting on

Leader changes per hour: more than a small number in a healthy cluster signals trouble.
Disk fsync and backend commit latency percentiles.
Heartbeat send failures and “took too long” warnings in the logs.
Proposals pending and failed: a growing backlog means the leader cannot commit.
Raft term: a rapidly climbing term indicates repeated elections.
Follower lag (match index gap) and snapshot transfer counts.

Jepsen-style testing

Kyle Kingsbury’s Jepsen framework tests distributed databases by running concurrent operations against a real cluster while injecting faults, process kills, pauses, clock skew and network partitions, then checking recorded histories against a consistency model such as linearizability with the Knossos or Elle checkers. Public Jepsen analyses have covered etcd, Consul, CockroachDB, TiDB and others, and they have found real bugs in systems built on proven protocols, which illustrates the central point: a correct algorithm does not guarantee a correct implementation. Several reports found problems in lease handling, membership change and storage layers, not in the core election logic.

For your own systems, build a nemesis suite even if you do not use Jepsen itself. Partition the leader away from a majority; partition a single follower into a one-way link; pause the leader with SIGSTOP past the election timeout then resume it; kill nodes during membership change; skew clocks forward and backward on the leader; restore a node from stale state; fill a disk. After each scenario assert that every acknowledged write is present, no two clients saw conflicting committed values, and the cluster recovers without manual intervention. Formal models help as well: the Raft authors published a TLA+ specification, and TLA+ and model checkers have been used to find bugs in membership change.

Practical Recommendations

Choose by the job. If you consume a system, treat Raft as an operational discipline: give consensus members dedicated, low-latency storage; deploy an odd number of members, three for most clusters and five where you must survive two simultaneous failures or a zone loss plus a node loss; spread members across failure domains with latency you have measured; and never run them on oversubscribed or throttled CPU. If you build a system, use a mature Raft library with pre-vote, check-quorum and learner support, and spend your effort on snapshots, client idempotency and testing rather than on the protocol itself. Pick Multi-Paxos or a Flexible Paxos variant only if you have a specific need, such as unusual quorum shapes, and the expertise to maintain it.

Match the consistency tool to the data. Keep small, critical, strongly consistent metadata in consensus, such as configuration, locks, leader election for other services and schema. Push high-volume, mergeable or geo-distributed state into replication models that do not need a global leader, and scope Raft groups so that cross-region write latency is a deliberate choice, as covered in the multi-region architecture article linked earlier.

Checklist before go-live:

Odd member count, 3 or 5, spread over distinct failure domains.
Heartbeat and election timeout set from measured round trips; election timeout at least ten times RTT.
Pre-vote and check-quorum enabled where available.
Dedicated low-latency disk; alert on fsync p99.
Learners used for adding members; one membership change at a time.
Tested restore procedure from backup, including quorum-loss recovery.
Clock sync monitored if leases are used; otherwise prefer ReadIndex reads.
Leader-kill, partition and pause scenarios rehearsed in a staging cluster.
Client retries are idempotent.
Dashboards and alerts on leader changes, term growth, proposal backlog and follower lag.

Frequently Asked Questions

Is Raft better than Paxos?

Neither dominates. Raft is easier to understand and implement completely, because its specification covers election, replication, membership change and snapshots, which is why most new systems adopt it. Multi-Paxos can allow more flexible leadership and out-of-order commit, at the cost of a less complete specification and more complex recovery. Both give the same safety guarantees under crash faults and need a majority round trip per committed write in steady state. Choose a mature implementation over an abstract protocol preference.

What is a leader election in Raft and how long does it take?

Leader election starts when a follower stops receiving heartbeats for longer than its randomised election timeout. It becomes a candidate, increments the term, votes for itself and requests votes; a majority makes it leader. Failover time is roughly the election timeout plus a network round trip, so a one-second timeout means about one to two seconds of write unavailability. Split votes repeat the process with new random timeouts, extending the delay.

How many nodes does a Raft cluster need?

A cluster of n voters needs floor(n/2)+1 for a quorum, so three nodes tolerate one failure and five tolerate two. Four nodes tolerate only one, the same as three, so even counts waste resources. More than five or seven is rare because every write is replicated to every member and the leader’s load grows, while fault tolerance rises slowly. Choose five when you need to survive a zone outage plus a single node failure.

Why does etcd keep changing leader?

Frequent leader changes in etcd usually stem from slow disks, network jitter or CPU throttling delaying heartbeats beyond the election timeout, not from a protocol bug. Check the wal fsync latency percentiles, round-trip times between members, container CPU limits and logs showing heartbeats took too long. Enabling pre-vote protects against isolated nodes disrupting a healthy leader, and raising heartbeat and election timeout together can ride out jitter, at the cost of slower failure detection.

What is joint consensus in Raft?

Joint consensus is Raft’s original technique for changing cluster membership safely. The leader first commits a transitional configuration that requires separate majorities from both the old and new member sets, then commits the new configuration alone. This prevents two disjoint majorities from existing at once and electing two leaders. Many implementations instead add or remove one server at a time, which is safe because consecutive majorities overlap, and etcd also offers non-voting learners for catch-up.

Does Raft guarantee linearizable reads?

Not automatically. Reads served from a follower or from a leader that has been deposed can be stale. To get linearizable reads, either route them through the log, use ReadIndex, where the leader confirms leadership with a quorum before serving, or use leader leases that rely on bounded clock drift. ReadIndex needs no clock assumptions but costs a quorum round trip; leases are faster but risk stale reads if clocks or pauses exceed the assumed bounds.