PostgreSQL High Availability: Patroni, Streaming Replication, Failover and PITR (2026)

PostgreSQL ships with the raw parts of a highly available database and none of the judgment. It will stream write-ahead log (WAL) to as many standbys as you like, it will promote a standby when told to, and it will happily let two nodes both believe they are the primary if nobody stops them. Everything between “a replica exists” and “the application survives the loss of a node without losing a committed transaction” is machinery you have to assemble, tune and, most importantly, rehearse.

That assembly is what PostgreSQL high availability actually means in production: a replication topology, a consensus-backed decision about who the primary is, a way to route clients to that primary, a way to bring the dead primary back safely, and a backup chain that survives the failure modes replication cannot. Replication protects you from a dead machine. It does nothing for a bad DELETE, because the bad DELETE replicates beautifully.

What this covers: how WAL and streaming replication work and what each synchronous_commit level really guarantees; Patroni’s architecture and its leader lock in a distributed configuration store; the timing arithmetic of failover; fencing and split-brain prevention; pg_rewind; client routing; quorum commit and strict mode; backups and point-in-time recovery with pgBackRest; worked RPO and RTO numbers; Kubernetes operators; upgrades; and the failure modes that actually page people.

Context and Background

Three generations of PostgreSQL HA tooling are worth separating. The first was shared storage with a cluster manager: a SAN volume, Pacemaker, and a floating IP. It works, but the storage is a single failure domain and failover means a cold crash-recovery start. The second was replication with a heartbeat tool (repmgr, Pacemaker with the PostgreSQL resource agent), where the cluster manager runs on every database host and negotiates among themselves. The third, which dominates new deployments, delegates the “who is primary” question to an external consensus store and runs a small agent beside each PostgreSQL instance. Patroni, originally from Zalando, is the best-known member of that third family, and operators such as CloudNativePG apply the same idea using the Kubernetes API as the consensus layer.

The design insight behind the third generation is that PostgreSQL nodes should not vote on their own leadership. Electing a leader is a distributed consensus problem, and consensus is hard to get right. etcd, Consul and ZooKeeper already implement it (Patroni also supports the Kubernetes API as its store; check the Patroni documentation for the current list of supported backends). The agent simply holds a lease in that store and refuses to run PostgreSQL as a primary when it cannot renew the lease. The database stays unaware of the election.

A second framing helps: HA and disaster recovery are different promises. HA targets a recovery time objective (RTO, how long you are down) measured in seconds to a minute, and a recovery point objective (RPO, how much committed data you may lose) of zero or near zero, within one failure domain such as a region or availability zone. Disaster recovery covers the correlated failures: a corrupted table, an operator error, a ransomware event, a lost region. Replication gives you the first. WAL archiving with base backups gives you the second. Mature setups run both and treat neither as a substitute for the other. The PostgreSQL project’s own documentation on high availability, load balancing and replication draws the same distinction, and our guide to multi-region active-active database architecture covers what changes when the failure domain becomes a whole region.

Finally, a note on versions. PostgreSQL 18 is the current major release in the documentation at the time of writing, and the behaviours described here are stable across the supported majors unless stated. Where a setting arrived in a specific release, this article says so. For what the next major brings to replication, see our PostgreSQL 19 upgrade guide.

The Reference Architecture: WAL, Replication Slots and a Consensus-Backed Leader

A highly available PostgreSQL cluster is a primary that emits WAL, one or more standbys that replay it, a distributed configuration store (DCS) that holds the leader lock, a Patroni agent on every database host that reconciles the two, and a connection-routing layer that follows the leader. An archive of WAL and base backups sits beside all of it for recovery that replication cannot provide.

Figure 1: Reference architecture. Patroni agents renew a leader key in the DCS, the primary streams WAL to standbys and archives it to object storage, and clients reach the primary through a routing tier.

The diagram shows the control plane (the DCS and the agents) separate from the data plane (WAL flowing from primary to standbys and to the archive). That separation is the point. The data plane can keep running briefly without the control plane, but only the control plane can authorize a change of leadership. Most HA incidents are the control plane and data plane disagreeing about reality, so it pays to understand each separately.

WAL, streaming replication and the guarantee ladder

Every change in PostgreSQL is first written to the write-ahead log. A commit is durable on the primary when its commit record has been flushed to local disk. Physical streaming replication ships that same byte stream to standbys over a long-lived connection served by a WAL sender process on the primary; the standby’s WAL receiver writes it to disk and the startup process replays it. A standby is therefore a byte-for-byte copy of the primary’s data files, which is why it must run the same major version and architecture, and why it can serve read-only queries when hot_standby is enabled.

By default replication is asynchronous: the primary reports commit success without waiting for any standby. That is fast and it is the source of every “we lost the last few seconds” story, because a primary that dies with unsent WAL takes those commits with it. The synchronous_commit setting controls how far a commit waits, and the PostgreSQL documentation defines the ladder precisely:

Level	Commit waits for	Survives
off	Nothing; success returns before local flush (delay up to three times wal_writer_delay)	Database stays consistent, but recent commits can be lost on a primary crash
local	Local WAL flush only	Primary process or host restart, but not primary loss
remote_write	Standby has received and written WAL to its filesystem	A standby PostgreSQL crash, but not a standby OS crash
on (default)	Standby has flushed WAL to durable storage	Loss of the primary plus an OS crash on the standby
remote_apply	Standby has replayed the record	Same as on, and the commit is visible to standby queries

Two details deserve emphasis. First, the levels beyond local only matter when synchronous_standby_names is non-empty; with it empty, remote_write and friends behave like on against the local disk. Second, the setting can be changed per transaction with SET LOCAL synchronous_commit, so a system can pay the replication latency for a payment record and skip it for a session-activity row. Mixing durability levels inside one cluster is a legitimate, underused tool.

remote_apply is the only level that gives read-your-writes against a standby, and it is also the most expensive because commit latency now includes replay time, which grows under a heavy single-threaded recovery workload. Use it only where a read-after-write guarantee on replicas is a stated requirement.

Replication slots and the disk-fill hazard

A replication slot tells the primary to retain WAL until a particular consumer has received it. Slots make standbys robust to long disconnects, because the primary will not recycle WAL the standby still needs. They also create the most common self-inflicted HA outage: an abandoned or stalled slot pins WAL forever. By default max_slot_wal_keep_size is -1, meaning slots may retain unlimited WAL, and the pg_wal directory grows until the volume fills and the primary stops accepting writes.

The mitigation is a deliberate cap. Setting max_slot_wal_keep_size to a value such as 50GB lets the primary invalidate a slot that falls too far behind rather than die. The cost is that the lagging standby can no longer catch up from the primary and must be rebuilt, or fed from the WAL archive via restore_command. The pg_replication_slots view exposes a wal_status column per slot that reports how close a slot is to losing its WAL; alert on anything other than the healthy states. PostgreSQL 18 also added idle_replication_slot_timeout, default 0 (disabled), which invalidates slots that have been inactive longer than the configured duration, checked at checkpoints. That is a useful backstop against forgotten logical-decoding slots, with the caveat that invalidation lag can be up to one checkpoint_timeout.

Patroni uses slots by default (use_slots is true), creating one per member so that each standby can reconnect without WAL gaps. That default is right, but it means the cap above is not optional hygiene; it is a required safety valve.

Why a DCS and a leader lock

Patroni’s agent runs a loop. On every iteration, the agent on the current primary renews a leader key in the DCS with a time-to-live (TTL). If it can renew, it keeps PostgreSQL running as primary. If it cannot, it must demote PostgreSQL before the TTL expires. The agents on standbys watch the key; when it vanishes they compare their replication position and race to create it. The winner promotes its local PostgreSQL and the rest repoint to it.

This turns split-brain prevention into a property of the lease protocol rather than a hope. A partitioned old primary that cannot reach the DCS cannot renew, so it demotes itself (or, as covered below, a watchdog resets the host). A new leader can only be elected after the old key has expired, so under correct timing there is a window with zero primaries, never two. The DCS cluster itself needs a majority to operate, which is why you run three or five DCS members across failure domains, and why the DCS becomes part of your availability budget.

Figure 2: Leader lock lifecycle. The old primary renews, loses contact, demotes; a healthy replica sees the key expire, wins the race and promotes.

The sequence matters in what it leaves out: nothing in it requires the old primary’s cooperation after it stops renewing. Correctness depends on the old primary noticing its own isolation within one TTL, which is the subject of the next section.

Failover Mechanics: Timing Budget, Fencing and Rejoining

Failover time in Patroni is the sum of three things: how long it takes to notice the primary is gone, how long it takes to agree on a replacement, and how long it takes to promote and reroute clients. The first term is governed by three parameters, and getting them wrong in either direction produces either needless failovers or unacceptable outages.

The three timers

According to the Patroni dynamic configuration documentation, ttl defaults to 30 seconds (minimum 20) and is the lifetime of the leader lock; loop_wait defaults to 10 seconds (minimum 1) and is how long the agent sleeps between cycles; retry_timeout defaults to 10 seconds (minimum 3) and bounds retries of DCS and PostgreSQL operations, so that short DCS or network blips do not demote a healthy leader. The documentation states the constraint that ties them together: loop_wait + 2 * retry_timeout <= ttl. With defaults, 10 + 20 = 30, which meets the limit exactly. That equality is not an accident; it means the defaults are the tightest safe configuration, and any change must move the three numbers together.

Why the constraint? The leader needs enough time inside one TTL to wait out a loop, fail a DCS write, retry once more, and still demote before another node can take over. If retry_timeout is too long relative to ttl, a leader could still be retrying (and believing itself leader) after a standby has already acquired the key. The inequality is the arithmetic form of “the old primary stops before the new one starts.”

A worked failover budget (illustrative)

Consider the defaults and a sudden power loss on the primary host. These numbers are illustrative reasoning from the timers, not a benchmark.

• Worst case, the primary renewed the key just before dying, so the key lives for up to ttl = 30 s.
• The standbys poll once per loop_wait, so they notice expiry up to 10 s later: up to 40 s elapsed.
• Leader race, lag comparison and pg_ctl promote typically add a few seconds (promotion of a healthy standby is fast; replaying a backlog of unapplied WAL lengthens it).
• Routing convergence (HAProxy health checks, DNS TTL, or a vIP move) adds its own term, commonly one to ten seconds depending on check intervals.

That yields a plausible envelope of roughly 35 to 55 seconds of write unavailability, with an average closer to 25 to 35 seconds because the primary rarely dies the instant after renewing. Halving it is possible by lowering ttl to the 20 s minimum, loop_wait to 5 and retry_timeout to 5 (5 + 10 = 15 <= 20), at the price of more DCS traffic and a lower tolerance for DCS latency spikes. The tradeoff is real: a tighter budget means a noisy network triggers more spurious failovers, and each spurious failover costs its own disruption.

The other path is the “Patroni alive, PostgreSQL dead” case. The documentation gives the worst-case formula loop_wait + primary_start_timeout + loop_wait, with primary_start_timeout defaulting to 300 seconds: Patroni will try to restart a crashed primary locally for up to five minutes before giving up and failing over, because a quick restart often beats a promotion. Setting it to 0 fails over immediately upon crash detection. Whether that default is right depends on how long your crash recovery takes; a database with a large WAL backlog may need many minutes to recover, but a five-minute grace period is long for a service with a 60-second RTO.

Figure 3: Failover time budget. Lock expiry, standby polling, promotion and client convergence add up to the user-visible RTO.

Choosing the candidate: lag and maximum_lag_on_failover

When the key expires, each standby evaluates itself. A standby that lags more than maximum_lag_on_failover bytes behind the last known primary position will not run for leader, because promoting it would discard more data than the operator has declared acceptable. This is an asynchronous-replication RPO knob: it says “I would rather be down than lose more than N bytes.” Among eligible candidates, the one with the most advanced WAL position wins. The setting is a ceiling on loss, not a guarantee of none; with asynchronous replication, even a fully caught-up standby may be missing the last unflushed transactions.

Fencing, watchdogs and split brain

Split brain is the state where two nodes accept writes as primary at once. It is destructive because the two histories diverge and cannot be merged; one must be discarded, and any data written to the loser is gone. Three layers of defence apply.

Layer one is the lease. The old primary demotes itself when it cannot renew within the loop. Demotion means stopping PostgreSQL and, in recent Patroni versions, restarting it as a standby or leaving it down for rejoin. This is software on the same machine that is misbehaving, and that is the weakness: if Patroni’s process is hung, paused by a hypervisor, or starved for CPU, it cannot execute the demotion at the moment the lease expires.

Layer two is the watchdog. Patroni can arm a Linux hardware or software watchdog device that it must keep pinging. If the agent stalls, the watchdog resets the machine, which is the strongest local fencing available without external hardware. The pattern is “if I cannot prove I am healthy, I stop existing.” Note the property this guarantees: the reset happens within the watchdog timeout even if the agent is wedged. It requires /dev/watchdog access and careful timeout tuning relative to ttl. Check the Patroni watchdog documentation for the exact safety_margin semantics for your version.

Layer three is the synchronous guarantee. If synchronous replication is enabled and the old primary is isolated, its commits cannot complete because no synchronous standby acknowledges them. The isolated primary can accept connections but its writes hang rather than commit. This does not remove the need for demotion, but it bounds the damage: transactions that were never acknowledged were never promised to clients.

A related Patroni feature is DCS failsafe mode (failsafe_mode, default false). Without it, a total DCS outage makes the leader demote because it can no longer renew the key, turning a control-plane failure into a full database outage. With failsafe mode, the leader may continue as primary as long as it can reach all the members it last knew about through Patroni’s REST API. This trades some partition-safety margin for survival of a DCS outage, and it should be enabled deliberately, with the understanding of how your network can partition. Verify the exact semantics against the version you run.

pg_rewind: bringing the old primary back

After failover the old primary has a WAL history that forked from the new timeline. Its tail may contain transactions that were never replicated. The brute-force fix is a fresh base backup, which for a multi-terabyte database takes hours. The pg_rewind documentation describes the shortcut: scan the target’s WAL from the last checkpoint before the fork, record every block it touched, copy exactly those blocks from the new primary, copy other files in full, then set up recovery to start at the failover checkpoint.

The prerequisites matter operationally. The cluster must have either data checksums or wal_log_hints = on, because pg_rewind needs to identify changed pages. full_page_writes must be on (the default). The old primary must have been cleanly shut down, and if not, pg_rewind will by default start it in single-user mode to complete crash recovery. The old primary’s pg_wal must still contain WAL back to the fork point, or pg_rewind with -c can fetch it using restore_command. Patroni’s use_pg_rewind defaults to false, so you must enable it explicitly, and enable the prerequisite at initdb time rather than discovering later that it is missing.

pg_rewind has a limitation worth stating plainly: if it fails midway, the target is not recoverable by retrying and you must take a new base backup. Patroni will fall back to reinitializing the member from a backup or from the leader when the rewind fails, but you should confirm that the fallback path (base backup bandwidth, available disk, pg_basebackup or pgBackRest delta restore) is fast enough for your RTO.

Routing Clients, Synchronous Modes and Quorum Commit

A failover is only visible to the business when clients follow the new primary. There is no PostgreSQL-native virtual IP, so the routing layer is a separate decision, and it is where an elegant HA design often leaks.

Four routing patterns

HAProxy with the Patroni REST API. Each Patroni agent exposes health endpoints; GET /primary returns 200 only on the leader (and /replica on standbys). HAProxy health-checks these endpoints and sends write traffic to whichever node answers 200. Two HAProxy instances plus keepalived provide a stable address. Health-check interval and fall/rise counts add to the failover budget, and on-marked-down shutdown-sessions drops existing connections to a demoted node so clients reconnect promptly.

A virtual IP moved by a callback. Patroni can run callback scripts on role change, which can bind or release a floating IP. It is simple and fast on a flat L2 network, but cloud environments often restrict gratuitous ARP, making it unreliable outside bare metal or specific cloud mechanisms.

Client-side multi-host connection strings. libpq accepts multiple hosts and target_session_attrs=read-write, so the client library itself finds the primary: postgresql://db1,db2,db3/app?target_session_attrs=read-write. No proxy tier, minimal moving parts, and it works with most libpq-based drivers. Behaviour depends on driver support and reconnect logic, and clients with long-lived pools may not re-probe until a connection errors.

Connection pooler in front. PgBouncer in front of the HAProxy endpoint absorbs connection storms and lets you pause traffic during a switchover. Pooling mode affects what survives a failover; see our comparison of PgBouncer, PgCat and RDS Proxy in transaction mode for the prepared-statement and session-state consequences.

Figure 4: Client routing. The application talks to a pooler, the proxy health-checks each node’s Patroni REST endpoint, and only the leader answers as primary.

Whichever pattern you choose, application behaviour decides the real RTO. A client that holds a pool of TCP connections to a dead primary will wait for TCP keepalive or a statement timeout before noticing. Set tcp_user_timeout, enable keepalives, and use short statement and connect timeouts. Retry logic must also be idempotent: a commit whose acknowledgement was lost in the failover may or may not have been applied, and a blind retry can double-insert.

Synchronous mode, strict mode and quorum

Asynchronous failover trades RPO for speed. Patroni’s replication modes let you buy the RPO back.

With synchronous_mode: on, Patroni manages synchronous_standby_names itself, and per the documentation “only the last leader or a synchronous replica can win the leader race.” In other words it will not promote a standby unless it is certain that standby has every transaction that might have been acknowledged. The consequence is stated openly: the cluster may refuse to fail over automatically when no safe candidate exists, though an administrator can force a promotion and accept the loss.

There is a subtlety that surprises people. If the synchronous standby disappears, plain synchronous mode lets Patroni disable synchronous replication so that the primary keeps accepting writes. Commits block until Patroni’s next loop makes that change, at most ttl seconds. This favours availability: you keep writing, but with no replication guarantee until a standby returns. If that primary then fails, nothing is promoted automatically.

synchronous_mode_strict removes that escape hatch. When no synchronous standby is available the primary blocks writes, guaranteeing every acknowledged write exists on at least two nodes. This is the consistency-over-availability choice, appropriate for ledgers and order systems, and it means a single replica failure can halt writes until a replacement synchronizes. The documentation notes that even strict mode cannot make every edge case perfect, for example when a backend is cancelled while waiting for acknowledgement.

synchronous_mode: quorum (PostgreSQL 10 and later) maps to the ANY num_sync (...) form of synchronous_standby_names rather than the priority-based FIRST. The PostgreSQL docs describe the difference: FIRST 2 (s1, s2, s3) waits for the two highest-priority connected standbys, while ANY 2 (s1, s2, s3) commits once any two reply. Quorum commit tolerates a slow standby without stalling, and because any quorum of voters overlaps in at least one node holding the latest commit, Patroni can promote safely by comparing a candidate’s LSN against at least quorum other voters. The cost of a bigger quorum is commit latency set by the second-fastest reply instead of the first, which is why quorum is a better fit for three-AZ clusters where the nearest standby is not always the same one.

A compact configuration for a three-node strict synchronous cluster looks like this:

scope: pg-prod
namespace: /db/
name: pg1
restapi:
  listen: 0.0.0.0:8008
  connect_address: 10.0.1.11:8008
etcd3:
  hosts: 10.0.0.1:2379,10.0.0.2:2379,10.0.0.3:2379
bootstrap:
  dcs:
    ttl: 30
    loop_wait: 10
    retry_timeout: 10
    maximum_lag_on_failover: 1048576
    synchronous_mode: true
    synchronous_mode_strict: true
    synchronous_node_count: 1
    postgresql:
      use_pg_rewind: true
      use_slots: true
      parameters:
        wal_level: replica
        wal_log_hints: "on"
        max_wal_senders: 10
        max_replication_slots: 10
        max_slot_wal_keep_size: 50GB
        hot_standby: "on"
        archive_mode: "on"
        archive_command: "pgbackrest --stanza=prod archive-push %p"
        archive_timeout: 60
postgresql:
  listen: 0.0.0.0:5432
  connect_address: 10.0.1.11:5432
  data_dir: /var/lib/postgresql/data
watchdog:
  mode: required
  device: /dev/watchdog
  safety_margin: 5

Treat this as a starting skeleton: key names such as the etcd section differ by DCS backend and Patroni version, so validate against the documentation for the release you deploy.

Backups and Point-in-Time Recovery: What Replication Cannot Do

A replica is a copy that agrees with the primary, which makes it a worthless defence against logical errors. A DROP TABLE executed by mistake is replayed on every standby within milliseconds. The only protection is a base backup plus a continuous archive of WAL, which lets you replay history up to a chosen moment. That is point-in-time recovery (PITR).

How PITR works

A base backup is a consistent copy of the data directory at a past instant. WAL archiving saves every completed WAL segment elsewhere via archive_command. To recover to time T, you restore the latest base backup that finished before T, then replay archived WAL up to T and stop. The PostgreSQL documentation notes that archiving only applies to completed segments, so on a quiet system a segment may sit unfinished for a long time; archive_timeout forces a segment switch after the interval (a minute or so is the documented suggestion), and shorter values bloat the archive because forced-switch segments are still full length. Remember that archive_command returning non-zero stalls archiving and WAL piles up on the primary, which is a second way to fill the disk after the slot problem above.

Archiving also bounds your DR RPO: you can lose up to the data in the current un-archived segment plus archive_timeout, unless you also stream WAL to a standby. This is why replication and archiving are complementary. Replication gives near-zero RPO for node loss; archiving gives a bounded RPO for everything else, with the unique ability to go back in time.

pgBackRest in practice

pgBackRest is a widely used backup tool for this job; WAL-G is a comparable alternative. Per its user guide, it supports full backups (self-contained), differential backups (changes since the last full), and incremental backups (changes since the last backup of any type), with restore needing the full chain back to the base. Retention is set with options such as repo1-retention-full and repo1-retention-diff, and expiring a full backup also expires its dependents. The guide cautions that archive retention needs care, because PITR needs a continuous WAL stream, and offers a dry-run mode to preview expiry. Setting archive-async=y batches archiving with a spool directory and greatly improves archive throughput, which matters on busy clusters where a synchronous archive_command per segment becomes a bottleneck.

# /etc/pgbackrest/pgbackrest.conf
[global]
repo1-type=s3
repo1-path=/pgbackrest
repo1-s3-bucket=acme-pg-backups
repo1-s3-endpoint=s3.amazonaws.com
repo1-s3-region=us-east-1
repo1-retention-full=4
repo1-retention-diff=7
repo1-cipher-type=aes-256-cbc
archive-async=y
spool-path=/var/spool/pgbackrest
start-fast=y
compress-type=zst

[prod]
pg1-path=/var/lib/postgresql/data

# weekly full, daily differential
pgbackrest --stanza=prod --type=full backup
pgbackrest --stanza=prod --type=diff backup

# prove the chain is restorable
pgbackrest --stanza=prod check
pgbackrest --stanza=prod info

# recover to just before a bad deploy
pg_ctl stop -D /var/lib/postgresql/data
pgbackrest --stanza=prod --delta \
  --type=time --target="2026-10-19 14:31:00+00" \
  --target-action=promote restore
pg_ctl start -D /var/lib/postgresql/data

Per the user guide, --type=time with --target selects a backup that ended before the target and writes the recovery settings into postgresql.auto.conf; --target-action=promote makes the cluster come online at the target. If the selected backup ends after the target, recovery fails with “recovery ended before configured recovery target was reached,” so choose an earlier backup with --set. The --delta option reuses files already in place and only restores what differs, which is far faster than a full restore of a large cluster.

Timelines, and why a restore is not a failover

Every promotion, whether from failover or recovery, starts a new timeline, and WAL files are tagged with the timeline ID. This is what lets pg_rewind find the fork point and what lets a PITR restore pick a branch of history. It also means a PITR restore target must be specified against the right timeline (recovery_target_timeline, which defaults to following the latest in recent releases; verify for your version). When you restore to a point before a bad change on a cluster that has since failed over, you are navigating history that has already forked at least once.

A practical rule is to restore into a new, separate cluster rather than over production. Recover to the minute before the damage, extract the lost rows with a query or pg_dump of the affected table, and re-insert them into the live database. Overwriting production with a rewind to the past discards every legitimate write since that moment.

RPO and RTO, worked out

The table below combines the mechanisms above into illustrative numbers. They are reasoning from timer defaults and typical behaviour, not measurements, and your hardware, network and workload will move them.

Scenario	Mechanism	Illustrative RPO	Illustrative RTO
Primary host dies, async replication	Patroni failover, default timers	Seconds of WAL not yet streamed (bounded by maximum_lag_on_failover)	30 to 55 s
Primary host dies, synchronous_mode_strict	Promote the synchronous standby	0 for acknowledged commits	30 to 55 s, plus risk of blocked writes if the sync standby is also down
Whole AZ lost, quorum across three AZs	Quorum commit, promote the standby with the latest LSN	0 for acknowledged commits	30 to 60 s
Bad DELETE at 14:32, discovered 14:50	pgBackRest PITR into a side cluster	Up to the last archived segment, plus whatever you choose to skip	Minutes to hours, dominated by restore size and WAL replay
Region lost	Cross-region standby or restored archive	Last archived segment, or async lag	Tens of minutes to hours, if rehearsed

A concrete example of the cost side: suppose the application commits 2,000 transactions per second at an average of 1 KB of WAL each, which is roughly 2 MB per second or about 170 GB of WAL per day (illustrative). A 16 MB segment fills in about eight seconds, so archive_timeout=60 is largely irrelevant at this load, and RPO to the archive is dominated by archiver throughput. On a quiet system the same setting would cap archive-based RPO near a minute. Recovery replay speed depends on I/O and the single-threaded redo process, so a 24-hour replay of 170 GB can run from tens of minutes to a few hours; the only honest way to know is to time a restore drill and record it.

Running It on Kubernetes: Operators and the API Server as the DCS

On Kubernetes, the DCS role is played by the Kubernetes API itself, and the operator pattern replaces a hand-managed agent fleet. Two operators dominate: CloudNativePG and Zalando’s postgres-operator (which runs Patroni inside the pods). They represent different philosophies.

CloudNativePG does not use Patroni. An instance manager runs as PID 1 in each pod, and the operator treats the Cluster custom resource as the source of truth. According to its failure-modes documentation, when the primary fails the operator promotes the standby with the most up-to-date data and the lowest replication lag, moves the read-write service to the new primary, and removes the failed pod from the read-write and read services. The former primary rejoins via pg_rewind if its persistent volume claim is available, or is rebuilt from a backup of the new primary otherwise. The same documentation warns that setting the cnpg.io/reconciliationLoop annotation to "disabled" prevents the operator from performing self-healing, including failover, so it is an emergency lever to be removed afterward. Release numbering moves quickly (the project documentation currently lists minor versions in the 1.2x to 1.30 range), so pin and test the specific version you adopt.

The Zalando operator takes the opposite bet: keep Patroni, which has years of production history, and let the operator manage the StatefulSets, services and credentials around it.

The Kubernetes setting adds its own hazards that bare-metal HA does not have. Storage is the large one. A pod rescheduled to another node must reattach its volume; with zonal network volumes the pod may be stuck pending when its zone is down. Local NVMe is fast but ties a replica to a node, so replacement means a full rebuild from backup. Anti-affinity and topology spread constraints must place replicas in distinct zones, or three replicas die together. Resource limits matter as well: a CPU-throttled Patroni or instance manager can miss its renewal window, which looks exactly like a partition.

Upgrades Without Losing the Property You Built

An HA cluster that cannot be upgraded safely is a liability, so design the upgrade path before you need it.

Minor version upgrades are binary swaps and are handled as a rolling restart: upgrade each standby in turn, then perform a controlled switchover to an upgraded standby and upgrade the old primary last. Patroni’s switchover is a planned role change that waits for the standby to catch up, so it avoids the TTL wait entirely and typically completes in a couple of seconds of write pause. Use patronictl switchover rather than killing the primary.

Major version upgrades change the on-disk format and are the dangerous case. Physical replicas cannot span majors. The two practical routes are pg_upgrade run on the primary (with --link for speed) followed by rebuilding the standbys, or logical replication to a new cluster at the new major with a brief cutover. The first is operationally simple and has an outage proportional to the upgrade; the second keeps downtime near seconds but requires every table to have a replica identity and treats sequences, large objects and DDL as manual concerns. Our PostgreSQL 19 upgrade guide walks the logical-replication path. Either way, take a verified backup first and make sure the new cluster is archiving before you send production traffic to it.

Trade-offs, Gotchas, and What Goes Wrong

The honest list of what actually causes incidents is shorter and less exotic than architecture diagrams suggest.

Runaway replication slots. An abandoned slot, often from a decommissioned replica, a CDC tool or a stalled logical consumer, pins WAL until the disk fills and the primary halts. The fix is max_slot_wal_keep_size, alerting on wal_status and slot lag, and on PostgreSQL 18 a nonzero idle_replication_slot_timeout. Know the trade: capping a slot makes the lagging consumer rebuild.

DCS outage or slow DCS. A DCS that loses quorum makes every leader demote unless failsafe mode is on. A DCS with high write latency (say, etcd on a noisy disk) makes lock renewals slow enough to trip the retry_timeout budget and trigger spurious failovers. Give etcd dedicated, fast disks, keep it off the database hosts if you can, and monitor its commit latency. Its availability is multiplied into the cluster’s.

Replica lag at the worst time. A standby that is minutes behind is not a failover candidate. If all standbys exceed maximum_lag_on_failover, the cluster stays leaderless until someone intervenes. Causes include long-running queries on a standby combined with max_standby_streaming_delay, single-threaded replay on write-heavy workloads, and saturated network links. hot_standby_feedback prevents query cancellations on a standby but can bloat the primary, because it delays vacuum cleanup; know which side you are paying.

Synchronous replication stalls. Strict mode converts a standby problem into a write outage. The same applies to plain synchronous mode during the window before Patroni notices. Application timeouts and circuit breakers must tolerate commit waits of up to ttl seconds.

Split brain via the sidecar. The classic mistake is putting a VIP or load balancer in front of two nodes without a lease protocol, or allowing a human to promote a replica by hand without stopping the old primary. Always use patronictl for role changes and never run pg_ctl promote on a Patroni-managed node.

Unverified backups. A backup that has never been restored is a hope. Archive gaps, missing WAL after a max_slot_wal_keep_size event, or a retention misconfiguration that expires archive needed by the oldest base backup are common silent failures. Run pgbackrest check in monitoring and a full restore drill on a schedule.

Clock and time-zone traps. PITR targets are timestamps. Specify them with an explicit offset, and remember that your log timestamps and the database server clock may differ.

Treating HA as backup. One more time: the replicas faithfully replicate your mistakes.

Practical Recommendations

Start from the failure you must survive and work backwards. If a node loss with zero acknowledged-data loss is the requirement, you need synchronous replication with a quorum that spans failure domains, and you must accept the availability cost of strict mode or its looser alternative. If seconds of loss are acceptable, asynchronous replication with a tight maximum_lag_on_failover is simpler and faster.

Run three DCS members across three failure domains, give them fast disks and keep them operationally boring. Keep the Patroni timer inequality intact whenever you change anything. Arm a watchdog if your platform supports it. Route clients through health checks on the Patroni REST API, not through a static address, and make application retries idempotent.

Treat backup and recovery as a product with an owner, an SLO and a drill schedule. Keep archive retention longer than the oldest base backup you intend to restore from. Test the failure you fear most, not the one that is easy to inject.

A short checklist:

• Set max_slot_wal_keep_size and alert on slot wal_status and pg_wal volume usage.
• Enable wal_log_hints or data checksums at initdb, and set use_pg_rewind: true.
• Verify loop_wait + 2 * retry_timeout <= ttl after every timer change.
• Decide synchronous mode explicitly (off, on, strict, quorum) and write the reasoning down.
• Use target_session_attrs=read-write or health-checked HAProxy, plus keepalives and tcp_user_timeout.
• Run pgbackrest check continuously and a timed restore drill at least quarterly.
• Practice switchover, then practice killing the primary, then practice a DCS outage.
• Pin operator and Patroni versions, and read release notes before upgrading either.
• For regional resilience, plan a cross-region path; see our Velero and Kubernetes disaster-recovery runbook for the workload side.

Frequently Asked Questions

What is the difference between Patroni and streaming replication?

Streaming replication is the PostgreSQL feature that ships WAL from a primary to standbys. It copies data but makes no decisions: it will not promote a standby, prevent two primaries, or reroute clients. Patroni is the layer on top that uses a distributed store such as etcd to hold a leader lock, promotes a standby when the lock expires, demotes an isolated primary, and exposes REST endpoints for routing. You need replication for HA; you need something like Patroni to automate it safely.

How long does a Patroni failover take?

With the documented defaults (ttl 30 s, loop_wait 10 s, retry_timeout 10 s), a sudden primary loss typically means up to about 30 seconds for the leader key to expire, up to another loop of detection, plus promotion and client rerouting, so roughly 30 to 55 seconds as an illustrative envelope. Lowering ttl toward its 20-second minimum shortens this, but the constraint loop_wait + 2 * retry_timeout <= ttl must still hold. Planned switchovers are much faster because they skip the expiry wait.

How do I prevent split brain in PostgreSQL?

Use a consensus-backed lease so only the node holding the leader key may run as primary, and make the old primary demote itself when it cannot renew. Add a watchdog so a hung agent still gets the host reset, enable synchronous replication so isolated writes cannot commit, and never promote nodes by hand outside the HA tool. Route clients by health checks rather than static addresses. Finally, run pg_rewind to rejoin the old primary instead of restarting it as-is.

Is a replica a substitute for a backup?

No. A replica applies every change from the primary, including accidental deletes, corrupt writes and dropped tables, within moments. Backups plus continuous WAL archiving let you restore to a chosen point in time before the damage. Replication handles hardware and node failure; backups handle human error, logic bugs and catastrophic loss. Production systems need both, and the backups must be restore-tested regularly or you cannot know they work.

Should I use Patroni or CloudNativePG?

If you run on virtual machines or bare metal, Patroni with etcd is the mature, well-documented choice. If your platform is Kubernetes and you want declarative clusters, built-in backup to object storage and rolling updates managed by an operator, CloudNativePG is designed for that environment and uses the Kubernetes API instead of an external DCS. Zalando’s operator keeps Patroni inside pods. Evaluate storage topology, team skills and upgrade cadence rather than features alone.

What does synchronous_mode_strict do?

In Patroni, synchronous_mode_strict prevents the cluster from turning synchronous replication off when no synchronous standby is available. The primary then blocks writes until a synchronous replica returns, which guarantees every acknowledged write exists on at least two nodes. It chooses consistency over availability: a single replica failure can pause writes. Without strict mode, Patroni lets the primary continue with no replication guarantee until a standby returns.

Further Reading

• PgBouncer vs PgCat vs RDS Proxy: PostgreSQL connection pooling in transaction mode
• PostgreSQL 19 release features and upgrade guide: partitioning and logical replication
• Multi-region active-active database architecture
• Velero 1.18 Kubernetes backup and disaster recovery runbook
• Patroni documentation: dynamic configuration and replication modes
• PostgreSQL documentation: WAL configuration and synchronous_commit
• pgBackRest user guide
• CloudNativePG documentation

By Riju — about