Erasure Coding vs Replication Explained: Reed-Solomon, Local Reconstruction Codes and Durability Math for Object Storage (2026)
Three full copies of every byte is the most expensive way to make data durable, and it is often not even the most durable. A Reed-Solomon 10+4 layout stores the same data on 1.4x raw capacity and survives four simultaneous losses, while triple replication burns 3x and survives two. So why does anyone still replicate? Because the real contest in erasure coding vs replication is not overhead. It is repair traffic, small-object waste, degraded-read latency and correlated failures, and those costs are invisible on a capacity spreadsheet. In 2026, with 20 TB-plus drives, Ceph’s new FastEC path and MinIO-style S3 stacks on commodity servers, choosing the wrong scheme means either paying double for disks or saturating your network every time a drive dies. This explainer works through the mechanism from the finite field up, the codes that hyperscalers actually ship, the durability math behind “eleven nines”, and where the math lies to you.
What this covers: Reed-Solomon over GF(2^8) and the MDS property, local reconstruction codes from Azure and Facebook, regenerating and Clay codes, HDFS, Ceph and MinIO EC defaults, worked MTTDL and repair-traffic numbers, a tested Python encoder, failure modes and a decision table.
Context and Background
Replication is the oldest redundancy scheme in distributed storage because it is trivially correct. A write lands on N nodes, a read can be served by any of them, and repair is a straight copy. Google File System and early HDFS defaulted to three replicas, and Haystack at Facebook combined triple replication with hardware RAID-6 underneath, which the f4 paper describes as an effective replication factor of 3.6. Every object store you can deploy today, from Ceph RADOS to MinIO and SeaweedFS, still offers replicated pools. Our S3-compatible object storage comparison of Ceph, MinIO, SeaweedFS and Garage covers how each product exposes these choices.
Erasure coding came from communications theory. Irving Reed and Gustave Solomon published “Polynomial Codes Over Certain Finite Fields” in the Journal of the Society for Industrial and Applied Mathematics in 1960. The same codes later protected CDs, QR codes, deep-space links and RAID-6 arrays. Storage systems use them in erasure mode: a missing shard is not corrupted but absent, and its position is known. That is the easy case for a code, because locating errors costs redundancy and erasures do not need locating.
The industrial turning point came around 2010 to 2014. Facebook ran HDFS-RAID with RS(10,4) on cold data. Microsoft published “Erasure Coding in Windows Azure Storage” at USENIX ATC 2012, introducing local reconstruction codes. Facebook’s f4 warm BLOB store, presented at OSDI 2014, moved photos off Haystack onto RS(10,4) with a 1.4x in-cell overhead. Hadoop 3.0 then made striped erasure coding a native HDFS feature.
Today the cloud providers do not publish their exact code parameters, but they do publish the outcome. AWS documentation states that S3 Standard is “designed to provide 99.999999999% durability and 99.99% availability of objects over a given year” and that it stores objects “on multiple devices across a minimum of three Availability Zones”. That “eleven nines” figure is a design target derived from models like the ones later in this post, not a measured loss rate. Understanding how such numbers are built is the best defence against misreading them.
Erasure Coding vs Replication: How Reed-Solomon Actually Works
Erasure coding splits an object into k data shards and computes m parity shards, so any k of the n = k + m shards rebuild the object. Replication stores n full copies, so any one copy suffices. Erasure coding cuts raw capacity from n× to n/k×, but every repair or degraded read must fetch k shards instead of one.

Figure 1: Lifecycle of one Reed-Solomon 6+3 stripe, from object split through encoding, placement and reconstruction after up to three losses.
The figure follows a 6 MiB object. It is cut into six 1 MiB data shards, a Cauchy matrix over GF(2^8) produces three parity shards, and placement puts each of the nine shards in a separate failure domain. If up to three domains vanish, any six survivors define an invertible 6×6 system and the original bytes come back exactly.
The finite field is what makes “any k of n” possible
Ordinary integer arithmetic does not work for parity on bytes, because sums overflow and division is not exact. Reed-Solomon codes in storage operate in the Galois field GF(2^8), which has exactly 256 elements, one per byte value. Addition is XOR. Multiplication is polynomial multiplication modulo a fixed irreducible polynomial; a common choice is x^8 + x^4 + x^3 + x^2 + 1, written 0x11D. Every non-zero element has a multiplicative inverse, so linear systems can be solved just as with real numbers.
Encoding is a matrix-vector product. Stack an identity matrix of size k on top of an m x k coding matrix to get a generator matrix G of size n x k. Multiply G by the vector of k data bytes at the same offset in each shard, and you get n output bytes: the first k are the data unchanged, which is what “systematic” means, and the last m are parity. A systematic code matters operationally because a healthy read never decodes anything. It just concatenates data shards.
Decoding picks any k surviving rows of G, inverts that k x k submatrix, and multiplies it by the surviving bytes. The construction guarantees every k x k submatrix is invertible. Vandermonde-derived and Cauchy matrices both have this property, which is why libraries such as Intel ISA-L and Jerasure offer them. With GF(2^8) the classical constructions limit a stripe to roughly 256 shards, far more than any storage system uses.
The MDS property and why it is the ceiling
A code that recovers from any m erasures with exactly m parity shards is called maximum distance separable, or MDS. The Singleton bound says a code of length n carrying k symbols of data has minimum distance at most n − k + 1, so it can never tolerate more than n − k erasures. Reed-Solomon meets that bound with equality. In plain terms, for a fixed overhead, no code tolerates more arbitrary failures than Reed-Solomon.
That optimality is why erasure coding vs replication often looks one-sided on paper. The table below uses only arithmetic, no vendor claims.
| Scheme | Raw bytes per usable byte | Arbitrary failures survived | Shards read to repair one lost shard |
|---|---|---|---|
| 2x replication | 2.00 | 1 | 1 |
| 3x replication | 3.00 | 2 | 1 |
| RS 4+2 | 1.50 | 2 | 4 |
| RS 6+3 | 1.50 | 3 | 6 |
| RS 8+3 | 1.375 | 3 | 8 |
| RS 10+4 | 1.40 | 4 | 10 |
| RS 12+4 | 1.33 | 4 | 12 |
| LRC 12+2+2 | 1.33 | 3, plus most 4-failure patterns | 6 for a data or local parity shard |
Read the last column carefully. It is where the true cost in erasure coding vs replication lives. Replication repairs a lost block by reading one block. RS 10+4 reads ten. The rest of this post is largely about that column.
Where the shards go matters as much as the code
The code only protects against failures that hit fewer than m + 1 shards of the same stripe. If two shards of an RS 6+3 stripe sit on the same host, a single host failure consumes two thirds of your tolerance. Every serious implementation therefore couples the code to a placement policy.
HDFS documents the rule explicitly. Its erasure coding guide says that for RS(6,3) you need “minimally 3 racks (calculated by (6 + 3) / 3 = 3), and ideally 9 or more to handle planned and unplanned outages”. The formula divides the stripe width by the parity count, so each rack holds at most m shards and a full rack loss is survivable. Ceph does the same through CRUSH, with a crush-failure-domain in every erasure code profile, defaulting to host. Backblaze places each of 20 shards on a different Storage Pod in a Vault, and its architecture post states that “all of the data is available as long as 17 of the 20 Storage Pods in the Vault are available”.
The practical consequence for erasure coding vs replication is that wide codes demand wide clusters. RS 10+4 with host failure domains needs at least 14 hosts, and realistically more so that a replacement target exists after a failure. Replication with three copies needs three. That cluster-size floor is one of the most common reasons small deployments stay on replication.
Local Reconstruction Codes, Regenerating Codes and the Repair Problem
The MDS property optimises one thing, failures tolerated per byte of overhead, and pays for it in repair cost. Large operators discovered that repair, not loss, dominated their day-to-day experience of erasure coding. Most failures are single shards: one disk, one node in maintenance, one slow server. With Reed-Solomon, each single-shard repair reads k shards.
The Facebook measurement that changed the field
The VLDB 2013 paper “XORing Elephants: Novel Erasure Codes for Big Data” by Sathiamoorthy, Asteris, Papailiopoulos, Dimakis, Vadali, Chen and Borthakur quantified the problem on a Facebook HDFS-RAID cluster running RS(10,4). The authors estimated repair traffic at 10 to 20 percent of the cluster’s average network traffic of about 2 PB per day. They also estimated that if half the cluster were RS-encoded, repair traffic would saturate the network links.
Their fix, deployed as HDFS-Xorbas, added two local XOR parities to the existing 14-block RS stripe, giving 16 blocks with locality 5. A lost block is rebuilt from five others instead of ten. In their EC2 experiments, Xorbas read 41 to 52 percent of the data that RS read for the same repairs. The price is stated plainly in the paper: “it requires 14% more storage compared to RS”. That trade, a little capacity for half the repair I/O, is the essence of a locally repairable code.
Azure’s LRC 12+2+2
Microsoft’s Windows Azure Storage paper, by Huang, Simitci, Xu, Ogus, Calder, Gopalan, Li and Yekhanin, formalised the idea as Local Reconstruction Codes and shipped it in production.

Figure 2: LRC 12+2+2 splits twelve data fragments into two local groups with one local parity each, plus two global parities computed over all twelve.
In LRC(12,2,2), twelve data fragments form two groups of six. Each group gets a local parity, and two global parities cover all twelve, for sixteen fragments and an overhead of 16/12, about 1.33x. A single lost data fragment, shown as x3 in the figure, is rebuilt from the five other data fragments and the local parity of its group, six reads in total. RS(12,4) at the same 1.33x overhead needs twelve.
The tolerance profile is subtle. The paper shows the construction decodes any three failures, and that the decodable four-failure patterns “account for 86% of all the four failures”. So LRC 12+2+2 is not MDS. It gives up some worst-case tolerance compared with RS(12,4) in exchange for halving common-case repair cost. The paper also defines a stronger property, Maximally Recoverable, meaning the code decodes every failure pattern that is information-theoretically decodable for its structure, and shows the construction achieves it.
The paper frames the comparison two ways. Holding reconstruction cost at six reads, LRC 12+2+2 reduces overhead from 1.5x, RS 6+3, to 1.33x, roughly an 11 percent capacity saving. Holding overhead at 1.5x, LRC 12+4+2 halves single-fragment reconstruction cost from 6 to 3. At exabyte scale, an 11 percent capacity saving is an enormous number of drives.
Degraded reads were the other motivation. Azure reconstructs data on the fly when a fragment is offline for an upgrade or when its server is merely hot, treating a slow node as missing. Reconstruction latency is set by the slowest fragment in the read set, so reading six fragments instead of twelve directly lowers tail latency. The paper reports a 4 MB reconstruction at 418 ms for LRC against 893 ms for RS reading twelve fragments in its test environment.
Regenerating codes and Clay
Local codes cut repair reads by adding parity. A second research line asks whether an MDS code can repair with less traffic without extra storage. Dimakis, Godfrey, Wainwright and Ramchandran’s “Network Coding for Distributed Storage Systems” introduced regenerating codes and characterised a trade-off curve between storage per node and repair bandwidth. At one end sit minimum-storage regenerating (MSR) codes, which keep MDS overhead but download only a fraction of each helper’s shard.
The insight is that repairing one shard does not require reconstructing the whole object. If each shard is divided into sub-chunks, the replacement can download a carefully chosen subset of sub-chunks from more helpers. Total bytes moved drop even though more nodes participate.
Clay codes, presented at USENIX FAST 2018 by Vajha and colleagues, made MSR practical by building one from layers of an ordinary MDS code. The paper reports that a code with 1.25x overhead reduced repair network traffic by a factor of 2.9 versus RS in its Ceph experiments. Ceph ships a clay plugin whose documentation works through k=8, m=4, d=11. Repairing one OSD contacts eleven helpers and downloads about 250 MiB from each, 2.75 GiB in total, where Jerasure or ISA reading eight full chunks would download 8 GiB. Note the operational caveat: the current Ceph documentation marks the Clay plugin deprecated since the Umbrella release, with removal planned for Vampire. Treat it as a reference design rather than a new deployment choice.

Figure 3: Illustrative repair traffic after a 20 TB drive failure. Replication copies about 20 TB; RS 10+4 reads about 200 TB; LRC 12+2+2 reads about 120 TB; Clay 8+4 with d=11 reads about 55 TB versus 160 TB for RS 8+4.
The figure turns the “shards read” column into network volume. These are first-order numbers. They assume the dead drive was full, every shard on it needs rebuilding and no compression applies. Local-group repairs in LRC still need the global path when two fragments of the same group are lost.
Durability Math: From MTTDL to “Eleven Nines”
Durability claims on both sides of erasure coding vs replication are probabilities derived from a model. The standard first-order tool is mean time to data loss, MTTDL, computed with a Markov chain over the number of failed shards in a group.
The Markov approximation
Consider one redundancy group of n fragments that survives any f concurrent failures. Each fragment fails independently at rate λ = 1/MTTF and each failure is repaired at rate μ = 1/MTTR. Data is lost when the chain reaches f + 1 failures. When MTTR is much smaller than MTTF, the chain collapses to a well-known approximation:
MTTDL ≈ MTTF^(f+1) / ( n × (n−1) × … × (n−f) × MTTR^f )
Intuitively, the first failure arrives at rate nλ. Each further failure must arrive inside the repair window of the previous ones, and each such coincidence costs a factor of roughly (n−i)λ × MTTR. The exponent f is why one extra parity shard is worth orders of magnitude, and why MTTR sits in the denominator with the same exponent.
A worked example, clearly illustrative
Take an annual drive failure rate of 2 percent. That gives MTTF ≈ 8,766 h / 0.02 ≈ 438,300 hours. Assume 24-hour repair for replication. For a three-copy group, n = 3 and f = 2:
MTTDL ≈ 438,300^3 / (3 × 2 × 1 × 24^2) ≈ 2.4 × 10^13 hours.
Now scale to a petabyte. If each group protects 1 TB of usable data, a petabyte has 1,000 groups. The probability of losing at least one group in a year is about 1,000 × 8,766 / 2.4 × 10^13 ≈ 3.6 × 10^-7. That is roughly 6.4 nines of “some data lost somewhere in the petabyte per year”.
The script below repeats the calculation for the common schemes. The repair windows differ on purpose, because erasure-coded repair reads k shards and generally takes longer. Every input is an assumption chosen for illustration, not a measurement from any vendor.
# Markov-approximation MTTDL for one redundancy group of n fragments that
# survives any f = n - k concurrent failures. Assumes independent exponential
# failures and repairs: an optimistic upper bound, not a prediction.
import math
HOURS_PER_YEAR = 8766
def mttdl_hours(n, f, mttf_h, mttr_h):
# Classic approximation: MTTF^(f+1) / (n(n-1)...(n-f) * MTTR^f)
denom = 1.0
for i in range(f + 1):
denom *= (n - i)
return mttf_h ** (f + 1) / (denom * mttr_h ** f)
def annual_loss_prob(n, f, groups, afr, mttr_h):
mttf_h = HOURS_PER_YEAR / afr
return groups * HOURS_PER_YEAR / mttdl_hours(n, f, mttf_h, mttr_h)
if __name__ == "__main__":
AFR = 0.02 # illustrative 2% annual drive failure rate
USABLE_TB = 1000 # 1 PB usable
FRAG_TB = 1 # pretend each group fragment is 1 TB of a drive
schemes = { # name: (n, k, f, mttr_hours)
"3x replication": (3, 1, 2, 24),
"RS 6+3": (9, 6, 3, 48),
"RS 10+4": (14, 10, 4, 72),
"LRC 12+2+2 *": (16, 12, 3, 48),
}
print(f"{'scheme':16}{'overhead':>9}{'raw TB':>8}{'P(loss)/yr':>14}{'nines':>7}")
for name, (n, k, f, mttr) in schemes.items():
groups = USABLE_TB / (k * FRAG_TB)
p = annual_loss_prob(n, f, groups, AFR, mttr)
print(f"{name:16}{n/k:>8.2f}x{USABLE_TB*n/k:>8.0f}{p:>14.2e}{-math.log10(p):>7.1f}")
Running it produces this output:
| Scheme (illustrative inputs) | Overhead | Raw TB for 1 PB | P(any loss) per year | Approx. nines |
|---|---|---|---|---|
| 3x replication, MTTR 24 h | 3.00x | 3,000 | 3.6 × 10^-7 | 6.4 |
| RS 6+3, MTTR 48 h | 1.50x | 1,500 | 1.3 × 10^-8 | 7.9 |
| RS 10+4, MTTR 72 h | 1.40x | 1,400 | 3.5 × 10^-10 | 9.5 |
| LRC 12+2+2, MTTR 48 h, modelled as f = 3 | 1.33x | 1,333 | 9.6 × 10^-8 | 7.0 |
The LRC row is deliberately pessimistic. It ignores the 86 percent of four-failure patterns the code survives, and it ignores that local repairs finish faster than global ones. A proper model gives LRC its own chain with separate local and global repair rates.
Three ways this model flatters you
First, it assumes independence. Google’s OSDI 2010 study “Availability in Globally Distributed Storage Systems” by Ford and colleagues found that “correlation among node failures dwarfs all other contributions to unavailability” and that “most large bursts of failures are associated with rack- or multirack level events”. It also notes that failing to model correlation “typically results in overestimating availability”. A power event that takes out a rack does not care about your exponent.
Second, it treats MTTR as a constant. In reality MTTR includes detection delay, the timeout before a down node is declared out, queueing behind other repairs and the rebuild itself. If repair traffic is throttled to protect foreground latency, MTTR stretches, and MTTR appears raised to the power f.
Third, it counts only drive failures. Latent sector errors, silent corruption, firmware bugs, operator mistakes and software defects that delete data on every replica at once are not in the chain. Checksums and scrubbing address the first two. Nothing in the code addresses the last two, which is why immutable copies and backups still matter; see our guide to ransomware-resilient backup with immutable storage and object lock.
For the erasure coding vs replication debate, the lesson is not that eleven nines is fake. It is that “storage durability eleven nines” is a property of an entire system, including placement across zones, fast detection, integrity scrubbing and operational discipline, and not of the code alone.
A Tested Reed-Solomon Encoder in Python
The fastest way to internalise the mechanism is to run it. The code below implements a systematic Reed-Solomon erasure code over GF(2^8) with a Cauchy parity matrix in about 80 lines of standard-library Python. It is educational. Production systems use SIMD-accelerated libraries such as ISA-L, which HDFS and Ceph both support, and per-byte Python loops are thousands of times slower.
# Minimal systematic Reed-Solomon erasure code over GF(2^8), Cauchy construction.
# Educational only: use ISA-L, Jerasure, klauspost/reedsolomon or pyeclib in production.
import os, random
PRIM = 0x11D # x^8 + x^4 + x^3 + x^2 + 1
EXP, LOG = [0] * 512, [0] * 256
x = 1
for i in range(255):
EXP[i], LOG[x] = x, i
x <<= 1
if x & 0x100:
x ^= PRIM
for i in range(255, 512):
EXP[i] = EXP[i - 255]
def mul(a, b):
return 0 if a == 0 or b == 0 else EXP[LOG[a] + LOG[b]]
def inv(a):
return EXP[255 - LOG[a]]
def cauchy_parity_matrix(k, m):
# Row i, col j = 1 / (x_i + y_j), with x and y disjoint sets. Addition in GF(2^8) is XOR.
xs, ys = range(k, k + m), range(k)
return [[inv(xi ^ yj) for yj in ys] for xi in xs]
def encode(data, m):
k = len(data)
P = cauchy_parity_matrix(k, m)
parity = []
for row in P:
out = bytearray(len(data[0]))
for coef, shard in zip(row, data):
for b, byte in enumerate(shard):
out[b] ^= mul(coef, byte)
parity.append(bytes(out))
return parity
def invert(M):
n = len(M)
A = [list(r) + [int(i == j) for j in range(n)] for i, r in enumerate(M)]
for c in range(n):
p = next(r for r in range(c, n) if A[r][c])
A[c], A[p] = A[p], A[c]
f = inv(A[c][c])
A[c] = [mul(f, v) for v in A[c]]
for r in range(n):
if r != c and A[r][c]:
g = A[r][c]
A[r] = [v ^ mul(g, w) for v, w in zip(A[r], A[c])]
return [row[n:] for row in A]
def decode(shards, k, m):
"""shards: list of k+m entries, None where lost. Returns the k data shards."""
G = [[int(i == j) for j in range(k)] for i in range(k)] + cauchy_parity_matrix(k, m)
alive = [i for i, s in enumerate(shards) if s is not None][:k]
if len(alive) < k:
raise ValueError("more than m shards lost: data unrecoverable")
D = invert([G[i] for i in alive])
size = len(shards[alive[0]])
data = []
for row in D:
out = bytearray(size)
for coef, i in zip(row, alive):
for b, byte in enumerate(shards[i]):
out[b] ^= mul(coef, byte)
data.append(bytes(out))
return data
if __name__ == "__main__":
k, m, size = 6, 3, 64
data = [os.urandom(size) for _ in range(k)]
stripe = data + encode(data, m)
trials = 0
for _ in range(200):
lost = random.sample(range(k + m), m) # any m shards
damaged = [None if i in lost else s for i, s in enumerate(stripe)]
assert decode(damaged, k, m) == data
trials += 1
print(f"RS({k},{m}): recovered data in {trials}/200 random {m}-shard losses")
try:
decode([None] * (m + 1) + stripe[m + 1:], k, m)
except ValueError as e:
print("lose m+1 shards ->", e)
On Python 3 it prints:
RS(6,3): recovered data in 200/200 random 3-shard losses
lose m+1 shards -> more than m shards lost: data unrecoverable
Three details are worth noticing. The parity matrix uses disjoint sets for x and y, which keeps every denominator non-zero and makes every square submatrix of the Cauchy matrix invertible; stacked under the identity, every k-row choice of G is invertible. Decoding cost is dominated by the matrix-vector work over the shard bytes, which is why SIMD table lookups in ISA-L matter. And when only data shards are lost, the decoder can reuse a cached inverse for that loss pattern, a trick real libraries use to make repeated degraded reads cheap.
What Ceph, HDFS and MinIO Actually Ship
Theory meets configuration in three popular open systems, and each frames erasure coding vs replication as a per-pool choice. The defaults differ in ways that reveal each project’s priorities.
Ceph erasure coding
Ceph erasure coding is configured through profiles. The current documentation shows the default profile as k=2, m=2 with plugin=isa, technique=reed_sol_van and crush-failure-domain=host. It notes that ISA-L “is the default plugin for pools created on Tentacle or later” and that “the Jerasure library is no longer maintained”. Older clusters will show Jerasure in their default profile, so check yours. The plugin list also includes lrc for locally repairable layouts, shec and the deprecated clay.
Two Ceph rules catch newcomers. By default an EC pool accepts only full-object writes, which suits RGW but not RBD or CephFS. Setting allow_ec_overwrites enables partial writes, and the docs say there is “no downside” and recommend doing so routinely. RBD and CephFS still keep metadata in a replicated pool. Second, Ceph recommends min_size of K+1 or greater to prevent loss of writes and data. A pool that keeps accepting writes with only K shards available has zero remaining redundancy for those writes.
Ceph Tentacle added FastEC, enabled per pool with allow_ec_optimizations. The Ceph team’s write-up describes partial reads that fetch only the needed data, partial writes that avoid rewriting the whole stripe, parity-delta writes and the removal of full-stripe padding for small objects. It warns that the flag “cannot be disabled” once set. The stated target is block and file workloads, the area where replication used to win by default.
The Ceph lrc plugin makes the local-group idea concrete. Its documentation shows k=8, m=4, l=4, where a lost OSD is recovered from four survivors instead of eight. A crush-locality=rack setting places each local group within a rack, so most repairs never cross the rack boundary.
HDFS erasure coding policies
HDFS names policies as codec-data-parity-cellsize. The built-in set is RS-3-2-1024k, RS-6-3-1024k, RS-10-4-1024k, RS-LEGACY-6-3-1024k and XOR-2-1-1024k. Only the system default, RS-6-3-1024k, is enabled out of the box; others must be enabled explicitly. The documentation contrasts the 200 percent overhead of 3x replication with “no more than 50%” for typical EC setups.
HDFS EC is striped. A file is cut into 1 MiB cells laid round-robin across the data blocks of a block group, unlike the contiguous blocks of replicated HDFS. Striping spreads large-file I/O across many DataNodes, but it also breaks data locality for MapReduce-style processing, because no single node holds a contiguous block. The default codec configuration prefers a native ISA-L implementation over pure Java when available, though ISA-L support must be built and enabled. As noted earlier, RS-6-3 needs at least nine DataNodes and, for rack tolerance, at least three racks.
MinIO erasure sets
MinIO uses Reed-Solomon across erasure sets of drives. Its documentation states that “each erasure set can contain 2 to 16 drives” by default, with newer AIStor releases able to configure up to 32. For a 16-drive set the default parity is EC:4, which is a 12+4 layout at 1.33x overhead. Reads need K shards. Writes need K drives, rising to K+1 when parity is exactly half the set size, which avoids split-brain writes in the symmetric case.
Because MinIO fixes erasure sets at deployment, the code width is tied to hardware layout. Choose parity per server pool with your node count in mind. A 16-drive set spread over four servers puts four shards on each server, so EC:4 survives one whole server, not two.
Erasure Coding vs Replication: A Decision Framework
With the mechanisms in place, erasure coding vs replication reduces to a handful of questions about workload shape and cluster topology.

Figure 4: A decision flow for choosing replication, narrow Reed-Solomon, wide Reed-Solomon or repair-optimised codes for a new pool or bucket.
The flow starts with latency sensitivity, because degraded reads under EC are expensive. It then checks object size and write pattern, then whether the cluster has enough failure domains for a wide stripe, and finally whether repair bandwidth is the limiting resource.
| Factor | Favours replication | Favours erasure coding |
|---|---|---|
| Object size | Small, under a few hundred KiB | Large, MiB and above |
| Write pattern | Random overwrites, appends | Write-once, read-many |
| Read latency | Tight p99 SLO, hot data | Throughput-bound or warm or cold data |
| Cluster size | Fewer than k + m + 1 failure domains | Many hosts or racks |
| Repair network | Constrained or shared | Ample, or LRC or Clay available |
| Capacity cost | Small datasets where 3x is cheap | Petabytes, where 1.4x vs 3x is decisive |
| Metadata and indexes | Almost always replicated | Rarely |
Two hybrid patterns are standard. Hot-warm tiering keeps fresh objects replicated and transitions them to EC once they cool, the strategy f4 used for photos. Pool splitting keeps metadata, bucket indexes and small objects in a replicated pool and bulk data in an EC pool, which Ceph explicitly requires for RBD and CephFS. Most production clusters run both erasure coding and replication side by side, so the real question is which bytes go where.
Geo-distribution adds a third layer. f4 paired RS(10,4) within a cell with a cross-datacenter XOR scheme, reducing its effective replication factor from 2.8 under double geo-replication to 2.1. Coding across regions saves capacity but turns every cross-region repair into wide-area traffic. For disaster recovery planning across sites, see our comparison of cloud disaster recovery strategies.
Trade-offs, Gotchas, and What Goes Wrong
Correlated failures defeat the exponent. Copysets research by Cidon and colleagues at USENIX ATC 2013 showed that random replica placement is “almost guaranteed to lose data” when many nodes fail together, as in a cluster-wide power outage. The same applies to EC stripes scattered randomly. Constrain placement so that the number of distinct shard combinations stays small, and align failure domains with real power and network boundaries.
Small objects waste space. Ceph EC pools without FastEC pad objects to a full stripe. With the default 4 KiB stripe unit, a 4 KiB object in an 8+3 pool becomes eight data chunks, seven of them padding, plus three parity chunks: 44 KiB on disk, an 11x overhead derived from that padding rule. That is worse than replication. Similar effects appear wherever a minimum shard size exists.
Degraded reads amplify load. When a data shard is missing, every read of that region fetches k shards and decodes. A rack outage on RS 10+4 can multiply read I/O on surviving nodes for the affected stripes just when capacity is reduced. Size headroom for the degraded state, not the healthy one.
Repair storms extend MTTR. A dense 20 TB-plus node holding shards from thousands of stripes generates repair reads across the cluster. If you throttle repair to protect clients, the window of reduced redundancy lengthens. If you do not throttle it, clients suffer. LRC, Clay and wide declustering are the structural fixes.
Partial writes are read-modify-write. Updating a block inside an EC stripe means reading the old data or the remaining stripe, recomputing parity and writing several shards. Without parity-delta logic this is far costlier than overwriting three replicas. Random-write workloads such as VM disks and databases suffer most.
Wide stripes and small clusters do not mix. RS 10+4 on 14 hosts leaves no spare host to rebuild into after a failure, so the pool stays degraded until hardware is replaced.
Code width limits growth. Changing k or m usually means rewriting data into a new pool. Pick a profile that fits the cluster you will have, not just the one you have now.
Practical Recommendations
Settle erasure coding vs replication from the workload, not the capacity bill. Replicate what is latency-critical, small, mutable or metadata. Erasure-code what is large, cold, write-once and voluminous. Then size the code to your failure domains, leaving at least one spare domain beyond k + m so repairs have somewhere to go.
For most mid-size object stores, RS 4+2 or 6+3 with host or rack failure domains is the sober default. Move to 8+3 or 10+4 only when you have many more than k + m failure domains and the capacity savings justify the extra repair traffic. When repair traffic, not capacity, is your binding constraint, evaluate local reconstruction codes; in Ceph that means the lrc plugin with crush-locality.
Treat durability numbers as model outputs. Rerun the MTTDL arithmetic with your own failure rates, measured MTTR and real placement, and then discount it for correlated failures. Pair the code with integrity checking. Scrubbing in Ceph and checksums in every serious object store catch silent corruption that the erasure model ignores, and Merkle-tree based anti-entropy is the scalable way to find divergent replicas.
A short checklist before you commit a profile:
- Count failure domains: at least k + m + 1 at the chosen level.
- Separate pools: metadata, indexes and small objects on replication.
- Set Ceph
min_sizeto at least K+1 and enableallow_ec_overwriteswhere needed. - Benchmark degraded reads and full-node rebuild time, not just healthy throughput.
- Budget repair bandwidth: drive size × k per failure, with throttles.
- Keep an independent immutable backup; codes do not protect against deletes.
Frequently Asked Questions
Is erasure coding more durable than replication?
At equal overhead, yes, by a wide margin. RS 10+4 uses 1.4x raw capacity and survives any four failures, while 3x replication uses 3x and survives two. Under a Markov model with independent failures, that extra tolerance adds orders of magnitude to MTTDL. However, erasure coding repairs more slowly, because it reads k shards per lost shard, and both schemes are vulnerable to correlated failures. Real durability depends on placement, repair speed and scrubbing as much as on the code itself.
What does RS 6+3 or 10+4 mean?
The notation gives data shards first, then parity shards. RS 6+3 splits each stripe into six data shards and adds three parity shards computed with Reed-Solomon over GF(2^8). Any six of the nine shards rebuild the data, so up to three losses are tolerated, at 9/6 = 1.5x raw overhead. RS 10+4 uses fourteen shards, tolerates four losses and costs 1.4x. HDFS writes these as RS-6-3-1024k and RS-10-4-1024k, where the suffix is the 1 MiB cell size.
How do local reconstruction codes differ from Reed-Solomon?
Local reconstruction codes add local parities computed over small groups of data shards, so a single lost shard is rebuilt from its group instead of from all k data shards. Azure’s LRC 12+2+2 rebuilds one fragment from six reads, versus twelve for RS 12+4, at the same 1.33x overhead. The trade-off is that LRC is not MDS. It tolerates any three failures and 86 percent of four-failure patterns, while RS 12+4 tolerates every four-failure pattern.
How does Amazon S3 achieve eleven nines of durability?
AWS documents that S3 Standard is designed for 99.999999999 percent annual durability and stores objects redundantly across a minimum of three Availability Zones. AWS also says the service detects and repairs lost redundancy quickly and verifies integrity with checksums. AWS does not publish its internal coding parameters, so any specific code attributed to S3 is speculation. The figure is a design target derived from failure models, not an observed loss rate.
Why is erasure coding bad for small files?
Each object is split into k shards plus parity, and storage systems impose minimum shard or stripe sizes. A tiny object can be padded up to a full stripe, so it may consume more space than three replicas would. Small objects also multiply metadata and the number of disk operations per read. Ceph’s Tentacle FastEC removes full-stripe padding for optimised pools, and most systems recommend keeping small objects and metadata in replicated pools.
Can I convert a replicated Ceph pool to erasure coding in place?
No. A pool’s redundancy type and erasure code profile are fixed at creation, so changing from replication to erasure coding, or changing k and m, requires creating a new pool and migrating data. For RGW you can direct new data to an EC-backed placement target or storage class and move existing objects with lifecycle transitions. For RBD and CephFS, the EC pool holds data while metadata stays replicated, and migration is a copy operation.
Further Reading
- S3-compatible object storage: Ceph vs MinIO vs SeaweedFS vs Garage architecture ADR — how the leading open object stores implement redundancy and where each fits.
- Ransomware-resilient backup architecture with immutable storage and object lock — the protection layer erasure codes cannot provide.
- Merkle trees explained: inclusion proofs and anti-entropy — how replicas detect and repair divergence efficiently.
- Raft vs Paxos: leader election, consensus and failure modes — the replicated logs that coordinate metadata in storage clusters.
References
- I. S. Reed and G. Solomon, “Polynomial Codes Over Certain Finite Fields,” Journal of the Society for Industrial and Applied Mathematics, vol. 8, no. 2, 1960.
- C. Huang et al., “Erasure Coding in Windows Azure Storage,” USENIX ATC 2012.
- M. Sathiamoorthy et al., “XORing Elephants: Novel Erasure Codes for Big Data,” VLDB 2013.
- S. Muralidhar et al., “f4: Facebook’s Warm BLOB Storage System,” USENIX OSDI 2014.
- A. G. Dimakis et al., “Network Coding for Distributed Storage Systems,” IEEE Transactions on Information Theory, 2010.
- M. Vajha et al., “Clay Codes: Moulding MDS Codes to Yield an MSR Code,” USENIX FAST 2018.
- D. Ford et al., “Availability in Globally Distributed Storage Systems,” USENIX OSDI 2010.
- A. Cidon et al., “Copysets: Reducing the Frequency of Data Loss in Cloud Storage,” USENIX ATC 2013.
- Apache Hadoop, HDFS Erasure Coding documentation.
- Ceph, Erasure Code, CLAY and LRC plugin documentation; FastEC for Tentacle.
- MinIO, Erasure Coding documentation.
- AWS, Data protection in Amazon S3.
- Backblaze, Backblaze Vaults: Zettabyte-Scale Cloud Storage Architecture.
By Riju — about
