S3-Compatible Object Storage: Ceph RGW vs MinIO vs SeaweedFS vs Garage (ADR, 2026)

If you run your own S3 compatible object storage in 2026, the decision you made two years ago may no longer be safe. MinIO, for a long time the default answer, now carries a repository banner stating that it “is no longer maintained” and that the community edition ships as source code only. That single fact reshuffles the shortlist for every team that needs a data lake, a backup target, or a model-artifact store on its own hardware.

This post is an architecture decision record (ADR) for choosing among the four systems that matter for self-hosting: Ceph RADOS Gateway (RGW), MinIO, SeaweedFS, and Garage. Rather than rank features, it compares the data-placement mechanism underneath each one, because placement determines durability, rebuild behaviour, and operational cost long before any benchmark does. You will leave with worked erasure-coding versus replication numbers, a clear view of each system’s S3 gaps, the current licensing state, a test script you can run today, and a decision matrix.

What this covers: the decision context, the shared architecture, erasure coding math, per-system internals (CRUSH, erasure sets, needles, layout rings), S3 compatibility gaps, failure and rebuild behaviour, licensing, operations, a boto3 test, and a recommendation.

Context and Background

Object storage became the substrate of modern data platforms because it makes one trade very well: it gives up POSIX semantics and in-place updates in exchange for flat namespaces, immutable objects, and near-unlimited horizontal scale. Amazon’s S3 API became the de facto interface, so tools such as Spark, Trino, Iceberg writers, backup agents, and ML frameworks all speak it. If you are choosing a table format on top, our comparison of Apache Iceberg versus Paimon lakehouse table formats shows how much of that stack assumes an S3-shaped endpoint underneath.

Self-hosting it is attractive for three recurring reasons. Egress and request pricing on public clouds punishes read-heavy analytics. Data residency rules force some data on-premises or into a specific country. And industrial or edge sites, a recurring theme in IoT and digital-twin work, often need a local store that keeps working when the WAN does not. Time-series and telemetry platforms are a typical consumer: the cold tier of the engines covered in our time-series database internals article is frequently Parquet on S3.

The four contenders sit at very different points in design space. Ceph is a unified distributed storage platform (block, file, object) with a mature object gateway, RGW. MinIO is a single-purpose Go object server built around erasure-coded sets of drives. SeaweedFS descends from Facebook’s Haystack paper: it packs many small files into large volume files and keeps a tiny in-memory index. Garage comes from the French collective Deuxfleurs and targets small, geo-distributed clusters on modest hardware, deliberately avoiding erasure coding.

What changed in 2026 is not technology but governance. According to the MinIO GitHub repository, the project was archived by its owner on 25 April 2026 and is read-only. Its README says the repository is no longer maintained and points to commercial AIStor editions. That is verified from the repository banner; third-party timelines of earlier steps (UI feature removal in 2025, the end of public Docker images and binaries later that year) are widely reported but I have not verified each date against primary announcements, so treat the intermediate dates as reported rather than confirmed. The licensing section below returns to this.

For regulated or cost-sensitive environments, the ADR question is therefore not “which is fastest” but “which is still a defensible dependency for the next five years, and which failure modes can my team actually operate”.

The Reference Architecture Shared by All Four

Every S3-compatible store, regardless of implementation, decomposes into the same pipeline: a stateless-ish S3 front end that authenticates and parses requests, a metadata path that maps bucket and key to object location, a data path that moves bytes, and a placement layer that decides which physical drives hold which pieces. The systems differ in how tightly they couple these and in how placement is computed.

Direct answer: S3-compatible object storage splits every request into a metadata lookup and a data transfer, then protects the data through either full replication or erasure coding. The practical differences between Ceph, MinIO, SeaweedFS, and Garage come from how each computes placement (CRUSH, hashed erasure sets, a volume master, or a zone-aware layout) and how each rebuilds after failure.

Figure 1: The common S3-compatible object storage pipeline. Differences between systems live in the placement and protection boxes.

The diagram shows the shared layers. A client using boto3 or awscli signs requests with Signature Version 4 and sends them to a gateway. The gateway consults a metadata structure to resolve the key, then reads or writes data through a placement function. Protection is applied either as whole-object copies or as erasure-coded shards, and both ultimately land on drives attached to nodes.

Placement is the real architecture

Placement answers one question: given an object, which drives hold it? There are four main answers in this comparison. Ceph computes it with CRUSH from a cluster map, so no central lookup table exists. MinIO groups drives into fixed-size erasure sets at deployment time and hashes objects to a set. SeaweedFS asks a master server to assign a volume ID and then writes to the servers hosting that volume. Garage computes a layout that assigns partitions of the key space to nodes subject to zone constraints, and replicates each partition.

Each choice has a signature failure mode. A computed placement (Ceph, MinIO) scales without a lookup bottleneck but makes reconfiguration expensive, because data must move when the map changes. A directory-based placement (SeaweedFS master) is easy to reason about but adds a coordination component. A layout ring (Garage) keeps operations simple but binds you to its replication-only protection.

Metadata is the hidden scaling limit

Objects are cheap to store in bulk; billions of tiny objects are hard to index. MinIO is generally described as storing object metadata alongside the shards on the same drives, so there is no separate metadata database to operate (my understanding of its design; not re-verified against current docs). Ceph RGW keeps bucket indexes in RADOS objects and can shard them. SeaweedFS keeps volume-level needle indexes in memory and delegates namespace metadata to a separate filer backed by an external store such as MySQL, Postgres, Redis or Cassandra. Garage replicates metadata tables across nodes with its own embedded database (LMDB or SQLite).

This difference matters for listing performance and small-object workloads. A listing operation is a metadata scan; a design that spreads metadata across the same drives as the data inherits their latency, while a design with a dedicated metadata store can put it on flash.

Protection: replication versus erasure coding

Replication stores N complete copies. Erasure coding splits an object into k data shards, computes m parity shards, and stores all k+m on distinct failure domains, so any k survive to reconstruct. The next section does the arithmetic, because this single choice dominates cost.

Erasure Coding vs Replication: The Math

Direct answer: Three-way replication stores 3.0 bytes of raw disk per byte of data and survives two simultaneous losses. A 4+2 erasure code stores 1.5 bytes per byte and also survives two losses. An 8+3 code stores about 1.375 bytes per byte and survives three. Erasure coding buys capacity efficiency in exchange for more network traffic, CPU, and slower small reads and rebuilds.

Figure 2: Erasure coding with k=4 data shards and m=2 parity shards. Any four of the six shards reconstruct the object.

Storage overhead

The overhead factor is (k+m)/k, a formula the Ceph documentation states directly (a 4,2 profile has overhead 1.5). Usable capacity is the inverse. For a cluster with 1,000 TB raw:

Scheme	Overhead	Usable of 1,000 TB raw	Failures tolerated
2x replication	2.0	500 TB	1
3x replication	3.0	333 TB	2
EC 4+2	1.5	667 TB	2
EC 8+3	1.375	727 TB	3
EC 10+4	1.4	714 TB	4
EC 12+4	1.333	750 TB	4

MinIO’s own capacity table, for a node with sixteen 1 TB drives, shows the same logic: parity EC:4 gives 12 TB usable (ratio 0.750), EC:6 gives 10 TB (0.625), and EC:8 gives 8 TB (0.500). Going from 3x replication to 8+3 on the same hardware lifts usable capacity by roughly 2.2 times, which is why erasure coding dominates at petabyte scale.

Durability, worked example

Durability depends on how often failures overlap inside the repair window. A simple model: let each drive fail independently with an annual failure rate (AFR), and let repair take w hours. The probability a given drive fails inside the window is p = AFR x w / 8760. For a group of n drives that loses data when more than t fail, the annual loss rate per group is approximately n x AFR x C(n-1, t) x p^t, counting the first failure as the trigger and requiring t further failures inside the window.

With illustrative assumptions of AFR = 2% and a 24-hour repair window (p of about 5.5e-5), the model yields:

Scheme	Group size n	Loss needs	Approx. annual loss probability per group
2x replication	2	2 failures	2.2e-6
3x replication	3	3 failures	1.8e-10
EC 4+2	6	3 failures	3.6e-9
EC 8+3	11	4 failures	4.3e-12
EC 10+4	14	5 failures	1.8e-15

These figures are the output of my own simple model with assumed inputs, not measured data or a vendor claim. Real systems have correlated failures (a bad batch of drives, a power event, a firmware bug), and the independence assumption flatters every row. Use the table for relative comparison only.

Three lessons survive the caveat. First, 4+2 is less durable than 3x replication at equal fault tolerance because it has more drives that can fail in the same window, even though both tolerate two losses; you pay for the capacity saving in exposure. Second, the repair window enters as w^t, so a 72-hour rebuild instead of 24 hours multiplies the 4+2 loss rate by 3^3 = 27 and the 3x rate by 3^2 = 9. Rebuild speed is a durability parameter, not just an operations nicety. Third, wider codes with more parity (8+3, 10+4) beat replication on both cost and durability, provided you have enough failure domains to place all shards.

The costs hidden in the overhead number

Erasure coding is not free. A write must encode and fan out k+m shards, so a small write touches many drives. MinIO attempts all shards in parallel and needs at least k to succeed, according to its documentation, which keeps latency tolerant of one slow drive. A degraded read must pull k shards and decode. A single-drive rebuild must read k shards for every shard it regenerates, so repair traffic is roughly k times the data rebuilt, against 1 times for replication.

Small objects suffer most. An object of 4 KiB split across 8 data shards yields 512-byte shards, below any sensible filesystem or flash page size, so metadata and alignment overhead can exceed the payload. This is one reason small-object workloads are often kept on replicated pools or packed into larger volumes, which is what Ceph replicated pools, Garage replication, and SeaweedFS volume packing each make possible.

Placement constraints also bite. Ceph’s documentation notes that most deployments need at least k+m failure domains and recommends k+m+1 so recovery has somewhere to put data after a loss. An 8+3 profile across racks therefore wants twelve racks, which few on-premises sites have; across hosts, twelve hosts is realistic.

Placement Internals: CRUSH, Erasure Sets, Needles, and Layout Rings

Understanding how each system chooses drives explains almost every operational behaviour you will see later, from rebalancing storms to why one cluster tolerates a rack loss and another does not.

Figure 3: Placement in Ceph (placement groups and CRUSH), SeaweedFS (master-assigned volumes of needles), and Garage (a zone-aware layout ring). MinIO’s erasure sets are described in text below.

Ceph RGW: CRUSH and placement groups

RGW is a gateway process that translates S3 into operations on RADOS, Ceph’s object layer. Objects land in pools; each pool is divided into placement groups (PGs); and CRUSH maps each PG to a set of OSDs (object storage daemons, normally one per drive) using the cluster’s CRUSH map and a failure-domain rule. No client ever asks a central server where data lives; it computes the location from the map.

For erasure-coded pools you define a profile. The Ceph documentation gives the default profile as k=2, m=2 with the jerasure plugin, technique reed_sol_van and a crush-failure-domain of host. A profile such as k=3, m=2 with crush-failure-domain=rack guarantees no two chunks share a rack, and the profile cannot be changed after the pool is created. This immutability is a real planning constraint: choose k and m before you have data.

Since the Octopus release, Ceph can recover a PG with at least k shards available, where earlier releases required min_size. The documentation still recommends min_size of k+1 or more, so that writes are not accepted when only k shards are available and a single further loss would destroy data. RGW itself needs only full-object writes and appends, which erasure-coded pools support without enabling allow_ec_overwrites; that flag, which requires BlueStore, matters for RBD and CephFS rather than for object workloads.

The cost of this generality is operational surface: monitors, managers, OSDs, a CRUSH map, PG autoscaling, and the RGW layer with its own realms, zonegroups and zones for multi-site. You gain the most mature S3 feature set of the four, plus block and file storage from the same cluster.

MinIO: erasure sets

MinIO drops the map. Drives in a server pool are grouped into erasure sets of equal size. Per the AIStor documentation, sets default to 2 to 16 drives, with a larger limit of 32 available from a February 2026 release through a configuration variable, and the set size is fixed when the pool is initialized. Each object is split into K data and M parity shards, N = K + M, using Reed-Solomon coding, with parity configurable from EC:0 up to half the set size.

Quorum rules are crisp. A read needs at least K healthy shards. A write attempts all N shards in parallel and needs at least K successes; when parity is exactly half the set, write quorum rises to K+1 to prevent split-brain writes. The docs give a 12-drive set with EC:3 as an example: twelve write attempts, nine required. Production deployments are told to use EC:3 or higher, and the documented default parity in the capacity table is EC:4.

One notable feature, documented for the current AIStor line, is parity auto-upgrade: if drives are offline during a PUT, that object is written with extra parity (EC:4 with two drives offline becomes EC:6), bounded by a budget defaulting to 1% of set capacity. Whether the abandoned community code line behaves identically is not something I verified, so treat this as a property of the vendor’s current product.

The structural trade-off is that capacity is added in whole server pools, not drive by drive. Because erasure sets are fixed at creation, you grow a MinIO deployment by adding another pool of identical shape, not by expanding an existing set.

SeaweedFS: volumes and needles

SeaweedFS follows the Haystack idea: instead of one filesystem file per object, many objects are appended into a large volume file, and a small in-memory index maps each object to an offset. The project README states that each data volume is 32 GB, that the master manages only volumes on volume servers (not individual files), and quotes figures of about 40 bytes of disk overhead per file’s metadata, and a 16-byte index entry in memory per file (a 64-bit key, 32-bit offset and 32-bit size). These come from the README and describe the design intent; I have not benchmarked them.

The consequence is that a read needs no per-file filesystem lookup on the volume server: one in-memory lookup, one disk read. This is why SeaweedFS shines with billions of small objects, the exact case where the erasure-coded designs struggle. The master assigns file IDs and tracks which servers host which volume; the README advertises automatic master failover. Namespace features, directories and the S3 gateway live in the filer, a separate stateless service with pluggable metadata stores.

Protection comes in two flavours: replication configured per volume, and erasure coding for older, colder volumes. The README lists a rack-aware 10.4 scheme (ten data, four parity) and notes that the Enterprise edition can customize the ratio. The license is Apache 2.0 according to the README.

Garage: layout rings and zones

Garage rejects erasure coding on principle. Its design goals list “extreme performance” and storage optimisation as non-goals and say erasure coding is avoided because it complicates data placement and synchronization. Data is duplicated, and the recommended configuration is a replication factor of three across zones, a zone being a datacenter or geographic location assigned with garage layout assign -z.

You also assign each node a capacity, and Garage balances data accordingly. A consequence the documentation spells out is that three copies must go to three different locations, so the smallest zone caps usable capacity: with 1.5 TB in one site and larger capacity elsewhere, usable space is bounded near 1.5 TB. Keep zones roughly equal.

Reads and writes follow a Dynamo-style quorum idea. With three replicas, Garage’s internals documentation says it queries the two preferred nodes for metadata and falls back to the third if one fails; at least two metadata responses must match. A node prefers itself, then nodes in its own zone, then the lowest-latency nodes. Garage maintains CRDT-based tables and keeps tombstones, because deleted values could otherwise reappear after a partition.

Metadata sits in an embedded database. The Garage cookbook describes LMDB as fastest and most tested but not portable across CPU architectures, with reports of corruption after unclean shutdowns (so periodic snapshots are advised, for instance a six-hour automatic snapshot interval), and SQLite as the more robust alternative. Block size can be raised to around 10 MB for fast networks and mostly large files.

The design is small and calm, but it carries two firm limits: three full copies of every byte, and a feature set limited to the S3 API.

S3 API Compatibility: Where Each System Stops

“S3 compatible” is a spectrum, not a boolean. The features that decide whether a workload runs are rarely GET and PUT; they are multipart upload, versioning, object lock, lifecycle rules, bucket policies, notifications, and conditional behaviour on listing.

Feature	Ceph RGW	MinIO	SeaweedFS	Garage
Multipart upload	Yes	Yes	Yes	Yes
Bucket versioning	Yes	Yes (reported)	Yes	No (not yet)
Object lock and legal hold	Reported yes, verify per release	Yes (reported)	Yes	No
Bucket policies and ACLs	Policies and canned ACLs	Policies (reported)	IAM-style policies	Own per-key permissions only
Lifecycle	Yes	Yes (reported)	Expiration yes, transition no	Expiration and multipart abort only
Event notifications	Yes	Yes (reported)	No	No
Replication API	Across zones only	Yes (reported)	No	No
Server-side encryption	Yes (reported)	Yes (reported)	Not verified	SSE-C only

Cells marked “reported” reflect common knowledge or secondary sources that I did not confirm on a primary page during this run. The Ceph, SeaweedFS and Garage cells for versioning, lifecycle and notifications come from their official documentation pages. Always re-verify the specific feature for the specific version you will deploy.

Ceph RGW

The Reef-era compatibility table lists bucket object versions, multipart, ACLs and bucket policies, lifecycle, notifications, tagging, storage classes, website, POST Object and Copy Object as supported, and bucket replication as supported only across zones. The page does not mention object lock, so I cannot confirm it from that page; RGW has documented object lock support elsewhere, but check your release.

Garage

The Garage S3 compatibility page is unusually candid. It supports Signature V4, presigned URLs, path and virtual-host addressing, SSE-C, and the full multipart set including UploadPartCopy. It does not support Signature V2, bucket versioning (“Garage does not (yet) support object versioning”), object lock, bucket policies, ACL writes, replication configuration, bucket-level encryption, tagging, or notifications. Missing endpoints return HTTP 501. Lifecycle is partial: only abort-incomplete-multipart and expiration work.

For many workloads that list is fine. For compliance retention (WORM) or ransomware-resistant backups that depend on object lock, it is disqualifying.

SeaweedFS

Its S3 documentation lists versioning (except MFA Delete), object lock with legal hold and retention, all seven multipart operations, IAM-style users, groups and policies, and STS. Unsupported or partial items include lifecycle transition rules, replication and notification configuration, website and logging configuration, and RestoreObject and SelectObjectContent. It also notes behavioural differences: a file and a folder cannot share the same path, and only “/” is accepted as a delimiter. The path rule is the one that surprises people: keys such as a/b and a/b/c cannot coexist, which some applications generate.

MinIO

MinIO historically had the most complete S3 behaviour of the group, including versioning, object lock, replication, and notifications. That is the case for the code that exists. The risk now is not compatibility with today’s clients but drift: the community repository will receive no new fixes, so new S3 features or security patches will not arrive.

The compatibility test you should run anyway

Vendor feature tables are claims. A short contract test that exercises exactly the calls your applications make is the real acceptance criterion. The script below uses boto3 and fails loudly on the first unsupported call.

import io, sys, boto3
from botocore.config import Config
from botocore.exceptions import ClientError

ENDPOINT = "http://127.0.0.1:9000"   # your gateway
s3 = boto3.client(
    "s3", endpoint_url=ENDPOINT,
    aws_access_key_id="TESTKEY", aws_secret_access_key="TESTSECRET",
    region_name="us-east-1",
    config=Config(signature_version="s3v4",
                  s3={"addressing_style": "path"}),
)
BUCKET = "contract-test"

def check(name, fn):
    try:
        fn(); print(f"PASS  {name}")
    except ClientError as e:
        print(f"FAIL  {name}: {e.response['Error']['Code']}")
    except Exception as e:
        print(f"FAIL  {name}: {type(e).__name__} {e}")

check("create bucket", lambda: s3.create_bucket(Bucket=BUCKET))
check("put/get small", lambda: (
    s3.put_object(Bucket=BUCKET, Key="a/b.txt", Body=b"hello"),
    s3.get_object(Bucket=BUCKET, Key="a/b.txt")["Body"].read()))
check("key as prefix and object", lambda: (
    s3.put_object(Bucket=BUCKET, Key="p", Body=b"x"),
    s3.put_object(Bucket=BUCKET, Key="p/q", Body=b"y")))

def multipart():
    key, size = "big.bin", 5 * 1024 * 1024   # S3 minimum part size
    mp = s3.create_multipart_upload(Bucket=BUCKET, Key=key)
    parts = []
    for n in (1, 2):
        r = s3.upload_part(Bucket=BUCKET, Key=key, UploadId=mp["UploadId"],
                           PartNumber=n, Body=io.BytesIO(b"z" * size))
        parts.append({"ETag": r["ETag"], "PartNumber": n})
    s3.complete_multipart_upload(Bucket=BUCKET, Key=key,
        UploadId=mp["UploadId"], MultipartUpload={"Parts": parts})
check("multipart 2x5MiB", multipart)

check("presigned url", lambda: s3.generate_presigned_url(
    "get_object", Params={"Bucket": BUCKET, "Key": "a/b.txt"}))
check("enable versioning", lambda: s3.put_bucket_versioning(
    Bucket=BUCKET, VersioningConfiguration={"Status": "Enabled"}))
check("list versions", lambda: s3.list_object_versions(Bucket=BUCKET))
check("lifecycle expiry", lambda: s3.put_bucket_lifecycle_configuration(
    Bucket=BUCKET, LifecycleConfiguration={"Rules": [{
        "ID": "exp", "Status": "Enabled", "Filter": {"Prefix": "tmp/"},
        "Expiration": {"Days": 1}}]}))
check("object lock config", lambda: s3.get_object_lock_configuration(
    Bucket=BUCKET))
check("bucket policy", lambda: s3.get_bucket_policy(Bucket=BUCKET))

The equivalent smoke check with awscli is aws --endpoint-url http://127.0.0.1:9000 s3 cp big.bin s3://contract-test/ followed by aws --endpoint-url ... s3api list-object-versions --bucket contract-test. Note that object lock normally must be enabled at bucket creation time on AWS, so an “object lock config” failure on a bucket created without it is expected on some systems; create a second bucket with ObjectLockEnabledForBucket=True to test that path properly. Running the same script against all candidates gives you a matrix grounded in your own workload rather than in documentation.

Failure, Rebuild, and Consistency Behaviour

Architecture diagrams look identical until a drive dies at 3 a.m. The questions that matter are how much data moves, who coordinates it, and what clients see in the meantime.

Consistency

Modern S3 on AWS is strongly consistent for reads after writes and list operations, and clients increasingly assume it. Ceph RGW reads and writes through RADOS, whose primary-copy model for each PG gives strong consistency for object operations within a zone; multi-site replication between zones is asynchronous, so a secondary zone can lag. MinIO’s quorum model (K shards to read, K to write) is likewise strict within a deployment. SeaweedFS’s volume layer writes to replicas synchronously according to its replication setting, while its filer inherits the consistency of whatever metadata store you choose. Garage uses quorum reads and writes over CRDT tables, which is designed to tolerate network partitions between sites, but its own documentation describes the internals page as incomplete, so I will not claim a stronger guarantee than the published pages support. If your application relies on read-after-write listing semantics, test it under partition rather than assuming it.

Rebuild behaviour

In Ceph, an OSD failure marks its PGs degraded; after a configurable interval the OSD is marked out and CRUSH remaps its PGs, so recovery traffic is spread across many surviving OSDs. This many-to-many repair is Ceph’s best durability property, because repair time shrinks as the cluster grows. It also means recovery competes with client I/O, and operators throttle it with backfill and recovery settings. For an erasure-coded pool, every regenerated chunk costs k reads.

In MinIO, healing works within an erasure set. A replaced drive is rebuilt from the other drives of its set, so repair parallelism is bounded by the set width (up to 16 drives by default), not by the whole pool. A bigger fleet therefore does not repair a single failure faster; it only means more sets are exposed to failures at any time.

SeaweedFS repairs at volume granularity. Because volumes are large, re-replicating a 32 GB volume is a coarse, sequential operation, which is efficient on spinning disks but slower to restore full redundancy per failure than a fine-grained scatter. Erasure-coded volumes use shards placed across racks per the README’s description.

Garage re-balances by recomputing its layout. When you add or remove nodes you apply a new layout and partitions migrate; because each partition has three copies in different zones, losing a whole zone leaves two copies of everything, and the cluster stays readable. A cluster with fewer than three zones cannot satisfy the placement rule fully, which is why the documentation recommends at least three nodes.

What a rack loss costs you

Scenario	Ceph (EC 4+2, rack domain)	MinIO (EC:4, 16 drives)	SeaweedFS (3 copies)	Garage (3 zones)
One drive fails	Auto repair, many-to-many	Heal within set	Re-replicate volumes	Resync partitions
One whole node fails	Auto repair; PGs degrade	Up to node’s drives per set must stay within parity	Volumes re-replicated	Two copies remain per partition
One rack or zone fails	Survives only if no more than m=2 chunks per rack	Survives only if shards were spread so losses stay within parity	Survives if replicas placed in other racks	Survives by design

The qualifier in each cell is the point: survivability under correlated failures is a placement-configuration property, not a product property. A MinIO pool with all drives of a set inside one chassis does not survive that chassis; a Ceph CRUSH rule with host as failure domain does not survive a rack.

Licensing and Project Health in 2026

Licensing is part of architecture because it determines whether you can patch, redistribute, or embed the system, and whether a vendor can change terms underneath you.

MinIO. Verified from the repository on GitHub: the repo was archived by its owner on 25 April 2026 and is read-only; the README says it is no longer maintained, that the community edition is now distributed as source only, that pre-compiled binaries are no longer provided, and that legacy binaries will not receive updates. The source is licensed under GNU AGPLv3 and the documentation under CC BY 4.0. The README points users toward commercial AIStor Free and AIStor Enterprise offerings. What this means practically: you may still build and run the AGPL code, and AGPL obligations (offering source to network users of a modified version) still apply, but you will not get upstream fixes. Third-party write-ups describe earlier steps (management features removed from the community console in 2025, public images and binaries ending later that year, maintenance-mode messaging, and an earlier archive-and-unarchive in 2026); those are secondary sources that I did not verify individually.

Ceph. Ceph is open source, backed by the Ceph Foundation under the Linux Foundation, and ships in distributions and commercial products from several vendors. I did not re-verify the license text in this run; its long-standing terms are LGPL for most components, and you should confirm for your release.

SeaweedFS. Apache License 2.0 per the README. There is an Enterprise edition with additional features (the README mentions customizable erasure coding ratios), so check which features you depend on are in the open-source tree.

Garage. Developed by Deuxfleurs and distributed from its own Git forge. I did not verify the license text in this run; the project is widely described as AGPL-licensed, which would matter for modified-network-service deployments, so confirm before embedding it in a product.

The AGPL deserves a plain explanation. It is a copyleft license that extends source-disclosure obligations to users who interact with a modified program over a network. Running an unmodified build for internal use is typically uncomplicated, but if you patch and offer the service externally, legal review is warranted. This is general information, not legal advice.

A practical project-health test: look at the last release date, the number of active maintainers, whether there is a security-advisory process, and whether the project has a funding model that survives key-person departure. By those yardsticks Ceph has the strongest institutional backing, SeaweedFS has an active single-lead community plus a commercial arm, Garage has a small dedicated team with grant-style funding, and MinIO’s community line is now effectively frozen.

Operational Cost: What You Actually Run

Figure 4: A simplified decision flow for S3 compatible object storage in 2026. MinIO’s community line is excluded for new greenfield builds because it is archived.

Capital cost is dominated by raw disk and therefore by the overhead factors computed earlier. Operating cost is dominated by people and by the number of moving parts.

Ceph has the largest surface: monitors (an odd number, typically three or five), managers, OSD daemons per drive, RGW daemons behind a load balancer, plus PG autoscaling, CRUSH rule design, and upgrade orchestration (cephadm or Rook on Kubernetes). A competent two-person platform team can run it, but expect a learning curve measured in months. Its reward is a single platform that also serves block volumes to Kubernetes and file shares.

MinIO (if you keep running it) was the simplest to deploy: identical servers, a handful of flags, no map. That simplicity now has to be weighed against the absence of upstream security patches; a team would need to build from the archived source, track CVEs itself, or migrate to a commercial edition or fork.

SeaweedFS has three roles: master, volume server and filer (with an external metadata store if you use the S3 and filer features). The metadata store is the part that needs real operations, because it is your namespace. If the filer’s backing database is lost, the objects still sit in volumes but their names are gone.

Garage is the lightest: one binary, an embedded metadata database, and a layout you apply with a CLI. It runs happily on small machines and mixed hardware, which is why it appears in homelab and edge stories. The operational cost is accepting its limits.

A rough sizing comparison

For 1 PB usable data, assuming 16 TB drives and ignoring spares and filesystem reserve: 3x replication needs 3 PB raw, about 188 drives; an 8+3 code needs about 1.375 PB, about 86 drives; EC 4+2 needs 1.5 PB, about 94 drives. These are straight arithmetic from the overhead factors. The 100-odd drives saved by moving from replication to 8+3 are worth tens of thousands of dollars in hardware, plus power and rack space, which is also why carbon accounting follows the same ratio; our article on FinOps and GreenOps cost and carbon-aware scheduling discusses how to turn such ratios into reporting.

Decision Matrix

Scores below are my judgement for a typical 2026 self-hosted deployment, not benchmark results. “Good” means few caveats, “OK” means workable with care, “Weak” means a known gap.

Criterion	Ceph RGW	MinIO (archived CE)	SeaweedFS	Garage
Petabyte-scale capacity efficiency	Good (EC, flexible profiles)	Good (EC sets)	Good (EC on warm volumes)	Weak (3 copies only)
Billions of small objects	OK	OK	Good (needle design)	OK
Versioning and object lock	Good (verify lock)	Good but frozen	Good	Weak (absent)
Multi-site and geo-distribution	Good (multi-site, async)	OK	OK	Good (core purpose)
Operational simplicity	Weak	Good	OK	Good
Edge or small hardware	Weak	OK	OK	Good
Upstream health and patches	Good	Weak (archived)	OK to Good	OK
Unified block, file and object	Good	Weak	OK (filer, mount)	Weak

Read it by workload. An enterprise data lake with retention rules and several hundred terabytes to petabytes points to Ceph RGW, accepting its operations burden. A fleet of thin edge sites that each need a local bucket and cross-site resilience points to Garage, provided you do not need versioning or object lock. A photo, log or artifact store with huge numbers of small files points to SeaweedFS. A new deployment of MinIO community edition is hard to justify; an existing one needs a migration plan, either to a commercial edition or to one of the other three.

Trade-offs, Gotchas, and What Goes Wrong

Immutable erasure parameters. In Ceph the EC profile cannot change after pool creation, and in MinIO the erasure set size is fixed at pool initialization. Picking 4+2 because you start with six nodes and then growing to forty leaves you with an inefficient layout for years. Migration means creating a new pool and copying.

Too few failure domains. An 8+3 profile with crush-failure-domain=host on eleven hosts leaves no room to recover after a host loss, because there is nowhere to place the missing chunk. Follow the k+m+1 guidance.

Rebuild storms. Wide erasure codes make repair read-heavy: regenerating one chunk of a 10+4 code reads ten. On a busy cluster this saturates network links and degrades client latency. Throttle recovery and size the network for the repair case, not only steady state.

Small objects on erasure-coded pools. Shards of a few hundred bytes waste space and IOPS. Put small objects on replicated pools or use a store designed for them.

Silent compatibility gaps. An SDK may succeed on PUT and then fail later on a call the target returns 501 for. Garage returns 501 for missing endpoints, which is honest, but a system that returns 200 and ignores a header is worse. Run the contract test above in CI against your real endpoint.

Path semantics. SeaweedFS refuses a key that is both a file and a folder prefix; applications that write p and p/q will fail. Test your key naming.

Frozen dependency risk. Choosing archived software is a conscious acceptance of owning its security lifecycle. If you do, pin the build, subscribe to the relevant CVE feeds, and schedule an exit.

Metadata backend as single point of failure. For SeaweedFS the filer store, for Garage the LMDB or SQLite files, and for Ceph the monitor store are the irreplaceable pieces. Snapshot and test restores of metadata, not just of data. The Garage cookbook explicitly reports LMDB corruption after unclean shutdowns and advises regular snapshots.

Correlated failures break the math. The durability table assumed independence. Same-batch drives, shared power feeds, and bad firmware invalidate it. Mix vendors and batches where possible and treat the durability numbers as a floor on risk, not a guarantee.

Practical Recommendations

Start from requirements, not products. Write down your needs for versioning, object lock, replication between sites, the largest bucket’s object count, typical object size, and the number of people available to operate the cluster. Then eliminate: if object lock is mandatory, Garage is out; if you have one infra engineer and three sites, Ceph is a stretch; if you have billions of KB-sized objects, take SeaweedFS seriously.

For a new on-premises platform with petabyte ambitions and a platform team, deploy Ceph with an erasure-coded data pool for RGW, a replicated pool for the bucket index and metadata, and at least k+m+1 failure domains. For a small or geo-distributed footprint without compliance locking, deploy Garage with three zones and equal capacity per zone. For small-object-heavy workloads, deploy SeaweedFS with an externally managed, backed-up filer store. If you already run MinIO community edition, treat the archive as a migration trigger, not an emergency: inventory the features you use, run the contract test against candidates, and use replication or a copy tool such as rclone to move buckets.

A short checklist before sign-off:

Contract test passes on the exact version you will run, including multipart and any lock or versioning calls.
Erasure profile chosen with at least k+m+1 failure domains, and the profile is documented as irreversible.
Rebuild time measured on a real failure drill; durability recalculated with that window.
Metadata backup and restore tested end to end.
License reviewed by someone qualified, especially AGPL components that you modify.
Exit plan written: how to copy all buckets elsewhere using standard S3 tools.

Frequently Asked Questions

What is S3 compatible object storage?

It is any storage system that exposes the Amazon S3 REST API so existing S3 clients and SDKs work against it unchanged. Data is held as immutable objects in flat buckets, addressed by key, and protected by replication or erasure coding. Compatibility varies: core calls like PUT, GET and multipart are nearly universal, while versioning, object lock, lifecycle transitions and notifications differ between implementations, so verify the specific features you need.

Ceph vs MinIO: which should I choose in 2026?

For new deployments, Ceph is the safer choice because it is actively maintained and covers block, file and object storage. MinIO’s community repository was archived on 25 April 2026 and is read-only, so it receives no upstream fixes. MinIO remains simpler to operate and is still usable from source under AGPLv3, but you would own security patching or move to a commercial edition. Existing MinIO users should plan a migration.

Is erasure coding better than replication?

For capacity, yes: 8+3 stores about 1.375 bytes per byte versus 3.0 for triple replication, while tolerating three losses instead of two. The trade-offs are higher CPU and network use, slower degraded reads, repair traffic of roughly k times the data rebuilt, and poor efficiency on small objects. Replication wins for small objects, tiny clusters, and simplicity. Many production systems use both on different pools.

Does Garage support erasure coding or versioning?

No to both, as of its published documentation. Garage deliberately avoids erasure coding and keeps full copies, typically three, placed across zones. Its S3 compatibility page states it does not yet support object versioning and does not support object lock; missing endpoints return HTTP 501. It does support multipart uploads, presigned URLs, SSE-C and basic lifecycle expiration, which covers many backup and static-hosting uses.

Is SeaweedFS good for small files?

Yes, that is its original design purpose. Following the Haystack approach it appends many files into 32 GB volumes and keeps a compact in-memory index, which the README describes as about 16 bytes per file in memory and about 40 bytes of on-disk metadata overhead, so reads cost roughly one disk access. The filer, which provides directories and the S3 API, needs its own metadata store that you must back up.

How do I test whether a store is really S3 compatible?

Write a contract test with boto3 or awscli that exercises the exact operations your applications use: create bucket, put and get, multipart with parts of at least 5 MiB, presigned URLs, versioning, lifecycle, object lock, and key names that overlap as prefixes. Run it in CI against each candidate and each upgrade. Treat any 501, ignored header, or listing discrepancy as a finding, because those fail silently in production.