Ceph

Last reviewed Aug 26, 2026 · 885 words

Ceph's entry price in a homelab is 3 nodes, about 4 GB of RAM for every disk you give it, and a 10 GbE network between them. Below that it works in the sense that the cluster comes up and reports healthy, and fails in the sense that a single rebuild will saturate a gigabit link for hours while your VMs crawl. The catalogue's 4 GB minimum is the floor for a demo, not for storage you would keep photos on. Budget honestly before you start, because Ceph is the one self-hosted project where under-provisioning does not produce a slow system, it produces a system you are afraid to touch.

What the bill buys: one cluster, three storage APIs

Ceph, LGPL-2.1, C++, first released in 2006 and around 17,000 stars, presents the same pool of disks as three different things. RBD gives block devices for virtual machine disks. CephFS gives a POSIX filesystem, served through metadata servers, that you can mount on any Linux box. RGW gives an S3-compatible object store on port 7480. All three sit on RADOS, which spreads objects across disks according to a placement map and heals itself when a disk or a node disappears. There is no controller node to lose; monitors form a quorum (3 is the practical minimum), managers run the dashboard and metrics, and OSD daemons own one disk each. That "no single point of failure" line in the catalogue is accurate and it is the entire reason to accept the complexity.

Replication maths: three copies means a third of your disks

The default pool keeps 3 copies and refuses writes below 2, so 12 TB of raw disk across 3 nodes becomes 4 TB usable, and a node can die without the cluster stopping. Erasure coding improves the ratio (a 4+2 profile gives 67% usable) but needs at least 6 failure domains, which in a homelab means 6 hosts, so almost everyone runs replication. The other number that matters is osd_memory_target, which defaults to 4 GiB per OSD. A node with 4 disks wants 16 GB for the OSDs alone before the monitor, manager, and whatever else lives there. Consumer SSDs without power-loss protection are the classic trap: Ceph does sync writes, and drives that fake them are either slow or unsafe.

Proxmox or MicroCeph is the sane on-ramp

Nobody should bootstrap Ceph from scratch for a first cluster. Proxmox VE has it integrated: pveceph install on each node, pveceph init on the first, then monitors, managers and OSDs are buttons in the web UI, and the resulting RBD pool appears as VM storage. That is the path most homelab Ceph clusters take, and the Proxmox, Docker, or both piece covers the hypervisor decision that precedes it. Outside Proxmox, Canonical's MicroCeph packages the whole thing as a snap:

sudo snap install microceph
sudo microceph cluster bootstrap
sudo microceph disk add /dev/sdb --wipe
sudo ceph status

Rook does the same job inside Kubernetes, and plain cephadm bootstrap --mon-ip 10.0.0.11 is the upstream route for everyone else. The dashboard listens on 8443 and is good enough that you rarely need the CLI for day-to-day work.

One node is a demo, not storage

You can force Ceph onto a single machine by setting the failure domain to osd instead of host, and every tutorial that shows this should carry a warning. Three copies on one box protects against a dead disk and nothing else, at three times the cost of a ZFS mirror with worse performance. If you own one server, ZFS with 3-2-1 backups is the correct answer and Ceph is a learning exercise. If you want S3 on one box, MinIO or SeaweedFS give you that in a single container on 512 MB, and the backup category has the tooling to point at it.

The operational reality

Day-to-day Ceph is quiet and then briefly terrifying. A failed disk triggers a rebuild that competes with client traffic; on gigabit that fight is visible, on 10 GbE it is not. Upgrades go a release at a time and monitors before OSDs. The two habits that keep it boring are watching ceph status for anything other than HEALTH_OK, and never letting the cluster pass 80% full, because a full Ceph cluster stops accepting writes and getting it back is a bad weekend. Expect to spend a few evenings learning placement groups and CRUSH rules; the concepts are not hard, but there are more of them than in any other storage system on this site.

What I'd do

Run Ceph if you already have 3 or more Proxmox nodes with 10 GbE between them and at least 2 enterprise SSDs each; in that setup it is the best shared storage you can get, live migration works, and a dead node is a Tuesday. Use the Proxmox integration, keep replication at 3, keep usage under 80% and treat the dashboard as your first stop. Below that hardware line, put ZFS on one box and MinIO beside it for S3, back up properly, and come back to Ceph when the third node arrives.