Ceph on Kubernetes: Rook, ceph-csi and CephFS

What is Ceph in Kubernetes?

Ceph is an open-source distributed storage system that delivers block, file and object storage. Kubernetes (K8s) uses it for persistent volumes through the ceph-csi driver. RBD images back ReadWriteOnce block volumes, and CephFS backs ReadWriteMany shared file systems. The Rook operator can run Ceph inside your cluster, or ceph-csi can connect to a separate Ceph cluster.

Kubernetes 1.31 removed its in-tree rbd and cephfs volume plugins and pointed users to the Ceph CSI drivers (v1.31 release). Older tutorials also use the external ceph.com/rbd and ceph.com/cephfs provisioners, whose repository is archived. This guide reflects Ceph Tentacle 20.2, Rook 1.21, ceph-csi 3.18 and Kubernetes 1.37, with sources checked in October 2026. We run Ceph in production.

How Ceph works

Ceph stores data as objects in pools, inside RADOS, its distributed object store (CephFS docs). The CRUSH algorithm computes which placement group (PG) holds each object and which OSDs hold each PG (Intro to Ceph). So clients talk to OSDs directly, with no central lookup server (CRUSH maps).

A Ceph cluster runs five main kinds of daemons (Intro to Ceph):

OSD stands for Object Storage Daemon. In most cases each OSD owns one drive, as Ceph's hardware recommendations advise. By default, Ceph keeps three copies of every object (pools). CRUSH rules place the copies in different failure domains, such as hosts or racks (CRUSH maps). Erasure-coded pools store data chunks plus coding chunks, which saves space at scale (erasure code).

Ceph offers three interfaces, and each maps to a different kind of Kubernetes storage (ceph-csi, Ceph's guide):

RGW's S3 interface is compatible with "a large subset of the Amazon S3 RESTful API" (Object Gateway). Rook runs RGW from a CephObjectStore resource, and apps request buckets with an ObjectBucketClaim (object storage).

Rook vs an external Ceph cluster

There are two ways to give Kubernetes Ceph storage. Rook, a CNCF graduated project, runs the Ceph daemons as pods in your cluster and manages them through custom resources such as CephCluster (storage architecture). Or a separate Ceph cluster serves the storage, and Kubernetes connects to it with ceph-csi.

Ceph's install docs recommend two tools, cephadm and Rook (installing Ceph). They call Rook "the way to run Ceph in Kubernetes or to connect an existing Ceph storage cluster to Kubernetes". The ceph-deploy tool from older tutorials is deprecated. Ceph's hardware guide also recommends separate hosts for processes that use the data cluster, Kubernetes included (hardware recommendations). Rook's external mode supports that setup.

Rook fits when:

• your nodes have spare local disks, as on bare metal, and one team runs both layers;
• you want storage defined in Kubernetes resources, like the rest of the cluster;
• you're ready for node drains and upgrades to move storage daemons too. Rook manages PodDisruptionBudgets for OSDs, mons, RGW and MDS, and blocks unsafe OSD evictions (CephCluster CRD).

An external cluster fits when:

• a storage team already runs Ceph, or Ceph also serves VMs;
• several Kubernetes clusters share it;
• Ceph upgrades must follow their own schedule.

Rook isn't in the Ceph data path, so apps read and write to Ceph directly (storage architecture). If apps and OSDs share nodes, keep volumes on ext4. Rook's example StorageClass warns that XFS can deadlock when a volume is mounted on the same node as the OSDs (storageclass.yaml). Node affinity and tolerations can keep OSDs on dedicated storage nodes (CephCluster CRD). Our Kubernetes on bare metal guide covers the rest of a self-hosted cluster.

How to deploy Ceph on Kubernetes with Rook

To deploy Ceph on Kubernetes with Rook, install the Rook operator, create a CephCluster, and then add pools and StorageClasses. Rook 1.21 supports Kubernetes 1.32 to 1.37 on amd64 and arm64 (prerequisites). Its example cluster runs Ceph Tentacle 20.2.4 (cluster.yaml).

Rook prerequisites

• Nodes. The production example, cluster.yaml, needs at least three worker nodes. cluster-test.yaml runs on one node without redundancy, for tests only (example configurations).
• Disks. OSDs need raw devices, raw partitions, LVM logical volumes or block-mode PVs, with no filesystem on them. Usable disks show an empty FSTYPE in lsblk -f (prerequisites).
• Packages. Hosts need lvm2 for encrypted OSDs, a separate metadata device or more than one OSD per device. Every storage node needs a running udev daemon (prerequisites).
• A safe test bed. Rook's quickstart says to test in virtual machines, never on a host whose local disks could be consumed by mistake (quickstart).

Kernel versions for Ceph clients

By default, ceph-csi mounts volumes with the Linux kernel's Ceph clients (Ceph's guide, CephFS deploy). So the node kernel decides which features work, with Rook and with an external cluster alike.

GKE's Container-Optimized OS, for example, has no RBD module (prerequisites). On kernels older than 5.11, set network.connections.requireMsgr2: false in the CephCluster (Rook upgrades). CentOS 8.4 or later supports msgr2 (CephCluster CRD), and CentOS Stream 9 and 10 support aes256k keys through backports (CVE-2025-30156). ceph-csi still calls its Arm64 support experimental (ceph-csi).

Install the Rook operator and a Ceph cluster

The quickstart installs Rook from the example manifests at the release tag (quickstart):

git clone --single-branch \
  --branch v1.21.0 \
  https://github.com/rook/rook.git
cd rook/deploy/examples
kubectl create -f crds.yaml \
  -f common.yaml \
  -f csi-operator.yaml
kubectl create -f operator.yaml
kubectl -n rook-ceph get pod
kubectl create -f cluster.yaml

Create the cluster once the rook-ceph-operator pod is Running. csi-operator.yaml installs the ceph-csi-operator, which runs the CSI drivers for Rook (CSI drivers chart). In Rook 1.21, cluster.yaml sets these defaults (cluster.yaml):

• three mons and two mgrs, each on a different node;
• an OSD on every empty device of every node, through useAllNodes and useAllDevices;
• the Ceph dashboard, with TLS;
• msgr2 required, and daemon data in /var/lib/rook on each host, a path that must survive reboots (quickstart).

Install Rook with Helm charts

With Helm, Rook uses three charts (Helm charts). rook-ceph installs the operator, ceph-csi-drivers the CSI drivers, and rook-ceph-cluster the CephCluster, pools and StorageClasses. The operator chart goes first (operator chart):

helm repo add rook-release \
  https://charts.rook.io/release
helm install -n rook-ceph \
  --create-namespace rook-ceph \
  rook-release/rook-ceph

Install the drivers chart from the ceph-csi-operator Helm repository with Rook's recommended values file, because the drivers fail with the chart defaults alone (CSI drivers chart). The cluster chart's defaults "are only an example", so write your own values.yaml for it (cluster chart).

Check the cluster with the toolbox

kubectl create -f toolbox.yaml
kubectl -n rook-ceph rollout \
  status deploy/rook-ceph-tools
kubectl -n rook-ceph exec -it \
  deploy/rook-ceph-tools -- \
  ceph status

A healthy cluster has all mons in quorum, an active mgr, at least three OSDs up and in, and HEALTH_OK (quickstart). On Ceph 19.2.6, 20.2.4 and later, a new cluster may report HEALTH_WARN for insecure key types. Rook's docs offer two fixes, rotating keys to AES256K on Linux 7.0 nodes or muting the warnings (common issues). The kubectl-rook-ceph plugin runs the same checks without a toolbox, as in kubectl rook-ceph ceph status (kubectl plugin).

How to connect an external Ceph cluster with ceph-csi

When Ceph runs outside Kubernetes, you can install ceph-csi yourself, without Rook, or let Rook run it in external mode.

Install ceph-csi with Helm

ceph-csi deprecated its Helm charts in 3.18 and will remove them in 3.19, in favor of the Ceph-CSI Operator that Rook runs (v3.18.1 release). The 3.18.1 charts still install.

Ceph's guide starts on the Ceph side (Block Devices and Kubernetes). Create and initialize a pool, create a client user for Kubernetes, and note the cluster fsid and monitor addresses:

ceph osd pool create kubernetes
rbd pool init kubernetes
ceph auth get-or-create \
  client.kubernetes \
  mon 'profile rbd' \
  osd 'profile rbd pool=kubernetes' \
  mgr 'profile rbd pool=kubernetes'
ceph mon dump

The key from get-or-create goes into a Kubernetes Secret, and the user ID drops the client. prefix (secret example). Clusters first installed on Ceph 19.2.6, 20.2.4 or later create aes256k keys by default (CVE-2025-30156), which kernel clients accept from Linux 7.0. For older nodes, run ceph mon set auth_allowed_ciphers aes,aes256k, then add --key-type aes to the get-or-create command (CephX config).

Then put the cluster details in a values.yaml for the ceph-csi-rbd chart, version 3.18.1:

csiConfig:
  - clusterID: "<fsid>"
    monitors:
      - "192.168.1.1:6789"
      - "192.168.1.2:6789"
      - "192.168.1.3:6789"
secret:
  create: true
  userID: kubernetes
  userKey: "<key>"
storageClass:
  create: true
  clusterID: "<fsid>"
  pool: kubernetes

helm repo add ceph-csi \
  https://ceph.github.io/csi-charts
helm install ceph-csi-rbd \
  ceph-csi/ceph-csi-rbd \
  -f values.yaml \
  -n ceph-csi-rbd \
  --create-namespace

The clusterID must be unique, at most 36 bytes and unchanged for the cluster's life, and ceph-csi suggests the cluster's fsid (RBD deploy). The chart then creates the csi-rbd-sc StorageClass, with the layering image feature and ext4 (chart values).

ceph-csi 3.18.1 lists Kubernetes 1.34, 1.35 and 1.36 as tested, not yet 1.37 (ceph-csi). Rook 1.21 supports 1.37 and runs the same ceph-csi release (prerequisites, ceph-csi-operator 1.1.0).

Rook in external mode

Run Rook's create-external-cluster-resources.py on the Ceph cluster to create users and keys (export config). Its --restricted-auth-permission flag limits the CSI keys to named pools and one Kubernetes cluster. If any node runs a kernel older than Linux 7.0, also add --cephx-key-type aes. Then install Rook with cluster-external.yaml and import the output with import-external-cluster.sh, which creates the ceph-rbd and cephfs StorageClasses (import config).

Ceph RBD StorageClass for block volumes

With Rook, one example manifest creates a replicated pool and its StorageClass (block storage):

kubectl create -f \
  csi/rbd/storageclass.yaml

It creates the replicapool CephBlockPool, with three replicas on different hosts, and the rook-ceph-block StorageClass. The example enables only the layering image feature, for older kernels, and formats volumes as ext4 (storageclass.yaml). A PersistentVolumeClaim then requests a volume:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbd-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: rook-ceph-block

ReadWriteOnce lets one node mount the volume read-write, and pods on that node can share it. ReadWriteOncePod limits it to a single pod (access modes). ceph-csi marks ReadWriteOncePod for RBD as alpha, which its README says is not recommended for production (support matrix). Raw block PVCs, with volumeMode: Block, can also be ReadWriteMany (Ceph's guide).

If a PVC stays Pending or a volume is stuck on another node, our troubleshooting guide maps the events to fixes.

CephFS on Kubernetes for ReadWriteMany volumes

CephFS lets pods on many nodes mount the same file system read-write. ceph-csi creates each volume as a CephFS subvolume (CephFS deploy), and the MDS daemons protect the data when several pods write (storage architecture). With Rook, create the file system and then its StorageClass (filesystem storage):

kubectl create -f filesystem.yaml
kubectl create -f \
  csi/cephfs/storageclass.yaml

filesystem.yaml creates the myfs file system with three-way replicated pools, one active MDS with a standby, and the csi subvolume group. The StorageClass is rook-cephfs (filesystem.yaml). A shared claim looks like this:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cephfs-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: rook-cephfs

With an external Ceph cluster, the ceph-csi-cephfs chart works like the RBD chart, with its own Ceph user and the file system name in storageClass.fsName. ceph-csi lists the caps that user needs (capabilities). Create the chart's subvolume group first, csi by default, with ceph fs subvolumegroup create <fs> csi. Without it, PVCs stay Pending (CephFS deploy, config sample, subvolume groups).

Two details matter for CephFS:

• Kernel 4.17 or later. On older kernels, ceph-csi falls back to the FUSE client. FUSE mounts can drop from their pods when the CSI driver restarts, during an upgrade for example (ceph-csi upgrade). Older kernels also don't enforce PVC sizes (prerequisites).
• Fast metadata. MDS daemons are single-threaded and CPU-intensive. Ceph's guide suggests CPUs with high clock speeds, and SSDs for the metadata pool (hardware recommendations).

Volume expansion, snapshots, clones and encryption

Expansion. Both Rook StorageClasses set allowVolumeExpansion: true. To grow a volume, raise spec.resources.requests.storage in its PVC. Volume expansion has been stable in Kubernetes since 1.24 (persistent volumes), and ceph-csi lists it as beta for RBD and CephFS (support matrix).

Snapshots and clones. VolumeSnapshot objects are CRDs, not part of the core API, and need a snapshot controller (volume snapshots). If your distribution doesn't bundle them, install the CRDs and the controller from external-snapshotter, once per cluster. Then Rook's examples create a snapshot class, a snapshot and a restored PVC:

cd csi/rbd
kubectl create -f snapshotclass.yaml
kubectl create -f snapshot.yaml
kubectl create -f pvc-restore.yaml

The restored PVC names the snapshot as its dataSource. A clone uses an existing PVC in the same namespace as its dataSource (Rook clones, data sources). ceph-csi lists snapshots and clones as GA (support matrix). With the ceph-csi-drivers Helm chart, set snapshotPolicy: volumeSnapshot for RBD, because the default none turns the snapshotter off (Rook snapshots, drivers chart).

Encryption. ceph-csi can encrypt each RBD volume with LUKS when the StorageClass sets encrypted: "true". The passphrase comes from a Secret or a KMS such as HashiCorp Vault, and nodes need the dm-crypt module. You can't turn encryption on for existing volumes (RBD deploy).

Running Ceph in production

Hardware and network

Ceph's hardware recommendations set the baseline:

• Memory. Each BlueStore OSD targets 4 GiB by default (osd_memory_target). Give a server more RAM than its OSD count × osd_memory_target × 2, and budget at least 20% extra for spikes.
• Drives. Use one drive per OSD in most cases, at least 1 TiB each. Enterprise SSDs have power-loss protection. Ceph strongly suggests SSDs for monitor and manager hosts. CephFS metadata pools and RGW index and log pools need SSDs at enterprise scale.
• Controllers. You don't need a RAID controller. RAID-mode HBAs may add latency compared with simpler JBOD mode.
• Network. Provision at least 10 Gb/s between Ceph hosts and to clients, and 25 Gb/s for substantial workloads. Network speed limits recovery. Replicating 1 TiB takes about twenty minutes at 10 Gb/s and three hours at 1 Gb/s.

Capacity and full ratios

By default, Ceph flags an OSD as nearfull at 85% used and stops backfilling to it at 90%. At 95%, the full ratio, Ceph stops serving writes to prevent data loss (monitor config, health checks). Keep headroom for a failed host. Ceph's guide warns that recovery after a host failure can push OSDs past the full ratio (hardware recommendations).

Monitoring Ceph

The manager's prometheus module serves metrics on port 9283 by default, and ceph-exporter daemons export the daemons' performance counters (Prometheus module). Rook runs both. With Helm, set monitoring.enabled in both charts and monitoring.createPrometheusRules in the cluster chart to load Ceph's alert rules. Prometheus must already run in the cluster (Rook monitoring, cluster chart). Start with these rules from Rook 1.21 (localrules.yaml):

Our Kubernetes monitoring guide covers the Prometheus stack and the PVC usage alerts on the Kubernetes side.

Upgrading Rook and Ceph

• Rook. Each Rook upgrade guide covers one minor step, so the 1.21 guide covers 1.20.x to 1.21.x (Rook upgrades).
• Ceph under Rook. Change spec.cephVersion.image in the CephCluster. Rook upgrades the daemons one by one, waits for each to settle, and refuses to start while health is HEALTH_ERR (Ceph upgrades).
• Image tags. Pin a full build tag such as v20.2.4-20260818, so every daemon runs the same build. Skip Tentacle 20.2.0, which Rook doesn't recommend because of a data corruption issue with read affinity (Ceph upgrades).
• Release lifecycle. Squid (19.2) reaches its estimated end of life on October 31, 2026, and Tentacle (20.2) on June 1, 2027 (Ceph releases).
• External clusters. cephadm upgrades with ceph orch upgrade start --image and the new image. With Rook or cephadm, once a cluster runs Tentacle, you can't downgrade it to Reef or Squid (Tentacle notes).
• Security fixes. Ceph 19.2.6 and 20.2.4, from August 19, 2026, fix four CVEs (Tentacle notes). The fix for CVE-2025-30156 adds aes256k keys, and Ceph's docs say the upgrade raises six new health warnings and errors, which is normal (CVE-2025-30156). On older releases, CVE-2026-50152 lets any key with monitor read access read the config-key store, which holds secrets such as OSD LUKS passphrases (CVE-2026-50152).

If your Ceph cluster is slow, near full or hard to upgrade, our Kubernetes consulting team starts with read-only access and a written list of what to fix first.

OSD failures and recovery

When an OSD stops responding, the monitors mark it down, and after 10 minutes by default also out (monitor and OSD interaction). Ceph then rebalances and recovers its data onto the other OSDs (Intro to Ceph).

For planned maintenance, drain the node with kubectl drain, which respects PodDisruptionBudgets (safely drain a node). Rook's budgets then let OSDs go down in only one failure domain at a time (PDB design). Rook also holds that failure domain in noout for 30 minutes by default, which keeps Ceph from rebalancing (CephCluster CRD, OSD troubleshooting).

Rook 1.21 can replace a failed disk on host-based OSDs. You annotate the OSD's deployment, and Rook drains the OSD while keeping its ID and CRUSH position. Ceph then backfills only that disk's data (OSD management):

kubectl -n rook-ceph annotate \
  deployment rook-ceph-osd-5 \
  osd.rook.io/replace=yes-really-replace-osd-5

Swap the disk in the same host, and only after Rook adds osd.rook.io/replace-ready-for-swap to the deployment. Rook then provisions the new disk on its next reconcile. The operator's discovery daemon starts that reconcile, but it is off by default, so otherwise restart the operator pod. To remove an OSD for good, run kubectl rook-ceph rook purge-osd with its ID (OSD management).

Backups and disaster recovery

Ceph describes an RBD snapshot as "a read-only logical copy of an image at a particular point in time" (RBD snapshots). It lives in the same cluster as the volume, so it won't help if you lose that cluster. Velero's CSI Snapshot Data Movement copies snapshot data to a backup storage location. Its docs name on-premises storage without durable snapshots as one use case (Velero 1.18).

For a second site, Ceph mirrors RBD images asynchronously to another Ceph cluster (RBD mirroring), and cephfs-mirror copies CephFS snapshots to a remote file system (CephFS mirroring). Rook runs both through its CephRBDMirror and CephFilesystemMirror resources.

When not to use Ceph on Kubernetes

In production, Ceph needs at least three nodes, fast networks and disks it owns outright, as the sections above show. Small clusters and cloud clusters have other options:

Our guide to Kubernetes CSI explains how these drivers plug into Kubernetes. Our performance guide covers local disks for low latency.

---

We have run production systems since 2009 and have managed more than 1,350 production servers, and we run Ceph in production. If you'd rather not run storage alone, our Kubernetes consulting team starts with read-only access and a written list of what to fix first. We can also run your clusters afterwards. On-call is by agreement, with up to 24-hour coverage under an SLA, and post-incident reviews are part of on-call work.

Frequently Asked Questions

Is Ceph better than RAID?

Ceph and RAID protect data at different levels. RAID protects disks inside one server. Ceph keeps three copies by default, or erasure-coded chunks, on separate hosts or racks chosen by CRUSH, so it can lose a whole server. Ceph's docs compare its simplest erasure-coded pool to RAID 5 and say Ceph needs no RAID controller (erasure code, hardware).

What are the common storage providers for Kubernetes?

Most clusters use a CSI driver. Cloud clusters use their provider's block and file services, such as Amazon EBS and EFS, Google Persistent Disk and Filestore, or Azure Disk and Azure Files. Self-hosted clusters use Ceph through Rook or ceph-csi, Longhorn, OpenEBS or local persistent volumes. The Kubernetes CSI project keeps a list of drivers.

Is Ceph S3 compatible?

Yes, for the core of the API. The Ceph Object Gateway, RGW, supports a RESTful API compatible with the basic data access model of the Amazon S3 API. Ceph's docs include a table of the S3 features it supports (S3 API). On Kubernetes, Rook runs RGW, and apps request buckets with ObjectBucketClaims.

How many nodes does Ceph need?

Plan for at least three in production. Ceph's docs say three monitors and three OSDs are normally required for redundancy and high availability (Intro to Ceph). Rook's production example needs three worker nodes, and with three replicas on different hosts one node can fail without data loss (Rook examples). Rook's single-node example is for tests only.

Related Reading

• Kubernetes Monitoring: Tools, Metrics and Alerting
• Kubernetes Troubleshooting: Guide and Cheat Sheet
• Kubernetes CSI: Leveraging Standards
• Kubernetes on Bare Metal: On-Prem Setup and Day 2
• Kubernetes Performance Tuning & Optimization (2026)