CephFS & RBD Storage

Berkelium provides over 4.5 Petabytes of persistent Ceph storage exposed to Kubernetes workloads through two primary StorageClasses:

• ceph-rbd (Ceph RADOS Block Device — ReadWriteOnce): High-IOPS block storage attached to a single pod at a time.
• cephfs (Ceph Shared Filesystem — ReadWriteMany): Distributed parallel POSIX filesystem that can be mounted by multiple pods across multiple nodes concurrently.

---

Choosing Between ceph-rbd and cephfs

Feature	ceph-rbd (Ceph Block Storage)	cephfs (Ceph Shared Filesystem)
Access Mode	ReadWriteOnce (RWO) — Single pod at a time	ReadWriteMany (RWX) — Multiple pods across nodes
Storage Type	Block device formatted with a local filesystem (ext4 / xfs)	Distributed POSIX network filesystem
Small-File IOPS	High (no distributed metadata server overhead per file)	Low (every file lookup/create queries shared Ceph Metadata Servers)
conda / pip / uv & Code Builds	Recommended	Prohibited / Strongly Discouraged
Large Parallel Streaming	Good (single-node throughput)	Best (high aggregate throughput for large files across many pods)
Volume Size & Expansion	Enforces resources.requests.storage limit; expandable online	Enforces resources.requests.storage limit; expandable online
Best Use Cases	Single-pod jobs, conda/pip environments, Git repos, databases (SQLite/Postgres/DuckDB), Coder /home/coder, KubeVirt VM disks	Multi-node PyTorch/Ray training, shared read/write datasets, large .tar/HDF5/Parquet shards, model checkpoints

When to Use RBD (ceph-rbd)

Ceph RBD (ceph-rbd) is preferred for all workloads and datasets that do not require simultaneous access from multiple pods.

Because RBD provisions a block volume formatted like a standard local disk rather than routing every file operation through shared CephFS Metadata Servers (MDS), it handles metadata-intensive workloads much faster. Use ceph-rbd when:

• Only one pod needs to mount the volume at a time (ReadWriteOnce).
• You are installing conda, pip, or uv virtual environments, cloning Git repositories, or compiling source code.
• Your dataset contains many small files or you are running a database (such as SQLite, PostgreSQL, DuckDB, or a vector database) that requires low-latency random I/O and standard file locking.
• You are provisioning root or data disks for KubeVirt Virtual Machines.

When to Use CephFS (cephfs)

CephFS (cephfs) is designed for distributed workloads that share data across multiple pods in parallel.

Use cephfs when:

• Multiple pods across different compute nodes must read from or write to the same volume at the same time (ReadWriteMany), such as Distributed PyTorch, Ray Clusters, or parallel Batch Jobs.
• Your data consists of fewer, larger files (such as WebDataset .tar archives, HDF5, Parquet, Zarr, or multi-gigabyte model checkpoints) rather than hundreds of thousands of tiny files.

Do Not Install Conda/Pip or Compile Code on CephFS

Installing conda or pip environments or compiling large codebases directly on a cephfs volume creates hundreds of thousands of small files and metadata operations. This overloads the shared CephFS Metadata Servers (MDS), severely slowing down your build and degrading filesystem performance across the cluster.

Always place conda/pip environments and source builds on a ceph-rbd volume, inside your container image, or on ephemeral local NVMe emptyDir scratch.

---

1. Using Ceph RBD Block Storage (ceph-rbd)

Provisioning a ceph-rbd PVC

Create a PersistentVolumeClaim with storageClassName: ceph-rbd and accessModes: [ReadWriteOnce]:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-rbd-workspace
  namespace: <your-namespace>
spec:
  storageClassName: ceph-rbd
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi

Apply and verify that the PVC reaches STATUS: Bound:

kubectl apply -f rbd-pvc.yaml
kubectl get pvc my-rbd-workspace -n <your-namespace>

Mounting a ceph-rbd PVC in a Pod

apiVersion: v1
kind: Pod
metadata:
  name: rbd-worker
  namespace: <your-namespace>
spec:
  containers:
    - name: worker
      image: ubuntu:24.04
      command: ["bash", "-c", "df -h /workspace && sleep 3600"]
      resources:
        requests:
          cpu: "2"
          memory: "8Gi"
        limits:
          cpu: "4"
          memory: "16Gi"
      volumeMounts:
        - name: workspace-vol
          mountPath: /workspace
  volumes:
    - name: workspace-vol
      persistentVolumeClaim:
        claimName: my-rbd-workspace

Expanding a ceph-rbd Volume

ceph-rbd volumes support online volume expansion. If you run out of space on a ceph-rbd PVC, increase spec.resources.requests.storage (volumes can only be increased in size, never shrunk):

kubectl patch pvc my-rbd-workspace -n <your-namespace> \
  -p '{"spec":{"resources":{"requests":{"storage":"100Gi"}}}}'

---

2. Using CephFS Shared Storage (cephfs)

Provisioning a cephfs PVC

Create a PersistentVolumeClaim with storageClassName: cephfs and accessModes: [ReadWriteMany]:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: team-shared-data
  namespace: <your-namespace>
spec:
  storageClassName: cephfs
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 500Gi

Volume Size Limits & Online Expansion

Kubernetes enforces the storage capacity specified in resources.requests.storage for both ceph-rbd and cephfs volumes. Start with the capacity you currently need—if you need more space later, you can increase spec.resources.requests.storage to expand the volume online without downtime (volumes can only be expanded, never shrunk):

kubectl patch pvc team-shared-data -n <your-namespace> \
  -p '{"spec":{"resources":{"requests":{"storage":"1Ti"}}}}'

Best Practices for Shared cephfs Volumes

When mounting a cephfs volume across multiple pods simultaneously, follow these rules to avoid performance bottlenecks and data corruption:

• Never write to the same file from multiple pods concurrently: Opening the same file for writing from multiple parallel workers causes file-locking contention and can corrupt the file. Have each worker write to a unique file (for example, output-rank-${RANK}.pt or results-${JOB_COMPLETION_INDEX}.parquet).
• Aggregate small files into larger archives: Instead of reading millions of individual 4 KB images or JSON files over cephfs, pack training datasets into larger .tar (WebDataset), .h5 (HDF5), .parquet, or .zarr shards, or unpack them onto local NVMe emptyDir scratch at the start of your job.
• Close file handles promptly: Avoid holding long-lived open write handles across idle workers so locks are released immediately.