Persistent Volumes & Storage Claims

By default, container filesystems are ephemeral. To persist code, environments, datasets, and trained model weights across pod restarts, create a PersistentVolumeClaim (PVC) using one of Berkelium’s two Ceph StorageClasses:

• ceph-rbd (ReadWriteOnce — Recommended for Single-Pod Workloads): High-IOPS Ceph RADOS Block Device attached to a single pod at a time. Best for conda/pip/uv environments, Git repositories, code compilation, databases, and workloads with many small files.
• cephfs (ReadWriteMany — Shared Multi-Pod Filesystem): Distributed POSIX filesystem that can be mounted by multiple pods across different compute nodes simultaneously. Best for shared datasets and large checkpoint files across distributed jobs.

For temporary, ultra-fast I/O operations (such as data preprocessing or unpacking dataset shards during training), you can also mount node-local NVMe storage via emptyDir.

See CephFS & RBD Storage for a detailed comparison and best practices.

---

1. Creating a Ceph RBD Block PVC (ceph-rbd, Single Pod)

If your volume only needs to be mounted by one pod at a time—or if you plan to install conda/pip packages, compile code, or work with many small files—use storageClassName: ceph-rbd with ReadWriteOnce.

Create rbd-claim.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-workspace-pvc
  namespace: <your-namespace>
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ceph-rbd
  resources:
    requests:
      storage: 50Gi

Apply the claim and verify that its status is Bound:

kubectl apply -f rbd-claim.yaml
kubectl get pvc my-workspace-pvc -n <your-namespace>

---

2. Creating a Shared CephFS PVC (cephfs, Multiple Pods)

CephFS provides a POSIX-compliant distributed filesystem that supports ReadWriteMany (RWX), allowing multiple pods across different compute nodes to read and write to the same volume simultaneously.

Create cephfs-claim.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: lab-shared-data-pvc
  namespace: <your-namespace>
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: cephfs
  resources:
    requests:
      storage: 500Gi

Apply the claim:

kubectl apply -f cephfs-claim.yaml
kubectl get pvc -n <your-namespace>

Volume Limits & Small-File Warning

• Volume Capacity & Expansion: Choose your volume limit via resources.requests.storage. Kubernetes enforces the requested volume size on both ceph-rbd and cephfs, and you can expand either volume online later by increasing spec.resources.requests.storage.
• Do not install conda/pip environments or compile code on cephfs: Use a ceph-rbd volume instead so metadata operations do not overload the shared CephFS Metadata Servers.

---

3. Mounting a PVC into a Pod

Once your ceph-rbd or cephfs PVC is Bound, reference its claimName under spec.volumes and mount it into your container with volumeMounts:

apiVersion: v1
kind: Pod
metadata:
  name: analysis-worker
  namespace: <your-namespace>
spec:
  containers:
    - name: app
      image: python:3.11-slim
      command: ["bash", "-c", "echo 'Experiment data logged' >> /mnt/shared/results.log && sleep 3600"]
      resources:
        requests:
          cpu: "2"
          memory: "4Gi"
        limits:
          cpu: "4"
          memory: "8Gi"
      volumeMounts:
        - name: workspace-mount
          mountPath: /workspace
        - name: dataset-mount
          mountPath: /mnt/shared
  volumes:
    - name: workspace-mount
      persistentVolumeClaim:
        claimName: my-workspace-pvc
    - name: dataset-mount
      persistentVolumeClaim:
        claimName: lab-shared-data-pvc

---

4. Ephemeral Fast Local NVMe Scratch (emptyDir)

Local NVMe storage on Berkelium is not persistent and does not use a StorageClass or PVC. It is exposed to pods via Kubernetes emptyDir volumes.

Scratch is Ephemeral — Deleted on Pod Restart

Data stored in emptyDir resides on the host node’s local NVMe disk and is tied directly to the pod’s lifecycle. When the pod stops, restarts, or crashes, all data on NVMe scratch is lost. Always save persistent results and checkpoints to a ceph-rbd or cephfs PVC.

apiVersion: v1
kind: Pod
metadata:
  name: scratch-example
  namespace: <your-namespace>
spec:
  containers:
    - name: worker
      image: ubuntu:22.04
      command: ["bash", "-c", "echo 'Fast scratch' > /scratch/temp.bin && sleep 3600"]
      resources:
        requests:
          cpu: "2"
          memory: "4Gi"
        limits:
          cpu: "4"
          memory: "8Gi"
      volumeMounts:
        - name: local-scratch
          mountPath: /scratch
  volumes:
    - name: local-scratch
      emptyDir:
        sizeLimit: 100Gi