Skip to content
Berkelium Docs

High-IOPS Local NVMe Scratch

For scientific workloads with intense random I/O demands (such as deep learning dataset shuffling, uncompressed HDF5 slicing, and temporary matrix caches), Berkelium worker nodes provide high-performance local NVMe storage.

Local NVMe storage on Berkelium is ephemeral and non-persistent. It is provisioned using Kubernetes emptyDir volumes and is tied directly to the lifecycle of your pod. When the pod stops, finishes, or restarts, all data residing on local NVMe scratch is permanently lost.

Using Local NVMe via emptyDir

Section titled “Using Local NVMe via emptyDir”

To utilize fast node-local scratch storage in your pod, define an emptyDir volume in your pod specification and mount it into your container:

apiVersion: v1
kind: Pod
metadata:
  name: fast-io-training
  namespace: <your-namespace>
spec:
  restartPolicy: OnFailure
  containers:
    - name: trainer
      image: pytorch/pytorch:2.3.0-cuda12.1-cudnn8-runtime
      command: ["python", "train.py", "--cache-dir=/scratch"]
      resources:
        limits:
          cpu: "16"
          memory: "64Gi"
          nvidia.com/gpu: "1"
      volumeMounts:
        - name: scratch-storage
          mountPath: /scratch
  volumes:
    - name: scratch-storage
      emptyDir: {}

NVMe Scratch is Ephemeral — Data Lost on Pod Restart

Local NVMe storage is not persistent:

  • It is created empty on the physical node when the pod starts.
  • If the pod crashes, restarts, or is rescheduled onto another node, the data in emptyDir is erased.
  • Never store persistent experimental results, checkpoints, or raw datasets exclusively on local NVMe scratch.
  • Always copy final model checkpoints and output artifacts to a persistent CephFS volume (storageClassName: cephfs) before your container completes.