Introduction to Kubernetes for HPC

If you have experience submitting Slurm batch scripts (sbatch, srun, squeue) on cluster systems, this guide provides a direct conceptual mapping to Kubernetes (kubectl, Pods, Jobs).

Conceptual Mapping: Slurm vs. Kubernetes

Slurm Concept	Kubernetes Equivalent	Purpose
sbatch job.sh	kubectl apply -f job.yaml	Submits a batch computation to the scheduler
squeue -u $USER	kubectl get pods -n <namespace>	Checks the status of active and queued tasks
scancel <jobid>	kubectl delete job <job-name>	Cancels and terminates running execution
#SBATCH --gpus=2	resources.limits: {"nvidia.com/gpu": 2}	Requests hardware GPU accelerators
#SBATCH --mem=64G	resources.limits: {memory: "64Gi"}	Enforces RAM limits
Cluster Scratch (/global/scratch)	PersistentVolumeClaim (ceph-rbd / cephfs)	Persistent block (ceph-rbd) or shared POSIX (cephfs) storage
Node Scratch (/tmp / /scratch)	emptyDir (Local NVMe)	Ultra-fast ephemeral node scratch (lost on pod restart)

Key Policy: Jobs Must Exit by Themselves (No sleep infinity)

Unlike holding an interactive Slurm node allocation idle, running sleep infinity, tail -f /dev/null, or any equivalent command in Kubernetes Jobs is strictly prohibited. Every batch job must execute its computation, write outputs to a persistent PVC (ceph-rbd or cephfs), and exit automatically (exit 0) upon completion. For interactive terminal sessions, use a bare Pod (6-hour limit) or Coder Development Environments.

---

Why Kubernetes for Science?

1. Complete Reproducibility: No missing shared libraries or conflicting Python virtualenvs. Everything is packaged in standard OCI containers.
2. Instant Microservices: Host interactive Dash/Streamlit visualization portals, FastAPI model servers, and notebooks with automatic URL routing.
3. Resilience & Auto-Healing: If a worker node suffers a hardware fault, Kubernetes automatically restarts your pod on a healthy node.
4. Dynamic Elastic Scaling: Auto-scale Ray or Dask workers based on real-time task queue depth.