Cluster Policies & Fair-Share

Persistent Data vs Pod Lifecycle

Ephemeral files in a pod container are lost upon termination. Always mount a Ceph RBD (ceph-rbd) or CephFS (cephfs) PersistentVolumeClaim (PVC) under /workspace or /data to preserve your code, datasets, and model checkpoints.

To ensure fair allocation and cluster reliability across hundreds of Berkeley Lab scientists, Berkelium enforces the following policies:

1. Interactive pods

Pods without controllers (e.g., kind: Pod without a Deployment, StatefulSet, or DaemonSet) are considered interactive and are automatically terminated after 6 hours. Use Deployments for persistent services and Jobs for batch processing.

---

2. Batch Jobs — No sleep infinity (Strictly Prohibited)

Strictly Prohibited: sleep infinity in Batch Jobs or GPU Workloads

Batch jobs (kind: Job) must terminate automatically without human intervention as soon as their computation completes.

Using sleep infinity, sleep 999999, tail -f /dev/null, while true; do sleep ...; done, or any equivalent command in a Job or to hold onto GPU/large compute allocations is strictly prohibited:

• Never use sleep infinity in a Job: Batch Jobs must run an unattended task and exit by themselves (exit 0). Do not submit a Job running sleep infinity for manual kubectl exec sessions, and do not append ; sleep infinity after a training command.
• Never hold a GPU or large resource reservation idle: Using sleep infinity in any controller (Job, Deployment, or StatefulSet) to keep a GPU allocated or reserve large CPU/memory requests is strictly forbidden.
• Lightweight CPU-only Deployment exception: sleep infinity is permitted only in lightweight CPU-only Deployments that do not reserve large resources (for example, a utility or data-management pod with a small CPU/memory request and a larger burst limit, and zero GPUs).
• Store outputs on persistent storage: Mount a ceph-rbd or cephfs PVC to save checkpoints, logs, and results so batch jobs can cleanly exit the moment computation finishes.

---

3. General GPU usage

Users should terminate their workloads, especially using GPUs, once those are not needed.

---

4. Container Resource Requests & Limits (Default LimitRange)

Always Specify Resource Requests and Limits

Always explicitly specify both resources.requests and resources.limits for every container in your workload.

If resource requests or limits are omitted, Kubernetes automatically applies the namespace default LimitRange:

• Default CPU Limit: 1 (1 core)
• Default CPU Request: 100m (0.1 core)
• Default Memory Limit & Request: 1Gi
• Minimum Allowable: 1m CPU, 5Mi Memory
• Maximum Pod Memory: 1Ti

Why this matters:

• Severe CPU Throttling: If your workload requires multi-core compute but lacks explicit CPU limits, Kubernetes caps the container at 1 CPU core, degrading performance.
• Out-Of-Memory (OOM) Kills: If your job consumes more than 1Gi RAM without an explicit memory limit, the Linux kernel OOM-killer terminates the container immediately (OOMKilled / Exit Code 137).
• Guaranteed Scheduling: Declaring requests ensures the Kubernetes scheduler reserves guaranteed node resources for your workload instead of placing it on an oversubscribed node.

Recommended Resource Specification

resources:
  requests:
    cpu: "4"           # Minimum guaranteed CPU cores
    memory: "16Gi"      # Minimum guaranteed RAM
  limits:
    cpu: "8"           # Maximum CPU burst ceiling
    memory: "32Gi"      # Memory limit before OOM kill
    nvidia.com/gpu: "1" # (Optional) Required when using GPUs