Cluster Policies & Fair-Share
To ensure fair allocation and cluster reliability across hundreds of Berkeley Lab scientists, Berkelium enforces the following policies:
1. Idle Pod Policy (Reaper Bot)
Section titled “1. Idle Pod Policy (Reaper Bot)”Pods that consume GPU or high-memory resources while remaining completely idle (0% compute utilization for > 4 hours) are automatically detected by the Berkelium cluster monitor:
- A warning notification is sent to the user via Slack / Email.
- If no active process is initiated within 60 minutes after the warning, the pod is terminated to free expensive GPU hardware for queued jobs.
2. Pod Lifetime Limits
Section titled “2. Pod Lifetime Limits”- Batch Jobs (
kind: Job): Allowed to run up to a maximum runtime of 7 days (168 hours). For longer calculations, design checkpointing intervals. - Interactive Pods: Maximum continuous session duration of 24 hours.
3. Container Security Standards
Section titled “3. Container Security Standards”Berkelium enforces Kubernetes Restricted Pod Security Standards:
- Containers must not run as root (
runAsNonRoot: true). - Privilege escalation is disabled (
allowPrivilegeEscalation: false). - Host path mounts (
hostPath) are restricted to cluster administrators. Researchers must use CephFS or local NVMe scratch storage classes.
4. Maintenance Windows
Section titled “4. Maintenance Windows”Regular kernel updates, NVIDIA driver upgrades, and Kubernetes control plane patches occur during the monthly scheduled maintenance window:
- First Tuesday of every month, 8:00 AM – 12:00 PM PT.
- Announcements are broadcast one week in advance in
#berkelium-userson Slack.