Batch Jobs & Indexed Workflows

Kubernetes Job resources run a task to completion, automatically handling node failures and retries.

Standard Batch Job Manifest

Create batch-simulation.yaml:

apiVersion: batch/v1
kind: Job
metadata:
  name: lbnl-diffraction-sweep
  namespace: sci-myproject
spec:
  backoffLimit: 3  # Retry up to 3 times if task exits with non-zero status
  activeDeadlineSeconds: 86400  # 24 hours max runtime
  template:
    spec:
      restartPolicy: OnFailure
      containers:
        - name: solver
          image: registry.berkelium.lbl.gov/sci-als/synchrotron-model:v1
          command: ["python", "run_sweep.py", "--angles", "0-180", "--out", "/data/results"]
          volumeMounts:
            - name: storage-volume
              mountPath: /data
          resources:
            limits:
              cpu: "16"
              memory: "64Gi"
              nvidia.com/gpu: "1"
      volumes:
        - name: storage-volume
          persistentVolumeClaim:
            claimName: my-project-cephfs-pvc

Submit and check execution:

kubectl apply -f batch-simulation.yaml
kubectl get jobs -n sci-myproject

---

Parallel Indexed Jobs (Slurm Array Equivalent)

To run 10 parallel sub-tasks where each worker receives its distinct index (JOB_COMPLETION_INDEX from 0 to 9):

apiVersion: batch/v1
kind: Job
metadata:
  name: parallel-array-simulation
  namespace: sci-myproject
spec:
  completions: 10
  parallelism: 5  # Run 5 concurrent pods at a time
  completionMode: Indexed
  template:
    spec:
      restartPolicy: OnFailure
      containers:
        - name: worker
          image: python:3.11-slim
          command: ["python", "-c", "import os; print(f'Processing shard {os.environ.get(\"JOB_COMPLETION_INDEX\")}')"]