Debugging Pods & Diagnosing Failures

When a pod fails or remains in a non-running state, follow this step-by-step diagnostic workflow:

Step 1: Inspect Pod Status and Events

The primary command for finding why Kubernetes rejected or failed a pod:

kubectl describe pod <pod-name> -n sci-myproject

Look at the Events section at the bottom:

• FailedScheduling: Namespace quota exceeded or no nodes have the requested GPU/RAM available.
• ImagePullBackOff: Docker registry authentication required or image tag misspelled.
• ErrImagePull: Harbor registry inaccessible.

---

Step 2: Read Container Logs

# View standard output and error streams
kubectl logs <pod-name> -n sci-myproject

# If the container crashed, inspect logs from the previous instance
kubectl logs <pod-name> --previous -n sci-myproject

# Stream live logs
kubectl logs -f <pod-name> -n sci-myproject

---

Step 3: Common Exit Codes

Exit Reason / Code	Meaning	Remediation
OOMKilled (Exit Code 137)	Container exceeded memory limits	Increase resources.limits.memory in your pod manifest
CrashLoopBackOff (Exit Code 1)	Python exception or fatal binary exit	Run kubectl logs to inspect traceback
Exit Code 127	Command or entrypoint binary not found	Verify $PATH or command syntax in your Dockerfile
CUDA Out of Memory	PyTorch allocated more VRAM than GPU has	Reduce batch size or switch from A100 (80GB) to H100 (80GB) / multi-GPU

---

Step 4: Launching an Ephemeral Debug Shell

To inspect the filesystem of a failing pod without altering its configuration:

kubectl debug -it <pod-name> --image=busybox:latest -n sci-myproject -- target=worker