1# Pod Failures & Application Issues
2 
3## Common Pod Diagnostic Commands
4 
5```bash
6# List unhealthy pods across all namespaces
7kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded
8# All pods wide view
9kubectl get pods -A -o wide
10# Detailed pod status - events section is critical
11kubectl describe pod <pod-name> -n <namespace>
12# Pod logs (current and previous crash)
13kubectl logs <pod-name> -n <namespace>
14kubectl logs <pod-name> -n <namespace> --previous
15```
16 
17---
18 
19## CrashLoopBackOff
20 
21Pod starts, crashes, restarts with exponential backoff (10s, 20s, 40s... up to 5m).
22 
23**Diagnostics:**
24 
25```bash
26kubectl describe pod <pod-name> -n <namespace>
27# Check: Exit Code, Reason, Last State, Events
28 
29kubectl logs <pod-name> -n <namespace> --previous
30# Shows stdout/stderr from the last crashed container
31```
32 
33**Decision tree:**
34 
35| Exit Code | Meaning                                               | Fix Path                                                      |
36| --------- | ----------------------------------------------------- | ------------------------------------------------------------- |
37| `0`       | App exited successfully (unexpected for long-running) | Check if entrypoint/command is correct; app may be a one-shot |
38| `1`       | Application error                                     | Read logs - unhandled exception, missing config, bad startup  |
39| `137`     | OOMKilled (SIGKILL)                                   | Increase `resources.limits.memory`; check for memory leaks    |
40| `139`     | Segfault (SIGSEGV)                                    | Binary compatibility issue or native code bug                 |
41| `143`     | SIGTERM - graceful shutdown                           | Pod was terminated; check if liveness probe killed it         |
42 
43**OOMKilled specifically:**
44 
45```bash
46kubectl describe pod <pod-name> -n <namespace> | grep -A2 "Last State"
47# Reason: OOMKilled -> container exceeded memory limit
48```
49 
50Fix: increase `resources.limits.memory` or optimize application memory usage. Check `kubectl top pod <pod-name> -n <namespace>` for actual usage.
51 
52**OOM kill tracing with Inspektor Gadget:** Use `trace_oomkill` (timeout 30) with `--k8s-namespace <namespace> --k8s-podname <pod-name>` to see which process was killed and memory at kill time. See [references/inspektor-gadget.md](references/inspektor-gadget.md).
53 
54**Deep diagnostics with Inspektor Gadget** (when logs and describe are inconclusive):
55 
56Use the [IG base command pattern](references/inspektor-gadget.md) with `--k8s-namespace <namespace> --k8s-podname <pod-name>` and these gadgets:
57 
58- `trace_exec` (timeout 30) — see what the container executes at startup
59- `trace_open` (timeout 30) — find missing configs/secrets (retval -2 = ENOENT, -13 = EACCES)
60- `snapshot_process` (timeout 5) — list running processes in the pod
61 
62See [references/inspektor-gadget.md](references/inspektor-gadget.md).
63 
64---
65 
66## ImagePullBackOff
67 
68Pod can't pull the container image.
69 
70**Diagnostics:**
71 
72```bash
73kubectl describe pod <pod-name> -n <namespace>
74# Events section shows the exact pull error
75```
76 
77| Error Message                           | Cause                        | Fix                                                            |
78| --------------------------------------- | ---------------------------- | -------------------------------------------------------------- |
79| `ErrImagePull` / `ImagePullBackOff`     | Image name or tag is wrong   | Verify image name and tag exist in the registry                |
80| `unauthorized: authentication required` | Missing or wrong pull secret | Create/update `imagePullSecrets` on the pod or service account |
81| `manifest unknown`                      | Tag doesn't exist            | Check available tags in the registry                           |
82| `context deadline exceeded`             | Registry unreachable         | Check network/firewall; for ACR, verify AKS -> ACR integration |
83 
84**ACR integration check:**
85 
86```bash
87# Verify AKS is attached to ACR
88az aks check-acr -g <rg> -n <cluster> --acr <acr-name>.azurecr.io
89```
90 
91---
92 
93## Pending Pods
94 
95Pod stays in `Pending` - scheduler can't place it.
96 
97**Diagnostics:**
98 
99```bash
100kubectl describe pod <pod-name> -n <namespace>
101# Events section shows why scheduling failed
102```
103 
104| Event Message                                                          | Cause                               | Fix                                                             |
105| ---------------------------------------------------------------------- | ----------------------------------- | --------------------------------------------------------------- |
106| `Insufficient cpu` / `Insufficient memory`                             | No node has enough resources        | Scale node pool; reduce resource requests; check for overcommit |
107| `node(s) had taint ... that the pod didn't tolerate`                   | Taint/toleration mismatch           | Add matching toleration or use a different node pool            |
108| `node(s) didn't match Pod's node affinity/selector`                    | Affinity rule unsatisfiable         | Check `nodeSelector` or `nodeAffinity` rules                    |
109| `persistentvolumeclaim ... not found` / `unbound`                      | PVC not ready                       | Check PVC status; verify storage class exists                   |
110| `0/N nodes are available: N node(s) had volume node affinity conflict` | Zonal disk vs pod in different zone | Use ZRS storage class or ensure same zone                       |
111 
112---
113 
114## Readiness & Liveness Probe Failures
115 
116**Readiness probe failure** -> pod removed from Service endpoints (no traffic). **Liveness probe failure** -> pod killed and restarted.
117 
118**Diagnostics:**
119 
120```bash
121kubectl describe pod <pod-name> -n <namespace>
122# Look for: "Readiness probe failed" or "Liveness probe failed" in Events
123 
124# Check the pod's READY column - must show n/n
125kubectl get pod <pod-name> -n <namespace>
126```
127 
128| Symptom                              | Cause                   | Fix                                                        |
129| ------------------------------------ | ----------------------- | ---------------------------------------------------------- |
130| READY shows `0/1` but pod is Running | Readiness probe failing | Check probe path, port, and app health endpoint            |
131| Pod restarts repeatedly              | Liveness probe failing  | Increase `initialDelaySeconds`; check if app starts slowly |
132| Probe timeout errors                 | App responds too slowly | Increase `timeoutSeconds`; check app performance           |
133 
134> 💡 **Tip:** Set `initialDelaySeconds` on liveness probes to be longer than your app's startup time. A common mistake is killing pods before they finish initializing.
135 
136---
137 
138## Resource Constraints (CPU/Memory)
139 
140**Check actual usage vs limits:**
141 
142```bash
143kubectl top pod <pod-name> -n <namespace>
144kubectl top pod -n <namespace> --sort-by=memory
145 
146# Compare with requests/limits
147kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[*].resources}'
148```
149 
150| Symptom                          | Cause                                   | Fix                                                 |
151| -------------------------------- | --------------------------------------- | --------------------------------------------------- |
152| OOMKilled (exit code 137)        | Container exceeded memory limit         | Increase `limits.memory` or fix memory leak         |
153| CPU throttling (slow responses)  | Container hitting CPU limit             | Increase `limits.cpu` or remove CPU limits          |
154| Pending - insufficient resources | Requests exceed available node capacity | Lower requests, scale nodes, or use larger VM sizes |
155 
156> ⚠️ **Warning:** Setting CPU limits can cause unnecessary throttling even when the node has spare capacity. Many teams set CPU requests but not limits. Memory limits should always be set.
157