1# Kubernetes Troubleshooting
2 
3## Essential kubectl Commands
4 
5### Pod Inspection
6 
7```bash
8# Get pods with details
9kubectl get pods -n production -o wide
10kubectl get pods --all-namespaces
11kubectl get pods --field-selector status.phase=Running
12kubectl get pods --selector app=web-app
13 
14# Describe pod (shows events)
15kubectl describe pod web-app-7d5c8b9f4-xk2pm -n production
16 
17# Get pod logs
18kubectl logs web-app-7d5c8b9f4-xk2pm -n production
19kubectl logs web-app-7d5c8b9f4-xk2pm -n production --previous  # Previous container
20kubectl logs web-app-7d5c8b9f4-xk2pm -n production -c init-container
21kubectl logs -f web-app-7d5c8b9f4-xk2pm -n production  # Follow logs
22kubectl logs --tail=100 web-app-7d5c8b9f4-xk2pm -n production
23kubectl logs --since=1h web-app-7d5c8b9f4-xk2pm -n production
24 
25# Get all pod logs from deployment
26kubectl logs deployment/web-app -n production --all-containers=true
27 
28# Execute commands in pod
29kubectl exec -it web-app-7d5c8b9f4-xk2pm -n production -- /bin/sh
30kubectl exec web-app-7d5c8b9f4-xk2pm -n production -- env
31kubectl exec web-app-7d5c8b9f4-xk2pm -n production -- cat /etc/config/app.yaml
32 
33# Copy files to/from pod
34kubectl cp web-app-7d5c8b9f4-xk2pm:/app/logs/app.log ./app.log -n production
35kubectl cp ./config.yaml web-app-7d5c8b9f4-xk2pm:/tmp/config.yaml -n production
36 
37# Port forward
38kubectl port-forward web-app-7d5c8b9f4-xk2pm 8080:8080 -n production
39kubectl port-forward service/web-app 8080:80 -n production
40```
41 
42### Deployment Debugging
43 
44```bash
45# Check deployment status
46kubectl get deployment web-app -n production
47kubectl describe deployment web-app -n production
48kubectl rollout status deployment/web-app -n production
49kubectl rollout history deployment/web-app -n production
50 
51# Check replica sets
52kubectl get rs -n production
53kubectl describe rs web-app-7d5c8b9f4 -n production
54 
55# Scale deployment
56kubectl scale deployment web-app --replicas=5 -n production
57 
58# Rollback deployment
59kubectl rollout undo deployment/web-app -n production
60kubectl rollout undo deployment/web-app --to-revision=2 -n production
61 
62# Restart deployment (recreate pods)
63kubectl rollout restart deployment/web-app -n production
64```
65 
66### Service and Network Debugging
67 
68```bash
69# Get services
70kubectl get svc -n production
71kubectl describe svc web-app -n production
72 
73# Get endpoints
74kubectl get endpoints web-app -n production
75kubectl describe endpoints web-app -n production
76 
77# Get ingress
78kubectl get ingress -n production
79kubectl describe ingress web-app -n production
80 
81# Get network policies
82kubectl get networkpolicy -n production
83kubectl describe networkpolicy frontend-to-backend -n production
84```
85 
86### Resource and Configuration
87 
88```bash
89# Get ConfigMaps and Secrets
90kubectl get configmap -n production
91kubectl describe configmap app-config -n production
92kubectl get configmap app-config -n production -o yaml
93 
94kubectl get secret -n production
95kubectl describe secret app-secrets -n production
96kubectl get secret app-secrets -n production -o jsonpath='{.data.password}' | base64 -d
97 
98# Get PVCs and PVs
99kubectl get pvc -n production
100kubectl describe pvc database-pvc -n production
101kubectl get pv
102 
103# Get events (sorted by timestamp)
104kubectl get events -n production --sort-by='.lastTimestamp'
105kubectl get events -n production --field-selector involvedObject.name=web-app-7d5c8b9f4-xk2pm
106```
107 
108## Debug Pod
109 
110### Ephemeral Debug Container
111 
112```bash
113# Attach debug container to running pod
114kubectl debug -it web-app-7d5c8b9f4-xk2pm -n production \
115  --image=busybox:latest \
116  --target=web-app
117 
118# Create copy of pod with debug tools
119kubectl debug web-app-7d5c8b9f4-xk2pm -n production \
120  -it \
121  --image=ubuntu:latest \
122  --share-processes \
123  --copy-to=web-app-debug
124 
125# Debug with different image
126kubectl debug web-app-7d5c8b9f4-xk2pm -n production \
127  -it \
128  --image=nicolaka/netshoot:latest \
129  --target=web-app
130```
131 
132### Debug on Node
133 
134```bash
135# Create privileged pod on specific node
136kubectl debug node/node-01 -it --image=ubuntu:latest
137 
138# Access node filesystem
139kubectl debug node/node-01 -it --image=ubuntu:latest -- chroot /host
140```
141 
142## Common Issues and Solutions
143 
144### Issue 1: Pod in Pending State
145 
146```bash
147# Check pod status and events
148kubectl describe pod web-app-7d5c8b9f4-xk2pm -n production
149 
150# Common causes:
151# 1. Insufficient resources
152kubectl top nodes
153kubectl describe nodes
154 
155# 2. PVC not bound
156kubectl get pvc -n production
157kubectl describe pvc database-pvc -n production
158 
159# 3. ImagePullBackOff
160kubectl describe pod web-app-7d5c8b9f4-xk2pm -n production | grep -A 10 Events
161 
162# 4. Node selector/affinity issues
163kubectl get pod web-app-7d5c8b9f4-xk2pm -n production -o yaml | grep -A 5 nodeSelector
164```
165 
166### Issue 2: CrashLoopBackOff
167 
168```bash
169# Check logs from crashed container
170kubectl logs web-app-7d5c8b9f4-xk2pm -n production --previous
171 
172# Check if liveness probe is failing
173kubectl describe pod web-app-7d5c8b9f4-xk2pm -n production | grep -A 10 "Liveness"
174 
175# Debug with different command
176kubectl run debug-pod --image=myapp:latest -it --rm --restart=Never -- /bin/sh
177 
178# Check resource limits
179kubectl describe pod web-app-7d5c8b9f4-xk2pm -n production | grep -A 10 "Limits"
180```
181 
182### Issue 3: ImagePullBackOff
183 
184```bash
185# Check image pull secret
186kubectl get secret registry-credentials -n production -o yaml
187 
188# Test image pull manually
189kubectl run test-pull --image=myregistry.io/myapp:v1.2.0 \
190  --image-pull-policy=Always \
191  --restart=Never \
192  -n production
193 
194# Create/update image pull secret
195kubectl create secret docker-registry registry-credentials \
196  --docker-server=myregistry.io \
197  --docker-username=myuser \
198  --docker-password=mypassword \
199  [email protected] \
200  -n production
201```
202 
203### Issue 4: Service Not Accessible
204 
205```bash
206# Check service endpoints
207kubectl get endpoints web-app -n production
208kubectl describe endpoints web-app -n production
209 
210# Verify pod labels match service selector
211kubectl get pod web-app-7d5c8b9f4-xk2pm -n production --show-labels
212kubectl get service web-app -n production -o yaml | grep -A 3 selector
213 
214# Test service connectivity from debug pod
215kubectl run debug --image=nicolaka/netshoot:latest -it --rm -n production -- bash
216# Inside pod:
217curl http://web-app.production.svc.cluster.local
218nslookup web-app.production.svc.cluster.local
219telnet web-app.production.svc.cluster.local 80
220```
221 
222### Issue 5: DNS Resolution Issues
223 
224```bash
225# Check CoreDNS pods
226kubectl get pods -n kube-system -l k8s-app=kube-dns
227kubectl logs -n kube-system -l k8s-app=kube-dns
228 
229# Test DNS resolution
230kubectl run dnsutils --image=tutum/dnsutils -it --rm -- bash
231# Inside pod:
232nslookup kubernetes.default
233nslookup web-app.production.svc.cluster.local
234dig web-app.production.svc.cluster.local
235 
236# Check DNS config in pod
237kubectl exec web-app-7d5c8b9f4-xk2pm -n production -- cat /etc/resolv.conf
238```
239 
240### Issue 6: NetworkPolicy Blocking Traffic
241 
242```bash
243# List network policies
244kubectl get networkpolicy -n production
245kubectl describe networkpolicy default-deny-all -n production
246 
247# Test connectivity
248kubectl run test-connectivity --image=nicolaka/netshoot:latest -it --rm -n production -- bash
249# Inside pod:
250curl -v http://web-app:80
251nc -zv web-app 80
252 
253# Temporarily allow all traffic (testing only)
254kubectl delete networkpolicy --all -n production
255```
256 
257### Issue 7: High Resource Usage
258 
259```bash
260# Check resource usage
261kubectl top nodes
262kubectl top pods -n production
263kubectl top pod web-app-7d5c8b9f4-xk2pm -n production --containers
264 
265# Check resource requests and limits
266kubectl describe pod web-app-7d5c8b9f4-xk2pm -n production | grep -A 10 "Limits"
267 
268# Get pods sorted by CPU/memory usage
269kubectl top pods -n production --sort-by=cpu
270kubectl top pods -n production --sort-by=memory
271 
272# Check node capacity
273kubectl describe node node-01 | grep -A 10 "Allocated resources"
274```
275 
276### Issue 8: PersistentVolumeClaim Issues
277 
278```bash
279# Check PVC status
280kubectl get pvc -n production
281kubectl describe pvc database-pvc -n production
282 
283# Check PV status
284kubectl get pv
285kubectl describe pv pvc-abc123
286 
287# Check storage class
288kubectl get storageclass
289kubectl describe storageclass fast-ssd
290 
291# Events related to PVC
292kubectl get events -n production --field-selector involvedObject.name=database-pvc
293```
294 
295## Advanced Debugging
296 
297### API Server Debugging
298 
299```bash
300# Enable verbose output
301kubectl get pods -n production -v=9
302 
303# Check API server logs (on master node)
304journalctl -u kube-apiserver -f
305 
306# Check cluster info
307kubectl cluster-info
308kubectl cluster-info dump > cluster-dump.txt
309```
310 
311### RBAC Debugging
312 
313```bash
314# Check if ServiceAccount can perform action
315kubectl auth can-i get pods --as=system:serviceaccount:production:web-app-sa -n production
316 
317# List permissions for ServiceAccount
318kubectl describe sa web-app-sa -n production
319kubectl describe role web-app-role -n production
320kubectl describe rolebinding web-app-rolebinding -n production
321 
322# Check all permissions
323kubectl auth can-i --list --as=system:serviceaccount:production:web-app-sa -n production
324```
325 
326### Performance Debugging
327 
328```bash
329# Get resource metrics
330kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes
331kubectl get --raw /apis/metrics.k8s.io/v1beta1/pods
332 
333# Check pod overhead
334kubectl get pod web-app-7d5c8b9f4-xk2pm -n production -o json | jq '.spec.overhead'
335 
336# Check priority classes
337kubectl get priorityclasses
338kubectl describe priorityclass high-priority
339```
340 
341## Diagnostic Tools
342 
343### Network Tools Container
344 
345```yaml
346apiVersion: v1
347kind: Pod
348metadata:
349  name: netshoot
350  namespace: production
351spec:
352  containers:
353  - name: netshoot
354    image: nicolaka/netshoot:latest
355    command: ["/bin/sleep", "3600"]
356  restartPolicy: Never
357```
358 
359### Database Client Container
360 
361```yaml
362apiVersion: v1
363kind: Pod
364metadata:
365  name: postgres-client
366  namespace: production
367spec:
368  containers:
369  - name: postgres
370    image: postgres:15-alpine
371    command: ["/bin/sleep", "3600"]
372    env:
373    - name: PGHOST
374      value: postgres-service
375    - name: PGUSER
376      value: myapp
377    - name: PGPASSWORD
378      valueFrom:
379        secretKeyRef:
380          name: postgres-secrets
381          key: password
382  restartPolicy: Never
383```
384 
385## Quick Reference
386 
387### Pod States
388- **Pending**: Waiting to be scheduled
389- **ContainerCreating**: Pulling image / creating container
390- **Running**: Pod is running
391- **Succeeded**: All containers exited successfully
392- **Failed**: At least one container failed
393- **CrashLoopBackOff**: Container keeps crashing
394- **ImagePullBackOff**: Cannot pull image
395- **ErrImagePull**: Image pull error
396- **Unknown**: Cannot get pod status
397 
398### Common Exit Codes
399- **0**: Success
400- **1**: General error
401- **137**: SIGKILL (OOMKilled - out of memory)
402- **139**: SIGSEGV (segmentation fault)
403- **143**: SIGTERM (graceful termination)
404 
405## Best Practices
406 
4071. **Logs**: Always check logs first with `kubectl logs`
4082. **Events**: Use `kubectl describe` to see events
4093. **Labels**: Use consistent labels for easier debugging
4104. **Resources**: Set appropriate requests and limits
4115. **Health Checks**: Implement proper liveness and readiness probes
4126. **Monitoring**: Set up comprehensive monitoring and alerting
4137. **Debug Tools**: Keep debug containers ready
4148. **Documentation**: Document common issues and solutions
415