1# Networking Troubleshooting
2 
3For CNI-specific issues, check CNI pod health and review [AKS networking concepts](https://learn.microsoft.com/azure/aks/concepts-network).
4 
5## Service Unreachable / Connection Refused
6 
7**Diagnostics - always start here:**
8 
9```bash
10# 1. Verify service exists and has endpoints (read-only)
11kubectl get svc <service-name> -n <ns>
12kubectl get endpoints <service-name> -n <ns>
13 
14# 2. Optional connectivity test from inside the namespace
15# This creates a temporary pod. Prefer read-only checks first.
16# Only use it after the user explicitly approves a mutating test.
17kubectl run netdebug --image=curlimages/curl -it --rm -n <ns> -- \
18  curl -sv http://<service>.<ns>.svc.cluster.local:<port>/healthz
19```
20 
21**Decision tree:**
22 
23| Observation                             | Cause                              | Fix                                             |
24| --------------------------------------- | ---------------------------------- | ----------------------------------------------- |
25| Endpoints shows `<none>`                | Label selector mismatch            | Align selector with pod labels; check for typos |
26| Endpoints has IPs but unreachable       | Port mismatch or app not listening | Confirm `targetPort` = actual container port    |
27| Works from some pods, fails from others | Network policy blocking            | See Network Policy section                      |
28| Works inside cluster, fails externally  | Load balancer issue                | See Load Balancer section                       |
29| `ECONNREFUSED` immediately              | App not listening on that port     | Check listening ports in the pod                |
30 
31Pods that are running but not Ready are removed from Endpoints. Check `kubectl get pod <pod> -n <ns>`.
32 
33---
34 
35## DNS Resolution Failures
36 
37**Diagnostics:**
38 
39The live DNS test creates a temporary pod. Prefer `get`, `describe`, `logs`, or `exec` into an existing pod first. Only use it after the user explicitly approves creating the test pod.
40 
41```bash
42# Confirm CoreDNS is running and healthy (read-only)
43kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide
44kubectl top pod -n kube-system -l k8s-app=kube-dns
45 
46# Optional live DNS test from the same namespace as the failing pod
47kubectl run dnstest --image=busybox:1.28 -it --rm -n <ns> -- \
48  nslookup <service-name>.<ns>.svc.cluster.local
49 
50# CoreDNS logs - errors show here first
51kubectl logs -n kube-system -l k8s-app=kube-dns --tail=100
52```
53 
54**DNS failure patterns:**
55 
56| Symptom                               | Cause                                        | Fix                                                                                                                        |
57| ------------------------------------- | -------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
58| `NXDOMAIN` for `svc.cluster.local`    | CoreDNS down or pod network broken           | After confirming the diagnostics above, coordinate with the cluster operator to restart or redeploy CoreDNS and verify CNI |
59| Internal resolves, external NXDOMAIN  | Custom DNS not forwarding to `168.63.129.16` | Fix upstream forwarder                                                                                                     |
60| Intermittent SERVFAIL under load      | CoreDNS CPU throttled                        | Remove CPU limits or add replicas                                                                                          |
61| Private cluster - external names fail | Custom DNS missing privatelink forwarder     | Add conditional forwarder to Azure DNS                                                                                     |
62| `i/o timeout` not `NXDOMAIN`          | Port 53 blocked by NetworkPolicy or NSG      | Allow UDP/TCP 53 from pods to kube-dns ClusterIP                                                                           |
63 
64> ⚠️ **Warning:** The fixes in this table can change cluster state. Use them only after performing the read-only diagnostics above, and only with explicit confirmation from the cluster owner or operator.
65 
66```bash
67kubectl get svc kube-dns -n kube-system -o jsonpath='{.spec.clusterIP}'
68```
69 
70Custom VNet DNS must forward `.cluster.local` to the CoreDNS ClusterIP and other lookups to `168.63.129.16`.
71 
72---
73 
74## Detailed Networking Guides
75 
76- [Load Balancer And Ingress Troubleshooting](load-balancer-and-ingress.md) for pending services, ingress controller state, backend routing, and TLS failures.
77- [Network Policy Troubleshooting](network-policy.md) for default-deny checks, Azure NPM or Calico validation, and ingress or egress rule audits.
78