1# AKS Cost Anomaly Investigation
2 
3Investigate user-reported cost or utilization spikes by correlating Azure Monitor metrics, scaling events, and Cost Management data.
4 
5## Step 1 - Confirm Timeframe
6 
7Ask the user: "When did you notice the spike? (e.g., 'last Tuesday', 'between 2 AM and 4 AM yesterday')"
8 
9## Step 2 - Pull Cost Data
10 
11```bash
12az rest --method post \
13  --url "https://management.azure.com/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.CostManagement/query?api-version=2023-11-01" \
14  --headers "ClientType=GitHubCopilotForAzure" \
15  --body '{
16    "type": "ActualCost",
17    "timeframe": "Custom",
18    "timePeriod": { "from": "<start-date>", "to": "<end-date>" },
19    "dataset": {
20      "granularity": "Daily",
21      "aggregation": { "totalCost": { "name": "Cost", "function": "Sum" } },
22      "grouping": [{ "type": "Dimension", "name": "ResourceId" }]
23    }
24  }'
25```
26 
27## Step 3 - Pull Node Count and Scaling Events
28 
29```bash
30# First, verify available metrics on your AKS resource
31az monitor metrics list-definitions \
32  --resource "<aks-resource-id>" \
33  --output table
34 
35# Node count over the anomaly window (use metric name from list-definitions output)
36az monitor metrics list \
37  --resource "<aks-resource-id>" \
38  --metrics "<verified-node-count-metric>" \
39  --interval PT5M --aggregation Count \
40  --start-time "<start-date>" --end-time "<end-date>"
41 
42# HPA scaling events
43kubectl get events --all-namespaces \
44  --field-selector reason=SuccessfulRescale \
45  --sort-by='.lastTimestamp'
46```
47 
48## Step 4 - Top Consumers
49 
50```bash
51kubectl top nodes
52kubectl top pods --all-namespaces --sort-by=cpu
53```
54 
55## Common Causes
56 
57| Symptom | Likely Cause | Action |
58|---------|-------------|--------|
59| Node count surged off-peak | HPA/VPA misconfiguration | Review HPA min replicas |
60| Single pod consuming all CPU | Memory leak or runaway process | Check logs, add resource limits |
61| Cost spike on specific day | Batch job ran unexpectedly | Review CronJob schedule |
62| Persistent high node count | CAS scale-down blocked | Check PodDisruptionBudgets, system pods |
63| Sudden namespace cost jump | New deployment with no resource limits | Add requests/limits |
64 
65## Set Up Budget Alert
66 
67```bash
68az consumption budget create \
69  --budget-name "aks-monthly-budget" \
70  --amount <budget-amount> \
71  --time-grain Monthly \
72  --start-date "<YYYY-MM-01>" \
73  --end-date "<YYYY-MM-01>" \
74  --resource-group "<resource-group>" \
75  --threshold 80 \
76  --contact-emails "<contact-email>"
77```
78 
79