1# Analyze Failures — Find and Cluster Failing Traces
2 
3Identify failing agent traces, group them by root cause, and produce a prioritized action table.
4 
5## Step 1 — Find Failing Traces
6 
7> ⚠️ **Hosted agents:** `gen_ai.agent.name` on `dependencies` holds the **code-level class name** (e.g., `BingSearchAgent`), NOT the Foundry agent name. To filter by Foundry name, use the [Hosted Agent Variant](#hosted-agent-variant--failures) below.
8 
9```kql
10dependencies
11| where timestamp > ago(24h)
12| where success == false or toint(resultCode) >= 400
13| extend
14    operation = tostring(customDimensions["gen_ai.operation.name"]),
15    errorType = tostring(customDimensions["error.type"]),
16    model = tostring(customDimensions["gen_ai.request.model"]),
17    agentName = tostring(customDimensions["gen_ai.agent.name"]),
18    conversationId = tostring(customDimensions["gen_ai.conversation.id"])
19| project timestamp, name, duration, resultCode, errorType, operation, model,
20    agentName, conversationId, operation_Id, id
21| order by timestamp desc
22| take 100
23```
24 
25## Step 2 — Cluster by Error Type
26 
27```kql
28dependencies
29| where timestamp > ago(24h)
30| where success == false or toint(resultCode) >= 400
31| extend
32    errorType = tostring(customDimensions["error.type"]),
33    operation = tostring(customDimensions["gen_ai.operation.name"])
34| summarize
35    count = count(),
36    firstSeen = min(timestamp),
37    lastSeen = max(timestamp),
38    avgDuration = avg(duration),
39    sampleOperationId = take_any(operation_Id)
40  by errorType, operation, resultCode
41| order by count desc
42```
43 
44## Step 3 — Prioritized Action Table
45 
46Present results as:
47 
48| Priority | Error Type | Operation | Count | Result Code | Suggested Action |
49|----------|-----------|-----------|-------|-------------|-----------------|
50| P0 | timeout | invoke_agent | 15 | 504 | Check agent container health, increase timeout |
51| P1 | rate_limited | chat | 8 | 429 | Check quota, add retry logic |
52| P2 | content_filter | chat | 5 | 400 | Review prompt for policy violations |
53| P3 | tool_error | execute_tool | 3 | 500 | Check tool implementation and permissions |
54 
55**Prioritization:** P0 = highest count or most severe (5xx), then by count × recency.
56 
57## Step 4 — Drill Into Specific Failure
58 
59When the user selects a cluster, show individual failing traces:
60 
61```kql
62dependencies
63| where timestamp > ago(24h)
64| where success == false
65| where customDimensions["error.type"] == "<selected_error_type>"
66| where customDimensions["gen_ai.operation.name"] == "<selected_operation>"
67| project timestamp, name, duration, resultCode,
68    conversationId = tostring(customDimensions["gen_ai.conversation.id"]),
69    responseId = tostring(customDimensions["gen_ai.response.id"]),
70    operation_Id
71| order by timestamp desc
72| take 20
73```
74 
75Also check `exceptions` table for stack traces:
76 
77```kql
78exceptions
79| where timestamp > ago(24h)
80| where operation_Id in ("<operation_id_1>", "<operation_id_2>")
81| project timestamp, type, message, outerMessage, details, operation_Id
82| order by timestamp desc
83```
84 
85Offer to view the full conversation for any trace via [Conversation Detail](conversation-detail.md).
86 
87## Hosted Agent Variant — Failures
88 
89For hosted agents, the Foundry agent name lives on `requests`, not `dependencies`. Use a two-step join:
90 
91```kql
92let reqIds = requests
93| where timestamp > ago(24h)
94| where customDimensions["gen_ai.agent.name"] == "<foundry-agent-name>"
95| distinct id;
96dependencies
97| where timestamp > ago(24h)
98| where operation_ParentId in (reqIds)
99| where success == false or toint(resultCode) >= 400
100| extend
101    operation = tostring(customDimensions["gen_ai.operation.name"]),
102    errorType = tostring(customDimensions["error.type"]),
103    model = tostring(customDimensions["gen_ai.request.model"]),
104    conversationId = tostring(customDimensions["gen_ai.conversation.id"])
105| project timestamp, name, duration, resultCode, errorType, operation, model,
106    conversationId, operation_ParentId, operation_Id
107| order by timestamp desc
108| take 100
109```
110