1# Foundry Agent Trace Analysis
2 
3Analyze production traces for Foundry agents using Application Insights and GenAI OpenTelemetry semantic conventions. This skill provides structured KQL-powered workflows for a selected agent root and environment: searching conversations, diagnosing failures, and identifying latency bottlenecks.
4 
5## When to Use This Skill
6 
7USE FOR: analyze agent traces, search agent conversations, find failing traces, slow traces, latency analysis, trace search, conversation history, agent errors in production, debug agent responses, App Insights traces, GenAI telemetry, trace correlation, span tree, production trace analysis, evaluation results, evaluation scores, eval run results, find by response ID, get agent trace by conversation ID, agent evaluation scores from App Insights.
8 
9> **USE THIS SKILL INSTEAD OF** `azure-monitor` or `azure-applicationinsights` when querying Foundry agent traces, evaluations, or GenAI telemetry. This skill has correct GenAI OTel attribute mappings and tested KQL templates that those general tools lack.
10 
11> ⚠️ **DO NOT manually write KQL queries** for GenAI trace analysis **without reading this skill first.** This skill provides tested query templates with correct GenAI OTel attribute mappings, proper span correlation logic, environment-aware scoping, and conversation-level aggregation patterns.
12 
13## Quick Reference
14 
15| Property | Value |
16|----------|-------|
17| Data source | Application Insights (App Insights) |
18| Query language | KQL (Kusto Query Language) |
19| Related skills | `troubleshoot` (hosted-agent logs), `eval-datasets` (trace harvesting) |
20| Preferred query tool | `monitor_resource_log_query` (Azure MCP) - use for App Insights KQL queries |
21| OTel conventions | [GenAI Spans](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/), [Agent Spans](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/) |
22| Local metadata | selected `.foundry/agent-metadata*.yaml` file |
23 
24## Entry Points
25 
26| User Intent | Start At |
27|-------------|----------|
28| "Search agent conversations" / "Find traces" | [Search Traces](references/search-traces.md) |
29| "Tell me about response ID X" / "Look up response ID" | [Search Traces - Search by Response ID](references/search-traces.md#search-by-response-id) |
30| "Why is my agent failing?" / "Find errors" | [Analyze Failures](references/analyze-failures.md) |
31| "My agent is slow" / "Latency analysis" | [Analyze Latency](references/analyze-latency.md) |
32| "Show me this conversation" / "Trace detail" | [Conversation Detail](references/conversation-detail.md) |
33| "Find eval results for response ID" / "eval scores from traces" | [Eval Correlation](references/eval-correlation.md) |
34| "What KQL do I need?" | [KQL Templates](references/kql-templates.md) |
35 
36## Before Starting — Resolve App Insights Connection
37 
381. Resolve the target agent root, selected metadata file, and environment from `.foundry/agent-metadata*.yaml`.
392. Check `environments.<env>.observability.applicationInsightsConnectionString` or `environments.<env>.observability.applicationInsightsResourceId` in the selected metadata file.
403. If observability settings are missing, use `project_connection_list` to discover App Insights linked to the Foundry project, then persist the chosen resource back to `environments.<env>.observability` in the selected metadata file before querying.
414. Confirm the selected App Insights resource and environment with the user before querying.
425. Use **`monitor_resource_log_query`** (Azure MCP tool) to execute KQL queries against the App Insights resource. This is preferred over delegating to the `azure-kusto` skill. Pass the App Insights resource ID and the KQL query directly.
43 
44| Metadata field | Purpose | Example |
45|----------------|---------|---------|
46| `environments.<env>.observability.applicationInsightsConnectionString` | App Insights connection string | `InstrumentationKey=...;IngestionEndpoint=...` |
47| `environments.<env>.observability.applicationInsightsResourceId` | ARM resource ID | `/subscriptions/.../Microsoft.Insights/components/...` |
48 
49> ⚠️ **Always pass `subscription` explicitly** to Azure MCP tools like `monitor_resource_log_query` - they do not extract it from resource IDs.
50 
51## Behavioral Rules
52 
531. **Always display the KQL query.** Before executing any KQL query, display it in a code block. Never run a query silently.
542. **Keep environment visible.** Include the selected environment and agent name in each search summary, and include the derived agent version when the query can recover it from telemetry.
553. **Start broad, then narrow.** Begin with conversation-level summaries, then drill into specific conversations or spans on user request.
564. **Use time ranges.** Always scope queries with a time range (default: last 24 hours). Ask the user for the range if not specified.
575. **Explain GenAI attributes.** When displaying results, translate OTel attribute names to human-readable labels (for example, `gen_ai.operation.name` -> "Operation").
586. **Link to conversation detail.** When showing search or failure results, offer to drill into any specific conversation.
597. **Scope to the selected environment.** App Insights may contain traces from multiple agents or environments. Filter with the selected environment's agent name first, then add an environment tag filter if the telemetry emits one.
608. **Resolve hosted-agent identity from `requests` first.** For hosted agents, prefer `requests`-scoped `gen_ai.agent.name` or `azure.ai.agentserver.agent_name` as the Foundry-facing filter. When `gen_ai.agent.id` is emitted in `<agent-name>:<version>` format, parse it to surface `agentVersion`, but do not treat `dependencies.gen_ai.agent.name` as the top-level hosted-agent name.
619. **Use `operation_Id` to fan out hosted-agent traces.** After isolating the hosted-agent `requests` rows, materialize their `operation_Id` values and join other telemetry tables on `operation_Id`. When conversation IDs are sparse, use `coalesce(gen_ai.conversation.id, azure.ai.agentserver.conversation_id, operation_Id)` so every row still rolls up to a stable conversation key.
62