1# Tracing Insights API
2 
3Automatically detect quality regressions and anomalies in agent traces using changepoint detection on evaluation scores stored in App Insights.
4 
5## When to Use
6 
7Use this instead of manual KQL queries when you want **automated anomaly detection** across evaluation dimensions (task adherence, intent resolution, fluency, latency, token usage). The API finds statistical changepoints in score distributions — no manual threshold tuning needed.
8 
9**Prerequisites:**
10- App Insights connected to the Foundry project (with `gen_ai.evaluation.result` custom events)
11- Evaluation data from portal playground sessions or batch evals (raw traces alone are not enough)
12 
13## Endpoint
14 
15```
16POST https://{region}.api.azureml.ms/notification/v1-beta2/subscriptions/{sub}/resourceGroups/{rg}/providers/microsoft.insights/components/{component}/:insights
17```
18 
19The API is region-agnostic — any regional endpoint can serve requests for any project. For lowest latency, use the same region as the Foundry project (e.g., `eastus2`, `westus2`, `westcentralus`). If the project region is unknown, use `eastus2` as the default.
20 
21**Query parameters:**
22| Parameter          | Required | Description                                                            |
23|--------------------|----------|------------------------------------------------------------------------|
24| `startDateTimeUtc` | Yes      | ISO 8601 start of analysis window                                      |
25| `endDateTimeUtc`   | Yes      | ISO 8601 end of analysis window                                        |
26| `agent`            | Yes      | Agent name (URL-encoded)                                               |
27| `projectId`        | Yes      | ARM resource ID of the Foundry project (URL-encoded — contains slashes)|
28| `top`              | No       | Max insights to return (default 50)                                    |
29 
30**Auth:** `az account get-access-token --resource https://ai.azure.com`
31 
32**Body:** Must send `{}` (empty JSON object) — POST with no body returns 400.
33 
34## Example
35 
36```powershell
37$token = az account get-access-token --resource https://ai.azure.com --query accessToken -o tsv
38$encodedAgent = [uri]::EscapeDataString("my-agent")
39$encodedProjectId = [uri]::EscapeDataString("/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.CognitiveServices/accounts/{account}/projects/{project}")
40 
41$uri = "https://{region}.api.azureml.ms/notification/v1-beta2/subscriptions/{sub}/resourceGroups/{rg}/providers/microsoft.insights/components/{component}/:insights?startDateTimeUtc=2025-01-01T00:00:00Z&endDateTimeUtc=2025-01-18T00:00:00Z&agent=$encodedAgent&projectId=$encodedProjectId&top=50"
42 
43$response = Invoke-RestMethod -Uri $uri -Method POST -Headers @{
44    "Authorization" = "Bearer $token"
45    "Content-Type" = "application/json"
46} -Body "{}"
47```
48 
49## Response Structure (v1-beta2)
50 
51Response is grouped by agent version. Each insight includes `relatedSpans` with `operationId` (App Insights trace ID) for querying full trace content.
52 
53```json
54{
55  "agents": [{
56    "agent": "my-agent:1",
57    "insights": [{
58      "id": "anomaly-token-shift-<hash>",
59      "type": "Token",
60      "severity": "Critical",
61      "message": "Token usage increased by 137%",
62      "agentVersion": "1",
63      "metadata": { "meanBefore": 2041, "meanAfter": 4831, "confidence": 0.91 },
64      "relatedSpans": {
65        "totalCount": 13,
66        "spans": [
67          { "responseId": "resp_...", "operationId": "<trace-id>", "evaluationRunId": null }
68        ]
69      }
70    }],
71    "insightCount": 3
72  }],
73  "totalCount": 3, "criticalCount": 1, "warningCount": 1, "improvementCount": 1
74}
75```
76 
77## Querying Traces from relatedSpans
78 
79Use `operationId` from `relatedSpans` to fetch full trace content from App Insights:
80 
81```kql
82dependencies
83| where operation_Id == "<operationId>"
84| where customDimensions has "invoke_agent"
85| project input = customDimensions["gen_ai.input.messages"],
86          output = customDimensions["gen_ai.output.messages"],
87          tokens = toint(customDimensions["gen_ai.usage.output_tokens"])
88```
89 
90This returns the user query and agent response for the specific trace flagged by the insight.
91 
92## How Changepoint Detection Works
93 
94The API finds **statistical inflection points within the queried time window**. `meanBefore`/`meanAfter` represent averages on either side of the detected shift — not comparisons to a historical baseline.
95 
96- 10+ data points give better signal for changepoint detection
97- `confidence` close to 1.0 = statistically significant shift
98 
99## Next Steps
100 
101After receiving insights with `Warning` or `Critical` severity:
1021. Use `relatedSpans.operationId` values to query full trace content from App Insights (see KQL above)
1032. Present the insights summary to the user with severity, type, evaluator name, and shift magnitude
1043. Offer to drill into specific traces for detailed analysis using the [trace analysis skill](../trace.md)
105