Eval Lineage — Full Traceability from Production to Deployment

Track the complete chain from production traces through dataset creation, evaluation runs, comparisons, and deployment decisions. Enables "why was this deployed?" audit queries and compliance reporting.

Lineage Chain

Production Trace (App Insights)
    │ conversationId, responseId
    ▼
Dataset Version (.foundry/datasets/*.jsonl, environment-scoped)
    │ metadata.conversationId, metadata.harvestRule
    ▼
Evaluation Run (evaluation_agent_batch_eval_create)
    │ evaluationId when creating, evalId when querying, evalRunId
    ▼
Comparison (evaluation_comparison_create)
    │ insightId, baselineRunId, treatmentRunIds
    ▼
Deployment Decision (agent_update)
    │ agentVersion
    ▼
Production Trace (cycle repeats)

Lineage Manifest

Track lineage in .foundry/datasets/manifest.json:

{
  "datasets": [
    {
      "name": "support-bot-prod-traces",
      "file": "support-bot-prod-traces-v3.jsonl",
      "version": "v3",
      "tag": "prod",
      "source": "trace-harvest",
      "harvestRule": "error+latency",
      "timeRange": "2025-02-01 to 2025-02-07",
      "exampleCount": 63,
      "createdAt": "2025-02-08T10:00:00Z",
      "evalRuns": [
        {
          "evalId": "eval-group-001",
          "runId": "run-abc-123",
          "agentVersion": "3",
          "date": "2025-02-08T12:00:00Z",
          "status": "completed"
        },
        {
          "evalId": "eval-group-001",
          "runId": "run-def-456",
          "agentVersion": "4",
          "date": "2025-02-10T09:00:00Z",
          "status": "completed"
        }
      ],
      "comparisons": [
        {
          "insightId": "insight-xyz-789",
          "baselineRunId": "run-abc-123",
          "treatmentRunIds": ["run-def-456"],
          "result": "v4 improved on 3/5 metrics",
          "date": "2025-02-10T10:00:00Z"
        }
      ],
      "deployments": [
        {
          "agentVersion": "4",
          "deployedAt": "2025-02-10T14:00:00Z",
          "reason": "v4 improved coherence +25%, relevance +10% vs v3"
        }
      ]
    }
  ]
}

Audit Queries

"Why was version X deployed?"

1. Read .foundry/datasets/manifest.json
2. Find entries where deployments[].agentVersion == X
3. Show the comparison that justified the deployment
4. Show the dataset and eval runs that informed the comparison

"What traces led to this dataset?"

1. Read the dataset JSONL file
2. Extract metadata.conversationId from each example
3. Look up each conversation in App Insights using the trace skill

"What evaluation history does this agent have?"

1. Use evaluation_get to list all evaluation groups
2. For each group, list runs with isRequestForRuns=true
3. Build the timeline from Eval Trending
4. Show comparisons from evaluation_comparison_get

"Did this dataset version catch any regressions?"

1. Find the dataset version in the manifest
2. Check evalRuns for runs that used this dataset
3. Check comparisons for any regression results
4. Cross-reference with tag == "regression-<date>" entries

Maintaining Lineage

Update .foundry/datasets/manifest.json at each step:

💡 Tip: Store the evaluation group identifier as evalId in lineage/manifest records, even if the create call used the parameter name evaluationId.

Next Steps

• View metric trends → Eval Trending
• Check for regressions → Eval Regression
• Harvest new traces → Trace-to-Dataset (start the next cycle)