1# Agent Metadata Contract
2 
3Use this contract for every agent source folder that participates in Microsoft Foundry workflows.
4 
5## Required Local Layout
6 
7```text
8<agent-root>/
9  .foundry/
10    agent-metadata.yaml
11    agent-metadata.prod.yaml
12    datasets/
13    evaluators/
14    results/
15```
16 
17- `agent-metadata.yaml` is the preferred local/dev metadata file.
18- Optional sidecar files such as `agent-metadata.prod.yaml` can hold a single prod or CI-targeted environment without mixing multiple environments in one file.
19- `datasets/` and `evaluators/` are local cache folders. Reuse existing files when they are current, and ask before refreshing or overwriting them.
20- `results/` stores local evaluation outputs and comparison artifacts by environment.
21 
22## Metadata File Model
23 
24| File | Typical use | Notes |
25|------|-------------|-------|
26| `.foundry/agent-metadata.yaml` | Preferred local/dev metadata | Default choice for local workflows when no file is specified |
27| `.foundry/agent-metadata.<env>.yaml` | Optional prod/CI or modular environment-specific metadata | Prefer this when the workflow explicitly targets that environment and the file exists |
28 
29New setups should prefer **one environment per metadata file** while keeping the current schema shape (`defaultEnvironment` + `environments.<name>`) for compatibility. Legacy multi-environment `agent-metadata.yaml` files remain supported.
30 
31## Environment Model
32 
33| Field | Required | Purpose |
34|-------|----------|---------|
35| `defaultEnvironment` | ✅ | Default environment inside the selected metadata file; in preferred single-environment files it should match the only environment key |
36| `environments.<name>.projectEndpoint` | ✅ | Foundry project endpoint for that environment |
37| `environments.<name>.agentName` | ✅ | Deployed Foundry agent name |
38| `environments.<name>.azureContainerRegistry` | ✅ for hosted agents | ACR used for deployment and image refresh |
39| `environments.<name>.observability.applicationInsightsResourceId` | Recommended | App Insights resource for trace workflows |
40| `environments.<name>.observability.applicationInsightsConnectionString` | Optional | Connection string when needed for tooling |
41| `environments.<name>.evaluationSuites[]` | ✅ | Dataset + local/remote references + evaluator + tag bundles for evaluation workflows |
42 
43## Example `.foundry/agent-metadata.yaml` (local/dev)
44 
45```yaml
46defaultEnvironment: dev
47environments:
48  dev:
49    projectEndpoint: https://contoso.services.ai.azure.com/api/projects/support-dev
50    agentName: support-agent-dev
51    azureContainerRegistry: contosoregistry.azurecr.io
52    observability:
53      applicationInsightsResourceId: /subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Insights/components/support-dev-ai
54    evaluationSuites:
55      - id: smoke-core
56        tags:
57          tier: smoke
58          purpose: baseline
59          stage: seed
60        dataset: support-agent-dev-eval-seed
61        datasetVersion: v1
62        datasetFile: .foundry/datasets/support-agent-dev-eval-seed-v1.jsonl
63        datasetUri: <foundry-dataset-uri>
64        evaluators:
65          - name: intent_resolution
66            threshold: 4
67          - name: task_adherence
68            threshold: 4
69          - name: citation_quality
70            threshold: 0.9
71            definitionFile: .foundry/evaluators/citation-quality.yaml
72      - id: trace-regression-suite
73        tags:
74          tier: regression
75          purpose: regression
76          stage: traces
77        dataset: support-agent-dev-traces
78        datasetVersion: v3
79        datasetFile: .foundry/datasets/support-agent-dev-traces-v3.jsonl
80        datasetUri: <foundry-dataset-uri>
81        evaluators:
82          - name: coherence
83            threshold: 4
84          - name: groundedness
85            threshold: 4
86```
87 
88## Example `.foundry/agent-metadata.prod.yaml` (prod/CI)
89 
90```yaml
91defaultEnvironment: prod
92environments:
93  prod:
94    projectEndpoint: https://contoso.services.ai.azure.com/api/projects/support-prod
95    agentName: support-agent-prod
96    azureContainerRegistry: contosoregistry.azurecr.io
97    evaluationSuites:
98      - id: production-guardrails
99        tags:
100          tier: smoke
101          purpose: safety
102          stage: prod
103        dataset: support-agent-prod-curated
104        datasetVersion: v2
105        datasetFile: .foundry/datasets/support-agent-prod-curated-v2.jsonl
106        datasetUri: <foundry-dataset-uri>
107        evaluators:
108          - name: violence
109            threshold: 1
110          - name: self_harm
111            threshold: 1
112```
113 
114## Workflow Rules
115 
1161. Auto-discover agent roots by searching for `.foundry/` folders that contain `agent-metadata.yaml` or `agent-metadata.<env>.yaml`.
1172. If exactly one agent root is found, use it. If multiple roots are found, require the user to choose one.
1183. Inside the selected agent root, select the metadata file in this order: explicit file/path from the user or workflow, then `.foundry/agent-metadata.<env>.yaml` when an explicit environment is already known and that file exists, then `.foundry/agent-metadata.yaml`. If `.foundry/agent-metadata.yaml` is absent, use the only matching sidecar file when exactly one `.foundry/agent-metadata.<env>.yaml` file exists; if multiple sidecar files exist and no explicit file/path was provided, require the user to choose the metadata file.
1194. Resolve environment in this order: explicit user choice, then the file's only environment when the selected metadata file is single-environment, then remembered session choice, then `defaultEnvironment`.
1205. Keep the selected agent root, metadata file, and environment visible in every deploy, eval, dataset, and trace summary.
1216. Once an agent root is selected, use only that root's `.foundry/` folders and source tree for local evaluation, dataset, trace, deploy, and prompt-optimization context. Do not merge sibling agent folders.
1227. Treat `datasets/` and `evaluators/` as cache folders. Reuse local files when present, but offer refresh when the user asks or when remote state is newer.
1238. Writes must target the selected metadata file only. For preferred single-environment files, update only that one environment block. For legacy multi-environment files, rewrite only the selected environment block. Never copy or merge environments across sibling metadata files automatically.
1249. Never overwrite cache files or metadata silently.
125 
126## Legacy Compatibility (`testCases[]` / `testSuites[]` -> `evaluationSuites[]`)
127 
128Use `evaluationSuites[]` as the canonical schema. If the selected environment still uses older `testSuites[]` and does not yet define `evaluationSuites[]`, treat that list as the current suite source, normalize it in memory, and migrate it on the next metadata write. If the selected environment is older still and uses legacy `testCases[]` without `evaluationSuites[]`, treat `testCases[]` as the suite source and normalize it the same way.
129 
130| Legacy field | Migration behavior |
131|--------------|--------------------|
132| `id` | Keep as-is |
133| `dataset`, `datasetVersion`, `datasetFile`, `datasetUri`, `evaluators` | Keep as-is |
134| `tags` | Preserve if already present |
135| `priority` | If `tags.tier` is missing, map `P0` -> `smoke`, `P1` -> `regression`, `P2` -> `coverage` |
136 
137When a workflow writes metadata, rewrite the selected metadata file so the target environment contains only `evaluationSuites[]`. Do not keep older `testSuites[]` or legacy `testCases[]` in the rewritten block.
138 
139## Evaluation-Suite Guidance
140 
141Use `tags` as a freeform key/value map on each evaluation suite. Suggested keys:
142 
143| Tag Key | Example Values | Typical Use |
144|---------|----------------|-------------|
145| `tier` | `smoke`, `regression`, `coverage` | Suggested run order / breadth |
146| `purpose` | `baseline`, `safety`, `tools`, `quality`, `regression` | Why the suite exists |
147| `stage` | `seed`, `traces`, `curated`, `prod` | Dataset lifecycle alignment |
148 
149Each evaluation suite should point to one dataset and one or more evaluators with explicit thresholds. Store `dataset` as the stable Foundry dataset name (without the `-vN` suffix), store the version separately in `datasetVersion`, and keep the local cache filename versioned (for example, `...-v3.jsonl`). Persist the local `datasetFile` and remote `datasetUri` together so every evaluation suite can resolve both the cache artifact and the Foundry-registered dataset. Add a `tags` map to each suite (for example, `tier: smoke`, `purpose: baseline`) so workflows can group or filter suites without a fixed priority enum. Local dataset filenames should start with the selected environment's Foundry `agentName` from the selected metadata file, followed by stage and version suffixes, so related cache files stay grouped by agent. If `agentName` already encodes the environment (for example, `support-agent-dev`), do not append the environment key again. Keep `datasets/`, `evaluators/`, and `results/` shared at the `.foundry/` root even when multiple metadata files exist. Use evaluation-suite IDs in evaluation names, result folders, and regression summaries so the flow remains traceable.
150 
151## Sync Guidance
152 
153- Pull/refresh when the user asks, when the workflow detects missing local cache, or when remote versions clearly differ from local metadata.
154- Push/register updates after the user confirms local changes that should be shared in Foundry.
155- Record remote dataset names, versions, dataset URIs, and last sync timestamps in `.foundry/datasets/manifest.json` or the relevant metadata section.
156