1# Agent Metadata Contract
2 
3Use this contract for Microsoft Foundry agent folders. In azd projects, `.foundry/agent-metadata*.yaml` is an overlay/cache, not the source of truth for azd-owned deployment context.
4 
5## Local Layout
6 
7```text
8<agent-root>/
9  .foundry/
10    agent-metadata.yaml
11    agent-metadata.<env>.yaml
12    suites/
13    datasets/
14    evaluators/
15    results/
16```
17 
18- `agent-metadata.yaml` is the preferred local/dev overlay.
19- Optional `agent-metadata.<env>.yaml` files can hold a single prod or CI-targeted overlay.
20- `suites/`, `datasets/`, `evaluators/`, and `results/` are local cache/result folders. Ask before overwriting user-edited files.
21 
22## Effective Context Model
23 
24Resolve deployment and evaluation context by layering sources in this order:
25 
26| Value | Preferred source | Fallbacks | Metadata write behavior |
27|-------|------------------|-----------|-------------------------|
28| Agent root | `azure.yaml` service `project` for `host: azure.ai.agent` | `.foundry` discovery, user path | Do not write except to initialize cache |
29| Environment | user/session, then azd env/default | metadata `defaultEnvironment` | Store azd binding only when useful |
30| Project endpoint | `azd env get-values` | metadata, user input | Do not duplicate azd values |
31| Agent name/version | azd `AGENT_<SERVICE>_*` vars | `agent.yaml`, metadata, user input | Do not duplicate azd values |
32| ACR | azd registry vars | metadata, user input | Do not duplicate azd values |
33| Observability | azd App Insights vars | metadata, user input | Do not copy secrets if azd has them |
34| Local eval draft | `eval.yaml` | metadata, user input | Sync to `.foundry` only after remote lookup/registration |
35| Remote suite/cache refs | metadata | Foundry lookups | Persist in `.foundry` |
36 
37If azd and metadata both provide the same value and differ, stop and ask which source is authoritative. If they match, use the azd value and omit the duplicate on future metadata rewrites.
38 
39## Environment Overlay Model
40 
41| Field | Required when | Purpose |
42|-------|---------------|---------|
43| `defaultEnvironment` | Any metadata file exists | Default key inside this overlay file |
44| `environments.<env>.azd.environmentName` | Optional | Binds overlay to an azd environment |
45| `environments.<env>.azd.service` | Optional | Binds overlay to an `azure.yaml` service |
46| `environments.<env>.projectEndpoint` | Required for non-azd/manual workflows | Explicit override when azd cannot resolve it |
47| `environments.<env>.agentName` / `agentVersion` | `agentName` required for non-azd/manual workflows; `agentVersion` optional | Explicit override when azd cannot resolve it |
48| `environments.<env>.azureContainerRegistry` | Required for non-azd/manual hosted-agent Docker/ACR deploy flow | Explicit override when azd cannot resolve it |
49| `environments.<env>.observability.*` | Required only for trace workflows when azd cannot resolve observability | Trace lookup config when azd cannot resolve it |
50| `environments.<env>.evaluationSuites[]` | Required after evaluation setup/sync | Remote suite/dataset/evaluator refs plus local cache paths |
51| `environments.<env>.lastEval` | Optional | Last local result summary and result file path |
52 
53## Example azd Overlay
54 
55```yaml
56defaultEnvironment: dev
57environments:
58  dev:
59    azd:
60      environmentName: <azd-env-name>
61      service: <azure-yaml-service-name>
62    evaluationSuites:
63      - id: smoke-core
64        suiteName: <foundry-suite-name>
65        suiteVersion: "1"
66        generationSource: eval-yaml
67        tags:
68          tier: smoke
69          purpose: baseline
70        suiteFile: .foundry/suites/<suite>-v1.json
71        dataset: <dataset-name>
72        datasetVersion: "1"
73        datasetFile: .foundry/datasets/<agent>-<dataset>-v1.ref.json
74        datasetUri: <foundry-dataset-uri>
75        evaluators:
76          - name: <evaluator-name>
77            version: "1"
78            threshold: 4
79            definitionFile: .foundry/evaluators/<evaluator>-v1.json
80```
81 
82## Example Manual Overlay
83 
84```yaml
85defaultEnvironment: dev
86environments:
87  dev:
88    projectEndpoint: https://<account>.services.ai.azure.com/api/projects/<project>
89    agentName: <agent-name>
90    azureContainerRegistry: <registry>.azurecr.io
91    evaluationSuites:
92      - id: smoke-core
93        datasetFile: .foundry/datasets/<agent>-smoke-v1.ref.json
94        evaluators:
95          - name: relevance
96            threshold: 4
97```
98 
99## eval.yaml Mapping
100 
101When `eval.yaml` exists in the selected agent root, treat it as local evaluation intent, not proof of a Foundry suite.
102 
103| eval.yaml field | Use |
104|-----------------|-----|
105| `agent.name` | Candidate target agent; verify it matches selected context |
106| `dataset.local_uri` | Local seed dataset candidate |
107| `dataset.name`, `dataset.version` | Registered dataset candidate |
108| `validation_dataset` | Optional validation dataset candidate |
109| `evaluators[]` | Candidate evaluator names; verify with `evaluator_catalog_get` |
110| `name` | Candidate eval/suite name; verify remotely before storing as `suiteName` |
111| `options.eval_model` | Candidate judge/generation deployment |
112| `options.optimization_model` | Candidate optimizer reasoning deployment |
113| `options.max_candidates` | Candidate optimization iteration limit |
114| `options.optimization_config.model_search_space` | Candidate target model search space |
115| `options.pass_threshold` | Candidate evaluator threshold/default pass gate |
116| `max_samples`, `trace_days`, `generation_instruction` | Suite setup defaults |
117 
118Legacy `dataset_file`, `dataset_reference`, and `validation_reference` keys may be normalized in memory when reading older files, but new files should use `dataset` and `validation_dataset`.
119 
120Persist eval.yaml-derived suite metadata only after the relevant dataset/evaluator/suite has been registered or found in Foundry. Use `generationSource: eval-yaml` for synced suite entries created from local eval config.
121 
122## Workflow Rules
123 
1241. Prefer azd service discovery before `.foundry` discovery when `azure.yaml` has `host: azure.ai.agent`.
1252. Once an agent root is selected, use only that root's `.foundry`, source tree, `agent.yaml`, and `eval.yaml` unless the user switches roots.
1263. Select metadata files in this order: explicit file/path, environment sidecar, `.foundry/agent-metadata.yaml`, then prompt if ambiguous.
1274. Resolve environment from user/session, azd env/default, single-environment metadata, then `defaultEnvironment`.
1285. Keep the selected root, environment, metadata overlay file, and primary context source visible in deploy/eval/trace summaries.
1296. Treat metadata deployment fields as overrides when azd cannot resolve the value.
1307. Treat `evaluationSuites[]` as the canonical synced suite model; normalize legacy fields in memory before use.
1318. Writes target only the selected metadata file and selected environment. Never merge sibling metadata files automatically.
1329. On metadata rewrites for azd projects, persist non-derivable overlay/cache state and omit azd-owned deployment duplicates.
13310. Never silently overwrite cache files or metadata. Show a summary before refreshing, pruning duplicate fields, or replacing suite refs.
134 
135## Legacy Compatibility
136 
137If the selected environment has `testSuites[]` but no `evaluationSuites[]`, treat `testSuites[]` as the current suite source and migrate it on the next metadata write. If it has only legacy `testCases[]`, normalize that list the same way.
138 
139Preserve `id`, `suiteName`, `suiteVersion`, `generationJobId`, `generationSource`, `dataset`, `datasetVersion`, `datasetFile`, `datasetUri`, `evaluators`, and existing `tags`. Map legacy `priority` to `tags.tier` only when `tags.tier` is missing: `P0` -> `smoke`, `P1` -> `regression`, `P2` -> `coverage`.
140 
141## Evaluation Suite Guidance
142 
143Use `tags` as freeform key/value metadata. Suggested keys: `tier` (`smoke`, `regression`, `coverage`), `purpose` (`baseline`, `safety`, `tools`, `quality`), and `stage` (`local`, `generated`, `traces`, `curated`, `prod`).
144 
145Each synced suite should point to one dataset and one or more evaluators with thresholds. Store stable remote names separately from versions, keep local cache filenames versioned, and persist `suiteFile`, `datasetFile`, `datasetContentPath`, `datasetUri`, and evaluator `definitionFile` when available. Local dataset filenames should start with the effective Foundry agent name. Use evaluation-suite IDs in evaluation names, result folders, and regression summaries.
146 
147For generated Foundry suites, persist `suiteName`, `suiteVersion`, `generationJobId`, and `generationSource`. `suiteName` must start with a letter (`A-Z` or `a-z`); prefix derived numeric names with an alphabetic label such as `suite-`. A suite with `suiteName` still runs batch eval through `evaluation_agent_batch_eval_create`; use `evaluation_suite_get` only to resolve reviewed dataset/evaluator metadata.
148