1# Step 1 - Auto-Setup Evaluation Suite
2 
3> **This step runs automatically after deployment.** If the agent was deployed via the [deploy skill](../../deploy/deploy.md), `.foundry` cache and metadata may already be configured. Check `.foundry/evaluators/`, `.foundry/datasets/`, and the selected metadata file under the selected agent root before re-creating them.
4 
5## Auto-Generate Suite
6 
7After deployment, immediately prepare a Foundry evaluation suite and local references for the selected environment without waiting for the user to request it.
8 
9### 1. Resolve Context
10 
11Use [Common Project Context Resolution](../../../SKILL.md#agent-common-project-context-resolution) to compute effective context. In azd projects, prefer `azd env get-values` for deployment context and use the selected `.foundry/agent-metadata*.yaml` file only as an overlay/cache. Use `agent_get`, local `agent.yaml`, and matching `eval.yaml` as needed to resolve:
12 
13| Value | Source |
14|-------|--------|
15| `projectEndpoint` | azd env, then metadata override |
16| `agentName` / `agentVersion` | azd agent vars, then metadata/`agent_get` |
17| `suiteName` | verified `eval.yaml` name or `<agent-name>-smoke` unless user provided one |
18| generation deployment | `model_deployment_get`; choose a chat-completions deployment |
19 
20`suiteName` must start with a letter (`A-Z` or `a-z`). If a derived name starts with a number, prefix it with an alphabetic label such as `suite-`.
21 
22Do not assume `gpt-4o` exists.
23 
24### 2. Reuse or Refresh Cache
25 
26Inspect `.foundry/suites/`, `.foundry/evaluators/`, `.foundry/datasets/`, matching `eval.yaml`, and the selected environment's `evaluationSuites[]` in the selected agent root only. Do **not** merge sibling agent folders.
27 
28- **Suite metadata has `suiteName` and current cache** -> call `evaluation_suite_get` to verify the remote suite, then reuse it.
29- **`eval.yaml` exists and matches the selected agent** -> verify its `dataset.local_uri` or registered `dataset.name`/`dataset.version`, `evaluators[]`, and optional `name` remotely or register them before persisting a synced suite entry. Normalize legacy `dataset_file` in memory only.
30- **Cache is missing/stale or user asks refresh** -> generate a new suite after confirming any overwrite.
31- **Legacy entry without `suiteName`** -> keep it as legacy fallback metadata unless the user approves generating a new suite.
32 
33### 3. Generate Suite
34 
35Read [Evaluation Suite Generation](evaluation-suite-generation.md). If the user selected existing `eval.yaml`, follow the local eval.yaml verification/registration path there before creating a generated suite. Otherwise call:
36 
37```text
38evaluation_suite_generation_job_create(
39  projectEndpoint,
40  suiteName,
41  agentName,
42  generationModelDeploymentName,
43  dataGenerationType,
44  maxSamples
45)
46```
47 
48For trace-informed suites, include `traceAgentName` or `traceAgentId`, `traceAgentVersion`, `traceStartTime`, `traceEndTime`, and `maxTraces`. Start background polling with `evaluation_suite_generation_job_get`, suppress intermediate `in_progress` output, then verify the generated suite with `evaluation_suite_get` after terminal success.
49 
50When refining an existing dataset, include `datasetName` and `datasetVersion`.
51 
52### 4. Persist Local References
53 
54Cache generated artifacts inside the selected root:
55 
56```text
57.foundry/
58  agent-metadata.yaml
59  agent-metadata.prod.yaml
60  suites/<suite-name>-v<version>.json
61  evaluators/<evaluator-name>-v<version>.json
62  datasets/<agent-name>-<dataset-name>-v<version>.ref.json
63  datasets/<dataset-name>-v<version>/<blob-name>
64  results/
65```
66 
67If the job result exposes only remote names/versions, fetch metadata with `evaluation_suite_get(projectEndpoint, suiteName, suiteVersion)`, `evaluation_dataset_get`, `evaluation_dataset_sas_url_get`, and `evaluator_catalog_get`, then materialize the full suite JSON, full evaluator JSON, dataset `.ref.json`, and downloaded dataset blobs. Never overwrite user-edited cache files without confirmation; deterministic re-fetch of the same immutable remote `<name>-v<version>` may replace the generated cache artifact for that exact version.
68 
69### 5. Update Metadata
70 
71Write only the selected metadata file and selected environment. In azd projects, persist only non-derivable overlay/cache state; do not copy azd-owned project endpoint, agent name/version, ACR, or observability values. Persist evaluation suites with:
72 
73- `id`, `tags`, `suiteName`, `suiteVersion`
74- `generationJobId`, `generationSource` (`synthetic`, `traces`, or `manual-fallback`)
75- `dataset`, `datasetVersion`, `datasetFile`, `datasetUri`
76- evaluator `name`, `version`, `threshold`, `definitionFile` (full cached JSON)
77 
78Use tags such as `tier: smoke`, `purpose: baseline`, and `stage: generated`. If metadata still uses older `testSuites[]` or legacy `testCases[]`, replace that list with `evaluationSuites[]` on write and map `priority` to `tags.tier` only when `tags.tier` is missing.
79 
80### 6. Fallback
81 
82If suite generation fails, is unavailable, or returns incomplete artifacts, explain the failure and fall back to the existing manual path: `evaluator_catalog_get`, local seed JSONL generation via [Generate Seed Evaluation Dataset](../../eval-datasets/references/generate-seed-dataset.md), `evaluation_dataset_create`, and `evaluationSuites[]` metadata with `generationSource: manual-fallback`.
83 
84### 7. Prompt User
85 
86Ask: *"Your agent is deployed and the selected environment has evaluation-suite metadata plus local dataset/evaluator references. Would you like to run an evaluation to identify optimization opportunities?"*
87 
88If yes -> proceed to [Step 2: Evaluate](evaluate-step.md). If no -> stop.
89