1# Step 6 — CI/CD Evals & Continuous Production Monitoring
2 
3After confirming the final agent version through the observe loop, present two complementary monitoring options. The user may choose one, both, or neither.
4 
5## Option 1 — CI/CD Pipeline Evaluations (Pre-Deploy Gate)
6 
7*"Would you like to add automated evaluations to your CI/CD pipeline so every deployment is evaluated before going live?"*
8 
9CI/CD evals run batch evaluations as part of your deployment pipeline, catching regressions **before** they reach production.
10 
11If yes, generate a GitHub Actions workflow (for example, `.github/workflows/agent-eval.yml`) that:
12 
131. Triggers on push to `main` or on pull request
142. Accepts a metadata-file input or environment variable such as `FOUNDRY_METADATA_FILE` and defaults it to `.foundry/agent-metadata.yaml`
153. Reads evaluation-suite definitions from the selected metadata file (for example, `.foundry/agent-metadata.prod.yaml` for prod CI)
164. Reads evaluator definitions from `.foundry/evaluators/` and test datasets from `.foundry/datasets/`
175. Runs `evaluation_agent_batch_eval_create` against the newly deployed agent version
186. Fails the workflow if any evaluator score falls below the configured thresholds for the environment and evaluation suite resolved from that metadata file
197. Posts a summary as a PR comment or workflow annotation
20 
21Use repository secrets for the selected environment's project endpoint and Azure credentials, and keep the metadata filename explicit in the workflow so prod rollouts do not depend on the local/dev default file. Confirm the workflow file with the user before committing.
22 
23## Option 2 — Continuous Production Monitoring (Post-Deploy)
24 
25*"Would you like to set up continuous evaluations to monitor your agent's quality in production?"*
26 
27Continuous evaluation uses Foundry-native MCP tools to automatically assess agent responses on an ongoing basis — no additional CI/CD pipeline setup is needed for this option. This catches regressions that emerge **after** deployment from changing data, user patterns, or upstream service drift.
28 
29### Enable Continuous Evaluation
30 
31Use the [continuous evaluation reference](continuous-eval.md) to configure monitoring. The workflow:
32 
331. **Check existing config** — call `continuous_eval_get` to see if monitoring is already active.
342. **Select evaluators** — recommend starting with the same evaluators used in batch evals for consistent comparison:
35   - **Quality evaluators** (require `deploymentName`): e.g., groundedness, coherence, relevance, task_adherence
36   - **Safety evaluators**: e.g., violence, indirect_attack, hate_unfairness
373. **Enable** — call `continuous_eval_create` with the selected evaluators. The tool auto-detects agent kind and configures the appropriate backend (real-time for prompt agents, scheduled for hosted agents).
384. **Confirm** — present the returned configuration to the user.
39 
40### Acting on Monitoring Results
41 
42Monitoring is only complete when score drops trigger investigation and remediation.
43 
44For instructions on how to read evaluation scores, triage regressions, and verify fixes, see [Acting on Results](continuous-eval.md#acting-on-results).
45 
46The observe loop does not end at deployment. Continuous monitoring closes the loop: **observe → optimize → deploy → monitor → observe**. Always offer to set up monitoring after completing an optimization cycle.
47 
48## Reference
49 
50- [Azure AI Foundry Cloud Evaluation](https://learn.microsoft.com/en-us/azure/ai-foundry/how-to/develop/cloud-evaluation)
51- [Hosted Agents](https://learn.microsoft.com/en-us/azure/ai-foundry/agents/concepts/hosted-agents)
52- [Continuous Evaluation Reference](continuous-eval.md)
53