docs(architect): weekly KB update — 106 files refreshed (2026-04)
Updates across all 5 skills: ms-ai-advisor, ms-ai-engineering, ms-ai-governance, ms-ai-security, ms-ai-infrastructure. Key changes: - Language Services (Custom Text Classification, Text Analytics, QnA): retirement warning 2029-03-31, migration guides to Foundry/GPT-4o - Agentic Retrieval: 50M free reasoning tokens/month (Public Preview) - Computer Use: Claude Sonnet 4.5 (preview) + OpenAI CUA models - Agent Registry: Risks column (M365 E7), user-shared/org-published types - Declarative agents: schema v1.5 → v1.6, Store validation requirements - MLflow 3: 13 built-in LLM judges, production monitoring, Genie Code - AG-UI HITL: ApprovalRequiredAIFunction (C#) + @tool(approval_mode) (Python) - Entra ID Ignite 2025: Agent ID Admin/Developer RBAC roles, Conditional Access - Security Copilot: 400 SCU/month per 1000 M365 E5 licenses, auto-provisioned - Fast Transcription API: phrase lists, 14-language multi-lingual transcription - Azure Monitor Workbooks: Bicep support, RBAC specifics - Power Platform Copilot: data residency (Norway/Europe → EU DB, Bing → USA) - RAG security-rbac: 4-approach table (GA + 3 preview access control methods) - IaC MLOps: Well-Architected OE:05 principles, Bicep/Terraform patterns - Translator: image file batch translation Preview (JPEG/PNG/BMP/WebP) All 106 files: Last updated 2026-04 | Verified: MCP 2026-04 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
0eb30fa853
commit
6645e93205
104 changed files with 1986 additions and 520 deletions
|
|
@ -1,6 +1,6 @@
|
|||
# Agent Evaluation and Testing Frameworks
|
||||
|
||||
**Last updated:** 2026-02
|
||||
**Last updated:** 2026-04 | Verified: MCP 2026-04
|
||||
**Status:** GA (Azure AI Evaluation SDK), Preview (Agent-specific evaluators)
|
||||
**Category:** Agent Orchestration & Automation
|
||||
|
||||
|
|
@ -276,6 +276,33 @@ evaluator = TaskAdherenceEvaluator(
|
|||
- **KQL queries:** Flexible querying av evaluation metrics over tid
|
||||
- **Alerts:** Sett opp alerts hvis pass rate dropper under threshold
|
||||
|
||||
### MLflow 3 (Databricks / Cross-platform)
|
||||
|
||||
MLflow 3 tilbyr komprehensiv GenAI-evaluering for agenter paa tvers av plattformer:
|
||||
|
||||
| Feature | Beskrivelse |
|
||||
|---------|-------------|
|
||||
| **Built-in LLM judges** | Innebygde dommere for kvalitetsmetrikker (relevance, groundedness, safety, etc.) |
|
||||
| **Custom scorers** | Definer egne kvalitetsmetrikker med Python-funksjoner |
|
||||
| **Eval harness** | Test GenAI-app mot eval-datasett under utvikling; sammenlign appversjoner |
|
||||
| **Conversation evaluation** | Vurder multi-turn samtalekvaltiet (completeness, user frustration, dialogue coherence) |
|
||||
| **Conversation simulation** | Generer syntetiske multi-turn samtaler for testing |
|
||||
| **Production monitoring** | Kjoer scorers og judges paa produksjons-traces automatisk (Beta) |
|
||||
| **Review App** | Samle ekspertfeedback og bygg eval-datasett |
|
||||
|
||||
MLflow Tracing gir real-time trace logging gjennom hele livssyklusen. Samme judges og scorers kan brukes i baade development og produksjon — konsistent evaluering.
|
||||
|
||||
```python
|
||||
# MLflow 3 evaluation eksempel
|
||||
import mlflow
|
||||
|
||||
results = mlflow.genai.evaluate(
|
||||
data=eval_dataset,
|
||||
predict_fn=my_agent,
|
||||
scorers=[mlflow.genai.scorers.groundedness(), mlflow.genai.scorers.safety()]
|
||||
)
|
||||
```
|
||||
|
||||
### Prompt Flow
|
||||
|
||||
- **Evaluation flows:** Custom evaluation logic som Prompt Flow (deprecated approach — bruk Azure AI Evaluation SDK i stedet)
|
||||
|
|
@ -488,7 +515,7 @@ evaluator = TaskAdherenceEvaluator(
|
|||
|
||||
9. **Evaluate and monitor AI agents (MLflow 3 on Databricks)**
|
||||
https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/eval-monitor/
|
||||
*Confidence: Verified* — MLflow-based evaluation for cross-platform agents
|
||||
*Confidence: Verified* — MLflow 3 GenAI evaluation: built-in LLM judges og scorers, eval-harness for development, production monitoring (Beta), conversation evaluation (multi-turn), conversation simulation, Review App for human feedback, Genie Code for observability; integrert med MLflow Tracing paa tvers av development/test/produksjon; oppdatert 2026-04
|
||||
|
||||
10. **Run automated tests for agent quality and reliability (Copilot Studio)**
|
||||
https://learn.microsoft.com/en-us/power-platform/release-plan/2025wave1/microsoft-copilot-studio/run-automated-tests-agent-quality-reliability
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue