open/ms-ai-architect
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity Actions

docs(architect): weekly KB update — 52 files refreshed (2026-04)

Browse source 
Key content changes:
- MLOps: MLflow 3 scorers expanded (RetrievalRelevance, Fluency, multi-turn judges)
- MLflow 3 A/B eval: mirror_traffic GA confirmed, new scorer catalog
- CI/CD: OIDC auth replaces deprecated --sdk-auth (Azure ML GitHub Actions)
- Agent framework A2A: updated SDK patterns (A2ACardResolver, BearerAuth)
- AG-UI backend tool rendering: accurate TOOL_CALL_* event shapes
- Computer Use agents: US region requirement, credentials patterns
- Purview governance: bulk term edit, expire/delete workflows
- CAF AI Secure: 3-phase structure confirmed current
- Copilot Studio: Claude Sonnet 4.5/4.6 GA, new orchestration controls
- M365 manifest: v1.26 GA (April 2026), copilotAgents node
- Power Platform: agent flow capacity enforcement corrected
- Azure Monitor: Simple Log Alerts GA, AMBA for policy-based alerting
- Security Copilot: SCU capacity model (400 SCU/1000 users)
- EU Data Boundary: all EU + EFTA countries confirmed
- gateway-multi-backend: added 4th topology, subscription-level quota note

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-04-10 11:31:11 +02:00
commit be4925a8ff
40 changed files with 398 additions and 239 deletions
Download patch file Download diff file Expand all files Collapse all files

15
skills/ms-ai-engineering/references/mlops-genaiops/llm-evaluation-production.md
View file

@ -1,7 +1,7 @@
# LLM Evaluation in Production Contexts
**Kategori:** MLOps & GenAIOps
**Sist oppdatert:** 2026-02-04
**Sist oppdatert:** 2026-04
**Confidence:** High (basert på offisiell Microsoft dokumentasjon, Azure AI Foundry SDK, og MLflow 3)
---
@ -575,15 +575,24 @@ MLflow 3 (SDK `mlflow[databricks]>=3.1`) introduces a unified evaluation model:
| Judge | Needs Ground Truth | Evaluates |
|-------|-------------------|-----------|
| `RelevanceToQuery` | No | Response relevance to user request |
| `RetrievalRelevance` | No | Retrieved context relevance to user request |
| `RetrievalGroundedness` | No | Hallucination detection |
| `Safety` | No | Harmful/toxic content |
| `Correctness` | Yes | Accuracy vs ground truth |
| `Completeness` | Yes | All questions addressed |
| `Fluency` | No | Grammatically correct and naturally flowing |
| `Equivalence` | Yes | Response equivalent to expected output |
| `RetrievalSufficiency` | Yes | Context provides all necessary information |
| `ToolCallCorrectness` | Yes | Tool calls and arguments |
| `ToolCallEfficiency` | No | Redundant tool usage |
| `Guidelines` | No | Custom natural-language rules |
| `ExpectationsGuidelines` | No (needs guidelines in expectations) | Per-example natural-language criteria |
**Multi-turn judges** (conversation-level): `ConversationCompleteness`, `UserFrustration`, `KnowledgeRetention`, `ConversationalSafety`
Verified (MCP 2026-04)
**Multi-turn judges** (conversation-level): `ConversationCompleteness`, `UserFrustration`, `KnowledgeRetention`, `ConversationalSafety`, `ConversationalGuidelines`, `ConversationalRoleAdherence`, `ConversationalToolCallEfficiency`
Verified (MCP 2026-04)
**Production monitoring**: Automatically runs scorers on production traces; uses Databricks-hosted LLM judges (EU workspaces: EU-hosted models). No prompts stored with Azure OpenAI (Abuse Monitoring opt-out).
@ -1088,7 +1097,7 @@ Production evaluation er ikke komplett uten human review loop. Anbefal:
- Power Platform evaluation gaps (product evolves rapidly)
- Human feedback loop implementation (no single canonical pattern)
**Ufullstendig informasjon (per feb 2026):**
**Ufullstendig informasjon (per april 2026):**
- Native Copilot Studio production evaluation features (roadmap item, not released)
- Detailed pricing for Azure AI Content Safety evaluators (bundled pricing, not per-call transparent)