docs(architect): weekly KB update — 52 files refreshed (2026-04)

Key content changes:
- MLOps: MLflow 3 scorers expanded (RetrievalRelevance, Fluency, multi-turn judges)
- MLflow 3 A/B eval: mirror_traffic GA confirmed, new scorer catalog
- CI/CD: OIDC auth replaces deprecated --sdk-auth (Azure ML GitHub Actions)
- Agent framework A2A: updated SDK patterns (A2ACardResolver, BearerAuth)
- AG-UI backend tool rendering: accurate TOOL_CALL_* event shapes
- Computer Use agents: US region requirement, credentials patterns
- Purview governance: bulk term edit, expire/delete workflows
- CAF AI Secure: 3-phase structure confirmed current
- Copilot Studio: Claude Sonnet 4.5/4.6 GA, new orchestration controls
- M365 manifest: v1.26 GA (April 2026), copilotAgents node
- Power Platform: agent flow capacity enforcement corrected
- Azure Monitor: Simple Log Alerts GA, AMBA for policy-based alerting
- Security Copilot: SCU capacity model (400 SCU/1000 users)
- EU Data Boundary: all EU + EFTA countries confirmed
- gateway-multi-backend: added 4th topology, subscription-level quota note

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-04-10 11:31:11 +02:00
commit be4925a8ff
40 changed files with 398 additions and 239 deletions

View file

@ -1,6 +1,6 @@
# A/B Testing and Experimentation for AI Models
**Last updated:** 2026-02
**Last updated:** 2026-04
**Verified:** MCP 2026-04
**Status:** GA
**Category:** MLOps & GenAIOps
@ -455,14 +455,15 @@ az ml online-endpoint update --name my-endpoint --traffic control=90 challenger=
# Use RelevanceToQuery, Correctness, custom business scorers
```
**MLflow 3 A/B evaluation pattern**:
**MLflow 3 A/B evaluation pattern** — Verified (MCP 2026-04):
- Use `mlflow.genai.evaluate()` on traces from each variant
- Compare scorers: `Correctness`, `RelevanceToQuery`, `ToolCallEfficiency`
- Compare scorers: `Correctness`, `RelevanceToQuery`, `RetrievalGroundedness`, `ToolCallEfficiency`, `Fluency` — expanded scorer set in MLflow 3
- Multi-turn scorers available: `ConversationCompleteness`, `UserFrustration` for conversational AI A/B testing
- Statistical significance: MLflow tracks Cohen's Kappa against human baseline
- Aliases in Prompt Registry: `@control` and `@challenger` for prompt A/B testing
**Azure ML safe rollout progression**:
1. **Shadow testing**: Mirror X% of traffic to new model (no user impact)
**Azure ML safe rollout progression** — Verified (MCP 2026-04):
1. **Shadow testing**: Mirror X% of traffic to new model (no user impact) — natively supported via `mirror_traffic` property on managed online endpoints
2. **Canary**: Route 10% live traffic, monitor bake time (hours/days)
3. **Progressive**: 10% → 50% → 100% with health gate at each step
4. **Rollback trigger**: Automatic halt on health signal degradation