docs(architect): weekly KB update — 52 files refreshed (2026-04)
Key content changes: - MLOps: MLflow 3 scorers expanded (RetrievalRelevance, Fluency, multi-turn judges) - MLflow 3 A/B eval: mirror_traffic GA confirmed, new scorer catalog - CI/CD: OIDC auth replaces deprecated --sdk-auth (Azure ML GitHub Actions) - Agent framework A2A: updated SDK patterns (A2ACardResolver, BearerAuth) - AG-UI backend tool rendering: accurate TOOL_CALL_* event shapes - Computer Use agents: US region requirement, credentials patterns - Purview governance: bulk term edit, expire/delete workflows - CAF AI Secure: 3-phase structure confirmed current - Copilot Studio: Claude Sonnet 4.5/4.6 GA, new orchestration controls - M365 manifest: v1.26 GA (April 2026), copilotAgents node - Power Platform: agent flow capacity enforcement corrected - Azure Monitor: Simple Log Alerts GA, AMBA for policy-based alerting - Security Copilot: SCU capacity model (400 SCU/1000 users) - EU Data Boundary: all EU + EFTA countries confirmed - gateway-multi-backend: added 4th topology, subscription-level quota note Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
6645e93205
commit
be4925a8ff
40 changed files with 398 additions and 239 deletions
|
|
@ -1,6 +1,6 @@
|
|||
# A/B Testing and Experimentation for AI Models
|
||||
|
||||
**Last updated:** 2026-02
|
||||
**Last updated:** 2026-04
|
||||
**Verified:** MCP 2026-04
|
||||
**Status:** GA
|
||||
**Category:** MLOps & GenAIOps
|
||||
|
|
@ -455,14 +455,15 @@ az ml online-endpoint update --name my-endpoint --traffic control=90 challenger=
|
|||
# Use RelevanceToQuery, Correctness, custom business scorers
|
||||
```
|
||||
|
||||
**MLflow 3 A/B evaluation pattern**:
|
||||
**MLflow 3 A/B evaluation pattern** — Verified (MCP 2026-04):
|
||||
- Use `mlflow.genai.evaluate()` on traces from each variant
|
||||
- Compare scorers: `Correctness`, `RelevanceToQuery`, `ToolCallEfficiency`
|
||||
- Compare scorers: `Correctness`, `RelevanceToQuery`, `RetrievalGroundedness`, `ToolCallEfficiency`, `Fluency` — expanded scorer set in MLflow 3
|
||||
- Multi-turn scorers available: `ConversationCompleteness`, `UserFrustration` for conversational AI A/B testing
|
||||
- Statistical significance: MLflow tracks Cohen's Kappa against human baseline
|
||||
- Aliases in Prompt Registry: `@control` and `@challenger` for prompt A/B testing
|
||||
|
||||
**Azure ML safe rollout progression**:
|
||||
1. **Shadow testing**: Mirror X% of traffic to new model (no user impact)
|
||||
**Azure ML safe rollout progression** — Verified (MCP 2026-04):
|
||||
1. **Shadow testing**: Mirror X% of traffic to new model (no user impact) — natively supported via `mirror_traffic` property on managed online endpoints
|
||||
2. **Canary**: Route 10% live traffic, monitor bake time (hours/days)
|
||||
3. **Progressive**: 10% → 50% → 100% with health gate at each step
|
||||
4. **Rollback trigger**: Automatic halt on health signal degradation
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue