docs(architect): weekly KB update — 52 files refreshed (2026-04)

Key content changes:
- MLOps: MLflow 3 scorers expanded (RetrievalRelevance, Fluency, multi-turn judges)
- MLflow 3 A/B eval: mirror_traffic GA confirmed, new scorer catalog
- CI/CD: OIDC auth replaces deprecated --sdk-auth (Azure ML GitHub Actions)
- Agent framework A2A: updated SDK patterns (A2ACardResolver, BearerAuth)
- AG-UI backend tool rendering: accurate TOOL_CALL_* event shapes
- Computer Use agents: US region requirement, credentials patterns
- Purview governance: bulk term edit, expire/delete workflows
- CAF AI Secure: 3-phase structure confirmed current
- Copilot Studio: Claude Sonnet 4.5/4.6 GA, new orchestration controls
- M365 manifest: v1.26 GA (April 2026), copilotAgents node
- Power Platform: agent flow capacity enforcement corrected
- Azure Monitor: Simple Log Alerts GA, AMBA for policy-based alerting
- Security Copilot: SCU capacity model (400 SCU/1000 users)
- EU Data Boundary: all EU + EFTA countries confirmed
- gateway-multi-backend: added 4th topology, subscription-level quota note

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-04-10 11:31:11 +02:00
commit 34c6db36fa
40 changed files with 398 additions and 239 deletions

View file

@ -45,7 +45,7 @@ Administratorer har 11 lifecycle management actions tilgjengelig i Admin Center:
| **Delete** | Permanent sletting (inkludert SharePoint Embedded containers) | Irreversibel cleanup (24t propagation) |
| **Approve Updates** | Godkjenn nye versjoner før deployment | Change management |
| **Manage Ownerless Agents** | Handling på agenter uten eier | Compliance og sikkerhet |
| **Reassign** | Tildel ny eier til ownerless/active agents | Kontinuitet |
| **Reassign** | Tildel ny eier til ownerless/active agents. Kun støttet for Agent Builder-agenter. Ny eier får full edit/delete-tilgang og tilgang til opplastede filer; forrige eier mister ALL tilgang inkl. lesetilgang. *(Verified MCP 2026-04)* | Kontinuitet |
| **Export Inventory** | Last ned full agent-liste (Excel) | Audit og rapportering |
**Verified (Microsoft Learn, 2026-02)**
@ -198,7 +198,7 @@ For agenter bygget i Agent Builder med embedded files (knowledge sources):
| Feil | Konsekvens | Løsning |
|------|------------|---------|
| Sletter SharePoint Embedded containers manuelt | Agent-functionality breaks | Aldri slett containers i SharePoint admin center |
| Blokkerer Microsoft-pinned agents (Researcher/Analyst) | Blokkerer for HELE tenant (kan ikke scope) | Bruk extensibility settings istedenfor Block |
| Blokkerer Researcher/Analyst feil | Edit users-panelet er deaktivert for disse agentene; de kan kun blokkeres for hele tenant. Scope til enkeltbrukere er ikke mulig. *(Verified MCP 2026-04)* | Bruk Block-action i Admin Center for hele tenant; Work-access styres separat via admin-innstillinger |
| Glemmer å approve agent updates | Brukere får ikke nye features/bugfixes | Sett opp notification for pending approvals |
| Ingen policy template ved aktivering | Agents opererer uten governance controls | Alltid bruk minimum Default Template |
@ -354,7 +354,7 @@ New-MgIdentityGovernanceLifecycleWorkflow -BodyParameter $params
## Kilder og verifisering
### Microsoft Learn (Verified, 2026-02)
- [Agent Registry i Microsoft 365 Admin Center](https://learn.microsoft.com/en-us/microsoft-365/admin/manage/agent-registry) **Confidence: Verified (2026-04)** — Oppdatert: Risks column (M365 E7), ownerless agent management, Researcher with Computer Use admin configuration, sensitivity labels for embedded files, GraphAPI for Agent Registry (preview)
- [Agent Registry i Microsoft 365 Admin Center](https://learn.microsoft.com/en-us/microsoft-365/admin/manage/agent-registry) **Confidence: Verified (2026-04)** — Oppdatert: Risks column (M365 E7), ownerless agent management, Researcher with Computer Use admin configuration, sensitivity labels for embedded files, GraphAPI for Agent Registry (preview), Reassign kun for Agent Builder-agenter, Application/Delegated permissions-tab i agent details
- [Microsoft 365 Copilot Agents Deployment Blueprint](https://learn.microsoft.com/en-us/copilot/microsoft-365/agent-essentials/m365-agents-blueprint) **Confidence: Verified**
- [Copilot Control System Management Controls](https://learn.microsoft.com/en-us/copilot/microsoft-365/copilot-control-system/management-controls) **Confidence: Verified**
- [Microsoft Entra Agent ID and Agent Identity Platform](https://learn.microsoft.com/en-us/microsoft-agent-365/admin/capabilities-entra) **Confidence: Verified**
@ -367,5 +367,5 @@ New-MgIdentityGovernanceLifecycleWorkflow -BodyParameter $params
- **Kostnadsoptimalisering** Baseline (generelle prinsipper, ikke produkt-spesifikke priser fra Microsoft Learn)
- **Modenhetsnivå-anbefalinger** Baseline (syntetisert fra Microsoft Maturity Framework-prinsipper)
**Total MCP calls:** 3 (microsoft_docs_search x3, microsoft_docs_fetch x2, microsoft_code_sample_search x1)
**Total MCP calls:** 4 (microsoft_docs_search x3, microsoft_docs_fetch x3, microsoft_code_sample_search x1)
**Unique URLs:** 7 Microsoft Learn-artikler

View file

@ -172,11 +172,20 @@ A2A skiller mellom **meldinger** (messages) for rask, synkron kommunikasjon, og
```python
import asyncio
import httpx
from a2a.client import A2ACardResolver
from agent_framework.a2a import A2AAgent
async def main():
a2a_host = "https://agents.nav.no/saksbehandler/a2a"
# Discover remote agent capabilities via AgentCard
async with httpx.AsyncClient(timeout=60.0) as http_client:
resolver = A2ACardResolver(httpx_client=http_client, base_url=a2a_host)
agent_card = await resolver.get_agent_card() # Verified MCP 2026-04
# Koble til ekstern A2A-agent
async with A2AAgent(name="saksbehandler", url="https://agents.nav.no/saksbehandler/a2a") as agent:
async with A2AAgent(name=agent_card.name, agent_card=agent_card, url=a2a_host) as agent:
# Synkron streaming
async with agent.run("Hva er min dagpengesats?", stream=True) as stream:
@ -586,6 +595,25 @@ Hver etat eier og drifter sin egen agent. Felles inngangsagent orkestrerer via A
## Installasjon og SDK-er
```python
# Autentisert A2A-kall (AuthInterceptor-mønster) — Verified MCP 2026-04
from a2a.client.auth.interceptor import AuthInterceptor
class BearerAuth(AuthInterceptor):
def __init__(self, token: str):
self.token = token
async def intercept(self, request):
request.headers["Authorization"] = f"Bearer {self.token}"
return request
async with A2AAgent(
name="secure-agent",
url="https://secure-a2a-agent.example.com",
auth_interceptor=BearerAuth("your-token"),
) as agent:
response = await agent.run("Hello!")
```
```bash
# Python — Agent Framework
pip install agent-framework-a2a --pre
@ -603,13 +631,15 @@ pip install microsoft-teams-a2a
**.NET (Semantic Kernel):**
```csharp
// Agent card tilgjengelig på: GET /a2a/my-agent/v1/card
// Message endpoint: POST /a2a/my-agent/v1/message:stream
app.MapA2A(agent, "/a2a/my-agent", agentCard: new()
{
Name = "Min Agent",
Description = "Hjelpsom assistent for norsk offentlig sektor",
Version = "1.0",
Capabilities = new() { Streaming = true }
});
}); // Verified MCP 2026-04
```
---
@ -638,9 +668,9 @@ app.MapA2A(agent, "/a2a/my-agent", agentCard: new()
- https://learn.microsoft.com/azure/api-management/agent-to-agent-api
- Confidence: **Verified** (APIM preview-støtte, februar 2026)
6. **Agent Framework — A2A Integration (Python)**
6. **Agent Framework — A2A Integration (Python og C#)**
- https://learn.microsoft.com/agent-framework/integrations/a2a
- Confidence: **Verified** (offisiell SDK-dokumentasjon, februar 2026)
- Confidence: **Verified (MCP 2026-04)** — A2ACardResolver-pattern (Python), A2AAgent med agent_card-parameter, AuthInterceptor for sikret kall, MapA2A /v1/card og /v1/message:stream endepunkt-paths (.NET), NuGet-pakker Microsoft.Agents.AI.Hosting.A2A og .AspNetCore
7. **Semantic Kernel Agent Orchestration**
- https://learn.microsoft.com/semantic-kernel/frameworks/agent/agent-orchestration/

View file

@ -107,6 +107,8 @@ if "pending_safety_checks" in response:
Copilot Studio tilbyr CUA som et lavkode **Computer Use Tool** — ingen koding nødvendig.
**Krav (preview):** Tilgjengelig kun for miljøer der regionen er satt til **United States**. Generativ orkestrering (generative orchestrator) må aktiveres på agenten. *(Verified MCP 2026-04)*
### Oppsett
1. Gå til **Tools** i agenten → **Add tool****New tool** → **Computer use**
@ -125,9 +127,9 @@ Copilot Studio tilbyr CUA som et lavkode **Computer Use Tool** — ingen koding
|---------------|-------------|
| **Maker-provided credentials** | Agenten bruker makerens innloggingsinfo (for autonome agenter) |
| **End user credentials** | Brukeren logger inn selv (for konversasjonelle agenter) |
| **Intern Power Platform-lagring** | Kryptert intern lagring — ingen forhåndskonfigurasjon nødvendig |
| **Azure Key Vault** | Passord lagres i Key Vault — anbefalt for produksjonsmiljøer |
| **Access control** | Begrens hvilke nettsider/applikasjoner CUA kan operere på |
| **Intern Power Platform-lagring** | Kryptert intern lagring — ingen forhåndskonfigurasjon nødvendig. Oppgi URL/app-navn + brukernavn + passord; wildcard (*) støttes for subdomener (f.eks. *.contoso.com). *(Verified MCP 2026-04)* |
| **Azure Key Vault** | Passord lagres i Key Vault — anbefalt for produksjonsmiljøer. Krever PowerPlatform resource provider registrert i Azure-abonnementet. *(Verified MCP 2026-04)* |
| **Access control** | Begrens hvilke nettsider/applikasjoner CUA kan operere på. Wildcards støttes (*.contoso.com). Desktop apps angis ved produkt-/prosessnavn (f.eks. "Microsoft Edge" eller "msedge"). *(Verified MCP 2026-04)* |
### Lisensiering (Copilot Studio, preview)
@ -485,7 +487,7 @@ Kostnader basert på:
2. **Automate web and desktop apps with computer use — Copilot Studio**
- https://learn.microsoft.com/microsoft-copilot-studio/computer-use
- Confidence: **Verified** (offisiell Copilot Studio preview-dokumentasjon, oppdatert 2026-04: støttede modeller, credentials, access control-semantikk)
- Confidence: **Verified** (offisiell Copilot Studio preview-dokumentasjon, oppdatert 2026-04: støttede modeller, credentials intern/Key Vault-detaljer, access control wildcard/desktop, US-only region-krav, generativ orkestrering påkrevd)
3. **Configure where computer use runs**
- https://learn.microsoft.com/microsoft-copilot-studio/configure-where-computer-use-runs

View file

@ -209,32 +209,55 @@ var result = await mainAgent.RunAsync("Hvordan er været i Oslo?");
AG-UI backend tool rendering stoetter HITL via to mekanismer:
**C# - ApprovalRequiredAIFunction:**
**C# - AIFunctionFactory med serializerOptions (Verified MCP 2026-04):**
```csharp
// Tool som krever human approval
var approvalTool = ApprovalRequiredAIFunction.Create(DeleteRecord);
// Workflow emitter RequestInfoEvent med ToolApprovalRequestContent
await foreach (var evt in workflow.WatchStreamAsync()) {
if (evt is RequestInfoEvent req && req.Data is ToolApprovalRequestContent tc) {
bool approved = await AskUserApproval(tc.ToolName);
await handle.SendResponseAsync(req.Request.CreateResponse(approved));
}
// Definer tool med Description-attributter
[Description("Search for restaurants in a location.")]
static RestaurantSearchResponse SearchRestaurants(
[Description("The restaurant search request")] RestaurantSearchRequest request)
{
// implementasjon
}
// Registrer tool - NB: serializerOptions PÅKREVD for komplekse typer
var jsonOptions = app.Services.GetRequiredService<IOptions<JsonOptions>>().Value;
AITool[] tools = [
AIFunctionFactory.Create(SearchRestaurants, serializerOptions: jsonOptions.SerializerOptions)
];
// FunctionCallContent og FunctionResultContent streames til klient
// FunctionCallContent: .Name, .Arguments (key-value pairs)
// FunctionResultContent: .CallId, .Result eller .Exception
```
**Python - @tool med approval_mode:**
**Python - @tool decorator (Verified MCP 2026-04):**
```python
@tool(approval_mode="always_require")
def delete_record(record_id: str) -> str:
# Sletter en post - krever alltid menneskelig godkjenning
return db.delete(record_id)
from typing import Annotated
from pydantic import Field
from agent_framework import tool
# Workflow pauser og emitter function_approval_request event
# Klient-loop maa haandtere og respondere
@tool
def get_weather(
location: Annotated[str, Field(description="The city")],
) -> str:
"""Get the current weather for a location."""
return f"The weather in {location} is 22 degrees C."
# Klasse-baserte tools for gruppering
class WeatherTools:
@tool
def get_current_weather(self, location: Annotated[str, Field(description="City")]) -> str:
"""Get current weather."""
return f"Current weather in {location}: Sunny"
```
**Backend tool events streames til klient i sanntid:** TOOL_CALL_START, TOOL_CALL_ARGS, TOOL_CALL_END, TOOL_CALL_RESULT.
**Backend tool events streames til klient i sanntid (Verified MCP 2026-04):**
```json
{"type": "TOOL_CALL_START", "toolCallId": "call_abc123", "toolCallName": "get_weather"}
{"type": "TOOL_CALL_ARGS", "toolCallId": "call_abc123", "delta": "{"location": "Oslo"}"}
{"type": "TOOL_CALL_END", "toolCallId": "call_abc123"}
{"type": "TOOL_CALL_RESULT","toolCallId": "call_abc123", "content": "The weather in Oslo is 22C."}
```
---
@ -428,7 +451,7 @@ def update_citizen_record(ssn: str, field: str, value: str) -> str:
1. [Azure OpenAI Function Calling](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/function-calling) — **Verified 2026-02**
2. [Semantic Kernel Agent Functions](https://learn.microsoft.com/en-us/semantic-kernel/frameworks/agent/agent-functions) — **Verified 2026-02**
3. [Agent Framework - Agent as Function Tool](https://learn.microsoft.com/en-us/agent-framework/tutorials/agents/agent-as-function-tool) — **Verified 2026-02**
4. [AG-UI Backend Tool Rendering](https://learn.microsoft.com/en-us/agent-framework/integrations/ag-ui/backend-tool-rendering) — **Verified 2026-04** (backend tool streaming, ApprovalRequiredAIFunction C#, @tool(approval_mode) Python, TOOL_CALL_* events)
4. [AG-UI Backend Tool Rendering](https://learn.microsoft.com/en-us/agent-framework/integrations/ag-ui/backend-tool-rendering) — **Verified (MCP 2026-04)** — AIFunctionFactory.Create() med serializerOptions for komplekse typer (C#), @tool decorator med Annotated/Field (Python), TOOL_CALL_START/ARGS/END/RESULT events, FunctionCallContent/.Arguments og FunctionResultContent/.Result (C#), klasse-baserte tools-moenster (Python)
5. [Azure OpenAI Assistants Function Calling](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/assistant-functions) — **Verified 2026-02**
6. [Structured Outputs](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/structured-outputs) — **Verified 2026-02**

View file

@ -549,7 +549,7 @@ TOTAL: ~46 700 NOK/måned (høyere cost, men forutsigbar)
3. [Baseline Foundry Chat Architecture (Foundry Agent Service + Microsoft Agent Framework)](https://learn.microsoft.com/en-us/azure/architecture/ai-ml/architecture/baseline-microsoft-foundry-chat) — Verified (MCP 2026-04)
4. [Azure API Management - AI Gateway Capabilities](https://learn.microsoft.com/en-us/azure/api-management/genai-gateway-capabilities)
5. [Reliability in Azure AI Search](https://learn.microsoft.com/en-us/azure/reliability/reliability-ai-search)
6. [Multi-Backend Gateway Guide](https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/azure-openai-gateway-multi-backend) — Verified MCP 2026-04: Dokumentet bekrefter fire gateway-topologier (single instance/multiple deployments, multi-instance same region/subscription, multi-instance multi-region). Tagger nå eksplisitt "Foundry Tools" og "Azure OpenAI in Foundry Models".
6. [Multi-Backend Gateway Guide](https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/azure-openai-gateway-multi-backend) — Verified MCP 2026-04: Dokumentet bekrefter fire gateway-topologier: (1) multiple model deployments i single instance, (2) multiple instances same region/single subscription, (3) multiple instances same region/multiple subscriptions (eksplisitt som egen topologi), (4) multiple instances multi-region. Tagger eksplisitt "Foundry Tools" og "Azure OpenAI in Foundry Models". Anbefaler sterkt credential termination og reestablishment ved gateway fremfor pass-through client credentials. Gateway muliggjør client-based usage tracking for chargeback-modeller. Verified (MCP 2026-04)
7. [Load Balancing Options - Azure Architecture](https://learn.microsoft.com/en-us/azure/architecture/guide/technology-choices/load-balancing-overview)
**GitHub Samples (Microsoft-verified):**

View file

@ -231,7 +231,7 @@ Purview Data Owner Policies muliggjør sentralisert tilgangsstyring:
### Governance Domains og OKR-er
Governance Domains er nå den sentrale organiseringsenhet for glossary terms i Unified Catalog. Workflow: opprett term (Draft) → rediger → publiser. Glossary terms kan flyttes mellom domains (begge domains krever Data Steward-rolle). Termer kan linkes til data products og critical data elements på tvers av domains. *(Verified MCP 2026-04)*
Governance Domains er nå den sentrale organiseringsenhet for glossary terms i Unified Catalog. Workflow: opprett term (Draft) → rediger → publiser. Governance domain MÅ publiseres FØR terms publiseres. Termer kan linkes til data products og critical data elements på tvers av domains. Bulk edit opptil 50 terms (kun Draft-state). Flytt terms mellom domains krever Data Steward-rolle i BEGGE domains; parent-term drar med seg child-terms. Expire-funksjon gjør termen usynlig for alle unntatt Data Stewards og Domain Owners. For å slette: unpublish → fjern alle lenker → delete. *(Verified MCP 2026-04)*
```
Governance Domain: "AI og Maskinlæring"
@ -346,7 +346,7 @@ Microsoft Purview gir nå governance-dekning for Fabric Copilots og agenter —
- [How to get lineage from Microsoft Fabric items into Microsoft Purview](https://learn.microsoft.com/en-us/purview/data-map-lineage-fabric) -- Lineage fra Fabric
- [Data lineage in classic Data Catalog](https://learn.microsoft.com/en-us/purview/data-gov-classic-lineage) -- Lineage-konsepter
- [Learn about sensitivity labels in Data Map](https://learn.microsoft.com/en-us/purview/data-map-sensitivity-labels) -- Sensitivitetsmerking
- [Create and manage glossary terms](https://learn.microsoft.com/en-us/purview/unified-catalog-glossary-terms-create-manage) -- Business glossary *(Verified MCP 2026-04)*Ny funksjonalitet: bulk edit opptil 50 terms, flytt terms mellom governance domains, custom attributes med filter, Data Steward-rolle påkrevd for opprettelse. Terms opprettes i Draft state, må publiseres for å bli synlige. Governance domain MÅ publiseres FØR terms publiseres.
- [Create and manage glossary terms](https://learn.microsoft.com/en-us/purview/unified-catalog-glossary-terms-create-manage) -- Business glossary *(Verified MCP 2026-04)*Bulk edit opptil 50 terms (Draft-state), flytt terms mellom governance domains (Data Steward i begge domains kreves), custom attribute-filter i Enterprise glossary, Expire-workflow, Delete-workflow (unpublish + fjern lenker → delete). Governance domain MÅ publiseres FØR terms publiseres. Parent-term drar med seg child-terms ved flytting. Related critical data elements kan linkes på tvers av domains.
- [Glossary terms in Unified Catalog](https://learn.microsoft.com/en-us/purview/unified-catalog-glossary-terms) -- Aktive glossary-termer
- [Learn about Microsoft Purview Unified Catalog](https://learn.microsoft.com/en-us/purview/unified-catalog) -- Oversikt over Unified Catalog
- [Set up data quality for Fabric Lakehouse data](https://learn.microsoft.com/en-us/purview/unified-catalog-data-quality-fabric-lakehouse) -- Datakvalitet for Fabric

View file

@ -1,6 +1,6 @@
# A/B Testing and Experimentation for AI Models
**Last updated:** 2026-02
**Last updated:** 2026-04
**Verified:** MCP 2026-04
**Status:** GA
**Category:** MLOps & GenAIOps
@ -455,14 +455,15 @@ az ml online-endpoint update --name my-endpoint --traffic control=90 challenger=
# Use RelevanceToQuery, Correctness, custom business scorers
```
**MLflow 3 A/B evaluation pattern**:
**MLflow 3 A/B evaluation pattern** — Verified (MCP 2026-04):
- Use `mlflow.genai.evaluate()` on traces from each variant
- Compare scorers: `Correctness`, `RelevanceToQuery`, `ToolCallEfficiency`
- Compare scorers: `Correctness`, `RelevanceToQuery`, `RetrievalGroundedness`, `ToolCallEfficiency`, `Fluency` — expanded scorer set in MLflow 3
- Multi-turn scorers available: `ConversationCompleteness`, `UserFrustration` for conversational AI A/B testing
- Statistical significance: MLflow tracks Cohen's Kappa against human baseline
- Aliases in Prompt Registry: `@control` and `@challenger` for prompt A/B testing
**Azure ML safe rollout progression**:
1. **Shadow testing**: Mirror X% of traffic to new model (no user impact)
**Azure ML safe rollout progression** — Verified (MCP 2026-04):
1. **Shadow testing**: Mirror X% of traffic to new model (no user impact) — natively supported via `mirror_traffic` property on managed online endpoints
2. **Canary**: Route 10% live traffic, monitor bake time (hours/days)
3. **Progressive**: 10% → 50% → 100% with health gate at each step
4. **Rollback trigger**: Automatic halt on health signal degradation

View file

@ -1,6 +1,6 @@
# Azure ML Pipelines - Orchestration and Automation
**Last updated:** 2026-02
**Last updated:** 2026-04
**Verified:** MCP 2026-04
**Status:** GA
**Category:** MLOps & GenAIOps
@ -22,59 +22,81 @@ Fra et kostnads- og effektivitetsperspektiv gir pipelines betydelige fordeler: d
### Pipeline Components (v2)
### Azure ML Pipelines — Python SDK v2 (Tutorial 2026)
### Azure ML Pipelines — Python SDK v2 (Tutorial, Verified MCP 2026-04)
**Key benefits**: Standardized MLOps, scalable team collaboration, training efficiency, cost reduction.
**Key benefits**: Standardized MLOps practice, scalable team collaboration, training efficiency, cost reduction.
**Pipeline creation pattern** (SDK v2):
**Pipeline creation pattern** (SDK v2 — from official tutorial):
```python
from azure.ai.ml import MLClient, dsl, Input, Output, command
from azure.identity import DefaultAzureCredential
from azure.identity import DefaultAzureCredential, InteractiveBrowserCredential
ml_client = MLClient(DefaultAzureCredential(), subscription_id, resource_group, workspace)
try:
credential = DefaultAzureCredential()
credential.get_token("https://management.azure.com/.default")
except Exception:
credential = InteractiveBrowserCredential()
# 1. Create reusable components
ml_client = MLClient(credential, subscription_id, resource_group, workspace)
# Note: MLClient initialization is lazy — no connection until first call
# 1. Create reusable components (programmatic definition)
data_prep_component = command(
name="data_prep",
name="data_prep_credit_defaults",
inputs={"data": Input(type="uri_folder"), "test_train_ratio": Input(type="number")},
outputs={"train_data": Output(type="uri_folder"), "test_data": Output(type="uri_folder")},
outputs={"train_data": Output(type="uri_folder", mode="rw_mount"),
"test_data": Output(type="uri_folder", mode="rw_mount")},
code="./components/data_prep",
command="python data_prep.py --data ${{inputs.data}} ...",
environment=f"{env.name}:{env.version}",
command="python data_prep.py --data ${{inputs.data}} --test_train_ratio ${{inputs.test_train_ratio}} ...",
environment=f"{pipeline_job_env.name}:{pipeline_job_env.version}",
)
# Register for reuse
ml_client.create_or_update(data_prep_component.component)
data_prep_component = ml_client.create_or_update(data_prep_component.component)
# 2. Define pipeline with @dsl.pipeline decorator
@dsl.pipeline(compute="serverless", description="E2E training pipeline")
def training_pipeline(data_input, test_train_ratio, learning_rate, model_name):
@dsl.pipeline(
compute="serverless", # "serverless" runs on serverless compute
description="E2E data_prep-train pipeline",
)
def credit_defaults_pipeline(data_input, test_train_ratio, learning_rate, registered_model_name):
prep_job = data_prep_component(data=data_input, test_train_ratio=test_train_ratio)
train_job = train_component(
train_data=prep_job.outputs.train_data,
test_data=prep_job.outputs.test_data,
learning_rate=learning_rate,
registered_model_name=model_name,
registered_model_name=registered_model_name,
)
return {
"pipeline_job_train_data": prep_job.outputs.train_data,
"pipeline_job_test_data": prep_job.outputs.test_data,
}
# 3. Submit pipeline
pipeline_job = ml_client.jobs.create_or_update(
training_pipeline(data_input=..., ...),
experiment_name="e2e_pipeline"
credit_defaults_pipeline(
data_input=Input(type="uri_file", path=credit_data.path),
test_train_ratio=0.25,
learning_rate=0.05,
registered_model_name="credit_defaults_model",
),
experiment_name="e2e_registered_components"
)
ml_client.jobs.stream(pipeline_job.name)
```
**Component lifecycle**:
1. Write YAML spec or create programmatically (`CommandComponent`)
2. Register with name+version in workspace or registry
3. Load and compose into pipeline
4. Submit via `ml_client.jobs.create_or_update()`
1. Write YAML spec (`train.yml`) or create programmatically (`CommandComponent` / `command()`)
2. Register with name+version: `ml_client.create_or_update(component)`
3. Load and compose into pipeline using `@dsl.pipeline` decorator
4. Submit via `ml_client.jobs.create_or_update()` with experiment name
**Compute options**: `serverless` (recommended), named compute cluster, or per-step compute override.
**Environment**: Curated environments (`azureml://registries/azureml/environments/sklearn-1.5/labels/latest`) or custom conda/Docker.
**Compute options**: `serverless` (recommended — zero config), named compute cluster, or per-step compute override (e.g., `train_step.compute = "cpu-cluster"`).
**Environment**: Curated environments (`azureml://registries/azureml/environments/sklearn-1.0/labels/latest`) or custom conda/Docker (base image: `mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu22.04:latest`).
**Output types**: `uri_folder` (data), `mlflow_model` (model), `uri_file` (file).
**MLflow integration**: Use `mlflow.start_run()` in scripts for automatic experiment tracking (metrics, parameters, models).
**MLflow integration**: Use `mlflow.start_run()` + `mlflow.sklearn.autolog()` in training scripts for automatic experiment tracking. Models registered via `mlflow.sklearn.log_model()` with `registered_model_name`.
**VNet note**: If workspace uses a managed virtual network, add outbound rules to allow access to public Python package repositories.
| Komponent-type | Beskrivelse | Bruksområde |
|----------------|-------------|-------------|
@ -564,41 +586,41 @@ Er det >3 steg i workflow?
1. **What are Azure Machine Learning pipelines?**
https://learn.microsoft.com/en-us/azure/machine-learning/concept-ml-pipelines?view=azureml-api-2
*Confidence: Verified (Feb 2026)*
*Confidence: Verified (April 2026)*
2. **Schedule machine learning pipeline jobs**
https://learn.microsoft.com/en-us/azure/machine-learning/how-to-schedule-pipeline-job?view=azureml-api-2
*Confidence: Verified (Feb 2026)*
*Confidence: Verified (April 2026)*
3. **Create and run machine learning pipelines using components with the Azure Machine Learning SDK v2**
https://learn.microsoft.com/en-us/azure/machine-learning/how-to-create-component-pipeline-python?view=azureml-api-2
*Confidence: Verified (Feb 2026)*
*Confidence: Verified (April 2026)*
4. **Tutorial: Create production machine learning pipelines**
https://learn.microsoft.com/en-us/azure/machine-learning/tutorial-pipeline-python-sdk?view=azureml-api-2
*Confidence: Verified (Feb 2026)*
*Confidence: Verified (April 2026)*
5. **Use parallel jobs in pipelines**
https://learn.microsoft.com/en-us/azure/machine-learning/how-to-use-parallel-job-in-pipeline?view=azureml-api-2
*Confidence: Verified (Feb 2026)*
*Confidence: Verified (April 2026)*
6. **Manage inputs and outputs for components and pipelines**
https://learn.microsoft.com/en-us/azure/machine-learning/how-to-manage-inputs-outputs-pipeline?view=azureml-api-2
*Confidence: Verified (Feb 2026)*
*Confidence: Verified (April 2026)*
7. **Create jobs and input data for batch endpoints**
https://learn.microsoft.com/en-us/azure/machine-learning/how-to-access-data-batch-endpoints-jobs?view=azureml-api-2
*Confidence: Verified (Feb 2026)*
*Confidence: Verified (April 2026)*
8. **Upgrade pipeline endpoints to SDK v2**
https://learn.microsoft.com/en-us/azure/machine-learning/migrate-to-v2-deploy-pipelines?view=azureml-api-2
*Confidence: Verified (Feb 2026)*
*Confidence: Verified (April 2026)*
### Code Samples (Verified)
- **Azure ML Examples Repository (azureml-examples/sdk/python/schedules):**
https://github.com/Azure/azureml-examples
*Confidence: Verified (Feb 2026)*
*Confidence: Verified (April 2026)*
### Konfidensgradering per seksjon
@ -613,5 +635,5 @@ Er det >3 steg i workflow?
| Kostnad og lisensiering | Verified + Baseline | MS Learn: cost considerations + Azure pricing |
| For arkitekten | Baseline | Arkitekturkonsulent-erfaring |
**Verified:** Informasjon hentet direkte fra Microsoft Learn MCP-dokumentasjon (februar 2026).
**Verified:** Informasjon hentet direkte fra Microsoft Learn MCP-dokumentasjon (april 2026).
**Baseline:** Informasjon basert på modellkunnskap og arkitekturerfaring, konsistent med Azure ML prinsipper.

View file

@ -1,7 +1,6 @@
# CI/CD Pipelines for Machine Learning Models
**Last updated:** 2026-02
**Verified:** MCP 2026-04
**Last updated:** 2026-04
**Status:** GA
**Category:** MLOps & GenAIOps
@ -290,8 +289,10 @@ Disse signalene indikerer at din ML CI/CD ikke er production-ready:
### GitHub Actions Integration
### GitHub Actions with Azure Machine Learning (2026 Update)
The recommended authentication approach is **OpenID Connect (OIDC) with federated credentials** — eliminates long-lived secrets.
### GitHub Actions with Azure Machine Learning (Verified MCP 2026-04)
The recommended authentication approach is **OpenID Connect (OIDC) with federated credentials** — eliminates long-lived secrets. Two options:
- **Option 1: Microsoft Entra application** — Create app registration, configure federated identity credential, assign role.
- **Option 2: User-assigned managed identity** — Create UAI, configure federated identity credential, assign role.
**Workflow structure** (`/.github/workflows/`):
```yaml
@ -300,6 +301,7 @@ permissions:
jobs:
build:
steps:
- uses: actions/checkout@v4
- uses: azure/login@v2
with:
client-id: ${{ secrets.AZURE_CLIENT_ID }}
@ -311,11 +313,13 @@ jobs:
**MLOps v2 GitHub setup** (recommended end-to-end):
1. Fork `Azure/mlops-v2-gha-demo` template repo
2. Set GitHub secrets: `ARM_CLIENT_ID`, `ARM_CLIENT_SECRET`, `ARM_SUBSCRIPTION_ID`, `ARM_TENANT_ID`
3. Deploy infrastructure via `tf-gha-deploy-infra.yml` workflow
3. Deploy infrastructure via `tf-gha-deploy-infra.yml` workflow (Terraform)
4. Run `deploy-model-training-pipeline` and `deploy-online-endpoint-pipeline` workflows
**Pipeline stages**: Prepare Data → Train Model → Evaluate Model → Register Model → Deploy Endpoint
**Note (2026-04):** The `--json-auth`/`--sdk-auth` parameters for `az ad sp create-for-rbac` are deprecated. New projects should use OIDC with federated credentials instead.
**Setup:**
- Opprett `.github/workflows/` directory i repo
@ -666,7 +670,7 @@ CI/CD pipelines for ML krever compute for training og deployment:
1. **Use GitHub Actions with Azure Machine Learning**
https://learn.microsoft.com/en-us/azure/machine-learning/how-to-github-actions-machine-learning
(Status: Verified 2026-02, fullstendig guide til GitHub Actions + Azure ML CLI v2)
(Status: Verified MCP 2026-04 — OIDC recommended; supports Entra app or user-assigned managed identity)
2. **MLOps and GenAIOps for AI workloads on Azure**
https://learn.microsoft.com/en-us/azure/well-architected/ai/mlops-genaiops
@ -674,7 +678,7 @@ CI/CD pipelines for ML krever compute for training og deployment:
3. **Set up MLOps with GitHub**
https://learn.microsoft.com/en-us/azure/machine-learning/how-to-setup-mlops-github-azure-ml
(Status: Verified 2026-02, end-to-end MLOps setup med GitHub Actions)
(Status: Verified MCP 2026-04 — uses mlops-v2-gha-demo accelerator; --json-auth deprecated, OIDC recommended)
4. **How does Databricks support CI/CD for machine learning?**
https://learn.microsoft.com/en-us/azure/databricks/machine-learning/mlops/ci-cd-for-ml

View file

@ -1,6 +1,6 @@
# Data Drift Monitoring and Detection
**Last updated:** 2026-02
**Last updated:** 2026-04
**Verified:** MCP 2026-04
**Status:** GA
**Category:** MLOps & GenAIOps
@ -350,7 +350,7 @@ Hvis kunden bruker legacy `DataDriftDetector` (azureml-datadrift SDK):
## Kilder og verifisering
**Verified (Microsoft Learn MCP, 2026-02):**
**Verified (Microsoft Learn MCP, 2026-04):**
- Azure Machine Learning model monitoring concept: https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2
- Monitor model performance in production: https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2
- Data drift (v1, deprecated): https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-datasets?view=azureml-api-1
@ -368,7 +368,7 @@ Hvis kunden bruker legacy `DataDriftDetector` (azureml-datadrift SDK):
**Unique Sources:** 12 Microsoft Learn URLs
### Azure ML Model Monitoring — Data Drift Detection (2026)
### Azure ML Model Monitoring — Data Drift Detection (2026) — Verified (MCP 2026-04)
**Model monitoring signals** (out-of-box for online endpoints):

View file

@ -2,6 +2,7 @@
**Kategori:** MLOps & GenAIOps
**Dato:** 2026-02-04
**Last updated:** 2026-04
**Confidence:** HIGH (basert på offisiell Microsoft-dokumentasjon)
**Verified:** MCP 2026-04
@ -711,7 +712,7 @@ mlflow.log_param("user_id_hash", user_id_hash) # Logged
**Primærkilder (Microsoft Learn):**
1. [MLflow for GenAI Apps and Agents - Continuous Improvement Cycle](https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/overview/)
1. [MLflow for GenAI Apps and Agents - Continuous Improvement Cycle](https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/overview/) (Verified MCP 2026-04 — updated 10-step cycle; new: Trace UI for pattern identification, evaluation harness, version/prompt management tracking)
2. [Machine Learning Operations v2 - Monitoring & Feedback](https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/machine-learning-operations-v2)
3. [Generative AI App Developer Workflow - Production Monitoring](https://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/genai-developer-workflow)
4. [Azure AI Foundry - Observability in Generative AI](https://learn.microsoft.com/en-us/azure/ai-foundry/concepts/observability)
@ -720,10 +721,10 @@ mlflow.log_param("user_id_hash", user_id_hash) # Logged
**Code samples:**
- MLflow feedback logging: [Azure Databricks - Agent Framework](https://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-framework/non-conversational-agents#log-user-feedback)
- Model monitoring setup: [Azure ML - Monitor Model Performance](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2)
- GenAI evaluation: [MLflow 3.x - Evaluate App](https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/eval-monitor/evaluate-app)
- Model monitoring setup: [Azure ML - Monitor Model Performance](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2) (Verified MCP 2026-04 — supports data quality, data drift, prediction drift, feature attribution drift, and custom signals; integrates with Azure Event Grid for alerting)
- GenAI evaluation: [MLflow 3.x - Evaluate App](https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/eval-monitor/evaluate-app) (Verified MCP 2026-04 — tutorial covers RAG email app evaluation; new scorers: RetrievalGroundedness, Guidelines, RelevanceToQuery, Safety; version comparison with mlflow.genai.evaluate())
**Dato for siste verifikasjon:** 2026-02-04
**Dato for siste verifikasjon:** 2026-04-10
**MCP calls:** 6 (microsoft_docs_search: 3, microsoft_docs_fetch: 3, microsoft_code_sample_search: 2)
@ -742,7 +743,7 @@ Dette dokumentet dekker hele feedback loop-syklusen for både classical ML og Ge
Bruk arkitekturmønstrene til å visualisere løsningen for kunden. Påpek at MLflow Tracing + Agent Evaluation gir "free" observability (built-in i Databricks).
### MLflow 3 Evaluation & Feedback Loop (2026)
### MLflow 3 Evaluation & Feedback Loop (Verified MCP 2026-04)
MLflow 3 introduces a unified evaluation-monitoring lifecycle for GenAI feedback loops:
@ -753,19 +754,27 @@ MLflow 3 introduces a unified evaluation-monitoring lifecycle for GenAI feedback
4. **Gather human feedback** via Review App (expert annotations)
5. **Improve** prompts/models based on evaluation datasets
**Built-in LLM judges (scorers)**:
- `RetrievalGroundedness` — checks if response is grounded in retrieved data
- `RelevanceToQuery` — checks if response addresses the user request
- `Safety` — checks for harmful/inappropriate content
- `Guidelines(name, guidelines)` — custom policy/tone/style checks
- `Correctness` — factual correctness with expected_facts
**Azure ML Model Monitoring signals**:
- Data quality: null values, out-of-range, type mismatch
- Data drift: statistical distribution changes between training and production data
- Prediction drift: distribution shift in model outputs
- Feature attribution drift: changes in feature importance
- Custom signals: user-defined metrics via custom scripts
- Integrates with **Azure Event Grid** for alerting on threshold breaches
**Monitoring setup**:
```python
# Set up out-of-box monitoring for Azure ML online endpoints
# Monitors data drift, prediction drift automatically
# Integrates with Azure Event Grid for alerting
```
**Evaluation dataset workflow (new 2026-04)**:
1. Search production traces → select problematic + high-quality examples
2. Save to versioned eval dataset in Unity Catalog (`mlflow.genai.datasets.create_dataset()`)
3. Run evaluation harness with `mlflow.genai.evaluate(data=eval_dataset, predict_fn=..., scorers=...)`
4. Compare runs in UI (`Evaluation runs` view) or SDK (`mlflow.search_runs`)
5. Identify regressions per-metric before promoting new versions
**Continuous improvement cycle**: Production traces → MLflow evaluation datasets → Scorer alignment → Prompt/model update → A/B test → Production rollout

View file

@ -1,7 +1,7 @@
# GenAIOps - LLM-Specific MLOps Practices
**Dato:** 2026-02-04
**Last updated:** 2026-04 | Verified: MCP 2026-04
**Last updated:** 2026-04
**Kategori:** MLOps & GenAIOps
**Konfidensgrad:** Høy (basert på 18 MCP-kilder fra Microsoft Learn)
@ -180,17 +180,25 @@ MLflow Tracing provides end-to-end observability for GenAI applications:
### API Management som LLM Gateway
**Hva:** Centralized gateway foran Azure OpenAI og eksterne LLM APIs.
**Hva:** Centralized gateway foran Azure OpenAI in Foundry Models og andre LLM APIs.
**GenAIOps use cases:**
- **Load balancing**: Distribuer trafikk over multiple Azure OpenAI instances
- **Throttling**: Rate limiting per user/subscription
- **Token tracking**: Centralized logging av token consumption
- **Cost allocation**: Chargeback til teams basert på usage
- **A/B testing**: Route 10% traffic til ny modell, 90% til gammel
- **Load balancing**: Distribuer trafikk over multiple Azure OpenAI instances (med health endpoint monitoring og circuit breaking)
- **Throttling**: Rate limiting per user/subscription (token-per-minute og requests-per-minute)
- **Token tracking**: Centralized logging av token consumption (cross-model observability)
- **Cost allocation**: Chargeback til teams basert på usage (showback/chargeback for multitenant)
- **A/B testing / Safe deployment**: Route 10% traffic til ny modell, 90% til gammel
- **Circuit breaker**: Failover til backup LLM provider (OpenAI → Mistral)
- **Federated authentication**: Extend client auth beyond Entra ID and API keys
- **Data sovereignty**: Enforce regional routing compliance for GDPR
**Konfidensgrad:** 90% — API Management for LLM er dokumentert pattern (2025).
**Implementasjonsalternativer (Verified MCP 2026-04):**
1. **Azure API Management** (anbefalt) — PaaS, built-in Azure OpenAI policies (`Limit Azure OpenAI API token usage`, `Emit metrics for consumption`), zone-redundant, multi-region. Bruk [GenAI gateway toolkit](https://github.com/Azure-Samples/apim-genai-gateway-toolkit) for custom policies + load-testing.
2. **Custom code** — Deploy gateway-logikk til App Service, Container Apps eller AKS. Kan frontes av API Management for HTTP-gateway capabilities.
**Viktig:** Global og data zone deployments i Azure OpenAI (som distribuerer kapasitet på tvers av datasentre) er i seg selv en gateway-implementasjon — vurder om disse dekker behovet FØR du legger til ekstra gateway-lag.
**Konfidensgrad:** 90% — API Management for LLM er dokumentert pattern (Verified MCP 2026-04).
---
@ -347,7 +355,7 @@ MLflow Tracing provides end-to-end observability for GenAI applications:
13. [GenAI app developer workflow](https://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/genai-developer-workflow)
14. [Plan and prepare a GenAIOps solution (Microsoft Learn Training)](https://learn.microsoft.com/en-us/training/modules/plan-prepare-genaiops/)
15. [Implement LLMOps in Azure Databricks (Microsoft Learn Training)](https://learn.microsoft.com/en-us/training/modules/implement-llmops-azure-databricks/)
16. [Azure OpenAI Gateway Guide](https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/azure-openai-gateway-guide)
16. [Access Azure OpenAI in Foundry Models through a gateway](https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/azure-openai-gateway-guide) (Verified MCP 2026-04)
17. [RAG solution design and evaluation guide](https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/rag/rag-solution-design-and-evaluation-guide)
18. [Microsoft GenAIOps Prompt Flow Template (GitHub)](https://github.com/microsoft/genaiops-promptflow-template)

View file

@ -1,7 +1,7 @@
# Inferencing Optimization and Caching
**Kategori:** MLOps & GenAIOps
**Dato:** 2026-02-04
**Dato:** 2026-04
**Forfattet av:** Cosmo Skyberg, Senior Microsoft AI Solution Architect
**Verified:** MCP 2026-04
@ -1015,7 +1015,7 @@ Diagnostikk:
**Confidence nivå: HIGH** — Denne referansen er basert på 12 MCP-kall til offisiell Microsoft-dokumentasjon og kodeeksempler.
### ONNX Inferencing Optimization for Computer Vision (Azure ML AutoML 2026)
### ONNX Inferencing Optimization for Computer Vision (Azure ML AutoML 2026) — Verified (MCP 2026-04)
ONNX (Open Neural Network Exchange) enables cross-framework interoperability and inference optimization:

View file

@ -1,6 +1,6 @@
# Infrastructure as Code for MLOps
**Dato:** 2026-02-04
**Dato:** 2026-04
**Kategori:** MLOps & GenAIOps
**Forfatter:** Cosmo Skyberg, Senior Microsoft AI Solution Architect
@ -572,7 +572,7 @@ terraform init && terraform apply
- Scan IaC repos for secrets (Microsoft Defender for Cloud: IaC vulnerability scanning)
- Immutable infrastructure preferred for business-critical workloads
**AI opportunity** (2026): AI tools (GitHub Copilot) can review IaC templates, identify misconfigurations, suggest security improvements, and generate templates from natural language.
**AI opportunity** (Verified MCP 2026-04): AI tools (GitHub Copilot) can review IaC templates for misconfigurations, suggest secure alternatives, and generate templates from natural language. Generative AI can analyze IaC templates and architectural diagrams, generate threat models, and recommend IaC updates from pull requests. Agent-based solutions can infer infrastructure needs from code and generate PRs with recommended IaC changes.
**MLOps v2 infrastructure**: `tf-gha-deploy-infra.yml` workflow in `Azure/mlops-v2-gha-demo` deploys full Azure ML infrastructure via Terraform + GitHub Actions.
@ -864,7 +864,7 @@ terraform {
## Kilder og verifisering
Denne kunnskapsreferansen er basert på følgende verifiserte kilder (hentet 2026-02-04):
Denne kunnskapsreferansen er basert på følgende verifiserte kilder (hentet 2026-04):
1. **Microsoft Learn - What is Infrastructure as Code (IaC)?**
- URL: https://learn.microsoft.com/devops/deliver/what-is-infrastructure-as-code
@ -921,7 +921,7 @@ Denne kunnskapsreferansen er basert på følgende verifiserte kilder (hentet 202
- **microsoft_docs_fetch calls:** 3
- **microsoft_code_sample_search calls:** 1
- **Total sources:** 10
- **Dato for research:** 2026-02-04
- **Dato for research:** 2026-04
**Confidence levels:**
- VERY_HIGH: Offisiell Microsoft-dokumentasjon, verifiserte code samples
@ -932,5 +932,5 @@ Alle kodeeksempler er hentet fra official Microsoft Learn eller GitHub repos und
---
**Oppdatert:** 2026-02-04
**Neste review:** 2026-05-04 (eller når Azure ML API major version oppdateres)
**Oppdatert:** 2026-04
**Neste review:** 2026-07-04 (eller når Azure ML API major version oppdateres)

View file

@ -1,7 +1,7 @@
# LLM Evaluation in Production Contexts
**Kategori:** MLOps & GenAIOps
**Sist oppdatert:** 2026-02-04
**Sist oppdatert:** 2026-04
**Confidence:** High (basert på offisiell Microsoft dokumentasjon, Azure AI Foundry SDK, og MLflow 3)
---
@ -575,15 +575,24 @@ MLflow 3 (SDK `mlflow[databricks]>=3.1`) introduces a unified evaluation model:
| Judge | Needs Ground Truth | Evaluates |
|-------|-------------------|-----------|
| `RelevanceToQuery` | No | Response relevance to user request |
| `RetrievalRelevance` | No | Retrieved context relevance to user request |
| `RetrievalGroundedness` | No | Hallucination detection |
| `Safety` | No | Harmful/toxic content |
| `Correctness` | Yes | Accuracy vs ground truth |
| `Completeness` | Yes | All questions addressed |
| `Fluency` | No | Grammatically correct and naturally flowing |
| `Equivalence` | Yes | Response equivalent to expected output |
| `RetrievalSufficiency` | Yes | Context provides all necessary information |
| `ToolCallCorrectness` | Yes | Tool calls and arguments |
| `ToolCallEfficiency` | No | Redundant tool usage |
| `Guidelines` | No | Custom natural-language rules |
| `ExpectationsGuidelines` | No (needs guidelines in expectations) | Per-example natural-language criteria |
**Multi-turn judges** (conversation-level): `ConversationCompleteness`, `UserFrustration`, `KnowledgeRetention`, `ConversationalSafety`
Verified (MCP 2026-04)
**Multi-turn judges** (conversation-level): `ConversationCompleteness`, `UserFrustration`, `KnowledgeRetention`, `ConversationalSafety`, `ConversationalGuidelines`, `ConversationalRoleAdherence`, `ConversationalToolCallEfficiency`
Verified (MCP 2026-04)
**Production monitoring**: Automatically runs scorers on production traces; uses Databricks-hosted LLM judges (EU workspaces: EU-hosted models). No prompts stored with Azure OpenAI (Abuse Monitoring opt-out).
@ -1088,7 +1097,7 @@ Production evaluation er ikke komplett uten human review loop. Anbefal:
- Power Platform evaluation gaps (product evolves rapidly)
- Human feedback loop implementation (no single canonical pattern)
**Ufullstendig informasjon (per feb 2026):**
**Ufullstendig informasjon (per april 2026):**
- Native Copilot Studio production evaluation features (roadmap item, not released)
- Detailed pricing for Azure AI Content Safety evaluators (bundled pricing, not per-call transparent)

View file

@ -1,6 +1,6 @@
# MLOps Fundamentals - Lifecycle and Principles
**Last updated:** 2026-02
**Last updated:** 2026-04
**Verified:** MCP 2026-04
**Status:** GA
**Category:** MLOps & GenAIOps
@ -296,7 +296,7 @@ jobs:
### DevOps-verktøy
### DevOps for Machine Learning — Azure DevOps Integration (2026)
### DevOps for Machine Learning — Azure DevOps Integration (Verified MCP 2026-04)
**Azure Pipelines + Azure ML** (how-to-devops-machine-learning):
@ -306,27 +306,36 @@ Automate the ML lifecycle via Azure DevOps pipelines:
3. Model deployment (public/private web service)
4. Monitoring (performance, data drift)
**Azure DevOps pipeline YAML pattern**:
**Prerequisite**: Python >=3.10 required for Azure ML SDK v2 scripts. Install [Azure Machine Learning extension for Azure Pipelines](https://marketplace.visualstudio.com/items?itemName=ms-air-aiagility.azureml-v2) from VS Marketplace.
**Authentication options** (Verified MCP 2026-04):
- **Azure Resource Manager service connection** (recommended) — use with `AzureMLJobWaitTask@1` from Azure ML extension
- **Generic service connection** — use with `InvokeRESTAPI` task calling REST API directly (api-version: `2024-04-01`)
**Azure DevOps pipeline YAML pattern** (ARM service connection):
```yaml
- task: AzureCLI@2
name: submit_azureml_job_task
inputs:
azureSubscription: $(service-connection)
scriptType: bash
inlineScript: |
job_name=$(az ml job create --file pipeline.yml -g $(resource-group) -w $(workspace) --query name -o tsv)
job_name=$(az ml job create --file pipeline.yml -g $(resource-group) -w $(workspace) --query name --output tsv)
echo "##vso[task.setvariable variable=JOB_NAME;isOutput=true;]$job_name"
- job: WaitForJobCompletion
pool: server # Server job — no agent costs
pool: server # Server job — no agent costs, runs on pipeline machine
dependsOn: SubmitAzureMLJob
steps:
- task: AzureMLJobWaitTask@1 # From Azure ML extension
- task: AzureMLJobWaitTask@1 # From Azure ML extension (not "classic")
inputs:
serviceConnection: $(service-connection)
azureMLJobName: $(azureml_job_name)
resourceGroupName: $(resource-group)
azureMLWorkspaceName: $(workspace)
azureMLJobName: $(azureml_job_name_from_submit_job)
```
**Authentication options**:
- Azure Resource Manager service connection (recommended with Azure ML extension)
- Generic service connection (uses InvokeRESTAPI task)
**Note**: `AzureMLJobWaitTask@1` runs as a server job (no agent pool costs). Max wait: 2 days (Azure DevOps hard limit). Use `AzureMLJobWaitTask@1`, not the legacy "Machine Learning (classic)" extension.
**MLOps maturity model**: Manual → Partial automation → Full CI/CD → Full MLOps with monitoring
@ -495,5 +504,5 @@ Er dette en POC?
### Sist verifisert
Alle kilder verifisert via `microsoft-learn` MCP-server **2026-02-04**.
Alle kilder verifisert via `microsoft-learn` MCP-server **2026-04**.
Azure ML dokumentasjon gjelder **API v2 (current)** med mindre annet er nevnt.

View file

@ -1,7 +1,7 @@
# Security and Access Control in MLOps
**Kategori:** MLOps & GenAIOps
**Last updated:** 2026-04 | Verified: MCP 2026-04
**Last updated:** 2026-04
**Dato:** 2026-04-10
**Confidence:** HIGH — Basert på offisiell Microsoft Learn dokumentasjon (8 MCP-oppslag, 16 kilder)
@ -738,9 +738,10 @@ AmlComputeClusterNodeEvent
6. [Configure a private endpoint for an Azure Machine Learning workspace](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-configure-private-link?view=azureml-api-2)
7. [Secure an Azure Machine Learning workspace with virtual networks](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-secure-workspace-vnet?view=azureml-api-2)
8. [Data encryption with Azure Machine Learning](https://learn.microsoft.com/en-us/azure/machine-learning/concept-data-encryption?view=azureml-api-2)
(Verified MCP 2026-04 — Key updates: Azure Data Lake Storage Gen1 retired 2024-02-29; Azure Database for PostgreSQL Single Server retired 2025-03-28; Azure Database for MySQL Single Server retired 2024-09-16. Use Gen2 / Flexible Server variants.)
**Sist verifisert:** 2026-02-04
**Neste review:** Q2 2026 (ved nye identity/network features i Azure ML)
**Sist verifisert:** 2026-04-10
**Neste review:** Q3 2026 (ved nye identity/network features i Azure ML)
---

View file

@ -1,7 +1,7 @@
# MLOps Team Collaboration and Tools Integration
**Kategori:** MLOps & GenAIOps
**Sist oppdatert:** 2026-02-04
**Sist oppdatert:** 2026-04
**Kilde:** Microsoft Learn, Azure Architecture Center
**Konfidensgradering:** ⭐⭐⭐⭐⭐ (Verifisert mot offisiell Microsoft-dokumentasjon)
@ -146,10 +146,12 @@ Azure DevOps provides end-to-end project management for ML teams:
- `azure/login@v2` + `az ml job create` pattern
- MLOps v2 solution accelerator: `Azure/mlops-v2-gha-demo`
**Databricks CI/CD best practices**:
- Feature branching with short-lived branches
- Automated notebook testing before merge
**Databricks CI/CD best practices (Verified MCP 2026-04)**:
- Feature branching with short-lived branches (Gitflow aligned with dev/staging/prod environments)
- Automated notebook testing before merge (bundle validate + pytest/ScalaTest)
- MLflow experiment tracking integrated into PR workflows
- **Declarative Automation Bundles** (formerly Databricks Asset Bundles) recommended for unified code+infra deployment
- Workload identity federation (eliminates Databricks secrets) recommended for CI/CD auth
**Formål:** CI/CD automation for ML lifecycle
**Nøkkelkapabiliteter:**
@ -675,13 +677,13 @@ Databricks MLOps Stacks demonstrerer best practice for multi-team collaboration:
3. **What is Azure DevOps?**
URL: https://learn.microsoft.com/en-us/azure/devops/user-guide/what-is-azure-devops
Hentet: 2026-02-04
Relevans: Azure Boards capabilities, team collaboration features
Hentet: 2026-04-10
Relevans: Azure Boards capabilities, team collaboration features (Verified MCP 2026-04 — new: Azure DevOps MCP Server for natural language project management queries, AI-Enhanced management with Copilot integration)
4. **Best Practices and Recommended CI/CD Workflows on Databricks**
URL: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/ci-cd/best-practices
Hentet: 2026-02-04
Relevans: MLOps Stacks team collaboration table
Hentet: 2026-04-10
Relevans: MLOps Stacks team collaboration table (Verified MCP 2026-04 — now covers Declarative Automation Bundles, workload identity federation for auth, SQL and dashboard CI/CD workflows)
5. **Set up MLOps with Azure DevOps**
URL: https://learn.microsoft.com/en-us/azure/machine-learning/how-to-setup-mlops-azureml
@ -690,8 +692,8 @@ Databricks MLOps Stacks demonstrerer best practice for multi-team collaboration:
6. **Use GitHub Actions with Azure Machine Learning**
URL: https://learn.microsoft.com/en-us/azure/machine-learning/how-to-github-actions-machine-learning
Hentet: 2026-02-04
Relevans: GitHub Actions integration patterns
Hentet: 2026-04-10
Relevans: GitHub Actions integration patterns (Verified MCP 2026-04 — OIDC recommended with Entra app or user-assigned managed identity)
7. **MLOps Workflows on Azure Databricks**
URL: https://learn.microsoft.com/en-us/azure/databricks/machine-learning/mlops/mlops-workflow

View file

@ -5,7 +5,7 @@
**Målgruppe:** Arkitekter som planlegger ML-modellutplassering i produksjon
**Konfidensgrad:** ⚡️⚡️⚡️ Høy (basert på Microsoft Learn + offisielle code samples)
**Verified:** MCP 2026-04
**Last updated:** 2026-04
## Introduksjon
@ -620,6 +620,8 @@ deployment = ManagedOnlineDeployment(
- Built-in support for scikit-learn, TensorFlow, PyTorch
- Enklere rollback (bare endre model version)
**Auth note (Verified MCP 2026-04):** For production deployments, use Microsoft Entra token-based authentication (`aad_token`) instead of key-based auth — provides identity-based access control.
**Referanse:** [Deploy MLflow models to online endpoints](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-mlflow-models-online-endpoints?view=azureml-api-2)
---
@ -1020,7 +1022,7 @@ Denne kunnskapsreferansen er basert på følgende Microsoft Learn-artikler og co
**Primære kilder:**
1. [Perform safe rollout of new deployments for real-time inference](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-safely-rollout-online-endpoints?view=azureml-api-2)
→ Komplett guide til blue-green deployment og traffic mirroring
→ Komplett guide til blue-green deployment og traffic mirroring (Verified MCP 2026-04)
2. [MLOps model management with Azure Machine Learning](https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-management-and-deployment?view=azureml-api-2)
→ Oversikt over deployment capabilities og controlled rollout
@ -1032,7 +1034,7 @@ Denne kunnskapsreferansen er basert på følgende Microsoft Learn-artikler og co
→ Canary deployment med Azure DevOps Pipelines
5. [Progressive rollout of MLflow models to Online Endpoints](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-mlflow-models-online-progressive?view=azureml-api-2)
→ MLflow-spesifikk progressive rollout
→ MLflow-spesifikk progressive rollout; supports model packaging (--with-package) for endpoints without egress connectivity (Verified MCP 2026-04)
**Code samples:**
- [azureml-examples/sdk/python/endpoints/online/managed/online-endpoints-safe-rollout.ipynb](https://github.com/Azure/azureml-examples/blob/main/sdk/python/endpoints/online/managed/online-endpoints-safe-rollout.ipynb)
@ -1040,7 +1042,7 @@ Denne kunnskapsreferansen er basert på følgende Microsoft Learn-artikler og co
**Well-Architected Framework:**
- [Architecture strategies for safe deployment practices](https://learn.microsoft.com/en-us/azure/well-architected/operational-excellence/safe-deployments)
→ Progressive exposure model, bake times, rollback strategies
→ Progressive exposure model, bake times, rollback strategies (Verified MCP 2026-04 — adds safe decommissioning guidance + AI opportunity note for GenAI-assisted rollout tuning)
**Pricing (sist verifisert: 2026-02-04):**
- [Azure Machine Learning pricing](https://azure.microsoft.com/en-us/pricing/details/machine-learning/)
@ -1050,8 +1052,8 @@ Denne kunnskapsreferansen er basert på følgende Microsoft Learn-artikler og co
---
**Sist oppdatert:** 2026-02-04
**Neste review:** 2026-05-04 (eller ved større endringer i Azure ML deployment capabilities)
**Sist oppdatert:** 2026-04-10
**Neste review:** 2026-07-10 (eller ved større endringer i Azure ML deployment capabilities)
### Safe Rollout / Blue-Green Deployment (Azure Well-Architected 2026)
@ -1085,3 +1087,7 @@ az ml online-endpoint update --name my-endpoint --traffic blue=90 green=10
**Emergency SDP**: Prescriptive protocols for hotfix acceleration — approval stage and bake time reduction — with explicit approval criteria.
**Safe decommissioning (new in 2026-04)**: Removing components is highest-risk. Steps: validate inactivity → preserve state (backup/export) → disable before deleting → monitor watch window covering full usage cycle → clean up residual references. Skip disable only if compliance requires immediate removal.
**AI opportunity**: AI can assist rollout tuning — analyze deployment docs, code reviews, incident history to suggest rollout strategies and parameters (low/medium GenAI approach). Advanced agentic solutions can auto-update rollout configurations.

View file

@ -1,6 +1,6 @@
# Model Drift and Performance Degradation Detection
**Last updated:** 2026-02
**Last updated:** 2026-04
**Verified:** MCP 2026-04
**Status:** GA
**Category:** MLOps & GenAIOps
@ -41,7 +41,7 @@ Azure Machine Learning støtter flere built-in signals (med GA- eller preview-st
| **Feature Attribution Drift** | Feature importance-endringer | Preview | Normalized Discounted Cumulative Gain |
| **Model Performance** | Objektiv ytelse (krever ground truth) | Preview | Accuracy, Precision, Recall (classification); MAE, MSE, RMSE (regression) |
**Verified (MCP):** Metrics og signal-typer hentet fra offisiell Microsoft Learn-dokumentasjon (2026-02).
**Verified (MCP):** Metrics og signal-typer hentet fra offisiell Microsoft Learn-dokumentasjon (2026-04).
### 2. Reference Data
@ -582,44 +582,44 @@ Email Alerts + Azure Monitor Dashboard
1. **Azure Machine Learning model monitoring (Concept)**
https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2
*Verified: 2026-02 via microsoft_docs_fetch*
*Verified: 2026-04 via microsoft_docs_fetch*
- Monitoring signals, metrics, reference data, lookback windows
2. **Monitor the performance of models deployed to production**
https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2
*Verified: 2026-02 via microsoft_docs_fetch*
*Verified: 2026-04 via microsoft_docs_fetch*
- Setup guides (CLI, SDK, Studio), Event Grid integration, interpret results
3. **Data drift (preview) will be retired, and replaced by Model Monitor**
https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-datasets?view=azureml-api-1
*Verified: 2026-02 via microsoft_docs_search (3 results)*
*Verified: 2026-04 via microsoft_docs_search (3 results)*
- Legacy DataDriftDetector (v1) vs. Model Monitor (v2)
4. **Trigger applications, processes, or CI/CD workflows based on Azure Machine Learning events**
https://learn.microsoft.com/en-us/azure/machine-learning/how-to-use-event-grid?view=azureml-api-2
*Verified: 2026-02 via microsoft_docs_search*
*Verified: 2026-04 via microsoft_docs_search*
- Event Grid integration, advanced filters
5. **Machine learning operations (MLOps v2)**
https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/machine-learning-operations-v2
*Verified: 2026-02 via microsoft_docs_search (multiple references)*
*Verified: 2026-04 via microsoft_docs_search (multiple references)*
- Data drift, prediction drift, resource monitoring
### Code Samples (MCP-verified)
1. **Model monitoring setup (Python SDK v2)**
https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance
*Verified: 2026-02 via microsoft_code_sample_search*
*Verified: 2026-04 via microsoft_code_sample_search*
- Out-of-box monitoring, advanced monitoring, model performance
2. **DataDriftDetector (Python SDK v1 deprecated)**
https://learn.microsoft.com/en-us/python/api/azureml-datadrift/azureml.datadrift.datadriftdetector
*Verified: 2026-02 via microsoft_code_sample_search*
*Verified: 2026-04 via microsoft_code_sample_search*
- Legacy API for comparison
3. **Custom signal component examples**
https://github.com/Azure/azureml-examples/tree/main/cli/monitoring/components/custom_signal
*Referenced: 2026-02 in Microsoft Learn documentation*
*Referenced: 2026-04 in Microsoft Learn documentation*
### Confidence Markers
@ -635,7 +635,7 @@ Email Alerts + Azure Monitor Dashboard
### Sist oppdatert
**2026-02** Basert på Microsoft Learn-dokumentasjon (azure-ai-ml SDK v2, API version 2).
**2026-04** Basert på Microsoft Learn-dokumentasjon (azure-ai-ml SDK v2, API version 2).
### Azure ML Model Drift & Performance Degradation Monitoring (2026)

View file

@ -1,6 +1,6 @@
# Model Evaluation Frameworks and Metrics
**Last updated:** 2026-02
**Last updated:** 2026-04
**Verified:** MCP 2026-04
**Status:** GA
**Category:** MLOps & GenAIOps
@ -119,10 +119,11 @@ MLflow 3 provides the evaluation framework for both traditional ML and GenAI app
| Type | Customization | Use Case |
|------|--------------|---------|
| Built-in judges | Minimal | Quick evaluation: `Correctness`, `RetrievalGroundedness`, `Safety` |
| Guidelines judges | Moderate | Custom natural-language rules (pass/fail) |
| Built-in judges | Minimal | Quick evaluation: `Correctness`, `RetrievalGroundedness`, `Safety`, `RelevanceToQuery`, `Fluency`, `Equivalence` — Verified (MCP 2026-04) |
| Guidelines judges | Moderate | Custom natural-language rules (pass/fail): `Guidelines`, `ExpectationsGuidelines` |
| Custom LLM judges | Full | Domain-specific criteria, detailed scoring |
| Code-based scorers | Full | Deterministic: exact match, format validation, business logic |
| Multi-turn judges | Minimal | Conversation-level: `ConversationCompleteness`, `UserFrustration`, `KnowledgeRetention`, `ConversationalSafety` — Verified (MCP 2026-04) |
**Key evaluation functions**:
```python

View file

@ -1,6 +1,6 @@
# Model Versioning and Registry Management
**Last updated:** 2026-02
**Last updated:** 2026-04
**Verified:** MCP 2026-04
**Status:** GA
**Category:** MLOps & GenAIOps
@ -306,6 +306,8 @@ Azure AI Foundry Model Catalog bruker samme underliggende registry-infrastruktur
### Power Platform AI
**Scenario:** Registrer Custom AI Builder model i Azure ML Registry for reuse.
> **Merk (Verified MCP 2026-04):** For production online endpoint deployments anbefaler Microsoft nå Microsoft Entra token-based authentication (`aad_token`) fremfor key-based authentication for forbedret sikkerhet via identity-based access control.
- Tren modell i AI Builder
- Eksporter modell (hvis tilgjengelig)
- Registrer i Azure ML Registry som MLflow model
@ -511,7 +513,7 @@ az ml model list --registry-name my-registry --query "[?created<'$cutoff_date'].
## Kilder og verifisering
### Microsoft Learn (Verified via MCP research, February 2026)
### Microsoft Learn (Verified via MCP research, April 2026)
1. **Share models, components, and environments across workspaces with registries**
- URL: https://learn.microsoft.com/en-us/azure/machine-learning/how-to-share-models-pipelines-across-workspaces-with-registries?view=azureml-api-2
@ -566,7 +568,7 @@ az ml model list --registry-name my-registry --query "[?created<'$cutoff_date'].
- **Document fetches:** 2 (Full registry guide, MLflow management guide)
- **Code samples:** 1 (MLflow Python SDK examples)
- **Unique sources:** 7 Microsoft Learn articles
- **Research timestamp:** February 2026
- **Research timestamp:** April 2026
---

View file

@ -1,7 +1,7 @@
# Monitoring and Observability for ML Systems
**Kategori:** MLOps & GenAIOps
**Dato:** 2026-02-04
**Dato:** 2026-04
**Kilder:** Microsoft Learn (azure-machine-learning, azure-monitor)
**Konfidensgrad:** ⭐⭐⭐⭐⭐ (Verifisert mot offisiell Microsoft-dokumentasjon)
@ -300,7 +300,7 @@ create_monitor:
### Azure Monitor
### Azure Machine Learning Monitoring Architecture (2026)
### Azure Machine Learning Monitoring Architecture (2026) — Verified (MCP 2026-04)
**Azure Monitor integration**:
- All metrics in namespace: `Machine Learning Service Workspace`
@ -630,7 +630,7 @@ Azure Machine Learning Model Monitoring gir production-ready overvåkning av ML-
## Kilder og verifisering
**Primærkilder** (✅ Verifisert 2026-02-04):
**Primærkilder** (✅ Verifisert 2026-04):
1. [Monitor the performance of models deployed to production](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2)
2. [Azure Machine Learning model monitoring](https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2)
3. [Detect and mitigate potential issues using AIOps and machine learning in Azure Monitor](https://learn.microsoft.com/en-us/azure/azure-monitor/aiops/aiops-machine-learning)
@ -646,5 +646,5 @@ Azure Machine Learning Model Monitoring gir production-ready overvåkning av ML-
- ⭐⭐⭐⭐ = Basert på Microsoft Learn, men med noe tolkning
- ⭐⭐⭐ = Community best practices (ikke offisiell Microsoft-guidance)
**Sist verifisert:** 2026-02-04
**Sist verifisert:** 2026-04
**Neste review:** Når Azure ML Model Monitoring v3 lanseres (roadmap Q2 2026)