docs(architect): weekly KB update — 106 files refreshed (2026-04)
Updates across all 5 skills: ms-ai-advisor, ms-ai-engineering, ms-ai-governance, ms-ai-security, ms-ai-infrastructure. Key changes: - Language Services (Custom Text Classification, Text Analytics, QnA): retirement warning 2029-03-31, migration guides to Foundry/GPT-4o - Agentic Retrieval: 50M free reasoning tokens/month (Public Preview) - Computer Use: Claude Sonnet 4.5 (preview) + OpenAI CUA models - Agent Registry: Risks column (M365 E7), user-shared/org-published types - Declarative agents: schema v1.5 → v1.6, Store validation requirements - MLflow 3: 13 built-in LLM judges, production monitoring, Genie Code - AG-UI HITL: ApprovalRequiredAIFunction (C#) + @tool(approval_mode) (Python) - Entra ID Ignite 2025: Agent ID Admin/Developer RBAC roles, Conditional Access - Security Copilot: 400 SCU/month per 1000 M365 E5 licenses, auto-provisioned - Fast Transcription API: phrase lists, 14-language multi-lingual transcription - Azure Monitor Workbooks: Bicep support, RBAC specifics - Power Platform Copilot: data residency (Norway/Europe → EU DB, Bing → USA) - RAG security-rbac: 4-approach table (GA + 3 preview access control methods) - IaC MLOps: Well-Architected OE:05 principles, Bicep/Terraform patterns - Translator: image file batch translation Preview (JPEG/PNG/BMP/WebP) All 106 files: Last updated 2026-04 | Verified: MCP 2026-04 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen
parent
0eb30fa853
commit
6645e93205
104 changed files with 1986 additions and 520 deletions
|
|
@ -1,6 +1,7 @@
|
|||
# GenAIOps - LLM-Specific MLOps Practices
|
||||
|
||||
**Dato:** 2026-02-04
|
||||
**Last updated:** 2026-04 | Verified: MCP 2026-04
|
||||
**Kategori:** MLOps & GenAIOps
|
||||
**Konfidensgrad:** Høy (basert på 18 MCP-kilder fra Microsoft Learn)
|
||||
|
||||
|
|
@ -10,252 +11,16 @@
|
|||
|
||||
GenAIOps (Generative AI Operations), også kalt LLMOps, beskriver operasjonelle praksiser og strategier for håndtering av store språkmodeller (LLMs) i produksjon. Mens tradisjonell MLOps fokuserer på å trene og deploye diskriminative modeller, handler GenAIOps om å **velge, tilpasse, orkestrere og overvåke** eksisterende foundation models.
|
||||
|
||||
### Forskjell mellom MLOps og GenAIOps
|
||||
|
||||
| Dimensjon | Tradisjonell MLOps | GenAIOps (LLMOps) |
|
||||
|-----------|-------------------|-------------------|
|
||||
| **Primært fokus** | Trene nye modeller fra scratch | Konsumere og fine-tune eksisterende foundation models |
|
||||
| **Artefakter** | Trainede modeller (pkl, ONNX) | Prompts, orchestrators, agents, chains, grounding data |
|
||||
| **Evaluering** | Accuracy, precision, recall (deterministiske) | Groundedness, relevance, coherence, fluency (LLM-as-judge) |
|
||||
| **Infrastruktur** | Modell-serving endepunkter | Orchestrators, vector stores, API gateways, LLM endpoints |
|
||||
| **Deployment** | Modellversjonering | Modell + prompt + grounding data + orchestrator |
|
||||
| **Monitoring** | Model drift, data drift | Data drift + prompt effectiveness + content safety + token usage |
|
||||
|
||||
**Konfidensgrad:** 95% — Microsoft dokumentasjon definerer eksplisitt disse forskjellene.
|
||||
### MLflow 3 Tracing — GenAI Observability
|
||||
MLflow Tracing provides end-to-end observability for GenAI applications:
|
||||
- Records inputs, outputs, intermediate steps, and metadata
|
||||
- Supports complex agent-based systems and multi-turn conversations
|
||||
- Integrates with Genie Code for natural language trace analysis
|
||||
- Enables: debugging, performance monitoring, cost optimization, auditability
|
||||
- Production monitoring reuses same scorers as development evaluation (consistent lifecycle)
|
||||
|
||||
---
|
||||
|
||||
## Kjernekomponenter
|
||||
|
||||
### 1. Prompt Engineering og Prompt Registry
|
||||
|
||||
**Hva:** Strukturert håndtering av system- og user prompts som versjonerte artefakter.
|
||||
|
||||
**Hvorfor:** Prompts er den primære "koden" i GenAI-løsninger. Endringer i prompts påvirker output like mye som kodeendringer.
|
||||
|
||||
**Hvordan (Azure):**
|
||||
- **MLflow Prompt Registry** (Databricks): Versjonert prompt-håndtering med aliaser (f.eks. `production`, `staging`)
|
||||
- **Azure AI Foundry Prompt Flow**: Visuell prompt designer med versjonering og CI/CD-integrasjon
|
||||
- **Semantic Kernel Prompt Functions**: Prompts som code-artefakter i `.txt`-filer med Handlebars-syntax
|
||||
|
||||
```python
|
||||
# MLflow Prompt Registry eksempel
|
||||
import mlflow
|
||||
|
||||
prompt = mlflow.genai.register_prompt(
|
||||
name="mycatalog.myschema.customer_support",
|
||||
template="You are a helpful assistant. Answer this question: {{question}}",
|
||||
commit_message="Initial customer support prompt"
|
||||
)
|
||||
|
||||
mlflow.genai.set_prompt_alias(
|
||||
name="mycatalog.myschema.customer_support",
|
||||
alias="production",
|
||||
version=1
|
||||
)
|
||||
|
||||
# I applikasjon
|
||||
prompt = mlflow.genai.load_prompt(
|
||||
name_or_uri="prompts:/mycatalog.myschema.customer_support@production"
|
||||
)
|
||||
response = llm.invoke(prompt.format(question="How do I reset my password?"))
|
||||
```
|
||||
|
||||
**Konfidensgrad:** 90% — Prompt Registry er dokumentert, men adoption rates varierer.
|
||||
|
||||
### 2. Orchestration Layer
|
||||
|
||||
**Hva:** Systemet som håndterer logikk, kaller datakilder/agenter, genererer prompts og kaller LLM-modeller.
|
||||
|
||||
**Hvorfor:** Generative AI-løsninger er ikke bare modellen — de er komplekse workflows som krever orkestrering.
|
||||
|
||||
**Microsoft-alternativer:**
|
||||
- **Azure AI Foundry Agent Service**: Low-code agent-orkestrering
|
||||
- **Microsoft Agent Framework SDK (Semantic Kernel)**: Code-first orkestrering med C#/Python
|
||||
- **Prompt Flow**: Visuell workflow-designer for LLM-chains
|
||||
- **LangChain/LlamaIndex**: Open source (støttes av Azure ML)
|
||||
|
||||
**Deployment:**
|
||||
- Azure App Service (containerized orchestrator)
|
||||
- Azure Container Apps (serverless orchestrator)
|
||||
- Azure Kubernetes Service (high-scale orchestrator)
|
||||
- Azure Machine Learning Managed Online Endpoints
|
||||
|
||||
**Konfidensgrad:** 85% — Mange deployment-alternativer, best practice varierer med use case.
|
||||
|
||||
### 3. Vector Stores og Grounding Data
|
||||
|
||||
**Hva:** Datalagringsløsninger for RAG (Retrieval-Augmented Generation) som støtter vektor-søk.
|
||||
|
||||
**Azure-alternativer:**
|
||||
- **Azure AI Search**: Hybrid search (full-text + vector + semantic)
|
||||
- **Azure Cosmos DB for MongoDB vCore**: Vector search capabilities
|
||||
- **Azure Database for PostgreSQL (pgvector)**: Open source vector extension
|
||||
- **Databricks Vector Search**: Delta table-basert, auto-syncing
|
||||
|
||||
**DataOps-utvidelser for GenAIOps:**
|
||||
- **Chunking pipelines**: Split dokumenter i semantisk meningsfulle chunks (Azure Machine Learning pipelines)
|
||||
- **Embedding generation**: Batch-generering av embeddings (Azure OpenAI text-embedding-ada-002 / text-embedding-3-small)
|
||||
- **Index maintenance**: Incremental updates vs. full rebuilds (compliance: right-to-be-forgotten)
|
||||
- **Data freshness**: Real-time vs. batch refresh (business requirements)
|
||||
|
||||
**Konfidensgrad:** 90% — Dokumentert arkitektur, men chunking-strategier er eksperimentelle.
|
||||
|
||||
### 4. Evaluation Framework
|
||||
|
||||
**Hva:** LLM-spesifikke evalueringsmetrikker og human-in-the-loop feedback.
|
||||
|
||||
**Azure AI Foundry Evaluation SDK:**
|
||||
```python
|
||||
from azure.ai.evaluation import evaluate, RelevanceEvaluator, CoherenceEvaluator
|
||||
|
||||
model_config = {
|
||||
"azure_endpoint": os.environ.get("AZURE_OPENAI_ENDPOINT"),
|
||||
"api_key": os.environ.get("AZURE_OPENAI_KEY"),
|
||||
"azure_deployment": os.environ.get("AZURE_OPENAI_DEPLOYMENT"),
|
||||
}
|
||||
|
||||
result = evaluate(
|
||||
data="test_data.jsonl",
|
||||
evaluators={
|
||||
"relevance": RelevanceEvaluator(model_config=model_config),
|
||||
"coherence": CoherenceEvaluator(model_config=model_config),
|
||||
},
|
||||
evaluator_config={
|
||||
"relevance": {
|
||||
"column_mapping": {
|
||||
"query": "${data.query}",
|
||||
"ground_truth": "${data.ground_truth}",
|
||||
"response": "${outputs.response}"
|
||||
}
|
||||
}
|
||||
},
|
||||
azure_ai_project=azure_ai_project,
|
||||
output_path="./evaluation_results.json"
|
||||
)
|
||||
```
|
||||
|
||||
**Evaluerings-dimensjoner:**
|
||||
| Use case | Metrikker |
|
||||
|----------|-----------|
|
||||
| **RAG** | Groundedness, relevance, coherence, fluency |
|
||||
| **Summarization** | ROUGE, BLEU, BERTScore, METEOR |
|
||||
| **Translation** | BLEU |
|
||||
| **Classification** | Precision, recall, accuracy, F1 |
|
||||
| **Content Safety** | Hate/violence/sexual/self-harm scores (Azure AI Content Safety) |
|
||||
|
||||
**Human Feedback Loop:**
|
||||
- **Mosaic AI Agent Framework Review App** (Databricks): UI for human reviewers
|
||||
- **Application Insights**: Thumbs up/down fra sluttbrukere
|
||||
- **Custom feedback APIs**: Integrasjon i enterprise workflows
|
||||
|
||||
**Konfidensgrad:** 95% — Built-in evaluators er godt dokumentert.
|
||||
|
||||
### 5. CI/CD for GenAIOps
|
||||
|
||||
**GenAIOps Prompt Flow Template** (Microsoft-anbefalt):
|
||||
- **Repository**: [microsoft/genaiops-promptflow-template](https://github.com/microsoft/genaiops-promptflow-template)
|
||||
- **CI/CD**: GitHub Actions eller Azure DevOps Pipelines
|
||||
- **Lifecycle**: Feature branch → PR → Dev → Staging → Production
|
||||
|
||||
**Pipeline-faser:**
|
||||
1. **PR Pipeline** (CI):
|
||||
- Flow validation
|
||||
- Unit testing av custom Python code
|
||||
- Variant experimentation
|
||||
- Evaluation runs mot test data
|
||||
2. **Dev Pipeline** (CI + CD):
|
||||
- Batch testing
|
||||
- Model/prompt registration (conditional)
|
||||
- Human-in-the-loop approval gate
|
||||
- Deployment til dev/staging endpoints
|
||||
3. **Production Pipeline** (CD):
|
||||
- Blue-green deployment
|
||||
- A/B testing (traffic splitting)
|
||||
- Canary deployment
|
||||
- Rollback capabilities
|
||||
|
||||
**Azure DevOps-integrasjon:**
|
||||
```yaml
|
||||
# Eksempel: Prompt Flow evaluation i Azure Pipelines
|
||||
- task: AzureCLI@2
|
||||
displayName: 'Run Prompt Flow Evaluation'
|
||||
inputs:
|
||||
azureSubscription: 'AzureML-ServiceConnection'
|
||||
scriptType: 'bash'
|
||||
scriptLocation: 'inlineScript'
|
||||
inlineScript: |
|
||||
az ml job create --file evaluation-job.yaml \
|
||||
--workspace-name $(ML_WORKSPACE) \
|
||||
--resource-group $(RESOURCE_GROUP)
|
||||
```
|
||||
|
||||
**Konfidensgrad:** 85% — Template er aktiv (2025), men requires customization.
|
||||
|
||||
### 6. Monitoring og Observability
|
||||
|
||||
**LLM-spesifikke overvåkningsdimensjoner:**
|
||||
|
||||
| Dimensjon | Hva overvåkes | Azure-verktøy |
|
||||
|-----------|---------------|---------------|
|
||||
| **Operational** | Latency, token usage, 429 errors, endpoint availability | Azure Monitor, Application Insights |
|
||||
| **Quality** | Groundedness, relevance, coherence, fluency (sampled) | Azure Machine Learning Model Monitoring (Generation Quality Signal) |
|
||||
| **Safety** | Harmful content detection (hate, violence, sexual, self-harm) | Azure AI Content Safety (real-time filtering) |
|
||||
| **Cost** | Token consumption per user/session, quota utilization | Azure Cost Management, API Management gateway logs |
|
||||
| **Data drift** | Changes in user query patterns, grounding data staleness | Azure ML Data Drift monitors |
|
||||
| **Feedback** | User ratings (thumbs up/down), session abandonment rate | Custom telemetry (Application Insights) |
|
||||
|
||||
**MLflow Tracing for GenAI:**
|
||||
```python
|
||||
import mlflow
|
||||
|
||||
# Automatisk tracing av OpenAI calls
|
||||
mlflow.openai.autolog()
|
||||
|
||||
# Custom trace decorators
|
||||
@mlflow.trace
|
||||
def my_rag_app(query: str):
|
||||
context = retrieve_from_vector_store(query)
|
||||
prompt = format_prompt(query, context)
|
||||
response = llm.invoke(prompt)
|
||||
return response
|
||||
```
|
||||
|
||||
**Azure AI Foundry Monitoring (SDK v2):**
|
||||
```python
|
||||
from azure.ai.ml.entities import (
|
||||
MonitorSchedule, GenerationSafetyQualitySignal,
|
||||
GenerationTokenStatisticsSignal
|
||||
)
|
||||
|
||||
# Quality monitoring
|
||||
gsq_signal = GenerationSafetyQualitySignal(
|
||||
connection_id=aoai_connection_id,
|
||||
metric_thresholds={
|
||||
"groundedness": {"aggregated_groundedness_pass_rate": 0.7},
|
||||
"relevance": {"aggregated_relevance_pass_rate": 0.7},
|
||||
},
|
||||
production_data=[production_data],
|
||||
sampling_rate=1.0
|
||||
)
|
||||
|
||||
# Token monitoring
|
||||
token_signal = GenerationTokenStatisticsSignal()
|
||||
|
||||
monitor = MonitorSchedule(
|
||||
name="genai-monitor",
|
||||
trigger=CronTrigger(expression="15 10 * * *"),
|
||||
create_monitor=MonitorDefinition(
|
||||
monitoring_signals={"quality": gsq_signal, "tokens": token_signal}
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
**Konfidensgrad:** 90% — Monitoring capabilities er dokumentert, men sampling rates må justeres for cost.
|
||||
|
||||
---
|
||||
|
||||
## Arkitekturmønstre
|
||||
|
||||
### 1. Fine-Tuning Pattern
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue