ms-ai-architect/skills/ms-ai-engineering/references/mlops-genaiops/prompt-flow-production-deployment.md

Prompt Flow and Production Deployment

Kategori: MLOps & GenAIOps Dato: 2026-02-04 Last updated: 2026-06-19 Confidence: 🟢 Høy (basert på offisiell Microsoft-dokumentasjon fra Microsoft Foundry og Azure Machine Learning)

---

⚠️ Retirement (verifisert mot Microsoft Learn 2026-06-19): Prompt Flow — i både Microsoft Foundry og Azure Machine Learning — pensjoneres 20. april 2027 og anbefales ikke for ny utvikling. Migrer eksisterende flows og deployments til Microsoft Agent Framework (MAF) før fristen. Web authoring-opplevelsen (Foundry + Azure ML), VS Code-utvidelsene og Prompt Flow container images (promptflow-runtime, promptflow-runtime-stable, promptflow-python) får ikke lenger oppdateringer, inkludert sikkerhetsoppdateringer. Migrasjonsguide. Innholdet under beskriver fortsatt gjeldende Prompt Flow-praksis for eksisterende løsninger frem til fristen.

Introduksjon

Prompt Flow er Microsofts rammeverk for å utvikle, teste og deploye LLM-baserte applikasjoner gjennom en visuell workflow-editor. Produksjonsdeployment av Prompt Flow handler om å ta en testet og evaluert flow fra utviklingsmiljø til skalerbar produksjon med robuste CI/CD-pipelines, overvåking og governance.

Dette dokumentet dekker hele produksjonsdeployment-spekteret: fra lokal utvikling til Azure Managed Online Endpoints, CI/CD-integrering, monitoring med Application Insights, og GenAIOps-praksiser for LLM-baserte applikasjoner.

Hvorfor dette er kritisk for produksjon:

• Lifecycle management: Strukturert overgang fra eksperiment til produksjon med versjonshåndtering
• Skalerbarhet: Automatisk skalering av endpoints basert på trafikk
• Observability: Fullstendig trace, metrics og logging via Application Insights
• Governance: Model registry, conditional registration, og audit trails
• Continuous deployment: Automatisert testing, evaluering og deployment via GitHub Actions eller Azure DevOps

---

Kjernekomponenter

1. Flow Development Lifecycle

Prompt Flow følger en fire-fase livssyklus:

Initialisering

• Definer business objective og samle sample data
• Bygg basic prompt structure i Prompt Flow editor (DAG-basert)
• Utvikle flow med nodes (LLM, Python, prompts) og connections

Eksperimentering

• Kjør flow mot sample data i Microsoft Foundry eller VS Code extension
• Test single inputs og batch runs
• Iterer på prompt variants og node-konfigurasjoner

Evaluering og refinement

• Kjør batch runs mot større datasett
• Bruk built-in evaluation flows (groundedness, relevance, etc.)
• Sammenlign variants og hyperparameters
• Register model i Azure Machine Learning Model Registry ved godkjente resultater

Produksjon

• Deploy til Azure Managed Online Endpoint eller Kubernetes
• Aktiver Application Insights for tracing og metrics
• Implementer A/B deployment for gradvis rollout
• Monitor performance og samle user feedback

2. Deployment Targets

Azure Managed Online Endpoint (anbefalt for de fleste scenarier)

• Fully managed infrastruktur med autoscaling
• Integrated med Azure RBAC og managed identities
• Built-in support for A/B testing via traffic splitting
• Krever Microsoft.PolicyInsights resource provider registrert

Kubernetes Online Endpoint

• For on-premises eller hybrid scenarios
• Krever custom instance types og manuell infrastruktur-oppsett
• Nyttig for air-gapped environments

Docker/Custom Platforms

• Flow kan eksporteres som Docker image basert på promptflow-runtime-stable base image
• Deploy til Azure App Service, Azure Container Apps, eller on-prem
• Krever custom monitoring-oppsett

3. Environment Configuration

Base Image

• Default: mcr.microsoft.com/azureml/promptflow/promptflow-runtime-stable:latest
• Kan spesifiseres i flow.dag.yaml under environment section
• Støtter custom images for private feeds eller spesialiserte dependencies

Requirements.txt

• Plasseres i flow root folder
• Dependencies installeres automatisk ved deployment
• Eksempel:

  openai>=1.0.0
  azure-identity
  promptflow-tools
  

Environment Variables

• Settes i deployment YAML under environment_variables
• Kritiske variabler:
  • APPLICATIONINSIGHTS_CONNECTION_STRING: For tracing til custom App Insights
  • PROMPTFLOW_SERVING_ENGINE: flask (default) eller fastapi (fra SDK 1.10.0+)
  • PROMPTFLOW_WORKER_NUM: Antall worker prosesser (default = CPU cores)
  • PROMPTFLOW_WORKER_THREADS: Threads per worker (default = 1, kun hvis flow er thread-safe)
  • PRT_CONFIG_OVERRIDE: Connection overrides for deployment

4. Deployment Process (Azure Foundry Portal)

Steg 1: Forbered Flow

1. Test flow grundig med batch runs og evaluations
2. Verifiser at connections fungerer (Azure OpenAI, AI Search, etc.)
3. Sjekk at requirements.txt inneholder alle dependencies

Steg 2: Deploy fra UI

1. Velg Deploy i flow editor eller run detail page

2. Konfigurer Basic Settings:

   • Endpoint name (nytt eller eksisterende)
   • Deployment name (unique per endpoint)
   • Virtual machine type (Standard_DS3_v2, Standard_F4s_v2, etc.)
   • Instance count (minimum 3 for high availability)
   • Inference data collection (enable for monitoring)

3. Advanced Settings - Endpoint:

   • Authentication type: Key-based (persistent keys) eller Token-based (rotating tokens)
   • Identity type: System-assigned (auto-created) eller User-assigned (pre-created)
   • For User-assigned: Grant Azure Machine Learning Workspace Connection Secrets Reader før deployment

4. Advanced Settings - Deployment:

   • Environment: Custom eller default (basert på flow.dag.yaml)
   • Tags for organisering
   • Application Insights diagnostics: Enable for tracing

5. Advanced Settings - Outputs & Connections:

   • Velg hvilke flow outputs som inkluderes i endpoint response
   • Override connections hvis produksjon bruker andre enn dev

Steg 3: Grant Permissions

• For System-assigned identity: Assign Azure Machine Learning Workspace Connection Secrets Reader role
• For connections med Entra ID auth (f.eks. Azure OpenAI): Assign Cognitive Services OpenAI User role
• For User-assigned: Grant ACR Pull + Storage Blob Data Reader på hub registry/storage

Deployment tar 15-20 minutter (endpoint creation, model registration, deployment creation)

5. CI/CD Integration

GitHub Actions Workflow (GenAIOps Template)

Microsoft tilbyr genaiops-promptflow-template med følgende process:

1. Feature branch → Dev branch (PR):

   • Build validation pipeline kjører
   • Experimentation flows testes
   • Manual approval kreves

2. Dev branch (merge):

   • CI pipeline kjører experimentation + evaluation flows sekvensielt
   • Registrerer flows i Azure ML Model Registry hvis metrics passerer threshold
   • CD pipeline deployer til dev environment (managed endpoint)
   • Integration og smoke tests kjøres

3. Dev → Release branch (PR):

   • Samme CI/CD loop for prod environment
   • A/B deployment støttes via traffic splitting

Key GitHub Actions Steps:

- name: Install promptflow CLI
  run: pip install promptflow promptflow-tools promptflow[azure]

- name: Run flow
  run: pf run create --flow <path> --data <data.jsonl>

- name: Evaluate flow
  run: pf run create --flow <eval-flow> --run <base-run-id>

- name: Register model
  run: az ml model create --name <model> --path <flow-path>

- name: Deploy endpoint
  run: az ml online-deployment create --file deployment.yml

Azure DevOps Pipelines:

• Tilsvarende struktur med Azure DevOps tasks
• Bruk AzureCLI@2 task for az ml commands
• Service principal autentisering via Azure Service Connection

6. Model Registry and Versioning

Conditional Registration:

• GenAIOps template registrerer kun nye versjoner hvis:
  • Dataset har endret seg (SHA hash comparison)
  • Evaluation metrics overstiger threshold
  • Flow definition har endret seg

Registration Format:

name: my-flow-model
version: 1
type: mlflow_model
path: azureml://jobs/<job-id>/outputs/artifacts/paths/model
properties:
  azureml.promptflow.source_flow_id: <flow-name>

Registry Benefits:

• Cross-workspace sharing av models
• Lineage tracking til training jobs
• Role-based access control per model
• Tagging for lifecycle stages (dev, staging, prod)

---

Arkitekturmønstre

Pattern 1: Single Environment Deployment

Bruk når:

• Enkel applikasjon uten kompleks governance
• Liten team med begrenset DevOps-kapasitet
• Proof-of-concept eller interne tools

Arkitektur:

Developer → Microsoft Foundry Portal → Manual Deploy → Single Endpoint

Fordeler:

• Rask time-to-deployment
• Ingen CI/CD overhead
• Enkel å forstå for ikke-DevOps-team

Ulemper:

• Ingen automated testing
• Mangler audit trail
• Vanskelig rollback

Pattern 2: Multi-Stage CI/CD Pipeline

Bruk når:

• Enterprise produksjon med compliance krav
• Team med DevOps/Platform engineering
• Kritiske applikasjoner med SLA

Arkitektur:

Feature Branch → PR → Dev CI/CD → Dev Endpoint
                                ↓
                          Manual Gate
                                ↓
               Release Branch → Prod CI/CD → Prod Endpoint (Blue-Green)

Fordeler:

• Automated evaluation og quality gates
• Audit trail via Git history
• Rollback via pipeline re-run
• A/B testing support

Ulemper:

• Høyere initial setup cost
• Krever CI/CD infrastruktur

Pattern 3: A/B Deployment for Gradual Rollout

Bruk når:

• Testing ny prompt versjon i produksjon
• Risikoreduksjon ved store endringer
• Data-driven prompt optimization

Arkitektur:

Endpoint: my-flow-endpoint
├── Deployment A (80% traffic): v1.0 (current stable)
└── Deployment B (20% traffic): v2.0 (new variant)

Implementation (Azure CLI):

# Deploy new version
az ml online-deployment create --name v2 --endpoint my-flow-endpoint \
  --file deployment-v2.yml --traffic-percentage 20

# Gradvis øk traffic
az ml online-endpoint update --name my-flow-endpoint \
  --traffic "v1=50 v2=50"

# Full rollout
az ml online-endpoint update --name my-flow-endpoint \
  --traffic "v2=100"

Pattern 4: Local-to-Cloud Development Loop

Bruk når:

• Rapid iteration på prompts
• Team collaboration på flows
• Hybrid dev environment (local + cloud compute)

Workflow:

1. Local Dev (VS Code + Prompt Flow extension)
   ↓
2. Test locally: pf flow test --flow .
   ↓
3. Submit batch run to cloud: pf run create --runtime serverless
   ↓
4. View results i Azure ML Studio
   ↓
5. Export flow til Git → CI/CD pipeline

Fordeler:

• Fast iteration cycle
• Cloud compute for batch testing
• Version control via Git

---

Beslutningsveiledning

Når velge Managed Online Endpoint vs. Kubernetes?

Kriterium	Managed Endpoint	Kubernetes Endpoint
Infrastruktur-overhead	Ingen (fully managed)	Høy (cluster management)
Skalerbarhet	Auto-scaling built-in	Manual HPA setup
Kostnads-transparens	Pay-per-instance	Cluster overhead + instances
Hybrid/On-prem	Nei (Azure only)	Ja (AKS eller on-prem K8s)
Compliance	Standard Azure compliance	Custom compliance setup
Anbefalt for	De fleste scenarier	Hybrid cloud, air-gapped

Anbefaling for offentlig sektor: Managed Endpoint i utgangspunktet, Kubernetes kun hvis hybrid cloud eller on-prem er lovpålagt.

Når bruke FastAPI vs. Flask serving engine?

Faktor	Flask (default)	FastAPI
Tilgjengelighet	Alle SDK-versjoner	SDK >= 1.10.0
Ytelse	Stabil, proven	Høyere throughput (async)
Concurrency	Process-based (multi-worker)	Async event loop + multi-worker
Thread-safety	Mindre kritisk	Krever thread-safe flow code

Aktivering:

environment_variables:
  PROMPTFLOW_SERVING_ENGINE: fastapi

Anbefaling: Start med Flask (default), switch til FastAPI hvis latency/throughput blir bottleneck OG flow code er thread-safe.

Concurrency Tuning

Formula:

max_concurrent_requests_per_instance = worker_num × worker_threads × multiplier

hvor multiplier =
  - 1.0 hvis request time > 200ms (CPU-bound)
  - 1.5-2.0 hvis request time <= 200ms (I/O-bound)

Eksempel for 4-core VM med 100ms request time:

request_settings:
  max_concurrent_requests_per_instance: 12  # 4 workers × 1 thread × 1.5

environment_variables:
  PROMPTFLOW_WORKER_NUM: 4
  PROMPTFLOW_WORKER_THREADS: 1

---

Integrasjon med Microsoft-stakken

Microsoft Foundry Integration

Flow Development:

• Drag-and-drop DAG editor for LLM, Python, Prompt nodes
• Built-in connections til Azure OpenAI, AI Search, Content Safety
• Variant experimentation (side-by-side prompt comparison)

Compute Session Management:

• Serverless compute (on-demand, billed per minute)
• Compute instance (dedicated, faster startup for iteration)
• Automatisk pause etter inaktivitet

Deployment Lifecycle:

• Flow → Test → Batch Run → Evaluation → Model Registry → Endpoint
• All steps traceable via Foundry portal UI

Azure Machine Learning Integration

Model Registry:

• Cross-workspace sharing via Azure ML Registry (multi-region)
• Lineage tracking: flow → training job → dataset
• RBAC per model version

Endpoints & Deployments:

• Same infrastructure som standard ML model deployments
• Supports managed identities for secure connection access
• Integrated med Azure Monitor for operational metrics

Application Insights Integration

Tracing:

• OpenTelemetry-compliant trace data
• Captures: LLM calls, node execution, token consumption, latency
• Transaction search for debugging individual requests

Metrics (under promptflow standard metrics namespace):

• token_consumption (counter): prompt_tokens, completion_tokens, total_tokens
• flow_latency (histogram): end-to-end request time
• flow_request (counter): request count per flow
• node_latency / node_request: per-node breakdown
• rpc_latency / rpc_request: external API call metrics
• flow_streaming_response_duration: for streaming endpoints

Enabling:

# I deployment.yml
app_insights_enabled: true

# Eller custom App Insights:
environment_variables:
  APPLICATIONINSIGHTS_CONNECTION_STRING: "InstrumentationKey=...;IngestionEndpoint=..."

Viewing Metrics:

1. Azure Portal → Application Insights → Metrics
2. Metric Namespace: promptflow standard metrics
3. Metric: Velg fra dropdown (token_consumption, flow_latency, etc.)
4. Split by dimension: flow, node, response_code

Feedback Collection API

Prompt Flow serving eksponerer /feedback endpoint for post-inference feedback:

Request:

POST https://<endpoint>.azureml.ms/feedback
Authorization: Bearer <token>
Content-Type: application/json
traceparent: 00-<trace-id>-<span-id>-01

{
  "rating": 5,
  "comment": "Excellent answer",
  "user_id": "user@example.com"
}

Trace Correlation:

• traceparent header linker feedback til original request trace
• Feedback lagres som span i Application Insights
• Enables correlation analysis (latency vs. rating, etc.)

Azure DevOps Integration

Pipeline Tasks:

• AzureCLI@2: For az ml commands
• PythonScript@0: For pf CLI commands
• PublishPipelineArtifact@1: Publish evaluation reports (CSV, HTML)

Artifact Management:

• Flow folder lagres i Azure Repos
• Evaluation results publiseres som pipeline artifacts
• Model versions linkes til Git commits

---

Offentlig sektor (Norge)

Compliance ved Deployment

Krav fra Digdir:

• Etterprøvbarhet: CI/CD pipeline gir audit trail (Git commits, pipeline runs, model versions)
• Versjonskontroll: Model registry sporer alle versjoner med lineage til training data
• Tilgangskontroll: Managed identities + Azure RBAC sikrer least privilege
• Datahåndtering: Inference data collection kan disabled hvis personvern krever det

DPIA for Deployment:

• Vurder om inference logs inneholder persondata (aktiveres via inference_data_collection)
• Application Insights trace data kan inneholde brukerinput → anonymiser i production
• Feedback API må ha consent-mekanisme hvis brukerdata lagres

Utredningsinstruksen: Teknologivalg

Deployment Target:

• Managed Endpoint: Standard valg, dokumenter kostnads-modell (instance count × VM cost)
• Kubernetes: Kun hvis hybrid cloud er påkrevd, dokumenter driftskostnader
• Docker on-prem: Kun hvis sky ikke er tillatt, dokumenter security patching-ansvar

Alternativer-analyse:

Alternativ	Fordel	Ulempe
Managed Endpoint	Fully managed, auto-scaling	Azure lock-in, cloud-only
AKS	Hybrid, full kontroll	Høy driftskostnad
On-prem Docker	Ingen sky-avhengighet	Manuell skalering, patching

Anbefaling: Managed Endpoint med fallback til AKS hvis hybrid cloud er lovpålagt.

ROS-analyse: Deployment Risiko

Trussel	Sannsynlighet	Konsekvens	Tiltak
Endpoint key leak	Middels	Høy	Bruk Token-based auth (roterende) + Key Vault
Connection credentials i logs	Lav	Høy	Disable inference data collection
Unauthorized model update	Lav	Middels	RBAC på Model Registry + approval gates
DDoS på endpoint	Middels	Middels	Azure DDoS Protection + rate limiting

---

Kostnad og lisensiering

Deployment Kostnader

Managed Online Endpoint:

Kostnad = (VM cost per hour × instance count × uptime hours)
        + (Azure ML deployment overhead)
        + (Application Insights ingestion + retention)

Eksempel (Standard_DS3_v2, 3 instances, 24/7):

• VM cost: ~70 NOK/time × 3 instances × 730 timer/måned = ~153 300 NOK/måned
• Application Insights: ~1000-5000 NOK/måned (avhengig av trace volume)
• Total: ~155 000-160 000 NOK/måned

Kostnadsoptimalisering:

• Bruk autoscaling (min 1 instance, max 5) for variabel trafikk
• Scheduled scaling (nedskalering utenfor arbeidstid)
• Reserved instances for forutsigbar last (opptil 72% rabatt)

Compute Session (Development):

• Serverless: ~5 NOK/time, billed per minute
• Compute instance: ~60-150 NOK/time avhengig av size, billed hourly
• Auto-pause etter 30 min inaktivitet (konfigurerbar)

Lisensiering

Microsoft Foundry:

• Included i Azure subscription, ingen separat lisens
• Betaler kun for underliggende resources (compute, storage, AI services)

Prompt Flow:

• Open source (MIT license) + Azure-managed variant
• Ingen lisenskostnad for SDK/CLI
• Azure-managed deployment krever Azure ML workspace (ingen ekstra lisens)

Nødvendige Azure Services:

• Azure Machine Learning workspace (gratis, betaler kun for compute/storage)
• Application Insights (pay-as-you-go)
• Optional: Azure ML Registry for cross-workspace sharing (ingen ekstra kostnad)

---

For arkitekten (Cosmo)

Når anbefale Prompt Flow Deployment?

Sterk anbefaling når:

• Kunden allerede bruker Microsoft Foundry for LLM-utvikling
• Behov for visuell DAG-editor (forenkler kommunikasjon med ikke-tekniske stakeholders)
• Team mangler dyp MLOps-kompetanse (Prompt Flow abstraherer bort mye kompleksitet)
• Krav om rapid iteration på prompts (variant experimentation built-in)

Vurder alternativer når:

• Kunden har eksisterende MLOps pipeline (f.eks. Kubeflow, MLflow) → integrer Prompt Flow som model format
• Kompleks custom orchestration logic → Semantic Kernel eller LangChain kan være bedre fit
• Pure API-basert workflow uten visuell editor-behov → Azure Functions + Azure OpenAI direkte

Red Flags å se etter

Deployment Anti-patterns:

• Deploying direkte fra developer laptop → alltid bruk CI/CD
• Hardkoding connection credentials i flow → bruk Azure Key Vault references
• Ingen evaluations før deployment → alltid kjør eval flows
• Single instance deployment for produksjon → minimum 3 instances for HA
• Ingen Application Insights → umulig å debugge production issues

Cost Traps:

• 24/7 high-end VMs uten autoscaling → kan koste 100K+ NOK/måned unødvendig
• Inference data collection enabled uten retention policy → App Insights storage kosten eksploderer
• Compute sessions som ikke auto-pauserer → betaler for idle compute

Spørsmål å stille kunden

1. Development Process: "Hvordan itererer teamet på prompts i dag? Lokalt eller i sky?"

   • Steer til: Local dev (VS Code) → cloud batch testing → CI/CD deployment

2. Deployment Frequency: "Hvor ofte oppdaterer dere prompts/flows i produksjon?"

   • Hvis daglig/ukentlig: CI/CD er kritisk
   • Hvis månedlig+: Manual deployment kan aksepteres

3. Traffic Pattern: "Er trafikken konstant eller variabel (dag vs. natt, virkedag vs. helg)?"

   • Hvis variabel: Autoscaling er must-have
   • Hvis konstant: Reserved instances for kostnadskutt

4. Compliance: "Har dere krav om on-prem eller hybrid cloud?"

   • Hvis ja: Kubernetes endpoint eller Docker export
   • Hvis nei: Managed endpoint (default)

5. Monitoring: "Hvordan måler dere kvalitet på LLM-output i dag?"

   • Hvis ingen: Setup evaluation flows + App Insights metrics
   • Hvis eksisterende: Integrer med /feedback API

Decision Tree: Deployment Strategy

Er dette første gang kunden deployer LLM-basert app?
├─ Ja → Start med Managed Endpoint + Manual deployment (rask learning)
│        Etter 1-2 måneder → Introduser CI/CD pipeline
│
└─ Nei (har erfaring) → Direkte til CI/CD pipeline
                        ├─ GitHub brukt? → GitHub Actions template
                        └─ Azure DevOps brukt? → Azure Pipelines template

Eksempel på anbefaling (offentlig sektor use case)

Scenario: NAV skal deploye chatbot for sykepenger-spørsmål.

Anbefalt arkitektur:

1. Development: Microsoft Foundry → Prompt Flow editor (DAG-basert)
2. CI/CD: GitHub (NAV sin standard) + GenAIOps template
   • Feature branch: PR trigger → build validation
   • Main branch: CI trigger → evaluation → model registry → dev endpoint
   • Prod branch: Manual approval gate → prod endpoint
3. Deployment: Managed Online Endpoint
   • 3 instances (Standard_DS3_v2) med autoscaling 1-5
   • Token-based auth (roterende credentials)
   • System-assigned managed identity
4. Monitoring: Application Insights
   • Token consumption metrics (budsjettsporing)
   • Latency metrics (SLA tracking)
   • Custom feedback via /feedback API (brukertilfredshet)
5. Compliance:
   • Inference data collection DISABLED (personvern)
   • Model registry for versjonssporing (etterprøvbarhet)
   • RBAC på endpoint + model registry (tilgangskontroll)

Kostnadsestimat:

• Deployment: ~160 000 NOK/måned (3 instances 24/7)
• Compute sessions (dev): ~10 000 NOK/måned (5 utviklere, 4 timer/dag)
• Application Insights: ~3 000 NOK/måned
• Total: ~173 000 NOK/måned

Alternativ (kostnadsoptimalisert):

• Autoscaling 1-3 instances med scheduled scaling (08:00-16:00 virkedager)
• Reserved instances (1-year commit)
• Redusert kostnad: ~80 000 NOK/måned

---

Kilder og verifisering

Microsoft Learn Dokumentasjon:

1. Deploy a flow for real-time inference (Microsoft Foundry) – Offisiell guide for deployment via portal
2. GenAIOps with Prompt Flow and GitHub – CI/CD pipeline patterns og lifecycle management
3. Enable tracing and collect feedback for a flow deployment – Application Insights integration og metrics
4. Deploy a flow to online endpoint with CLI/SDK – Advanced deployment configuration (concurrency, FastAPI, etc.)
5. Integrate Prompt Flow with DevOps – Local-to-cloud development workflow

GitHub Resources:

• GenAIOps Prompt Flow Template – Reference implementation for CI/CD
• Prompt Flow SDK Examples – Code samples for deployment automation

Verifisert: 2026-06-19 via microsoft-learn MCP server (søk + fetch på 5 offisielle docs)