ms-ai-architect/skills/ms-ai-governance/references/monitoring-observability/custom-dashboards-ai-operations.md

Custom Dashboards for AI Operations

Category: Monitoring & Observability Last updated: 2026-06-19 Brukes av: Cosmo Skyberg, Microsoft AI Solution Architect Type: reference Source: https://learn.microsoft.com/azure/azure-monitor/visualize/workbooks-overview

---

Innhold

• Innledning
• Azure Workbooks for AI
• Grafana for AI Operational Dashboards
• Power BI for Executive AI Dashboards
• Real-Time Intelligence Dashboards (Fabric)
• Dashboard Sharing and Governance
• Cost and Usage Visualizations
• Dashboard Anti-Patterns
• For Cosmo Skyberg
• Ressurser

Innledning

Custom dashboards er essensielle for å visualisere og forstå AI-driften i sanntid. Mens standard metrics-visninger gir grunnleggende innsikt, tilbyr tilpassede dashboards mulighet til å kombinere data fra flere kilder, skreddersy visualiseringer for ulike interessenter, og bygge operasjonelle kommandosentral for AI-systemer.

Microsoft-stakken tilbyr flere dashboarding-løsninger med ulike styrker: Azure Workbooks for teknisk dybde, Grafana for operasjonelle sanntidsvisninger, Power BI for executive insights, og Real-Time Intelligence dashboards for streaming-data.

---

Azure Workbooks for AI

Azure Workbooks er Microsofts native dashboarding-løsning i Azure Monitor. De kombinerer tekst, KQL-queries, metrics, og interaktive parametere i én fleksibel canvas.

Hvorfor Workbooks for AI-monitoring?

• Unified data sources: Kombinerer Application Insights, Log Analytics, metrics, og Azure Resource Graph i én view
• KQL-powered: Direkte tilgang til Kusto Query Language for avanserte aggregeringer
• Template-drevet: Distribuer standardiserte dashboards programmatisk via ARM templates
• Resource-centric: Visualiser data på tvers av flere AI-ressurser samtidig
• Built-in for AI Foundry: Microsoft Foundry leverer ferdig "Application Analytics" workbook

Microsoft Foundry Application Analytics Workbook

Microsoft Foundry tilbyr en out-of-box workbook som sporer:

• Generative AI metrics: Total conversations, latency, exceptions
• Tool usage: Hvilke extensions og tools brukes mest
• Topic analytics: Hvilke conversation topics dominerer
• Operational health: Success rates, error patterns, response times

Tilgang:

1. Gå til Application Insights ressurs
2. Velg Monitoring → Workbooks
3. Åpne "Copilot Studio Dashboard" fra galleriet

Tilpasning:

// Eksempel: Track custom attribute for AI responses
customEvents
| where name == "AIResponse"
| extend ResponseQuality = tostring(customDimensions.quality)
| summarize Count = count() by ResponseQuality, bin(timestamp, 1h)
| render timechart

Workbook Architecture for AI

Typiske seksjoner i et AI operations workbook:

1. Executive summary (stat tiles)

   • Total requests today
   • Average latency
   • Token consumption
   • Success rate

2. Request trends (timecharts)

   • API calls per hour
   • Per-model distribution
   • Geographic distribution

3. Token economics (barcharts)

   • Token usage by deployment
   • Cost per request
   • Top consumers

4. Error analysis (grids + pie charts)

   • Error codes by frequency
   • Failed requests by model
   • Retry patterns

5. Performance drill-down (interactive queries)

   • Parametere for time range, model, region
   • Query-backed visualizations som oppdateres live

Programmatic Deployment

Workbooks kan deployes via ARM templates for consistency across teams:

{
  "name": "ai-operations-workbook",
  "type": "microsoft.insights/workbooks",
  "location": "[resourceGroup().location]",
  "apiVersion": "2018-06-17-preview",  // For workbook instances; workbook templates bruker 2019-10-17-preview (workbooktemplates resource type). Bicep støttes nå offisielt som alternativ til ARM JSON. *(Verified MCP 2026-04)*
  "properties": {
    "displayName": "AI Operations Dashboard",
    "serializedData": "{\"version\":\"Notebook/1.0\",\"items\":[...]}",
    "category": "AI Monitoring",
    "sourceId": "[resourceId('Microsoft.Insights/components', parameters('appInsightsName'))]"
  }
}

Best practices: (Verified MCP 2026-04)

• Bruk parametere for time ranges og resource filters
• Inkluder markdown-tekst for kontekst og aksjonspunkter
• Legg til links til troubleshooting-docs
• Del workbooks via Azure RBAC (Monitoring Contributor for redigering, Monitoring Reader for visning)
• Vurder Bicep-templates som alternativ til ARM JSON for ny infrastruktur (støttes nå offisielt)

---

Grafana for AI Operational Dashboards

Azure Managed Grafana er ideell for sanntids-operasjonssentre. Grafana excels i streaming visualizations, multi-source aggregation, og alert-integrasjon.

Microsoft Foundry Grafana Dashboard (Verified MCP 2026-04)

Microsoft tilbyr en ferdig Grafana dashboard (ID: 24039) for Microsoft Foundry/Foundry ressurser. Dashboard-tittelen er nå "Microsoft Foundry dashboard" i offisiell dokumentasjon.

Key metrics:

• Model performance: Inference latency (time to last byte), throughput, success rates
• Token tracking: Total tokens, prompt tokens, completion tokens
• Request trends: API call volume per deployment
• Cost visibility: Token consumption patterns for cost optimization
• Per-deployment comparison: Side-by-side metrics for GPT-4 vs GPT-3.5

Import prosess:

1. Gå til Azure Managed Grafana workspace
2. Dashboards → New → Import
3. Enter dashboard ID: 24039
4. Velg Azure Monitor data source
5. Assign Monitoring Reader role til Grafana managed identity

Alternativ (direktelenke fra Azure Portal): Monitor → Dashboards with Grafana (preview) → AI Foundry (Verified MCP 2026-04)

Metric namespace: Microsoft.CognitiveServices/accounts

Key metrics:

• AzureOpenAIRequests – API call volume and success rates
• TokenTransaction – Total inference tokens for cost tracking
• ProcessedPromptTokens – Input tokens consumed
• GeneratedTokens – Output tokens produced
• AzureOpenAITTLTInMS – Inference latency (time to last byte)

Grouping: All metrics split by ModelDeploymentName

Custom Grafana Panels

Legg til nytt panel:

1. Edit → Add → Visualization
2. Data source: Azure Monitor
3. Resource: Velg AI Foundry resource
4. Metric: Velg metric (f.eks. TokenTransaction)
5. Aggregation: Average, Sum, Count, Min, Max
6. Visualization type: Time series, Stat, Gauge, Bar chart
7. Thresholds: Definer warning/critical levels for visual alerts

Eksempel på custom panel for token cost:

• Data source: Azure Monitor
• Metric: TokenTransaction
• Aggregation: Sum
• Transform: Math operation × 0.000002 (cost per token in NOK)
• Visualization: Stat panel med "NOK spent today"
• Threshold: Red over 5000 NOK

---

Power BI for Executive AI Dashboards

Power BI tilbyr business-orienterte visualiseringer med kraftig datamodellering. Ideell for executive dashboards som kombinerer AI metrics med business KPIs.

Power BI + Azure Monitor Integration

Dataflyt:

1. Azure Monitor logs → Log Analytics workspace
2. Power BI connector → Import eller DirectQuery
3. Power BI semantic model → Transform og model data
4. Power BI report → Visualiser for executives

Setup:

1. I Power BI Desktop: Get Data → Azure → Azure Monitor Logs
2. Enter workspace resource ID
3. Write KQL query:

AzureDiagnostics
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
| where TimeGenerated > ago(30d)
| summarize
    TotalRequests = count(),
    AvgLatency = avg(DurationMs),
    TotalTokens = sum(toint(customDimensions.tokens))
  by bin(TimeGenerated, 1d), ModelDeployment = tostring(customDimensions.model)

Executive Dashboard Layout

Typical executive AI dashboard:

1. Top KPIs (cards)

   • Monthly AI spend
   • Total conversations handled
   • Average user satisfaction (fra feedback)
   • Cost per interaction

2. Trends (line charts)

   • AI usage growth over time
   • Cost efficiency trend
   • User adoption rate

3. Business impact (combo charts)

   • Support tickets vs AI conversations (korrelasjon)
   • Customer satisfaction vs AI usage
   • Cost savings from automation

4. Model performance (tables)

   • Ranker modeller etter success rate, cost, speed
   • Benchmark mot SLA

Scheduling:

• Scheduled refresh, deling, dataflows og incremental refresh krever Power BI Pro eller Premium (gratis-tier dekker kun lokal rapport-/dashboardbygging). Incremental refresh forutsetter et datetime-felt i resultatsettet.
• Email subscriptions for stakeholders
• Power BI mobile app for on-the-go access

---

Real-Time Intelligence Dashboards (Fabric)

Microsoft Fabric Real-Time Intelligence tilbyr sanntids-dashboards drevet av KQL queries mot Eventhouse.

AI Monitoring i Fabric

Use case: Streaming AI telemetry for øyeblikkelig innsikt.

Architecture:

1. Microsoft Foundry → Event Hub → Fabric Eventhouse
2. KQL Database → Continuous queries
3. Real-Time Dashboard → Live visualizations

Dashboard tiles:

Stat tile (max temperature pattern):

AITelemetry
| where Timestamp between (_startTime.._endTime)
| where ModelDeploymentName == _deployment
| top 1 by Latency desc
| summarize by Latency

Time chart (request rate):

AITelemetry
| where Timestamp between (_startTime.._endTime)
| where ModelDeploymentName == _deployment
| summarize RequestCount = count() by bin(Timestamp, 1m)
| render timechart

Parameters:

// Deployment selector
AITelemetry
| summarize by ModelDeploymentName

Best practices:

• Bruk parameters for interactive filtering
• Auto-refresh interval: 30 sek for operations, 5 min for analytics
• Conditional formatting for thresholds (red/yellow/green)

---

Dashboard Sharing and Governance

Access Control

Azure Workbooks: (Verified MCP 2026-04)

• Monitoring Contributor role: Inkluderer workbooks/write — kan redigere og lagre delte workbooks
• Monitoring Reader role: Kan se workbooks, men ikke endre
• Custom roles: Krev microsoft.insights/workbooks/write for redigering
• Resource-based permissions: Brukere ser kun data fra ressurser de har tilgang til

Grafana:

• Grafana Admin role: Full tilgang
• Grafana Editor role: Kan redigere dashboards
• Grafana Viewer role: Read-only
• Azure RBAC: Monitoring Reader på subscription/resource group

Power BI:

• Workspace roles: Admin, Member, Contributor, Viewer
• Row-level security (RLS): Filtrer data basert på brukeridentitet
• App distribution: Del read-only versjon via Power BI app

Governance Best Practices

Standardisering:

• Opprett dashboard templates for ulike roller (DevOps, Leadership, Security)
• Bruk naming conventions: [Team]-[Purpose]-[Environment] (f.eks. AITeam-Operations-Prod)
• Version control for workbook ARM templates i Git

Dokumentasjon:

• Inkluder markdown-seksjoner i workbooks med:
  • Hva viser denne dashboard?
  • Hvilke actions skal jeg ta ved alerts?
  • Links til runbooks og troubleshooting guides
• README i Power BI workspace med metric definitions

Update cadence:

• Operations dashboards: Live/1 min refresh
• Analytics dashboards: 15 min refresh
• Executive dashboards: Daily refresh (for kostnad-effektivitet)

Arkivering:

• Fjern dashboards som ikke har vært brukt på 90 dager
• Eksporter historiske dashboards som snapshots (PDF fra Grafana, PBIX backup)

---

Cost and Usage Visualizations

Token Economics Dashboard

Kritisk for AI-budsjett: Visualiser token costs i sanntid.

KQL query for daily cost:

AzureDiagnostics
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
| where OperationName == "ChatCompletions_Create"
| extend
    PromptTokens = toint(customDimensions.prompt_tokens),
    CompletionTokens = toint(customDimensions.completion_tokens),
    Model = tostring(customDimensions.model)
| extend TotalCost = case(
    Model == "gpt-4", (PromptTokens * 0.00003 + CompletionTokens * 0.00006),
    Model == "gpt-35-turbo", (PromptTokens * 0.0000015 + CompletionTokens * 0.000002),
    0
  )
| summarize DailyCost = sum(TotalCost) by bin(TimeGenerated, 1d)
| render areachart

Visualization types:

• Waterfall chart: Vis cost breakdown per model, per team, per use case
• Gauge: Daily spend vs budget
• Heat map: Peak usage hours (for PTU optimization)

PTU Utilization Dashboard

For Provisioned Throughput Units (PTU):

Key metrics:

• PTU utilization percentage
• Requests per PTU
• Cost per request (PTU vs PayGo comparison)

Grafana panel:

• Data source: Azure Monitor
• Metric: ProcessedPromptTokens + GeneratedTokens
• Transform: Divide by PTU capacity → percentage
• Visualization: Gauge med thresholds (green <80%, yellow 80-95%, red >95%)

---

Dashboard Anti-Patterns

Feil å unngå:

❌ Information overload: 20+ metrics på én side – Splitt i multiple views ❌ Stale data: Refresh rate som ikke matcher use case (real-time ops trenger <1 min) ❌ No context: Metrics uten thresholds eller trend-indikatorer ❌ Static dashboards: Ingen parameters for filtering eller drill-down ❌ Isolated metrics: Ikke kombiner business outcomes med technical metrics ❌ No alerts configured: Dashboards er reactive, du trenger proactive alerts også

Best practices:

✅ Progressive disclosure: Summary view → Drill-down details ✅ Thresholds everywhere: Visual indicators (red/yellow/green) ✅ Contextual annotations: Markdown-tekst som forklarer hva er normalt, hva er alarming ✅ Role-based views: Ulike dashboards for DevOps, managers, finance ✅ Mobile-friendly: Test på mobile devices (Grafana/Power BI mobile apps) ✅ Integration with incidents: Link fra dashboard tile til incident management (ServiceNow, Linear)

---

For Cosmo Skyberg

Når kunden spør om dashboards for AI operations:

Discovery Questions

1. Hvem er dashboardet for? (DevOps, executives, security team, finance?)
2. Hva er decision-kriteriene? (Real-time troubleshooting, cost control, compliance, capacity planning?)
3. Hvilke data sources? (Kun Azure Monitor, eller også custom app telemetry?)
4. Refresh requirements? (Live, minutt, time, daglig?)
5. Mobile access? (Grafana/Power BI mobile, eller kun desktop?)
6. Compliance constraints? (Hvem kan se hvilke data? RLS nødvendig?)

Anbefalingsmatrise

Use Case	Anbefalt Løsning	Begrunnelse
Real-time operations center	Grafana (Azure Managed)	Streaming metrics, alert-integrasjon, 24/7 NOC-friendly
Deep technical troubleshooting	Azure Workbooks	KQL-drevet, resource-centric, kan kombinere logs+metrics
Executive monthly reviews	Power BI	Business-oriented visuals, kombinerer AI med business KPIs
Streaming IoT/Edge AI telemetry	Fabric Real-Time Dashboard	Sub-second refresh, event-driven
Quick ad-hoc analysis	Log Analytics + Metrics Explorer	Ingen setup, direkte i portal

Implementation Checklist

Fase 1: Design (1-2 uker)

• Definer målgrupper og deres behov
• Skissér dashboard layout (wireframes)
• Identifiser data sources og KQL queries
• Etablér thresholds og alert-kriterier

Fase 2: Prototype (1 uke)

• Bygg workbook/Grafana dashboard med sample data
• Test queries for performance (< 5 sek load time)
• Validér med pilot-brukere

Fase 3: Production (1 uke)

• Deploy via ARM template (Workbooks) eller import (Grafana)
• Konfigurer RBAC og sharing
• Sett opp refresh schedules
• Dokumentér i README

Fase 4: Iterate (kontinuerlig)

• Samle feedback fra brukere
• Monitor dashboard usage (Application Insights for Grafana/PBI)
• Optimaliser trege queries
• Legg til nye metrics basert på operasjonelle behov

Technical Guidance

Når velge Workbooks:

• Teamet er komfortable med KQL
• Trenger resource-centric views (mange AI-ressurser samtidig)
• Ønsker programmatic deployment (IaC)
• Budget-bevisst (ingen ekstra lisenskostnad)

Når velge Grafana:

• 24/7 operations center
• Multi-cloud (kombinerer Azure med AWS/GCP metrics)
• Alert-drevet kultur (Grafana alerting er kraftig)
• Eksisterende Grafana-kompetanse

Når velge Power BI:

• Executive audience (ikke-tekniske interessenter)
• Kombinerer AI metrics med ERP/CRM data
• Trenger mobile app access
• Ønsker scheduled email reports

Når velge Fabric Real-Time:

• Sub-second latency requirements
• Massive scale (millioner av events per sekund)
• Allerede investert i Microsoft Fabric
• Event-driven architecture (Event Hub → Eventhouse)

Example Deliverables

Eksempel 1: DevOps Operations Workbook

• Sections: Health Overview, Request Trends, Error Analysis, Token Economics
• Parametere: Time range, Model deployment, Region
• Refresh: Live (1 min)
• RBAC: DevOps team (Contributor), Leadership (Reader)

Eksempel 2: Executive Grafana Dashboard

• Panels: KPI cards (top row), Time series (middle), Tables (bottom)
• Variables: Environment (prod/test), Cost center
• Refresh: 5 min
• Alerts: Email til leadership ved cost > threshold

Eksempel 3: Finance Power BI Report

• Pages: Monthly spend, Cost per business unit, Forecast vs Actual
• Data sources: Azure Monitor + Finance system (via Dataverse)
• Refresh: Daily (6 AM)
• RLS: Finance team ser all data, business units ser kun sine egne

---

Ressurser

Microsoft Learn

• Azure Workbooks overview
• Create an Microsoft Foundry dashboard
• Monitor Azure OpenAI
• Workbooks programmatic management (Verified MCP 2026-04) — ARM/Bicep deployment, RBAC (Monitoring Contributor for redigering, Monitoring Reader for visning), microsoft.insights/workbooks/write for custom roles
• Power BI + Azure Monitor (Verified MCP 2026-06-19) — eksport fra Log Analytics som M-query (.txt → Power BI Desktop) eller new Dataset (Power BI-tjenesten); dataflows + incremental refresh; scheduled refresh/deling krever Pro/Premium

Code Samples

• Workbook ARM/Bicep template samples — workbook templates bruker apiVersion 2019-10-17-preview (type: microsoft.insights/workbooktemplates); workbook instances bruker 2018-06-17-preview (type: Microsoft.Insights/workbooks) (Verified MCP 2026-04)
• Microsoft Foundry Grafana dashboard ID: 24039
• KQL query examples for AI monitoring

GitHub

• Azure Monitor Community – Workbook templates
• Grafana dashboards – Community dashboards
• Power BI samples – BI report templates

---

Status: Komplett Neste steg: Kombiner med "alert-strategies-ai-systems.md" for helhetlig monitoring approach.