ms-ai-architect/skills/ms-ai-governance/references/monitoring-observability/custom-dashboards-ai-operations.md

# Custom Dashboards for AI Operations

**Category:** Monitoring & Observability
**Last updated:** 2026-06-19
**Brukes av:** Cosmo Skyberg, Microsoft AI Solution Architect
**Type:** reference
**Source:** https://learn.microsoft.com/azure/azure-monitor/visualize/workbooks-overview

---

## Innhold

- [Innledning](#innledning)
- [Azure Workbooks for AI](#azure-workbooks-for-ai)
- [Grafana for AI Operational Dashboards](#grafana-for-ai-operational-dashboards)
- [Power BI for Executive AI Dashboards](#power-bi-for-executive-ai-dashboards)
- [Real-Time Intelligence Dashboards (Fabric)](#real-time-intelligence-dashboards-fabric)
- [Dashboard Sharing and Governance](#dashboard-sharing-and-governance)
- [Cost and Usage Visualizations](#cost-and-usage-visualizations)
- [Dashboard Anti-Patterns](#dashboard-anti-patterns)
- [For Cosmo Skyberg](#for-cosmo-skyberg)
- [Ressurser](#ressurser)

## Innledning

Custom dashboards er essensielle for å visualisere og forstå AI-driften i sanntid. Mens standard metrics-visninger gir grunnleggende innsikt, tilbyr tilpassede dashboards mulighet til å kombinere data fra flere kilder, skreddersy visualiseringer for ulike interessenter, og bygge operasjonelle kommandosentral for AI-systemer.

Microsoft-stakken tilbyr flere dashboarding-løsninger med ulike styrker: Azure Workbooks for teknisk dybde, Grafana for operasjonelle sanntidsvisninger, Power BI for executive insights, og Real-Time Intelligence dashboards for streaming-data.

---

## Azure Workbooks for AI

Azure Workbooks er Microsofts native dashboarding-løsning i Azure Monitor. De kombinerer tekst, KQL-queries, metrics, og interaktive parametere i én fleksibel canvas.

### Hvorfor Workbooks for AI-monitoring?

- **Unified data sources:** Kombinerer Application Insights, Log Analytics, metrics, og Azure Resource Graph i én view
- **KQL-powered:** Direkte tilgang til Kusto Query Language for avanserte aggregeringer
- **Template-drevet:** Distribuer standardiserte dashboards programmatisk via ARM templates
- **Resource-centric:** Visualiser data på tvers av flere AI-ressurser samtidig
- **Built-in for AI Foundry:** Microsoft Foundry leverer ferdig "Application Analytics" workbook

### Microsoft Foundry Application Analytics Workbook

Microsoft Foundry tilbyr en out-of-box workbook som sporer:

- **Generative AI metrics:** Total conversations, latency, exceptions
- **Tool usage:** Hvilke extensions og tools brukes mest
- **Topic analytics:** Hvilke conversation topics dominerer
- **Operational health:** Success rates, error patterns, response times

**Tilgang:**
1. Gå til Application Insights ressurs
2. Velg **Monitoring** → **Workbooks**
3. Åpne "Copilot Studio Dashboard" fra galleriet

**Tilpasning:**
```kusto
// Eksempel: Track custom attribute for AI responses
customEvents
| where name == "AIResponse"
| extend ResponseQuality = tostring(customDimensions.quality)
| summarize Count = count() by ResponseQuality, bin(timestamp, 1h)
| render timechart
```

### Workbook Architecture for AI

**Typiske seksjoner i et AI operations workbook:**

1. **Executive summary** (stat tiles)
   - Total requests today
   - Average latency
   - Token consumption
   - Success rate

2. **Request trends** (timecharts)
   - API calls per hour
   - Per-model distribution
   - Geographic distribution

3. **Token economics** (barcharts)
   - Token usage by deployment
   - Cost per request
   - Top consumers

4. **Error analysis** (grids + pie charts)
   - Error codes by frequency
   - Failed requests by model
   - Retry patterns

5. **Performance drill-down** (interactive queries)
   - Parametere for time range, model, region
   - Query-backed visualizations som oppdateres live

### Programmatic Deployment

Workbooks kan deployes via ARM templates for consistency across teams:

```json
{
  "name": "ai-operations-workbook",
  "type": "microsoft.insights/workbooks",
  "location": "[resourceGroup().location]",
  "apiVersion": "2018-06-17-preview",  // For workbook instances; workbook templates bruker 2019-10-17-preview (workbooktemplates resource type). Bicep støttes nå offisielt som alternativ til ARM JSON. *(Verified MCP 2026-04)*
  "properties": {
    "displayName": "AI Operations Dashboard",
    "serializedData": "{\"version\":\"Notebook/1.0\",\"items\":[...]}",
    "category": "AI Monitoring",
    "sourceId": "[resourceId('Microsoft.Insights/components', parameters('appInsightsName'))]"
  }
}
```

**Best practices:** *(Verified MCP 2026-04)*
- Bruk parametere for time ranges og resource filters
- Inkluder markdown-tekst for kontekst og aksjonspunkter
- Legg til links til troubleshooting-docs
- Del workbooks via Azure RBAC (Monitoring Contributor for redigering, Monitoring Reader for visning)
- Vurder Bicep-templates som alternativ til ARM JSON for ny infrastruktur (støttes nå offisielt)

---

## Grafana for AI Operational Dashboards

Azure Managed Grafana er ideell for sanntids-operasjonssentre. Grafana excels i streaming visualizations, multi-source aggregation, og alert-integrasjon.

### Microsoft Foundry Grafana Dashboard *(Verified MCP 2026-04)*

Microsoft tilbyr en ferdig Grafana dashboard (ID: **24039**) for Microsoft Foundry/Foundry ressurser. Dashboard-tittelen er nå "Microsoft Foundry dashboard" i offisiell dokumentasjon.

**Key metrics:**
- **Model performance:** Inference latency (time to last byte), throughput, success rates
- **Token tracking:** Total tokens, prompt tokens, completion tokens
- **Request trends:** API call volume per deployment
- **Cost visibility:** Token consumption patterns for cost optimization
- **Per-deployment comparison:** Side-by-side metrics for GPT-4 vs GPT-3.5

**Import prosess:**
1. Gå til Azure Managed Grafana workspace
2. Dashboards → New → Import
3. Enter dashboard ID: **24039**
4. Velg Azure Monitor data source
5. Assign Monitoring Reader role til Grafana managed identity

**Alternativ (direktelenke fra Azure Portal):** Monitor → Dashboards with Grafana (preview) → AI Foundry *(Verified MCP 2026-04)*

**Metric namespace:** `Microsoft.CognitiveServices/accounts`

**Key metrics:**
- `AzureOpenAIRequests` – API call volume and success rates
- `TokenTransaction` – Total inference tokens for cost tracking
- `ProcessedPromptTokens` – Input tokens consumed
- `GeneratedTokens` – Output tokens produced
- `AzureOpenAITTLTInMS` – Inference latency (time to last byte)

**Grouping:** All metrics split by `ModelDeploymentName`

### Custom Grafana Panels

**Legg til nytt panel:**
1. Edit → Add → Visualization
2. Data source: Azure Monitor
3. Resource: Velg AI Foundry resource
4. Metric: Velg metric (f.eks. `TokenTransaction`)
5. Aggregation: Average, Sum, Count, Min, Max
6. Visualization type: Time series, Stat, Gauge, Bar chart
7. Thresholds: Definer warning/critical levels for visual alerts

**Eksempel på custom panel for token cost:**
- Data source: Azure Monitor
- Metric: `TokenTransaction`
- Aggregation: Sum
- Transform: Math operation × 0.000002 (cost per token in NOK)
- Visualization: Stat panel med "NOK spent today"
- Threshold: Red over 5000 NOK

---

## Power BI for Executive AI Dashboards

Power BI tilbyr business-orienterte visualiseringer med kraftig datamodellering. Ideell for executive dashboards som kombinerer AI metrics med business KPIs.

### Power BI + Azure Monitor Integration

**Dataflyt:**
1. Azure Monitor logs → Log Analytics workspace
2. Power BI connector → Import eller DirectQuery
3. Power BI semantic model → Transform og model data
4. Power BI report → Visualiser for executives

**Setup:**
1. I Power BI Desktop: Get Data → Azure → Azure Monitor Logs
2. Enter workspace resource ID
3. Write KQL query:
```kusto
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
| where TimeGenerated > ago(30d)
| summarize
    TotalRequests = count(),
    AvgLatency = avg(DurationMs),
    TotalTokens = sum(toint(customDimensions.tokens))
  by bin(TimeGenerated, 1d), ModelDeployment = tostring(customDimensions.model)
```

### Executive Dashboard Layout

**Typical executive AI dashboard:**

1. **Top KPIs** (cards)
   - Monthly AI spend
   - Total conversations handled
   - Average user satisfaction (fra feedback)
   - Cost per interaction

2. **Trends** (line charts)
   - AI usage growth over time
   - Cost efficiency trend
   - User adoption rate

3. **Business impact** (combo charts)
   - Support tickets vs AI conversations (korrelasjon)
   - Customer satisfaction vs AI usage
   - Cost savings from automation

4. **Model performance** (tables)
   - Ranker modeller etter success rate, cost, speed
   - Benchmark mot SLA

**Scheduling:**
- Scheduled refresh, deling, dataflows og incremental refresh krever Power BI Pro eller Premium (gratis-tier dekker kun lokal rapport-/dashboardbygging). Incremental refresh forutsetter et **datetime**-felt i resultatsettet.
- Email subscriptions for stakeholders
- Power BI mobile app for on-the-go access

---

## Real-Time Intelligence Dashboards (Fabric)

Microsoft Fabric Real-Time Intelligence tilbyr sanntids-dashboards drevet av KQL queries mot Eventhouse.

### AI Monitoring i Fabric

**Use case:** Streaming AI telemetry for øyeblikkelig innsikt.

**Architecture:**
1. Microsoft Foundry → Event Hub → Fabric Eventhouse
2. KQL Database → Continuous queries
3. Real-Time Dashboard → Live visualizations

**Dashboard tiles:**

**Stat tile (max temperature pattern):**
```kusto
AITelemetry
| where Timestamp between (_startTime.._endTime)
| where ModelDeploymentName == _deployment
| top 1 by Latency desc
| summarize by Latency
```

**Time chart (request rate):**
```kusto
AITelemetry
| where Timestamp between (_startTime.._endTime)
| where ModelDeploymentName == _deployment
| summarize RequestCount = count() by bin(Timestamp, 1m)
| render timechart
```

**Parameters:**
```kusto
// Deployment selector
AITelemetry
| summarize by ModelDeploymentName
```

**Best practices:**
- Bruk parameters for interactive filtering
- Auto-refresh interval: 30 sek for operations, 5 min for analytics
- Conditional formatting for thresholds (red/yellow/green)

---

## Dashboard Sharing and Governance

### Access Control

**Azure Workbooks:** *(Verified MCP 2026-04)*
- **Monitoring Contributor role:** Inkluderer `workbooks/write` — kan redigere og lagre delte workbooks
- **Monitoring Reader role:** Kan se workbooks, men ikke endre
- **Custom roles:** Krev `microsoft.insights/workbooks/write` for redigering
- **Resource-based permissions:** Brukere ser kun data fra ressurser de har tilgang til

**Grafana:**
- **Grafana Admin role:** Full tilgang
- **Grafana Editor role:** Kan redigere dashboards
- **Grafana Viewer role:** Read-only
- Azure RBAC: Monitoring Reader på subscription/resource group

**Power BI:**
- **Workspace roles:** Admin, Member, Contributor, Viewer
- **Row-level security (RLS):** Filtrer data basert på brukeridentitet
- **App distribution:** Del read-only versjon via Power BI app

### Governance Best Practices

**Standardisering:**
- Opprett dashboard templates for ulike roller (DevOps, Leadership, Security)
- Bruk naming conventions: `[Team]-[Purpose]-[Environment]` (f.eks. `AITeam-Operations-Prod`)
- Version control for workbook ARM templates i Git

**Dokumentasjon:**
- Inkluder markdown-seksjoner i workbooks med:
  - Hva viser denne dashboard?
  - Hvilke actions skal jeg ta ved alerts?
  - Links til runbooks og troubleshooting guides
- README i Power BI workspace med metric definitions

**Update cadence:**
- **Operations dashboards:** Live/1 min refresh
- **Analytics dashboards:** 15 min refresh
- **Executive dashboards:** Daily refresh (for kostnad-effektivitet)

**Arkivering:**
- Fjern dashboards som ikke har vært brukt på 90 dager
- Eksporter historiske dashboards som snapshots (PDF fra Grafana, PBIX backup)

---

## Cost and Usage Visualizations

### Token Economics Dashboard

**Kritisk for AI-budsjett:** Visualiser token costs i sanntid.

**KQL query for daily cost:**
```kusto
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
| where OperationName == "ChatCompletions_Create"
| extend
    PromptTokens = toint(customDimensions.prompt_tokens),
    CompletionTokens = toint(customDimensions.completion_tokens),
    Model = tostring(customDimensions.model)
| extend TotalCost = case(
    Model == "gpt-4", (PromptTokens * 0.00003 + CompletionTokens * 0.00006),
    Model == "gpt-35-turbo", (PromptTokens * 0.0000015 + CompletionTokens * 0.000002),
    0
  )
| summarize DailyCost = sum(TotalCost) by bin(TimeGenerated, 1d)
| render areachart
```

**Visualization types:**
- **Waterfall chart:** Vis cost breakdown per model, per team, per use case
- **Gauge:** Daily spend vs budget
- **Heat map:** Peak usage hours (for PTU optimization)

### PTU Utilization Dashboard

For Provisioned Throughput Units (PTU):

**Key metrics:**
- PTU utilization percentage
- Requests per PTU
- Cost per request (PTU vs PayGo comparison)

**Grafana panel:**
- Data source: Azure Monitor
- Metric: `ProcessedPromptTokens` + `GeneratedTokens`
- Transform: Divide by PTU capacity → percentage
- Visualization: Gauge med thresholds (green <80%, yellow 80-95%, red >95%)

---

## Dashboard Anti-Patterns

**Feil å unngå:**

❌ **Information overload:** 20+ metrics på én side – Splitt i multiple views
❌ **Stale data:** Refresh rate som ikke matcher use case (real-time ops trenger <1 min)
❌ **No context:** Metrics uten thresholds eller trend-indikatorer
❌ **Static dashboards:** Ingen parameters for filtering eller drill-down
❌ **Isolated metrics:** Ikke kombiner business outcomes med technical metrics
❌ **No alerts configured:** Dashboards er reactive, du trenger proactive alerts også

**Best practices:**

✅ **Progressive disclosure:** Summary view → Drill-down details
✅ **Thresholds everywhere:** Visual indicators (red/yellow/green)
✅ **Contextual annotations:** Markdown-tekst som forklarer hva er normalt, hva er alarming
✅ **Role-based views:** Ulike dashboards for DevOps, managers, finance
✅ **Mobile-friendly:** Test på mobile devices (Grafana/Power BI mobile apps)
✅ **Integration with incidents:** Link fra dashboard tile til incident management (ServiceNow, Linear)

---

## For Cosmo Skyberg

Når kunden spør om dashboards for AI operations:

### Discovery Questions

1. **Hvem er dashboardet for?** (DevOps, executives, security team, finance?)
2. **Hva er decision-kriteriene?** (Real-time troubleshooting, cost control, compliance, capacity planning?)
3. **Hvilke data sources?** (Kun Azure Monitor, eller også custom app telemetry?)
4. **Refresh requirements?** (Live, minutt, time, daglig?)
5. **Mobile access?** (Grafana/Power BI mobile, eller kun desktop?)
6. **Compliance constraints?** (Hvem kan se hvilke data? RLS nødvendig?)

### Anbefalingsmatrise

| Use Case | Anbefalt Løsning | Begrunnelse |
|----------|------------------|-------------|
| Real-time operations center | Grafana (Azure Managed) | Streaming metrics, alert-integrasjon, 24/7 NOC-friendly |
| Deep technical troubleshooting | Azure Workbooks | KQL-drevet, resource-centric, kan kombinere logs+metrics |
| Executive monthly reviews | Power BI | Business-oriented visuals, kombinerer AI med business KPIs |
| Streaming IoT/Edge AI telemetry | Fabric Real-Time Dashboard | Sub-second refresh, event-driven |
| Quick ad-hoc analysis | Log Analytics + Metrics Explorer | Ingen setup, direkte i portal |

### Implementation Checklist

**Fase 1: Design (1-2 uker)**
- [ ] Definer målgrupper og deres behov
- [ ] Skissér dashboard layout (wireframes)
- [ ] Identifiser data sources og KQL queries
- [ ] Etablér thresholds og alert-kriterier

**Fase 2: Prototype (1 uke)**
- [ ] Bygg workbook/Grafana dashboard med sample data
- [ ] Test queries for performance (< 5 sek load time)
- [ ] Validér med pilot-brukere

**Fase 3: Production (1 uke)**
- [ ] Deploy via ARM template (Workbooks) eller import (Grafana)
- [ ] Konfigurer RBAC og sharing
- [ ] Sett opp refresh schedules
- [ ] Dokumentér i README

**Fase 4: Iterate (kontinuerlig)**
- [ ] Samle feedback fra brukere
- [ ] Monitor dashboard usage (Application Insights for Grafana/PBI)
- [ ] Optimaliser trege queries
- [ ] Legg til nye metrics basert på operasjonelle behov

### Technical Guidance

**Når velge Workbooks:**
- Teamet er komfortable med KQL
- Trenger resource-centric views (mange AI-ressurser samtidig)
- Ønsker programmatic deployment (IaC)
- Budget-bevisst (ingen ekstra lisenskostnad)

**Når velge Grafana:**
- 24/7 operations center
- Multi-cloud (kombinerer Azure med AWS/GCP metrics)
- Alert-drevet kultur (Grafana alerting er kraftig)
- Eksisterende Grafana-kompetanse

**Når velge Power BI:**
- Executive audience (ikke-tekniske interessenter)
- Kombinerer AI metrics med ERP/CRM data
- Trenger mobile app access
- Ønsker scheduled email reports

**Når velge Fabric Real-Time:**
- Sub-second latency requirements
- Massive scale (millioner av events per sekund)
- Allerede investert i Microsoft Fabric
- Event-driven architecture (Event Hub → Eventhouse)

### Example Deliverables

**Eksempel 1: DevOps Operations Workbook**
- Sections: Health Overview, Request Trends, Error Analysis, Token Economics
- Parametere: Time range, Model deployment, Region
- Refresh: Live (1 min)
- RBAC: DevOps team (Contributor), Leadership (Reader)

**Eksempel 2: Executive Grafana Dashboard**
- Panels: KPI cards (top row), Time series (middle), Tables (bottom)
- Variables: Environment (prod/test), Cost center
- Refresh: 5 min
- Alerts: Email til leadership ved cost > threshold

**Eksempel 3: Finance Power BI Report**
- Pages: Monthly spend, Cost per business unit, Forecast vs Actual
- Data sources: Azure Monitor + Finance system (via Dataverse)
- Refresh: Daily (6 AM)
- RLS: Finance team ser all data, business units ser kun sine egne

---

## Ressurser

### Microsoft Learn
- [Azure Workbooks overview](https://learn.microsoft.com/en-us/azure/azure-monitor/visualize/workbooks-overview)
- [Create an Microsoft Foundry dashboard](https://learn.microsoft.com/en-us/azure/managed-grafana/azure-ai-foundry-dashboard)
- [Monitor Azure OpenAI](https://learn.microsoft.com/en-us/azure/foundry-classic/openai/how-to/monitor-openai)
- [Workbooks programmatic management](https://learn.microsoft.com/en-us/azure/azure-monitor/visualize/workbooks-automate) *(Verified MCP 2026-04)* — ARM/Bicep deployment, RBAC (Monitoring Contributor for redigering, Monitoring Reader for visning), `microsoft.insights/workbooks/write` for custom roles
- [Power BI + Azure Monitor](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/log-powerbi) *(Verified MCP 2026-06-19)* — eksport fra Log Analytics som M-query (.txt → Power BI Desktop) eller new Dataset (Power BI-tjenesten); dataflows + incremental refresh; scheduled refresh/deling krever Pro/Premium

### Code Samples
- [Workbook ARM/Bicep template samples](https://learn.microsoft.com/en-us/azure/azure-monitor/visualize/workbooks-samples) — workbook templates bruker apiVersion `2019-10-17-preview` (type: microsoft.insights/workbooktemplates); workbook instances bruker `2018-06-17-preview` (type: Microsoft.Insights/workbooks) *(Verified MCP 2026-04)*
- [Microsoft Foundry Grafana dashboard ID: 24039](https://grafana.com/grafana/dashboards/24039)
- [KQL query examples for AI monitoring](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/samples)

### GitHub
- [Azure Monitor Community](https://github.com/microsoft/AzureMonitorCommunity) – Workbook templates
- [Grafana dashboards](https://github.com/grafana/grafana) – Community dashboards
- [Power BI samples](https://github.com/microsoft/powerbi-samples) – BI report templates

---

**Status:** Komplett
**Neste steg:** Kombiner med "alert-strategies-ai-systems.md" for helhetlig monitoring approach.