feat(ultraplan-local): v1.6.0 — /ultraresearch-local deep research command

Add /ultraresearch-local for structured research combining local codebase analysis with external knowledge via parallel agent swarms. Produces research briefs with triangulation, confidence ratings, and source quality assessment. New command: /ultraresearch-local with modes --quick, --local, --external, --fg. New agents: research-orchestrator (opus), docs-researcher, community-researcher, security-researcher, contrarian-researcher, gemini-bridge (all sonnet). New template: research-brief-template.md. Integration: --research flag in /ultraplan-local accepts pre-built research briefs (up to 3), enriches the interview and exploration phases. Planning orchestrator cross-references brief findings during synthesis. Design principle: Context Engineering — right information to right agent at right time. Research briefs are structured artifacts in the pipeline: ultraresearch → brief → ultraplan --research → plan → ultraexecute. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-08 08:58:35 +02:00 · 2026-04-08 08:58:35 +02:00 · baa2d0220b
commit baa2d0220b
488 changed files with 213221 additions and 0 deletions
--- a/skills/ms-ai-governance/references/monitoring-observability/custom-dashboards-ai-operations.md
+++ b/skills/ms-ai-governance/references/monitoring-observability/custom-dashboards-ai-operations.md
@ -0,0 +1,499 @@
+# Custom Dashboards for AI Operations
+
+**Kategori:** Monitoring & Observability
+**Sist oppdatert:** 2026-02-05
+**Brukes av:** Cosmo Skyberg, Microsoft AI Solution Architect
+
+---
+
+## Innledning
+
+Custom dashboards er essensielle for å visualisere og forstå AI-driften i sanntid. Mens standard metrics-visninger gir grunnleggende innsikt, tilbyr tilpassede dashboards mulighet til å kombinere data fra flere kilder, skreddersy visualiseringer for ulike interessenter, og bygge operasjonelle kommandosentral for AI-systemer.
+
+Microsoft-stakken tilbyr flere dashboarding-løsninger med ulike styrker: Azure Workbooks for teknisk dybde, Grafana for operasjonelle sanntidsvisninger, Power BI for executive insights, og Real-Time Intelligence dashboards for streaming-data.
+
+---
+
+## Azure Workbooks for AI
+
+Azure Workbooks er Microsofts native dashboarding-løsning i Azure Monitor. De kombinerer tekst, KQL-queries, metrics, og interaktive parametere i én fleksibel canvas.
+
+### Hvorfor Workbooks for AI-monitoring?
+
+- **Unified data sources:** Kombinerer Application Insights, Log Analytics, metrics, og Azure Resource Graph i én view
+- **KQL-powered:** Direkte tilgang til Kusto Query Language for avanserte aggregeringer
+- **Template-drevet:** Distribuer standardiserte dashboards programmatisk via ARM templates
+- **Resource-centric:** Visualiser data på tvers av flere AI-ressurser samtidig
+- **Built-in for AI Foundry:** Azure AI Foundry leverer ferdig "Application Analytics" workbook
+
+### Azure AI Foundry Application Analytics Workbook
+
+Azure AI Foundry tilbyr en out-of-box workbook som sporer:
+
+- **Generative AI metrics:** Total conversations, latency, exceptions
+- **Tool usage:** Hvilke extensions og tools brukes mest
+- **Topic analytics:** Hvilke conversation topics dominerer
+- **Operational health:** Success rates, error patterns, response times
+
+**Tilgang:**
+1. Gå til Application Insights ressurs
+2. Velg **Monitoring** → **Workbooks**
+3. Åpne "Copilot Studio Dashboard" fra galleriet
+
+**Tilpasning:**
+```kusto
+// Eksempel: Track custom attribute for AI responses
+customEvents
+| where name == "AIResponse"
+| extend ResponseQuality = tostring(customDimensions.quality)
+| summarize Count = count() by ResponseQuality, bin(timestamp, 1h)
+| render timechart
+```
+
+### Workbook Architecture for AI
+
+**Typiske seksjoner i et AI operations workbook:**
+
+1. **Executive summary** (stat tiles)
+   - Total requests today
+   - Average latency
+   - Token consumption
+   - Success rate
+
+2. **Request trends** (timecharts)
+   - API calls per hour
+   - Per-model distribution
+   - Geographic distribution
+
+3. **Token economics** (barcharts)
+   - Token usage by deployment
+   - Cost per request
+   - Top consumers
+
+4. **Error analysis** (grids + pie charts)
+   - Error codes by frequency
+   - Failed requests by model
+   - Retry patterns
+
+5. **Performance drill-down** (interactive queries)
+   - Parametere for time range, model, region
+   - Query-backed visualizations som oppdateres live
+
+### Programmatic Deployment
+
+Workbooks kan deployes via ARM templates for consistency across teams:
+
+```json
+{
+  "name": "ai-operations-workbook",
+  "type": "microsoft.insights/workbooks",
+  "location": "[resourceGroup().location]",
+  "apiVersion": "2022-04-01",
+  "properties": {
+    "displayName": "AI Operations Dashboard",
+    "serializedData": "{\"version\":\"Notebook/1.0\",\"items\":[...]}",
+    "category": "AI Monitoring",
+    "sourceId": "[resourceId('Microsoft.Insights/components', parameters('appInsightsName'))]"
+  }
+}
+```
+
+**Best practices:**
+- Bruk parametere for time ranges og resource filters
+- Inkluder markdown-tekst for kontekst og aksjonspunkter
+- Legg til links til troubleshooting-docs
+- Del workbooks via Azure RBAC (Workbook Contributor role)
+
+---
+
+## Grafana for AI Operational Dashboards
+
+Azure Managed Grafana er ideell for sanntids-operasjonssentre. Grafana excels i streaming visualizations, multi-source aggregation, og alert-integrasjon.
+
+### Azure AI Foundry Grafana Dashboard
+
+Microsoft tilbyr en ferdig Grafana dashboard (ID: 24039) for Azure AI Foundry ressurser.
+
+**Key metrics:**
+- **Model performance:** Inference latency (time to last byte), throughput, success rates
+- **Token tracking:** Total tokens, prompt tokens, completion tokens
+- **Request trends:** API call volume per deployment
+- **Cost visibility:** Token consumption patterns for cost optimization
+- **Per-deployment comparison:** Side-by-side metrics for GPT-4 vs GPT-3.5
+
+**Import prosess:**
+1. Gå til Azure Managed Grafana workspace
+2. Dashboards → New → Import
+3. Enter dashboard ID: **24039**
+4. Velg Azure Monitor data source
+5. Assign Monitoring Reader role til Grafana managed identity
+
+**Metric namespace:** `Microsoft.CognitiveServices/accounts`
+
+**Key metrics:**
+- `AzureOpenAIRequests` – API call volume and success rates
+- `TokenTransaction` – Total inference tokens for cost tracking
+- `ProcessedPromptTokens` – Input tokens consumed
+- `GeneratedTokens` – Output tokens produced
+- `AzureOpenAITTLTInMS` – Inference latency (time to last byte)
+
+**Grouping:** All metrics split by `ModelDeploymentName`
+
+### Custom Grafana Panels
+
+**Legg til nytt panel:**
+1. Edit → Add → Visualization
+2. Data source: Azure Monitor
+3. Resource: Velg AI Foundry resource
+4. Metric: Velg metric (f.eks. `TokenTransaction`)
+5. Aggregation: Average, Sum, Count, Min, Max
+6. Visualization type: Time series, Stat, Gauge, Bar chart
+7. Thresholds: Definer warning/critical levels for visual alerts
+
+**Eksempel på custom panel for token cost:**
+- Data source: Azure Monitor
+- Metric: `TokenTransaction`
+- Aggregation: Sum
+- Transform: Math operation × 0.000002 (cost per token in NOK)
+- Visualization: Stat panel med "NOK spent today"
+- Threshold: Red over 5000 NOK
+
+---
+
+## Power BI for Executive AI Dashboards
+
+Power BI tilbyr business-orienterte visualiseringer med kraftig datamodellering. Ideell for executive dashboards som kombinerer AI metrics med business KPIs.
+
+### Power BI + Azure Monitor Integration
+
+**Dataflyt:**
+1. Azure Monitor logs → Log Analytics workspace
+2. Power BI connector → Import eller DirectQuery
+3. Power BI semantic model → Transform og model data
+4. Power BI report → Visualiser for executives
+
+**Setup:**
+1. I Power BI Desktop: Get Data → Azure → Azure Monitor Logs
+2. Enter workspace resource ID
+3. Write KQL query:
+```kusto
+AzureDiagnostics
+| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
+| where TimeGenerated > ago(30d)
+| summarize
+    TotalRequests = count(),
+    AvgLatency = avg(DurationMs),
+    TotalTokens = sum(toint(customDimensions.tokens))
+  by bin(TimeGenerated, 1d), ModelDeployment = tostring(customDimensions.model)
+```
+
+### Executive Dashboard Layout
+
+**Typical executive AI dashboard:**
+
+1. **Top KPIs** (cards)
+   - Monthly AI spend
+   - Total conversations handled
+   - Average user satisfaction (fra feedback)
+   - Cost per interaction
+
+2. **Trends** (line charts)
+   - AI usage growth over time
+   - Cost efficiency trend
+   - User adoption rate
+
+3. **Business impact** (combo charts)
+   - Support tickets vs AI conversations (korrelasjon)
+   - Customer satisfaction vs AI usage
+   - Cost savings from automation
+
+4. **Model performance** (tables)
+   - Ranker modeller etter success rate, cost, speed
+   - Benchmark mot SLA
+
+**Scheduling:**
+- Sett opp scheduled refresh (8x per dag for free, hourly for Pro)
+- Email subscriptions for stakeholders
+- Power BI mobile app for on-the-go access
+
+---
+
+## Real-Time Intelligence Dashboards (Fabric)
+
+Microsoft Fabric Real-Time Intelligence tilbyr sanntids-dashboards drevet av KQL queries mot Eventhouse.
+
+### AI Monitoring i Fabric
+
+**Use case:** Streaming AI telemetry for øyeblikkelig innsikt.
+
+**Architecture:**
+1. Azure AI Foundry → Event Hub → Fabric Eventhouse
+2. KQL Database → Continuous queries
+3. Real-Time Dashboard → Live visualizations
+
+**Dashboard tiles:**
+
+**Stat tile (max temperature pattern):**
+```kusto
+AITelemetry
+| where Timestamp between (_startTime.._endTime)
+| where ModelDeploymentName == _deployment
+| top 1 by Latency desc
+| summarize by Latency
+```
+
+**Time chart (request rate):**
+```kusto
+AITelemetry
+| where Timestamp between (_startTime.._endTime)
+| where ModelDeploymentName == _deployment
+| summarize RequestCount = count() by bin(Timestamp, 1m)
+| render timechart
+```
+
+**Parameters:**
+```kusto
+// Deployment selector
+AITelemetry
+| summarize by ModelDeploymentName
+```
+
+**Best practices:**
+- Bruk parameters for interactive filtering
+- Auto-refresh interval: 30 sek for operations, 5 min for analytics
+- Conditional formatting for thresholds (red/yellow/green)
+
+---
+
+## Dashboard Sharing and Governance
+
+### Access Control
+
+**Azure Workbooks:**
+- **Workbook Contributor role:** Kan redigere og lagre delte workbooks
+- **Monitoring Reader role:** Kan se workbooks, men ikke endre
+- **Resource-based permissions:** Brukere ser kun data fra ressurser de har tilgang til
+
+**Grafana:**
+- **Grafana Admin role:** Full tilgang
+- **Grafana Editor role:** Kan redigere dashboards
+- **Grafana Viewer role:** Read-only
+- Azure RBAC: Monitoring Reader på subscription/resource group
+
+**Power BI:**
+- **Workspace roles:** Admin, Member, Contributor, Viewer
+- **Row-level security (RLS):** Filtrer data basert på brukeridentitet
+- **App distribution:** Del read-only versjon via Power BI app
+
+### Governance Best Practices
+
+**Standardisering:**
+- Opprett dashboard templates for ulike roller (DevOps, Leadership, Security)
+- Bruk naming conventions: `[Team]-[Purpose]-[Environment]` (f.eks. `AITeam-Operations-Prod`)
+- Version control for workbook ARM templates i Git
+
+**Dokumentasjon:**
+- Inkluder markdown-seksjoner i workbooks med:
+  - Hva viser denne dashboard?
+  - Hvilke actions skal jeg ta ved alerts?
+  - Links til runbooks og troubleshooting guides
+- README i Power BI workspace med metric definitions
+
+**Update cadence:**
+- **Operations dashboards:** Live/1 min refresh
+- **Analytics dashboards:** 15 min refresh
+- **Executive dashboards:** Daily refresh (for kostnad-effektivitet)
+
+**Arkivering:**
+- Fjern dashboards som ikke har vært brukt på 90 dager
+- Eksporter historiske dashboards som snapshots (PDF fra Grafana, PBIX backup)
+
+---
+
+## Cost and Usage Visualizations
+
+### Token Economics Dashboard
+
+**Kritisk for AI-budsjett:** Visualiser token costs i sanntid.
+
+**KQL query for daily cost:**
+```kusto
+AzureDiagnostics
+| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
+| where OperationName == "ChatCompletions_Create"
+| extend
+    PromptTokens = toint(customDimensions.prompt_tokens),
+    CompletionTokens = toint(customDimensions.completion_tokens),
+    Model = tostring(customDimensions.model)
+| extend TotalCost = case(
+    Model == "gpt-4", (PromptTokens * 0.00003 + CompletionTokens * 0.00006),
+    Model == "gpt-35-turbo", (PromptTokens * 0.0000015 + CompletionTokens * 0.000002),
+    0
+  )
+| summarize DailyCost = sum(TotalCost) by bin(TimeGenerated, 1d)
+| render areachart
+```
+
+**Visualization types:**
+- **Waterfall chart:** Vis cost breakdown per model, per team, per use case
+- **Gauge:** Daily spend vs budget
+- **Heat map:** Peak usage hours (for PTU optimization)
+
+### PTU Utilization Dashboard
+
+For Provisioned Throughput Units (PTU):
+
+**Key metrics:**
+- PTU utilization percentage
+- Requests per PTU
+- Cost per request (PTU vs PayGo comparison)
+
+**Grafana panel:**
+- Data source: Azure Monitor
+- Metric: `ProcessedPromptTokens` + `GeneratedTokens`
+- Transform: Divide by PTU capacity → percentage
+- Visualization: Gauge med thresholds (green <80%, yellow 80-95%, red >95%)
+
+---
+
+## Dashboard Anti-Patterns
+
+**Feil å unngå:**
+
+❌ **Information overload:** 20+ metrics på én side – Splitt i multiple views
+❌ **Stale data:** Refresh rate som ikke matcher use case (real-time ops trenger <1 min)
+❌ **No context:** Metrics uten thresholds eller trend-indikatorer
+❌ **Static dashboards:** Ingen parameters for filtering eller drill-down
+❌ **Isolated metrics:** Ikke kombiner business outcomes med technical metrics
+❌ **No alerts configured:** Dashboards er reactive, du trenger proactive alerts også
+
+**Best practices:**
+
+✅ **Progressive disclosure:** Summary view → Drill-down details
+✅ **Thresholds everywhere:** Visual indicators (red/yellow/green)
+✅ **Contextual annotations:** Markdown-tekst som forklarer hva er normalt, hva er alarming
+✅ **Role-based views:** Ulike dashboards for DevOps, managers, finance
+✅ **Mobile-friendly:** Test på mobile devices (Grafana/Power BI mobile apps)
+✅ **Integration with incidents:** Link fra dashboard tile til incident management (ServiceNow, Linear)
+
+---
+
+## For Cosmo Skyberg
+
+Når kunden spør om dashboards for AI operations:
+
+### Discovery Questions
+
+1. **Hvem er dashboardet for?** (DevOps, executives, security team, finance?)
+2. **Hva er decision-kriteriene?** (Real-time troubleshooting, cost control, compliance, capacity planning?)
+3. **Hvilke data sources?** (Kun Azure Monitor, eller også custom app telemetry?)
+4. **Refresh requirements?** (Live, minutt, time, daglig?)
+5. **Mobile access?** (Grafana/Power BI mobile, eller kun desktop?)
+6. **Compliance constraints?** (Hvem kan se hvilke data? RLS nødvendig?)
+
+### Anbefalingsmatrise
+
+| Use Case | Anbefalt Løsning | Begrunnelse |
+|----------|------------------|-------------|
+| Real-time operations center | Grafana (Azure Managed) | Streaming metrics, alert-integrasjon, 24/7 NOC-friendly |
+| Deep technical troubleshooting | Azure Workbooks | KQL-drevet, resource-centric, kan kombinere logs+metrics |
+| Executive monthly reviews | Power BI | Business-oriented visuals, kombinerer AI med business KPIs |
+| Streaming IoT/Edge AI telemetry | Fabric Real-Time Dashboard | Sub-second refresh, event-driven |
+| Quick ad-hoc analysis | Log Analytics + Metrics Explorer | Ingen setup, direkte i portal |
+
+### Implementation Checklist
+
+**Fase 1: Design (1-2 uker)**
+- [ ] Definer målgrupper og deres behov
+- [ ] Skissér dashboard layout (wireframes)
+- [ ] Identifiser data sources og KQL queries
+- [ ] Etablér thresholds og alert-kriterier
+
+**Fase 2: Prototype (1 uke)**
+- [ ] Bygg workbook/Grafana dashboard med sample data
+- [ ] Test queries for performance (< 5 sek load time)
+- [ ] Validér med pilot-brukere
+
+**Fase 3: Production (1 uke)**
+- [ ] Deploy via ARM template (Workbooks) eller import (Grafana)
+- [ ] Konfigurer RBAC og sharing
+- [ ] Sett opp refresh schedules
+- [ ] Dokumentér i README
+
+**Fase 4: Iterate (kontinuerlig)**
+- [ ] Samle feedback fra brukere
+- [ ] Monitor dashboard usage (Application Insights for Grafana/PBI)
+- [ ] Optimaliser trege queries
+- [ ] Legg til nye metrics basert på operasjonelle behov
+
+### Technical Guidance
+
+**Når velge Workbooks:**
+- Teamet er komfortable med KQL
+- Trenger resource-centric views (mange AI-ressurser samtidig)
+- Ønsker programmatic deployment (IaC)
+- Budget-bevisst (ingen ekstra lisenskostnad)
+
+**Når velge Grafana:**
+- 24/7 operations center
+- Multi-cloud (kombinerer Azure med AWS/GCP metrics)
+- Alert-drevet kultur (Grafana alerting er kraftig)
+- Eksisterende Grafana-kompetanse
+
+**Når velge Power BI:**
+- Executive audience (ikke-tekniske interessenter)
+- Kombinerer AI metrics med ERP/CRM data
+- Trenger mobile app access
+- Ønsker scheduled email reports
+
+**Når velge Fabric Real-Time:**
+- Sub-second latency requirements
+- Massive scale (millioner av events per sekund)
+- Allerede investert i Microsoft Fabric
+- Event-driven architecture (Event Hub → Eventhouse)
+
+### Example Deliverables
+
+**Eksempel 1: DevOps Operations Workbook**
+- Sections: Health Overview, Request Trends, Error Analysis, Token Economics
+- Parametere: Time range, Model deployment, Region
+- Refresh: Live (1 min)
+- RBAC: DevOps team (Contributor), Leadership (Reader)
+
+**Eksempel 2: Executive Grafana Dashboard**
+- Panels: KPI cards (top row), Time series (middle), Tables (bottom)
+- Variables: Environment (prod/test), Cost center
+- Refresh: 5 min
+- Alerts: Email til leadership ved cost > threshold
+
+**Eksempel 3: Finance Power BI Report**
+- Pages: Monthly spend, Cost per business unit, Forecast vs Actual
+- Data sources: Azure Monitor + Finance system (via Dataverse)
+- Refresh: Daily (6 AM)
+- RLS: Finance team ser all data, business units ser kun sine egne
+
+---
+
+## Ressurser
+
+### Microsoft Learn
+- [Azure Workbooks overview](https://learn.microsoft.com/en-us/azure/azure-monitor/visualize/workbooks-overview)
+- [Create an Azure AI Foundry dashboard](https://learn.microsoft.com/en-us/azure/managed-grafana/azure-ai-foundry-dashboard)
+- [Monitor Azure OpenAI](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/monitor-openai)
+- [Workbooks programmatic management](https://learn.microsoft.com/en-us/azure/azure-monitor/visualize/workbooks-automate)
+- [Power BI + Azure Monitor](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/log-powerbi)
+
+### Code Samples
+- [Workbook ARM template sample](https://learn.microsoft.com/en-us/azure/azure-monitor/visualize/workbooks-samples)
+- [Azure AI Foundry Grafana dashboard ID: 24039](https://grafana.com/grafana/dashboards/24039)
+- [KQL query examples for AI monitoring](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/samples)
+
+### GitHub
+- [Azure Monitor Community](https://github.com/microsoft/AzureMonitorCommunity) – Workbook templates
+- [Grafana dashboards](https://github.com/grafana/grafana) – Community dashboards
+- [Power BI samples](https://github.com/microsoft/powerbi-samples) – BI report templates
+
+---
+
+**Status:** Komplett
+**Neste steg:** Kombiner med "alert-strategies-ai-systems.md" for helhetlig monitoring approach.