To operasjoner, én økt (⊥ R7), begge ren metadata-normalisering (verdi aldri fabrikkert). Premiss-korreksjon (ground truth 2026-07-07): roadmap sa «27 none + 4 ai-act». Målt: 29 mangler bold **Status:** = 25 rene none + 4 ai-act (plain Status: GA). De «27» inkluderte 2 for mye — 2 filer (custom-dashboards-ai-operations, zero-trust-ai-services) har bold **Status:** KUN forbi byte 500 (present for full-fil-audit, usynlig for 500B header-parser) → egen header-slanking-residual (§8-register), utenfor Enhet 3. Op A — Status-backfill 25 rene none: utvidet backfill-status.mjs MANIFEST 14→39 (samme statusForFile + insertMetaField + hard per-fil-invariant, idempotent skip på de 14 R21-gjorte). Alle 25 → **Status:** Established Practice (ingen matcher template|matrix|benchmarks|register). Diff +25/-0. Op B — ai-act dual-header-dedup (4 filer): ny driver dedup-plain-header.mjs + 2 rene primitiver i transform.mjs — boldifyPlainField (plain→bold, verdi bevart byte-eksakt, header-scoped, idempotent) + dropRedundantPlainField (sletter plain KUN når bold m/ identisk verdi beviser redundans; kaster ved avvik/manglende bold). Per fil: plain Last updated: + Status: GA → bold (2026-06-18/2026-02, GA bevart), redundant plain Category: fjernet. Hard per-fil-invariant (net -1 linje, begge felt bold m/ bevart verdi, ingen plain-header igjen, body byte-identisk). Diff -12/+8. Verifisering: test-backfill-status 8/8 + test-dedup-plain-header 13/13; audit Missing Status 29→0, Missing English Last updated 4→0; skills-diff 29 filer +33/-12 (kun **Status:** + 8 bold-swaps), diff-kontekst inspisert per fil; begge drivere idempotent (re-run 0 writes); suite 806/806 exit 0; none=8 uendret (Enhet 4).
768 lines
25 KiB
Markdown
768 lines
25 KiB
Markdown
# Log Analytics KQL Queries for AI
|
|
|
|
**Category:** Monitoring & Observability
|
|
**Last updated:** 2026-05
|
|
**Forfatter:** Cosmo Skyberg, AI Solution Architect
|
|
**Type:** reference
|
|
**Source:** https://learn.microsoft.com/azure/machine-learning/monitor-azure-machine-learning
|
|
**Status:** Established Practice
|
|
|
|
## Innhold
|
|
|
|
- [Oversikt](#oversikt)
|
|
- [Essential KQL Queries for AI Monitoring](#essential-kql-queries-for-ai-monitoring)
|
|
- [Performance Analysis Queries](#performance-analysis-queries)
|
|
- [Error Investigation Patterns](#error-investigation-patterns)
|
|
- [Cost Analysis Queries](#cost-analysis-queries)
|
|
- [Query Optimization Techniques](#query-optimization-techniques)
|
|
- [Advanced Patterns](#advanced-patterns)
|
|
- [For Cosmo: Anvendelse i Arkitekturrådgivning](#for-cosmo-anvendelse-i-arkitekturrådgivning)
|
|
- [Viktige KQL-ressurser](#viktige-kql-ressurser)
|
|
- [Nøkkelinnsikter](#nøkkelinnsikter)
|
|
- [Referanser](#referanser)
|
|
|
|
## Oversikt
|
|
|
|
Kusto Query Language (KQL) er det primære språket for å analysere monitoring-data i Azure Monitor Logs og Log Analytics. For AI-løsninger gir KQL kraftig innsikt i ytelse, kostnader, feil og bruksmønstre på tvers av Azure OpenAI, Azure AI Search, Azure Machine Learning og andre AI-tjenester.
|
|
|
|
Denne referansen gir essential KQL-queries skreddersydd for AI-monitoring, med fokus på praktiske mønstre for feilsøking, ytelsesanalyse, kostnadskontroll og query-optimalisering.
|
|
|
|
## Essential KQL Queries for AI Monitoring
|
|
|
|
### Grunnleggende Query-struktur
|
|
|
|
Alle KQL-queries følger pipe-syntaks der data flyter gjennom operatorer:
|
|
|
|
```kusto
|
|
TableName
|
|
| where <filter>
|
|
| project <columns>
|
|
| summarize <aggregation>
|
|
| render <visualization>
|
|
```
|
|
|
|
**Viktige tabeller for AI-monitoring:**
|
|
|
|
- `AzureDiagnostics` — resource logs fra Azure-tjenester
|
|
- `AzureMetrics` — platform metrics
|
|
- `CDBCassandraRequests` — Cosmos DB (hvis brukt for AI-lagring)
|
|
- `ABSBotRequests` — Azure Bot Service
|
|
- `AmlComputeJobEvent` — Azure Machine Learning job events
|
|
- `AmlComputeClusterEvent` — Azure ML cluster events
|
|
- `AmlOnlineEndpointTrafficLog` — Azure ML online endpoint traffic *(Verified MCP 2026-04)*
|
|
|
|
### Azure OpenAI: Grunnleggende Diagnostics Query
|
|
|
|
```kusto
|
|
// Initial analysis av Azure OpenAI resource logs
|
|
AzureDiagnostics
|
|
| where ResourceProvider == "MICROSOFT.OPENAI"
|
|
| take 100
|
|
| project TimeGenerated, _ResourceId, Category, OperationName, DurationMs, ResultSignature, properties_s
|
|
```
|
|
|
|
**Output:** Sample av 100 entries med key columns. For å se alle kolonner, fjern `| project ...` linjen.
|
|
|
|
### Azure OpenAI: Token-bruk Over Tid
|
|
|
|
```kusto
|
|
// Visualiser request volume over tid
|
|
AzureDiagnostics
|
|
| where ResourceProvider == "MICROSOFT.OPENAI"
|
|
| where Category == "RequestResponse"
|
|
| summarize count() by bin(TimeGenerated, 10m), OperationName
|
|
| render timechart
|
|
```
|
|
|
|
**Forklaring:** `bin(TimeGenerated, 10m)` grupperer data i 10-minutters intervaller. `render timechart` genererer tidsseriegraf.
|
|
|
|
### Azure OpenAI: Feilrate og Status Codes
|
|
|
|
```kusto
|
|
// Identifiser feilede requests med status code
|
|
AzureDiagnostics
|
|
| where ResourceProvider == "MICROSOFT.OPENAI"
|
|
| where ResultSignature != "200"
|
|
| summarize ErrorCount = count() by ResultSignature, OperationName
|
|
| order by ErrorCount desc
|
|
```
|
|
|
|
**Bruk:** Finn hvilke operasjoner som feiler hyppigst og hvilke HTTP-statuskoder som returneres.
|
|
|
|
## Performance Analysis Queries
|
|
|
|
### Azure OpenAI: Latency Percentiles
|
|
|
|
```kusto
|
|
// Beregn p50, p95, p99 latency for OpenAI requests
|
|
AzureDiagnostics
|
|
| where ResourceProvider == "MICROSOFT.OPENAI"
|
|
| where TimeGenerated > ago(24h)
|
|
| summarize
|
|
p50 = percentile(DurationMs, 50),
|
|
p95 = percentile(DurationMs, 95),
|
|
p99 = percentile(DurationMs, 99),
|
|
avg = avg(DurationMs),
|
|
max = max(DurationMs)
|
|
by OperationName
|
|
| order by p99 desc
|
|
```
|
|
|
|
**Forklaring:** Percentil-analyse er kritisk for å forstå "tail latency". p99 = 500ms betyr at 99% av requests er raskere enn 500ms.
|
|
|
|
### Azure AI Search: Long-running Queries
|
|
|
|
```kusto
|
|
// Finn tregeste search queries
|
|
AzureDiagnostics
|
|
| where ResourceProvider == "MICROSOFT.SEARCH"
|
|
| where OperationName == "Query.Search"
|
|
| project TimeGenerated, DurationMs, Query_s, IndexName_s, Documents_d
|
|
| where DurationMs > 1000 // > 1 sekund
|
|
| order by DurationMs desc
|
|
| take 20
|
|
```
|
|
|
|
**Bruk:** Identifiser queries som trenger optimalisering (indeksering, filtrering, caching).
|
|
|
|
### Azure AI Search: Query Volume (QPS)
|
|
|
|
```kusto
|
|
// Search queries per second over tid
|
|
AzureDiagnostics
|
|
| where ResourceProvider == "MICROSOFT.SEARCH"
|
|
| where OperationName == "Query.Search"
|
|
| summarize QPS = count() by bin(TimeGenerated, 1m)
|
|
| render timechart
|
|
```
|
|
|
|
**Forklaring:** Visualiserer query load. Spikes kan indikere traffic-mønstre eller potensielle throttling-situasjoner.
|
|
|
|
### Azure Machine Learning: Failed Jobs
|
|
|
|
```kusto
|
|
// ML jobs som har feilet siste 5 dager
|
|
AmlComputeJobEvent
|
|
| where TimeGenerated > ago(5d) and EventType == "JobFailed"
|
|
| project TimeGenerated, ClusterId, EventType, ExecutionState, ToolType
|
|
| order by TimeGenerated desc
|
|
```
|
|
|
|
**Bruk:** Rask oversikt over failed training/inference jobs. Drill ned med `JobName` for detaljert analyse.
|
|
|
|
### Azure Machine Learning: Cluster Node Allocation
|
|
|
|
```kusto
|
|
// Node allocation over tid (capacity planning)
|
|
AmlComputeClusterEvent
|
|
| where TimeGenerated > ago(1d)
|
|
| summarize avgRunningNodes=avg(TargetNodeCount), maxRunningNodes=max(TargetNodeCount)
|
|
by Workspace=tostring(split(_ResourceId, "/")[8]), ClusterName, VmSize
|
|
| order by maxRunningNodes desc
|
|
```
|
|
|
|
**Forklaring:** Identifiser peak node-bruk for å optimalisere cluster sizing og kostnader.
|
|
|
|
### Azure Machine Learning: Failed Online Endpoint Requests
|
|
|
|
*(Verified MCP 2026-04)*
|
|
|
|
```kusto
|
|
// Failed online endpoint requests siste dag
|
|
AmlOnlineEndpointTrafficLog
|
|
| where TimeGenerated > ago(1d) and ResponseCode != 200
|
|
| project TimeGenerated, EndpointName, DeploymentName, ResponseCode, ResponseCodeReason
|
|
| order by TimeGenerated desc
|
|
```
|
|
|
|
**Bruk:** Overvåk inference-endepunkter i produksjon. `ResponseCodeReason` gir detaljert feilinfo for debugging.
|
|
|
|
### Azure Machine Learning: Anbefalte Alert Rules
|
|
|
|
*(Verified MCP 2026-04)*
|
|
|
|
Microsoft dokumenterer tre standard alert rules for Azure ML:
|
|
|
|
| Alert type | Betingelse | Beskrivelse |
|
|
|-----------|-----------|-------------|
|
|
| Model Deploy Failed | Total > 0 | Én eller flere modelldeploy-jobber har feilet |
|
|
| Quota Utilization Percentage | Average > 90% | Kvoteutnyttelse over 90% |
|
|
| Unusable Nodes | Total > 0 | Én eller flere noder er i unusable-tilstand |
|
|
|
|
**KQL for quota-overvåkning:**
|
|
|
|
```kusto
|
|
// Overvåk cluster quota-utnyttelse
|
|
AmlComputeClusterEvent
|
|
| where TimeGenerated > ago(1h)
|
|
| summarize AvgQuotaUtilization = avg(todouble(QuotaUtilized) / todouble(QuotaAllocated) * 100)
|
|
by ClusterName
|
|
| where AvgQuotaUtilization > 90
|
|
| project ClusterName, AvgQuotaUtilization
|
|
```
|
|
|
|
## Error Investigation Patterns
|
|
|
|
### Pattern 1: Error Spike Detection
|
|
|
|
```kusto
|
|
// Finn tidspunkter med unormal feilrate
|
|
let baselineErrorRate = toscalar(
|
|
AzureDiagnostics
|
|
| where TimeGenerated > ago(7d)
|
|
| where ResourceProvider == "MICROSOFT.OPENAI"
|
|
| summarize ErrorRate = todouble(countif(ResultSignature != "200")) / count()
|
|
);
|
|
AzureDiagnostics
|
|
| where TimeGenerated > ago(24h)
|
|
| where ResourceProvider == "MICROSOFT.OPENAI"
|
|
| summarize ErrorRate = todouble(countif(ResultSignature != "200")) / count() by bin(TimeGenerated, 5m)
|
|
| where ErrorRate > (baselineErrorRate * 2) // 2x baseline
|
|
| project TimeGenerated, ErrorRate, Threshold = baselineErrorRate * 2
|
|
| render timechart
|
|
```
|
|
|
|
**Forklaring:** Baseline-basert anomaly detection. Flagg perioder der feilrate er 2x over 7-dagers gjennomsnitt.
|
|
|
|
### Pattern 2: Error Message Analysis
|
|
|
|
```kusto
|
|
// Grupper feilmeldinger for pattern-analyse
|
|
AzureDiagnostics
|
|
| where TimeGenerated > ago(24h)
|
|
| where ResultSignature != "200"
|
|
| extend ErrorDetails = parse_json(properties_s)
|
|
| project TimeGenerated, OperationName, ResultSignature, ErrorMessage = tostring(ErrorDetails.error.message)
|
|
| summarize Count = count() by ErrorMessage
|
|
| order by Count desc
|
|
| take 10
|
|
```
|
|
|
|
**Bruk:** Identifiser vanligste feilmeldinger. Nyttig for å finne repeterende problemer (auth, quota, invalid input).
|
|
|
|
### Pattern 3: Throttling Detection (429 Errors)
|
|
|
|
```kusto
|
|
// Azure OpenAI throttling events
|
|
AzureDiagnostics
|
|
| where ResourceProvider == "MICROSOFT.OPENAI"
|
|
| where ResultSignature == "429"
|
|
| summarize ThrottleCount = count() by bin(TimeGenerated, 10m), OperationName
|
|
| render timechart
|
|
```
|
|
|
|
**Cosmos DB variant (for AI-backends):**
|
|
|
|
```kusto
|
|
CDBCassandraRequests
|
|
| where ErrorCode == 4097 // Cassandra error code for throttling
|
|
| where TimeGenerated > ago(1h)
|
|
| project TimeGenerated, DatabaseName, CollectionName, OperationName, RateLimitingDelayMs
|
|
```
|
|
|
|
### Pattern 4: Cross-service Correlation
|
|
|
|
```kusto
|
|
// Korrelasjoner mellom Azure OpenAI errors og AI Search errors
|
|
let openaiErrors =
|
|
AzureDiagnostics
|
|
| where ResourceProvider == "MICROSOFT.OPENAI"
|
|
| where ResultSignature != "200"
|
|
| summarize OpenAIErrors = count() by bin(TimeGenerated, 5m);
|
|
let searchErrors =
|
|
AzureDiagnostics
|
|
| where ResourceProvider == "MICROSOFT.SEARCH"
|
|
| where resultSignature_d >= 400
|
|
| summarize SearchErrors = count() by bin(TimeGenerated, 5m);
|
|
openaiErrors
|
|
| join kind=inner searchErrors on TimeGenerated
|
|
| project TimeGenerated, OpenAIErrors, SearchErrors
|
|
| render timechart
|
|
```
|
|
|
|
**Bruk:** Finn om feil i én AI-tjeneste samvarierer med feil i en annen (f.eks. RAG pipeline: search → OpenAI).
|
|
|
|
## Cost Analysis Queries
|
|
|
|
### Token Consumption by Operation
|
|
|
|
```kusto
|
|
// Aggreger token-bruk per operasjonstype
|
|
AzureMetrics
|
|
| where TimeGenerated > ago(7d)
|
|
| where ResourceProvider == "MICROSOFT.OPENAI"
|
|
| where MetricName in ("TokenTransaction", "TotalTokens", "PromptTokens", "CompletionTokens")
|
|
| summarize TotalTokens = sum(Total) by MetricName, bin(TimeGenerated, 1d)
|
|
| render columnchart
|
|
```
|
|
|
|
**Forklaring:** Visualiserer token-forbruk over tid. Nyttig for å identifisere kostnadsdrivere.
|
|
|
|
### Cost Estimation (NOK)
|
|
|
|
```kusto
|
|
// Estimer kostnader basert på token-bruk (GPT-4 Turbo priser)
|
|
// Anta: Prompt = 0.01 USD / 1K tokens, Completion = 0.03 USD / 1K tokens
|
|
// USD/NOK = 10.5 (juster etter gjeldende kurs)
|
|
AzureMetrics
|
|
| where TimeGenerated > ago(30d)
|
|
| where ResourceProvider == "MICROSOFT.OPENAI"
|
|
| where MetricName in ("PromptTokens", "CompletionTokens")
|
|
| summarize
|
|
PromptTokens = sumif(Total, MetricName == "PromptTokens"),
|
|
CompletionTokens = sumif(Total, MetricName == "CompletionTokens")
|
|
by bin(TimeGenerated, 1d)
|
|
| extend
|
|
PromptCostUSD = PromptTokens / 1000 * 0.01,
|
|
CompletionCostUSD = CompletionTokens / 1000 * 0.03,
|
|
TotalCostUSD = (PromptTokens / 1000 * 0.01) + (CompletionTokens / 1000 * 0.03),
|
|
TotalCostNOK = ((PromptTokens / 1000 * 0.01) + (CompletionTokens / 1000 * 0.03)) * 10.5
|
|
| project TimeGenerated, PromptTokens, CompletionTokens, TotalCostNOK
|
|
| render timechart
|
|
```
|
|
|
|
**Viktig:** Oppdater priser og valutakurs regelmessig. Bruk Azure Cost Management for offisielle kostnader.
|
|
|
|
### Top Costly Operations
|
|
|
|
```kusto
|
|
// Finn operasjoner med høyest RU-forbruk (Cosmos DB AI-backend)
|
|
CDBPartitionKeyRUConsumption
|
|
| where TimeGenerated > ago(24h)
|
|
| where DatabaseName == "ai_vectors"
|
|
| summarize TotalRU = sum(RequestCharge) by OperationName
|
|
| order by TotalRU desc
|
|
| take 10
|
|
```
|
|
|
|
**Bruk:** Identifiser hvilke operasjoner som driver Cosmos DB-kostnader i AI-løsninger (vector search, embedding storage).
|
|
|
|
### Hot Partition Detection (Cost & Performance Impact)
|
|
|
|
```kusto
|
|
// Identifiser "hot partitions" som kan drive opp kostnader
|
|
CDBPartitionKeyStatistics
|
|
| where DatabaseName == "ai_vectors"
|
|
| where TimeGenerated > ago(8h)
|
|
| summarize StorageUsed = sum(SizeKb), RequestCharge = sum(RequestCharge) by PartitionKey
|
|
| order by RequestCharge desc
|
|
| take 20
|
|
```
|
|
|
|
**Forklaring:** Hot partitions = ubalansert load → throttling → høyere kostnader. Vurder re-partitioning.
|
|
|
|
## Query Optimization Techniques
|
|
|
|
### 1. Filter Early and Often
|
|
|
|
**❌ Ineffektivt:**
|
|
|
|
```kusto
|
|
AzureDiagnostics
|
|
| project TimeGenerated, OperationName, DurationMs
|
|
| where TimeGenerated > ago(1d)
|
|
| where OperationName == "ChatCompletion"
|
|
```
|
|
|
|
**✅ Optimalisert:**
|
|
|
|
```kusto
|
|
AzureDiagnostics
|
|
| where TimeGenerated > ago(1d) // Filter først
|
|
| where OperationName == "ChatCompletion"
|
|
| project TimeGenerated, OperationName, DurationMs
|
|
```
|
|
|
|
**Regel:** `where` alltid før `project`. Reduserer datamengde tidlig i pipeline.
|
|
|
|
### 2. Use `top` Instead of `sort` + `take`
|
|
|
|
**❌ Ineffektivt:**
|
|
|
|
```kusto
|
|
AzureDiagnostics
|
|
| where TimeGenerated > ago(1d)
|
|
| sort by TimeGenerated desc
|
|
| take 100
|
|
```
|
|
|
|
**✅ Optimalisert:**
|
|
|
|
```kusto
|
|
AzureDiagnostics
|
|
| where TimeGenerated > ago(1d)
|
|
| top 100 by TimeGenerated desc
|
|
```
|
|
|
|
**Forklaring:** `top` sorterer server-side og returnerer kun N records. Raskere enn `sort` + `take`.
|
|
|
|
### 3. Limit Time Range Explicitly
|
|
|
|
**❌ Unngå:**
|
|
|
|
```kusto
|
|
AzureDiagnostics
|
|
| where OperationName == "Completion" // Søker ALL data!
|
|
```
|
|
|
|
**✅ Best practice:**
|
|
|
|
```kusto
|
|
AzureDiagnostics
|
|
| where TimeGenerated > ago(7d) // Eksplisitt tidsfilter
|
|
| where OperationName == "Completion"
|
|
```
|
|
|
|
**Forklaring:** Alltid definer time range. Uten `TimeGenerated`-filter kan queries time out på store datasett.
|
|
|
|
### 4. Avoid `search *` — Use Specific Tables
|
|
|
|
**❌ Tregt:**
|
|
|
|
```kusto
|
|
search *
|
|
| where TimeGenerated > ago(1d)
|
|
| where * has "OpenAI"
|
|
```
|
|
|
|
**✅ Raskere:**
|
|
|
|
```kusto
|
|
AzureDiagnostics
|
|
| where TimeGenerated > ago(1d)
|
|
| where ResourceProvider == "MICROSOFT.OPENAI"
|
|
```
|
|
|
|
**Forklaring:** `search *` scanner alle tabeller. Alltid spesifiser tabell og kolonner.
|
|
|
|
### 5. Use `summarize` with `bin()` for Time-series
|
|
|
|
**Pattern:**
|
|
|
|
```kusto
|
|
AzureDiagnostics
|
|
| where TimeGenerated > ago(24h)
|
|
| summarize avg(DurationMs), count() by bin(TimeGenerated, 5m), OperationName
|
|
| render timechart
|
|
```
|
|
|
|
**Forklaring:** `bin(TimeGenerated, 5m)` grupperer data i 5-minutters buckets. Reduserer output-size og gjør visualisering mulig.
|
|
|
|
### 6. Leverage `has` Over `contains` for Performance
|
|
|
|
**❌ Tregt (substring match):**
|
|
|
|
```kusto
|
|
| where OperationName contains "Chat"
|
|
```
|
|
|
|
**✅ Raskere (word match):**
|
|
|
|
```kusto
|
|
| where OperationName has "Chat"
|
|
```
|
|
|
|
**Forklaring:** `has` søker etter hele ord, ikke substring. Raskere indeks-lookup.
|
|
|
|
### 7. Pre-aggregate with `let` Statements
|
|
|
|
```kusto
|
|
// Beregn baseline én gang, reuse
|
|
let baseline = toscalar(
|
|
AzureDiagnostics
|
|
| where TimeGenerated > ago(7d)
|
|
| summarize avg(DurationMs)
|
|
);
|
|
AzureDiagnostics
|
|
| where TimeGenerated > ago(1h)
|
|
| summarize CurrentAvg = avg(DurationMs)
|
|
| extend BaselineAvg = baseline, Diff = CurrentAvg - baseline
|
|
```
|
|
|
|
**Forklaring:** `let` lagrer intermediære resultater. Unngå duplicate beregninger.
|
|
|
|
### 8. Limit Output with `take` During Development
|
|
|
|
```kusto
|
|
// Test query med begrenset output
|
|
AzureDiagnostics
|
|
| where TimeGenerated > ago(30d)
|
|
| take 100 // Bare 100 rows for testing
|
|
```
|
|
|
|
**Best practice:** Bruk `take 10` eller `take 100` mens du utvikler queries. Fjern før produksjon.
|
|
|
|
### 9. Bruk Query Details-panelet for ytelsesdiagnose
|
|
|
|
*(Verified MCP 2026-04)*
|
|
|
|
Log Analytics har et **Query Details**-panel (klikk "Query details" nede til høyre etter kjøring) med tre faner:
|
|
|
|
- **Overview** — KPI-er: CPU, tidsomfang, alder på data, antall workspaces, antall regioner, parallellisme, **Memory peak** (nytt)
|
|
- **Raw statistics** — Detaljert eksekusjonsstatistikk
|
|
- **Errors** — Feil under kjøring
|
|
|
|
**Execution time** er nå delt i tre komponenter:
|
|
|
|
| Komponent | Betydning |
|
|
|-----------|-----------|
|
|
| Engine Execution Time | Tid i underliggende data-engine (Azure Data Explorer). Høy verdi → optimaliser selve queryen |
|
|
| Service Execution Time | Intern Azure Monitor-prosessering og orkestrering |
|
|
| Service Queue Time | Ventetid i kø pga. concurrency-grenser. Høy verdi → reduser samtidige queries |
|
|
|
|
**Memory peak** er maksimal RAM observert under kjøring. Høy memory peak kan trigge `E_RUNAWAY_QUERY`- eller `E_LOW_MEMORY_CONDITION`-feil. Reduseres med tidlig filtrering og `shuffle`-hint på `join`/`summarize`.
|
|
|
|
### 10. Bryt opp store `parse`-kommandoer
|
|
|
|
*(Verified MCP 2026-04)*
|
|
|
|
**Regel:** Maks 5 kolonne-ekstraksjoner per `parse`-setning. Over 5 øker prosesseringstiden markant.
|
|
|
|
**❌ Tregere (mange ekstraksjoner i én setning):**
|
|
|
|
```kusto
|
|
LogData
|
|
| parse Message with
|
|
* "field1=" Field1: string " field2=" Field2: string
|
|
" field3=" Field3: string " field4=" Field4: string
|
|
" field5=" Field5: string " field6=" Field6: string
|
|
" field7=" Field7: string " field8=" Field8: string *
|
|
```
|
|
|
|
**✅ Raskere (del opp i flere setninger):**
|
|
|
|
```kusto
|
|
LogData
|
|
| parse Message with
|
|
* "field1=" Field1: string " field2=" Field2: string
|
|
" field3=" Field3: string " field4=" Field4: string
|
|
" field5=" Field5: string *
|
|
| parse Message with
|
|
* " field6=" Field6: string " field7=" Field7: string
|
|
" field8=" Field8: string *
|
|
```
|
|
|
|
**Merk:** I transformasjoner er grensen 10 ekstraksjoner per `parse`-setning.
|
|
|
|
### 11. Bruk `materialize()` for subqueries som gjenbrukes
|
|
|
|
*(Verified MCP 2026-04)*
|
|
|
|
Når samme datakilde brukes i flere subqueries, kan `materialize()` cache mellomresultater og forhindre multiple gjennomganger av kilde-data:
|
|
|
|
```kusto
|
|
let CachedData = materialize(
|
|
AzureDiagnostics
|
|
| where TimeGenerated > ago(1h)
|
|
| where ResourceProvider == "MICROSOFT.OPENAI"
|
|
);
|
|
CachedData | summarize ErrorCount = countif(ResultSignature != "200") by OperationName
|
|
| join kind=inner (CachedData | summarize TotalCount = count() by OperationName) on OperationName
|
|
| extend ErrorRate = todouble(ErrorCount) / TotalCount
|
|
```
|
|
|
|
**Effektivt når:** Output fra subquery er mye mindre enn input, og subquery kjøres flere ganger i samme query.
|
|
|
|
## Advanced Patterns
|
|
|
|
### Multi-region Aggregation
|
|
|
|
```kusto
|
|
// Aggreger Azure OpenAI metrics på tvers av regions
|
|
AzureMetrics
|
|
| where ResourceProvider == "MICROSOFT.OPENAI"
|
|
| where TimeGenerated > ago(24h)
|
|
| extend Region = tostring(split(_ResourceId, "/")[8])
|
|
| summarize TotalRequests = sum(Total) by Region, MetricName
|
|
| order by TotalRequests desc
|
|
```
|
|
|
|
**Bruk:** Sammenlign load på tvers av Azure-regioner for global AI-deployment.
|
|
|
|
### Anomaly Detection med `series_decompose_anomalies()`
|
|
|
|
```kusto
|
|
// Automatisk anomaly detection i latency
|
|
AzureDiagnostics
|
|
| where TimeGenerated > ago(7d)
|
|
| where ResourceProvider == "MICROSOFT.OPENAI"
|
|
| make-series AvgLatency=avg(DurationMs) on TimeGenerated step 10m
|
|
| extend anomalies = series_decompose_anomalies(AvgLatency, 1.5)
|
|
| render anomalychart
|
|
```
|
|
|
|
**Forklaring:** `series_decompose_anomalies()` bruker ML-basert anomaly detection. Threshold 1.5 = moderat sensitivitet.
|
|
|
|
### Workload Patterns (Peak Hours)
|
|
|
|
```kusto
|
|
// Identifiser peak-hours for capacity planning
|
|
AzureDiagnostics
|
|
| where TimeGenerated > ago(30d)
|
|
| where ResourceProvider == "MICROSOFT.OPENAI"
|
|
| extend Hour = datetime_part("Hour", TimeGenerated)
|
|
| summarize RequestCount = count() by Hour
|
|
| render columnchart
|
|
```
|
|
|
|
**Bruk:** Finn når AI-løsningen har høyest trafikk. Optimaliser autoscaling og PTU-allokeringer.
|
|
|
|
### User Behavior Analysis (fra query strings)
|
|
|
|
```kusto
|
|
// Analyser bruker-queries i AI Search
|
|
AzureDiagnostics
|
|
| where ResourceProvider == "MICROSOFT.SEARCH"
|
|
| where OperationName == "Query.Search"
|
|
| where Query_s != "?api-version=2025-09-01&search=*" // Filtrer health checks
|
|
| project TimeGenerated, Query_s, Documents_d
|
|
| summarize SearchCount = count() by Query_s
|
|
| order by SearchCount desc
|
|
| take 20
|
|
```
|
|
|
|
**Forklaring:** Finn hyppigst brukte søk. Optimaliser indekser og suggestions basert på reelt bruksmønster.
|
|
|
|
## For Cosmo: Anvendelse i Arkitekturrådgivning
|
|
|
|
### Scenario 1: RAG Performance Troubleshooting
|
|
|
|
**Problem:** Kunde rapporterer treg respons i RAG-løsning (Azure AI Search + Azure OpenAI).
|
|
|
|
**Tilnærming:**
|
|
|
|
1. **Mål latency per komponent:**
|
|
|
|
```kusto
|
|
// AI Search query latency
|
|
AzureDiagnostics
|
|
| where ResourceProvider == "MICROSOFT.SEARCH"
|
|
| where TimeGenerated > ago(1h)
|
|
| summarize p95_search = percentile(DurationMs, 95);
|
|
|
|
// OpenAI completion latency
|
|
AzureDiagnostics
|
|
| where ResourceProvider == "MICROSOFT.OPENAI"
|
|
| where TimeGenerated > ago(1h)
|
|
| summarize p95_openai = percentile(DurationMs, 95);
|
|
```
|
|
|
|
2. **Korreler tidsstempler** for å finne bottleneck (search vs. completion).
|
|
|
|
3. **Drill ned** med queries fra "Long-running Queries" og "Latency Percentiles" seksjoner over.
|
|
|
|
### Scenario 2: Overspent AI Budget
|
|
|
|
**Problem:** Kunde har brukt 80% av månedlig AI-budsjett på dag 15.
|
|
|
|
**Tilnærming:**
|
|
|
|
1. **Identifiser kostnadsdrivere:**
|
|
|
|
```kusto
|
|
// Hvilke operasjoner bruker mest tokens?
|
|
AzureMetrics
|
|
| where TimeGenerated > ago(15d)
|
|
| where MetricName == "TotalTokens"
|
|
| summarize TotalTokens = sum(Total) by tostring(parse_json(properties).ModelName)
|
|
| order by TotalTokens desc
|
|
```
|
|
|
|
2. **Finn hot users/apps** (krever custom dimensions i logging):
|
|
|
|
```kusto
|
|
AzureDiagnostics
|
|
| where TimeGenerated > ago(15d)
|
|
| extend AppId = tostring(parse_json(properties_s).appId)
|
|
| summarize RequestCount = count() by AppId
|
|
| order by RequestCount desc
|
|
```
|
|
|
|
3. **Anbefalinger:** Implementer caching, prompt-optimalisering, eller switch til billigere modeller for visse operasjoner.
|
|
|
|
### Scenario 3: Proaktiv Alerting Setup
|
|
|
|
**Anbefaling til kunde:**
|
|
|
|
Sett opp Azure Monitor alerts basert på KQL-queries:
|
|
|
|
- **Latency alert:**
|
|
|
|
```kusto
|
|
AzureDiagnostics
|
|
| where ResourceProvider == "MICROSOFT.OPENAI"
|
|
| summarize p95 = percentile(DurationMs, 95) by bin(TimeGenerated, 5m)
|
|
| where p95 > 2000 // Alert hvis p95 > 2s
|
|
```
|
|
|
|
- **Error rate alert:**
|
|
|
|
```kusto
|
|
AzureDiagnostics
|
|
| where ResourceProvider == "MICROSOFT.OPENAI"
|
|
| summarize ErrorRate = todouble(countif(ResultSignature != "200")) / count() by bin(TimeGenerated, 5m)
|
|
| where ErrorRate > 0.05 // Alert hvis > 5% feilrate
|
|
```
|
|
|
|
- **Cost anomaly alert:**
|
|
|
|
```kusto
|
|
AzureMetrics
|
|
| where MetricName == "TotalTokens"
|
|
| make-series TokensPerHour=sum(Total) on TimeGenerated step 1h
|
|
| extend anomalies = series_decompose_anomalies(TokensPerHour, 2.0)
|
|
| where anomalies > 0 // Alert på token-spikes
|
|
```
|
|
|
|
### Scenario 4: Compliance & Audit Logging
|
|
|
|
**Problem:** Kunde i offentlig sektor må dokumentere AI-bruk for revisjon.
|
|
|
|
**Løsning:** KQL-queries for audit trail:
|
|
|
|
```kusto
|
|
// Hvilke brukere har aksessert AI-tjenester?
|
|
AzureDiagnostics
|
|
| where TimeGenerated > ago(90d)
|
|
| where ResourceProvider in ("MICROSOFT.OPENAI", "MICROSOFT.SEARCH")
|
|
| extend User = tostring(parse_json(properties_s).userId)
|
|
| summarize RequestCount = count(), FirstAccess = min(TimeGenerated), LastAccess = max(TimeGenerated) by User
|
|
| order by RequestCount desc
|
|
```
|
|
|
|
**Export til CSV** for arkivering:
|
|
|
|
```kusto
|
|
// Kjør query i Log Analytics → "Export" → "CSV (all columns)"
|
|
```
|
|
|
|
## Viktige KQL-ressurser
|
|
|
|
- **KQL Quick Reference:** [learn.microsoft.com/kusto/query/kql-quick-reference](https://learn.microsoft.com/en-us/kusto/query/kql-quick-reference)
|
|
- **Azure Monitor KQL Samples:** [learn.microsoft.com/azure/azure-monitor/logs/queries](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/queries)
|
|
- **Azure OpenAI Monitoring:** [learn.microsoft.com/azure/foundry-classic/openai/how-to/monitor-openai](https://learn.microsoft.com/en-us/azure/foundry-classic/openai/how-to/monitor-openai)
|
|
- **Optimize Log Queries:** [learn.microsoft.com/azure/azure-monitor/logs/query-optimization](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/query-optimization)
|
|
|
|
## Nøkkelinnsikter
|
|
|
|
- **Filter tidlig:** `where TimeGenerated` alltid først for å begrense datamengde.
|
|
- **Bruk `top` over `sort` + `take`:** Server-side optimalisering.
|
|
- **Percentiler > gjennomsnitt:** p95/p99 gir bedre innsikt i brukeropplevelse enn avg.
|
|
- **`has` > `contains`:** Raskere word-match vs. substring-match.
|
|
- **Pre-aggreger med `let`:** Unngå duplicate beregninger.
|
|
- **Test med `take`:** Begrens output under query-utvikling.
|
|
- **Korreler på tvers av tjenester:** `join` for å finne cross-service dependencies.
|
|
- **Visualiser med `render`:** `timechart`, `columnchart`, `anomalychart` for innsikt.
|
|
- **Bruk Query Details-panel:** Engine/Service/Queue execution time + Memory peak for diagnose. *(Verified MCP 2026-04)*
|
|
- **Maks 5 per `parse`:** Del opp store parse-setninger for å redusere prosesseringstid. *(Verified MCP 2026-04)*
|
|
- **`materialize()` for gjentatte subqueries:** Cache mellomresultater, unngå multiple datascans. *(Verified MCP 2026-04)*
|
|
- **`AmlOnlineEndpointTrafficLog`:** Ny tabell for inference-endepunktovervåkning i Azure ML. *(Verified MCP 2026-04)*
|
|
|
|
## Referanser
|
|
|
|
- Microsoft Learn: [Monitor Azure OpenAI](https://learn.microsoft.com/en-us/azure/foundry-classic/openai/how-to/monitor-openai)
|
|
- Microsoft Learn: [Get started with log queries in Azure Monitor](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/get-started-queries)
|
|
- Microsoft Learn: [Optimize log queries in Azure Monitor](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/query-optimization) *(Verified MCP 2026-04)*
|
|
- Microsoft Learn: [Configure diagnostic logging for Azure AI Search](https://learn.microsoft.com/en-us/azure/search/search-monitor-enable-logging)
|
|
- Microsoft Learn: [Monitor Azure Machine Learning](https://learn.microsoft.com/en-us/azure/machine-learning/monitor-azure-machine-learning) *(Verified MCP 2026-04)*
|
|
- Microsoft Learn: [KQL quick reference](https://learn.microsoft.com/en-us/kusto/query/kql-quick-reference)
|