ms-ai-architect/skills/ms-ai-governance/references/monitoring-observability/log-analytics-kql-ai-queries.md

# Log Analytics KQL Queries for AI

**Kategori:** Monitoring & Observability
**Dato:** 2026-02-05
**Forfatter:** Cosmo Skyberg, AI Solution Architect

## Oversikt

Kusto Query Language (KQL) er det primære språket for å analysere monitoring-data i Azure Monitor Logs og Log Analytics. For AI-løsninger gir KQL kraftig innsikt i ytelse, kostnader, feil og bruksmønstre på tvers av Azure OpenAI, Azure AI Search, Azure Machine Learning og andre AI-tjenester.

Denne referansen gir essential KQL-queries skreddersydd for AI-monitoring, med fokus på praktiske mønstre for feilsøking, ytelsesanalyse, kostnadskontroll og query-optimalisering.

## Essential KQL Queries for AI Monitoring

### Grunnleggende Query-struktur

Alle KQL-queries følger pipe-syntaks der data flyter gjennom operatorer:

```kusto
TableName
| where <filter>
| project <columns>
| summarize <aggregation>
| render <visualization>
```

**Viktige tabeller for AI-monitoring:**

- `AzureDiagnostics` — resource logs fra Azure-tjenester
- `AzureMetrics` — platform metrics
- `CDBCassandraRequests` — Cosmos DB (hvis brukt for AI-lagring)
- `ABSBotRequests` — Azure Bot Service
- `AmlComputeJobEvent` — Azure Machine Learning job events
- `AmlComputeClusterEvent` — Azure ML cluster events

### Azure OpenAI: Grunnleggende Diagnostics Query

```kusto
// Initial analysis av Azure OpenAI resource logs
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.OPENAI"
| take 100
| project TimeGenerated, _ResourceId, Category, OperationName, DurationMs, ResultSignature, properties_s
```

**Output:** Sample av 100 entries med key columns. For å se alle kolonner, fjern `| project ...` linjen.

### Azure OpenAI: Token-bruk Over Tid

```kusto
// Visualiser request volume over tid
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.OPENAI"
| where Category == "RequestResponse"
| summarize count() by bin(TimeGenerated, 10m), OperationName
| render timechart
```

**Forklaring:** `bin(TimeGenerated, 10m)` grupperer data i 10-minutters intervaller. `render timechart` genererer tidsseriegraf.

### Azure OpenAI: Feilrate og Status Codes

```kusto
// Identifiser feilede requests med status code
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.OPENAI"
| where ResultSignature != "200"
| summarize ErrorCount = count() by ResultSignature, OperationName
| order by ErrorCount desc
```

**Bruk:** Finn hvilke operasjoner som feiler hyppigst og hvilke HTTP-statuskoder som returneres.

## Performance Analysis Queries

### Azure OpenAI: Latency Percentiles

```kusto
// Beregn p50, p95, p99 latency for OpenAI requests
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.OPENAI"
| where TimeGenerated > ago(24h)
| summarize
    p50 = percentile(DurationMs, 50),
    p95 = percentile(DurationMs, 95),
    p99 = percentile(DurationMs, 99),
    avg = avg(DurationMs),
    max = max(DurationMs)
    by OperationName
| order by p99 desc
```

**Forklaring:** Percentil-analyse er kritisk for å forstå "tail latency". p99 = 500ms betyr at 99% av requests er raskere enn 500ms.

### Azure AI Search: Long-running Queries

```kusto
// Finn tregeste search queries
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.SEARCH"
| where OperationName == "Query.Search"
| project TimeGenerated, DurationMs, Query_s, IndexName_s, Documents_d
| where DurationMs > 1000  // > 1 sekund
| order by DurationMs desc
| take 20
```

**Bruk:** Identifiser queries som trenger optimalisering (indeksering, filtrering, caching).

### Azure AI Search: Query Volume (QPS)

```kusto
// Search queries per second over tid
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.SEARCH"
| where OperationName == "Query.Search"
| summarize QPS = count() by bin(TimeGenerated, 1m)
| render timechart
```

**Forklaring:** Visualiserer query load. Spikes kan indikere traffic-mønstre eller potensielle throttling-situasjoner.

### Azure Machine Learning: Failed Jobs

```kusto
// ML jobs som har feilet siste 5 dager
AmlComputeJobEvent
| where TimeGenerated > ago(5d) and EventType == "JobFailed"
| project TimeGenerated, ClusterId, EventType, ExecutionState, ToolType
| order by TimeGenerated desc
```

**Bruk:** Rask oversikt over failed training/inference jobs. Drill ned med `JobName` for detaljert analyse.

### Azure Machine Learning: Cluster Node Allocation

```kusto
// Node allocation over tid (capacity planning)
AmlComputeClusterEvent
| where TimeGenerated > ago(1d)
| summarize avgRunningNodes=avg(TargetNodeCount), maxRunningNodes=max(TargetNodeCount)
    by Workspace=tostring(split(_ResourceId, "/")[8]), ClusterName, VmSize
| order by maxRunningNodes desc
```

**Forklaring:** Identifiser peak node-bruk for å optimalisere cluster sizing og kostnader.

## Error Investigation Patterns

### Pattern 1: Error Spike Detection

```kusto
// Finn tidspunkter med unormal feilrate
let baselineErrorRate = toscalar(
    AzureDiagnostics
    | where TimeGenerated > ago(7d)
    | where ResourceProvider == "MICROSOFT.OPENAI"
    | summarize ErrorRate = todouble(countif(ResultSignature != "200")) / count()
);
AzureDiagnostics
| where TimeGenerated > ago(24h)
| where ResourceProvider == "MICROSOFT.OPENAI"
| summarize ErrorRate = todouble(countif(ResultSignature != "200")) / count() by bin(TimeGenerated, 5m)
| where ErrorRate > (baselineErrorRate * 2)  // 2x baseline
| project TimeGenerated, ErrorRate, Threshold = baselineErrorRate * 2
| render timechart
```

**Forklaring:** Baseline-basert anomaly detection. Flagg perioder der feilrate er 2x over 7-dagers gjennomsnitt.

### Pattern 2: Error Message Analysis

```kusto
// Grupper feilmeldinger for pattern-analyse
AzureDiagnostics
| where TimeGenerated > ago(24h)
| where ResultSignature != "200"
| extend ErrorDetails = parse_json(properties_s)
| project TimeGenerated, OperationName, ResultSignature, ErrorMessage = tostring(ErrorDetails.error.message)
| summarize Count = count() by ErrorMessage
| order by Count desc
| take 10
```

**Bruk:** Identifiser vanligste feilmeldinger. Nyttig for å finne repeterende problemer (auth, quota, invalid input).

### Pattern 3: Throttling Detection (429 Errors)

```kusto
// Azure OpenAI throttling events
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.OPENAI"
| where ResultSignature == "429"
| summarize ThrottleCount = count() by bin(TimeGenerated, 10m), OperationName
| render timechart
```

**Cosmos DB variant (for AI-backends):**

```kusto
CDBCassandraRequests
| where ErrorCode == 4097  // Cassandra error code for throttling
| where TimeGenerated > ago(1h)
| project TimeGenerated, DatabaseName, CollectionName, OperationName, RateLimitingDelayMs
```

### Pattern 4: Cross-service Correlation

```kusto
// Korrelasjoner mellom Azure OpenAI errors og AI Search errors
let openaiErrors =
    AzureDiagnostics
    | where ResourceProvider == "MICROSOFT.OPENAI"
    | where ResultSignature != "200"
    | summarize OpenAIErrors = count() by bin(TimeGenerated, 5m);
let searchErrors =
    AzureDiagnostics
    | where ResourceProvider == "MICROSOFT.SEARCH"
    | where resultSignature_d >= 400
    | summarize SearchErrors = count() by bin(TimeGenerated, 5m);
openaiErrors
| join kind=inner searchErrors on TimeGenerated
| project TimeGenerated, OpenAIErrors, SearchErrors
| render timechart
```

**Bruk:** Finn om feil i én AI-tjeneste samvarierer med feil i en annen (f.eks. RAG pipeline: search → OpenAI).

## Cost Analysis Queries

### Token Consumption by Operation

```kusto
// Aggreger token-bruk per operasjonstype
AzureMetrics
| where TimeGenerated > ago(7d)
| where ResourceProvider == "MICROSOFT.OPENAI"
| where MetricName in ("TokenTransaction", "TotalTokens", "PromptTokens", "CompletionTokens")
| summarize TotalTokens = sum(Total) by MetricName, bin(TimeGenerated, 1d)
| render columnchart
```

**Forklaring:** Visualiserer token-forbruk over tid. Nyttig for å identifisere kostnadsdrivere.

### Cost Estimation (NOK)

```kusto
// Estimer kostnader basert på token-bruk (GPT-4 Turbo priser)
// Anta: Prompt = 0.01 USD / 1K tokens, Completion = 0.03 USD / 1K tokens
// USD/NOK = 10.5 (juster etter gjeldende kurs)
AzureMetrics
| where TimeGenerated > ago(30d)
| where ResourceProvider == "MICROSOFT.OPENAI"
| where MetricName in ("PromptTokens", "CompletionTokens")
| summarize
    PromptTokens = sumif(Total, MetricName == "PromptTokens"),
    CompletionTokens = sumif(Total, MetricName == "CompletionTokens")
    by bin(TimeGenerated, 1d)
| extend
    PromptCostUSD = PromptTokens / 1000 * 0.01,
    CompletionCostUSD = CompletionTokens / 1000 * 0.03,
    TotalCostUSD = (PromptTokens / 1000 * 0.01) + (CompletionTokens / 1000 * 0.03),
    TotalCostNOK = ((PromptTokens / 1000 * 0.01) + (CompletionTokens / 1000 * 0.03)) * 10.5
| project TimeGenerated, PromptTokens, CompletionTokens, TotalCostNOK
| render timechart
```

**Viktig:** Oppdater priser og valutakurs regelmessig. Bruk Azure Cost Management for offisielle kostnader.

### Top Costly Operations

```kusto
// Finn operasjoner med høyest RU-forbruk (Cosmos DB AI-backend)
CDBPartitionKeyRUConsumption
| where TimeGenerated > ago(24h)
| where DatabaseName == "ai_vectors"
| summarize TotalRU = sum(RequestCharge) by OperationName
| order by TotalRU desc
| take 10
```

**Bruk:** Identifiser hvilke operasjoner som driver Cosmos DB-kostnader i AI-løsninger (vector search, embedding storage).

### Hot Partition Detection (Cost & Performance Impact)

```kusto
// Identifiser "hot partitions" som kan drive opp kostnader
CDBPartitionKeyStatistics
| where DatabaseName == "ai_vectors"
| where TimeGenerated > ago(8h)
| summarize StorageUsed = sum(SizeKb), RequestCharge = sum(RequestCharge) by PartitionKey
| order by RequestCharge desc
| take 20
```

**Forklaring:** Hot partitions = ubalansert load → throttling → høyere kostnader. Vurder re-partitioning.

## Query Optimization Techniques

### 1. Filter Early and Often

**❌ Ineffektivt:**

```kusto
AzureDiagnostics
| project TimeGenerated, OperationName, DurationMs
| where TimeGenerated > ago(1d)
| where OperationName == "ChatCompletion"
```

**✅ Optimalisert:**

```kusto
AzureDiagnostics
| where TimeGenerated > ago(1d)  // Filter først
| where OperationName == "ChatCompletion"
| project TimeGenerated, OperationName, DurationMs
```

**Regel:** `where` alltid før `project`. Reduserer datamengde tidlig i pipeline.

### 2. Use `top` Instead of `sort` + `take`

**❌ Ineffektivt:**

```kusto
AzureDiagnostics
| where TimeGenerated > ago(1d)
| sort by TimeGenerated desc
| take 100
```

**✅ Optimalisert:**

```kusto
AzureDiagnostics
| where TimeGenerated > ago(1d)
| top 100 by TimeGenerated desc
```

**Forklaring:** `top` sorterer server-side og returnerer kun N records. Raskere enn `sort` + `take`.

### 3. Limit Time Range Explicitly

**❌ Unngå:**

```kusto
AzureDiagnostics
| where OperationName == "Completion"  // Søker ALL data!
```

**✅ Best practice:**

```kusto
AzureDiagnostics
| where TimeGenerated > ago(7d)  // Eksplisitt tidsfilter
| where OperationName == "Completion"
```

**Forklaring:** Alltid definer time range. Uten `TimeGenerated`-filter kan queries time out på store datasett.

### 4. Avoid `search *` — Use Specific Tables

**❌ Tregt:**

```kusto
search *
| where TimeGenerated > ago(1d)
| where * has "OpenAI"
```

**✅ Raskere:**

```kusto
AzureDiagnostics
| where TimeGenerated > ago(1d)
| where ResourceProvider == "MICROSOFT.OPENAI"
```

**Forklaring:** `search *` scanner alle tabeller. Alltid spesifiser tabell og kolonner.

### 5. Use `summarize` with `bin()` for Time-series

**Pattern:**

```kusto
AzureDiagnostics
| where TimeGenerated > ago(24h)
| summarize avg(DurationMs), count() by bin(TimeGenerated, 5m), OperationName
| render timechart
```

**Forklaring:** `bin(TimeGenerated, 5m)` grupperer data i 5-minutters buckets. Reduserer output-size og gjør visualisering mulig.

### 6. Leverage `has` Over `contains` for Performance

**❌ Tregt (substring match):**

```kusto
| where OperationName contains "Chat"
```

**✅ Raskere (word match):**

```kusto
| where OperationName has "Chat"
```

**Forklaring:** `has` søker etter hele ord, ikke substring. Raskere indeks-lookup.

### 7. Pre-aggregate with `let` Statements

```kusto
// Beregn baseline én gang, reuse
let baseline = toscalar(
    AzureDiagnostics
    | where TimeGenerated > ago(7d)
    | summarize avg(DurationMs)
);
AzureDiagnostics
| where TimeGenerated > ago(1h)
| summarize CurrentAvg = avg(DurationMs)
| extend BaselineAvg = baseline, Diff = CurrentAvg - baseline
```

**Forklaring:** `let` lagrer intermediære resultater. Unngå duplicate beregninger.

### 8. Limit Output with `take` During Development

```kusto
// Test query med begrenset output
AzureDiagnostics
| where TimeGenerated > ago(30d)
| take 100  // Bare 100 rows for testing
```

**Best practice:** Bruk `take 10` eller `take 100` mens du utvikler queries. Fjern før produksjon.

## Advanced Patterns

### Multi-region Aggregation

```kusto
// Aggreger Azure OpenAI metrics på tvers av regions
AzureMetrics
| where ResourceProvider == "MICROSOFT.OPENAI"
| where TimeGenerated > ago(24h)
| extend Region = tostring(split(_ResourceId, "/")[8])
| summarize TotalRequests = sum(Total) by Region, MetricName
| order by TotalRequests desc
```

**Bruk:** Sammenlign load på tvers av Azure-regioner for global AI-deployment.

### Anomaly Detection med `series_decompose_anomalies()`

```kusto
// Automatisk anomaly detection i latency
AzureDiagnostics
| where TimeGenerated > ago(7d)
| where ResourceProvider == "MICROSOFT.OPENAI"
| make-series AvgLatency=avg(DurationMs) on TimeGenerated step 10m
| extend anomalies = series_decompose_anomalies(AvgLatency, 1.5)
| render anomalychart
```

**Forklaring:** `series_decompose_anomalies()` bruker ML-basert anomaly detection. Threshold 1.5 = moderat sensitivitet.

### Workload Patterns (Peak Hours)

```kusto
// Identifiser peak-hours for capacity planning
AzureDiagnostics
| where TimeGenerated > ago(30d)
| where ResourceProvider == "MICROSOFT.OPENAI"
| extend Hour = datetime_part("Hour", TimeGenerated)
| summarize RequestCount = count() by Hour
| render columnchart
```

**Bruk:** Finn når AI-løsningen har høyest trafikk. Optimaliser autoscaling og PTU-allokeringer.

### User Behavior Analysis (fra query strings)

```kusto
// Analyser bruker-queries i AI Search
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.SEARCH"
| where OperationName == "Query.Search"
| where Query_s != "?api-version=2025-09-01&search=*"  // Filtrer health checks
| project TimeGenerated, Query_s, Documents_d
| summarize SearchCount = count() by Query_s
| order by SearchCount desc
| take 20
```

**Forklaring:** Finn hyppigst brukte søk. Optimaliser indekser og suggestions basert på reelt bruksmønster.

## For Cosmo: Anvendelse i Arkitekturrådgivning

### Scenario 1: RAG Performance Troubleshooting

**Problem:** Kunde rapporterer treg respons i RAG-løsning (Azure AI Search + Azure OpenAI).

**Tilnærming:**

1. **Mål latency per komponent:**

```kusto
// AI Search query latency
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.SEARCH"
| where TimeGenerated > ago(1h)
| summarize p95_search = percentile(DurationMs, 95);

// OpenAI completion latency
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.OPENAI"
| where TimeGenerated > ago(1h)
| summarize p95_openai = percentile(DurationMs, 95);
```

2. **Korreler tidsstempler** for å finne bottleneck (search vs. completion).

3. **Drill ned** med queries fra "Long-running Queries" og "Latency Percentiles" seksjoner over.

### Scenario 2: Overspent AI Budget

**Problem:** Kunde har brukt 80% av månedlig AI-budsjett på dag 15.

**Tilnærming:**

1. **Identifiser kostnadsdrivere:**

```kusto
// Hvilke operasjoner bruker mest tokens?
AzureMetrics
| where TimeGenerated > ago(15d)
| where MetricName == "TotalTokens"
| summarize TotalTokens = sum(Total) by tostring(parse_json(properties).ModelName)
| order by TotalTokens desc
```

2. **Finn hot users/apps** (krever custom dimensions i logging):

```kusto
AzureDiagnostics
| where TimeGenerated > ago(15d)
| extend AppId = tostring(parse_json(properties_s).appId)
| summarize RequestCount = count() by AppId
| order by RequestCount desc
```

3. **Anbefalinger:** Implementer caching, prompt-optimalisering, eller switch til billigere modeller for visse operasjoner.

### Scenario 3: Proaktiv Alerting Setup

**Anbefaling til kunde:**

Sett opp Azure Monitor alerts basert på KQL-queries:

- **Latency alert:**

```kusto
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.OPENAI"
| summarize p95 = percentile(DurationMs, 95) by bin(TimeGenerated, 5m)
| where p95 > 2000  // Alert hvis p95 > 2s
```

- **Error rate alert:**

```kusto
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.OPENAI"
| summarize ErrorRate = todouble(countif(ResultSignature != "200")) / count() by bin(TimeGenerated, 5m)
| where ErrorRate > 0.05  // Alert hvis > 5% feilrate
```

- **Cost anomaly alert:**

```kusto
AzureMetrics
| where MetricName == "TotalTokens"
| make-series TokensPerHour=sum(Total) on TimeGenerated step 1h
| extend anomalies = series_decompose_anomalies(TokensPerHour, 2.0)
| where anomalies > 0  // Alert på token-spikes
```

### Scenario 4: Compliance & Audit Logging

**Problem:** Kunde i offentlig sektor må dokumentere AI-bruk for revisjon.

**Løsning:** KQL-queries for audit trail:

```kusto
// Hvilke brukere har aksessert AI-tjenester?
AzureDiagnostics
| where TimeGenerated > ago(90d)
| where ResourceProvider in ("MICROSOFT.OPENAI", "MICROSOFT.SEARCH")
| extend User = tostring(parse_json(properties_s).userId)
| summarize RequestCount = count(), FirstAccess = min(TimeGenerated), LastAccess = max(TimeGenerated) by User
| order by RequestCount desc
```

**Export til CSV** for arkivering:

```kusto
// Kjør query i Log Analytics → "Export" → "CSV (all columns)"
```

## Viktige KQL-ressurser

- **KQL Quick Reference:** [learn.microsoft.com/kusto/query/kql-quick-reference](https://learn.microsoft.com/en-us/kusto/query/kql-quick-reference)
- **Azure Monitor KQL Samples:** [learn.microsoft.com/azure/azure-monitor/logs/queries](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/queries)
- **Azure OpenAI Monitoring:** [learn.microsoft.com/azure/ai-foundry/openai/how-to/monitor-openai](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/monitor-openai)
- **Optimize Log Queries:** [learn.microsoft.com/azure/azure-monitor/logs/query-optimization](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/query-optimization)

## Nøkkelinnsikter

- **Filter tidlig:** `where TimeGenerated` alltid først for å begrense datamengde.
- **Bruk `top` over `sort` + `take`:** Server-side optimalisering.
- **Percentiler > gjennomsnitt:** p95/p99 gir bedre innsikt i brukeropplevelse enn avg.
- **`has` > `contains`:** Raskere word-match vs. substring-match.
- **Pre-aggreger med `let`:** Unngå duplicate beregninger.
- **Test med `take`:** Begrens output under query-utvikling.
- **Korreler på tvers av tjenester:** `join` for å finne cross-service dependencies.
- **Visualiser med `render`:** `timechart`, `columnchart`, `anomalychart` for innsikt.

## Referanser

- Microsoft Learn: [Monitor Azure OpenAI](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/monitor-openai)
- Microsoft Learn: [Get started with log queries in Azure Monitor](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/get-started-queries)
- Microsoft Learn: [Optimize log queries in Azure Monitor](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/query-optimization)
- Microsoft Learn: [Configure diagnostic logging for Azure AI Search](https://learn.microsoft.com/en-us/azure/search/search-monitor-enable-logging)
- Microsoft Learn: [Monitor Azure Machine Learning](https://learn.microsoft.com/en-us/azure/machine-learning/monitor-azure-machine-learning)
- Microsoft Learn: [KQL quick reference](https://learn.microsoft.com/en-us/kusto/query/kql-quick-reference)