ms-ai-architect/skills/ms-ai-governance/references/monitoring-observability/distributed-tracing-ai-pipelines.md

# Distributed Tracing for AI Pipelines

**Category:** Monitoring & Observability
**Last updated:** 2026-06-19
**Status:** ✅ Komplett
**Type:** reference
**Source:** https://learn.microsoft.com/azure/azure-monitor/app/opentelemetry-overview

## Innhold

- [Innledning](#innledning)
- [Nøkkelkonsepter](#nøkkelkonsepter)
- [OpenTelemetry for AI Pipelines](#opentelemetry-for-ai-pipelines)
- [Implementering i Microsoft-stakken](#implementering-i-microsoft-stakken)
- [End-to-End Trace Correlation](#end-to-end-trace-correlation)
- [Trace Visualization og Analysis](#trace-visualization-og-analysis)
- [Performance Bottleneck Identification](#performance-bottleneck-identification)
- [Best Practices](#best-practices)
- [Azure Functions OpenTelemetry Pattern](#azure-functions-opentelemetry-pattern)
- [Integrasjon med AI Foundry Tracing](#integrasjon-med-ai-foundry-tracing)
- [Troubleshooting Common Issues](#troubleshooting-common-issues)
- [For Cosmo](#for-cosmo)
- [Kilder og verifisering](#kilder-og-verifisering)

## Innledning

Distributed tracing (distribuert sporing) gir end-to-end synlighet gjennom hele AI-pipelinens kjede av operasjoner — fra brukerforespørsel, via LLM-kall, tool-anrop og multi-agent-samarbeid, til ferdig respons. Dette er kritisk for å diagnostisere ytelsesflaskehalser, identifisere feiltilstander, og optimalisere komplekse agentic AI-systemer.

Microsoft sin tilnærming er bygget på **OpenTelemetry**-standarder og integrerer sømløst med **Azure Monitor Application Insights**, med native støtte for AI-spesifikke semantiske konvensjoner (OpenTelemetry Gen AI Semantic Conventions).

## Nøkkelkonsepter

### Traces, Spans og Correlation

- **Trace:** Fullstendig reise for en operasjon gjennom systemet (f.eks. én brukerforespørsel til en AI-agent)
- **Span:** Individuell operasjon innenfor en trace (LLM-kall, tool-invokasjon, HTTP-request)
- **Attributes:** Key-value metadata knyttet til spans (model name, token count, tool parameters)
- **Correlation ID:** `operation_Id` og `operation_ParentId` som knytter alle spans i en trace sammen

### W3C Trace Context

Microsoft støtter W3C Trace Context-standarden for cross-service propagation:

- **traceparent:** Globally unique operation ID + span ID (propageres via HTTP-headers)
- **tracestate:** System-spesifikk trace-kontekst
- **Bakoverkompatibilitet:** Application Insights SDK støtter både W3C og legacy Request-Id-protokoller

## OpenTelemetry for AI Pipelines

### Semantic Conventions for Generative AI

OpenTelemetry definerer standardiserte span-navn og attributter for AI-operasjoner:

**Standard AI Spans:**
- `gen_ai.model.completion` — LLM-inferens
- `gen_ai.tool.execution` — Tool/function-kall
- `gen_ai.agent.invoke` — Agent-invokasjon
- `gen_ai.agent_planning` — Agent-planleggingssteg
- `gen_ai.agent_to_agent_interaction` — Multi-agent-kommunikasjon

**Standard Attributter:**
- `gen_ai.system` — AI-system (OpenAI, Azure AI, etc.)
- `gen_ai.request.model` — Modellnavn
- `gen_ai.usage.prompt_tokens` — Prompt-tokens
- `gen_ai.usage.completion_tokens` — Completion-tokens
- `gen_ai.response.finish_reason` — Årsak til ferdigstillelse

### Multi-Agent Observability

Microsoft har utviklet nye semantic conventions for multi-agent-systemer (i samarbeid med Cisco Outshift):

| Span Type | Formål | Eksempel |
|-----------|--------|----------|
| `execute_task` | Overvåker task-dekomponering og event-propagering | Bryter ned kompleks forespørsel |
| `agent_to_agent_interaction` | Sporer kommunikasjon mellom agenter | Agent A ber Agent B om data |
| `agent.state.management` | Kontekst- og minnehåndtering | Long-term memory-oppdatering |
| `agent_planning` | Agentens interne planleggingssteg | Reasoning-steg før tool-valg |
| `agent_orchestration` | Agent-til-agent-orkestrering | Main agent delegerer til sub-agents |

## Implementering i Microsoft-stakken

### 1. Microsoft Foundry + Azure Monitor

**Setup (Python):**

```python
import os
from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential
from azure.monitor.opentelemetry import configure_azure_monitor
from opentelemetry import trace

# Enable content recording (valgfritt - kan inneholde sensitive data)
os.environ["AZURE_TRACING_GEN_AI_CONTENT_RECORDING_ENABLED"] = "true"

# Koble til AI Foundry-prosjekt
project_client = AIProjectClient(
    credential=DefaultAzureCredential(),
    endpoint=os.environ["PROJECT_ENDPOINT"]
)

# Hent Application Insights connection string
connection_string = project_client.telemetry.get_application_insights_connection_string()

# Konfigurer Azure Monitor
configure_azure_monitor(connection_string=connection_string)

# Start tracing
tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("ai-agent-session"):
    agent = project_client.agents.create_agent(
        model="gpt-4o",
        name="support-agent",
        instructions="Du er en supportagent"
    )
    thread = project_client.agents.threads.create()
    message = project_client.agents.messages.create(
        thread_id=thread.id,
        role="user",
        content="Hjelp meg med å feilsøke"
    )
    run = project_client.agents.runs.create_and_process(
        thread_id=thread.id,
        agent_id=agent.id
    )
```

### 2. Azure Functions + OpenTelemetry

**Konfigurer host.json:**

```json
{
  "version": "2.0",
  "telemetryMode": "OpenTelemetry",
  "extensions": {
    "serviceBus": {
      "maxConcurrentCalls": 10
    }
  },
  "extensionBundle": {
    "id": "Microsoft.Azure.Functions.ExtensionBundle",
    "version": "[4.*, 5.0.0)"
  }
}
```

**Python Function med tracing:**

```python
import azure.functions as func
from azure.monitor.opentelemetry import configure_azure_monitor
import os

# Konfigurer Azure Monitor
configure_azure_monitor(
    connection_string=os.environ["APPLICATIONINSIGHTS_CONNECTION_STRING"]
)

app = func.FunctionApp()

@app.function_name("orchestrator")
@app.route(route="orchestrator", auth_level=func.AuthLevel.ANONYMOUS)
def orchestrator(req: func.HttpRequest) -> func.HttpResponse:
    # Automatisk tracet av Azure Functions OpenTelemetry-integrasjon
    # Alle HTTP-kall, Service Bus-meldinger, og dependencies trackes
    return func.HttpResponse("OK", status_code=200)
```

**Viktige forbehold for OpenTelemetry-modus i Functions** *(Verified MCP 2026-06-19)*:
- OTel aktiveres på app-nivå i både `host.json` (`"telemetryMode": "OpenTelemetry"`) og i koden. Når du oppgir både `APPLICATIONINSIGHTS_CONNECTION_STRING` og en OTLP-eksportør (`OTEL_EXPORTER_OTLP_ENDPOINT`/`OTEL_EXPORTER_OTLP_HEADERS`), sendes telemetri til begge endepunktene.
- I OTel-modus støtter Azure-portalen **ikke** log streaming, og `Recent function invocation`-traces vises kun hvis telemetri sendes til Azure Monitor. Logging-konfigurasjon under `logging.applicationInsights` i `host.json` gjelder ikke.
- **Parent-based sampling er standard.** Triggere som HTTP, Service Bus og Event Hubs avhenger av context propagation; request-telemetri genereres ikke når den innkommende requesten/meldingen ikke samples. `OperationId` hentes direkte fra `traceparent` — gjenbruk av samme `traceparent` gir samme `OperationId`.
- Filtre i `host.json` gjelder kun host-prosessens logger; worker-prosessens logger filtreres via språkspesifikke OTel-innstillinger. Go-worker har egen opt-in OTel-middleware (`middleware/otelfunc`).

### 3. LangChain/LangGraph + Azure AI Tracing

**Setup:**

```python
from langchain_azure_ai.callbacks.tracers import AzureAIOpenTelemetryTracer
from langchain_openai import AzureChatOpenAI
import os

# Opprett tracer
azure_tracer = AzureAIOpenTelemetryTracer(
    connection_string=os.environ["APPLICATION_INSIGHTS_CONNECTION_STRING"],
    enable_content_recording=True,
    name="LangChain Agent",
    id="langchain_agent_v1"
)

# Konfigurer model med callbacks
model = AzureChatOpenAI(
    azure_deployment=os.environ["AZURE_OPENAI_CHAT_DEPLOYMENT"],
    azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
    api_version="2024-08-01-preview",
    callbacks=[azure_tracer]
)

# Alle LLM-kall, tool-invokasjon, og agent-steg trackes automatisk
```

### 4. Semantic Kernel

Semantic Kernel har innebygd OpenTelemetry-støtte:

**Automatisk metrics:**
- `semantic_kernel.function.invocation.duration` (Histogram) — Funksjonsutførelsestid
- `semantic_kernel.function.streaming.duration` (Histogram) — Streaming-utførelsestid
- `semantic_kernel.function.invocation.token_usage.prompt` — Prompt-tokens
- `semantic_kernel.function.invocation.token_usage.completion` — Completion-tokens

**Aktiviteter (Spans):**
- Hver kernel function-execution genererer en Activity
- Hver AI-modellkall genereres som egen Activity
- Activity source: `"Microsoft.SemanticKernel"`

### 5. Custom Functions og Tools

**Trace egne funksjoner:**

```python
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

def rag_retrieval(query: str) -> list[str]:
    with tracer.start_as_current_span("rag_retrieval") as span:
        span.set_attribute("query", query)
        span.set_attribute("retrieval.database", "azure_ai_search")

        # Utfør retrieval
        results = search_index(query)

        span.set_attribute("retrieval.results_count", len(results))
        span.set_attribute("retrieval.latency_ms", 120)

        return results

def agent_tool_call(tool_name: str, arguments: dict):
    with tracer.start_as_current_span("execute_tool") as span:
        span.set_attribute("tool.name", tool_name)
        span.set_attribute("tool.call.arguments", str(arguments))

        result = execute_tool(tool_name, arguments)

        span.set_attribute("tool.call.results", str(result))
        return result
```

## End-to-End Trace Correlation

### Distribuert Tracing Across Services

**Scenario:** Bruker → Azure Functions → Azure OpenAI → Azure AI Search → Response

**Trace Flow:**

1. **HTTP Request** (traceparent-header propageres automatisk)
   - `operation_Id`: `abc123def456`
   - Span: `GET /api/chat`

2. **Azure Function Processing**
   - `operation_ParentId`: `abc123def456`
   - Span: `process_chat_request`

3. **Azure OpenAI API Call** (dependency tracked)
   - `operation_ParentId`: `process_chat_request`
   - Span: `gen_ai.model.completion`
   - Attributes: `model=gpt-4o`, `prompt_tokens=150`, `completion_tokens=75`

4. **Azure AI Search Query** (dependency tracked)
   - `operation_ParentId`: `process_chat_request`
   - Span: `azure_ai_search.query`
   - Attributes: `index=knowledge_base`, `results_count=5`

5. **Service Bus Message** (context propageres via message properties)
   - `operation_ParentId`: `process_chat_request`
   - Span: `servicebus.send`

**Resultat i Application Insights:**
- Application Map viser alle tjenester grafisk
- Transaction Search viser fullstendig call stack
- End-to-End Transaction Details viser timing for hver operasjon

### Query Traces i Application Insights

**Kusto Query for å finne relatert telemetri:**

```kusto
let operationId = "abc123def456";
(requests | union dependencies | union traces | union exceptions)
| where operation_Id == operationId
| project timestamp, itemType, name, id, operation_ParentId, operation_Id, duration
| order by timestamp asc
```

**Analyse AI-spesifikke spans:**

```kusto
dependencies
| where type == "AI"
| extend model = tostring(customDimensions.["gen_ai.request.model"])
| extend promptTokens = toint(customDimensions.["gen_ai.usage.prompt_tokens"])
| extend completionTokens = toint(customDimensions.["gen_ai.usage.completion_tokens"])
| summarize
    avgDuration = avg(duration),
    totalPromptTokens = sum(promptTokens),
    totalCompletionTokens = sum(completionTokens),
    requestCount = count()
    by model
| order by avgDuration desc
```

## Trace Visualization og Analysis

### Application Insights Features

**1. Application Map**
- Visuell representasjon av tjeneste-dependencies
- Automatisk deteksjon av performance-problemer
- Highlighting av feiltilstander

**2. Transaction Search**
- Søk etter spesifikke traces basert på:
  - Operation ID
  - Tidsvindu
  - Resultat (success/failure)
  - Duration threshold

**3. End-to-End Transaction Details**
- Komplett trace timeline
- Span-detaljer (start/end times, attributes)
- Korrelerte logger
- Performance metrics per span

**4. Performance View**
- Gjennomsnittlig duration per operation
- P95/P99 latency
- Dependency latency breakdown

**5. Failures Blade**
- Exception tracking korrelert med traces
- Failure rate per endpoint
- Root cause analysis

### Local Tracing (Development)

**Aspire Dashboard (lokal OTLP viewer):**

```bash
pip install opentelemetry-exporter-otlp

# Start Aspire Dashboard
docker run --rm -it -p 18888:18888 -p 4317:18889 \
  mcr.microsoft.com/dotnet/aspire-dashboard:latest
```

**Console Export (debugging):**

```python
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor
from opentelemetry.sdk.trace import TracerProvider

span_exporter = ConsoleSpanExporter()
tracer_provider = TracerProvider()
tracer_provider.add_span_processor(SimpleSpanProcessor(span_exporter))
trace.set_tracer_provider(tracer_provider)
```

## Performance Bottleneck Identification

### Analyse Latency Distribution

**Identifiser trege spans:**

```kusto
dependencies
| where operation_Name == "chat_completion"
| summarize
    p50 = percentile(duration, 50),
    p90 = percentile(duration, 90),
    p99 = percentile(duration, 99)
    by name
| where p99 > 5000 // Over 5 sekunder
```

**Finn flaskehalser i multi-step pipeline:**

```kusto
let traceId = "abc123";
dependencies
| where operation_Id == traceId
| project timestamp, name, duration, operation_ParentId
| order by timestamp asc
// Visualiser i Timeline-chart for å se hvor tid brukes
```

### Token Usage Analysis

```kusto
traces
| where message contains "gen_ai.usage"
| extend promptTokens = toint(customDimensions.["gen_ai.usage.prompt_tokens"])
| extend completionTokens = toint(customDimensions.["gen_ai.usage.completion_tokens"])
| summarize
    totalCost = sum((promptTokens * 0.00003) + (completionTokens * 0.00006))
    by bin(timestamp, 1h)
| render timechart
```

## Best Practices

### 1. Consistent Span Attributes

Bruk standardiserte attributt-navn:
- `gen_ai.*` for AI-spesifikke spans
- `tool.*` for tool-invokasjon
- `agent.*` for agent-metadata
- Følg OpenTelemetry Semantic Conventions

### 2. Redact Sensitive Content

**Ikke log sensitive data i spans:**

```python
# IKKE gjør dette:
span.set_attribute("user.password", password)

# Gjør dette i stedet:
span.set_attribute("user.id", user_id)
span.set_attribute("request.sanitized", True)
```

**Deaktiver content recording i prod:**

```python
# Development
os.environ["AZURE_TRACING_GEN_AI_CONTENT_RECORDING_ENABLED"] = "true"

# Production
os.environ["AZURE_TRACING_GEN_AI_CONTENT_RECORDING_ENABLED"] = "false"
```

### 3. Correlate Evaluation Runs

Knytt trace IDs til evaluation-runs:

```python
span.set_attribute("evaluation.run_id", evaluation_run_id)
span.set_attribute("evaluation.metrics", json.dumps(metrics))
```

### 4. Service Name for Multi-App Scenarios

Identifiser tjenester via `OTEL_SERVICE_NAME`:

```bash
export OTEL_SERVICE_NAME="support-agent-api"
export OTEL_RESOURCE_ATTRIBUTES="service.namespace=production,service.instance.id=instance-01"
```

I Application Insights mappes dette til `cloud_RoleName`:

```kusto
traces
| where cloud_RoleName == "support-agent-api"
```

### 5. Sampling for High-Volume Scenarios

**Adaptive sampling (Application Insights SDK / klassisk):**
- Reduserer volum uten å miste viktige traces
- Prioriterer feil og trege forespørsler

> **Merk:** Azure Monitor OpenTelemetry-distroen sampler **ikke** som standard. Distroen støtter fixed-rate og rate-limited samplere som må konfigureres eksplisitt; trace-basert sampling for logger er default-på først når sampling er aktivert. Adaptive sampling over gjelder den klassiske Application Insights SDK-en, ikke OTel-distroen. *(Verified MCP 2026-06-19)*

**Custom sampling (avansert):**

```python
from opentelemetry.sdk.trace.sampling import TraceIdRatioBased

# Sample 10% av traces
sampler = TraceIdRatioBased(rate=0.1)
tracer_provider = TracerProvider(sampler=sampler)
```

## Azure Functions OpenTelemetry Pattern

### Multi-Function Distributed Trace

**Function 1 (HTTP Trigger):**

```python
@app.route(route="function1")
def function1(req: func.HttpRequest) -> func.HttpResponse:
    # Caller function2 (automatic trace propagation)
    response = requests.get(f"{base_url}/api/function2")
    return func.HttpResponse(response.text)
```

**Function 2 (HTTP Trigger + Service Bus Output):**

```python
@app.route(route="function2")
@app.service_bus_queue_output(
    arg_name="outputmsg",
    queue_name="processing-queue",
    connection="ServiceBusConnection"
)
def function2(req: func.HttpRequest, outputmsg: func.Out[str]):
    # Send message (trace context propageres automatisk)
    outputmsg.set("Process this")
    return func.HttpResponse("OK")
```

**Function 3 (Service Bus Trigger):**

```python
@app.service_bus_queue_trigger(
    arg_name="msg",
    queue_name="processing-queue",
    connection="ServiceBusConnection"
)
def function3(msg: func.ServiceBusMessage):
    # Automatisk korrelert med function1 og function2
    logging.info(f"Processing: {msg.get_body().decode()}")
```

**Resultat:** En enkelt HTTP-request til function1 genererer en komplett trace som viser:
- HTTP request → function1
- function1 → function2 (HTTP dependency)
- function2 → Service Bus (messaging dependency)
- Service Bus → function3 (queue trigger)

## Integrasjon med AI Foundry Tracing

### View Traces i Foundry Portal

1. Naviger til **Tracing** i AI Foundry-prosjekt
2. Filtrer traces etter:
   - Tidsvindu
   - Status (success/failed)
   - Agent/model
3. Drill-down i individual trace for span-detaljer

### Thread Logs i Agents Playground

- **Thread details:** Fullstendig konversasjonshistorikk
- **Run information:** Agent execution metadata
- **Ordered run steps:** Sekvens av operasjoner
- **Tool calls:** Input/output for hver tool-invokasjon
- **Linked evaluations:** Automatic quality metrics (hvis aktivert)

## Troubleshooting Common Issues

### Problem: Traces not appearing in Application Insights

**Løsning:**
1. Verifiser connection string:
   ```python
   print(os.environ["APPLICATIONINSIGHTS_CONNECTION_STRING"])
   ```
2. Sjekk at `configure_azure_monitor()` kalles tidlig i app lifecycle
3. Vent 2-5 minutter (ingestion lag)
4. Sjekk sampling rate (hvis custom sampling)

### Problem: Missing trace context across services

**Løsning:**
1. Verifiser W3C Trace Context headers propageres:
   ```python
   # Inspect outgoing request headers
   print(request.headers.get("traceparent"))
   ```
2. Bruk instrumentation libraries (ikke manual HTTP calls uten context propagation)
3. For Azure Functions: Sjekk at alle functions har `"telemetryMode": "OpenTelemetry"`

### Problem: High cardinality attributes causing performance issues

**Løsning:**
- Unngå unique IDs som span attributes (bruk aggregated metrics i stedet)
- Reduser sampling rate for høy-volum scenarios
- Bruk tags/dimensions med lav cardinality

## For Cosmo

Ved arkitekturveiledning:

**Når bruker spør om:**
- "Hvordan kan jeg feilsøke min AI-pipeline?"
- "Hvordan tracke end-to-end ytelse i multi-agent-systemet?"
- "Hvordan finne flaskehalser i RAG-pipeline?"
- "Hvordan korrelere LLM-kall med tool-invokasjon?"

**Svar med:**
1. **Beskriv trace-arkitektur:** Spans → Traces → Operation ID correlation
2. **Anbefal OpenTelemetry + Azure Monitor:** Native støtte, AI-spesifikke semantics
3. **Gi konkret implementering:** Vis code snippets for brukerens plattform (Foundry, Functions, LangChain, etc.)
4. **Highlight Application Insights features:** Application Map, Transaction Search, Performance View
5. **Sikkerhet:** Påminn om content recording (deaktiver i prod hvis sensitive data)
6. **Query-eksempler:** Gi Kusto-queries for vanlige analyse-scenarioer

**Decision factors:**
- **High-volume scenarios:** Vurder adaptive sampling
- **Multi-region deployments:** Bruk `cloud_RoleName` og `cloud_RoleInstance` for å skille instances
- **Compliance-krav:** Deaktiver content recording, bruk private Application Insights
- **Local development:** Anbefal Aspire Dashboard for rask feedback

**Trade-offs:**
- **Detailed tracing vs. storage cost:** Mer spans = høyere Application Insights-kostnad
- **Content recording vs. privacy:** Recording av prompts/completions kan eksponere PII
- **Real-time vs. historical analysis:** Live Metrics vs. Kusto queries

---

## Kilder og verifisering

Adapted from Microsoft Learn documentation ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)):

- [Tracing in Microsoft Foundry](https://learn.microsoft.com/en-us/azure/ai-studio/how-to/develop/trace-local-sdk)
- [Azure Monitor OpenTelemetry overview](https://learn.microsoft.com/en-us/azure/azure-monitor/app/opentelemetry-overview)
- [Azure Functions OpenTelemetry](https://learn.microsoft.com/en-us/azure/azure-functions/opentelemetry-howto) *(Verified MCP 2026-06-19 — parent-based sampling default; OTLP + App Insights dual-export; portal log-streaming/recent-invocation-traces krever Azure Monitor)*
- [Distributed tracing in Application Insights](https://learn.microsoft.com/en-us/azure/azure-monitor/app/distributed-trace-data)
- [Semantic Kernel observability](https://learn.microsoft.com/en-us/semantic-kernel/concepts/enterprise-readiness/observability/)

Content has been translated to Norwegian, reorganized, and augmented with implementation guidance.

**Relaterte referanser:**
- `azure-monitor-foundations.md` — Application Insights-grunnlag
- `token-tracking.md` — Token usage monitoring
- `alerting-ai-systems.md` — Alerting på trace data
- `app-insights-ai-integration.md` — Application Insights AI-features