Kjell Tore Guttormsen 1ab4fc34cf feat(ms-ai-architect): decision-b Enhet 2 — Dato→Last-updated relabel 16 filer (21 header-kandidater: 16 rene + 5 dual utskilt til dedup), isRealDateValue placeholder-guard [skip-docs]

2026-07-06 11:08:42 +02:00

22 KiB

Raw Blame History

Distributed Tracing for AI Pipelines

Category: Monitoring & Observability Last updated: 2026-06-19 Status: ✅ Komplett Type: reference Source: https://learn.microsoft.com/azure/azure-monitor/app/opentelemetry-overview

Innledning

Distributed tracing (distribuert sporing) gir end-to-end synlighet gjennom hele AI-pipelinens kjede av operasjoner — fra brukerforespørsel, via LLM-kall, tool-anrop og multi-agent-samarbeid, til ferdig respons. Dette er kritisk for å diagnostisere ytelsesflaskehalser, identifisere feiltilstander, og optimalisere komplekse agentic AI-systemer.

Microsoft sin tilnærming er bygget på OpenTelemetry-standarder og integrerer sømløst med Azure Monitor Application Insights, med native støtte for AI-spesifikke semantiske konvensjoner (OpenTelemetry Gen AI Semantic Conventions).

Nøkkelkonsepter

Traces, Spans og Correlation

Trace: Fullstendig reise for en operasjon gjennom systemet (f.eks. én brukerforespørsel til en AI-agent)
Span: Individuell operasjon innenfor en trace (LLM-kall, tool-invokasjon, HTTP-request)
Attributes: Key-value metadata knyttet til spans (model name, token count, tool parameters)
Correlation ID: operation_Id og operation_ParentId som knytter alle spans i en trace sammen

W3C Trace Context

Microsoft støtter W3C Trace Context-standarden for cross-service propagation:

traceparent: Globally unique operation ID + span ID (propageres via HTTP-headers)
tracestate: System-spesifikk trace-kontekst
Bakoverkompatibilitet: Application Insights SDK støtter både W3C og legacy Request-Id-protokoller

OpenTelemetry for AI Pipelines

Semantic Conventions for Generative AI

OpenTelemetry definerer standardiserte span-navn og attributter for AI-operasjoner:

Standard AI Spans:

gen_ai.model.completion — LLM-inferens
gen_ai.tool.execution — Tool/function-kall
gen_ai.agent.invoke — Agent-invokasjon
gen_ai.agent_planning — Agent-planleggingssteg
gen_ai.agent_to_agent_interaction — Multi-agent-kommunikasjon

Standard Attributter:

gen_ai.system — AI-system (OpenAI, Azure AI, etc.)
gen_ai.request.model — Modellnavn
gen_ai.usage.prompt_tokens — Prompt-tokens
gen_ai.usage.completion_tokens — Completion-tokens
gen_ai.response.finish_reason — Årsak til ferdigstillelse

Multi-Agent Observability

Microsoft har utviklet nye semantic conventions for multi-agent-systemer (i samarbeid med Cisco Outshift):

Span Type	Formål	Eksempel
`execute_task`	Overvåker task-dekomponering og event-propagering	Bryter ned kompleks forespørsel
`agent_to_agent_interaction`	Sporer kommunikasjon mellom agenter	Agent A ber Agent B om data
`agent.state.management`	Kontekst- og minnehåndtering	Long-term memory-oppdatering
`agent_planning`	Agentens interne planleggingssteg	Reasoning-steg før tool-valg
`agent_orchestration`	Agent-til-agent-orkestrering	Main agent delegerer til sub-agents

Implementering i Microsoft-stakken

1. Microsoft Foundry + Azure Monitor

Setup (Python):

import os
from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential
from azure.monitor.opentelemetry import configure_azure_monitor
from opentelemetry import trace

# Enable content recording (valgfritt - kan inneholde sensitive data)
os.environ["AZURE_TRACING_GEN_AI_CONTENT_RECORDING_ENABLED"] = "true"

# Koble til AI Foundry-prosjekt
project_client = AIProjectClient(
    credential=DefaultAzureCredential(),
    endpoint=os.environ["PROJECT_ENDPOINT"]
)

# Hent Application Insights connection string
connection_string = project_client.telemetry.get_application_insights_connection_string()

# Konfigurer Azure Monitor
configure_azure_monitor(connection_string=connection_string)

# Start tracing
tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("ai-agent-session"):
    agent = project_client.agents.create_agent(
        model="gpt-4o",
        name="support-agent",
        instructions="Du er en supportagent"
    )
    thread = project_client.agents.threads.create()
    message = project_client.agents.messages.create(
        thread_id=thread.id,
        role="user",
        content="Hjelp meg med å feilsøke"
    )
    run = project_client.agents.runs.create_and_process(
        thread_id=thread.id,
        agent_id=agent.id
    )

2. Azure Functions + OpenTelemetry

Konfigurer host.json:

{
  "version": "2.0",
  "telemetryMode": "OpenTelemetry",
  "extensions": {
    "serviceBus": {
      "maxConcurrentCalls": 10
    }
  },
  "extensionBundle": {
    "id": "Microsoft.Azure.Functions.ExtensionBundle",
    "version": "[4.*, 5.0.0)"
  }
}

Python Function med tracing:

import azure.functions as func
from azure.monitor.opentelemetry import configure_azure_monitor
import os

# Konfigurer Azure Monitor
configure_azure_monitor(
    connection_string=os.environ["APPLICATIONINSIGHTS_CONNECTION_STRING"]
)

app = func.FunctionApp()

@app.function_name("orchestrator")
@app.route(route="orchestrator", auth_level=func.AuthLevel.ANONYMOUS)
def orchestrator(req: func.HttpRequest) -> func.HttpResponse:
    # Automatisk tracet av Azure Functions OpenTelemetry-integrasjon
    # Alle HTTP-kall, Service Bus-meldinger, og dependencies trackes
    return func.HttpResponse("OK", status_code=200)

Viktige forbehold for OpenTelemetry-modus i Functions (Verified MCP 2026-06-19):

OTel aktiveres på app-nivå i både host.json ("telemetryMode": "OpenTelemetry") og i koden. Når du oppgir både APPLICATIONINSIGHTS_CONNECTION_STRING og en OTLP-eksportør (OTEL_EXPORTER_OTLP_ENDPOINT/OTEL_EXPORTER_OTLP_HEADERS), sendes telemetri til begge endepunktene.
I OTel-modus støtter Azure-portalen ikke log streaming, og Recent function invocation-traces vises kun hvis telemetri sendes til Azure Monitor. Logging-konfigurasjon under logging.applicationInsights i host.json gjelder ikke.
Parent-based sampling er standard. Triggere som HTTP, Service Bus og Event Hubs avhenger av context propagation; request-telemetri genereres ikke når den innkommende requesten/meldingen ikke samples. OperationId hentes direkte fra traceparent — gjenbruk av samme traceparent gir samme OperationId.
Filtre i host.json gjelder kun host-prosessens logger; worker-prosessens logger filtreres via språkspesifikke OTel-innstillinger. Go-worker har egen opt-in OTel-middleware (middleware/otelfunc).

3. LangChain/LangGraph + Azure AI Tracing

Setup:

from langchain_azure_ai.callbacks.tracers import AzureAIOpenTelemetryTracer
from langchain_openai import AzureChatOpenAI
import os

# Opprett tracer
azure_tracer = AzureAIOpenTelemetryTracer(
    connection_string=os.environ["APPLICATION_INSIGHTS_CONNECTION_STRING"],
    enable_content_recording=True,
    name="LangChain Agent",
    id="langchain_agent_v1"
)

# Konfigurer model med callbacks
model = AzureChatOpenAI(
    azure_deployment=os.environ["AZURE_OPENAI_CHAT_DEPLOYMENT"],
    azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
    api_version="2024-08-01-preview",
    callbacks=[azure_tracer]
)

# Alle LLM-kall, tool-invokasjon, og agent-steg trackes automatisk

4. Semantic Kernel

Semantic Kernel har innebygd OpenTelemetry-støtte:

Automatisk metrics:

semantic_kernel.function.invocation.duration (Histogram) — Funksjonsutførelsestid
semantic_kernel.function.streaming.duration (Histogram) — Streaming-utførelsestid
semantic_kernel.function.invocation.token_usage.prompt — Prompt-tokens
semantic_kernel.function.invocation.token_usage.completion — Completion-tokens

Aktiviteter (Spans):

Hver kernel function-execution genererer en Activity
Hver AI-modellkall genereres som egen Activity
Activity source: "Microsoft.SemanticKernel"

5. Custom Functions og Tools

Trace egne funksjoner:

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

def rag_retrieval(query: str) -> list[str]:
    with tracer.start_as_current_span("rag_retrieval") as span:
        span.set_attribute("query", query)
        span.set_attribute("retrieval.database", "azure_ai_search")

        # Utfør retrieval
        results = search_index(query)

        span.set_attribute("retrieval.results_count", len(results))
        span.set_attribute("retrieval.latency_ms", 120)

        return results

def agent_tool_call(tool_name: str, arguments: dict):
    with tracer.start_as_current_span("execute_tool") as span:
        span.set_attribute("tool.name", tool_name)
        span.set_attribute("tool.call.arguments", str(arguments))

        result = execute_tool(tool_name, arguments)

        span.set_attribute("tool.call.results", str(result))
        return result

End-to-End Trace Correlation

Distribuert Tracing Across Services

Scenario: Bruker → Azure Functions → Azure OpenAI → Azure AI Search → Response

Trace Flow:

HTTP Request (traceparent-header propageres automatisk)
- operation_Id: abc123def456
- Span: GET /api/chat
Azure Function Processing
- operation_ParentId: abc123def456
- Span: process_chat_request
Azure OpenAI API Call (dependency tracked)
- operation_ParentId: process_chat_request
- Span: gen_ai.model.completion
- Attributes: model=gpt-4o, prompt_tokens=150, completion_tokens=75
Azure AI Search Query (dependency tracked)
- operation_ParentId: process_chat_request
- Span: azure_ai_search.query
- Attributes: index=knowledge_base, results_count=5
Service Bus Message (context propageres via message properties)
- operation_ParentId: process_chat_request
- Span: servicebus.send

Resultat i Application Insights:

Application Map viser alle tjenester grafisk
Transaction Search viser fullstendig call stack
End-to-End Transaction Details viser timing for hver operasjon

Query Traces i Application Insights

Kusto Query for å finne relatert telemetri:

let operationId = "abc123def456";
(requests | union dependencies | union traces | union exceptions)
| where operation_Id == operationId
| project timestamp, itemType, name, id, operation_ParentId, operation_Id, duration
| order by timestamp asc

Analyse AI-spesifikke spans:

dependencies
| where type == "AI"
| extend model = tostring(customDimensions.["gen_ai.request.model"])
| extend promptTokens = toint(customDimensions.["gen_ai.usage.prompt_tokens"])
| extend completionTokens = toint(customDimensions.["gen_ai.usage.completion_tokens"])
| summarize
    avgDuration = avg(duration),
    totalPromptTokens = sum(promptTokens),
    totalCompletionTokens = sum(completionTokens),
    requestCount = count()
    by model
| order by avgDuration desc

Trace Visualization og Analysis

Application Insights Features

1. Application Map

Visuell representasjon av tjeneste-dependencies
Automatisk deteksjon av performance-problemer
Highlighting av feiltilstander

2. Transaction Search

Søk etter spesifikke traces basert på:
- Operation ID
- Tidsvindu
- Resultat (success/failure)
- Duration threshold

3. End-to-End Transaction Details

Komplett trace timeline
Span-detaljer (start/end times, attributes)
Korrelerte logger
Performance metrics per span

4. Performance View

Gjennomsnittlig duration per operation
P95/P99 latency
Dependency latency breakdown

5. Failures Blade

Exception tracking korrelert med traces
Failure rate per endpoint
Root cause analysis

Local Tracing (Development)

Aspire Dashboard (lokal OTLP viewer):

pip install opentelemetry-exporter-otlp

# Start Aspire Dashboard
docker run --rm -it -p 18888:18888 -p 4317:18889 \
  mcr.microsoft.com/dotnet/aspire-dashboard:latest

Console Export (debugging):

from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor
from opentelemetry.sdk.trace import TracerProvider

span_exporter = ConsoleSpanExporter()
tracer_provider = TracerProvider()
tracer_provider.add_span_processor(SimpleSpanProcessor(span_exporter))
trace.set_tracer_provider(tracer_provider)

Performance Bottleneck Identification

Analyse Latency Distribution

Identifiser trege spans:

dependencies
| where operation_Name == "chat_completion"
| summarize
    p50 = percentile(duration, 50),
    p90 = percentile(duration, 90),
    p99 = percentile(duration, 99)
    by name
| where p99 > 5000 // Over 5 sekunder

Finn flaskehalser i multi-step pipeline:

let traceId = "abc123";
dependencies
| where operation_Id == traceId
| project timestamp, name, duration, operation_ParentId
| order by timestamp asc
// Visualiser i Timeline-chart for å se hvor tid brukes

Token Usage Analysis

traces
| where message contains "gen_ai.usage"
| extend promptTokens = toint(customDimensions.["gen_ai.usage.prompt_tokens"])
| extend completionTokens = toint(customDimensions.["gen_ai.usage.completion_tokens"])
| summarize
    totalCost = sum((promptTokens * 0.00003) + (completionTokens * 0.00006))
    by bin(timestamp, 1h)
| render timechart

Best Practices

1. Consistent Span Attributes

Bruk standardiserte attributt-navn:

gen_ai.* for AI-spesifikke spans
tool.* for tool-invokasjon
agent.* for agent-metadata
Følg OpenTelemetry Semantic Conventions

2. Redact Sensitive Content

Ikke log sensitive data i spans:

# IKKE gjør dette:
span.set_attribute("user.password", password)

# Gjør dette i stedet:
span.set_attribute("user.id", user_id)
span.set_attribute("request.sanitized", True)

Deaktiver content recording i prod:

# Development
os.environ["AZURE_TRACING_GEN_AI_CONTENT_RECORDING_ENABLED"] = "true"

# Production
os.environ["AZURE_TRACING_GEN_AI_CONTENT_RECORDING_ENABLED"] = "false"

3. Correlate Evaluation Runs

Knytt trace IDs til evaluation-runs:

span.set_attribute("evaluation.run_id", evaluation_run_id)
span.set_attribute("evaluation.metrics", json.dumps(metrics))

4. Service Name for Multi-App Scenarios

Identifiser tjenester via OTEL_SERVICE_NAME:

export OTEL_SERVICE_NAME="support-agent-api"
export OTEL_RESOURCE_ATTRIBUTES="service.namespace=production,service.instance.id=instance-01"

I Application Insights mappes dette til cloud_RoleName:

traces
| where cloud_RoleName == "support-agent-api"

5. Sampling for High-Volume Scenarios

Adaptive sampling (Application Insights SDK / klassisk):

Reduserer volum uten å miste viktige traces
Prioriterer feil og trege forespørsler

Merk: Azure Monitor OpenTelemetry-distroen sampler ikke som standard. Distroen støtter fixed-rate og rate-limited samplere som må konfigureres eksplisitt; trace-basert sampling for logger er default-på først når sampling er aktivert. Adaptive sampling over gjelder den klassiske Application Insights SDK-en, ikke OTel-distroen. (Verified MCP 2026-06-19)

Custom sampling (avansert):

from opentelemetry.sdk.trace.sampling import TraceIdRatioBased

# Sample 10% av traces
sampler = TraceIdRatioBased(rate=0.1)
tracer_provider = TracerProvider(sampler=sampler)

Azure Functions OpenTelemetry Pattern

Multi-Function Distributed Trace

Function 1 (HTTP Trigger):

@app.route(route="function1")
def function1(req: func.HttpRequest) -> func.HttpResponse:
    # Caller function2 (automatic trace propagation)
    response = requests.get(f"{base_url}/api/function2")
    return func.HttpResponse(response.text)

Function 2 (HTTP Trigger + Service Bus Output):

@app.route(route="function2")
@app.service_bus_queue_output(
    arg_name="outputmsg",
    queue_name="processing-queue",
    connection="ServiceBusConnection"
)
def function2(req: func.HttpRequest, outputmsg: func.Out[str]):
    # Send message (trace context propageres automatisk)
    outputmsg.set("Process this")
    return func.HttpResponse("OK")

Function 3 (Service Bus Trigger):

@app.service_bus_queue_trigger(
    arg_name="msg",
    queue_name="processing-queue",
    connection="ServiceBusConnection"
)
def function3(msg: func.ServiceBusMessage):
    # Automatisk korrelert med function1 og function2
    logging.info(f"Processing: {msg.get_body().decode()}")

Resultat: En enkelt HTTP-request til function1 genererer en komplett trace som viser:

HTTP request → function1
function1 → function2 (HTTP dependency)
function2 → Service Bus (messaging dependency)
Service Bus → function3 (queue trigger)

Integrasjon med AI Foundry Tracing

View Traces i Foundry Portal

Naviger til Tracing i AI Foundry-prosjekt
Filtrer traces etter:
- Tidsvindu
- Status (success/failed)
- Agent/model
Drill-down i individual trace for span-detaljer

Thread Logs i Agents Playground

Thread details: Fullstendig konversasjonshistorikk
Run information: Agent execution metadata
Ordered run steps: Sekvens av operasjoner
Tool calls: Input/output for hver tool-invokasjon
Linked evaluations: Automatic quality metrics (hvis aktivert)

Troubleshooting Common Issues

Problem: Traces not appearing in Application Insights

Løsning:

Verifiser connection string:

print(os.environ["APPLICATIONINSIGHTS_CONNECTION_STRING"])

Sjekk at configure_azure_monitor() kalles tidlig i app lifecycle
Vent 2-5 minutter (ingestion lag)
Sjekk sampling rate (hvis custom sampling)

Problem: Missing trace context across services

Løsning:

Verifiser W3C Trace Context headers propageres:

# Inspect outgoing request headers
print(request.headers.get("traceparent"))

Bruk instrumentation libraries (ikke manual HTTP calls uten context propagation)
For Azure Functions: Sjekk at alle functions har "telemetryMode": "OpenTelemetry"

Problem: High cardinality attributes causing performance issues

Løsning:

Unngå unique IDs som span attributes (bruk aggregated metrics i stedet)
Reduser sampling rate for høy-volum scenarios
Bruk tags/dimensions med lav cardinality

For Cosmo

Ved arkitekturveiledning:

Når bruker spør om:

"Hvordan kan jeg feilsøke min AI-pipeline?"
"Hvordan tracke end-to-end ytelse i multi-agent-systemet?"
"Hvordan finne flaskehalser i RAG-pipeline?"
"Hvordan korrelere LLM-kall med tool-invokasjon?"

Svar med:

Beskriv trace-arkitektur: Spans → Traces → Operation ID correlation
Anbefal OpenTelemetry + Azure Monitor: Native støtte, AI-spesifikke semantics
Gi konkret implementering: Vis code snippets for brukerens plattform (Foundry, Functions, LangChain, etc.)
Highlight Application Insights features: Application Map, Transaction Search, Performance View
Sikkerhet: Påminn om content recording (deaktiver i prod hvis sensitive data)
Query-eksempler: Gi Kusto-queries for vanlige analyse-scenarioer

Decision factors:

High-volume scenarios: Vurder adaptive sampling
Multi-region deployments: Bruk cloud_RoleName og cloud_RoleInstance for å skille instances
Compliance-krav: Deaktiver content recording, bruk private Application Insights
Local development: Anbefal Aspire Dashboard for rask feedback

Trade-offs:

Detailed tracing vs. storage cost: Mer spans = høyere Application Insights-kostnad
Content recording vs. privacy: Recording av prompts/completions kan eksponere PII
Real-time vs. historical analysis: Live Metrics vs. Kusto queries

Kilder og verifisering

Adapted from Microsoft Learn documentation (CC BY 4.0):

Tracing in Microsoft Foundry
Azure Monitor OpenTelemetry overview
Azure Functions OpenTelemetry (Verified MCP 2026-06-19 — parent-based sampling default; OTLP + App Insights dual-export; portal log-streaming/recent-invocation-traces krever Azure Monitor)
Distributed tracing in Application Insights
Semantic Kernel observability

Content has been translated to Norwegian, reorganized, and augmented with implementation guidance.

Relaterte referanser:

azure-monitor-foundations.md — Application Insights-grunnlag
token-tracking.md — Token usage monitoring
alerting-ai-systems.md — Alerting på trace data
app-insights-ai-integration.md — Application Insights AI-features

22 KiB Raw Blame History

Distributed Tracing for AI Pipelines

Innhold

Innledning

Nøkkelkonsepter

Traces, Spans og Correlation

W3C Trace Context

OpenTelemetry for AI Pipelines

Semantic Conventions for Generative AI

Multi-Agent Observability

Implementering i Microsoft-stakken

1. Microsoft Foundry + Azure Monitor

2. Azure Functions + OpenTelemetry

3. LangChain/LangGraph + Azure AI Tracing

4. Semantic Kernel

5. Custom Functions og Tools

End-to-End Trace Correlation

Distribuert Tracing Across Services

Query Traces i Application Insights

Trace Visualization og Analysis

Application Insights Features

Local Tracing (Development)

Performance Bottleneck Identification

Analyse Latency Distribution

Token Usage Analysis

Best Practices

1. Consistent Span Attributes

2. Redact Sensitive Content

3. Correlate Evaluation Runs

4. Service Name for Multi-App Scenarios

5. Sampling for High-Volume Scenarios

Azure Functions OpenTelemetry Pattern

Multi-Function Distributed Trace

Integrasjon med AI Foundry Tracing

View Traces i Foundry Portal

Thread Logs i Agents Playground

Troubleshooting Common Issues

Problem: Traces not appearing in Application Insights

Problem: Missing trace context across services

Problem: High cardinality attributes causing performance issues

For Cosmo

Kilder og verifisering

22 KiB

Raw Blame History