ms-ai-architect/skills/ms-ai-governance/references/monitoring-observability/observability-for-copilot-extensions.md

Observability Patterns for Copilot Extensions and Plugins

Last updated: 2026-05 Status: GA Category: Monitoring & Observability Type: reference Source: https://learn.microsoft.com/microsoft-cloud/dev/copilot/isv/observability-for-ai

---

Innhold

• Introduksjon
• Kjernekomponenter
• Arkitekturmønstre
• Beslutningsveiledning
• Integrasjon med Microsoft-stakken
• Offentlig sektor (Norge)
• Kostnad og lisensiering
• For arkitekten (Cosmo)
• Kilder og verifisering

Introduksjon

Når organisasjoner utvider Microsoft Copilot med custom plugins, connectors og extensions, blir observability kritisk for å sikre pålitelighet, ytelse og compliance. I motsetning til standalone applikasjoner opererer Copilot-extensions i et distribuert økosystem hvor telemetri må samles fra flere lag: plugin-kjøretid, API-kall, LLM-interaksjoner og brukeropplevelse.

Microsoft tilbyr et helhetlig observability-rammeverk basert på Azure Application Insights, Copilot Studio analytics og Azure Monitor. Dette gir innsikt i plugin performance, token-forbruk, error rates, user engagement og sikkerhetshendelser. For Copilot Studio-agenter er Application Insights-integrasjon nå en out-of-the-box feature som logger incoming/outgoing messages, topic triggers og custom telemetry events.

Utfordringen ligger i å instrumentere extensions korrekt, definere relevante metrics (både system- og business-metrics), og bygge dashboards som gir actionable insights for både utviklere, data scientists og business stakeholders. I offentlig sektor må observability også dekke compliance-logging for Forvaltningsloven § 11 (journalføring av vedtak) og GDPR Article 35 (DPIA monitoring).

---

Kjernekomponenter

Telemetry Layers for Copilot Extensions

Layer	Data Captured	Tool	Purpose
Copilot Studio Agent	Messages, topics, custom events, design mode	Application Insights	Track agent behavior, conversation flow, topic performance
Plugin/Connector Runtime	API calls, latency, errors, token usage	Application Insights SDK	Monitor external integrations, debug failures
LLM Interaction	Prompt tokens, completion tokens, model latency, groundedness	Azure OpenAI metrics	Cost tracking, performance optimization
User Engagement	Thumbs up/down, edit distance, session duration	Custom events	Measure usefulness, iterate on UX
Security/Compliance	Filtered prompts, PII detection, audit logs	Microsoft Sentinel, Purview	Governance, risk management

Application Insights Integration for Copilot Studio

Configuration Steps:

1. Navigate to Settings → Advanced in Copilot Studio
2. Add Application Insights Connection string (from Azure Portal)
3. Enable optional settings:
   • Log activities: Incoming/outgoing messages and events
   • Log sensitive Activity properties: userid, name, text, speak (vurder GDPR-implikasjoner)
   • Log custom telemetry events: Via "Log custom telemetry event" node in topics

Custom Dimensions (customDimensions field):

Field	Description	Sample Values
type	Activity type	message, conversationUpdate, event, invoke
channelId	Channel identifier	emulator, directline, msteams, webchat
designMode	Test canvas vs. production	True / False
locale	User locale	en-us, nb-no, sv-se
text	Message text (if logging enabled)	User prompt/agent response

Pre-Built Dashboards

Copilot Studio Workbook (Preview) – Tilgjengelig i Application Insights:

• Path: Application Insights → Monitoring → Workbooks → "Copilot Studio Dashboard"
• Metrics: Total conversations, latency, exceptions, tool usage, topic analytics
• Customization: Edit mode for adding KQL queries (e.g., track custom attributes)

---

Arkitekturmønstre

Pattern 1: Centralized Telemetry Hub

Bruk: Enterprise med mange Copilot-extensions på tvers av teams.

Arkitektur:

Copilot Studio Agent(s) → Application Insights (Workspace 1)
                      ↓
M365 Copilot Plugin(s) → Application Insights (Workspace 2) → Azure Workbook (Consolidated)
                      ↓
Power Platform Connector(s) → Application Insights (Workspace 3)
                      ↓
Microsoft Sentinel (Audit Logs via Purview)

Fordeler:

• Felles sikkerhetspolicy og RBAC (Reader role for team members)
• Cross-correlation av events på tvers av extensions
• Compliance-logging aggregert i Sentinel

Ulemper:

• Krever Application Insights API-tilgang for cross-workspace queries
• Høyere kostnad ved separate workspaces (vurder single workspace hvis <500GB/month)

---

Pattern 2: Plugin-Specific Instrumentation

Bruk: Custom plugin/connector utviklet med pro-code (C#, TypeScript).

Implementering:

// C# example - Application Insights SDK
using Microsoft.ApplicationInsights;
using Microsoft.ApplicationInsights.DataContracts;

var telemetryClient = new TelemetryClient();
telemetryClient.Context.GlobalProperties["PluginName"] = "SalesforceConnector";

// Track plugin execution
var stopwatch = Stopwatch.StartNew();
try
{
    var result = await ExecutePluginAsync(request);
    telemetryClient.TrackEvent("PluginSuccess", new Dictionary<string, string>
    {
        { "operation", request.Operation },
        { "duration_ms", stopwatch.ElapsedMilliseconds.ToString() }
    });
}
catch (Exception ex)
{
    telemetryClient.TrackException(ex);
    telemetryClient.TrackMetric("PluginErrorRate", 1);
}

Fordeler:

• Full kontroll over metrics og custom properties
• Multi-layer instrumentation (tokenize → infer → generate → detokenize)
• Granular performance debugging

Ulemper:

• Requires code changes for every extension
• DevOps overhead (ensure SDK updates)

---

Pattern 3: User Feedback Loop

Bruk: Kontinuerlig forbedring basert på brukerrespons.

Flow:

1. User interacts with Copilot → Agent response
2. Thumbs up/down → Custom telemetry event: UserFeedback
3. Edit distance tracked → Metric: avg_edit_distance
4. KQL query identifies low-rated topics → Trigger re-evaluation

KQL Example:

customEvents
| where name == "UserFeedback"
| extend rating = customDimensions['rating']
| where rating == "down"
| summarize count() by tostring(customDimensions['topicName'])
| order by count_ desc

Fordeler:

• Direkte input fra sluttbrukere
• Data-driven topic/prompt iteration

Ulemper:

• Feedback bias (users rarely rate neutral experiences)
• Privacy concerns (GDPR Article 6 – lawful basis for processing feedback)

---

Beslutningsveiledning

Når bruke hvilken løsning?

Scenario	Anbefalt Tool	Reasoning
Copilot Studio agent (low/no-code)	Built-in analytics + App Insights	No SDK required, out-of-the-box setup
Custom M365 Copilot plugin (TypeScript)	Application Insights SDK	Full control, correlation with Azure OpenAI metrics
Power Platform connector	Power Platform telemetry + App Insights	Hybrid (connector-level + custom events)
Compliance audit (Forvaltningsloven)	Microsoft Sentinel + Purview	Audit logs for decisions/actions
Cost tracking (Azure OpenAI)	Azure Monitor (OpenAI resource metrics)	Token-level billing data

Vanlige feil

Feil	Konsekvens	Løsning
Logger sensitive data (PII) uten consent	GDPR Article 5 brudd, Datatilsynet-varsel	Disable "Log sensitive Activity properties" OR anonymize/hash userid/text
Inkluderer test-data i production metrics	Falske performance-trender	Filter designMode == "False" in all KQL queries
Mangler correlation IDs	Kan ikke tracke multi-step flows	Use Activity.Current.RootId (ASP.NET Core) eller custom correlation headers
Ignorer latency breakdown	Identify bottleneck i feil lag	Instrument tokenize, infer, generate, detokenize separat
Ingen alerting på error spikes	Incidents oppdages for sent	Set up Azure Monitor alerts (e.g., >5% error rate in 5 min window)

Røde flagg

• Manglende telemetri i 30+ dager → Brukere ikke adoptert extension?
• Edit distance >50% of response length → Agent gir irrelevante svar
• >10% filtered prompts → Content filtering blokkerer legitim bruk (juster policy)
• Token cost øker 3x uten brukerøkning → Prompt inefficiency eller token leakage

---

Integrasjon med Microsoft-stakken

Azure Monitor Ecosystem

Application Insights ← Copilot Studio, Plugins, Connectors
        ↓
Azure Monitor Workspace → Azure Copilot observability agent (preview)
        ↓
Microsoft Sentinel ← Audit logs (via Purview)
        ↓
Power BI / Azure Workbooks → Executive dashboards

Key Integrations:

Integration	Use Case	Configuration
Azure OpenAI metrics	Token usage, model latency, throttling	No extra config – auto-emitted to Azure Monitor
Purview audit logs	Copilot Studio events (BotCreate, BotPublish, BotShare)	Enable audit logging for Microsoft 365 license holders
Sentinel analytics	Custom detection rules (e.g., unusual token spikes)	Ingest App Insights logs, create KQL-based rules
Copilot Studio Kit	Automated testing + telemetry enrichment	Register Azure AD app, grant App Insights API permissions

Cross-Service Correlation

Scenario: M365 Copilot plugin calls Azure Function → Azure OpenAI → Cosmos DB

Solution:

1. Distributed tracing: Use traceparent HTTP header (W3C standard)
2. Correlation ID: Propagate operation_Id through all layers
3. KQL join:

requests
| join (dependencies) on operation_Id
| join (customEvents | where name == "LLMInvocation") on operation_Id
| project timestamp, request_name, dependency_name, llm_tokens=customDimensions['tokens']

---

Offentlig sektor (Norge)

GDPR og Schrems II

Utfordring: Application Insights lagrer data i Azure-region (e.g., West Europe). Schrems II krever vurdering av USA-baserte sub-processors.

Mitigering:

• Bruk EU Data Boundary (Microsoft commitment per nov 2024)
• Aktivér Data Residency i Application Insights (Settings → Data retention)
• DPIA for logging av text (personopplysninger i meldinger)

Forvaltningsloven § 11

Krav: Journalføring av vedtak truffet av forvaltning.

Implementering:

1. Custom telemetry event når Copilot-agent treffer "decision topic":

telemetryClient.TrackEvent("DecisionMade", new Dictionary<string, string>
{
    { "decisionType", "LoanApproval" },
    { "caseId", "2026-001234" },
    { "timestamp", DateTime.UtcNow.ToString("o") }
});

2. Sentinels analytics rule → arkivering i case management system (e.g., ePhorte)

AI Act (EU 2024/1689)

Artikkel 12: High-risk AI-systemer skal logge operations for traceability.

Relevans: Copilot-agent som automatiserer saksbehandling = high-risk.

Compliance:

• Log alle inputs (prompts), outputs (responses), intermediate steps (topic flow)
• Retention: Minimum 6 måneder (AI Act Article 12(1))
• Access control: Kun autoriserte brukere (RBAC via Application Insights)

---

Kostnad og lisensiering

Prismodell (Application Insights)

Component	Pricing	Optimization Tips
Data ingestion	$2.76/GB (first 5GB free/month)	Use sampling (e.g., 50% for non-critical events)
Data retention	Free (90 days), $0.12/GB/month beyond	Archive to Azure Storage for long-term compliance
Web tests	$5.75 per test/month	Not required for Copilot extensions (use synthetic monitoring via Azure Functions)

Estimert cost for Copilot Studio agent (1000 users, 50k msgs/month):

• Telemetry volume: ~10GB/month (hvis logging av messages enabled)
• Cost: $13.80/month (ingestion) + retention cost
• Tip: Disable "Log sensitive Activity properties" → reduser volume 30%

Lisens-krav

Feature	License Requirement
Copilot Studio analytics (built-in)	Power Virtual Agents license / Copilot Studio capacity
Application Insights integration	Azure subscription (free tier available)
Microsoft Sentinel (audit logs)	Microsoft 365 E5 OR Sentinel standalone
Power BI dashboards	Power BI Pro per user OR Premium capacity

---

For arkitekten (Cosmo)

5 spørsmål å stille kunden

1. Scope: Hvilke Copilot-extensions skal overvåkes? (Copilot Studio agents, M365 plugins, Power Platform connectors?)
2. Compliance: Er dette high-risk AI under AI Act? Trenger dere audit trail for Forvaltningsloven?
3. Sensitive data: Logger dere meldingstekst? Har dere DPIA for logging av personopplysninger?
4. Alerting: Hvem skal varsles ved error spikes, cost overruns eller security events?
5. Retention: Hvor lenge må telemetri oppbevares? (GDPR minimums vs. compliance-krav)

Fallgruver

Fallgruve	Impact	Hvordan unngå
Over-logging i test-fase	Kostnadssprekk	Filter designMode == "False" i KQL
Manglende sampling strategy	Unødvendig detaljnivå → dyrt	100% logging for errors, 10-50% for success events
Ingen incident response plan	Treg respons på security events	Set up Azure Monitor action groups (email, SMS, webhook til Teams/Slack)
Siloed telemetry	Kan ikke correlate plugin + LLM + backend	Bruk distributed tracing (W3C traceparent)

Anbefalinger per modenhetsnivå

Level 1: MVP (First Copilot Extension)

• Bruk Copilot Studio built-in analytics
• Enable Application Insights med basic logging (ikke sensitive properties)
• Set up 2-3 alerts (error rate >5%, response time >3s)

Level 2: Production Scale (5+ extensions)

• Centralized Application Insights workspace
• Custom telemetry events for business metrics (e.g., "LoanApprovalGranted")
• Pre-built dashboards (Copilot Studio Workbook + custom Azure Workbook)

Level 3: Enterprise/Compliance-heavy

• Microsoft Sentinel integration for audit logs
• Distributed tracing across all tiers (plugin → LLM → backend)
• Automated anomaly detection (Azure Monitor ML-based alerts)
• Quarterly compliance audit exports (GDPR, AI Act)

---

Kilder og verifisering

Verified (fra Microsoft Learn MCP):

1. Capture telemetry with Application Insights - Copilot Studio – Full guide til App Insights setup, KQL queries, custom dimensions
2. Observability for pro-code generative AI solutions – ISV-guidance: lifecycle phases, metrics categories, evaluation techniques
3. Monitor operations, compliance, and capacity - Copilot Studio – Operational monitoring, Sentinel integration, compliance auditing
4. Application Insights telemetry with Microsoft Copilot Studio (Dynamics 365) – Prerequisites, custom events, topic tracking
5. Enable Application Insights support in Copilot Studio Kit – Azure AD app registration, API permissions for telemetry enrichment

Baseline (modellkunnskap, verifisert mot docs):

• GDPR Article 5 (data minimization), Article 6 (lawful basis), Article 35 (DPIA)
• AI Act (EU 2024/1689) Article 12 (logging for high-risk AI)
• Forvaltningsloven § 11 (journalføring av vedtak)

Konfidensnivå per seksjon:

• Kjernekomponenter: Verified (App Insights docs, Copilot Studio Workbook)
• Arkitekturmønstre: Baseline (patterns basert på Azure Well-Architected Framework + docs)
• Offentlig sektor: Verified (GDPR/AI Act legal text + Microsoft EU Data Boundary docs)
• Kostnad: Verified (Azure pricing calculator, Application Insights pricing page)