open/ms-ai-architect
1
0
Fork 0
Code Issues Releases 4 Activity

feat(ultraplan-local): v1.6.0 — /ultraresearch-local deep research command

Browse source 
Add /ultraresearch-local for structured research combining local codebase
analysis with external knowledge via parallel agent swarms. Produces research
briefs with triangulation, confidence ratings, and source quality assessment.

New command: /ultraresearch-local with modes --quick, --local, --external, --fg.
New agents: research-orchestrator (opus), docs-researcher, community-researcher,
security-researcher, contrarian-researcher, gemini-bridge (all sonnet).
New template: research-brief-template.md.

Integration: --research flag in /ultraplan-local accepts pre-built research
briefs (up to 3), enriches the interview and exploration phases. Planning
orchestrator cross-references brief findings during synthesis.

Design principle: Context Engineering — right information to right agent at
right time. Research briefs are structured artifacts in the pipeline:
ultraresearch → brief → ultraplan --research → plan → ultraexecute.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-04-08 08:58:35 +02:00
commit baa2d0220b
488 changed files with 213221 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

354
skills/ms-ai-governance/references/monitoring-observability/observability-for-copilot-extensions.md Normal file
View file

@ -0,0 +1,354 @@
# Observability Patterns for Copilot Extensions and Plugins
**Last updated:** 2026-02
**Status:** GA
**Category:** Monitoring & Observability
---
## Introduksjon
Når organisasjoner utvider Microsoft Copilot med custom plugins, connectors og extensions, blir observability kritisk for å sikre pålitelighet, ytelse og compliance. I motsetning til standalone applikasjoner opererer Copilot-extensions i et distribuert økosystem hvor telemetri må samles fra flere lag: plugin-kjøretid, API-kall, LLM-interaksjoner og brukeropplevelse.
Microsoft tilbyr et helhetlig observability-rammeverk basert på Azure Application Insights, Copilot Studio analytics og Azure Monitor. Dette gir innsikt i plugin performance, token-forbruk, error rates, user engagement og sikkerhetshendelser. For Copilot Studio-agenter er Application Insights-integrasjon nå en out-of-the-box feature som logger incoming/outgoing messages, topic triggers og custom telemetry events.
Utfordringen ligger i å instrumentere extensions korrekt, definere relevante metrics (både system- og business-metrics), og bygge dashboards som gir actionable insights for både utviklere, data scientists og business stakeholders. I offentlig sektor må observability også dekke compliance-logging for Forvaltningsloven § 11 (journalføring av vedtak) og GDPR Article 35 (DPIA monitoring).
---
## Kjernekomponenter
### Telemetry Layers for Copilot Extensions
| Layer | Data Captured | Tool | Purpose |
|-------|---------------|------|---------|
| **Copilot Studio Agent** | Messages, topics, custom events, design mode | Application Insights | Track agent behavior, conversation flow, topic performance |
| **Plugin/Connector Runtime** | API calls, latency, errors, token usage | Application Insights SDK | Monitor external integrations, debug failures |
| **LLM Interaction** | Prompt tokens, completion tokens, model latency, groundedness | Azure OpenAI metrics | Cost tracking, performance optimization |
| **User Engagement** | Thumbs up/down, edit distance, session duration | Custom events | Measure usefulness, iterate on UX |
| **Security/Compliance** | Filtered prompts, PII detection, audit logs | Microsoft Sentinel, Purview | Governance, risk management |
### Application Insights Integration for Copilot Studio
**Configuration Steps:**
1. Navigate to **Settings → Advanced** in Copilot Studio
2. Add Application Insights **Connection string** (from Azure Portal)
3. Enable optional settings:
- **Log activities**: Incoming/outgoing messages and events
- **Log sensitive Activity properties**: userid, name, text, speak (vurder GDPR-implikasjoner)
- **Log custom telemetry events**: Via "Log custom telemetry event" node in topics
**Custom Dimensions (customDimensions field):**
| Field | Description | Sample Values |
|-------|-------------|---------------|
| `type` | Activity type | `message`, `conversationUpdate`, `event`, `invoke` |
| `channelId` | Channel identifier | `emulator`, `directline`, `msteams`, `webchat` |
| `designMode` | Test canvas vs. production | `True` / `False` |
| `locale` | User locale | `en-us`, `nb-no`, `sv-se` |
| `text` | Message text (if logging enabled) | User prompt/agent response |
### Pre-Built Dashboards
**Copilot Studio Workbook (Preview)** – Tilgjengelig i Application Insights:
- **Path:** Application Insights → Monitoring → Workbooks → "Copilot Studio Dashboard"
- **Metrics:** Total conversations, latency, exceptions, tool usage, topic analytics
- **Customization:** Edit mode for adding KQL queries (e.g., track custom attributes)
---
## Arkitekturmønstre
### Pattern 1: Centralized Telemetry Hub
**Bruk:** Enterprise med mange Copilot-extensions på tvers av teams.
**Arkitektur:**
```
Copilot Studio Agent(s) → Application Insights (Workspace 1)
↓
M365 Copilot Plugin(s) → Application Insights (Workspace 2) → Azure Workbook (Consolidated)
↓
Power Platform Connector(s) → Application Insights (Workspace 3)
↓
Microsoft Sentinel (Audit Logs via Purview)
```
**Fordeler:**
- Felles sikkerhetspolicy og RBAC (Reader role for team members)
- Cross-correlation av events på tvers av extensions
- Compliance-logging aggregert i Sentinel
**Ulemper:**
- Krever Application Insights API-tilgang for cross-workspace queries
- Høyere kostnad ved separate workspaces (vurder single workspace hvis <500GB/month)
---
### Pattern 2: Plugin-Specific Instrumentation
**Bruk:** Custom plugin/connector utviklet med pro-code (C#, TypeScript).
**Implementering:**
```csharp
// C# example - Application Insights SDK
using Microsoft.ApplicationInsights;
using Microsoft.ApplicationInsights.DataContracts;
var telemetryClient = new TelemetryClient();
telemetryClient.Context.GlobalProperties["PluginName"] = "SalesforceConnector";
// Track plugin execution
var stopwatch = Stopwatch.StartNew();
try
{
var result = await ExecutePluginAsync(request);
telemetryClient.TrackEvent("PluginSuccess", new Dictionary<string, string>
{
{ "operation", request.Operation },
{ "duration_ms", stopwatch.ElapsedMilliseconds.ToString() }
});
}
catch (Exception ex)
{
telemetryClient.TrackException(ex);
telemetryClient.TrackMetric("PluginErrorRate", 1);
}
```
**Fordeler:**
- Full kontroll over metrics og custom properties
- Multi-layer instrumentation (tokenize → infer → generate → detokenize)
- Granular performance debugging
**Ulemper:**
- Requires code changes for every extension
- DevOps overhead (ensure SDK updates)
---
### Pattern 3: User Feedback Loop
**Bruk:** Kontinuerlig forbedring basert på brukerrespons.
**Flow:**
1. User interacts with Copilot → Agent response
2. Thumbs up/down → Custom telemetry event: `UserFeedback`
3. Edit distance tracked → Metric: `avg_edit_distance`
4. KQL query identifies low-rated topics → Trigger re-evaluation
**KQL Example:**
```kusto
customEvents
| where name == "UserFeedback"
| extend rating = customDimensions['rating']
| where rating == "down"
| summarize count() by tostring(customDimensions['topicName'])
| order by count_ desc
```
**Fordeler:**
- Direkte input fra sluttbrukere
- Data-driven topic/prompt iteration
**Ulemper:**
- Feedback bias (users rarely rate neutral experiences)
- Privacy concerns (GDPR Article 6 – lawful basis for processing feedback)
---
## Beslutningsveiledning
### Når bruke hvilken løsning?
| Scenario | Anbefalt Tool | Reasoning |
|----------|---------------|-----------|
| Copilot Studio agent (low/no-code) | Built-in analytics + App Insights | No SDK required, out-of-the-box setup |
| Custom M365 Copilot plugin (TypeScript) | Application Insights SDK | Full control, correlation with Azure OpenAI metrics |
| Power Platform connector | Power Platform telemetry + App Insights | Hybrid (connector-level + custom events) |
| Compliance audit (Forvaltningsloven) | Microsoft Sentinel + Purview | Audit logs for decisions/actions |
| Cost tracking (Azure OpenAI) | Azure Monitor (OpenAI resource metrics) | Token-level billing data |
### Vanlige feil
| Feil | Konsekvens | Løsning |
|------|------------|---------|
| **Logger sensitive data (PII) uten consent** | GDPR Article 5 brudd, Datatilsynet-varsel | Disable "Log sensitive Activity properties" OR anonymize/hash userid/text |
| **Inkluderer test-data i production metrics** | Falske performance-trender | Filter `designMode == "False"` in all KQL queries |
| **Mangler correlation IDs** | Kan ikke tracke multi-step flows | Use `Activity.Current.RootId` (ASP.NET Core) eller custom correlation headers |
| **Ignorer latency breakdown** | Identify bottleneck i feil lag | Instrument tokenize, infer, generate, detokenize separat |
| **Ingen alerting på error spikes** | Incidents oppdages for sent | Set up Azure Monitor alerts (e.g., >5% error rate in 5 min window) |
### Røde flagg
- **Manglende telemetri i 30+ dager** → Brukere ikke adoptert extension?
- **Edit distance >50% of response length** → Agent gir irrelevante svar
- **>10% filtered prompts** → Content filtering blokkerer legitim bruk (juster policy)
- **Token cost øker 3x uten brukerøkning** → Prompt inefficiency eller token leakage
---
## Integrasjon med Microsoft-stakken
### Azure Monitor Ecosystem
```
Application Insights ← Copilot Studio, Plugins, Connectors
↓
Azure Monitor Workspace → Azure Copilot observability agent (preview)
↓
Microsoft Sentinel ← Audit logs (via Purview)
↓
Power BI / Azure Workbooks → Executive dashboards
```
**Key Integrations:**
| Integration | Use Case | Configuration |
|-------------|----------|---------------|
| **Azure OpenAI metrics** | Token usage, model latency, throttling | No extra config – auto-emitted to Azure Monitor |
| **Purview audit logs** | Copilot Studio events (BotCreate, BotPublish, BotShare) | Enable audit logging for Microsoft 365 license holders |
| **Sentinel analytics** | Custom detection rules (e.g., unusual token spikes) | Ingest App Insights logs, create KQL-based rules |
| **Copilot Studio Kit** | Automated testing + telemetry enrichment | Register Azure AD app, grant App Insights API permissions |
### Cross-Service Correlation
**Scenario:** M365 Copilot plugin calls Azure Function → Azure OpenAI → Cosmos DB
**Solution:**
1. **Distributed tracing**: Use `traceparent` HTTP header (W3C standard)
2. **Correlation ID**: Propagate `operation_Id` through all layers
3. **KQL join**:
```kusto
requests
| join (dependencies) on operation_Id
| join (customEvents | where name == "LLMInvocation") on operation_Id
| project timestamp, request_name, dependency_name, llm_tokens=customDimensions['tokens']
```
---
## Offentlig sektor (Norge)
### GDPR og Schrems II
**Utfordring:** Application Insights lagrer data i Azure-region (e.g., West Europe). Schrems II krever vurdering av USA-baserte sub-processors.
**Mitigering:**
- Bruk **EU Data Boundary** (Microsoft commitment per nov 2024)
- Aktivér **Data Residency** i Application Insights (Settings → Data retention)
- DPIA for logging av `text` (personopplysninger i meldinger)
### Forvaltningsloven § 11
**Krav:** Journalføring av vedtak truffet av forvaltning.
**Implementering:**
1. Custom telemetry event når Copilot-agent treffer "decision topic":
```csharp
telemetryClient.TrackEvent("DecisionMade", new Dictionary<string, string>
{
{ "decisionType", "LoanApproval" },
{ "caseId", "2026-001234" },
{ "timestamp", DateTime.UtcNow.ToString("o") }
});
```
2. Sentinels analytics rule → arkivering i case management system (e.g., ePhorte)
### AI Act (EU 2024/1689)
**Artikkel 12:** High-risk AI-systemer skal logge operations for traceability.
**Relevans:** Copilot-agent som automatiserer saksbehandling = high-risk.
**Compliance:**
- Log alle inputs (prompts), outputs (responses), intermediate steps (topic flow)
- Retention: Minimum 6 måneder (AI Act Article 12(1))
- Access control: Kun autoriserte brukere (RBAC via Application Insights)
---
## Kostnad og lisensiering
### Prismodell (Application Insights)
| Component | Pricing | Optimization Tips |
|-----------|---------|-------------------|
| **Data ingestion** | $2.76/GB (first 5GB free/month) | Use sampling (e.g., 50% for non-critical events) |
| **Data retention** | Free (90 days), $0.12/GB/month beyond | Archive to Azure Storage for long-term compliance |
| **Web tests** | $5.75 per test/month | Not required for Copilot extensions (use synthetic monitoring via Azure Functions) |
**Estimert cost for Copilot Studio agent (1000 users, 50k msgs/month):**
- Telemetry volume: ~10GB/month (hvis logging av messages enabled)
- Cost: $13.80/month (ingestion) + retention cost
- **Tip:** Disable "Log sensitive Activity properties" → reduser volume 30%
### Lisens-krav
| Feature | License Requirement |
|---------|---------------------|
| Copilot Studio analytics (built-in) | Power Virtual Agents license / Copilot Studio capacity |
| Application Insights integration | Azure subscription (free tier available) |
| Microsoft Sentinel (audit logs) | Microsoft 365 E5 OR Sentinel standalone |
| Power BI dashboards | Power BI Pro per user OR Premium capacity |
---
## For arkitekten (Cosmo)
### 5 spørsmål å stille kunden
1. **Scope:** Hvilke Copilot-extensions skal overvåkes? (Copilot Studio agents, M365 plugins, Power Platform connectors?)
2. **Compliance:** Er dette high-risk AI under AI Act? Trenger dere audit trail for Forvaltningsloven?
3. **Sensitive data:** Logger dere meldingstekst? Har dere DPIA for logging av personopplysninger?
4. **Alerting:** Hvem skal varsles ved error spikes, cost overruns eller security events?
5. **Retention:** Hvor lenge må telemetri oppbevares? (GDPR minimums vs. compliance-krav)
### Fallgruver
| Fallgruve | Impact | Hvordan unngå |
|-----------|--------|---------------|
| **Over-logging i test-fase** | Kostnadssprekk | Filter `designMode == "False"` i KQL |
| **Manglende sampling strategy** | Unødvendig detaljnivå → dyrt | 100% logging for errors, 10-50% for success events |
| **Ingen incident response plan** | Treg respons på security events | Set up Azure Monitor action groups (email, SMS, webhook til Teams/Slack) |
| **Siloed telemetry** | Kan ikke correlate plugin + LLM + backend | Bruk distributed tracing (W3C traceparent) |
### Anbefalinger per modenhetsnivå
#### Level 1: MVP (First Copilot Extension)
- Bruk Copilot Studio built-in analytics
- Enable Application Insights med basic logging (ikke sensitive properties)
- Set up 2-3 alerts (error rate >5%, response time >3s)
#### Level 2: Production Scale (5+ extensions)
- Centralized Application Insights workspace
- Custom telemetry events for business metrics (e.g., "LoanApprovalGranted")
- Pre-built dashboards (Copilot Studio Workbook + custom Azure Workbook)
#### Level 3: Enterprise/Compliance-heavy
- Microsoft Sentinel integration for audit logs
- Distributed tracing across all tiers (plugin → LLM → backend)
- Automated anomaly detection (Azure Monitor ML-based alerts)
- Quarterly compliance audit exports (GDPR, AI Act)
---
## Kilder og verifisering
**Verified (fra Microsoft Learn MCP):**
1. [Capture telemetry with Application Insights - Copilot Studio](https://learn.microsoft.com/en-us/microsoft-copilot-studio/advanced-bot-framework-composer-capture-telemetry) – Full guide til App Insights setup, KQL queries, custom dimensions
2. [Observability for pro-code generative AI solutions](https://learn.microsoft.com/en-us/microsoft-cloud/dev/copilot/isv/observability-for-ai) – ISV-guidance: lifecycle phases, metrics categories, evaluation techniques
3. [Monitor operations, compliance, and capacity - Copilot Studio](https://learn.microsoft.com/en-us/microsoft-copilot-studio/guidance/sec-gov-phase5) – Operational monitoring, Sentinel integration, compliance auditing
4. [Application Insights telemetry with Microsoft Copilot Studio (Dynamics 365)](https://learn.microsoft.com/en-us/dynamics365/guidance/resources/copilot-studio-appinsights) – Prerequisites, custom events, topic tracking
5. [Enable Application Insights support in Copilot Studio Kit](https://learn.microsoft.com/en-us/microsoft-copilot-studio/guidance/kit-enable-application-insights) – Azure AD app registration, API permissions for telemetry enrichment
**Baseline (modellkunnskap, verifisert mot docs):**
- GDPR Article 5 (data minimization), Article 6 (lawful basis), Article 35 (DPIA)
- AI Act (EU 2024/1689) Article 12 (logging for high-risk AI)
- Forvaltningsloven § 11 (journalføring av vedtak)
**Konfidensnivå per seksjon:**
- Kjernekomponenter: **Verified** (App Insights docs, Copilot Studio Workbook)
- Arkitekturmønstre: **Baseline** (patterns basert på Azure Well-Architected Framework + docs)
- Offentlig sektor: **Verified** (GDPR/AI Act legal text + Microsoft EU Data Boundary docs)
- Kostnad: **Verified** (Azure pricing calculator, Application Insights pricing page)