open/ms-ai-architect
1
0
Fork 0
Code Issues Pull requests Releases Activity

feat(ms-ai-architect): R7.3 bølge 4 (delvis) — payload-36/40/44/46 dømt+ingestet (80 claims, 16 flagg; ledger 129→133); 10 dommere døde på session-grensen [skip-docs]

Browse source
This commit is contained in:
Kjell Tore Guttormsen 2026-07-25 12:08:33 +02:00
commit e625645739
1 changed files with 237 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

238
scripts/kb-eval/data/judge-pass-manifest.json
View file

@ -4,7 +4,7 @@
"derived_from": "scripts/kb-update/data/full-pass-worklist.json (243 due, all never-verified)",
"cadence": "R7–R10 (5 økter × ~49, 8–10 samtidige)",
"batch": "R7.1",
"count": 129,
"count": 133,
"generated": "2026-07-18"
},
"files": [
@ -6097,6 +6097,65 @@
}
]
},
{
"file": "skills/ms-ai-engineering/references/rag-architecture/semantic-ranker-reranking.md",
"batch": "R7.3",
"judged_at": "2026-07-25",
"per_file_verdict": "flagged",
"claim_count": 20,
"verified": null,
"verified_by": null,
"flags": [
{
"id": "ms-ai-engineering/rag-architecture/semantic-ranker-reranking.md#14",
"judge_verdict": "not_grounded",
"rule": "",
"evidence_url": "https://learn.microsoft.com/en-us/azure/search/semantic-how-to-enable-disable",
"evidence_quote": "| Free (default) | Provides a monthly free request allowance. After the free allowance is consumed, semantic ranker requests return a billing error. | Available on all pricing tiers. | | Standard | Pay-as-you-go pricing after the monthly free allowance is consumed. | Requires the Basic tier or higher. |",
"reason": "Siden sier gratisplanen er tilgjengelig på alle pricing tiers og at standard-planen krever Basic tier or higher - det finnes ikke noe S1-krav, så tier-terskelen i påstanden er feil.",
"file": "skills/ms-ai-engineering/references/rag-architecture/semantic-ranker-reranking.md",
"line": 171,
"claim": "Bruk av semantic ranking i RAG-mønsteret krever S1-tier.",
"disposition": "outdated"
},
{
"id": "ms-ai-engineering/rag-architecture/semantic-ranker-reranking.md#15",
"judge_verdict": "not_grounded",
"rule": "R8",
"evidence_url": "https://learn.microsoft.com/en-us/azure/search/agentic-retrieval-overview",
"evidence_quote": "Some agentic retrieval features are generally available in the 2026-04-01 REST API via programmatic access.",
"reason": "Preview-statusen er superseded (deler av agentic retrieval er GA i 2026-04-01), og siden beskriver semantic ranker i pipelinen som L2 reranking - ingen L3-ranking finnes; begge lastbærende deler svikter.",
"file": "skills/ms-ai-engineering/references/rag-architecture/semantic-ranker-reranking.md",
"line": 184,
"claim": "Agentic Retrieval med LLM-assistert L3-ranking er en preview-funksjon (2025).",
"disposition": "outdated"
},
{
"id": "ms-ai-engineering/rag-architecture/semantic-ranker-reranking.md#17",
"judge_verdict": "not_grounded",
"rule": "R2",
"evidence_url": "https://learn.microsoft.com/en-us/azure/search/search-region-support",
"evidence_quote": "| Norway East | ✅ | ✅ | | ✅ | | |",
"reason": "I den kanoniske regionstabellen (kolonner: AI enrichment, Availability zones, Agentic retrieval, Confidential computing, Semantic ranker, Query rewrite) er Semantic ranker-kolonnen tom for Norway East, og Norway West står ikke i Europa-tabellen i det hele tatt - fraværet motbeviser regionpåstanden.",
"file": "skills/ms-ai-engineering/references/rag-architecture/semantic-ranker-reranking.md",
"line": 230,
"claim": "Semantic ranker er tilgjengelig i regionene Norway East og Norway West, og all prosessering skjer i valgt region uten at data sendes ut av regionen.",
"disposition": "outdated"
},
{
"id": "ms-ai-engineering/rag-architecture/semantic-ranker-reranking.md#18",
"judge_verdict": "source_silent",
"rule": "",
"evidence_url": "https://learn.microsoft.com/en-us/azure/search/semantic-how-to-enable-disable",
"evidence_quote": "",
"reason": "To-plan-strukturen (Free som default på alle tiers, Standard pay-as-you-go etter gratiskvoten) er bekreftet, men ingen learn.microsoft.com-side oppgir tallet 1000 requests per måned - Learn omtaler kun a monthly free request allowance og henviser til den JS-rendrede prissiden, så kvantumet kan verken bekreftes eller motbevises.",
"file": "skills/ms-ai-engineering/references/rag-architecture/semantic-ranker-reranking.md",
"line": 245,
"claim": "Prismodellen for semantic ranker har to planer: Gratis (1000 semantic ranker-requests per måned, tilgjengelig på alle tier inkludert Free) | Standard (pay-as-you-go per 1000 requests etter gratis kvote).",
"disposition": "unsourced"
}
]
},
{
"file": "skills/ms-ai-governance/references/monitoring-observability/alerting-strategies-escalation.md",
"batch": "R7.1",
@ -6132,6 +6191,183 @@
}
]
},
{
"file": "skills/ms-ai-governance/references/monitoring-observability/compliance-monitoring-ai-governance.md",
"batch": "R7.3",
"judged_at": "2026-07-25",
"per_file_verdict": "flagged",
"claim_count": 26,
"verified": null,
"verified_by": null,
"flags": [
{
"id": "ms-ai-governance/monitoring-observability/compliance-monitoring-ai-governance.md#1",
"judge_verdict": "not_grounded",
"rule": "R2",
"evidence_url": "https://learn.microsoft.com/en-us/purview/compliance-manager-assessments#assessments-for-ai-regulations",
"evidence_quote": "The AI regulations listed below align with compliance requirements such as monitoring AI interactions and preventing data loss in AI applications: - EU Artificial Intelligence Act - ISO/IEC 23894:2023 - ISO/IEC 42001:2023 - NIST AI Risk Management Framework (RMF) 1.0",
"reason": "Den kanoniske enumereringen av AI-maler lister ISO/IEC 23894:2023 og ISO/IEC 42001:2023 - ISO/IEC 23053:2022 er fraværende, så den påståtte malen finnes ikke (AI Act og NIST AI RMF stemmer).",
"file": "skills/ms-ai-governance/references/monitoring-observability/compliance-monitoring-ai-governance.md",
"line": 41,
"claim": "Microsoft Purview Compliance Manager har AI-spesifikke assessment-templates: AI Act | ISO/IEC 23053:2022 | NIST AI RMF.",
"disposition": "outdated"
},
{
"id": "ms-ai-governance/monitoring-observability/compliance-monitoring-ai-governance.md#3",
"judge_verdict": "not_grounded",
"rule": "R8",
"evidence_url": "https://learn.microsoft.com/en-us/purview/compliance-manager-assessments#assessments-for-ai-regulations",
"evidence_quote": "Compliance Manager provides four premium regulatory templates to help your organization assess, implement, and strengthen its compliance against AI regulations. ... - EU Artificial Intelligence Act - ISO/IEC 23894:2023 - ISO/IEC 42001:2023 - NIST AI Risk Management Framework (RMF) 1.0",
"reason": "360+-tallet er grunnet (over 360 regulatory templates), men den load-bearende AI-listen er feil: de AI-spesifikke malene er AI Act, ISO/IEC 23894:2023, ISO/IEC 42001:2023 og NIST AI RMF 1.0 - GDPR AI-tillegg finnes ikke, og CCPA/HIPAA står under Premium templates, ikke Premium AI templates.",
"file": "skills/ms-ai-governance/references/monitoring-observability/compliance-monitoring-ai-governance.md",
"line": 45,
"claim": "Compliance Manager har 360+ ferdigbygde regulatory templates, inkludert AI-spesifikke: EU AI Act | GDPR AI-tillegg | CCPA | HIPAA.",
"disposition": "outdated"
},
{
"id": "ms-ai-governance/monitoring-observability/compliance-monitoring-ai-governance.md#6",
"judge_verdict": "not_grounded",
"rule": "R2",
"evidence_url": "https://learn.microsoft.com/en-us/security/security-for-ai/security-dashboard-for-ai",
"evidence_quote": "Identity and access risk | Data security risk | Cloud security risk | Misconfigurations and attack paths | Agents with sensitive interactions",
"reason": "Dashbordets faktiske sider er Overview, AI inventory og AI risk (med risikokategoriene over) - Threat Detection, Access Control og Compliance Status står ikke som seksjoner noe sted på den kanoniske siden, og inventar-siden heter AI inventory, ikke AI Agent Inventory.",
"file": "skills/ms-ai-governance/references/monitoring-observability/compliance-monitoring-ai-governance.md",
"line": 81,
"claim": "Security Dashboard for AI har seksjonene AI Agent Inventory | Threat Detection | Data Security | Access Control | Compliance Status.",
"disposition": "outdated"
},
{
"id": "ms-ai-governance/monitoring-observability/compliance-monitoring-ai-governance.md#24",
"judge_verdict": "source_silent",
"rule": "R8",
"evidence_url": "https://learn.microsoft.com/en-us/azure/foundry/how-to/create-projects",
"evidence_quote": "",
"reason": "Første del er grunnet (Select a Location or use the default. The location is the region where the project resources are hosted), men ingen offisiell Learn-side sier at prosjektet ikke kan flyttes til en annen region senere - create-projects dokumenterer ingen flytte-operasjon, og eneste treff på påstanden er et community-svar i Microsoft Q&A, ikke dokumentasjon.",
"file": "skills/ms-ai-governance/references/monitoring-observability/compliance-monitoring-ai-governance.md",
"line": 355,
"claim": "I Microsoft Foundry velges region ved opprettelse av prosjektet, og prosjektet kan ikke flyttes til en annen region senere.",
"disposition": "unsourced"
},
{
"id": "ms-ai-governance/monitoring-observability/compliance-monitoring-ai-governance.md#25",
"judge_verdict": "not_grounded",
"rule": "R8",
"evidence_url": "https://learn.microsoft.com/en-us/privacy/eudb/eu-data-boundary-transfers-for-optional-capabilities",
"evidence_quote": "When the customer selects any \"Global\" deployment type (for example, Global Standard), prompts and completions sent to and output by that deployment may be processed, for inferencing or fine-tuning, in any Azure AI Foundry Models region globally, including outside the EU for an Azure AI Foundry Models resource created in a region within the EU.",
"reason": "Kilden motsier garantien direkte: med Global-deploymenttyper kan prompts og completions prosesseres utenfor EU, og EU Data Boundary-oversikten sier selv at forpliktelsen er subject to limited circumstances where Customer Data ... will continue to be transferred outside the EU Data Boundary.",
"file": "skills/ms-ai-governance/references/monitoring-observability/compliance-monitoring-ai-governance.md",
"line": 356,
"claim": "Azure OpenAI med EU Data Boundary garanterer at prompts og responses ikke forlater EU.",
"disposition": "outdated"
},
{
"id": "ms-ai-governance/monitoring-observability/compliance-monitoring-ai-governance.md#26",
"judge_verdict": "not_grounded",
"rule": "R8",
"evidence_url": "https://learn.microsoft.com/en-us/purview/compliance-manager-regulations",
"evidence_quote": "The Microsoft Data Protection Baseline regulatory template is available for organizations at all subscription levels to use. The regulations designated as premium require purchase of a license to use them.",
"reason": "Lisensmodellen er en annen enn claimet: premium-maler (inkludert Premium AI templates) krever kjøpt lisens uavhengig av E5, ikke at E5 gir 360+ maler; i tillegg er ISO/IEC 23053 ikke blant AI-malene (kanonisk liste: ISO/IEC 23894:2023 og ISO/IEC 42001:2023).",
"file": "skills/ms-ai-governance/references/monitoring-observability/compliance-monitoring-ai-governance.md",
"line": 388,
"claim": "Compliance Manager-lisensiering: Microsoft 365 E3 gir kun basic assessments (Microsoft baseline) uten AI-spesifikke templates | Microsoft 365 E5 gir 360+ regulatory templates, custom templates og automatiserte assessments inkl. AI Act, ISO/IEC 23053 og NIST AI RMF | Purview Compliance standalone gir full Compliance Manager + DLP + eDiscovery med DSPM for AI, AI audit logs og retention for AI-apper.",
"disposition": "outdated"
}
]
},
{
"file": "skills/ms-ai-governance/references/monitoring-observability/distributed-tracing-ai-pipelines.md",
"batch": "R7.3",
"judged_at": "2026-07-25",
"per_file_verdict": "flagged",
"claim_count": 18,
"verified": null,
"verified_by": null,
"flags": [
{
"id": "ms-ai-governance/monitoring-observability/distributed-tracing-ai-pipelines.md#2",
"judge_verdict": "not_grounded",
"rule": "R7",
"evidence_url": "https://learn.microsoft.com/azure/foundry/observability/concepts/trace-agent-concept",
"evidence_quote": "Child Span | invoke_agent | agent_to_agent_interaction | Traces communication between agents.",
"reason": "De kanoniske span-navnene på Learn er execute_task, invoke_agent, execute_tool, create_agent og agent_to_agent_interaction — uten gen_ai.-prefiks; ingen Learn-side navngir gen_ai.model.completion, gen_ai.tool.execution eller gen_ai.agent.invoke, så de påståtte strengene (som ER selve assertionen) er feil.",
"file": "skills/ms-ai-governance/references/monitoring-observability/distributed-tracing-ai-pipelines.md",
"line": 55,
"claim": "OpenTelemetrys semantiske konvensjoner for generativ AI definerer standard AI-span: gen_ai.model.completion | gen_ai.tool.execution | gen_ai.agent.invoke | gen_ai.agent_planning | gen_ai.agent_to_agent_interaction.",
"disposition": "outdated"
},
{
"id": "ms-ai-governance/monitoring-observability/distributed-tracing-ai-pipelines.md#3",
"judge_verdict": "not_grounded",
"rule": "R7",
"evidence_url": "https://learn.microsoft.com/microsoft-agent-365/developer/observability-attribute-reference",
"evidence_quote": "gen_ai.usage.input_tokens | CH | O | -- | Input token count, string-encoded.",
"reason": "Learns kanoniske gen_ai-attributtabell oppgir gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.response.finish_reasons (flertall) og gen_ai.provider.name — påstandens gen_ai.usage.prompt_tokens, gen_ai.usage.completion_tokens, gen_ai.response.finish_reason og gen_ai.system er superserte navn; kun gen_ai.request.model stemmer.",
"file": "skills/ms-ai-governance/references/monitoring-observability/distributed-tracing-ai-pipelines.md",
"line": 62,
"claim": "Standard gen_ai-attributter er: gen_ai.system | gen_ai.request.model | gen_ai.usage.prompt_tokens | gen_ai.usage.completion_tokens | gen_ai.response.finish_reason.",
"disposition": "outdated"
},
{
"id": "ms-ai-governance/monitoring-observability/distributed-tracing-ai-pipelines.md#11",
"judge_verdict": "source_silent",
"rule": "",
"evidence_url": "https://learn.microsoft.com/azure/foundry/how-to/develop/langchain-models",
"evidence_quote": "",
"reason": "Den siterte siden omtaler ikke LangChain-konfigurasjon i det hele tatt, og ingen Learn-side oppgir api_version 2024-08-01-preview for AzureChatOpenAI; Learns kanoniske LangChain-side dokumenterer AZURE_OPENAI_API_VERSION med eksempelverdiene v1 eller preview, men enumererer ikke AzureChatOpenAIs api_version, så verdien kan verken bekreftes eller motbevises.",
"file": "skills/ms-ai-governance/references/monitoring-observability/distributed-tracing-ai-pipelines.md",
"line": 198,
"claim": "AzureChatOpenAI konfigureres med api_version 2024-08-01-preview i LangChain-integrasjonen mot Azure OpenAI.",
"disposition": "unsourced"
},
{
"id": "ms-ai-governance/monitoring-observability/distributed-tracing-ai-pipelines.md#17",
"judge_verdict": "not_grounded",
"rule": "R8",
"evidence_url": "https://learn.microsoft.com/azure/azure-monitor/app/opentelemetry-sampling",
"evidence_quote": "The Application Insights OpenTelemetry distros include a default sampler. The specific sampler and its rate depend on the language and distro version.",
"reason": "Den bærende delpåstanden om at distroen ikke sampler som standard og at samplerne må konfigureres eksplisitt er motsagt: Learn oppgir rate-limited sampling som default for Python (fra 1.8.6), Node.js (fra 1.16.0) og Java (fra 3.4.0), og ApplicationInsightsSampler som default for ASP.NET Core/.NET.",
"file": "skills/ms-ai-governance/references/monitoring-observability/distributed-tracing-ai-pipelines.md",
"line": 478,
"claim": "Azure Monitor OpenTelemetry-distroen sampler ikke som standard; den støtter fixed-rate og rate-limited samplere som må konfigureres eksplisitt, og trace-basert sampling for logger er default-på først når sampling er aktivert — adaptive sampling gjelder kun den klassiske Application Insights SDK-en.",
"disposition": "outdated"
}
]
},
{
"file": "skills/ms-ai-governance/references/monitoring-observability/log-analytics-kql-ai-queries.md",
"batch": "R7.3",
"judged_at": "2026-07-25",
"per_file_verdict": "flagged",
"claim_count": 16,
"verified": null,
"verified_by": null,
"flags": [
{
"id": "ms-ai-governance/monitoring-observability/log-analytics-kql-ai-queries.md#9",
"judge_verdict": "not_grounded",
"rule": "R7",
"evidence_url": "https://learn.microsoft.com/en-us/azure/foundry-classic/openai/monitor-openai-reference",
"evidence_quote": "| Input token volume per request | Processed Prompt Tokens | `ProcessedPromptTokens` |",
"reason": "TokenTransaction og TotalTokens finnes i metrikklisten, men PromptTokens og CompletionTokens gjør det ikke - de heter ProcessedPromptTokens og GeneratedTokens (alternativt InputTokens/OutputTokens under Models - Usage); metrikknavnene ER påstanden, så R7s load-bearing carve-out gjelder og to av fire navn er feil.",
"file": "skills/ms-ai-governance/references/monitoring-observability/log-analytics-kql-ai-queries.md",
"line": 294,
"claim": "Azure OpenAI-metrikker i AzureMetrics omfatter TokenTransaction | TotalTokens | PromptTokens | CompletionTokens.",
"disposition": "outdated"
},
{
"id": "ms-ai-governance/monitoring-observability/log-analytics-kql-ai-queries.md#15",
"judge_verdict": "not_grounded",
"rule": "R4",
"evidence_url": "https://learn.microsoft.com/en-us/azure/search/search-monitor-queries",
"evidence_quote": "| where Query_s != \"?api-version=2026-04-01&search=*\"",
"reason": "Den kanoniske siden bruker ?api-version=2026-04-01&search=* i nøyaktig samme query-streng-eksempel (og datareferansen viser api-version=2026-04-01), mens REST-versjonslisten oppgir 2026-04-01 som latest stable og 2025-09-01 som en eldre stabil versjon - påstanden peker på en superseded rad.",
"file": "skills/ms-ai-governance/references/monitoring-observability/log-analytics-kql-ai-queries.md",
"line": 617,
"claim": "Azure AI Search bruker api-version 2025-09-01 i query-strenger (?api-version=2025-09-01&search=*).",
"disposition": "outdated"
}
]
},
{
"file": "skills/ms-ai-governance/references/monitoring-observability/security-and-audit-logging-ai.md",
"batch": "R7.1",