open/ms-ai-architect
1
0
Fork 0
Code Issues Releases Activity

feat(ms-ai-architect): G5b gull-friskhets-spot-sjekk LUKKET — 4 v3-FP re-adjudert mot live, ALLE stale gull (v3 flagget korrekt), baseline løftet v3 P89.7/R92.1 → P100/R92.9/0FP; v3.1 forfattet (ren recall-hardning, FP-vakt droppet) [skip-docs]

Browse source
This commit is contained in:
Kjell Tore Guttormsen 2026-06-30 13:43:52 +02:00
commit 045db566ba
8 changed files with 842 additions and 12 deletions
Download patch file Download diff file Expand all files Collapse all files

19
scripts/kb-eval/data/gold-correctness-set.json
View file

@ -8,7 +8,8 @@
"claim_count": 373,
"reconciliation_log": [
"2026-06-30 Spor 2b: 12 judge-vs-gold-uenigheter adjudert mot live; 4 gull-feil rettet + 1 note-fiks. Logg: docs/ref-kb-gold-reconciliation-2026-06.md.",
"2026-06-30 G5 friskhets-mikropass: 5 omstridte claims (gold=outdated, v3=grounded) re-adjudert mot live MS Learn. 2 stale gull rettet outdated->correct (genaiops-llm-specific-practices.md#2 '1600+' live 1900 tett nedre grense; model-selection-price-performance.md#8 Model Router GA nov 2025). 3 opprettholdt som outdated (judge-feil bekreftet: multi-model-strategy-costs.md#2, token-usage-tracking-attribution.md#3, ai-foundry-disaster-recovery-planning.md#9). Logg: docs/ref-kb-correctness-program-2026-06.md §8 G5."
"2026-06-30 G5 friskhets-mikropass: 5 omstridte claims (gold=outdated, v3=grounded) re-adjudert mot live MS Learn. 2 stale gull rettet outdated->correct (genaiops-llm-specific-practices.md#2 '1600+' live 1900 tett nedre grense; model-selection-price-performance.md#8 Model Router GA nov 2025). 3 opprettholdt som outdated (judge-feil bekreftet: multi-model-strategy-costs.md#2, token-usage-tracking-attribution.md#3, ai-foundry-disaster-recovery-planning.md#9). Logg: docs/ref-kb-correctness-program-2026-06.md §8 G5.",
"2026-06-30 G5b friskhets-spot-sjekk: de 4 v3-FP-claims (gold=correct, v3=not_grounded) re-adjudert mot live MS Learn (4 Opus-subagenter, blinde). ALLE 4 var stale gull (v3 flagget korrekt): adr-template.md#1 correct->wrong (zero permission management motsies, Graph connectors krever ACL); multi-region-azure-openai-deployment.md#2 correct->outdated (gpt-35-turbo retired); network-resilience-patterns-ai.md#4 correct->wrong (obligatorisk vs anbefalt); vector-storage-cost-optimization.md#7 correct->wrong (GA-dato 2024-07-01, ikke 2024-11-01-preview). v3 hadde 0 ekte FP. Logg: docs/ref-kb-correctness-program-2026-06.md §8 G5b."
]
},
"claims": [
@ -344,11 +345,11 @@
"stratum": "volatile",
"claim": "SharePoint Embedded/Graph Connectors: zero permission management, permissions respekteres automatisk",
"claim_type": "status",
"verdict": "correct",
"verdict": "wrong",
"evidence_url": "https://learn.microsoft.com/microsoft-365/copilot/extensibility/data-privacy-security",
"lastmod_changed": false,
"file_last_updated": "2026-06-24",
"notes": "Permission inheritance bekreftet."
"notes": "RECONCILED 2026-06-30 (G5b friskhets-spot-sjekk): correct->wrong. Live (m365/copilot/extensibility/data-privacy-security + graph/connecting-external-content-manage-items): \"You can manage permissions to view external items by associating an access control list (ACL)\"; hver externalItem MA ha ACL (ikke-Entra-brukere ma mappes til Entra). Del B (permissions respekteres automatisk ved grounding) stemmer, men del A \"zero permission management\" motsies - Graph connectors krever ACL-forfatting. Fil-fiks (Spor 0/1): fjern \"zero permission management\", behold permission-honoring. Judge not_grounded var korrekt. Confidence: medium (innsats-overdrivelse pa last-baerende del A)."
},
{
"id": "ms-ai-advisor/architecture/adr-template.md#2",
@ -3165,11 +3166,11 @@
"stratum": "volatile",
"claim": "Sweden Central sekundær, bred (gpt-4o, o1, gpt-35-turbo)",
"claim_type": "region",
"verdict": "correct",
"verdict": "outdated",
"evidence_url": "https://learn.microsoft.com/azure/foundry/foundry-models/concepts/models-sold-directly-by-azure-region-availability",
"lastmod_changed": false,
"file_last_updated": "2026-06-24",
"notes": "Sweden Central bred støtte bekreftet."
"notes": "RECONCILED 2026-06-30 (G5b friskhets-spot-sjekk): correct->outdated. Live (models-sold-directly-by-azure-region-availability + retirements): gpt-4o og o1 tilgjengelig i Sweden Central, men gpt-35-turbo er RETIRED (0301/0613 feb 2025; 0125/1106 fra sep 2025) og finnes ikke lenger i katalogen. Claim listet gpt-35-turbo som tilgjengelig - var sant, na utdatert. Fil-fiks (Spor 0/1): bytt gpt-35-turbo med gjeldende modell (gpt-4.1-mini/gpt-4o-mini). Judge not_grounded (R2 entitet-fravaer) var korrekt. Confidence: high."
},
{
"id": "ms-ai-infrastructure/bcdr/multi-region-azure-openai-deployment.md#3",
@ -3399,11 +3400,11 @@
"stratum": "volatile",
"claim": "Circuit Breaker + Retry exponential backoff obligatorisk for alle Azure AI API-kall",
"claim_type": "status",
"verdict": "correct",
"verdict": "wrong",
"evidence_url": "https://learn.microsoft.com/azure/foundry-classic/openai/how-to/quota",
"lastmod_changed": false,
"file_last_updated": "2026-06-24",
"notes": "Retry m/backoff + circuit breaker offisielt anbefalt; 'obligatorisk' sterk formulering."
"notes": "RECONCILED 2026-06-30 (G5b friskhets-spot-sjekk): correct->wrong. Live (foundry-classic/openai/how-to/quota): MS rammer dette som \"Rate limit best practices\" / \"recommended\", ikke obligatorisk; circuit breaker nevnes kun som valgfritt avansert Polly-monster. Claim \"obligatorisk for alle Azure AI API-kall\" overdriver modaliteten (anbefalt -> palagt) og omfanget (alle kall). Monstrene er reelle MS-anbefalinger, men \"obligatorisk\" er ikke grunnet. Fil-fiks (Spor 0/1): bytt \"obligatorisk\" med \"anbefalt\". Judge not_grounded (R6) var korrekt. Confidence: medium (modalitets-overdrivelse)."
},
{
"id": "ms-ai-infrastructure/bcdr/service-level-documentation-dr.md#1",
@ -4533,11 +4534,11 @@
"stratum": "volatile",
"claim": "Vector quantization GA siden 2024-11-01",
"claim_type": "status",
"verdict": "correct",
"verdict": "wrong",
"evidence_url": "https://learn.microsoft.com/azure/search/vector-search-index-size",
"lastmod_changed": false,
"file_last_updated": "2026-06-19",
"notes": "GA-status bekreftet."
"notes": "RECONCILED 2026-06-30 (G5b friskhets-spot-sjekk): correct->wrong. Live (search-api-migration + vector-search-index-size): vector quantization ER GA, MEN GA-dato var 2024-07-01 (stable release), ikke 2024-11-01. \"2024-11-01\" finnes kun som preview-API-versjon (2024-11-01-preview). Last-baerende dato er feil. Fil-fiks (Spor 0/1): rett GA-dato til 2024-07-01. Judge not_grounded var korrekt. Confidence: high."
},
{
"id": "ms-ai-security/cost-optimization/vector-storage-cost-optimization.md#8",