feat(ms-ai-architect): Spor 2b gull-rekonsiliering — 12 uenigheter løst, fasit herdet (P 84→92%, R 84→88%) [skip-docs]

4 gull-feil rettet (FP1/FP2/FP6/FN1) + 1 note-fiks (FP4); 8 judge-feil dokumentert som kalibreringsmål for Spor 2a/judge-prompt-v3. Hver endring live-belagt mot MS Learn. Nedre-grense-policy vedtatt. 0 uløste uenigheter. Suite 637/637. Logg: docs/ref-kb-gold-reconciliation-2026-06.md; herdet rapport: judge-bakeoff-report-v2-reconciled (frozen v2 beholdt).
2026-06-30 07:20:21 +02:00 · 2026-06-30 07:20:21 +02:00 · 4ab1138760
commit 4ab1138760
parent f508380131
5 changed files with 404 additions and 12 deletions
--- a/scripts/kb-eval/data/gold-correctness-set.json
+++ b/scripts/kb-eval/data/gold-correctness-set.json
@ -405,11 +405,11 @@
      "stratum": "volatile",
      "claim": "1900+ Foundry Models sold by Azure; 11000+ totalt; 40+ regioner",
      "claim_type": "taxonomy",
-      "verdict": "correct",
+      "verdict": "wrong",
      "evidence_url": "https://learn.microsoft.com/azure/foundry/what-is-foundry",
      "lastmod_changed": false,
      "file_last_updated": "2026-06-24",
-      "notes": "1900+ bekreftet, men kilden rammer det som total fler-leverandør-tilgang, ikke smalt 'sold by Azure'. 11000+/40+ uverifisert."
+      "notes": "RECONCILED 2026-06-30 (Spor 2b): correct->wrong. Live (what-is-foundry): \"1,900+\" rammer HELE katalogen (Microsoft+OpenAI+partnere+community), ikke \"sold by Azure\" (en delmengde). \"11000+ totalt\" og \"40+ regioner\" finnes ikke pa kilden (kilden sier kun \"most regions where Foundry Tools are available\"). Fil-fiks (Spor 0/1): omramme til \"1900+ modeller totalt i katalogen\", fjern 11000+/40+. Judge not_grounded var korrekt."
    },
    {
      "id": "ms-ai-advisor/platforms/azure-ai-foundry.md#3",
@ -730,11 +730,11 @@
      "stratum": "volatile",
      "claim": "Image-to-vector queries kun med Azure Vision multimodal embeddings vectorizer",
      "claim_type": "taxonomy",
-      "verdict": "correct",
+      "verdict": "wrong",
      "evidence_url": "https://learn.microsoft.com/azure/search/multimodal-search-overview",
      "lastmod_changed": false,
      "file_last_updated": "2026-06-19",
-      "notes": "Bekreftet."
+      "notes": "RECONCILED 2026-06-30 (Spor 2b): correct->wrong. Live (multimodal-search-overview): image-to-vector ved query krever en multimodal-embedding-vectorizer, men det finnes TO veier - AML-skill ELLER Azure Vision multimodal embeddings skill. \"kun Azure Vision\" er for restriktivt (utelukker AML-veien). Fil-fiks: myk opp \"kun\" -> \"f.eks./primaert\". Judge not_grounded var korrekt pa eksklusiviteten."
    },
    {
      "id": "ms-ai-advisor/prompt-engineering/multimodal-prompt-design.md#8",
@ -912,11 +912,11 @@
      "stratum": "volatile",
      "claim": "Realtime API regions: East US 2, Sweden Central (global)",
      "claim_type": "region",
-      "verdict": "outdated",
+      "verdict": "correct",
      "evidence_url": "https://learn.microsoft.com/azure/foundry/openai/how-to/realtime-audio",
      "lastmod_changed": false,
      "file_last_updated": "2026-06-24",
-      "notes": "Kilde: tilgjengelig for global deployments, ikke begrenset til de to. To-region-begrensning avløst."
+      "notes": "RECONCILED 2026-06-30 (Spor 2b): outdated->correct. Live (realtime-audio + WebRTC/SIP/WebSockets-soesken): Realtime-modellene er global deployment-type tilgjengelig i NOEYAKTIG East US 2 + Sweden Central. To-region-bildet er IKKE avloest - forrige gull-note misleste \"global deployments\" som \"tilgjengelig overalt\". Judge grounded var korrekt."
    },
    {
      "id": "ms-ai-advisor/prompt-engineering/real-time-reasoning-performance.md#6",
@ -2060,7 +2060,7 @@
      "evidence_url": "https://learn.microsoft.com/azure/foundry/foundry-models/concepts/deployment-types",
      "lastmod_changed": false,
      "file_last_updated": "2026-06-24",
-      "notes": "Alle fire data-residency-oppførsler bekreftet ordrett."
+      "notes": "RECONCILED 2026-06-30 (Spor 2b): verdikt correct beholdt; note korrigert. Live (deployment-types): de tre kjerne-residency-atferdene bekreftet (Standard=deployment-region, Data Zone=innen US/EU-sone, Global Standard=any region). MEN \"alle fire ordrett\" overdrev - kilden grupperer residency etter Global/DataZone/Regional, og Global Provisioned = any region (ikke single-region). \"Standard/Provisioned=region\" er rimelig kortform for kjernemodellen men utelater Global/DataZone Provisioned. Judge not_grounded var over-streng (kjernen ER grunnet)."
    },
    {
      "id": "ms-ai-engineering/api-management/multi-region-ai-gateway-design.md#3",
@ -3603,11 +3603,11 @@
      "stratum": "volatile",
      "claim": "Global training (Public Preview) rimeligere, ingen datasuverenitet; bruk regional i Norway East/Sweden Central",
      "claim_type": "status",
-      "verdict": "correct",
+      "verdict": "outdated",
      "evidence_url": "https://learn.microsoft.com/azure/ai-foundry/openai/concepts/models",
      "lastmod_changed": false,
      "file_last_updated": "2026-06-24",
-      "notes": "Global training public preview, billigere, ingen residency; regional gir residency. Bekreftet."
+      "notes": "RECONCILED 2026-06-30 (Spor 2b): correct->outdated. Live (openai/concepts/models): Global training har INGEN egen \"Public Preview\"-status - tilbys for GA-modeller (gpt-4o/gpt-4.1 m.fl. = GA). \"billigere\" + \"ingen residency\" stemmer. MEN Norway East er listet som en GLOBAL trenings-region (uten residency), ikke en regional/residency-region - sa \"bruk regional i Norway East for residency\" er feil (Standard fine-tuning-regioner: North Central US/Sweden Central/East US 2). Fil-fiks: fjern \"(Public Preview)\", rett Norway-East-anbefalingen. Judge not_grounded var defensibel."
    },
    {
      "id": "ms-ai-security/cost-optimization/gpt5-gpt41-pricing-models.md#1",
--- a/scripts/kb-eval/data/judge-bakeoff-report-v2-reconciled.json
+++ b/scripts/kb-eval/data/judge-bakeoff-report-v2-reconciled.json
@ -0,0 +1,234 @@
+{
+  "_meta": {
+    "source": "gold-correctness-set.json + judge-bakeoff-results.json",
+    "thresholds": {
+      "minRecall": 0.8,
+      "minPrecision": 0.7
+    },
+    "judged": 255
+  },
+  "population": {
+    "total": 255,
+    "verifiable": 240,
+    "positives": 40,
+    "negatives": 200,
+    "unsourcedInP": 15
+  },
+  "arms": {
+    "staleness": {
+      "tp": 0,
+      "fp": 0,
+      "fn": 40,
+      "tn": 200,
+      "positives": 40,
+      "negatives": 200,
+      "flagged": 0,
+      "precision": null,
+      "recall": 0,
+      "f1": null,
+      "recallWilson": {
+        "p": 0,
+        "low": 0,
+        "high": 0.08762453925039232
+      },
+      "precisionWilson": null
+    },
+    "judge": {
+      "tp": 35,
+      "fp": 3,
+      "fn": 5,
+      "tn": 197,
+      "positives": 40,
+      "negatives": 200,
+      "flagged": 38,
+      "precision": 0.9210526315789473,
+      "recall": 0.875,
+      "f1": 0.8974358974358975,
+      "recallWilson": {
+        "p": 0.875,
+        "low": 0.7388757932976187,
+        "high": 0.9454058022645873
+      },
+      "precisionWilson": {
+        "p": 0.9210526315789473,
+        "low": 0.792003210797347,
+        "high": 0.972785898605735
+      }
+    },
+    "hybrid": {
+      "tp": 35,
+      "fp": 3,
+      "fn": 5,
+      "tn": 197,
+      "positives": 40,
+      "negatives": 200,
+      "flagged": 38,
+      "precision": 0.9210526315789473,
+      "recall": 0.875,
+      "f1": 0.8974358974358975,
+      "recallWilson": {
+        "p": 0.875,
+        "low": 0.7388757932976187,
+        "high": 0.9454058022645873
+      },
+      "precisionWilson": {
+        "p": 0.9210526315789473,
+        "low": 0.792003210797347,
+        "high": 0.972785898605735
+      }
+    }
+  },
+  "sourceSilent": {
+    "onVerifiableNegative": 3,
+    "onVerifiableError": 2,
+    "agreesWithUnsourced": 5,
+    "disagreesWithUnsourced": 10
+  },
+  "byClaimType": {
+    "version": {
+      "tp": 6,
+      "fp": 0,
+      "fn": 1,
+      "tn": 21,
+      "positives": 7,
+      "negatives": 21,
+      "flagged": 6,
+      "precision": 1,
+      "recall": 0.8571428571428571,
+      "f1": 0.923076923076923,
+      "recallWilson": {
+        "p": 0.8571428571428571,
+        "low": 0.4868654966809701,
+        "high": 0.9743210440510252
+      },
+      "precisionWilson": {
+        "p": 1,
+        "low": 0.6096569663469354,
+        "high": 0.9999999999999999
+      }
+    },
+    "tpm": {
+      "tp": 4,
+      "fp": 0,
+      "fn": 1,
+      "tn": 20,
+      "positives": 5,
+      "negatives": 20,
+      "flagged": 4,
+      "precision": 1,
+      "recall": 0.8,
+      "f1": 0.888888888888889,
+      "recallWilson": {
+        "p": 0.8,
+        "low": 0.3755282641185388,
+        "high": 0.9637768390302125
+      },
+      "precisionWilson": {
+        "p": 1,
+        "low": 0.5100999795960008,
+        "high": 1
+      }
+    },
+    "region": {
+      "tp": 1,
+      "fp": 0,
+      "fn": 0,
+      "tn": 14,
+      "positives": 1,
+      "negatives": 14,
+      "flagged": 1,
+      "precision": 1,
+      "recall": 1,
+      "f1": 1,
+      "recallWilson": {
+        "p": 1,
+        "low": 0.2065432914738929,
+        "high": 1
+      },
+      "precisionWilson": {
+        "p": 1,
+        "low": 0.2065432914738929,
+        "high": 1
+      }
+    },
+    "status": {
+      "tp": 7,
+      "fp": 0,
+      "fn": 1,
+      "tn": 45,
+      "positives": 8,
+      "negatives": 45,
+      "flagged": 7,
+      "precision": 1,
+      "recall": 0.875,
+      "f1": 0.9333333333333333,
+      "recallWilson": {
+        "p": 0.875,
+        "low": 0.5291051942301386,
+        "high": 0.9775830911367038
+      },
+      "precisionWilson": {
+        "p": 1,
+        "low": 0.6456611570247934,
+        "high": 1
+      }
+    },
+    "taxonomy": {
+      "tp": 11,
+      "fp": 3,
+      "fn": 0,
+      "tn": 83,
+      "positives": 11,
+      "negatives": 86,
+      "flagged": 14,
+      "precision": 0.7857142857142857,
+      "recall": 1,
+      "f1": 0.88,
+      "recallWilson": {
+        "p": 1,
+        "low": 0.7411599827511859,
+        "high": 1
+      },
+      "precisionWilson": {
+        "p": 0.7857142857142857,
+        "low": 0.5241027622679172,
+        "high": 0.9242875166308363
+      }
+    },
+    "sku": {
+      "tp": 6,
+      "fp": 0,
+      "fn": 2,
+      "tn": 14,
+      "positives": 8,
+      "negatives": 14,
+      "flagged": 6,
+      "precision": 1,
+      "recall": 0.75,
+      "f1": 0.8571428571428571,
+      "recallWilson": {
+        "p": 0.75,
+        "low": 0.40926987910258916,
+        "high": 0.9285223111419724
+      },
+      "precisionWilson": {
+        "p": 1,
+        "low": 0.6096569663469354,
+        "high": 0.9999999999999999
+      }
+    }
+  },
+  "gate": {
+    "pass": true,
+    "recallOk": true,
+    "precisionOk": true,
+    "beatsStaleness": true,
+    "thresholds": {
+      "minRecall": 0.8,
+      "minPrecision": 0.7
+    },
+    "reasons": [
+      "all criteria met"
+    ]
+  }
+}
--- a/scripts/kb-eval/data/judge-bakeoff-report-v2-reconciled.md
+++ b/scripts/kb-eval/data/judge-bakeoff-report-v2-reconciled.md
@ -0,0 +1,54 @@
+# Judge bake-off-rapport — S1 (Fase 3 de-risk)
+
+_Generert deterministisk av `run-judge-bakeoff.mjs` over `gold-correctness-set.json` + `judge-bakeoff-results.json`. Tall fra testet `lib/judge-bakeoff.mjs`. Ikke rediger for hånd — regenerer._
+
+**Forhåndsregistrert gate (låst FØR fan-out):** recall ≥ 0.8, presisjon ≥ 0.7, OG judge-recall > staleness-recall.
+
+## Evaluerings-populasjon (P)
+
+Volatil stratum + fetchbare claim_types (price ekskludert) — der feilene bor; unngår «invertert leverage».
+
+| metrikk | verdi |
+|---|---|
+| P totalt | 255 |
+| Verifiserbare (correct/outdated/wrong) | 240 |
+| Positive (reelle feil å fange) | 40 |
+| Negative (correct) | 200 |
+| Unsourced i P (kjørt, men utenfor P/R) | 15 |
+
+## Arm-sammenligning (detektering over de 240 verifiserbare)
+
+| arm | TP | FP | FN | TN | presisjon | recall | recall Wilson 95% | F1 |
+|---|---|---|---|---|---|---|---|---|
+| staleness (billig baseline) | 0 | 0 | 40 | 200 | n/a | 0.0% | [0.0%, 8.8%] | n/a |
+| judge (per-påstand groundedness) | 35 | 3 | 5 | 197 | 92.1% | 87.5% | [73.9%, 94.5%] | 0.897 |
+| hybrid (union) | 35 | 3 | 5 | 197 | 92.1% | 87.5% | [73.9%, 94.5%] | 0.897 |
+
+## Judge per claim_type (verifiserbar delmengde)
+
+| claim_type | positive | TP | FP | FN | presisjon | recall |
+|---|---|---|---|---|---|---|
+| taxonomy | 11 | 11 | 3 | 0 | 78.6% | 100.0% |
+| status | 8 | 7 | 0 | 1 | 100.0% | 87.5% |
+| sku | 8 | 6 | 0 | 2 | 100.0% | 75.0% |
+| version | 7 | 6 | 0 | 1 | 100.0% | 85.7% |
+| tpm | 5 | 4 | 0 | 1 | 100.0% | 80.0% |
+| region | 1 | 1 | 0 | 0 | 100.0% | 100.0% |
+
+## source_silent-diagnostikk
+
+Judgen hentet siden men fant ikke verdien. Diagnostisk, ikke et flagg.
+
+| signal | antall | tolkning |
+|---|---|---|
+| På verifiserbar feil | 2 | judge-bom: reell feil oversett via «kan ikke verifisere» |
+| På verifiserbar correct | 3 | judge reproduserte ikke et korrekt faktum mennesket fant |
+| Enig med unsourced | 5 | judge reproduserer den uverifiserbare grensen (godt) |
+| Uenig med unsourced | 10 | judge hevdet grunnet/ugrunnet der mennesket ikke fant kilde |
+
+## GATE: ✅ PASS — bygg S3
+
+- recall 0.875 ≥ 0.8? **ja**
+- presisjon 0.921 ≥ 0.7? **ja**
+- slår staleness (recall 0.000)? **ja**
+- begrunnelse: all criteria met