diff --git a/docs/ref-kb-correctness-program-2026-06.md b/docs/ref-kb-correctness-program-2026-06.md index 974e6ec..b382f32 100644 --- a/docs/ref-kb-correctness-program-2026-06.md +++ b/docs/ref-kb-correctness-program-2026-06.md @@ -96,13 +96,13 @@ Status-nøkkel: 🔴 ikke startet · 🟡 pågår · 🟢 lukket. | # | Gap (mekanismen mangler) | Forhindrer feilklasse | Lukke-fase | Status | MÅ lukkes før | |---|---|---|---|---|---| -| **G1** | Judgen er ikke herdet mot de 8 dokumenterte feilmodusene (`source_silent`-maskerer-fravær, legacy-rad-match, ramme-skifte-tall-overlever, nedre-grense-understatement, eksakt-streng-pedanteri, taksonomi-nyanse, kapabilitet-bom) | Judge-FN/FP påvist i Spor 2b (8 mål, se `ref-kb-gold-reconciliation-2026-06.md`) | **Spor 2a** — judge-prompt-v3, MÅLT single vs v3/ensemble på herdet gull; adopter kun ved målt forbedring (ad-hoc-patch overfitter + bytter P/R) | 🟡 **v3 REHABILITERT 2026-06-30 av G5** — mot fersk gull slår v3 (P 89,7/R 92,1) v2 (P 86,8/R 86,8) på BEGGE akser; «regresjonen» var stale gull. v3 = interim baseline; **v3.1 neste** (slå v3, ikke v2) | Spor 1 korpus-pass (judgen brukes i ~2700 fetches) | +| **G1** | Judgen er ikke herdet mot de 8 dokumenterte feilmodusene (`source_silent`-maskerer-fravær, legacy-rad-match, ramme-skifte-tall-overlever, nedre-grense-understatement, eksakt-streng-pedanteri, taksonomi-nyanse, kapabilitet-bom) | Judge-FN/FP påvist i Spor 2b (8 mål, se `ref-kb-gold-reconciliation-2026-06.md`) | **Spor 2a** — judge-prompt-v3, MÅLT single vs v3/ensemble på herdet gull; adopter kun ved målt forbedring (ad-hoc-patch overfitter + bytter P/R) | 🟡 **v3 = adoptert baseline; G5b løftet den til P 100 % / R 92,9 % (0 FP) på G5b-korrigert gull.** v3.1 FORFATTET (`judge-claim-prompt-v3.1.md`) — ren recall-hardning av 3 bekreftede FN (R1 øvre-grense, R7 last-bærende-streng, ny R8 fler-delt); FP-vakt DROPPET (G5b: v3 har 0 ekte FP). 45-veis fan-out IKKE kjørt (operatør-gate, stor spend) — bar er nå v3 (hold P=100 ∧ løft R) | Spor 1 korpus-pass (judgen brukes i ~2700 fetches) | | **G2** | Herdet judge er ikke wired inn i Port 2 (born-verified create-guard) + Port 3 (kadens) — uten innplugging binder ikke hardningen mekanisk | Re-introdusert drift ved nye/regenererte filer + kadens-bom | Del av Spor 2a→3: bytt ut v2 med v3 i `transform.mjs`-judge-passet + kadens-runneren | 🔴 (avh. G1) | Spor 1 | | **G3** | Ingen deterministisk gull-intern-konsistens-vakt (`verdict=correct` mens egen `notes` sier «uverifisert/illustrativ») | Gull-labeling-feil av FP1-klassen (selvmotsigende annotasjon) | Liten TDD-lint over `gold-correctness-set.json` (+ kjøres på fremtidige gull-bygg) | 🟢 **lukket 2026-06-30** | Spor 1 (nytt gull bygges) / §7 friskt utvalg | | **G4** | Nedre-grense-policyen lever kun i prosa (denne dok + reconciliation-logg) — ikke kodet i judge-prompt ELLER `build-gold-set`-instruks | Re-introdusert nedre-grense-ambivalens i fremtidige gull-bygg + judge-kjøringer | Kod policyen inn i judge-prompt-v3 (G1) + build-gold-set-instruks | 🟢 **kodet 2026-06-30** (build-instruks + v3 R1); håndheving rir på G1/G2-adopsjon | Spor 1 / §7 friskt utvalg | -| **G5** | Gull-fasiten kan aldre — ingen friskhets-/re-adjuderings-vakt på selve svarnøkkelen. v3-målingen avdekket at flere judge-«feil» trolig er *utdatert gull*, ikke judge-feil (`genaiops-llm-specific#2`: claim «1600+», live=1900 ⇒ 1,19× tett nedre grense, R1 sier korrekt `grounded`, gull sier `outdated` — gull-standarden er her for streng) | Feil adopsjonsbeslutning bygd på aldrende baseline; falsk feilrate i §7-nordstjernen | Friskhets-mikropass: re-adjuder de ~5 omstridte v3-vs-v2-claims mot live MS Learn (avgjør gull-feil vs judge-feil) + periodisk gull-re-adjudering knyttet til §7 friskt utvalg | 🟢 **lukket 2026-06-30** (2 gull-feil rettet, 3 judge-feil bekreftet; **reverserte adopsjonsbeslutningen** — se lukke-logg) | v3.1-adopsjon (baseline må være til å stole på FØR ny prompt måles mot den) | +| **G5** | Gull-fasiten kan aldre — ingen friskhets-/re-adjuderings-vakt på selve svarnøkkelen. v3-målingen avdekket at flere judge-«feil» trolig er *utdatert gull*, ikke judge-feil (`genaiops-llm-specific#2`: claim «1600+», live=1900 ⇒ 1,19× tett nedre grense, R1 sier korrekt `grounded`, gull sier `outdated` — gull-standarden er her for streng) | Feil adopsjonsbeslutning bygd på aldrende baseline; falsk feilrate i §7-nordstjernen | Friskhets-mikropass: re-adjuder de ~5 omstridte v3-vs-v2-claims mot live MS Learn (avgjør gull-feil vs judge-feil) + periodisk gull-re-adjudering knyttet til §7 friskt utvalg | 🟢 **lukket 2026-06-30** (G5: 2 gull-feil rettet, 3 judge-feil bekreftet, **reverserte adopsjonsbeslutningen**; **G5b: completeness-caveat lukket** — de 4 v3-FP re-sjekket, ALLE 4 stale gull, v3 → P 100 % / R 92,9 % / 0 FP — se lukke-logg) | v3.1-adopsjon (baseline må være til å stole på FØR ny prompt måles mot den) | -**Ikke mekanisme-gap, men sporet backlog (innhold, ikke loop):** reference-`.md`-fil-fiksene fra Spor 2b (FP1 11000+/40+, FP2 «kun», FP6 Preview/Norway-East, FN2–FN6 utdaterte tall) er **Spor 0/1**-innholdsarbeid — pekt per-claim i `notes`, ikke gjentakelses-mekanisme. Føres i Spor 0-manifest / Spor 1-korpus-pass, ikke her. +**Ikke mekanisme-gap, men sporet backlog (innhold, ikke loop):** reference-`.md`-fil-fiksene fra Spor 2b (FP1 11000+/40+, FP2 «kun», FP6 Preview/Norway-East, FN2–FN6 utdaterte tall) **+ G5b** (`adr-template.md` fjern «zero permission management»; `multi-region-azure-openai-deployment.md` bytt retired `gpt-35-turbo` → gjeldende modell; `network-resilience-patterns-ai.md` «obligatorisk» → «anbefalt»; `vector-storage-cost-optimization.md` GA-dato `2024-11-01` → `2024-07-01`) er **Spor 0/1**-innholdsarbeid — pekt per-claim i `notes`, ikke gjentakelses-mekanisme. Føres i Spor 0-manifest / Spor 1-korpus-pass, ikke her. ### Lukke-logg @@ -124,3 +124,12 @@ Status-nøkkel: 🔴 ikke startet · 🟡 pågår · 🟢 lukket. - **Korreksjon av linje 114 (v3-måling, mot stale gull):** `model-selection#8` var IKKE «R7 for ettergivende» — det var R7 vindisert (gull-feil). v3.1 R7-vakten gjelder kun load-bearing-strenger (`token-usage#3`). - **Baseline-revurdering (re-score, samme gull begge):** v2 falt 92,1/87,5 → **86,8/86,8** (mistet 2 TP→FP på de rettede claims — var oppblåst av stale gull). v3 steg 89,7/87,5 → **89,7/92,1** (2 FN→TN). **v3 slår nå v2 på BEGGE akser.** Den opprinnelige «v3 regredierte» var et stale-gull-artefakt. Den detaljerte 22-flip-tellingen i linje 113 var mot stale gull (2 «regresjoner» = claims 1+3 er nå presisjon-forbedringer FP→TN) — superseded av re-scoren. Artefakter: `judge-bakeoff-report-v2-g5gold.{json,md}`, `judge-bakeoff-report-v3-g5gold.{json,md}`. Gull-`_meta.reconciliation_log` + lint (373 claims, 0 flagget) + suite 641/641 grønt. - **Beslutning:** v3 = **interim adoptert baseline** (slår v2 på fersk gull per forhåndsregistrert gate). v3.1-baren er nå **v3 (89,7/92,1)**, ikke v2 — strengere og ærligere. **Completeness-caveat:** kun de 5 omstridte ble re-sjekket; de 4 v3-FP-claims (`adr-template#1` m.fl.) sitt gull er IKKE re-verifisert (antatt `correct`; spot-sjekk under v3.1). Periodisk gull-re-adjudering knyttes til §7 friskt utvalg (G5-mekanismen er nå et mønster, ikke engangs). +- **G5b 🟢 lukket (2026-06-30) — completeness-caveat innfridd; baseline løftet til P 100 %.** G5s eksplisitte gjenstående caveat (de 4 v3-FP-claims hadde antatt, ikke verifisert, `correct`-gull) lukket: de 4 (`adr-template#1`, `multi-region-azure-openai-deployment#2`, `network-resilience-patterns-ai#4`, `vector-storage-cost-optimization#7`) re-adjudert mot live MS Learn, **én Opus-4.8-subagent per claim, blind for gull/v3-verdikt** (anti-anchoring, samme protokoll som G5). + - **Utfall — ALLE 4 var stale gull; v3 flagget hver korrekt (0 ekte FP):** + 1. `adr-template#1` (status) «zero permission management, permissions respekteres automatisk» — live (`data-privacy-security` + `connecting-external-content-manage-items`): del B (auto-respektert ved grounding) stemmer, men del A motsies — Graph connectors krever ACL per `externalItem` + identitets-mapping. **Gull-feil** `correct→wrong`. v3 `not_grounded` (R6) korrekt. + 2. `multi-region-azure-openai-deployment#2` (region) «Sweden Central … gpt-4o, o1, gpt-35-turbo» — live: gpt-4o + o1 tilgjengelig, men **gpt-35-turbo er retired** (0301/0613 feb 2025; 0125/1106 fra sep 2025), borte fra katalogen. **Gull-feil** `correct→outdated`. v3 `not_grounded` (R2 entitet-fravær) korrekt. + 3. `network-resilience-patterns-ai#4` (status) «Circuit Breaker + Retry … obligatorisk for alle Azure AI API-kall» — live (`how-to/quota`): MS rammer dette som «Rate limit best practices / recommended», circuit breaker kun valgfri Polly-utvidelse. «Obligatorisk for alle» overdriver modalitet + omfang. **Gull-feil** `correct→wrong`. v3 `not_grounded` (R6) korrekt. + 4. `vector-storage-cost-optimization#7` (status) «Vector quantization GA siden 2024-11-01» — live (`search-api-migration`): quantization ER GA, men GA-dato var **2024-07-01** (stable release); «2024-11-01» finnes kun som *preview*-API-versjon (`2024-11-01-preview`). Last-bærende dato feil. **Gull-feil** `correct→wrong`. v3 `not_grounded` korrekt. + - **Baseline-revurdering (re-score, G5b-korrigert gull, samme gull begge):** de 4 flyttet FP→TP for v3. **v3: P 89,7/92,1 → 100,0 / 92,9 (TP 39, FP 0, FN 3, TN 198).** v2: 86,8/86,8 → **86,8 / 78,6** (de 4 ble FN for v2 — v2 flagget ingen). v3 dominerer nå v2 på begge akser med større margin; gull var fortsatt kontaminert. Artefakter: `judge-bakeoff-report-v3-g5bgold.{json,md}`, `judge-bakeoff-report-v2-g5bgold.{json,md}`. Gull `_meta.reconciliation_log` + lint (373 claims, 0 flagget) + suite 641/641 grønt. + - **Konsekvens for v3.1-design (PLAN-INVERSJON):** STATEs planlagte v3.1-endring #4 (R2/R6 «FP-vakt» for de 4) er **droppet** — R2/R6 fanget disse korrekt; en vakt ville re-knekt 3 reelle treff. v3.1 er nå **ren recall-hardning** av de 3 gjenstående FN (R1 øvre-grense-skille, R7 last-bærende-streng-carve-out, ny R8 fler-delt-fullstendighet) — `judge-claim-prompt-v3.1.md` forfattet. **Adopsjonsgate strammet:** v3 sitter på presisjonstaket (P=100), så v3.1 må **holde P=100 OG løfte R over 92,9** — enhver ny FP feller den. 45-veis fan-out gjenstår (operatør-gate, stor spend). + - **Mønster bekreftet:** G5b er andre gang gull-friskhet inverterte en adopsjonskonklusjon ([[gold-freshness-can-invert-adoption]]). Gull-re-adjudering FØR baseline stoles på er nå fast disiplin, ikke engangs — knyttes til §7 friskt utvalg. diff --git a/docs/ref-kb-gold-reconciliation-2026-06.md b/docs/ref-kb-gold-reconciliation-2026-06.md index b138a51..b3bb424 100644 --- a/docs/ref-kb-gold-reconciliation-2026-06.md +++ b/docs/ref-kb-gold-reconciliation-2026-06.md @@ -102,3 +102,22 @@ Disse er **ikke** gull-endringer — gull sto, judgen bommet. Grupperte feilmodu ## Hva som IKKE ble gjort (scope-grense) Spor 2b retter **fasiten** (gull-settet), ikke reference-`.md`-filene. Fil-fiksene (FP1 11000+/40+, FP2 «kun», FP6 Public-Preview/Norway-East, FN2–FN6 utdaterte tall) er **Spor 0/1**-arbeid og er pekt ut i hver claims `notes`. Mange av FN-ene er reelle korpus-feil som hører til Spor 0-manifestet / Spor 1-korpus-passet. + +## Addendum — etterfølgende gull-friskhets-flips (G5 + G5b, kanonisk logg i programdok §8) + +Spor 2b var ikke siste ord: gull-fasiten eldes (G5-gapet). Senere friskhets-passes (full logg + belegg i `ref-kb-correctness-program-2026-06.md` §8 lukke-logg) flyttet ytterligere **6 gull-verdikt** mot live MS Learn. Samlet flip-ledger for komplett sporbarhet: + +| Pass | Claim | Før | Etter | Retning | +|---|---|---|---|---| +| 2b | `azure-ai-foundry.md#2` | correct | wrong | for streng gull → feil avslørt | +| 2b | `multimodal-prompt-design.md#7` | correct | wrong | — | +| 2b | `ai-foundry-disaster-recovery-planning.md#9` | correct | outdated | — | +| 2b | `real-time-reasoning-performance.md#5` | outdated | correct | gull for streng → rettet | +| **G5** | `genaiops-llm-specific-practices.md#2` | outdated | **correct** | aldrende gull (1600+ vs live 1900, tett nedre grense) | +| **G5** | `model-selection-price-performance.md#8` | outdated | **correct** | aldrende gull (Model Router GA nov 2025) | +| **G5b** | `adr-template.md#1` | correct | **wrong** | stale gull (Graph connectors krever ACL) | +| **G5b** | `multi-region-azure-openai-deployment.md#2` | correct | **outdated** | stale gull (gpt-35-turbo retired) | +| **G5b** | `network-resilience-patterns-ai.md#4` | correct | **wrong** | stale gull («obligatorisk» vs «recommended») | +| **G5b** | `vector-storage-cost-optimization.md#7` | correct | **wrong** | stale gull (GA-dato 2024-07-01, ikke 2024-11-01-preview) | + +**Mønster:** gull-friskhet inverterte adopsjonskonklusjonen **to ganger** (G5 og G5b). Re-adjudering av omstridt gull mot live FØR en baseline stoles på er nå fast disiplin ([[gold-freshness-can-invert-adoption]]), knyttet til §7 friskt utvalg. Effekt på adoptert baseline: v3 målt **P 100 % / R 92,9 % / 0 FP** på G5b-korrigert gull (`judge-bakeoff-report-v3-g5bgold.{json,md}`). diff --git a/scripts/kb-eval/data/gold-correctness-set.json b/scripts/kb-eval/data/gold-correctness-set.json index 606c529..63ebed4 100644 --- a/scripts/kb-eval/data/gold-correctness-set.json +++ b/scripts/kb-eval/data/gold-correctness-set.json @@ -8,7 +8,8 @@ "claim_count": 373, "reconciliation_log": [ "2026-06-30 Spor 2b: 12 judge-vs-gold-uenigheter adjudert mot live; 4 gull-feil rettet + 1 note-fiks. Logg: docs/ref-kb-gold-reconciliation-2026-06.md.", - "2026-06-30 G5 friskhets-mikropass: 5 omstridte claims (gold=outdated, v3=grounded) re-adjudert mot live MS Learn. 2 stale gull rettet outdated->correct (genaiops-llm-specific-practices.md#2 '1600+' live 1900 tett nedre grense; model-selection-price-performance.md#8 Model Router GA nov 2025). 3 opprettholdt som outdated (judge-feil bekreftet: multi-model-strategy-costs.md#2, token-usage-tracking-attribution.md#3, ai-foundry-disaster-recovery-planning.md#9). Logg: docs/ref-kb-correctness-program-2026-06.md §8 G5." + "2026-06-30 G5 friskhets-mikropass: 5 omstridte claims (gold=outdated, v3=grounded) re-adjudert mot live MS Learn. 2 stale gull rettet outdated->correct (genaiops-llm-specific-practices.md#2 '1600+' live 1900 tett nedre grense; model-selection-price-performance.md#8 Model Router GA nov 2025). 3 opprettholdt som outdated (judge-feil bekreftet: multi-model-strategy-costs.md#2, token-usage-tracking-attribution.md#3, ai-foundry-disaster-recovery-planning.md#9). Logg: docs/ref-kb-correctness-program-2026-06.md §8 G5.", + "2026-06-30 G5b friskhets-spot-sjekk: de 4 v3-FP-claims (gold=correct, v3=not_grounded) re-adjudert mot live MS Learn (4 Opus-subagenter, blinde). ALLE 4 var stale gull (v3 flagget korrekt): adr-template.md#1 correct->wrong (zero permission management motsies, Graph connectors krever ACL); multi-region-azure-openai-deployment.md#2 correct->outdated (gpt-35-turbo retired); network-resilience-patterns-ai.md#4 correct->wrong (obligatorisk vs anbefalt); vector-storage-cost-optimization.md#7 correct->wrong (GA-dato 2024-07-01, ikke 2024-11-01-preview). v3 hadde 0 ekte FP. Logg: docs/ref-kb-correctness-program-2026-06.md §8 G5b." ] }, "claims": [ @@ -344,11 +345,11 @@ "stratum": "volatile", "claim": "SharePoint Embedded/Graph Connectors: zero permission management, permissions respekteres automatisk", "claim_type": "status", - "verdict": "correct", + "verdict": "wrong", "evidence_url": "https://learn.microsoft.com/microsoft-365/copilot/extensibility/data-privacy-security", "lastmod_changed": false, "file_last_updated": "2026-06-24", - "notes": "Permission inheritance bekreftet." + "notes": "RECONCILED 2026-06-30 (G5b friskhets-spot-sjekk): correct->wrong. Live (m365/copilot/extensibility/data-privacy-security + graph/connecting-external-content-manage-items): \"You can manage permissions to view external items by associating an access control list (ACL)\"; hver externalItem MA ha ACL (ikke-Entra-brukere ma mappes til Entra). Del B (permissions respekteres automatisk ved grounding) stemmer, men del A \"zero permission management\" motsies - Graph connectors krever ACL-forfatting. Fil-fiks (Spor 0/1): fjern \"zero permission management\", behold permission-honoring. Judge not_grounded var korrekt. Confidence: medium (innsats-overdrivelse pa last-baerende del A)." }, { "id": "ms-ai-advisor/architecture/adr-template.md#2", @@ -3165,11 +3166,11 @@ "stratum": "volatile", "claim": "Sweden Central sekundær, bred (gpt-4o, o1, gpt-35-turbo)", "claim_type": "region", - "verdict": "correct", + "verdict": "outdated", "evidence_url": "https://learn.microsoft.com/azure/foundry/foundry-models/concepts/models-sold-directly-by-azure-region-availability", "lastmod_changed": false, "file_last_updated": "2026-06-24", - "notes": "Sweden Central bred støtte bekreftet." + "notes": "RECONCILED 2026-06-30 (G5b friskhets-spot-sjekk): correct->outdated. Live (models-sold-directly-by-azure-region-availability + retirements): gpt-4o og o1 tilgjengelig i Sweden Central, men gpt-35-turbo er RETIRED (0301/0613 feb 2025; 0125/1106 fra sep 2025) og finnes ikke lenger i katalogen. Claim listet gpt-35-turbo som tilgjengelig - var sant, na utdatert. Fil-fiks (Spor 0/1): bytt gpt-35-turbo med gjeldende modell (gpt-4.1-mini/gpt-4o-mini). Judge not_grounded (R2 entitet-fravaer) var korrekt. Confidence: high." }, { "id": "ms-ai-infrastructure/bcdr/multi-region-azure-openai-deployment.md#3", @@ -3399,11 +3400,11 @@ "stratum": "volatile", "claim": "Circuit Breaker + Retry exponential backoff obligatorisk for alle Azure AI API-kall", "claim_type": "status", - "verdict": "correct", + "verdict": "wrong", "evidence_url": "https://learn.microsoft.com/azure/foundry-classic/openai/how-to/quota", "lastmod_changed": false, "file_last_updated": "2026-06-24", - "notes": "Retry m/backoff + circuit breaker offisielt anbefalt; 'obligatorisk' sterk formulering." + "notes": "RECONCILED 2026-06-30 (G5b friskhets-spot-sjekk): correct->wrong. Live (foundry-classic/openai/how-to/quota): MS rammer dette som \"Rate limit best practices\" / \"recommended\", ikke obligatorisk; circuit breaker nevnes kun som valgfritt avansert Polly-monster. Claim \"obligatorisk for alle Azure AI API-kall\" overdriver modaliteten (anbefalt -> palagt) og omfanget (alle kall). Monstrene er reelle MS-anbefalinger, men \"obligatorisk\" er ikke grunnet. Fil-fiks (Spor 0/1): bytt \"obligatorisk\" med \"anbefalt\". Judge not_grounded (R6) var korrekt. Confidence: medium (modalitets-overdrivelse)." }, { "id": "ms-ai-infrastructure/bcdr/service-level-documentation-dr.md#1", @@ -4533,11 +4534,11 @@ "stratum": "volatile", "claim": "Vector quantization GA siden 2024-11-01", "claim_type": "status", - "verdict": "correct", + "verdict": "wrong", "evidence_url": "https://learn.microsoft.com/azure/search/vector-search-index-size", "lastmod_changed": false, "file_last_updated": "2026-06-19", - "notes": "GA-status bekreftet." + "notes": "RECONCILED 2026-06-30 (G5b friskhets-spot-sjekk): correct->wrong. Live (search-api-migration + vector-search-index-size): vector quantization ER GA, MEN GA-dato var 2024-07-01 (stable release), ikke 2024-11-01. \"2024-11-01\" finnes kun som preview-API-versjon (2024-11-01-preview). Last-baerende dato er feil. Fil-fiks (Spor 0/1): rett GA-dato til 2024-07-01. Judge not_grounded var korrekt. Confidence: high." }, { "id": "ms-ai-security/cost-optimization/vector-storage-cost-optimization.md#8", diff --git a/scripts/kb-eval/data/judge-bakeoff-report-v2-g5bgold.json b/scripts/kb-eval/data/judge-bakeoff-report-v2-g5bgold.json new file mode 100644 index 0000000..1c57e20 --- /dev/null +++ b/scripts/kb-eval/data/judge-bakeoff-report-v2-g5bgold.json @@ -0,0 +1,234 @@ +{ + "_meta": { + "source": "gold-correctness-set.json + judge-bakeoff-results.json", + "thresholds": { + "minRecall": 0.7, + "minPrecision": 0.6 + }, + "judged": 255 + }, + "population": { + "total": 255, + "verifiable": 240, + "positives": 42, + "negatives": 198, + "unsourcedInP": 15 + }, + "arms": { + "staleness": { + "tp": 0, + "fp": 0, + "fn": 42, + "tn": 198, + "positives": 42, + "negatives": 198, + "flagged": 0, + "precision": null, + "recall": 0, + "f1": null, + "recallWilson": { + "p": 0, + "low": 0, + "high": 0.08380161250916199 + }, + "precisionWilson": null + }, + "judge": { + "tp": 33, + "fp": 5, + "fn": 9, + "tn": 193, + "positives": 42, + "negatives": 198, + "flagged": 38, + "precision": 0.868421052631579, + "recall": 0.7857142857142857, + "f1": 0.825, + "recallWilson": { + "p": 0.7857142857142857, + "low": 0.6405986210195627, + "high": 0.8829433146894876 + }, + "precisionWilson": { + "p": 0.868421052631579, + "low": 0.7267282994850112, + "high": 0.9424621712426856 + } + }, + "hybrid": { + "tp": 33, + "fp": 5, + "fn": 9, + "tn": 193, + "positives": 42, + "negatives": 198, + "flagged": 38, + "precision": 0.868421052631579, + "recall": 0.7857142857142857, + "f1": 0.825, + "recallWilson": { + "p": 0.7857142857142857, + "low": 0.6405986210195627, + "high": 0.8829433146894876 + }, + "precisionWilson": { + "p": 0.868421052631579, + "low": 0.7267282994850112, + "high": 0.9424621712426856 + } + } + }, + "sourceSilent": { + "onVerifiableNegative": 1, + "onVerifiableError": 4, + "agreesWithUnsourced": 5, + "disagreesWithUnsourced": 10 + }, + "byClaimType": { + "version": { + "tp": 6, + "fp": 0, + "fn": 1, + "tn": 21, + "positives": 7, + "negatives": 21, + "flagged": 6, + "precision": 1, + "recall": 0.8571428571428571, + "f1": 0.923076923076923, + "recallWilson": { + "p": 0.8571428571428571, + "low": 0.4868654966809701, + "high": 0.9743210440510252 + }, + "precisionWilson": { + "p": 1, + "low": 0.6096569663469354, + "high": 0.9999999999999999 + } + }, + "tpm": { + "tp": 4, + "fp": 0, + "fn": 1, + "tn": 20, + "positives": 5, + "negatives": 20, + "flagged": 4, + "precision": 1, + "recall": 0.8, + "f1": 0.888888888888889, + "recallWilson": { + "p": 0.8, + "low": 0.3755282641185388, + "high": 0.9637768390302125 + }, + "precisionWilson": { + "p": 1, + "low": 0.5100999795960008, + "high": 1 + } + }, + "region": { + "tp": 1, + "fp": 0, + "fn": 1, + "tn": 13, + "positives": 2, + "negatives": 13, + "flagged": 1, + "precision": 1, + "recall": 0.5, + "f1": 0.6666666666666666, + "recallWilson": { + "p": 0.5, + "low": 0.09452865480086614, + "high": 0.9054713451991339 + }, + "precisionWilson": { + "p": 1, + "low": 0.2065432914738929, + "high": 1 + } + }, + "status": { + "tp": 6, + "fp": 1, + "fn": 4, + "tn": 42, + "positives": 10, + "negatives": 43, + "flagged": 7, + "precision": 0.8571428571428571, + "recall": 0.6, + "f1": 0.7058823529411764, + "recallWilson": { + "p": 0.6, + "low": 0.3126695474501863, + "high": 0.8318224187964902 + }, + "precisionWilson": { + "p": 0.8571428571428571, + "low": 0.4868654966809701, + "high": 0.9743210440510252 + } + }, + "taxonomy": { + "tp": 11, + "fp": 3, + "fn": 0, + "tn": 83, + "positives": 11, + "negatives": 86, + "flagged": 14, + "precision": 0.7857142857142857, + "recall": 1, + "f1": 0.88, + "recallWilson": { + "p": 1, + "low": 0.7411599827511859, + "high": 1 + }, + "precisionWilson": { + "p": 0.7857142857142857, + "low": 0.5241027622679172, + "high": 0.9242875166308363 + } + }, + "sku": { + "tp": 5, + "fp": 1, + "fn": 2, + "tn": 14, + "positives": 7, + "negatives": 15, + "flagged": 6, + "precision": 0.8333333333333334, + "recall": 0.7142857142857143, + "f1": 0.7692307692307692, + "recallWilson": { + "p": 0.7142857142857143, + "low": 0.35892909014821267, + "high": 0.9177828342909844 + }, + "precisionWilson": { + "p": 0.8333333333333334, + "low": 0.43649056343635395, + "high": 0.9699474141282697 + } + } + }, + "gate": { + "pass": true, + "recallOk": true, + "precisionOk": true, + "beatsStaleness": true, + "thresholds": { + "minRecall": 0.7, + "minPrecision": 0.6 + }, + "reasons": [ + "all criteria met" + ] + } +} diff --git a/scripts/kb-eval/data/judge-bakeoff-report-v2-g5bgold.md b/scripts/kb-eval/data/judge-bakeoff-report-v2-g5bgold.md new file mode 100644 index 0000000..6c9d9ba --- /dev/null +++ b/scripts/kb-eval/data/judge-bakeoff-report-v2-g5bgold.md @@ -0,0 +1,54 @@ +# Judge bake-off-rapport — S1 (Fase 3 de-risk) + +_Generert deterministisk av `run-judge-bakeoff.mjs` over `gold-correctness-set.json` + `judge-bakeoff-results.json`. Tall fra testet `lib/judge-bakeoff.mjs`. Ikke rediger for hånd — regenerer._ + +**Forhåndsregistrert gate (låst FØR fan-out):** recall ≥ 0.7, presisjon ≥ 0.6, OG judge-recall > staleness-recall. + +## Evaluerings-populasjon (P) + +Volatil stratum + fetchbare claim_types (price ekskludert) — der feilene bor; unngår «invertert leverage». + +| metrikk | verdi | +|---|---| +| P totalt | 255 | +| Verifiserbare (correct/outdated/wrong) | 240 | +| Positive (reelle feil å fange) | 42 | +| Negative (correct) | 198 | +| Unsourced i P (kjørt, men utenfor P/R) | 15 | + +## Arm-sammenligning (detektering over de 240 verifiserbare) + +| arm | TP | FP | FN | TN | presisjon | recall | recall Wilson 95% | F1 | +|---|---|---|---|---|---|---|---|---| +| staleness (billig baseline) | 0 | 0 | 42 | 198 | n/a | 0.0% | [0.0%, 8.4%] | n/a | +| judge (per-påstand groundedness) | 33 | 5 | 9 | 193 | 86.8% | 78.6% | [64.1%, 88.3%] | 0.825 | +| hybrid (union) | 33 | 5 | 9 | 193 | 86.8% | 78.6% | [64.1%, 88.3%] | 0.825 | + +## Judge per claim_type (verifiserbar delmengde) + +| claim_type | positive | TP | FP | FN | presisjon | recall | +|---|---|---|---|---|---|---| +| taxonomy | 11 | 11 | 3 | 0 | 78.6% | 100.0% | +| status | 10 | 6 | 1 | 4 | 85.7% | 60.0% | +| version | 7 | 6 | 0 | 1 | 100.0% | 85.7% | +| sku | 7 | 5 | 1 | 2 | 83.3% | 71.4% | +| tpm | 5 | 4 | 0 | 1 | 100.0% | 80.0% | +| region | 2 | 1 | 0 | 1 | 100.0% | 50.0% | + +## source_silent-diagnostikk + +Judgen hentet siden men fant ikke verdien. Diagnostisk, ikke et flagg. + +| signal | antall | tolkning | +|---|---|---| +| På verifiserbar feil | 4 | judge-bom: reell feil oversett via «kan ikke verifisere» | +| På verifiserbar correct | 1 | judge reproduserte ikke et korrekt faktum mennesket fant | +| Enig med unsourced | 5 | judge reproduserer den uverifiserbare grensen (godt) | +| Uenig med unsourced | 10 | judge hevdet grunnet/ugrunnet der mennesket ikke fant kilde | + +## GATE: ✅ PASS — bygg S3 + +- recall 0.786 ≥ 0.7? **ja** +- presisjon 0.868 ≥ 0.6? **ja** +- slår staleness (recall 0.000)? **ja** +- begrunnelse: all criteria met diff --git a/scripts/kb-eval/data/judge-bakeoff-report-v3-g5bgold.json b/scripts/kb-eval/data/judge-bakeoff-report-v3-g5bgold.json new file mode 100644 index 0000000..7be2462 --- /dev/null +++ b/scripts/kb-eval/data/judge-bakeoff-report-v3-g5bgold.json @@ -0,0 +1,234 @@ +{ + "_meta": { + "source": "gold-correctness-set.json + judge-bakeoff-results.json", + "thresholds": { + "minRecall": 0.7, + "minPrecision": 0.6 + }, + "judged": 255 + }, + "population": { + "total": 255, + "verifiable": 240, + "positives": 42, + "negatives": 198, + "unsourcedInP": 15 + }, + "arms": { + "staleness": { + "tp": 0, + "fp": 0, + "fn": 42, + "tn": 198, + "positives": 42, + "negatives": 198, + "flagged": 0, + "precision": null, + "recall": 0, + "f1": null, + "recallWilson": { + "p": 0, + "low": 0, + "high": 0.08380161250916199 + }, + "precisionWilson": null + }, + "judge": { + "tp": 39, + "fp": 0, + "fn": 3, + "tn": 198, + "positives": 42, + "negatives": 198, + "flagged": 39, + "precision": 1, + "recall": 0.9285714285714286, + "f1": 0.962962962962963, + "recallWilson": { + "p": 0.9285714285714286, + "low": 0.8099028671147483, + "high": 0.9754100364488272 + }, + "precisionWilson": { + "p": 1, + "low": 0.9103301463997611, + "high": 1 + } + }, + "hybrid": { + "tp": 39, + "fp": 0, + "fn": 3, + "tn": 198, + "positives": 42, + "negatives": 198, + "flagged": 39, + "precision": 1, + "recall": 0.9285714285714286, + "f1": 0.962962962962963, + "recallWilson": { + "p": 0.9285714285714286, + "low": 0.8099028671147483, + "high": 0.9754100364488272 + }, + "precisionWilson": { + "p": 1, + "low": 0.9103301463997611, + "high": 1 + } + } + }, + "sourceSilent": { + "onVerifiableNegative": 0, + "onVerifiableError": 0, + "agreesWithUnsourced": 2, + "disagreesWithUnsourced": 13 + }, + "byClaimType": { + "version": { + "tp": 7, + "fp": 0, + "fn": 0, + "tn": 21, + "positives": 7, + "negatives": 21, + "flagged": 7, + "precision": 1, + "recall": 1, + "f1": 1, + "recallWilson": { + "p": 1, + "low": 0.6456611570247934, + "high": 1 + }, + "precisionWilson": { + "p": 1, + "low": 0.6456611570247934, + "high": 1 + } + }, + "tpm": { + "tp": 5, + "fp": 0, + "fn": 0, + "tn": 20, + "positives": 5, + "negatives": 20, + "flagged": 5, + "precision": 1, + "recall": 1, + "f1": 1, + "recallWilson": { + "p": 1, + "low": 0.5655085052479191, + "high": 1 + }, + "precisionWilson": { + "p": 1, + "low": 0.5655085052479191, + "high": 1 + } + }, + "region": { + "tp": 2, + "fp": 0, + "fn": 0, + "tn": 13, + "positives": 2, + "negatives": 13, + "flagged": 2, + "precision": 1, + "recall": 1, + "f1": 1, + "recallWilson": { + "p": 1, + "low": 0.34237195288961925, + "high": 1 + }, + "precisionWilson": { + "p": 1, + "low": 0.34237195288961925, + "high": 1 + } + }, + "status": { + "tp": 9, + "fp": 0, + "fn": 1, + "tn": 43, + "positives": 10, + "negatives": 43, + "flagged": 9, + "precision": 1, + "recall": 0.9, + "f1": 0.9473684210526316, + "recallWilson": { + "p": 0.9, + "low": 0.5958436145024278, + "high": 0.9821242504842788 + }, + "precisionWilson": { + "p": 1, + "low": 0.7008472464490407, + "high": 1 + } + }, + "taxonomy": { + "tp": 9, + "fp": 0, + "fn": 2, + "tn": 86, + "positives": 11, + "negatives": 86, + "flagged": 9, + "precision": 1, + "recall": 0.8181818181818182, + "f1": 0.9, + "recallWilson": { + "p": 0.8181818181818182, + "low": 0.5230138624217553, + "high": 0.9486333993289995 + }, + "precisionWilson": { + "p": 1, + "low": 0.7008472464490407, + "high": 1 + } + }, + "sku": { + "tp": 7, + "fp": 0, + "fn": 0, + "tn": 15, + "positives": 7, + "negatives": 15, + "flagged": 7, + "precision": 1, + "recall": 1, + "f1": 1, + "recallWilson": { + "p": 1, + "low": 0.6456611570247934, + "high": 1 + }, + "precisionWilson": { + "p": 1, + "low": 0.6456611570247934, + "high": 1 + } + } + }, + "gate": { + "pass": true, + "recallOk": true, + "precisionOk": true, + "beatsStaleness": true, + "thresholds": { + "minRecall": 0.7, + "minPrecision": 0.6 + }, + "reasons": [ + "all criteria met" + ] + } +} diff --git a/scripts/kb-eval/data/judge-bakeoff-report-v3-g5bgold.md b/scripts/kb-eval/data/judge-bakeoff-report-v3-g5bgold.md new file mode 100644 index 0000000..ec14f97 --- /dev/null +++ b/scripts/kb-eval/data/judge-bakeoff-report-v3-g5bgold.md @@ -0,0 +1,54 @@ +# Judge bake-off-rapport — S1 (Fase 3 de-risk) + +_Generert deterministisk av `run-judge-bakeoff.mjs` over `gold-correctness-set.json` + `judge-bakeoff-results.json`. Tall fra testet `lib/judge-bakeoff.mjs`. Ikke rediger for hånd — regenerer._ + +**Forhåndsregistrert gate (låst FØR fan-out):** recall ≥ 0.7, presisjon ≥ 0.6, OG judge-recall > staleness-recall. + +## Evaluerings-populasjon (P) + +Volatil stratum + fetchbare claim_types (price ekskludert) — der feilene bor; unngår «invertert leverage». + +| metrikk | verdi | +|---|---| +| P totalt | 255 | +| Verifiserbare (correct/outdated/wrong) | 240 | +| Positive (reelle feil å fange) | 42 | +| Negative (correct) | 198 | +| Unsourced i P (kjørt, men utenfor P/R) | 15 | + +## Arm-sammenligning (detektering over de 240 verifiserbare) + +| arm | TP | FP | FN | TN | presisjon | recall | recall Wilson 95% | F1 | +|---|---|---|---|---|---|---|---|---| +| staleness (billig baseline) | 0 | 0 | 42 | 198 | n/a | 0.0% | [0.0%, 8.4%] | n/a | +| judge (per-påstand groundedness) | 39 | 0 | 3 | 198 | 100.0% | 92.9% | [81.0%, 97.5%] | 0.963 | +| hybrid (union) | 39 | 0 | 3 | 198 | 100.0% | 92.9% | [81.0%, 97.5%] | 0.963 | + +## Judge per claim_type (verifiserbar delmengde) + +| claim_type | positive | TP | FP | FN | presisjon | recall | +|---|---|---|---|---|---|---| +| taxonomy | 11 | 9 | 0 | 2 | 100.0% | 81.8% | +| status | 10 | 9 | 0 | 1 | 100.0% | 90.0% | +| version | 7 | 7 | 0 | 0 | 100.0% | 100.0% | +| sku | 7 | 7 | 0 | 0 | 100.0% | 100.0% | +| tpm | 5 | 5 | 0 | 0 | 100.0% | 100.0% | +| region | 2 | 2 | 0 | 0 | 100.0% | 100.0% | + +## source_silent-diagnostikk + +Judgen hentet siden men fant ikke verdien. Diagnostisk, ikke et flagg. + +| signal | antall | tolkning | +|---|---|---| +| På verifiserbar feil | 0 | judge-bom: reell feil oversett via «kan ikke verifisere» | +| På verifiserbar correct | 0 | judge reproduserte ikke et korrekt faktum mennesket fant | +| Enig med unsourced | 2 | judge reproduserer den uverifiserbare grensen (godt) | +| Uenig med unsourced | 13 | judge hevdet grunnet/ugrunnet der mennesket ikke fant kilde | + +## GATE: ✅ PASS — bygg S3 + +- recall 0.929 ≥ 0.7? **ja** +- presisjon 1.000 ≥ 0.6? **ja** +- slår staleness (recall 0.000)? **ja** +- begrunnelse: all criteria met diff --git a/scripts/kb-eval/judge-claim-prompt-v3.1.md b/scripts/kb-eval/judge-claim-prompt-v3.1.md new file mode 100644 index 0000000..a38a725 --- /dev/null +++ b/scripts/kb-eval/judge-claim-prompt-v3.1.md @@ -0,0 +1,225 @@ +# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs) + +v3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file +design, same three verdicts, same output schema, same evidence discipline. v3.1 +changes only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the +**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed +were genuine judge misses, not stale gold). + +**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b +freshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false +positives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3 +flagged every one correctly** (`adr-template#1` "zero permission management" is +contradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4` +overstates "recommended" as "obligatorisk"; `vector-storage#7` cites the wrong GA +date). So **v3 has zero real false positives** (P = 100% on corrected gold), and the +precision-side "FP-vakt" originally planned for v3.1 is **dropped — there is nothing to +defend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the +judge catches 3 documented failure modes it currently misses, without touching the +precision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged). + +**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% / +R 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only +be adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without +introducing a single new false positive). Any new FP drops P below 100% and fails the +gate — keep v3. Recall rules are double-edged over the full population (v3's own +bake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay +frozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the +corrected `gold-correctness-set.json`. + +--- + +You are a correctness judge for Microsoft AI reference documentation. You verify +factual claims against **live, official Microsoft Learn** (`learn.microsoft.com`). +Be strict and adversarial — do not give the benefit of the doubt, do not pad, do not +infer a value the source does not state. + +You are judging claims extracted from ``. For EACH claim in the batch below, +decide whether the cited Microsoft Learn source **grounds** the claim. + +## The three verdicts (exhaustive, mutually exclusive) + +- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed + value(s). The page supports the claim. (Maps to gold `correct`.) +- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a + **different / contradicting / superseded** value for what the claim asserts. The + claim disagrees with the source. (Maps to gold `outdated` + `wrong`.) +- **`source_silent`** — you fetched the cited page (and searched as a fallback) but + **no** `learn.microsoft.com` page states the claimed value at all. You cannot + confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure + pages typically lands here — that is expected, not a failure. **Exception: existence + claims — see Rule R2.** + +## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force) + +A claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s). +Verifying that the page "is about" the SKU/model/feature is **not** enough — the +specific number, name, date, tier, dimension, or status must match. If the claim +asserts value **X** and the page states a **different** value **Y** (even if adjacent +or plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for +`not_grounded`: you still need a fetched quote stating the **differing** value. + +Applies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`. + +--- + +## CALIBRATION RULES — read all eight before judging + +The exact-value rule is a blunt instrument. The 8 rules below sharpen it on both +edges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop +over-flagging where the core is grounded** (more correctly `grounded`). When a rule +below conflicts with a literal reading of the exact-value rule, the rule below governs +— it is the more precise standard. + +### Recall side — flag these as `not_grounded` + +**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).** +A claim may assert a **bound**. Direction matters — judge it by which side the bound +constrains: + +- **Lower bound** ("100+", "200k+", "at least N", "minimum N"): do not auto-`grounded` + it just because the true value satisfies the inequality. Apply the **lower-bound + policy:** if the true current value **grossly exceeds** the stated bound — roughly + **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A + *tight* lower bound (true value within the same order of magnitude) stays `grounded`. + *Example (FN2): "200k+ context" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.* +- **Upper bound** ("up to N", "opptil N", "maximum N", "no more than N", "as many as N"): + this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The + exact-value rule governs: if the live page states a current maximum **higher** than N, + the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even + a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it + is really higher. *Example (v3-FN): claim "up to 18 underlying models" while the page + states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true + maximum is N or the claim's ceiling still binds.) + +**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the +claim asserts that a named entity **exists / is offered / is in a list** ("X is a +built-in judge", "feature Y is available", "tier Z exists"), and you fetch the +authoritative page that *would* enumerate it and the entity is **absent**, that absence +is **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve +`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered +prices). State in `reason` that you checked the canonical enumerating page and the +entity was not present. *Example (FN5): claim "99.99% SLA tier" while the reliability +page lists only 99.9% → absence of any 99.99% tier = `not_grounded`.* + +**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can +be superseded even when derived ratios survive. If the page shows the claim's framing +has been **replaced** (e.g. "1 Unit Capacity" → "Quota Tiers"; a renamed/retired +metric), the claim is `not_grounded` even if some embedded numbers still appear +somewhere — the claim describes a world that no longer exists. Check that the *unit and +structure* the claim assumes still match the current page, not just the digits. + +**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or +effective-dated rows ("Before April 3, 2024", "Legacy", "Retiring"). A claim is +`grounded` only if it matches the **current/effective** row. Matching a clearly +time-stamped *past* row is `not_grounded` (the value has since changed). Always locate +the row that applies *today*. *Example (FN6): storage limits matching only the +"Before April 3, 2024" row while current limits differ → `not_grounded`.* + +**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the +`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing +sub-assertions** (a status AND a region; a capability AND a named target; a date AND a +GA level). Verify **each load-bearing part separately**. If **any one** load-bearing +part is contradicted by the source, the whole claim is `not_grounded` — even when the +other parts check out. Do not let a correct first half earn a `grounded` for a wrong +second half. *Example (v3-FN): "Global training (Public Preview), cheaper, no data +residency; use regional in Norway East" — the GA-vs-Preview part and the "no residency" +part hold, but **Norway East is a Global (non-residency) training region, not a regional +one** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6: +R6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated, +load-bearing** part that is wrong. Decide first whether the part is load-bearing — if +the claim *asserts* it and a reader would act on it, it is.) + +### Precision side — keep these `grounded` (do not over-flag) + +**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a +numeric claim merely because the exact string is not verbatim, when the asserted value +is the **documented theoretical or benchmark equivalent** of what the page states and +both trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the +same technique, same order of magnitude, same direction). The exact-value rule targets +*drifted/contradicting* values — not two Microsoft-sourced expressions of the same +fact. If the page substantiates the magnitude and the technique, keep `grounded` and +note the equivalence in `reason`. + +**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish "the +claim's **core** assertion is grounded but it omits a sub-category" from "the core is +ungrounded." If the page confirms the claim's **central** behavior/categorization and +the only gap is an *unstated additional* case the claim did not deny, that is +`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the +page **maps the core differently** or the claim **asserts** something the page +contradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here; +a *stated* but wrong load-bearing part is not — that is R8's domain.) + +**R7 — Follow the capability to its canonical page; don't punish illustrative numbers +(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real +capability** and the cited `evidence_url` does not foreground it, search for the +**canonical** page that documents the capability before judging — do not return +`not_grounded` merely because the *cited* page was a weak choice. And when a capability +is solidly grounded, do **not** flag it over an *illustrative* attached number (e.g. +"~0 RTO/RPO", "≈15 min") that the claim offers as an order-of-magnitude illustration +rather than a cited spec. Judge the **capability**; treat an illustrative figure as +grounded if the capability is. + +> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency +> covers only *illustrative* values. It does **NOT** cover a value or **exact string** +> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum +> value, a specific date/version. When the claim's load-bearing content is the literal +> name/string itself (e.g. "the metrics are `PromptTokens` and `CompletionTokens`"), +> the exact-value rule applies in full: if the live page names them differently +> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the +> claim is `not_grounded`. "Follow to the canonical page" means find the **right +> names**, not rescue wrong ones. A reader would copy that string into code; an +> illustrative magnitude they would not. + +--- + +## Procedure (per claim) + +1. **Identify the volatile assertion(s)** in the claim text — and when the claim + bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells + you what to check: + - `version` → model/API version, GA date, context window, max output, training cutoff + - `tpm` → tokens-per-minute / throughput / quota numbers + - `sku` → SKU name, tier, PTU minimums, deployment type + - `region` → regional availability + - `status` → GA / preview / retirement / deprecation status + - `taxonomy` → categorization, capability mapping, which-feature-does-what +2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's + `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not + address the assertion, run `microsoft_docs_search` to find the authoritative page. + **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak + cited URL is not the last word. +3. **Exact-value entailment check** each checkable value (and each load-bearing part + under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any, + govern the claim. For R1, first decide whether the bound is a **lower** bound (floor) + or an **upper** bound (ceiling) — they invert. +4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim + quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence + absence), the quote is the canonical enumeration in which the entity does **not** + appear — quote the enumeration and state the entity is absent. No quote → `source_silent`. + +## Hard rules + +- Verify against the fetched page only. Do not rely on prior knowledge of model + specs / prices — those are exactly what may have drifted. +- Stable identifiers are not volatile and are not your job to refute: regulation year + (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10 + 2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on + whatever volatile value it carries, else `source_silent`. +- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence. +- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the + verdict (empty string for `source_silent`). `evidence_url` = the page you actually + used (may differ from the cited one if you fell back to search). +- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty. + +## Batch to judge (from ``) + + + +## Output (strict JSON, no fence) + +``` +{"file":"","results":[ + {"id":"","judge_verdict":"grounded|not_grounded|source_silent","rule":"","evidence_url":"","evidence_quote":"","reason":""} +]} +```