diff --git a/docs/ref-kb-workflow-plan-2026-06.md b/docs/ref-kb-workflow-plan-2026-06.md index aacfd8d..f052374 100644 --- a/docs/ref-kb-workflow-plan-2026-06.md +++ b/docs/ref-kb-workflow-plan-2026-06.md @@ -150,6 +150,8 @@ Fase 0 ✅ lukket; gaten sa **BYGG Fase 3 (scoped)**. Gjenstående arbeid har ** > **FORHÅNDSREGISTRERT GATE (låst 2026-06-26, operatørvalg, FØR fan-out):** judge **recall ≥ 0,80 OG presisjon ≥ 0,70** (punktestimat), målt på den verifiserbare evaluerings-populasjonen P = volatil + fetchbar claim_type (price ekskl.) = **240 påstander, 38 positive**. PLUSS nødvendig betingelse: judge-recall > staleness-recall (staleness = 0/38). Wilson 95 %-bånd rapporteres som kontekst (n=38 → bredt bånd); grensetilfeller flagges for operatør, ikke mekanisk avvist. Strengt nivå valgt: bygg S3 KUN hvis judgen er svært sterk. **Harness (testet, 14 tester):** `lib/judge-bakeoff.mjs` + `extract-judge-claims.mjs` (blind manifest, 0 label-lekkasje) + `judge-claim-prompt.md` (blind per-påstands-judge) + `run-judge-bakeoff.mjs --min-recall 0.80 --min-precision 0.70`. Blindhet: judgen ser aldri gull-verdict; join på `id` i koden etterpå. +> **S1-v2-RESULTAT (2026-06-26, målrettet iterasjon — operatørvalg (c)): GATE ✅ PASS — recall 84,2 %, presisjon 84,2 %.** v2-prompt `judge-claim-prompt-v2.md` (eksakt-verdi-entailment, prinsipiell fiks på diagnostisert grounded-men-feil-feilmode; terskel uendret). Rapport: `judge-bakeoff-report-v2.{json,md}`; resultater: `judge-bakeoff-results-v2.json`. **Judge: recall 84,2 % (32/38, ✅ ≥0,80, Wilson [69,6–92,6 %]), presisjon 84,2 % (✅ ≥0,70), F1 0,842, slår staleness 0/38.** Fiksen løste målet: **sku 37,5 %→75,0 %, taxonomy 66,7 %→100 % recall**; +6 ekte fangster (26→32) UTEN netto nye FP (6→6) ⇒ recall OG presisjon opp samtidig (mekanistisk koherent, ikke støy). **Ærlige forbehold:** (1) andre måling på samme frosne gull-sett etter v1 (åpent erkjent; v1 frosset); (2) Wilson nedre grense 69,6 % < 0,80 (n=38) ⇒ sann recall ≥0,80 ikke statistisk garantert; (3) region 50 % (n=2) for lite. **Gate-logikk ⇒ vei mot S3 (scoped/hybrid).** NESTE OPERATØR-BESLUTNING: gå til S2 (type-tag) + S3 (backfill/frontmatter)? (stoppet her — eskalerer ikke selv.) + > **S1-RESULTAT (2026-06-26, blind fan-out 15 batcher Opus xhigh, 255 påstander dekket): GATE ❌ FAIL — recall 68,4 % < 0,80.** Rapport: `scripts/kb-eval/data/judge-bakeoff-report.{json,md}`; resultater: `judge-bakeoff-results.json`. **Judge: presisjon 81,3 % (✅ ≥0,70), recall 68,4 % (❌ <0,80, Wilson [52,5–80,9 %]), fanget 26/38, slår staleness 0/38 desisivt.** Recall-draget er **konsentrert**: sku 37,5 % (3/8) + taxonomy 66,7 % + region 50 % (n=2); version/status/tpm er allerede 80–86 % recall ved 75–100 % presisjon. 10 av 12 bommer var «grounded»-men-feil (brittle claim-dekomponering, isolert til sku/taxonomy). Per pre-registrert gate ⇒ **STOPP, bygg IKKE S3.** **ÅPEN FORK (operatør):** (a) honorer FAIL — fall tilbake på staleness + operatør-gating [ren pre-registrering]; (b) scoped judge — auto-flag kun sterke typer (version/status/tpm), operatør-gate sku/taxonomy [data-støttet mellomvei]; (c) én målrettet prompt-iterasjon på sku/taxonomy + re-kjør [flagg p-hacking-risiko: frys v1 som ærlig pre-registrert utfall]. **S2 — Fase 2: minimal type-tag (judge-uavhengig, nyttig uansett).** Klassifiser ~389 filer `reference|template|methodology|regulatory` (sidecar-manifest el. mappekonvensjon — IKKE full YAML ennå). Skiller de ~83 kildeløse legitimt (mal/metodikk: `decision-trees`, `cost-models`) fra MS-fakta-uten-kilde. NY `scripts/kb-eval/classify-ref-type.mjs` (TDD-først). Kan kjøres før/parallelt med S1 (billig). **Scope:** klassifiser advisor-filer, men MUTÉR dem ikke (Cosmo-kollisjon). **Gate:** hver fil har én type; kildeløse delt i to bøtter; reproduserbar; suite grønn. diff --git a/scripts/kb-eval/data/judge-bakeoff-report-v2.json b/scripts/kb-eval/data/judge-bakeoff-report-v2.json new file mode 100644 index 0000000..1896d08 --- /dev/null +++ b/scripts/kb-eval/data/judge-bakeoff-report-v2.json @@ -0,0 +1,234 @@ +{ + "_meta": { + "source": "gold-correctness-set.json + judge-bakeoff-results.json", + "thresholds": { + "minRecall": 0.8, + "minPrecision": 0.7 + }, + "judged": 255 + }, + "population": { + "total": 255, + "verifiable": 240, + "positives": 38, + "negatives": 202, + "unsourcedInP": 15 + }, + "arms": { + "staleness": { + "tp": 0, + "fp": 0, + "fn": 38, + "tn": 202, + "positives": 38, + "negatives": 202, + "flagged": 0, + "precision": null, + "recall": 0, + "f1": null, + "recallWilson": { + "p": 0, + "low": 0, + "high": 0.09181293258383999 + }, + "precisionWilson": null + }, + "judge": { + "tp": 32, + "fp": 6, + "fn": 6, + "tn": 196, + "positives": 38, + "negatives": 202, + "flagged": 38, + "precision": 0.8421052631578947, + "recall": 0.8421052631578947, + "f1": 0.8421052631578947, + "recallWilson": { + "p": 0.8421052631578947, + "low": 0.6958287736272311, + "high": 0.9255623777627731 + }, + "precisionWilson": { + "p": 0.8421052631578947, + "low": 0.6958287736272311, + "high": 0.9255623777627731 + } + }, + "hybrid": { + "tp": 32, + "fp": 6, + "fn": 6, + "tn": 196, + "positives": 38, + "negatives": 202, + "flagged": 38, + "precision": 0.8421052631578947, + "recall": 0.8421052631578947, + "f1": 0.8421052631578947, + "recallWilson": { + "p": 0.8421052631578947, + "low": 0.6958287736272311, + "high": 0.9255623777627731 + }, + "precisionWilson": { + "p": 0.8421052631578947, + "low": 0.6958287736272311, + "high": 0.9255623777627731 + } + } + }, + "sourceSilent": { + "onVerifiableNegative": 3, + "onVerifiableError": 2, + "agreesWithUnsourced": 5, + "disagreesWithUnsourced": 10 + }, + "byClaimType": { + "version": { + "tp": 6, + "fp": 0, + "fn": 1, + "tn": 21, + "positives": 7, + "negatives": 21, + "flagged": 6, + "precision": 1, + "recall": 0.8571428571428571, + "f1": 0.923076923076923, + "recallWilson": { + "p": 0.8571428571428571, + "low": 0.4868654966809701, + "high": 0.9743210440510252 + }, + "precisionWilson": { + "p": 1, + "low": 0.6096569663469354, + "high": 0.9999999999999999 + } + }, + "tpm": { + "tp": 4, + "fp": 0, + "fn": 1, + "tn": 20, + "positives": 5, + "negatives": 20, + "flagged": 4, + "precision": 1, + "recall": 0.8, + "f1": 0.888888888888889, + "recallWilson": { + "p": 0.8, + "low": 0.3755282641185388, + "high": 0.9637768390302125 + }, + "precisionWilson": { + "p": 1, + "low": 0.5100999795960008, + "high": 1 + } + }, + "region": { + "tp": 1, + "fp": 0, + "fn": 1, + "tn": 13, + "positives": 2, + "negatives": 13, + "flagged": 1, + "precision": 1, + "recall": 0.5, + "f1": 0.6666666666666666, + "recallWilson": { + "p": 0.5, + "low": 0.09452865480086614, + "high": 0.9054713451991339 + }, + "precisionWilson": { + "p": 1, + "low": 0.2065432914738929, + "high": 1 + } + }, + "status": { + "tp": 6, + "fp": 1, + "fn": 1, + "tn": 45, + "positives": 7, + "negatives": 46, + "flagged": 7, + "precision": 0.8571428571428571, + "recall": 0.8571428571428571, + "f1": 0.8571428571428571, + "recallWilson": { + "p": 0.8571428571428571, + "low": 0.4868654966809701, + "high": 0.9743210440510252 + }, + "precisionWilson": { + "p": 0.8571428571428571, + "low": 0.4868654966809701, + "high": 0.9743210440510252 + } + }, + "taxonomy": { + "tp": 9, + "fp": 5, + "fn": 0, + "tn": 83, + "positives": 9, + "negatives": 88, + "flagged": 14, + "precision": 0.6428571428571429, + "recall": 1, + "f1": 0.782608695652174, + "recallWilson": { + "p": 1, + "low": 0.7008472464490407, + "high": 1 + }, + "precisionWilson": { + "p": 0.6428571428571429, + "low": 0.3876400468214041, + "high": 0.8365550926279728 + } + }, + "sku": { + "tp": 6, + "fp": 0, + "fn": 2, + "tn": 14, + "positives": 8, + "negatives": 14, + "flagged": 6, + "precision": 1, + "recall": 0.75, + "f1": 0.8571428571428571, + "recallWilson": { + "p": 0.75, + "low": 0.40926987910258916, + "high": 0.9285223111419724 + }, + "precisionWilson": { + "p": 1, + "low": 0.6096569663469354, + "high": 0.9999999999999999 + } + } + }, + "gate": { + "pass": true, + "recallOk": true, + "precisionOk": true, + "beatsStaleness": true, + "thresholds": { + "minRecall": 0.8, + "minPrecision": 0.7 + }, + "reasons": [ + "all criteria met" + ] + } +} diff --git a/scripts/kb-eval/data/judge-bakeoff-report-v2.md b/scripts/kb-eval/data/judge-bakeoff-report-v2.md new file mode 100644 index 0000000..130ef03 --- /dev/null +++ b/scripts/kb-eval/data/judge-bakeoff-report-v2.md @@ -0,0 +1,54 @@ +# Judge bake-off-rapport — S1 (Fase 3 de-risk) + +_Generert deterministisk av `run-judge-bakeoff.mjs` over `gold-correctness-set.json` + `judge-bakeoff-results.json`. Tall fra testet `lib/judge-bakeoff.mjs`. Ikke rediger for hånd — regenerer._ + +**Forhåndsregistrert gate (låst FØR fan-out):** recall ≥ 0.8, presisjon ≥ 0.7, OG judge-recall > staleness-recall. + +## Evaluerings-populasjon (P) + +Volatil stratum + fetchbare claim_types (price ekskludert) — der feilene bor; unngår «invertert leverage». + +| metrikk | verdi | +|---|---| +| P totalt | 255 | +| Verifiserbare (correct/outdated/wrong) | 240 | +| Positive (reelle feil å fange) | 38 | +| Negative (correct) | 202 | +| Unsourced i P (kjørt, men utenfor P/R) | 15 | + +## Arm-sammenligning (detektering over de 240 verifiserbare) + +| arm | TP | FP | FN | TN | presisjon | recall | recall Wilson 95% | F1 | +|---|---|---|---|---|---|---|---|---| +| staleness (billig baseline) | 0 | 0 | 38 | 202 | n/a | 0.0% | [0.0%, 9.2%] | n/a | +| judge (per-påstand groundedness) | 32 | 6 | 6 | 196 | 84.2% | 84.2% | [69.6%, 92.6%] | 0.842 | +| hybrid (union) | 32 | 6 | 6 | 196 | 84.2% | 84.2% | [69.6%, 92.6%] | 0.842 | + +## Judge per claim_type (verifiserbar delmengde) + +| claim_type | positive | TP | FP | FN | presisjon | recall | +|---|---|---|---|---|---|---| +| taxonomy | 9 | 9 | 5 | 0 | 64.3% | 100.0% | +| sku | 8 | 6 | 0 | 2 | 100.0% | 75.0% | +| version | 7 | 6 | 0 | 1 | 100.0% | 85.7% | +| status | 7 | 6 | 1 | 1 | 85.7% | 85.7% | +| tpm | 5 | 4 | 0 | 1 | 100.0% | 80.0% | +| region | 2 | 1 | 0 | 1 | 100.0% | 50.0% | + +## source_silent-diagnostikk + +Judgen hentet siden men fant ikke verdien. Diagnostisk, ikke et flagg. + +| signal | antall | tolkning | +|---|---|---| +| På verifiserbar feil | 2 | judge-bom: reell feil oversett via «kan ikke verifisere» | +| På verifiserbar correct | 3 | judge reproduserte ikke et korrekt faktum mennesket fant | +| Enig med unsourced | 5 | judge reproduserer den uverifiserbare grensen (godt) | +| Uenig med unsourced | 10 | judge hevdet grunnet/ugrunnet der mennesket ikke fant kilde | + +## GATE: ✅ PASS — bygg S3 + +- recall 0.842 ≥ 0.8? **ja** +- presisjon 0.842 ≥ 0.7? **ja** +- slår staleness (recall 0.000)? **ja** +- begrunnelse: all criteria met diff --git a/scripts/kb-eval/data/judge-bakeoff-results-v2.json b/scripts/kb-eval/data/judge-bakeoff-results-v2.json new file mode 100644 index 0000000..bccbda1 --- /dev/null +++ b/scripts/kb-eval/data/judge-bakeoff-results-v2.json @@ -0,0 +1,1030 @@ +{ + "_meta": { + "source": "BLIND judge fan-out v2 (targeted iteration, judge-claim-prompt-v2.md exact-value rule), Opus 4.8 xhigh", + "note": "v1 frozen as pre-registered result; v2 is a transparent second attempt on the same frozen gold set", + "population": "volatile + fetchable claim_type (price excluded)", + "claim_count": 255 + }, + "results": [ + { + "id": "ms-ai-advisor/platforms/azure-ai-foundry.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/platforms/azure-ai-foundry.md#2", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-advisor/platforms/azure-ai-foundry.md#3", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/platforms/azure-ai-foundry.md#4", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/platforms/azure-ai-foundry.md#5", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/platforms/azure-ai-foundry.md#6", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-advisor/platforms/azure-ai-foundry.md#7", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-advisor/platforms/azure-ai-foundry.md#8", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/mlops-genaiops/llm-evaluation-production.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/mlops-genaiops/llm-evaluation-production.md#2", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/mlops-genaiops/llm-evaluation-production.md#3", + "judge_verdict": "source_silent" + }, + { + "id": "ms-ai-engineering/mlops-genaiops/llm-evaluation-production.md#4", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/mlops-genaiops/llm-evaluation-production.md#5", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/mlops-genaiops/llm-evaluation-production.md#6", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/mlops-genaiops/llm-evaluation-production.md#7", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-engineering/mlops-genaiops/llm-evaluation-production.md#8", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/mlops-genaiops/llm-evaluation-production.md#9", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/mlops-genaiops/llm-evaluation-production.md#10", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/platforms/model-catalog-2026.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/platforms/model-catalog-2026.md#2", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/platforms/model-catalog-2026.md#3", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/platforms/model-catalog-2026.md#4", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-advisor/platforms/model-catalog-2026.md#5", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/platforms/model-catalog-2026.md#6", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-advisor/platforms/model-catalog-2026.md#7", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-advisor/platforms/model-catalog-2026.md#8", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/platforms/model-catalog-2026.md#9", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/api-management/multi-region-ai-gateway-design.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/api-management/multi-region-ai-gateway-design.md#2", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-engineering/api-management/multi-region-ai-gateway-design.md#3", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/api-management/multi-region-ai-gateway-design.md#4", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/api-management/multi-region-ai-gateway-design.md#5", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/api-management/multi-region-ai-gateway-design.md#6", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/api-management/multi-region-ai-gateway-design.md#7", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/api-management/multi-region-ai-gateway-design.md#8", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/api-management/multi-region-ai-gateway-design.md#9", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/prompt-engineering/multimodal-prompt-design.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/prompt-engineering/multimodal-prompt-design.md#2", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/prompt-engineering/multimodal-prompt-design.md#3", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/prompt-engineering/multimodal-prompt-design.md#4", + "judge_verdict": "source_silent" + }, + { + "id": "ms-ai-advisor/prompt-engineering/multimodal-prompt-design.md#5", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/prompt-engineering/multimodal-prompt-design.md#6", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/prompt-engineering/multimodal-prompt-design.md#7", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-advisor/prompt-engineering/multimodal-prompt-design.md#8", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-governance/responsible-ai/algorithmic-accountability-auditability.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-infrastructure/bcdr/ai-foundry-disaster-recovery-planning.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-infrastructure/bcdr/ai-foundry-disaster-recovery-planning.md#2", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-infrastructure/bcdr/ai-foundry-disaster-recovery-planning.md#3", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-infrastructure/bcdr/ai-foundry-disaster-recovery-planning.md#4", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-infrastructure/bcdr/ai-foundry-disaster-recovery-planning.md#5", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-infrastructure/bcdr/ai-foundry-disaster-recovery-planning.md#6", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-infrastructure/bcdr/ai-foundry-disaster-recovery-planning.md#7", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-infrastructure/bcdr/ai-foundry-disaster-recovery-planning.md#8", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-infrastructure/bcdr/ai-foundry-disaster-recovery-planning.md#9", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-advisor/architecture/regional-availability-verification.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/architecture/regional-availability-verification.md#2", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/architecture/regional-availability-verification.md#3", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/architecture/regional-availability-verification.md#4", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/architecture/regional-availability-verification.md#5", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/architecture/regional-availability-verification.md#6", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/architecture/regional-availability-verification.md#7", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/architecture/regional-availability-verification.md#8", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-engineering/rag-architecture/rag-cost-optimization.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/rag-architecture/rag-cost-optimization.md#2", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/rag-architecture/rag-cost-optimization.md#3", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/rag-architecture/rag-cost-optimization.md#4", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/rag-architecture/rag-cost-optimization.md#5", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-engineering/rag-architecture/rag-cost-optimization.md#6", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-engineering/rag-architecture/rag-cost-optimization.md#7", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/rag-architecture/rag-cost-optimization.md#8", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-advisor/architecture/cost-models.md#3", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/architecture/cost-models.md#6", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/azure-ai-services/ai-services-enterprise-architecture.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/azure-ai-services/ai-services-enterprise-architecture.md#2", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/azure-ai-services/ai-services-enterprise-architecture.md#3", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/azure-ai-services/ai-services-enterprise-architecture.md#4", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/azure-ai-services/ai-services-enterprise-architecture.md#5", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/azure-ai-services/ai-services-enterprise-architecture.md#6", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/azure-ai-services/ai-services-enterprise-architecture.md#7", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/azure-ai-services/ai-services-enterprise-architecture.md#8", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-engineering/mlops-genaiops/genaiops-llm-specific-practices.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/mlops-genaiops/genaiops-llm-specific-practices.md#2", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-governance/monitoring-observability/token-usage-tracking-attribution.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-governance/monitoring-observability/token-usage-tracking-attribution.md#2", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-governance/monitoring-observability/token-usage-tracking-attribution.md#3", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-governance/monitoring-observability/token-usage-tracking-attribution.md#4", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-governance/monitoring-observability/token-usage-tracking-attribution.md#5", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-governance/monitoring-observability/token-usage-tracking-attribution.md#6", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-governance/monitoring-observability/token-usage-tracking-attribution.md#7", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-governance/monitoring-observability/token-usage-tracking-attribution.md#8", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-advisor/prompt-engineering/reasoning-models-o1-o3-optimization.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/prompt-engineering/reasoning-models-o1-o3-optimization.md#2", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/prompt-engineering/reasoning-models-o1-o3-optimization.md#3", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/prompt-engineering/reasoning-models-o1-o3-optimization.md#4", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/prompt-engineering/reasoning-models-o1-o3-optimization.md#5", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/prompt-engineering/reasoning-models-o1-o3-optimization.md#6", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/prompt-engineering/reasoning-models-o1-o3-optimization.md#7", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-advisor/architecture/adr-template.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/architecture/adr-template.md#2", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/architecture/adr-template.md#3", + "judge_verdict": "source_silent" + }, + { + "id": "ms-ai-security/cost-optimization/reserved-capacity-planning.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/reserved-capacity-planning.md#2", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/reserved-capacity-planning.md#3", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/reserved-capacity-planning.md#4", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/reserved-capacity-planning.md#5", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/reserved-capacity-planning.md#6", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-security/cost-optimization/reserved-capacity-planning.md#7", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/prompt-engineering/real-time-reasoning-performance.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/prompt-engineering/real-time-reasoning-performance.md#2", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/prompt-engineering/real-time-reasoning-performance.md#3", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/prompt-engineering/real-time-reasoning-performance.md#4", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/prompt-engineering/real-time-reasoning-performance.md#5", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/prompt-engineering/real-time-reasoning-performance.md#6", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-advisor/prompt-engineering/real-time-reasoning-performance.md#7", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/prompt-engineering/token-optimization-and-efficiency.md#3", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/prompt-engineering/token-optimization-and-efficiency.md#4", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/prompt-engineering/token-optimization-and-efficiency.md#5", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/prompt-engineering/token-optimization-and-efficiency.md#7", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/rag-architecture/rag-context-windows.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/rag-architecture/rag-context-windows.md#2", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/rag-architecture/rag-context-windows.md#3", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-engineering/rag-architecture/rag-context-windows.md#4", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/rag-architecture/rag-context-windows.md#5", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-engineering/rag-architecture/rag-context-windows.md#6", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/rag-architecture/rag-context-windows.md#7", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/mlops-genaiops/monitoring-observability-ml-systems.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/mlops-genaiops/monitoring-observability-ml-systems.md#2", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/mlops-genaiops/monitoring-observability-ml-systems.md#3", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/mlops-genaiops/monitoring-observability-ml-systems.md#4", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/mlops-genaiops/monitoring-observability-ml-systems.md#5", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/mlops-genaiops/monitoring-observability-ml-systems.md#6", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/mlops-genaiops/monitoring-observability-ml-systems.md#7", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-infrastructure/bcdr/multi-region-azure-openai-deployment.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-infrastructure/bcdr/multi-region-azure-openai-deployment.md#2", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-infrastructure/bcdr/multi-region-azure-openai-deployment.md#3", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-infrastructure/bcdr/multi-region-azure-openai-deployment.md#4", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-infrastructure/bcdr/multi-region-azure-openai-deployment.md#5", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-infrastructure/bcdr/multi-region-azure-openai-deployment.md#6", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-infrastructure/bcdr/multi-region-azure-openai-deployment.md#7", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-infrastructure/bcdr/capacity-planning-dr-configurations.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-infrastructure/bcdr/capacity-planning-dr-configurations.md#2", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-infrastructure/bcdr/capacity-planning-dr-configurations.md#3", + "judge_verdict": "source_silent" + }, + { + "id": "ms-ai-infrastructure/bcdr/capacity-planning-dr-configurations.md#5", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-infrastructure/bcdr/network-resilience-patterns-ai.md#1", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-infrastructure/bcdr/network-resilience-patterns-ai.md#2", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-infrastructure/bcdr/network-resilience-patterns-ai.md#3", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-infrastructure/bcdr/network-resilience-patterns-ai.md#4", + "judge_verdict": "source_silent" + }, + { + "id": "ms-ai-infrastructure/bcdr/service-level-documentation-dr.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-infrastructure/bcdr/service-level-documentation-dr.md#2", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-infrastructure/bcdr/service-level-documentation-dr.md#3", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-infrastructure/bcdr/service-level-documentation-dr.md#4", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-infrastructure/bcdr/service-level-documentation-dr.md#5", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-infrastructure/bcdr/service-level-documentation-dr.md#6", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-infrastructure/bcdr/service-level-documentation-dr.md#7", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/multi-model-strategy-costs.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/multi-model-strategy-costs.md#2", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-security/cost-optimization/multi-model-strategy-costs.md#3", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-security/cost-optimization/multi-model-strategy-costs.md#4", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/multi-model-strategy-costs.md#5", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/multi-model-strategy-costs.md#6", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/multi-model-strategy-costs.md#8", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/gpt5-gpt41-pricing-models.md#4", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/gpt5-gpt41-pricing-models.md#5", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/gpt5-gpt41-pricing-models.md#6", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/gpt5-gpt41-pricing-models.md#7", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/azure-ai-foundry-cost-governance.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/azure-ai-foundry-cost-governance.md#2", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/azure-ai-foundry-cost-governance.md#3", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/azure-ai-foundry-cost-governance.md#4", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/azure-ai-foundry-cost-governance.md#5", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/azure-ai-foundry-cost-governance.md#6", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-security/cost-optimization/azure-ai-foundry-cost-governance.md#7", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/vector-storage-cost-optimization.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/vector-storage-cost-optimization.md#2", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/vector-storage-cost-optimization.md#3", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/vector-storage-cost-optimization.md#4", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/vector-storage-cost-optimization.md#5", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-security/cost-optimization/vector-storage-cost-optimization.md#6", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/vector-storage-cost-optimization.md#7", + "judge_verdict": "source_silent" + }, + { + "id": "ms-ai-engineering/azure-ai-services/ai-services-cost-optimization.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/azure-ai-services/ai-services-cost-optimization.md#2", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/azure-ai-services/ai-services-cost-optimization.md#3", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/azure-ai-services/ai-services-cost-optimization.md#5", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-engineering/azure-ai-services/ai-services-cost-optimization.md#6", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/azure-ai-services/ai-services-cost-optimization.md#8", + "judge_verdict": "source_silent" + }, + { + "id": "ms-ai-engineering/azure-ai-services/ai-services-vs-foundry-tools-selection.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/azure-ai-services/ai-services-vs-foundry-tools-selection.md#3", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-engineering/azure-ai-services/ai-services-vs-foundry-tools-selection.md#4", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/azure-ai-services/ai-services-vs-foundry-tools-selection.md#5", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-engineering/azure-ai-services/ai-services-vs-foundry-tools-selection.md#7", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/small-language-models-economics.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/small-language-models-economics.md#2", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/small-language-models-economics.md#3", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/small-language-models-economics.md#4", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/small-language-models-economics.md#5", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/small-language-models-economics.md#6", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/small-language-models-economics.md#9", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/rag-architecture/embedding-models-selection.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-engineering/rag-architecture/embedding-models-selection.md#2", + "judge_verdict": "source_silent" + }, + { + "id": "ms-ai-engineering/rag-architecture/embedding-models-selection.md#4", + "judge_verdict": "source_silent" + }, + { + "id": "ms-ai-engineering/rag-architecture/embedding-models-selection.md#5", + "judge_verdict": "source_silent" + }, + { + "id": "ms-ai-engineering/rag-architecture/embedding-models-selection.md#6", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-engineering/rag-architecture/embedding-models-selection.md#7", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-governance/monitoring-observability/endpoint-health-and-capacity-planning.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-governance/monitoring-observability/endpoint-health-and-capacity-planning.md#2", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-governance/monitoring-observability/endpoint-health-and-capacity-planning.md#3", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-governance/monitoring-observability/endpoint-health-and-capacity-planning.md#4", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-governance/monitoring-observability/endpoint-health-and-capacity-planning.md#5", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-governance/monitoring-observability/endpoint-health-and-capacity-planning.md#6", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-governance/monitoring-observability/response-quality-metrics-rag.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-governance/monitoring-observability/response-quality-metrics-rag.md#2", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-governance/monitoring-observability/response-quality-metrics-rag.md#3", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-governance/monitoring-observability/response-quality-metrics-rag.md#4", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-governance/monitoring-observability/response-quality-metrics-rag.md#5", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-governance/monitoring-observability/response-quality-metrics-rag.md#6", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-governance/monitoring-observability/cost-monitoring-cost-attribution.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-governance/monitoring-observability/cost-monitoring-cost-attribution.md#2", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-governance/monitoring-observability/cost-monitoring-cost-attribution.md#3", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-governance/monitoring-observability/cost-monitoring-cost-attribution.md#4", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-governance/monitoring-observability/cost-monitoring-cost-attribution.md#7", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-infrastructure/bcdr/cost-analysis-dr-configurations.md#3", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/ptu-vs-paygo-economics.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/ptu-vs-paygo-economics.md#2", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/ptu-vs-paygo-economics.md#3", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/ptu-vs-paygo-economics.md#4", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/ptu-vs-paygo-economics.md#5", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/ptu-vs-paygo-economics.md#6", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/ai-builder-credits-transition.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/ai-builder-credits-transition.md#2", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/ai-builder-credits-transition.md#3", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/ai-builder-credits-transition.md#6", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/ai-builder-credits-transition.md#7", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/ai-builder-credits-transition.md#8", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/prompt-engineering/chain-of-thought-prompting.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/prompt-engineering/chain-of-thought-prompting.md#2", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-advisor/prompt-engineering/chain-of-thought-prompting.md#3", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-governance/responsible-ai/stakeholder-communication-ai-decisions.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-governance/responsible-ai/stakeholder-communication-ai-decisions.md#3", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-security/cost-optimization/model-selection-price-performance.md#4", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/model-selection-price-performance.md#6", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/model-selection-price-performance.md#7", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/model-selection-price-performance.md#8", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-security/cost-optimization/model-selection-price-performance.md#9", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-security/cost-optimization/token-counting-optimization.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/token-counting-optimization.md#2", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/token-counting-optimization.md#4", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/token-counting-optimization.md#8", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/rag-query-cost-reduction.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/rag-query-cost-reduction.md#2", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/rag-query-cost-reduction.md#4", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/rag-query-cost-reduction.md#7", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-governance/responsible-ai/responsible-ai-training-awareness.md#1", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-governance/responsible-ai/responsible-ai-training-awareness.md#2", + "judge_verdict": "not_grounded" + }, + { + "id": "ms-ai-security/cost-optimization/deterministic-cost-calculation-model.md#9", + "judge_verdict": "grounded" + }, + { + "id": "ms-ai-security/cost-optimization/reserved-capacity-planning.md#8", + "judge_verdict": "grounded" + } + ] +} diff --git a/scripts/kb-eval/judge-claim-prompt-v2.md b/scripts/kb-eval/judge-claim-prompt-v2.md new file mode 100644 index 0000000..df1a5cc --- /dev/null +++ b/scripts/kb-eval/judge-claim-prompt-v2.md @@ -0,0 +1,110 @@ +# Per-claim groundedness judge — S1 bake-off **v2** (targeted iteration) + +v2 of `judge-claim-prompt.md`. Same blind, per-claim, one-subagent-per-file design. +**Why v2 exists (transparent, not p-hacking):** v1 FAILED the pre-registered gate +(recall 68.4%, frozen as the honest result). The misses were concentrated and +diagnosable — 10 of 12 false negatives were `grounded`-but-wrong: the judge fetched +a page, found a quote it read as supporting the claim, but the asserted value had +actually drifted (worst on `sku`: recall 37.5%). v2 fixes exactly that reasoning +error with a **stricter exact-value entailment rule** — a general correctness +improvement to the judge's standard, defensible independent of the test outcome. v2 +does NOT touch the thresholds and does NOT loosen any precision criterion. + +The v1 result stays frozen (`judge-bakeoff-results.json`, `...-report.*`). v2 writes +to `judge-bakeoff-results-v2.json` and is graded against the same frozen gold set. + +--- + +You are a correctness judge for Microsoft AI reference documentation. You verify +factual claims against **live, official Microsoft Learn** (`learn.microsoft.com`). +Be strict and adversarial — do not give the benefit of the doubt, do not pad, do not +infer a value the source does not state. + +You are judging claims extracted from ``. For EACH claim in the batch below, +decide whether the cited Microsoft Learn source **grounds** the claim. + +## The three verdicts (exhaustive, mutually exclusive) + +- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed + value(s). The page supports the claim. (Maps to gold `correct`.) +- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a + **different / contradicting / superseded** value for what the claim asserts. The + claim disagrees with the source. (Maps to gold `outdated` + `wrong`.) +- **`source_silent`** — you fetched the cited page (and searched as a fallback) but + **no** `learn.microsoft.com` page states the claimed value at all. You cannot + confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure + pages typically lands here — that is expected, not a failure. + +## ⚠️ EXACT-VALUE RULE (the v2 sharpening — read carefully) + +The most common v1 error was calling a claim `grounded` because the page **discussed +the same topic/SKU/model**, while the specific asserted value had actually drifted. +Fix that: + +- A claim is `grounded` ONLY if the fetched page states the **exact** asserted + value(s). Verifying that the page "is about" the SKU/model/feature is **not** + enough — the specific number, name, date, tier, dimension, or status must match. +- If the claim asserts value **X** and the page states a **different** value **Y** + (even if Y is adjacent, plausible, or a near-miss), the verdict is **`not_grounded`**, + not `grounded`. Do not round, approximate, or accept "close enough." +- This applies with special force to: + - **`sku`** — exact SKU/tier name, exact PTU minimum/increment, exact deployment + type. A different SKU value on the page = `not_grounded`. + - **`taxonomy`** — the exact categorization/mapping. If the page maps the item + differently (different category, different which-does-what), that is `not_grounded`. + - **`version` / `tpm` / `region` / `status`** — exact date/number/region/GA-preview + status. A superseded date or changed number is `not_grounded`. +- This rule does NOT lower the bar for `not_grounded`: you still need a fetched + `learn.microsoft.com` quote that states the **differing** value. "I couldn't find + the value" remains `source_silent`, never `not_grounded`. + +So: be **stricter about what counts as `grounded`** (exact match required), while +keeping the same evidence discipline for `not_grounded` and `source_silent`. + +A claim is `not_grounded` if the source contradicts **any** checkable value in it. +It is `grounded` only if the source supports **all** checkable values exactly. If the +source states none of them, it is `source_silent`. + +## Procedure (per claim) + +1. **Identify the volatile assertion(s)** in the claim text. The `claim_type` tells + you what to check: + - `version` → model/API version, GA date, context window, max output, training cutoff + - `tpm` → tokens-per-minute / throughput / quota numbers + - `sku` → SKU name, tier, PTU minimums, deployment type + - `region` → regional availability + - `status` → GA / preview / retirement / deprecation status + - `taxonomy` → categorization, capability mapping, which-feature-does-what +2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's + `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not + address the assertion, run `microsoft_docs_search` to find the authoritative page. +3. **Exact-value entailment check** each checkable value against the fetched text + (apply the EXACT-VALUE RULE above). +4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a + verbatim quote you actually fetched from a `learn.microsoft.com` URL that states + the relevant value. No quote → `source_silent`. Never quote from memory. + +## Hard rules + +- Verify against the fetched page only. Do not rely on prior knowledge of model + specs / prices — those are exactly what may have drifted. +- Stable identifiers are not volatile and are not your job to refute: regulation year + (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10 + 2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on + whatever volatile value it carries, else `source_silent`. +- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence. +- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the + verdict (empty string for `source_silent`). `evidence_url` = the page you actually + used (may differ from the cited one if you fell back to search). + +## Batch to judge (from ``) + + + +## Output (strict JSON, no fence) + +``` +{"file":"","results":[ + {"id":"","judge_verdict":"grounded|not_grounded|source_silent","evidence_url":"","evidence_quote":"","reason":""} +]} +``` diff --git a/scripts/kb-eval/run-judge-bakeoff.mjs b/scripts/kb-eval/run-judge-bakeoff.mjs index 326a456..1faa6fc 100644 --- a/scripts/kb-eval/run-judge-bakeoff.mjs +++ b/scripts/kb-eval/run-judge-bakeoff.mjs @@ -36,7 +36,10 @@ if (!Number.isFinite(minRecall) || !Number.isFinite(minPrecision)) { const thresholds = { minRecall, minPrecision }; const gold = JSON.parse(fs.readFileSync(path.join(DATA, 'gold-correctness-set.json'), 'utf8')); -const resultsPath = path.join(DATA, 'judge-bakeoff-results.json'); +// --results / --report-prefix let a second iteration (v2) be graded without +// clobbering the frozen v1 artifacts. Defaults preserve the v1 file names. +const resultsPath = path.join(DATA, flag('--results') || 'judge-bakeoff-results.json'); +const reportPrefix = flag('--report-prefix') || 'judge-bakeoff-report'; if (!fs.existsSync(resultsPath)) { console.error(`error: ${resultsPath} not found — run the judge fan-out first`); process.exit(2); @@ -126,8 +129,8 @@ Judgen hentet siden men fant ikke verdien. Diagnostisk, ikke et flagg. `; if (process.argv.includes('--write')) { - const jsonOut = path.join(DATA, 'judge-bakeoff-report.json'); - const mdOut = path.join(DATA, 'judge-bakeoff-report.md'); + const jsonOut = path.join(DATA, `${reportPrefix}.json`); + const mdOut = path.join(DATA, `${reportPrefix}.md`); fs.writeFileSync( jsonOut, JSON.stringify(