open/ms-ai-architect
1
0
Fork 0
Code Issues Pull requests Releases Activity

feat(ms-ai-architect): G5b gull-friskhets-spot-sjekk LUKKET — 4 v3-FP re-adjudert mot live, ALLE stale gull (v3 flagget korrekt), baseline løftet v3 P89.7/R92.1 → P100/R92.9/0FP; v3.1 forfattet (ren recall-hardning, FP-vakt droppet) [skip-docs]

Browse source
This commit is contained in:
Kjell Tore Guttormsen 2026-06-30 13:43:52 +02:00
commit 045db566ba
8 changed files with 842 additions and 12 deletions
Download patch file Download diff file Expand all files Collapse all files

19
scripts/kb-eval/data/gold-correctness-set.json
View file

@ -8,7 +8,8 @@
"claim_count": 373,
"reconciliation_log": [
"2026-06-30 Spor 2b: 12 judge-vs-gold-uenigheter adjudert mot live; 4 gull-feil rettet + 1 note-fiks. Logg: docs/ref-kb-gold-reconciliation-2026-06.md.",
"2026-06-30 G5 friskhets-mikropass: 5 omstridte claims (gold=outdated, v3=grounded) re-adjudert mot live MS Learn. 2 stale gull rettet outdated->correct (genaiops-llm-specific-practices.md#2 '1600+' live 1900 tett nedre grense; model-selection-price-performance.md#8 Model Router GA nov 2025). 3 opprettholdt som outdated (judge-feil bekreftet: multi-model-strategy-costs.md#2, token-usage-tracking-attribution.md#3, ai-foundry-disaster-recovery-planning.md#9). Logg: docs/ref-kb-correctness-program-2026-06.md §8 G5."
"2026-06-30 G5 friskhets-mikropass: 5 omstridte claims (gold=outdated, v3=grounded) re-adjudert mot live MS Learn. 2 stale gull rettet outdated->correct (genaiops-llm-specific-practices.md#2 '1600+' live 1900 tett nedre grense; model-selection-price-performance.md#8 Model Router GA nov 2025). 3 opprettholdt som outdated (judge-feil bekreftet: multi-model-strategy-costs.md#2, token-usage-tracking-attribution.md#3, ai-foundry-disaster-recovery-planning.md#9). Logg: docs/ref-kb-correctness-program-2026-06.md §8 G5.",
"2026-06-30 G5b friskhets-spot-sjekk: de 4 v3-FP-claims (gold=correct, v3=not_grounded) re-adjudert mot live MS Learn (4 Opus-subagenter, blinde). ALLE 4 var stale gull (v3 flagget korrekt): adr-template.md#1 correct->wrong (zero permission management motsies, Graph connectors krever ACL); multi-region-azure-openai-deployment.md#2 correct->outdated (gpt-35-turbo retired); network-resilience-patterns-ai.md#4 correct->wrong (obligatorisk vs anbefalt); vector-storage-cost-optimization.md#7 correct->wrong (GA-dato 2024-07-01, ikke 2024-11-01-preview). v3 hadde 0 ekte FP. Logg: docs/ref-kb-correctness-program-2026-06.md §8 G5b."
]
},
"claims": [
@ -344,11 +345,11 @@
"stratum": "volatile",
"claim": "SharePoint Embedded/Graph Connectors: zero permission management, permissions respekteres automatisk",
"claim_type": "status",
"verdict": "correct",
"verdict": "wrong",
"evidence_url": "https://learn.microsoft.com/microsoft-365/copilot/extensibility/data-privacy-security",
"lastmod_changed": false,
"file_last_updated": "2026-06-24",
"notes": "Permission inheritance bekreftet."
"notes": "RECONCILED 2026-06-30 (G5b friskhets-spot-sjekk): correct->wrong. Live (m365/copilot/extensibility/data-privacy-security + graph/connecting-external-content-manage-items): \"You can manage permissions to view external items by associating an access control list (ACL)\"; hver externalItem MA ha ACL (ikke-Entra-brukere ma mappes til Entra). Del B (permissions respekteres automatisk ved grounding) stemmer, men del A \"zero permission management\" motsies - Graph connectors krever ACL-forfatting. Fil-fiks (Spor 0/1): fjern \"zero permission management\", behold permission-honoring. Judge not_grounded var korrekt. Confidence: medium (innsats-overdrivelse pa last-baerende del A)."
},
{
"id": "ms-ai-advisor/architecture/adr-template.md#2",
@ -3165,11 +3166,11 @@
"stratum": "volatile",
"claim": "Sweden Central sekundær, bred (gpt-4o, o1, gpt-35-turbo)",
"claim_type": "region",
"verdict": "correct",
"verdict": "outdated",
"evidence_url": "https://learn.microsoft.com/azure/foundry/foundry-models/concepts/models-sold-directly-by-azure-region-availability",
"lastmod_changed": false,
"file_last_updated": "2026-06-24",
"notes": "Sweden Central bred støtte bekreftet."
"notes": "RECONCILED 2026-06-30 (G5b friskhets-spot-sjekk): correct->outdated. Live (models-sold-directly-by-azure-region-availability + retirements): gpt-4o og o1 tilgjengelig i Sweden Central, men gpt-35-turbo er RETIRED (0301/0613 feb 2025; 0125/1106 fra sep 2025) og finnes ikke lenger i katalogen. Claim listet gpt-35-turbo som tilgjengelig - var sant, na utdatert. Fil-fiks (Spor 0/1): bytt gpt-35-turbo med gjeldende modell (gpt-4.1-mini/gpt-4o-mini). Judge not_grounded (R2 entitet-fravaer) var korrekt. Confidence: high."
},
{
"id": "ms-ai-infrastructure/bcdr/multi-region-azure-openai-deployment.md#3",
@ -3399,11 +3400,11 @@
"stratum": "volatile",
"claim": "Circuit Breaker + Retry exponential backoff obligatorisk for alle Azure AI API-kall",
"claim_type": "status",
"verdict": "correct",
"verdict": "wrong",
"evidence_url": "https://learn.microsoft.com/azure/foundry-classic/openai/how-to/quota",
"lastmod_changed": false,
"file_last_updated": "2026-06-24",
"notes": "Retry m/backoff + circuit breaker offisielt anbefalt; 'obligatorisk' sterk formulering."
"notes": "RECONCILED 2026-06-30 (G5b friskhets-spot-sjekk): correct->wrong. Live (foundry-classic/openai/how-to/quota): MS rammer dette som \"Rate limit best practices\" / \"recommended\", ikke obligatorisk; circuit breaker nevnes kun som valgfritt avansert Polly-monster. Claim \"obligatorisk for alle Azure AI API-kall\" overdriver modaliteten (anbefalt -> palagt) og omfanget (alle kall). Monstrene er reelle MS-anbefalinger, men \"obligatorisk\" er ikke grunnet. Fil-fiks (Spor 0/1): bytt \"obligatorisk\" med \"anbefalt\". Judge not_grounded (R6) var korrekt. Confidence: medium (modalitets-overdrivelse)."
},
{
"id": "ms-ai-infrastructure/bcdr/service-level-documentation-dr.md#1",
@ -4533,11 +4534,11 @@
"stratum": "volatile",
"claim": "Vector quantization GA siden 2024-11-01",
"claim_type": "status",
"verdict": "correct",
"verdict": "wrong",
"evidence_url": "https://learn.microsoft.com/azure/search/vector-search-index-size",
"lastmod_changed": false,
"file_last_updated": "2026-06-19",
"notes": "GA-status bekreftet."
"notes": "RECONCILED 2026-06-30 (G5b friskhets-spot-sjekk): correct->wrong. Live (search-api-migration + vector-search-index-size): vector quantization ER GA, MEN GA-dato var 2024-07-01 (stable release), ikke 2024-11-01. \"2024-11-01\" finnes kun som preview-API-versjon (2024-11-01-preview). Last-baerende dato er feil. Fil-fiks (Spor 0/1): rett GA-dato til 2024-07-01. Judge not_grounded var korrekt. Confidence: high."
},
{
"id": "ms-ai-security/cost-optimization/vector-storage-cost-optimization.md#8",

234
scripts/kb-eval/data/judge-bakeoff-report-v2-g5bgold.json Normal file
View file

@ -0,0 +1,234 @@
{
"_meta": {
"source": "gold-correctness-set.json + judge-bakeoff-results.json",
"thresholds": {
"minRecall": 0.7,
"minPrecision": 0.6
},
"judged": 255
},
"population": {
"total": 255,
"verifiable": 240,
"positives": 42,
"negatives": 198,
"unsourcedInP": 15
},
"arms": {
"staleness": {
"tp": 0,
"fp": 0,
"fn": 42,
"tn": 198,
"positives": 42,
"negatives": 198,
"flagged": 0,
"precision": null,
"recall": 0,
"f1": null,
"recallWilson": {
"p": 0,
"low": 0,
"high": 0.08380161250916199
},
"precisionWilson": null
},
"judge": {
"tp": 33,
"fp": 5,
"fn": 9,
"tn": 193,
"positives": 42,
"negatives": 198,
"flagged": 38,
"precision": 0.868421052631579,
"recall": 0.7857142857142857,
"f1": 0.825,
"recallWilson": {
"p": 0.7857142857142857,
"low": 0.6405986210195627,
"high": 0.8829433146894876
},
"precisionWilson": {
"p": 0.868421052631579,
"low": 0.7267282994850112,
"high": 0.9424621712426856
}
},
"hybrid": {
"tp": 33,
"fp": 5,
"fn": 9,
"tn": 193,
"positives": 42,
"negatives": 198,
"flagged": 38,
"precision": 0.868421052631579,
"recall": 0.7857142857142857,
"f1": 0.825,
"recallWilson": {
"p": 0.7857142857142857,
"low": 0.6405986210195627,
"high": 0.8829433146894876
},
"precisionWilson": {
"p": 0.868421052631579,
"low": 0.7267282994850112,
"high": 0.9424621712426856
}
}
},
"sourceSilent": {
"onVerifiableNegative": 1,
"onVerifiableError": 4,
"agreesWithUnsourced": 5,
"disagreesWithUnsourced": 10
},
"byClaimType": {
"version": {
"tp": 6,
"fp": 0,
"fn": 1,
"tn": 21,
"positives": 7,
"negatives": 21,
"flagged": 6,
"precision": 1,
"recall": 0.8571428571428571,
"f1": 0.923076923076923,
"recallWilson": {
"p": 0.8571428571428571,
"low": 0.4868654966809701,
"high": 0.9743210440510252
},
"precisionWilson": {
"p": 1,
"low": 0.6096569663469354,
"high": 0.9999999999999999
}
},
"tpm": {
"tp": 4,
"fp": 0,
"fn": 1,
"tn": 20,
"positives": 5,
"negatives": 20,
"flagged": 4,
"precision": 1,
"recall": 0.8,
"f1": 0.888888888888889,
"recallWilson": {
"p": 0.8,
"low": 0.3755282641185388,
"high": 0.9637768390302125
},
"precisionWilson": {
"p": 1,
"low": 0.5100999795960008,
"high": 1
}
},
"region": {
"tp": 1,
"fp": 0,
"fn": 1,
"tn": 13,
"positives": 2,
"negatives": 13,
"flagged": 1,
"precision": 1,
"recall": 0.5,
"f1": 0.6666666666666666,
"recallWilson": {
"p": 0.5,
"low": 0.09452865480086614,
"high": 0.9054713451991339
},
"precisionWilson": {
"p": 1,
"low": 0.2065432914738929,
"high": 1
}
},
"status": {
"tp": 6,
"fp": 1,
"fn": 4,
"tn": 42,
"positives": 10,
"negatives": 43,
"flagged": 7,
"precision": 0.8571428571428571,
"recall": 0.6,
"f1": 0.7058823529411764,
"recallWilson": {
"p": 0.6,
"low": 0.3126695474501863,
"high": 0.8318224187964902
},
"precisionWilson": {
"p": 0.8571428571428571,
"low": 0.4868654966809701,
"high": 0.9743210440510252
}
},
"taxonomy": {
"tp": 11,
"fp": 3,
"fn": 0,
"tn": 83,
"positives": 11,
"negatives": 86,
"flagged": 14,
"precision": 0.7857142857142857,
"recall": 1,
"f1": 0.88,
"recallWilson": {
"p": 1,
"low": 0.7411599827511859,
"high": 1
},
"precisionWilson": {
"p": 0.7857142857142857,
"low": 0.5241027622679172,
"high": 0.9242875166308363
}
},
"sku": {
"tp": 5,
"fp": 1,
"fn": 2,
"tn": 14,
"positives": 7,
"negatives": 15,
"flagged": 6,
"precision": 0.8333333333333334,
"recall": 0.7142857142857143,
"f1": 0.7692307692307692,
"recallWilson": {
"p": 0.7142857142857143,
"low": 0.35892909014821267,
"high": 0.9177828342909844
},
"precisionWilson": {
"p": 0.8333333333333334,
"low": 0.43649056343635395,
"high": 0.9699474141282697
}
}
},
"gate": {
"pass": true,
"recallOk": true,
"precisionOk": true,
"beatsStaleness": true,
"thresholds": {
"minRecall": 0.7,
"minPrecision": 0.6
},
"reasons": [
"all criteria met"
]
}
}

54
scripts/kb-eval/data/judge-bakeoff-report-v2-g5bgold.md Normal file
View file

@ -0,0 +1,54 @@
# Judge bake-off-rapport — S1 (Fase 3 de-risk)
_Generert deterministisk av `run-judge-bakeoff.mjs` over `gold-correctness-set.json` + `judge-bakeoff-results.json`. Tall fra testet `lib/judge-bakeoff.mjs`. Ikke rediger for hånd — regenerer._
**Forhåndsregistrert gate (låst FØR fan-out):** recall ≥ 0.7, presisjon ≥ 0.6, OG judge-recall > staleness-recall.
## Evaluerings-populasjon (P)
Volatil stratum + fetchbare claim_types (price ekskludert) — der feilene bor; unngår «invertert leverage».
| metrikk | verdi |
|---|---|
| P totalt | 255 |
| Verifiserbare (correct/outdated/wrong) | 240 |
| Positive (reelle feil å fange) | 42 |
| Negative (correct) | 198 |
| Unsourced i P (kjørt, men utenfor P/R) | 15 |
## Arm-sammenligning (detektering over de 240 verifiserbare)
| arm | TP | FP | FN | TN | presisjon | recall | recall Wilson 95% | F1 |
|---|---|---|---|---|---|---|---|---|
| staleness (billig baseline) | 0 | 0 | 42 | 198 | n/a | 0.0% | [0.0%, 8.4%] | n/a |
| judge (per-påstand groundedness) | 33 | 5 | 9 | 193 | 86.8% | 78.6% | [64.1%, 88.3%] | 0.825 |
| hybrid (union) | 33 | 5 | 9 | 193 | 86.8% | 78.6% | [64.1%, 88.3%] | 0.825 |
## Judge per claim_type (verifiserbar delmengde)
| claim_type | positive | TP | FP | FN | presisjon | recall |
|---|---|---|---|---|---|---|
| taxonomy | 11 | 11 | 3 | 0 | 78.6% | 100.0% |
| status | 10 | 6 | 1 | 4 | 85.7% | 60.0% |
| version | 7 | 6 | 0 | 1 | 100.0% | 85.7% |
| sku | 7 | 5 | 1 | 2 | 83.3% | 71.4% |
| tpm | 5 | 4 | 0 | 1 | 100.0% | 80.0% |
| region | 2 | 1 | 0 | 1 | 100.0% | 50.0% |
## source_silent-diagnostikk
Judgen hentet siden men fant ikke verdien. Diagnostisk, ikke et flagg.
| signal | antall | tolkning |
|---|---|---|
| På verifiserbar feil | 4 | judge-bom: reell feil oversett via «kan ikke verifisere» |
| På verifiserbar correct | 1 | judge reproduserte ikke et korrekt faktum mennesket fant |
| Enig med unsourced | 5 | judge reproduserer den uverifiserbare grensen (godt) |
| Uenig med unsourced | 10 | judge hevdet grunnet/ugrunnet der mennesket ikke fant kilde |
## GATE: ✅ PASS — bygg S3
- recall 0.786 ≥ 0.7? **ja**
- presisjon 0.868 ≥ 0.6? **ja**
- slår staleness (recall 0.000)? **ja**
- begrunnelse: all criteria met

234
scripts/kb-eval/data/judge-bakeoff-report-v3-g5bgold.json Normal file
View file

@ -0,0 +1,234 @@
{
"_meta": {
"source": "gold-correctness-set.json + judge-bakeoff-results.json",
"thresholds": {
"minRecall": 0.7,
"minPrecision": 0.6
},
"judged": 255
},
"population": {
"total": 255,
"verifiable": 240,
"positives": 42,
"negatives": 198,
"unsourcedInP": 15
},
"arms": {
"staleness": {
"tp": 0,
"fp": 0,
"fn": 42,
"tn": 198,
"positives": 42,
"negatives": 198,
"flagged": 0,
"precision": null,
"recall": 0,
"f1": null,
"recallWilson": {
"p": 0,
"low": 0,
"high": 0.08380161250916199
},
"precisionWilson": null
},
"judge": {
"tp": 39,
"fp": 0,
"fn": 3,
"tn": 198,
"positives": 42,
"negatives": 198,
"flagged": 39,
"precision": 1,
"recall": 0.9285714285714286,
"f1": 0.962962962962963,
"recallWilson": {
"p": 0.9285714285714286,
"low": 0.8099028671147483,
"high": 0.9754100364488272
},
"precisionWilson": {
"p": 1,
"low": 0.9103301463997611,
"high": 1
}
},
"hybrid": {
"tp": 39,
"fp": 0,
"fn": 3,
"tn": 198,
"positives": 42,
"negatives": 198,
"flagged": 39,
"precision": 1,
"recall": 0.9285714285714286,
"f1": 0.962962962962963,
"recallWilson": {
"p": 0.9285714285714286,
"low": 0.8099028671147483,
"high": 0.9754100364488272
},
"precisionWilson": {
"p": 1,
"low": 0.9103301463997611,
"high": 1
}
}
},
"sourceSilent": {
"onVerifiableNegative": 0,
"onVerifiableError": 0,
"agreesWithUnsourced": 2,
"disagreesWithUnsourced": 13
},
"byClaimType": {
"version": {
"tp": 7,
"fp": 0,
"fn": 0,
"tn": 21,
"positives": 7,
"negatives": 21,
"flagged": 7,
"precision": 1,
"recall": 1,
"f1": 1,
"recallWilson": {
"p": 1,
"low": 0.6456611570247934,
"high": 1
},
"precisionWilson": {
"p": 1,
"low": 0.6456611570247934,
"high": 1
}
},
"tpm": {
"tp": 5,
"fp": 0,
"fn": 0,
"tn": 20,
"positives": 5,
"negatives": 20,
"flagged": 5,
"precision": 1,
"recall": 1,
"f1": 1,
"recallWilson": {
"p": 1,
"low": 0.5655085052479191,
"high": 1
},
"precisionWilson": {
"p": 1,
"low": 0.5655085052479191,
"high": 1
}
},
"region": {
"tp": 2,
"fp": 0,
"fn": 0,
"tn": 13,
"positives": 2,
"negatives": 13,
"flagged": 2,
"precision": 1,
"recall": 1,
"f1": 1,
"recallWilson": {
"p": 1,
"low": 0.34237195288961925,
"high": 1
},
"precisionWilson": {
"p": 1,
"low": 0.34237195288961925,
"high": 1
}
},
"status": {
"tp": 9,
"fp": 0,
"fn": 1,
"tn": 43,
"positives": 10,
"negatives": 43,
"flagged": 9,
"precision": 1,
"recall": 0.9,
"f1": 0.9473684210526316,
"recallWilson": {
"p": 0.9,
"low": 0.5958436145024278,
"high": 0.9821242504842788
},
"precisionWilson": {
"p": 1,
"low": 0.7008472464490407,
"high": 1
}
},
"taxonomy": {
"tp": 9,
"fp": 0,
"fn": 2,
"tn": 86,
"positives": 11,
"negatives": 86,
"flagged": 9,
"precision": 1,
"recall": 0.8181818181818182,
"f1": 0.9,
"recallWilson": {
"p": 0.8181818181818182,
"low": 0.5230138624217553,
"high": 0.9486333993289995
},
"precisionWilson": {
"p": 1,
"low": 0.7008472464490407,
"high": 1
}
},
"sku": {
"tp": 7,
"fp": 0,
"fn": 0,
"tn": 15,
"positives": 7,
"negatives": 15,
"flagged": 7,
"precision": 1,
"recall": 1,
"f1": 1,
"recallWilson": {
"p": 1,
"low": 0.6456611570247934,
"high": 1
},
"precisionWilson": {
"p": 1,
"low": 0.6456611570247934,
"high": 1
}
}
},
"gate": {
"pass": true,
"recallOk": true,
"precisionOk": true,
"beatsStaleness": true,
"thresholds": {
"minRecall": 0.7,
"minPrecision": 0.6
},
"reasons": [
"all criteria met"
]
}
}

54
scripts/kb-eval/data/judge-bakeoff-report-v3-g5bgold.md Normal file
View file

@ -0,0 +1,54 @@
# Judge bake-off-rapport — S1 (Fase 3 de-risk)
_Generert deterministisk av `run-judge-bakeoff.mjs` over `gold-correctness-set.json` + `judge-bakeoff-results.json`. Tall fra testet `lib/judge-bakeoff.mjs`. Ikke rediger for hånd — regenerer._
**Forhåndsregistrert gate (låst FØR fan-out):** recall ≥ 0.7, presisjon ≥ 0.6, OG judge-recall > staleness-recall.
## Evaluerings-populasjon (P)
Volatil stratum + fetchbare claim_types (price ekskludert) — der feilene bor; unngår «invertert leverage».
| metrikk | verdi |
|---|---|
| P totalt | 255 |
| Verifiserbare (correct/outdated/wrong) | 240 |
| Positive (reelle feil å fange) | 42 |
| Negative (correct) | 198 |
| Unsourced i P (kjørt, men utenfor P/R) | 15 |
## Arm-sammenligning (detektering over de 240 verifiserbare)
| arm | TP | FP | FN | TN | presisjon | recall | recall Wilson 95% | F1 |
|---|---|---|---|---|---|---|---|---|
| staleness (billig baseline) | 0 | 0 | 42 | 198 | n/a | 0.0% | [0.0%, 8.4%] | n/a |
| judge (per-påstand groundedness) | 39 | 0 | 3 | 198 | 100.0% | 92.9% | [81.0%, 97.5%] | 0.963 |
| hybrid (union) | 39 | 0 | 3 | 198 | 100.0% | 92.9% | [81.0%, 97.5%] | 0.963 |
## Judge per claim_type (verifiserbar delmengde)
| claim_type | positive | TP | FP | FN | presisjon | recall |
|---|---|---|---|---|---|---|
| taxonomy | 11 | 9 | 0 | 2 | 100.0% | 81.8% |
| status | 10 | 9 | 0 | 1 | 100.0% | 90.0% |
| version | 7 | 7 | 0 | 0 | 100.0% | 100.0% |
| sku | 7 | 7 | 0 | 0 | 100.0% | 100.0% |
| tpm | 5 | 5 | 0 | 0 | 100.0% | 100.0% |
| region | 2 | 2 | 0 | 0 | 100.0% | 100.0% |
## source_silent-diagnostikk
Judgen hentet siden men fant ikke verdien. Diagnostisk, ikke et flagg.
| signal | antall | tolkning |
|---|---|---|
| På verifiserbar feil | 0 | judge-bom: reell feil oversett via «kan ikke verifisere» |
| På verifiserbar correct | 0 | judge reproduserte ikke et korrekt faktum mennesket fant |
| Enig med unsourced | 2 | judge reproduserer den uverifiserbare grensen (godt) |
| Uenig med unsourced | 13 | judge hevdet grunnet/ugrunnet der mennesket ikke fant kilde |
## GATE: ✅ PASS — bygg S3
- recall 0.929 ≥ 0.7? **ja**
- presisjon 1.000 ≥ 0.6? **ja**
- slår staleness (recall 0.000)? **ja**
- begrunnelse: all criteria met

225
scripts/kb-eval/judge-claim-prompt-v3.1.md Normal file
View file

@ -0,0 +1,225 @@
# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)
v3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file
design, same three verdicts, same output schema, same evidence discipline. v3.1
changes only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the
**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed
were genuine judge misses, not stale gold).
**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b
freshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false
positives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3
flagged every one correctly** (`adr-template#1` "zero permission management" is
contradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`
overstates "recommended" as "obligatorisk"; `vector-storage#7` cites the wrong GA
date). So **v3 has zero real false positives** (P = 100% on corrected gold), and the
precision-side "FP-vakt" originally planned for v3.1 is **dropped — there is nothing to
defend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the
judge catches 3 documented failure modes it currently misses, without touching the
precision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).
**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /
R 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only
be adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without
introducing a single new false positive). Any new FP drops P below 100% and fails the
gate — keep v3. Recall rules are double-edged over the full population (v3's own
bake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay
frozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the
corrected `gold-correctness-set.json`.
---
You are a correctness judge for Microsoft AI reference documentation. You verify
factual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).
Be strict and adversarial — do not give the benefit of the doubt, do not pad, do not
infer a value the source does not state.
You are judging claims extracted from `<FILE>`. For EACH claim in the batch below,
decide whether the cited Microsoft Learn source **grounds** the claim.
## The three verdicts (exhaustive, mutually exclusive)
- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed
value(s). The page supports the claim. (Maps to gold `correct`.)
- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a
**different / contradicting / superseded** value for what the claim asserts. The
claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)
- **`source_silent`** — you fetched the cited page (and searched as a fallback) but
**no** `learn.microsoft.com` page states the claimed value at all. You cannot
confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure
pages typically lands here — that is expected, not a failure. **Exception: existence
claims — see Rule R2.**
## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)
A claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).
Verifying that the page "is about" the SKU/model/feature is **not** enough — the
specific number, name, date, tier, dimension, or status must match. If the claim
asserts value **X** and the page states a **different** value **Y** (even if adjacent
or plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for
`not_grounded`: you still need a fetched quote stating the **differing** value.
Applies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.
---
## CALIBRATION RULES — read all eight before judging
The exact-value rule is a blunt instrument. The 8 rules below sharpen it on both
edges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop
over-flagging where the core is grounded** (more correctly `grounded`). When a rule
below conflicts with a literal reading of the exact-value rule, the rule below governs
— it is the more precise standard.
### Recall side — flag these as `not_grounded`
**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**
A claim may assert a **bound**. Direction matters — judge it by which side the bound
constrains:
- **Lower bound** ("100+", "200k+", "at least N", "minimum N"): do not auto-`grounded`
it just because the true value satisfies the inequality. Apply the **lower-bound
policy:** if the true current value **grossly exceeds** the stated bound — roughly
**>2× and decision-changing** — the bound materially misleads → `not_grounded`. A
*tight* lower bound (true value within the same order of magnitude) stays `grounded`.
*Example (FN2): "200k+ context" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*
- **Upper bound** ("up to N", "opptil N", "maximum N", "no more than N", "as many as N"):
this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The
exact-value rule governs: if the live page states a current maximum **higher** than N,
the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even
a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it
is really higher. *Example (v3-FN): claim "up to 18 underlying models" while the page
states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true
maximum is N or the claim's ceiling still binds.)
**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the
claim asserts that a named entity **exists / is offered / is in a list** ("X is a
built-in judge", "feature Y is available", "tier Z exists"), and you fetch the
authoritative page that *would* enumerate it and the entity is **absent**, that absence
is **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve
`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered
prices). State in `reason` that you checked the canonical enumerating page and the
entity was not present. *Example (FN5): claim "99.99% SLA tier" while the reliability
page lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*
**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can
be superseded even when derived ratios survive. If the page shows the claim's framing
has been **replaced** (e.g. "1 Unit Capacity" → "Quota Tiers"; a renamed/retired
metric), the claim is `not_grounded` even if some embedded numbers still appear
somewhere — the claim describes a world that no longer exists. Check that the *unit and
structure* the claim assumes still match the current page, not just the digits.
**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or
effective-dated rows ("Before April 3, 2024", "Legacy", "Retiring"). A claim is
`grounded` only if it matches the **current/effective** row. Matching a clearly
time-stamped *past* row is `not_grounded` (the value has since changed). Always locate
the row that applies *today*. *Example (FN6): storage limits matching only the
"Before April 3, 2024" row while current limits differ → `not_grounded`.*
**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the
`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing
sub-assertions** (a status AND a region; a capability AND a named target; a date AND a
GA level). Verify **each load-bearing part separately**. If **any one** load-bearing
part is contradicted by the source, the whole claim is `not_grounded` — even when the
other parts check out. Do not let a correct first half earn a `grounded` for a wrong
second half. *Example (v3-FN): "Global training (Public Preview), cheaper, no data
residency; use regional in Norway East" — the GA-vs-Preview part and the "no residency"
part hold, but **Norway East is a Global (non-residency) training region, not a regional
one** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:
R6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,
load-bearing** part that is wrong. Decide first whether the part is load-bearing — if
the claim *asserts* it and a reader would act on it, it is.)
### Precision side — keep these `grounded` (do not over-flag)
**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a
numeric claim merely because the exact string is not verbatim, when the asserted value
is the **documented theoretical or benchmark equivalent** of what the page states and
both trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the
same technique, same order of magnitude, same direction). The exact-value rule targets
*drifted/contradicting* values — not two Microsoft-sourced expressions of the same
fact. If the page substantiates the magnitude and the technique, keep `grounded` and
note the equivalence in `reason`.
**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish "the
claim's **core** assertion is grounded but it omits a sub-category" from "the core is
ungrounded." If the page confirms the claim's **central** behavior/categorization and
the only gap is an *unstated additional* case the claim did not deny, that is
`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the
page **maps the core differently** or the claim **asserts** something the page
contradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;
a *stated* but wrong load-bearing part is not — that is R8's domain.)
**R7 — Follow the capability to its canonical page; don't punish illustrative numbers
(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real
capability** and the cited `evidence_url` does not foreground it, search for the
**canonical** page that documents the capability before judging — do not return
`not_grounded` merely because the *cited* page was a weak choice. And when a capability
is solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.
"~0 RTO/RPO", "≈15 min") that the claim offers as an order-of-magnitude illustration
rather than a cited spec. Judge the **capability**; treat an illustrative figure as
grounded if the capability is.
> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency
> covers only *illustrative* values. It does **NOT** cover a value or **exact string**
> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum
> value, a specific date/version. When the claim's load-bearing content is the literal
> name/string itself (e.g. "the metrics are `PromptTokens` and `CompletionTokens`"),
> the exact-value rule applies in full: if the live page names them differently
> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the
> claim is `not_grounded`. "Follow to the canonical page" means find the **right
> names**, not rescue wrong ones. A reader would copy that string into code; an
> illustrative magnitude they would not.
---
## Procedure (per claim)
1. **Identify the volatile assertion(s)** in the claim text — and when the claim
bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells
you what to check:
- `version` → model/API version, GA date, context window, max output, training cutoff
- `tpm` → tokens-per-minute / throughput / quota numbers
- `sku` → SKU name, tier, PTU minimums, deployment type
- `region` → regional availability
- `status` → GA / preview / retirement / deprecation status
- `taxonomy` → categorization, capability mapping, which-feature-does-what
2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's
`evidence_url`. If the claim has no `evidence_url`, or the fetched page does not
address the assertion, run `microsoft_docs_search` to find the authoritative page.
**Under R2/R7, actively seek the canonical enumerating/capability page** — a weak
cited URL is not the last word.
3. **Exact-value entailment check** each checkable value (and each load-bearing part
under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,
govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)
or an **upper** bound (ceiling) — they invert.
4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim
quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence
absence), the quote is the canonical enumeration in which the entity does **not**
appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.
## Hard rules
- Verify against the fetched page only. Do not rely on prior knowledge of model
specs / prices — those are exactly what may have drifted.
- Stable identifiers are not volatile and are not your job to refute: regulation year
(2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10
2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on
whatever volatile value it carries, else `source_silent`.
- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.
- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the
verdict (empty string for `source_silent`). `evidence_url` = the page you actually
used (may differ from the cited one if you fell back to search).
- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.
## Batch to judge (from `<FILE>`)
<CLAIMS>
## Output (strict JSON, no fence)
```
{"file":"<FILE>","results":[
{"id":"<claim id>","judge_verdict":"grounded|not_grounded|source_silent","rule":"<R1-R8 or empty>","evidence_url":"<url actually used>","evidence_quote":"<verbatim quote or empty>","reason":"<one sentence: what the source said vs the claim>"}
]}
```