open/ms-ai-architect
1
0
Fork 0
Code Issues Releases 1 Activity

feat(ms-ai-architect): S1 judge bake-off RESULT — GATE FAIL (recall 68.4% < 80%) [skip-docs]

Browse source 
Blind fan-out: 15 batcher Opus 4.8 xhigh, 255 P-påstander dekket (0 missing/dupes/extra),
hver judget mot live MS Learn via microsoft_docs_fetch (judgen så aldri gull-verdict).

Resultat mot forhåndsregistrert gate (recall >=0.80, presisjon >=0.70):
- judge: presisjon 81.3% (PASS), recall 68.4% (FAIL, Wilson [52.5-80.9%]), F1 0.743, 26/38 fanget
- staleness baseline: recall 0/38 (judge slår den desisivt)
- recall-drag konsentrert: sku 37.5%, taxonomy 66.7%; version/status/tpm allerede 80-86%
- 10/12 bommer var grounded-men-feil (brittle dekomponering, isolert til sku/taxonomy)

Per pre-registrert gate: STOPP, bygg IKKE S3. Fork (operatør) registrert i plan-doc:
(a) honorer FAIL, (b) scoped judge, (c) målrettet iterasjon + re-kjør.
This commit is contained in:
Kjell Tore Guttormsen 2026-06-26 20:45:09 +02:00
commit 79e34312ea
4 changed files with 1666 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

1376
scripts/kb-eval/data/judge-bakeoff-results.json Normal file
View file

File diff suppressed because it is too large Load diff