open/ms-ai-architect
1
0
Fork 0
Code Issues Releases 1 Activity

feat(ms-ai-architect): S1 v2 targeted iteration — GATE PASS (recall 84.2%, precision 84.2%) [skip-docs]

Browse source 
Operatørvalg (c): én målrettet, prinsipiell prompt-iterasjon på den diagnostiserte
grounded-men-feil-feilmoden (eksakt-verdi-entailment). Terskel uendret; v1 frosset.

Full blind v2 fan-out (15 batcher Opus 4.8 xhigh, 255 P-påstander dekket):
- judge v2: recall 84.2% (32/38, PASS >=0.80, Wilson [69.6-92.6%]), presisjon 84.2%
  (PASS >=0.70), F1 0.842, slår staleness 0/38.
- Fiksen løste målet: sku recall 37.5%->75.0%, taxonomy 66.7%->100%.
- +6 ekte fangster (26->32) uten netto nye FP (6->6) => recall OG presisjon opp.

Forbehold (ærlig): andre måling på samme frosne sett etter v1 (erkjent); Wilson nedre
grense 69.6% < 0.80 ved n=38; én iterasjon. Gate-logikk => vei mot S3. Stoppet for
operatør-beslutning (S2/S3), eskalerer ikke selv.

run-judge-bakeoff.mjs: --results/--report-prefix flagg (v2 uten å klobbe v1). Suite 552/552.
This commit is contained in:
Kjell Tore Guttormsen 2026-06-26 21:23:50 +02:00
commit 4cd290c14b
6 changed files with 1436 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

9
scripts/kb-eval/run-judge-bakeoff.mjs
View file

@ -36,7 +36,10 @@ if (!Number.isFinite(minRecall) || !Number.isFinite(minPrecision)) {
const thresholds = { minRecall, minPrecision };
const gold = JSON.parse(fs.readFileSync(path.join(DATA, 'gold-correctness-set.json'), 'utf8'));
const resultsPath = path.join(DATA, 'judge-bakeoff-results.json');
// --results / --report-prefix let a second iteration (v2) be graded without
// clobbering the frozen v1 artifacts. Defaults preserve the v1 file names.
const resultsPath = path.join(DATA, flag('--results') || 'judge-bakeoff-results.json');
const reportPrefix = flag('--report-prefix') || 'judge-bakeoff-report';
if (!fs.existsSync(resultsPath)) {
console.error(`error: ${resultsPath} not found — run the judge fan-out first`);
process.exit(2);
@ -126,8 +129,8 @@ Judgen hentet siden men fant ikke verdien. Diagnostisk, ikke et flagg.
`;
if (process.argv.includes('--write')) {
const jsonOut = path.join(DATA, 'judge-bakeoff-report.json');
const mdOut = path.join(DATA, 'judge-bakeoff-report.md');
const jsonOut = path.join(DATA, `${reportPrefix}.json`);
const mdOut = path.join(DATA, `${reportPrefix}.md`);
fs.writeFileSync(
jsonOut,
JSON.stringify(