Kjell Tore Guttormsen
3e39f2df6b
Deterministisk de-risk-harness for Fase 3-judgen (kjøres på frosset 373-påstands gull-sett): - lib/judge-bakeoff.mjs: P-filter (volatil+fetchbar, price ekskl.), confusion-matrix for 3 armer (staleness/judge/hybrid), Wilson-bånd, forhåndsregistrert gate. 14 tester. - extract-judge-claims.mjs: blind manifest (255 påstander, 0 label-lekkasje — testet invariant). - judge-claim-prompt.md: blind per-påstands groundedness-judge (Opus xhigh, microsoft_docs_fetch). - run-judge-bakeoff.mjs: join gull+results på id, gate-rapport (.json/.md). Gate FORHÅNDSREGISTRERT (operatørvalg, før fan-out): recall ≥0.80, presisjon ≥0.70, OG slår staleness (0/38). Evalueringspop P = 240 verifiserbare, 38 positive. Suite 551/551 (538 + 13 nye).
51 lines
2.3 KiB
JavaScript
51 lines
2.3 KiB
JavaScript
#!/usr/bin/env node
|
|
// extract-judge-claims.mjs — S1 step: emit the BLIND claim manifest the judge
|
|
// subagents are graded on. Reads the frozen gold set, filters to the eval
|
|
// population P (volatile + fetchable, price excluded) via the tested lib, strips
|
|
// every claim to the fields the judge may see (no gold verdict / notes /
|
|
// lastmod_changed / stratum — no label leakage), and writes a flat manifest.
|
|
//
|
|
// The fan-out groups these by file at dispatch time (one judge subagent per file,
|
|
// mirroring how the gold set was built). The harness later joins the judge results
|
|
// back to gold by id and runs run-judge-bakeoff.mjs.
|
|
//
|
|
// Usage: node scripts/kb-eval/extract-judge-claims.mjs [--write]
|
|
// (default: print population summary; --write persists data/judge-bakeoff-claims.json)
|
|
|
|
import fs from 'node:fs';
|
|
import path from 'node:path';
|
|
import { fileURLToPath } from 'node:url';
|
|
import { blindClaims, evalPopulation } from './lib/judge-bakeoff.mjs';
|
|
|
|
const __dirname = path.dirname(fileURLToPath(import.meta.url));
|
|
const DATA = path.join(__dirname, 'data');
|
|
|
|
const gold = JSON.parse(fs.readFileSync(path.join(DATA, 'gold-correctness-set.json'), 'utf8'));
|
|
const blind = blindClaims(gold.claims);
|
|
|
|
// File -> claim count, for fan-out sizing (read-only summary).
|
|
const byFile = {};
|
|
for (const c of blind) (byFile[c.file] ||= 0), (byFile[c.file] += 1);
|
|
const files = Object.keys(byFile).length;
|
|
const withUrl = blind.filter((c) => c.evidence_url).length;
|
|
|
|
if (process.argv.includes('--write')) {
|
|
const out = path.join(DATA, 'judge-bakeoff-claims.json');
|
|
const payload = {
|
|
_meta: {
|
|
source: 'gold-correctness-set.json',
|
|
population: 'volatile + fetchable claim_type (price excluded)',
|
|
blind: 'gold verdict/notes/lastmod_changed/stratum withheld — no label leakage',
|
|
claim_count: blind.length,
|
|
files,
|
|
},
|
|
claims: blind,
|
|
};
|
|
fs.writeFileSync(out, JSON.stringify(payload, null, 2) + '\n');
|
|
console.log(`wrote ${out} (${blind.length} blind claims across ${files} files)`);
|
|
} else {
|
|
console.log(`eval population P: ${evalPopulation(gold.claims).length} claims`);
|
|
console.log(`blind manifest: ${blind.length} claims across ${files} files`);
|
|
console.log(`with evidence_url: ${withUrl} | without (judge falls back to docs_search): ${blind.length - withUrl}`);
|
|
console.log('(dry run — pass --write to persist judge-bakeoff-claims.json)');
|
|
}
|