open/ms-ai-architect
1
0
Fork 0
Code Issues Releases Activity

feat(ms-ai-architect): Spor D Steg B — skill-quality lifecycle-gate + score-cache (TDD)

Browse source 
B1: gateSkill(report, name) — pure post-mutation verdict (blocked/provisional/
improvements), wired into apply-skill-op.mjs to re-score the affected skill from
fresh disk after create/merge/sanitize-apply. K10 floor enforced immediately;
unjudged → provisional + nudge to re-run the judge pass. retire → skipped.

B2: buildScoreCache(result) — compact, deterministic cache shape + score-skill.mjs
--write [path] → data/skill-score-report.json (always whole-corpus). Gitignored
(derived/regenerable; avoids churn in the public repo). Consumed by Steg C surfacing.

8 new tests (tests/kb-eval/test-skill-score-gate.test.mjs). kb-eval 150 pass,
validate 239 pass. Live score unchanged: advisor 91, eng/gov/infra/sec 96, 0 < 90.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-06-23 18:26:29 +02:00
commit 92cd93771b
5 changed files with 294 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

28
scripts/kb-eval/score-skill.mjs
View file

@ -13,16 +13,25 @@
// node scripts/kb-eval/score-skill.mjs --skill <name> # scope output to one skill
// # (still scores the full corpus
// # so K10 sibling-overlap is correct)
// node scripts/kb-eval/score-skill.mjs --write [path] # persist the whole-corpus cache
// # (default: data/skill-score-report.json;
// # consumed by the STEG C surfacing)
//
// K10 requires every sibling description, so even single-skill (incremental)
// scoring evaluates the whole corpus and then filters the reported set — this is
// the building block for the lifecycle-gate in apply-skill-op.mjs.
// the building block for the lifecycle-gate in apply-skill-op.mjs. The --write
// cache is ALWAYS whole-corpus (surfacing needs every skill), independent of --skill.
//
// Zero dependencies. Pure scoring lives in lib/skill-score.mjs.
import { writeFileSync } from 'node:fs';
import { dirname, join } from 'node:path';
import { fileURLToPath } from 'node:url';
import { buildReport } from './eval.mjs';
import { scoreReport, formatScoreReport, TARGET } from './lib/skill-score.mjs';
import { scoreReport, formatScoreReport, buildScoreCache, TARGET } from './lib/skill-score.mjs';
const __dirname = dirname(fileURLToPath(import.meta.url));
const DEFAULT_CACHE = join(__dirname, 'data', 'skill-score-report.json');
function argValue(args, flag) {
const i = args.indexOf(flag);
@ -36,9 +45,24 @@ function main() {
const gate = gateRaw !== null ? Number(gateRaw) : null;
const skillFilter = argValue(args, '--skill');
// --write [path]: bare flag uses the default cache path; an explicit non-flag
// argument overrides it (the tests write to a temp path to avoid clobbering).
const writeIdx = args.indexOf('--write');
const doWrite = writeIdx !== -1;
const writePath = doWrite && args[writeIdx + 1] && !args[writeIdx + 1].startsWith('--')
? args[writeIdx + 1]
: DEFAULT_CACHE;
const target = Number.isFinite(gate) ? gate : TARGET;
const full = scoreReport(buildReport(), { target });
// The cache is whole-corpus regardless of --skill (surfacing needs every skill).
if (doWrite) {
const cache = buildScoreCache(full, { generatedAt: new Date().toISOString().slice(0, 10) });
writeFileSync(writePath, JSON.stringify(cache, null, 2) + '\n');
process.stdout.write(`✓ Skrev skill-score-cache → ${writePath} (${cache.scored.length} skills, ${cache.below.length} under mål).\n`);
}
// Scope the reported/gated set to one skill if requested (scoring already ran
// over the whole corpus, so K10 is computed correctly).
let scored = full.scored;