open/ms-ai-architect
1
0
Fork 0
Code Issues Releases 1 Activity

feat(ms-ai-architect): generator-anti-regresjon Fase 1c — composeKbFile + TOC-gate

Browse source 
Nye ref-filer fodes best-practice-konforme sa neste KB-update ikke
reverserer TOC-arbeidet (write-path-regresjon). MA fore 1b.

- buildToc(body): deterministisk ## Innhold fra ##-seksjoner, GitHub-slugs,
  fence-aware, idempotent (lister aldri seg selv)
- composeKbFile(meta, body): generatoren — header + (store filer >100 linjer) TOC
  + brodtekst; erstatter bar buildKbHeader+body-konkatenering
- validateKbFile: stor fil uten TOC -> missing:['toc'] (write-path-gaten)
- Terskel 100 speiler eval.mjs N4_LARGE_FILE_LINES; bundet atferdsmessig via
  test som importerer ekte checkN4 (anti-regresjons-orakel), ikke cross-import
- Wiret composeKbFile inn i transform-prompt.md + kb-update.md (create + in-place)

TDD: +11 transform-tester. kb-update 349/349, kb-eval 154/154 (503 totalt).
Fikset pre-eksisterende skjor assertion i criterion-testen (stale
'Azure AI Foundry' -> 'AI Foundry'; produktet omdopt) — var rod for 1c.
This commit is contained in:
Kjell Tore Guttormsen 2026-06-26 10:59:40 +02:00
commit 2240f1efdd
5 changed files with 215 additions and 19 deletions
Download patch file Download diff file Expand all files Collapse all files

24
tests/kb-eval/test-transform-criterion.test.mjs
View file

@ -4,12 +4,14 @@
// Two halves, both runnable, no LLM:
// (1) The live deterministic eval for a target skill is ≥ the recorded baseline
// — the literal criterion, a regression guard over eval-baseline.json.
// (2) Regenerating one REAL reference file through the lag-4 pipeline (real body +
// a fresh buildKbHeader) yields a file that is valid, dated, source-anchored,
// (2) Regenerating one REAL reference file through the lag-4 pipeline (real body
// composed via composeKbFile) yields a file that is valid, dated, source-anchored,
// routed back to the same skill dir, and preserves the body verbatim — so
// writing it in place is score-preserving (same path → same ref count → same
// deterministic eval). It is in fact strictly BETTER: the pre-lag-4 file has
// no **Source:** header (the 0%-coverage failure mode), the regenerated one does.
// deterministic eval). It is in fact strictly BETTER on two axes: the pre-lag-4
// file has no **Source:** header (the 0%-coverage failure mode) and no TOC (N4) —
// the regenerated one carries both (composeKbFile births a TOC for large files,
// so `valid: true` here transitively proves the large file is TOC'd: Fase 1c).
import { test } from 'node:test';
import assert from 'node:assert/strict';
@ -17,7 +19,7 @@ import { readFileSync } from 'node:fs';
import { dirname, join, basename } from 'node:path';
import { fileURLToPath } from 'node:url';
import { evalSkill } from '../../scripts/kb-eval/eval.mjs';
import { buildKbHeader, validateKbFile, resolveTargetPath } from '../../scripts/kb-update/lib/transform.mjs';
import { composeKbFile, validateKbFile, resolveTargetPath } from '../../scripts/kb-update/lib/transform.mjs';
import { loadTaxonomy } from '../../scripts/kb-update/lib/taxonomy.mjs';
const __dirname = dirname(fileURLToPath(import.meta.url));
@ -69,21 +71,25 @@ test('regenerate 1 real file → valid, dated, source-anchored, body preserved,
const category = parts[3];
const filename = basename(refRel);
const regenerated = buildKbHeader({
const regenerated = composeKbFile({
title: 'AI Foundry Disaster Recovery Planning',
status: 'GA',
category,
source: 'https://learn.microsoft.com/azure/ai-foundry/concepts/disaster-recovery',
lastUpdated: '2026-06',
}) + body;
}, body);
// Strictly better than the original: now valid (Source present), still dated.
// Strictly better than the original: now valid (Source present + TOC for the large
// body), still dated. validateKbFile requires a TOC on large files, so valid:true
// here is also the Fase 1c anti-regression proof.
const v = validateKbFile(regenerated);
assert.equal(v.valid, true, `regenerated file invalid: ${v.missing}`);
// Body preserved verbatim — transformation does not drop content.
assert.ok(regenerated.includes(body), 'body not preserved');
assert.ok(regenerated.includes('Azure AI Foundry'), 'expected body content missing');
// Spot-check a stable body marker. ("Azure AI Foundry" was the old product name;
// the file now uses "Microsoft Foundry"/"AI Foundry" after the rename.)
assert.ok(regenerated.includes('AI Foundry'), 'expected body content missing');
// Routes back to the SAME skill/category dir → in-place → count-preserving → eval-score preserved.
assert.equal(resolveTargetPath(tax, category, filename), refRel);