feat(ms-ai-architect): Spor 3 Port 2 — create-time-guard (born-verified contract) [skip-docs]
Type-aware create-guard for reference-filer (Spor 3 Port 1/Port 2): - kb-headers.mjs: parseTypeHeader/parseVerifiedHeader/parseVerifiedByHeader (samme top-500-byte bold-label-skann som parseSourceHeader). - transform.mjs: buildKbHeader er type-aware (reference krever source+verified+verified_by; non-reference kaster pa MS-Learn-source) + emitterer Type/Verified/Verified by; validateKbFile type-aware; stampVerifiedMeta = fodt-verifisert-gate (stempler KUN ved bestatt judge-verdikt, ellers kaster). - validate-kb-file.mjs (ny): kjorbar gate generatorer kaller for commit (exit != 0 ved kontraktbrudd). - generate-skills.md + kb-update.md + transform-prompt.md: wiret til fodt-verifisert kontrakt (header-felt + judge-steg + create-guard-gate). Suite 586/586 (33 nye + 1 invariant). Plugin-validering 239/0/0. Utenfor scope (flagget): generate-skills.sh legacy (sonnet/Cosmo/no-source), Cosmo-persona i generatorene (S-Cosmo), korpus-migrering av 306 filer (Spor 1).
This commit is contained in:
parent
0ded2f9413
commit
5a0ba1fc9e
10 changed files with 621 additions and 54 deletions
|
|
@ -86,9 +86,13 @@ Format (STRENGT — alle seksjoner påkrevd):
|
|||
|
||||
# {SKILL_TITLE}
|
||||
|
||||
**Last updated:** 2026-02
|
||||
**Last updated:** {dagens måned, YYYY-MM}
|
||||
**Status:** [GA | Preview | Announced]
|
||||
**Category:** {CATEGORY_NAME}
|
||||
**Type:** reference
|
||||
**Source:** {den autoritative Microsoft Learn-URLen fra research — den faktiske doc-siden påstandene hviler på, ikke et søketreff}
|
||||
**Verified:** {SETTES i Steg 2.5 etter judge — IKKE her}
|
||||
**Verified by:** {SETTES i Steg 2.5 etter judge — IKKE her}
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -127,6 +131,23 @@ Format (STRENGT — alle seksjoner påkrevd):
|
|||
- Confidence markers: "Verified" (fra MCP), "Baseline" (modellkunnskap)
|
||||
- Konkret og balansert — vis fordeler OG ulemper
|
||||
|
||||
## Steg 2.5: Født-verifisert (judge FØR du returnerer)
|
||||
|
||||
Filen skal være *født verifisert*: ingen reference-fil forlater agenten uten at de
|
||||
maskin-verifiserbare påstandene er bekreftet mot `Source`-URLen.
|
||||
|
||||
1. Kjør den gjeldende claim-judgen (`scripts/kb-eval/judge-claim-prompt-v2.md`) over
|
||||
filens maskin-verifiserbare påstander (sku/version/tpm/status/region/taxonomy) mot
|
||||
`Source`-URLen du hentet i Steg 1.
|
||||
2. **Hvis alle påstander er `grounded`:** sett i headeren
|
||||
`**Verified:** {dagens dato, YYYY-MM-DD}` og `**Verified by:** judge-v2`.
|
||||
3. **Hvis noen påstand IKKE er grounded:** IKKE stempl `Verified`. Rett påstanden mot
|
||||
kilden og kjør judgen på nytt, eller — hvis verdien er maskin-uverifiserbar (pris/
|
||||
JS-rendret) — merk den eksplisitt og la filen gå til menneske-gaten (den blir IKKE
|
||||
committet; se batch-gaten under).
|
||||
|
||||
Aldri stempl `Verified` spekulativt. Stempelet betyr «judgen bekreftet mot kilde».
|
||||
|
||||
## Steg 3: Returner kvittering
|
||||
|
||||
Returner KUN dette (ingenting annet):
|
||||
|
|
@ -159,11 +180,19 @@ Task(general-purpose, sonnet): "Research + write skill: Multi-Index..."
|
|||
|
||||
1. **Parse kvitteringer** fra agentene
|
||||
2. **Verifiser filer finnes** med Glob
|
||||
3. **Oppdater state.json:**
|
||||
- Legg til ferdige skill-IDer i `completed`
|
||||
- Legg til eventuelle feilede i `failed`
|
||||
3. **Create-guard (OBLIGATORISK — Spor 3 Port 2):** kjør gaten over batchens nye filer:
|
||||
```bash
|
||||
node scripts/kb-update/validate-kb-file.mjs <fil1.md> <fil2.md> ...
|
||||
```
|
||||
Gaten exit-er ≠0 hvis en fil mangler `Source`, ikke er født-verifisert (mangler
|
||||
`Verified`/`Verified by`), eller er en stor fil uten TOC. **En fil som FAIL-er
|
||||
committes IKKE** — logg den i `state.failed` og send den til menneske-gaten. Dette er
|
||||
det som hindrer generatoren i å re-introdusere drift (avhengighet før Spor 1).
|
||||
4. **Oppdater state.json:**
|
||||
- Legg til ferdige (gate-PASS) skill-IDer i `completed`
|
||||
- Legg til feilede/gate-FAIL i `failed`
|
||||
- Oppdater `stats.total_generated` og `stats.total_bytes`
|
||||
4. **Neste batch** eller avslutt
|
||||
5. **Neste batch** eller avslutt
|
||||
|
||||
## Etter hele sesjonen
|
||||
|
||||
|
|
@ -180,13 +209,17 @@ Task(general-purpose, sonnet): "Research + write skill: Multi-Index..."
|
|||
Gjenstår: N skills
|
||||
```
|
||||
|
||||
2. **Commit:**
|
||||
2. **Create-guard FØR commit (OBLIGATORISK):** kjør gaten over ALLE filer du er i ferd
|
||||
med å committe. Bare gate-PASS-filer skal med:
|
||||
```bash
|
||||
node scripts/kb-update/validate-kb-file.mjs $(git diff --name-only --diff-filter=AM -- 'skills/ms-ai-*/references/**/*.md')
|
||||
git add skills/ms-ai-*/references/<dirs>/ scripts/skill-gen/state.json
|
||||
git commit -m "docs(architect): generate N knowledge skills (category-names)"
|
||||
```
|
||||
Hvis gaten exit-er ≠0: IKKE commit de feilende filene — fjern dem fra staging og send
|
||||
til menneske-gaten.
|
||||
|
||||
3. **Oppdater REMEMBER.md** med ny status
|
||||
3. **Oppdater STATE.md** med ny status
|
||||
|
||||
## Feilhåndtering
|
||||
|
||||
|
|
@ -263,8 +296,16 @@ Bruk Edit-verktøyet (IKKE Write) for å:
|
|||
- Oppdatere "Last updated" til gjeldende måned
|
||||
- Oppdatere utdaterte fakta, priser, datoer
|
||||
- Oppdatere Microsoft Learn-URLer
|
||||
- Markere oppdatert innhold med "Verified (MCP {måned})"
|
||||
- Beholde eksisterende struktur og seksjoner
|
||||
- **Backfill kontrakt-headeren** (Spor 3 Port 1) hvis legacy-fila mangler den: legg til
|
||||
`**Type:** reference` og `**Source:** {autoritativ Learn-URL}` i header-blokken (en
|
||||
berørt fil skal forlate oppdateringen kontrakt-kompatibel — inkrementell Spor 1).
|
||||
|
||||
## Steg 2.5: Født-verifisert (judge FØR du returnerer)
|
||||
Kjør claim-judgen (`scripts/kb-eval/judge-claim-prompt-v2.md`) over de oppdaterte maskin-
|
||||
verifiserbare påstandene mot `Source`. Alle `grounded` ⇒ sett `**Verified:** {dagens dato}`
|
||||
+ `**Verified by:** judge-v2`. Ikke grounded ⇒ IKKE stempl; rett mot kilde og kjør på nytt,
|
||||
eller send til menneske-gaten (fila committes ikke; se batch-gaten). Aldri stempl spekulativt.
|
||||
|
||||
## Steg 3: Returner kvittering
|
||||
SKILL_UPDATED
|
||||
|
|
@ -275,7 +316,8 @@ status: success|no_changes|failed
|
|||
```
|
||||
|
||||
4. Track in `state.json` under a new `"updated"` array
|
||||
5. After each batch, verify files still pass `validate-plugin.sh`
|
||||
5. **Create-guard (OBLIGATORISK):** `node scripts/kb-update/validate-kb-file.mjs <oppdaterte filer>` — gate-FAIL committes IKKE
|
||||
6. After each batch, verify files still pass `validate-plugin.sh`
|
||||
|
||||
**Key difference from generation:** Update uses Edit (preserves structure), generation uses Write (creates from scratch).
|
||||
|
||||
|
|
|
|||
|
|
@ -127,11 +127,11 @@ saveDecisions(led);
|
|||
d. **For `approved`:** registrér URLen i `url-registry` (gated) så den fanges av polling heretter — via `lib/registry-io.mjs` `saveRegistry`. Bruk `suggested_skill`/`suggested_category` fra kandidaten. Deretter kjør **transformasjonslaget (lag 4)** for å lage KB-fila:
|
||||
1. `microsoft_docs_fetch` på den godkjente URLen → kildedokument.
|
||||
2. Destillér via `scripts/kb-update/transform-prompt.md` (doc→KB-fil; status-påstander holdes eksplisitte). Multi-agent parallell fan-out foreslås i produksjon (roadmap §71), speilet på `generate-skills`-mønsteret.
|
||||
3. `content = composeKbFile({title, status, category, source: <godkjent URL>, lastUpdated: <YYYY-MM>}, body)` (`lib/transform.mjs`) — bygger header + (store filer >100 linjer) en deterministisk `## Innhold`-TOC + brødtekst. **`Status` + `Source` er obligatoriske** — `Source` i header-blokka (øverste 500 bytes) er det som lar lag 3 backfille `authority_source`. TOC-en fødes inn så regenerering ikke stripper den (Fase 1c).
|
||||
4. `validateKbFile(content)` MÅ være `valid: true` (title + Last updated + Status + Source; **store filer også TOC**) før noe gates videre (ellers be modellen fylle manglende felt).
|
||||
3. **Født-verifisert (Spor 3 Port 2):** kjør claim-judgen (`scripts/kb-eval/judge-claim-prompt-v2.md`) over brødtekstens maskin-verifiserbare påstander mot den godkjente URLen → `verdict = {pass}`. Så `meta = stampVerifiedMeta({title, status, category, source: <godkjent URL>, lastUpdated: <YYYY-MM>}, verdict, <i dag>)` — stempler `type='reference'` + `verified` + `verified_by='judge-v2'` **kun** ved `pass`; ellers KASTER (ingen fil, flagg for menneske). Deretter `content = composeKbFile(meta, body)` (`lib/transform.mjs`) — header + (store filer >100 linjer) en deterministisk `## Innhold`-TOC + brødtekst. **`Status` + `Source` + `Verified` + `Verified by` er obligatoriske** — `Source` i header-blokka (øverste 500 bytes) er det som lar lag 3 backfille `authority_source`. TOC-en fødes inn så regenerering ikke stripper den (Fase 1c).
|
||||
4. `validateKbFile(content)` MÅ være `valid: true` (title + Last updated + Status + Source + **Verified + Verified by**; **store filer også TOC**) før noe gates videre (ellers be modellen fylle manglende felt).
|
||||
5. For hver status-/load-bearing-påstand: `buildChange({...})` → **lag 5** `classifyChange(...)` (samme gate som §4 c2). Status-påstander er alltid `flagged`.
|
||||
6. `resolveTargetPath(tax, category, filename)` → eierskill-sti via taksonomien (`null` = ukjent kategori → flagg for operatør, ikke skriv).
|
||||
7. Først etter operatør-gate: atomisk skriving (`lib/atomic-write.mjs` + `lib/backup.mjs`). **`transform.mjs` skriver aldri selv** — verifisert av `tests/kb-update/test-transform.test.mjs` (import-invariant). Kriterium verifisert av `tests/kb-eval/test-transform-criterion.test.mjs` (regenerer 1 fil → eval ≥ baseline).
|
||||
7. **Create-guard FØR skriving:** `node scripts/kb-update/validate-kb-file.mjs <sti>` (exit ≠0 ⇒ ikke skriv). Først etter operatør-gate + grønn create-guard: atomisk skriving (`lib/atomic-write.mjs` + `lib/backup.mjs`). **`transform.mjs` skriver aldri selv** — verifisert av `tests/kb-update/test-transform.test.mjs` (import-invariant). Kriterium verifisert av `tests/kb-eval/test-transform-criterion.test.mjs` (regenerer 1 fil → eval ≥ baseline).
|
||||
|
||||
e. **Invariant:** `discover-new-urls.mjs` (deteksjon) skriver **aldri** ledgeren — den kun leser. Bare denne gaten skriver. Verifisert av `tests/kb-update/test-discover-invariant.test.mjs`.
|
||||
|
||||
|
|
|
|||
|
|
@ -10,6 +10,21 @@ const SOURCE_PATTERNS = [
|
|||
/\*\*Primary source:\*\*\s*(\S+)/i,
|
||||
];
|
||||
|
||||
// Spor 3 Port 1 frontmatter contract: these live in the SAME top-500-byte bold-label
|
||||
// region as **Source:**, so build-registry and the create-guard read one header format.
|
||||
// `Verified:` and `Verified by:` are distinct labels — the date pattern below is
|
||||
// anchored on the colon right after "Verified" so it never swallows the "by:" line.
|
||||
const TYPE_PATTERN = /\*\*Type:\*\*\s*(\S+)/i;
|
||||
const VERIFIED_PATTERN = /\*\*Verified:\*\*\s*(\S+)/i;
|
||||
const VERIFIED_BY_PATTERN = /\*\*Verified by:\*\*\s*(\S+)/i;
|
||||
|
||||
/** Scan only the header region for the first capture of `pattern`, trimmed. */
|
||||
function scanHeader(content, pattern) {
|
||||
if (!content || typeof content !== 'string') return null;
|
||||
const match = content.slice(0, HEADER_REGION_BYTES).match(pattern);
|
||||
return match ? match[1].trim() : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract the declared authority source URL from a KB reference file's header.
|
||||
* @param {string} content — full file content (only the first 500 bytes are scanned)
|
||||
|
|
@ -24,3 +39,32 @@ export function parseSourceHeader(content) {
|
|||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract the Port 1 `type` (reference|template|methodology|regulatory), lowercased.
|
||||
* @param {string} content — full file content (only the first 500 bytes are scanned)
|
||||
* @returns {string|null} the declared type, or null if absent
|
||||
*/
|
||||
export function parseTypeHeader(content) {
|
||||
const t = scanHeader(content, TYPE_PATTERN);
|
||||
return t === null ? null : t.toLowerCase();
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract the Port 1 `verified` date (when the file was last confirmed against source).
|
||||
* Anchored on `**Verified:**` so it never matches the `**Verified by:**` line.
|
||||
* @param {string} content — full file content (only the first 500 bytes are scanned)
|
||||
* @returns {string|null} the verified date (YYYY-MM or YYYY-MM-DD), or null if absent
|
||||
*/
|
||||
export function parseVerifiedHeader(content) {
|
||||
return scanHeader(content, VERIFIED_PATTERN);
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract the Port 1 `verified_by` field (`judge-vN` / `human`).
|
||||
* @param {string} content — full file content (only the first 500 bytes are scanned)
|
||||
* @returns {string|null} the verifier, or null if absent
|
||||
*/
|
||||
export function parseVerifiedByHeader(content) {
|
||||
return scanHeader(content, VERIFIED_BY_PATTERN);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -15,12 +15,38 @@
|
|||
// validateKbFile uses the SAME source detection as build-registry — one source of
|
||||
// truth for "what counts as a Source header" (the top-500-byte region).
|
||||
|
||||
import { parseSourceHeader } from './kb-headers.mjs';
|
||||
import {
|
||||
parseSourceHeader,
|
||||
parseTypeHeader,
|
||||
parseVerifiedHeader,
|
||||
parseVerifiedByHeader,
|
||||
} from './kb-headers.mjs';
|
||||
import { getCategorySkill } from './taxonomy.mjs';
|
||||
|
||||
// Mandatory header fields. Status is obligatory per spec ("status-felt obligatorisk");
|
||||
// Source is what lets lag 3 backfill authority_source and lag-5 rule 3 eventually fire.
|
||||
const REQUIRED_META = ['title', 'status', 'category', 'source', 'lastUpdated'];
|
||||
// Base mandatory header fields for EVERY file type. Status is obligatory per spec
|
||||
// ("status-felt obligatorisk").
|
||||
const REQUIRED_BASE_META = ['title', 'status', 'category', 'lastUpdated'];
|
||||
|
||||
// Spor 3 Port 1 — the frontmatter-contract `type`. Only `reference` files are in the
|
||||
// correctness scope (they carry a Microsoft source we verify against). Template /
|
||||
// methodology / regulatory are out of scope and carry NO Microsoft source.
|
||||
const VALID_TYPES = ['reference', 'template', 'methodology', 'regulatory'];
|
||||
const DEFAULT_TYPE = 'reference';
|
||||
|
||||
// Extra fields a `reference` file MUST carry to be "born verified" (Spor 3 Port 2):
|
||||
// `source` lets lag-5 authority checks fire; `verified`/`verified_by` record that the
|
||||
// content cleared the judge before it shipped. A reference file without these is the
|
||||
// drift-introducing failure mode the create-guard exists to stop.
|
||||
const REQUIRED_REFERENCE_META = ['source', 'verified', 'verified_by'];
|
||||
|
||||
// The current GATE-PASS claim judge (judge-claim-prompt-v2). stampVerifiedMeta records
|
||||
// `verified_by: judge-v<JUDGE_VERSION>` unless the caller overrides or marks `human`.
|
||||
const JUDGE_VERSION = 2;
|
||||
|
||||
// A Microsoft Learn / docs authority URL — a non-reference file may NOT carry one.
|
||||
const MS_LEARN_RE = /\b(?:learn|docs)\.microsoft\.com/i;
|
||||
|
||||
const DATE_RE = /^\d{4}-\d{2}(?:-\d{2})?$/;
|
||||
|
||||
// A reference file longer than this many lines should carry a table of contents
|
||||
// (best-practices §Structure). MIRRORS eval.mjs N4_LARGE_FILE_LINES — kept in sync
|
||||
|
|
@ -60,33 +86,84 @@ function slugify(heading) {
|
|||
}
|
||||
|
||||
/**
|
||||
* Build the deterministic KB-file header block. Status and Source live IN the
|
||||
* header (top 500 bytes) so kb-headers.parseSourceHeader and the "Last updated:"
|
||||
* scanner can see them — the old bottom-of-file source section was invisible to
|
||||
* those scanners, which is why authority_source coverage was 0%.
|
||||
* Build the deterministic KB-file header block. Status, Source and the Port 1 contract
|
||||
* fields (Type / Verified / Verified by) live IN the header (top 500 bytes) so
|
||||
* kb-headers.* and the "Last updated:" scanner can see them — the old bottom-of-file
|
||||
* source section was invisible to those scanners, which is why authority_source
|
||||
* coverage was 0%.
|
||||
*
|
||||
* @param {{title: string, status: string, category: string, source: string,
|
||||
* lastUpdated: string}} meta
|
||||
* Type-aware create-guard (Spor 3 Port 2): a `reference` file MUST carry
|
||||
* source + verified + verified_by (it is "born verified"); a non-reference file
|
||||
* (template/methodology/regulatory) is out of correctness scope and may NOT carry a
|
||||
* Microsoft Learn source. Either violation is a hard throw — the create-time gate.
|
||||
*
|
||||
* @param {{title: string, status: string, category: string, lastUpdated: string,
|
||||
* type?: string, source?: string, verified?: string, verified_by?: string}} meta
|
||||
* @returns {string} the header block, ending with the `---` rule + trailing newline
|
||||
* @throws if any mandatory field is missing/empty
|
||||
* @throws if the type is unknown, a mandatory field is missing, or a non-reference
|
||||
* file carries a Microsoft Learn source
|
||||
*/
|
||||
export function buildKbHeader(meta) {
|
||||
const m = meta ?? {};
|
||||
const missing = REQUIRED_META.filter((k) => {
|
||||
const type = m.type ?? DEFAULT_TYPE;
|
||||
if (!VALID_TYPES.includes(type)) {
|
||||
throw new Error(`buildKbHeader: invalid type '${type}' (expected ${VALID_TYPES.join('|')})`);
|
||||
}
|
||||
const isReference = type === 'reference';
|
||||
const empty = (k) => {
|
||||
const v = m[k];
|
||||
return v === undefined || v === null || String(v).trim() === '';
|
||||
});
|
||||
};
|
||||
const required = isReference
|
||||
? [...REQUIRED_BASE_META, ...REQUIRED_REFERENCE_META]
|
||||
: REQUIRED_BASE_META;
|
||||
const missing = required.filter(empty);
|
||||
if (missing.length) {
|
||||
throw new Error(`buildKbHeader: missing mandatory field(s): ${missing.join(', ')}`);
|
||||
}
|
||||
return (
|
||||
if (!isReference && !empty('source') && MS_LEARN_RE.test(String(m.source))) {
|
||||
throw new Error(
|
||||
`buildKbHeader: a ${type} file may not carry a Microsoft Learn source (out of correctness scope)`,
|
||||
);
|
||||
}
|
||||
let header =
|
||||
`# ${m.title}\n\n` +
|
||||
`**Last updated:** ${m.lastUpdated}\n` +
|
||||
`**Status:** ${m.status}\n` +
|
||||
`**Category:** ${m.category}\n` +
|
||||
`**Source:** ${m.source}\n\n` +
|
||||
`---\n`
|
||||
);
|
||||
`**Type:** ${type}\n`;
|
||||
if (!empty('source')) header += `**Source:** ${m.source}\n`;
|
||||
if (!empty('verified')) header += `**Verified:** ${m.verified}\n`;
|
||||
if (!empty('verified_by')) header += `**Verified by:** ${m.verified_by}\n`;
|
||||
return `${header}\n---\n`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Born-verified gate (Spor 3 Port 2): stamp `verified`/`verified_by` onto a reference
|
||||
* meta IFF the judge verdict passed — otherwise REFUSE (throw). This encodes
|
||||
* "nytt/regenerert innhold kjøres gjennom judgen FØR commit → født verifisert; ellers
|
||||
* ingen fil". transform.mjs never calls the judge itself (pure lib); the command runs
|
||||
* the v2 claim judge, aggregates per-claim results into `{pass}`, and calls this. A
|
||||
* failing verdict means no file is born.
|
||||
*
|
||||
* @param {object} baseMeta — the reference meta (title/status/category/source/lastUpdated)
|
||||
* @param {{pass: boolean, judgeVersion?: number, by?: 'human'}} verdict
|
||||
* @param {string} today — YYYY-MM or YYYY-MM-DD, supplied by the caller (lib stays pure)
|
||||
* @returns {object} baseMeta with type='reference' + verified + verified_by set
|
||||
* @throws if the verdict is not passing, or today is not a valid date
|
||||
*/
|
||||
export function stampVerifiedMeta(baseMeta, verdict, today) {
|
||||
if (!verdict || verdict.pass !== true) {
|
||||
throw new Error('stampVerifiedMeta: refusing to stamp verified — judge verdict not passing');
|
||||
}
|
||||
if (!DATE_RE.test(String(today ?? ''))) {
|
||||
throw new Error(`stampVerifiedMeta: invalid today date '${today}' (expected YYYY-MM[-DD])`);
|
||||
}
|
||||
const verifiedBy =
|
||||
verdict.by === 'human'
|
||||
? 'human'
|
||||
: `judge-v${Number.isInteger(verdict.judgeVersion) ? verdict.judgeVersion : JUDGE_VERSION}`;
|
||||
return { ...baseMeta, type: 'reference', verified: today, verified_by: verifiedBy };
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -181,10 +258,16 @@ const RE_STATUS = /\*\*Status:\*\*\s*\S/i;
|
|||
/**
|
||||
* Validate that a candidate KB file carries the mandatory contract: a title, a
|
||||
* "Last updated:" header (so it is never dateless → never the 'unverified' bucket),
|
||||
* a Status field, a Source header in the scannable region, and — for a large file
|
||||
* (>100 lines, per N4) — a table of contents. The TOC clause is the write-path
|
||||
* regression gate: a regenerated large file that lost its TOC fails here and the
|
||||
* caller re-prompts, so KB-update can never silently strip a TOC back out.
|
||||
* a Status field, and — for a large file (>100 lines, per N4) — a table of contents.
|
||||
* The TOC clause is the write-path regression gate: a regenerated large file that lost
|
||||
* its TOC fails here and the caller re-prompts, so KB-update can never silently strip a
|
||||
* TOC back out.
|
||||
*
|
||||
* Type-aware (Spor 3 Port 2): a `reference` file MUST additionally carry source +
|
||||
* verified + verified_by (born-verified). A file with no `**Type:**` header is treated
|
||||
* as a reference — the create path always stamps Type, and the legacy corpus (no Type)
|
||||
* is migrated under Spor 1, where this same contract is the target. Non-reference files
|
||||
* (template/methodology/regulatory) are out of correctness scope and need no source.
|
||||
*
|
||||
* @param {string} content — full file content
|
||||
* @returns {{valid: boolean, missing: string[]}}
|
||||
|
|
@ -195,7 +278,12 @@ export function validateKbFile(content) {
|
|||
if (!RE_TITLE.test(s)) missing.push('title');
|
||||
if (!RE_LAST_UPDATED.test(s)) missing.push('last_updated');
|
||||
if (!RE_STATUS.test(s)) missing.push('status');
|
||||
if (parseSourceHeader(s) === null) missing.push('source');
|
||||
const type = parseTypeHeader(s) ?? DEFAULT_TYPE;
|
||||
if (type === 'reference') {
|
||||
if (parseSourceHeader(s) === null) missing.push('source');
|
||||
if (parseVerifiedHeader(s) === null) missing.push('verified');
|
||||
if (parseVerifiedByHeader(s) === null) missing.push('verified_by');
|
||||
}
|
||||
if (isLargeContent(s) && !hasToc(s)) missing.push('toc');
|
||||
return { valid: missing.length === 0, missing };
|
||||
}
|
||||
|
|
|
|||
|
|
@ -23,13 +23,16 @@ regel 3 (autoritets-mismatch) faktisk fyre.
|
|||
- `{{CATEGORY}}` — taksonomi-kategorinøkkel (f.eks. `rag-architecture`) — bestemmer eierskill via `resolveTargetPath`
|
||||
- `{{SOURCE_URL}}` — den **utpekte autoritetskilden** (kanonisk Learn-URL dokumentet ble hentet fra)
|
||||
- `{{LAST_UPDATED}}` — `YYYY-MM` (dagens måned, caller-injisert)
|
||||
- `{{VERIFIED_DATE}}` — `YYYY-MM-DD` (dagen judgen bekreftet innholdet; settes av `stampVerifiedMeta` etter steg 2, ikke før)
|
||||
- `{{FETCHED_DOC}}` — den hentede markdownen fra `microsoft_docs_fetch`
|
||||
- `{{EXISTING_CONTEXT}}` — sammendrag av overlappende eksisterende ref-filer (unngå duplisering — K4)
|
||||
|
||||
## Header (deterministisk — IKKE la LLM-en skrive den)
|
||||
|
||||
Header-blokka bygges av `buildKbHeader({title, status, category, source, lastUpdated})`,
|
||||
ikke av modellen. Den ser slik ut og MÅ stå først:
|
||||
Header-blokka bygges av `buildKbHeader({title, status, category, type, source, lastUpdated,
|
||||
verified, verified_by})`, ikke av modellen. En `reference`-fil er **født verifisert**
|
||||
(Spor 3 Port 1/Port 2): den MÅ bære `Type` + `Source` + `Verified` + `Verified by`.
|
||||
Headeren ser slik ut og MÅ stå først:
|
||||
|
||||
```markdown
|
||||
# {{TITLE}}
|
||||
|
|
@ -37,17 +40,22 @@ ikke av modellen. Den ser slik ut og MÅ stå først:
|
|||
**Last updated:** {{LAST_UPDATED}}
|
||||
**Status:** {{STATUS}}
|
||||
**Category:** {{CATEGORY}}
|
||||
**Type:** reference
|
||||
**Source:** {{SOURCE_URL}}
|
||||
**Verified:** {{VERIFIED_DATE}}
|
||||
**Verified by:** judge-v2
|
||||
|
||||
---
|
||||
```
|
||||
|
||||
Modellen skriver **kun brødteksten under `---`**. `composeKbFile({...}, body)` setter så
|
||||
sammen hele fila — header + (for **store filer >100 linjer**) en deterministisk
|
||||
`## Innhold`-TOC fra brødtekstens `##`-seksjoner + brødteksten. Etterpå kjører
|
||||
`validateKbFile(content)` og MÅ returnere `valid: true` (title + Last updated + Status +
|
||||
Source til stede; **for store filer også TOC**) før noe gates videre. TOC-en fødes inn så
|
||||
en senere regenerering ikke stripper den (write-path-regresjon — Fase 1c).
|
||||
`Verified`/`Verified by` settes IKKE før innholdet har bestått judgen (se «Etter
|
||||
destillasjon» steg 2). Modellen skriver **kun brødteksten under `---`**.
|
||||
`composeKbFile(stampVerifiedMeta({...}, verdict, today), body)` setter så sammen hele fila
|
||||
— header + (for **store filer >100 linjer**) en deterministisk `## Innhold`-TOC fra
|
||||
brødtekstens `##`-seksjoner + brødteksten. Etterpå kjører `validateKbFile(content)` og MÅ
|
||||
returnere `valid: true` (title + Last updated + Status + Source + **Verified + Verified by**
|
||||
til stede; **for store filer også TOC**) før noe gates videre. TOC-en fødes inn så en
|
||||
senere regenerering ikke stripper den (write-path-regresjon — Fase 1c).
|
||||
|
||||
## Prompt (brødtekst)
|
||||
|
||||
|
|
@ -90,10 +98,19 @@ Ingen innledning eller avslutningskommentar — kun markdown-brødteksten.
|
|||
|
||||
## Etter destillasjon (caller, gated)
|
||||
|
||||
1. `content = composeKbFile({title, status, category, source, lastUpdated}, body)` — header + (store filer) `## Innhold`-TOC + brødtekst. Erstatter den bare `buildKbHeader(...) + body`-konkateneringen.
|
||||
2. `validateKbFile(content)` → må være `valid: true` (title + Last updated + Status + Source; **store filer også TOC**) (ellers: be modellen fikse manglende felt).
|
||||
3. For hver status-/load-bearing-påstand: `buildChange({field, oldValue, newValue, sourceUrl, authoritySource})`
|
||||
1. **Født-verifisert judge (FØR commit):** kjør claim-judgen (`scripts/kb-eval/judge-claim-prompt-v2.md`)
|
||||
over brødtekstens maskin-verifiserbare påstander mot `{{SOURCE_URL}}`. Aggreger til `verdict = {pass}`
|
||||
(alle maskin-verifiserbare påstander `grounded` ⇒ `pass: true`).
|
||||
2. `meta = stampVerifiedMeta({title, status, category, source, lastUpdated}, verdict, today)` —
|
||||
stempler `type='reference'` + `verified=today` + `verified_by='judge-v2'` **kun** ved `pass: true`;
|
||||
ellers KASTER den (ingen fil fødes). Deretter `content = composeKbFile(meta, body)` —
|
||||
header + (store filer) `## Innhold`-TOC + brødtekst.
|
||||
3. `validateKbFile(content)` → må være `valid: true` (title + Last updated + Status + Source +
|
||||
**Verified + Verified by**; **store filer også TOC**) (ellers: be modellen fikse manglende felt).
|
||||
4. For hver status-/load-bearing-påstand: `buildChange({field, oldValue, newValue, sourceUrl, authoritySource})`
|
||||
→ `classifyChange(...)` (lag 5). `flagged` → operatør avgjør; status-påstander er alltid `flagged`.
|
||||
4. `resolveTargetPath(tax, category, filename)` → eierskill-sti (null = ukjent kategori → gate, ikke skriv).
|
||||
5. Først etter operatør-gate: atomisk skriving (`lib/atomic-write.mjs` + `lib/backup.mjs`). Transform-libben skriver aldri selv.
|
||||
5. `resolveTargetPath(tax, category, filename)` → eierskill-sti (null = ukjent kategori → gate, ikke skriv).
|
||||
6. **Create-guard FØR skriving:** `node scripts/kb-update/validate-kb-file.mjs <sti>` (exit ≠0 = ikke skriv).
|
||||
Først etter operatør-gate + grønn create-guard: atomisk skriving (`lib/atomic-write.mjs` +
|
||||
`lib/backup.mjs`). Transform-libben skriver aldri selv.
|
||||
```
|
||||
|
|
|
|||
69
scripts/kb-update/validate-kb-file.mjs
Normal file
69
scripts/kb-update/validate-kb-file.mjs
Normal file
|
|
@ -0,0 +1,69 @@
|
|||
#!/usr/bin/env node
|
||||
// validate-kb-file.mjs — the executable create-time-guard (Spor 3 Port 2).
|
||||
//
|
||||
// The gate any generator (generate-skills, kb-update) calls AFTER writing a candidate
|
||||
// KB file and BEFORE committing it. It wraps transform.validateKbFile over one or more
|
||||
// paths and exits NON-ZERO if any file violates the contract — so a regenerated file
|
||||
// that lost its Source, is not born-verified, or dropped its TOC can never ship. This
|
||||
// is what makes "create-guard MUST be in place before Spor 1 corpus-fixing" enforceable:
|
||||
// the generator physically cannot re-introduce the drift the corpus pass is removing.
|
||||
//
|
||||
// Usage:
|
||||
// node scripts/kb-update/validate-kb-file.mjs <file.md> [<file2.md> ...]
|
||||
//
|
||||
// Exit code: 0 = all files valid; 1 = at least one violation (or usage error).
|
||||
|
||||
import { readFileSync, realpathSync } from 'node:fs';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { validateKbFile } from './lib/transform.mjs';
|
||||
|
||||
/**
|
||||
* Validate a batch of candidate KB files. Pure + dependency-injected (readFile) so it is
|
||||
* unit-testable without disk. A path that cannot be read is reported as invalid (with a
|
||||
* `read-error` marker) rather than throwing — one unreadable file must not crash the gate.
|
||||
*
|
||||
* @param {string[]} paths
|
||||
* @param {(p: string) => string} [readFile] — reader (defaults to readFileSync utf8)
|
||||
* @returns {{ok: boolean, results: Array<{path: string, valid: boolean, missing: string[]}>}}
|
||||
*/
|
||||
export function validatePaths(paths, readFile = (p) => readFileSync(p, 'utf8')) {
|
||||
const results = (paths ?? []).map((path) => {
|
||||
let content;
|
||||
try {
|
||||
content = readFile(path);
|
||||
} catch (err) {
|
||||
return { path, valid: false, missing: [`read-error: ${err.message}`] };
|
||||
}
|
||||
const { valid, missing } = validateKbFile(content);
|
||||
return { path, valid, missing };
|
||||
});
|
||||
return { ok: results.every((r) => r.valid), results };
|
||||
}
|
||||
|
||||
function main(argv) {
|
||||
const paths = argv.slice(2);
|
||||
if (paths.length === 0) {
|
||||
process.stderr.write('usage: validate-kb-file.mjs <file.md> [<file2.md> ...]\n');
|
||||
process.exit(1);
|
||||
}
|
||||
const { ok, results } = validatePaths(paths);
|
||||
for (const r of results) {
|
||||
if (r.valid) {
|
||||
process.stdout.write(`OK ${r.path}\n`);
|
||||
} else {
|
||||
process.stdout.write(`FAIL ${r.path} — missing: ${r.missing.join(', ')}\n`);
|
||||
}
|
||||
}
|
||||
process.exit(ok ? 0 : 1);
|
||||
}
|
||||
|
||||
// Robust direct-invocation check (mirrors detect-courses.mjs): only run main when this
|
||||
// file is the entrypoint, not when imported by a test.
|
||||
const isMain = (() => {
|
||||
try {
|
||||
return realpathSync(process.argv[1]) === realpathSync(fileURLToPath(import.meta.url));
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
})();
|
||||
if (isMain) main(process.argv);
|
||||
|
|
@ -19,7 +19,7 @@ import { readFileSync } from 'node:fs';
|
|||
import { dirname, join, basename } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { evalSkill } from '../../scripts/kb-eval/eval.mjs';
|
||||
import { composeKbFile, validateKbFile, resolveTargetPath } from '../../scripts/kb-update/lib/transform.mjs';
|
||||
import { composeKbFile, validateKbFile, resolveTargetPath, stampVerifiedMeta } from '../../scripts/kb-update/lib/transform.mjs';
|
||||
import { loadTaxonomy } from '../../scripts/kb-update/lib/taxonomy.mjs';
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
|
|
@ -71,16 +71,20 @@ test('regenerate 1 real file → valid, dated, source-anchored, body preserved,
|
|||
const category = parts[3];
|
||||
const filename = basename(refRel);
|
||||
|
||||
const regenerated = composeKbFile({
|
||||
// Born-verified (Spor 3 Port 2): the regenerated file is stamped only after a passing
|
||||
// judge verdict. Here the verdict is supplied directly (no LLM in this test) — the
|
||||
// point is the regenerated file carries the full Port 1 contract (source + verified).
|
||||
const stamped = stampVerifiedMeta({
|
||||
title: 'AI Foundry Disaster Recovery Planning',
|
||||
status: 'GA',
|
||||
category,
|
||||
source: 'https://learn.microsoft.com/azure/ai-foundry/concepts/disaster-recovery',
|
||||
lastUpdated: '2026-06',
|
||||
}, body);
|
||||
}, { pass: true }, '2026-06-29');
|
||||
const regenerated = composeKbFile(stamped, body);
|
||||
|
||||
// Strictly better than the original: now valid (Source present + TOC for the large
|
||||
// body), still dated. validateKbFile requires a TOC on large files, so valid:true
|
||||
// Strictly better than the original: now valid (Source + Verified present + TOC for the
|
||||
// large body), still dated. validateKbFile requires a TOC on large files, so valid:true
|
||||
// here is also the Fase 1c anti-regression proof.
|
||||
const v = validateKbFile(regenerated);
|
||||
assert.equal(v.valid, true, `regenerated file invalid: ${v.missing}`);
|
||||
|
|
|
|||
|
|
@ -6,7 +6,12 @@
|
|||
|
||||
import { test } from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { parseSourceHeader } from '../../scripts/kb-update/lib/kb-headers.mjs';
|
||||
import {
|
||||
parseSourceHeader,
|
||||
parseTypeHeader,
|
||||
parseVerifiedHeader,
|
||||
parseVerifiedByHeader,
|
||||
} from '../../scripts/kb-update/lib/kb-headers.mjs';
|
||||
|
||||
test('parseSourceHeader — **Source:** header', () => {
|
||||
const content = '# Title\n\n**Source:** https://learn.microsoft.com/azure/foundry/concepts/x\n**Status:** GA\n';
|
||||
|
|
@ -50,3 +55,62 @@ test('parseSourceHeader — trims trailing whitespace from URL', () => {
|
|||
const content = '**Source:** https://learn.microsoft.com/azure/trim \n';
|
||||
assert.equal(parseSourceHeader(content), 'https://learn.microsoft.com/azure/trim');
|
||||
});
|
||||
|
||||
// --- parseTypeHeader: the Port 1 frontmatter-contract `type` field --------------
|
||||
// Spor 3 Port 1: every file carries type (reference|template|methodology|regulatory).
|
||||
// Same top-500-byte bold-label scan as parseSourceHeader (one source of truth).
|
||||
|
||||
test('parseTypeHeader — **Type:** header', () => {
|
||||
const content = '# T\n\n**Status:** GA\n**Type:** reference\n**Source:** https://learn.microsoft.com/x\n';
|
||||
assert.equal(parseTypeHeader(content), 'reference');
|
||||
});
|
||||
|
||||
test('parseTypeHeader — case-insensitive label, lowercased value', () => {
|
||||
assert.equal(parseTypeHeader('**type:** Methodology\n'), 'methodology');
|
||||
});
|
||||
|
||||
test('parseTypeHeader — no header returns null', () => {
|
||||
assert.equal(parseTypeHeader('# T\n\n**Status:** GA\n'), null);
|
||||
assert.equal(parseTypeHeader(''), null);
|
||||
assert.equal(parseTypeHeader(null), null);
|
||||
});
|
||||
|
||||
test('parseTypeHeader — only scans the top of the file (header region)', () => {
|
||||
const buried = '# T\n' + 'x'.repeat(800) + '\n**Type:** template\n';
|
||||
assert.equal(parseTypeHeader(buried), null);
|
||||
});
|
||||
|
||||
// --- parseVerifiedHeader: the Port 1 `verified` date (born-verified stamp) ------
|
||||
|
||||
test('parseVerifiedHeader — full date', () => {
|
||||
assert.equal(parseVerifiedHeader('**Verified:** 2026-06-29\n'), '2026-06-29');
|
||||
});
|
||||
|
||||
test('parseVerifiedHeader — YYYY-MM date', () => {
|
||||
assert.equal(parseVerifiedHeader('**Verified:** 2026-06\n'), '2026-06');
|
||||
});
|
||||
|
||||
test('parseVerifiedHeader — does NOT match the "**Verified by:**" line', () => {
|
||||
// "Verified by:" is a distinct label; the date parser must not swallow it.
|
||||
assert.equal(parseVerifiedHeader('**Verified by:** judge-v2\n'), null);
|
||||
});
|
||||
|
||||
test('parseVerifiedHeader — no header / garbage returns null', () => {
|
||||
assert.equal(parseVerifiedHeader('**Status:** GA\n'), null);
|
||||
assert.equal(parseVerifiedHeader(null), null);
|
||||
});
|
||||
|
||||
// --- parseVerifiedByHeader: the Port 1 `verified_by` field (judge-vN | human) ---
|
||||
|
||||
test('parseVerifiedByHeader — judge version', () => {
|
||||
assert.equal(parseVerifiedByHeader('**Verified by:** judge-v2\n'), 'judge-v2');
|
||||
});
|
||||
|
||||
test('parseVerifiedByHeader — human', () => {
|
||||
assert.equal(parseVerifiedByHeader('**Verified by:** human\n'), 'human');
|
||||
});
|
||||
|
||||
test('parseVerifiedByHeader — no header returns null', () => {
|
||||
assert.equal(parseVerifiedByHeader('**Verified:** 2026-06-29\n'), null);
|
||||
assert.equal(parseVerifiedByHeader(''), null);
|
||||
});
|
||||
|
|
|
|||
|
|
@ -24,8 +24,14 @@ import {
|
|||
buildToc,
|
||||
composeKbFile,
|
||||
insertToc,
|
||||
stampVerifiedMeta,
|
||||
} from '../../scripts/kb-update/lib/transform.mjs';
|
||||
import { parseSourceHeader } from '../../scripts/kb-update/lib/kb-headers.mjs';
|
||||
import {
|
||||
parseSourceHeader,
|
||||
parseTypeHeader,
|
||||
parseVerifiedHeader,
|
||||
parseVerifiedByHeader,
|
||||
} from '../../scripts/kb-update/lib/kb-headers.mjs';
|
||||
import { classifyChange } from '../../scripts/kb-update/lib/verify-out.mjs';
|
||||
import { loadTaxonomy } from '../../scripts/kb-update/lib/taxonomy.mjs';
|
||||
// The N4 (TOC-in-large-files) check is the cross-subsystem best-practice contract.
|
||||
|
|
@ -44,12 +50,17 @@ const SHORT_BODY = '## Introduksjon\n\nKort.\n';
|
|||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
|
||||
// A born-verified reference meta: Spor 3 Port 1 contract adds type/verified/verified_by
|
||||
// (reference files are "born verified" — stamped only after the v2 judge clears them).
|
||||
const META = {
|
||||
title: 'Azure AI Search - Hybrid Retrieval',
|
||||
status: 'GA',
|
||||
category: 'rag-architecture',
|
||||
type: 'reference',
|
||||
source: 'https://learn.microsoft.com/azure/search/hybrid-search-overview',
|
||||
lastUpdated: '2026-06',
|
||||
verified: '2026-06-29',
|
||||
verified_by: 'judge-v2',
|
||||
};
|
||||
|
||||
// --- buildKbHeader: mandatory fields, parseable by the real header scanners ----
|
||||
|
|
@ -76,6 +87,154 @@ test('buildKbHeader throws when a mandatory field is missing (status-felt obliga
|
|||
assert.throws(() => buildKbHeader({ ...META, lastUpdated: undefined }), /last.?updated/i);
|
||||
});
|
||||
|
||||
// --- Spor 3 Port 1/Port 2: type-aware create-guard contract --------------------
|
||||
// Every reference file is "born verified": carries Type + Verified + Verified by in
|
||||
// the header, all within the top-500-byte region the scanners read.
|
||||
|
||||
test('buildKbHeader emits Type, Verified, Verified by in the header (Port 1 contract)', () => {
|
||||
const h = buildKbHeader(META);
|
||||
assert.match(h, /\*\*Type:\*\*\s*reference/);
|
||||
assert.match(h, /\*\*Verified:\*\*\s*2026-06-29/);
|
||||
assert.match(h, /\*\*Verified by:\*\*\s*judge-v2/);
|
||||
});
|
||||
|
||||
test('buildKbHeader: the new fields are parseable in the top-500-byte region', () => {
|
||||
const h = buildKbHeader(META);
|
||||
assert.equal(parseTypeHeader(h), 'reference');
|
||||
assert.equal(parseVerifiedHeader(h), '2026-06-29');
|
||||
assert.equal(parseVerifiedByHeader(h), 'judge-v2');
|
||||
});
|
||||
|
||||
test('buildKbHeader: Verified by (the last header line) stays inside the 500-byte scan region for a realistic long title + URL', () => {
|
||||
// Verified by sits after Source (the long field) — guard that a realistic worst case
|
||||
// still fits, so a valid file is never wrongly flagged as missing verified_by.
|
||||
const h = buildKbHeader({
|
||||
title: 'Azure AI Foundry Model Router and Deployment Strategies for Enterprise',
|
||||
status: 'Public Preview',
|
||||
category: 'azure-ai-services',
|
||||
type: 'reference',
|
||||
source: 'https://learn.microsoft.com/en-us/azure/ai-foundry/concepts/model-router-deployment-strategies-overview',
|
||||
lastUpdated: '2026-06',
|
||||
verified: '2026-06-29',
|
||||
verified_by: 'judge-v2',
|
||||
});
|
||||
assert.equal(parseVerifiedByHeader(h), 'judge-v2', 'verified_by fell outside the 500-byte header region');
|
||||
});
|
||||
|
||||
test('buildKbHeader: a reference file MUST carry verified + verified_by (born-verified guard)', () => {
|
||||
assert.throws(() => buildKbHeader({ ...META, verified: undefined }), /verified/i);
|
||||
assert.throws(() => buildKbHeader({ ...META, verified_by: '' }), /verified_by|verified by/i);
|
||||
});
|
||||
|
||||
test('buildKbHeader: type defaults to reference when omitted (back-compat for callers)', () => {
|
||||
const { type, ...noType } = META;
|
||||
const h = buildKbHeader(noType);
|
||||
assert.equal(parseTypeHeader(h), 'reference');
|
||||
});
|
||||
|
||||
test('buildKbHeader: an unknown type is rejected', () => {
|
||||
assert.throws(() => buildKbHeader({ ...META, type: 'blogpost' }), /type/i);
|
||||
});
|
||||
|
||||
test('buildKbHeader: a non-reference file (methodology) needs NO source/verified', () => {
|
||||
// Template/methodology/regulatory are out of correctness-scope — they carry no MS source.
|
||||
const h = buildKbHeader({
|
||||
title: 'ROS-metodikk (NS 5814)',
|
||||
status: 'Stable',
|
||||
category: 'norwegian-public-sector-ai-governance',
|
||||
type: 'methodology',
|
||||
lastUpdated: '2026-06',
|
||||
});
|
||||
assert.match(h, /\*\*Type:\*\*\s*methodology/);
|
||||
assert.doesNotMatch(h, /\*\*Source:\*\*/);
|
||||
assert.doesNotMatch(h, /\*\*Verified:\*\*/);
|
||||
});
|
||||
|
||||
test('buildKbHeader: a non-reference file may NOT carry a Microsoft Learn source (contract)', () => {
|
||||
assert.throws(
|
||||
() => buildKbHeader({
|
||||
title: 'X', status: 'Stable', category: 'c', type: 'template',
|
||||
lastUpdated: '2026-06', source: 'https://learn.microsoft.com/azure/x',
|
||||
}),
|
||||
/source/i,
|
||||
);
|
||||
});
|
||||
|
||||
// --- validateKbFile: type-aware enforcement of the full contract ---------------
|
||||
|
||||
test('validateKbFile: a reference file missing verified/verified_by is invalid', () => {
|
||||
// Header has Source but no Verified / Verified by → not born-verified → rejected.
|
||||
const file =
|
||||
'# T\n\n**Last updated:** 2026-06\n**Status:** GA\n**Type:** reference\n' +
|
||||
'**Source:** https://learn.microsoft.com/x\n\n---\n\n## A\n\ntekst\n';
|
||||
const r = validateKbFile(file);
|
||||
assert.equal(r.valid, false);
|
||||
assert.ok(r.missing.includes('verified'), `expected 'verified' in ${r.missing}`);
|
||||
assert.ok(r.missing.includes('verified_by'), `expected 'verified_by' in ${r.missing}`);
|
||||
});
|
||||
|
||||
test('validateKbFile: a complete born-verified reference file is valid', () => {
|
||||
const file = buildKbHeader(META) + '\n## Introduksjon\n\nNoe innhold.\n';
|
||||
assert.equal(validateKbFile(file).valid, true);
|
||||
});
|
||||
|
||||
test('validateKbFile: a methodology file without source/verified is valid (out of scope)', () => {
|
||||
const file =
|
||||
'# ROS-metodikk\n\n**Last updated:** 2026-06\n**Status:** Stable\n**Type:** methodology\n\n' +
|
||||
'---\n\n## Metode\n\nNS 5814.\n';
|
||||
const r = validateKbFile(file);
|
||||
assert.equal(r.valid, true, `unexpected missing: ${r.missing}`);
|
||||
});
|
||||
|
||||
// --- stampVerifiedMeta: the born-verified gate (judge clears → stamp; else refuse) --
|
||||
// transform.mjs never calls the judge (pure lib). The command runs the v2 claim judge,
|
||||
// aggregates per-claim results into {pass}, and calls this. A failing verdict throws —
|
||||
// "nytt innhold gjennom judgen FØR commit; ellers ingen fil." (§4 Port 2.)
|
||||
|
||||
test('stampVerifiedMeta stamps verified=today + verified_by=judge-vN on a passing verdict', () => {
|
||||
const base = { title: 'T', status: 'GA', category: 'c',
|
||||
source: 'https://learn.microsoft.com/x', lastUpdated: '2026-06' };
|
||||
const m = stampVerifiedMeta(base, { pass: true }, '2026-06-29');
|
||||
assert.equal(m.type, 'reference');
|
||||
assert.equal(m.verified, '2026-06-29');
|
||||
assert.equal(m.verified_by, 'judge-v2'); // current GATE-PASS judge
|
||||
});
|
||||
|
||||
test('stampVerifiedMeta honours an explicit judgeVersion', () => {
|
||||
const base = { title: 'T', status: 'GA', category: 'c',
|
||||
source: 'https://learn.microsoft.com/x', lastUpdated: '2026-06' };
|
||||
assert.equal(stampVerifiedMeta(base, { pass: true, judgeVersion: 3 }, '2026-06-29').verified_by, 'judge-v3');
|
||||
});
|
||||
|
||||
test('stampVerifiedMeta supports a human verification path', () => {
|
||||
const base = { title: 'T', status: 'GA', category: 'c',
|
||||
source: 'https://learn.microsoft.com/x', lastUpdated: '2026-06' };
|
||||
assert.equal(stampVerifiedMeta(base, { pass: true, by: 'human' }, '2026-06-29').verified_by, 'human');
|
||||
});
|
||||
|
||||
test('stampVerifiedMeta REFUSES to stamp when the judge verdict is not passing', () => {
|
||||
const base = { title: 'T', status: 'GA', category: 'c',
|
||||
source: 'https://learn.microsoft.com/x', lastUpdated: '2026-06' };
|
||||
assert.throws(() => stampVerifiedMeta(base, { pass: false }, '2026-06-29'), /judge|verdict|verified/i);
|
||||
assert.throws(() => stampVerifiedMeta(base, null, '2026-06-29'), /judge|verdict|verified/i);
|
||||
});
|
||||
|
||||
test('stampVerifiedMeta rejects an invalid today date', () => {
|
||||
const base = { title: 'T', status: 'GA', category: 'c',
|
||||
source: 'https://learn.microsoft.com/x', lastUpdated: '2026-06' };
|
||||
assert.throws(() => stampVerifiedMeta(base, { pass: true }, 'i går'), /date/i);
|
||||
});
|
||||
|
||||
test('stampVerifiedMeta → composeKbFile is a complete born-verified file (end-to-end)', () => {
|
||||
const base = { title: 'Azure AI Search', status: 'GA', category: 'rag-architecture',
|
||||
source: 'https://learn.microsoft.com/azure/search/x', lastUpdated: '2026-06' };
|
||||
const stamped = stampVerifiedMeta(base, { pass: true }, '2026-06-29');
|
||||
const file = composeKbFile(stamped, '## Introduksjon\n\nKort.\n');
|
||||
assert.equal(validateKbFile(file).valid, true);
|
||||
assert.equal(parseVerifiedHeader(file), '2026-06-29');
|
||||
assert.equal(parseVerifiedByHeader(file), 'judge-v2');
|
||||
});
|
||||
|
||||
// --- validateKbFile: catches files missing the mandatory contract -------------
|
||||
|
||||
test('validateKbFile accepts a header-conformant file', () => {
|
||||
|
|
|
|||
80
tests/kb-update/test-validate-kb-file.test.mjs
Normal file
80
tests/kb-update/test-validate-kb-file.test.mjs
Normal file
|
|
@ -0,0 +1,80 @@
|
|||
// tests/kb-update/test-validate-kb-file.test.mjs
|
||||
// The executable create-guard (Spor 3 Port 2): scripts/kb-update/validate-kb-file.mjs
|
||||
// is the gate any generator calls AFTER writing a candidate file and BEFORE committing.
|
||||
// It wraps transform.validateKbFile over one or more paths and exits non-zero if any
|
||||
// file violates the contract (no source / not born-verified / large file without a TOC).
|
||||
//
|
||||
// validatePaths is a pure function with an injectable reader, so these tests never
|
||||
// touch disk — they feed synthetic content and assert the verdict + ok flag.
|
||||
|
||||
import { test } from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { validatePaths } from '../../scripts/kb-update/validate-kb-file.mjs';
|
||||
import { buildKbHeader } from '../../scripts/kb-update/lib/transform.mjs';
|
||||
|
||||
const GOOD = buildKbHeader({
|
||||
title: 'Azure AI Search - Hybrid Retrieval',
|
||||
status: 'GA',
|
||||
category: 'rag-architecture',
|
||||
type: 'reference',
|
||||
source: 'https://learn.microsoft.com/azure/search/hybrid-search-overview',
|
||||
lastUpdated: '2026-06',
|
||||
verified: '2026-06-29',
|
||||
verified_by: 'judge-v2',
|
||||
}) + '\n## Introduksjon\n\nNoe innhold.\n';
|
||||
|
||||
// A reference file with a Source but NOT born-verified (no Verified / Verified by).
|
||||
const NOT_VERIFIED =
|
||||
'# T\n\n**Last updated:** 2026-06\n**Status:** GA\n**Type:** reference\n' +
|
||||
'**Source:** https://learn.microsoft.com/x\n\n---\n\n## A\n\ntekst\n';
|
||||
|
||||
// A reference file with no Source header at all (the 0%-coverage failure mode).
|
||||
const NO_SOURCE = '# T\n\n**Last updated:** 2026-06\n**Status:** GA\n\n---\n\n## A\n\ntekst\n';
|
||||
|
||||
function reader(map) {
|
||||
return (p) => {
|
||||
if (!(p in map)) throw new Error(`unexpected read: ${p}`);
|
||||
return map[p];
|
||||
};
|
||||
}
|
||||
|
||||
test('validatePaths — all files valid → ok:true', () => {
|
||||
const r = validatePaths(['a.md', 'b.md'], reader({ 'a.md': GOOD, 'b.md': GOOD }));
|
||||
assert.equal(r.ok, true);
|
||||
assert.equal(r.results.length, 2);
|
||||
assert.ok(r.results.every((x) => x.valid));
|
||||
});
|
||||
|
||||
test('validatePaths — a file missing source fails the gate', () => {
|
||||
const r = validatePaths(['bad.md'], reader({ 'bad.md': NO_SOURCE }));
|
||||
assert.equal(r.ok, false);
|
||||
assert.equal(r.results[0].valid, false);
|
||||
assert.ok(r.results[0].missing.includes('source'));
|
||||
});
|
||||
|
||||
test('validatePaths — a reference file that is not born-verified fails the gate', () => {
|
||||
const r = validatePaths(['nv.md'], reader({ 'nv.md': NOT_VERIFIED }));
|
||||
assert.equal(r.ok, false);
|
||||
assert.ok(r.results[0].missing.includes('verified'));
|
||||
assert.ok(r.results[0].missing.includes('verified_by'));
|
||||
});
|
||||
|
||||
test('validatePaths — one bad file in a batch fails the whole gate', () => {
|
||||
const r = validatePaths(['ok.md', 'bad.md'], reader({ 'ok.md': GOOD, 'bad.md': NO_SOURCE }));
|
||||
assert.equal(r.ok, false);
|
||||
assert.equal(r.results.find((x) => x.path === 'ok.md').valid, true);
|
||||
assert.equal(r.results.find((x) => x.path === 'bad.md').valid, false);
|
||||
});
|
||||
|
||||
test('validatePaths — a path that cannot be read is reported as invalid, not thrown', () => {
|
||||
const r = validatePaths(['missing.md'], () => { throw new Error('ENOENT'); });
|
||||
assert.equal(r.ok, false);
|
||||
assert.equal(r.results[0].valid, false);
|
||||
assert.ok(r.results[0].missing.some((m) => /read|enoent/i.test(m)));
|
||||
});
|
||||
|
||||
test('validatePaths — no paths → ok:true (nothing to gate)', () => {
|
||||
const r = validatePaths([], reader({}));
|
||||
assert.equal(r.ok, true);
|
||||
assert.deepEqual(r.results, []);
|
||||
});
|
||||
Loading…
Add table
Add a link
Reference in a new issue