feat(ms-ai-architect): G7 lukket som mekanisme — navngitt review-kø (form b), valgt på måling
Operatør-ratifisert 2026-08-03. G7 var gapet for korreksjoner som er RIKTIGE men større enn O2-konvolutten (én lokator + kun-sletting). Uten eier falt de stille ut: O2-triagen avviser dem, O3 dekker dem ikke, og review-sporet hadde ingen inngangskø. Formvalget ble tatt på §9.6-målingen, ikke på preferanse: - 2 av 4 anvendte subtraksjoner etterlot en rest -> rester er delete-only- konvoluttens NORMALE biprodukt, ikke et unntak. - 2 av 5 medlemmer (idx 17, 33) er ERSTATNINGER, ikke fler-lokator. En delete-orientert O4-klasse med egen retur-kontrakt ville ikke fikset dem — altså feil dimensjonert mot evidensen. Køen absorberer begge klasser. Artefakter (TDD, test-først): - data/g7-review-queue.json — 6 entries, tracked - lib/g7-queue.mjs — validateQueue, lukkede vokabular - check-g7-queue.mjs — exit 1 ved drift eller skjemafeil - tests/kb-eval/test-g7-queue.test.mjs — 15 tester Kontrakten: ankere er ORDRETTE strenger, aldri linjenummer (line != real_line i 9 av 17 R11-records). En åpen entry hvis anker slutter å matche gir anchor_drift og exit 1 — den kan ikke falle stille ut, som er hele hensikten. En resolved entry MÅ føre resolution, ellers er "resolved" ikke til å skille fra "stille droppet". Ingenting i køen er maskin-anvendbart per definisjon. Innhold: 5 åpne (26, 27, 33, 36, 18), 1 lukket (17). idx 27 kom hit ved å falle ut av O2 på cond 3; idx 26 ved operatørens avvisning av delvis fiks. Suite 1047/1047. [skip-docs]
This commit is contained in:
parent
c569bdc10e
commit
59c6c280b1
5 changed files with 379 additions and 1 deletions
|
|
@ -106,7 +106,7 @@ Status-nøkkel: 🔴 ikke startet · 🟡 pågår · 🟢 lukket.
|
|||
|
||||
| **G6** | Ingen sikkerhetsgate på ingestion-kjeden (hentet eksternt innhold → korpus): llm-security-pluginen er **deaktivert globalt** (verifisert 2026-07-03 i `~/.claude/settings.json`), så `post-mcp-verify`-hooken (injection-skann på all tool-output, inkl. `microsoft_docs_fetch`) fyrer ikke; commit-gaten dekker kun secrets (gitleaks), ikke injeksjon/steganografi i `.md`-innhold. Tillit til MS Learn dekker faktisk korrekthet — ikke adversarielt innhold i kanalen eller i kodeeksempler/lokalisert stoff | Indirekte prompt-injeksjon/steganografi persistert i offentlig distribuert KB: references-filene blir instruksjonsnær kontekst i fremtidige agent-sesjoner, én forgiftet fil re-serveres til alle brukere (hele Norge) | **R6-briefen designer gaten, to lag** (`docs/ingestion-security-brief-2026-07.md`, committet 2026-07-04): (a) llm-security AKTIV + verifisert fyrende i enhver fetch-økt (kb-update, research, generate-skills, judge-pass); headless-caveat GH #36071 → foreground eller kompenserende skann; (b) deterministisk node-skann (unicode/decode/injection — de DELTE llm-security-detektorene importert in-process, ikke kopiert; operatør-valg 2026-07-04) over endrede `skills/**/*.md` før commit. Håndheves fra R7 og i kb-update-kadensen | 🟢 **Layer B (b) LUKKET 2026-07-04 (TDD).** Bærende gaten bygget: `scan-adversarial-content.mjs` (+ `lib/adversarial-scan.mjs` disposition-kjerne, `lib/adversarial-detect.mjs` llm-security-bro), wiret som sibling til `validate-kb-file.mjs` ved det eneste skrive-chokepunktet (kb-update §3b.d.7/§4/§5 + generate-skills per-batch/pre-commit). Provenance-tiered BLOCK/WARN; injection-flagg → samme menneske-i-loop som status-påstand. 30 tester (692/0). Premiss-korr.: research/research-agent skriver ingenting (kun Layer A); CLI-scan alene misset injection+base64 → importerer rene primitiver. **Baseline-adjudikering (Enhet A2) LUKKET 2026-07-18 (TDD):** korpus-baseline 55/389 flagget (83 funn) → 4 ekte defekter fikset (2 Cyrillic-homoglyph-ord, 2 filer med U+00AD) + 75 funn human-adjudikert inn i innholds-basert allowlist (`scripts/kb-update/data/layerb-allowlist.json`: class+evidence+tier+eksakt trimmet linjeinnhold per entry, med begrunnelse; flyttet linje forblir grønn, endret innhold GJENOPPSTÅR som flagg — scanneren selv usvekket, korrupt/manglende allowlist → tom = full strenghet). Korpus 389/389 OK exit 0; commit-gaten (pre-commit-scan) leser samme allowlist. **Layer A (a) LUKKET 2026-07-18 (Enhet B):** `post-mcp-verify` aktivert kirurgisk i `~/.claude/settings.json` (PostToolUse-matcher `mcp__microsoft-learn__.*`) + live-verifisert fyrende med 2 foreground `microsoft_docs_fetch` — og en **ekte hook-defekt funnet+fikset underveis** (hooken leste `tool_output`, live-protokollen sender `tool_response` → hooken skannet ingenting; fiks TDD i llm-security `44aa390` v7.8.3). Se lukke-logg. | Layer A: R7 (første judge-pass-fetch-økt) / enhver `/architect:kb-update`/`generate-skills`-fetch-økt |
|
||||
|
||||
| **G7** | **Ingen rute for korreksjoner som er RIKTIGE, men større enn O2-konvolutten.** O2 er definert som én lokator + kun-sletting. R11 §9.3/§9.4 produserte fire funn der den korrekte fiksen beviselig ligger utenfor: idx 18 (`rag-caching-optimization.md:29` — den overlevende påstanden er en hel titulert seksjon 303-318 **pluss** en `**Verified**`-rad på 510; ingen sletting begrenset til linje 29 kan reparere fila), idx 36 (`ai-threat-modeling-stride.md:38` — companion-edit på 310 kreves for at prosaen skal matche den innsnevrede severity-tabellen), idx 17 (`**Verified**`-stemplet på 258 stempler etter editen kun retnings-utsagnet), idx 33 (innholdet overlever på 357 under CAF-attribusjon, så editens gevinst er mindre enn den ser ut). Alle fire er i dag kun prosa i `r11-pilot-results.md` | **Sanne defekter som stille faller ut av programmet fordi ingen mekanisme eier dem.** O2-triagen avviser dem (utenfor konvolutt), O3 dekker dem ikke (fiksen er ikke en verdi-swap), og det menneskelige review-sporet har ingen inngangskø. Nettoeffekten er at den *vanskeligste* klassen — der fila motsier seg selv — er den eneste uten eier | **Ikke designet.** Minimum: en eksplisitt fler-lokator-klasse (O4?) med egen retur-kontrakt og egen maskin-sjekk, ELLER en navngitt kø inn i den menneskelige review-fasen. Beslutningen henger sammen med ÅPEN OPERATØRBESLUTNING #2 (lønner korpus-bred O2-klassifisering seg) — hvis svaret er «hopp til menneskelig review», er G7 samme kø | 🔴 **ikke startet** (reist 2026-08-03, R11 §9.4) | Før R11s menneskelige review-fase erklæres ferdig — ellers er de fire funnene tapt |
|
||||
| **G7** | **Ingen rute for korreksjoner som er RIKTIGE, men større enn O2-konvolutten.** O2 er definert som én lokator + kun-sletting. R11 §9.3/§9.4 produserte fire funn der den korrekte fiksen beviselig ligger utenfor: idx 18 (`rag-caching-optimization.md:29` — den overlevende påstanden er en hel titulert seksjon 303-318 **pluss** en `**Verified**`-rad på 510; ingen sletting begrenset til linje 29 kan reparere fila), idx 36 (`ai-threat-modeling-stride.md:38` — companion-edit på 310 kreves for at prosaen skal matche den innsnevrede severity-tabellen), idx 17 (`**Verified**`-stemplet på 258 stempler etter editen kun retnings-utsagnet), idx 33 (innholdet overlever på 357 under CAF-attribusjon, så editens gevinst er mindre enn den ser ut). Alle fire er i dag kun prosa i `r11-pilot-results.md` | **Sanne defekter som stille faller ut av programmet fordi ingen mekanisme eier dem.** O2-triagen avviser dem (utenfor konvolutt), O3 dekker dem ikke (fiksen er ikke en verdi-swap), og det menneskelige review-sporet har ingen inngangskø. Nettoeffekten er at den *vanskeligste* klassen — der fila motsier seg selv — er den eneste uten eier | **DESIGNET + BYGGET 2026-08-03 (form b — navngitt kø).** Valget ble tatt på måling, ikke på form-preferanse: av de fire subtraksjonene i `957ebef` etterlot **to** en rest (§9.6), så rester er delete-only-konvoluttens normale biprodukt, ikke et unntak. Og **to av de fem medlemmene (idx 17, 33) er erstatninger, ikke fler-lokator** — en delete-orientert O4-klasse med egen retur-kontrakt ville ikke fikset dem, altså vært feil dimensjonert mot evidensen. Køen absorberer begge klasser. Artefakter: `scripts/kb-eval/data/g7-review-queue.json` (tracked, 6 entries) + `lib/g7-queue.mjs` + `check-g7-queue.mjs` + 15 tester. **Ankere er ordrette strenger, ALDRI linjenummer** (`line` ≠ `real_line` i 9 av 17 R11-records); en åpen entry hvis anker slutter å matche gir `anchor_drift` og exit 1 — den kan ikke falle stille ut. Ingenting i køen er maskin-anvendbart per definisjon; lukking er en menneskelig review-handling som MÅ føre `resolution`. Kobles fra ÅPEN OPERATØRBESLUTNING #2: køen står uansett hvordan den lander | 🟡 **pågår — mekanismen står, køen er ikke tømt.** 5 åpne (idx 26, 27, 33, 36, 18), 1 lukket (idx 17: dinglende ledetekst → kolon-til-punktum, operatør-ratifisert). idx 27 kom hit ved å FALLE UT av O2 på cond 3 (§9.6), idx 26 ved operatørens avvisning av delvis fiks | Før R11s menneskelige review-fase erklæres ferdig. **Mekanismen er nå lukke-vilkåret oppfylt for; det som gjenstår er innholdet i køen** |
|
||||
|
||||
**Ikke mekanisme-gap, men sporet backlog (innhold, ikke loop):** reference-`.md`-fil-fiksene fra Spor 2b (FP1 11000+/40+, FP2 «kun», FP6 Preview/Norway-East, FN2–FN6 utdaterte tall) **+ G5b** (`adr-template.md` fjern «zero permission management»; `multi-region-azure-openai-deployment.md` bytt retired `gpt-35-turbo` → gjeldende modell; `network-resilience-patterns-ai.md` «obligatorisk» → «anbefalt»; `vector-storage-cost-optimization.md` GA-dato `2024-11-01` → `2024-07-01`) er **Spor 0/1**-innholdsarbeid — pekt per-claim i `notes`, ikke gjentakelses-mekanisme. Føres i Spor 0-manifest / Spor 1-korpus-pass, ikke her.
|
||||
|
||||
|
|
|
|||
64
scripts/kb-eval/check-g7-queue.mjs
Normal file
64
scripts/kb-eval/check-g7-queue.mjs
Normal file
|
|
@ -0,0 +1,64 @@
|
|||
#!/usr/bin/env node
|
||||
/**
|
||||
* Check the G7 review queue against the live corpus.
|
||||
*
|
||||
* Exit 0 = every open entry still anchors to real text; exit 1 = drift or a
|
||||
* schema fault. Drift is a finding, never a silent pass: an entry that stops
|
||||
* matching is exactly the case G7 exists to prevent — a real defect leaving the
|
||||
* programme unnoticed because someone edited around it.
|
||||
*
|
||||
* Read-only. Nothing in the queue is machine-appliable; resolution is a human
|
||||
* review act (form b, ratified 2026-08-03).
|
||||
*
|
||||
* node scripts/kb-eval/check-g7-queue.mjs [--json]
|
||||
*/
|
||||
import { readFileSync } from 'node:fs';
|
||||
import { validateQueue } from './lib/g7-queue.mjs';
|
||||
|
||||
const QUEUE = 'scripts/kb-eval/data/g7-review-queue.json';
|
||||
|
||||
const asJson = process.argv.includes('--json');
|
||||
|
||||
let queue;
|
||||
try {
|
||||
queue = JSON.parse(readFileSync(QUEUE, 'utf8'));
|
||||
} catch (err) {
|
||||
console.error(`cannot read ${QUEUE}: ${err.message}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const entries = queue.entries ?? [];
|
||||
const { ok, findings } = validateQueue(entries, (p) => readFileSync(p, 'utf8'));
|
||||
|
||||
const open = entries.filter((e) => e.status === 'open');
|
||||
const resolved = entries.filter((e) => e.status === 'resolved');
|
||||
|
||||
if (asJson) {
|
||||
console.log(JSON.stringify({ ok, open: open.length, resolved: resolved.length, findings }, null, 2));
|
||||
process.exit(ok ? 0 : 1);
|
||||
}
|
||||
|
||||
console.log(`G7 review queue — ${open.length} open, ${resolved.length} resolved\n`);
|
||||
|
||||
const byClass = (cls) => open.filter((e) => e.class === cls);
|
||||
for (const cls of ['multi-locator', 'replacement']) {
|
||||
const rows = byClass(cls);
|
||||
if (rows.length === 0) continue;
|
||||
console.log(` ${cls} (${rows.length}):`);
|
||||
for (const e of rows) {
|
||||
console.log(` ${e.id.padEnd(8)} ${e.file.split('/').pop()}`);
|
||||
}
|
||||
console.log();
|
||||
}
|
||||
|
||||
if (findings.length > 0) {
|
||||
console.log(`FINDINGS (${findings.length}):`);
|
||||
for (const f of findings) {
|
||||
console.log(` [${f.kind}] ${f.id ?? ''} ${f.message}`);
|
||||
}
|
||||
console.log('\nA drifted anchor means the file changed under a queued defect.');
|
||||
console.log('Re-derive the anchor from the live file — do not delete the entry.');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
console.log('All open entries still anchor to live corpus text. exit 0');
|
||||
86
scripts/kb-eval/data/g7-review-queue.json
Normal file
86
scripts/kb-eval/data/g7-review-queue.json
Normal file
|
|
@ -0,0 +1,86 @@
|
|||
{
|
||||
"_meta": {
|
||||
"gap": "G7",
|
||||
"form": "(b) named queue into the human review phase",
|
||||
"ratified": "2026-08-03",
|
||||
"rationale": "Measured in R11 §9.6: 2 of the 4 subtractions applied in 957ebef left a residue, so residues are the normal by-product of a delete-only envelope rather than an exception. Two of the members are replacements, not multi-locator cases, which a deletion-oriented O4 class would not have fixed. A queue absorbs both classes; an O4 return contract would have been mis-sized against the evidence.",
|
||||
"contract": "Anchors are verbatim strings, never line numbers (line ≠ real_line in 9 of 17 R11 records). An open entry whose anchor no longer occurs in its file is drift, and check-g7-queue.mjs fails rather than passing it silently. Nothing in this queue is machine-appliable by definition — every entry is outside the O2 envelope. Resolution is a human review act.",
|
||||
"evidence": "docs/r11-pilot-results.md §9.4, §9.5, §9.6; docs/ref-kb-correctness-program-2026-06.md §8 G7"
|
||||
},
|
||||
"entries": [
|
||||
{
|
||||
"id": "idx-17",
|
||||
"file": "skills/ms-ai-engineering/references/rag-architecture/rag-caching-optimization.md",
|
||||
"class": "replacement",
|
||||
"status": "resolved",
|
||||
"raised": "2026-08-03",
|
||||
"summary": "The idx 17 subtraction (957ebef) deleted the three score-threshold bands but left the lead-in ending in a colon, promising an enumeration that no longer existed, immediately followed by a **Verified** stamp. The subtraction was correct; the paragraph it left was not. V1/V2/V2b/V3 are string invariants over deleted text and cannot see document coherence, so the machine could not have caught it.",
|
||||
"evidence": "docs/r11-pilot-results.md §9.6",
|
||||
"anchors": [],
|
||||
"resolution": "Operator-ratified 2026-08-03: colon changed to a period, making the lead-in a complete and independently true sentence that the **Verified** stamp correctly covers. Corpus swept for the same defect shape (bold lead-in ending in colon, blank line, **Verified**) — no other occurrence."
|
||||
},
|
||||
{
|
||||
"id": "idx-33",
|
||||
"file": "skills/ms-ai-security/references/ai-security-engineering/ai-threat-modeling-stride.md",
|
||||
"class": "replacement",
|
||||
"status": "open",
|
||||
"raised": "2026-08-03",
|
||||
"summary": "The idx 33 subtraction removed AI asset inventory via Azure Resource Graph and the Purview Insider Risk Management bullet from an unsupported Defender for Cloud AISPM attribution. Correct — but the same capabilities survive in this file under Cloud Adoption Framework Secure AI attribution, stamped 'Verified MCP 2026-04', and are asserted in four other corpus files. The edit's benefit is corpus-wide smaller than the single line suggested. Whether the CAF attribution is itself supported has not been checked.",
|
||||
"evidence": "docs/r11-pilot-results.md §9.5 (cross-corpus check), §9.6",
|
||||
"anchors": [
|
||||
"Oppdatert 2026-04: inkluderer nå AI asset inventory via Azure Resource Graph"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "idx-26",
|
||||
"file": "skills/ms-ai-governance/references/responsible-ai/transparency-documentation-standards.md",
|
||||
"class": "multi-locator",
|
||||
"status": "open",
|
||||
"raised": "2026-08-03",
|
||||
"summary": "The Responsible AI Scorecard component list names Error analysis (item 4) and Counterfactual analysis (item 5). Both were measured false against first-party docs 2026-08-03: the canonical scorecard segments are summary/model overview, data analysis, model performance, cohorts, top important factors, fairness insights and causal insights; Error analysis and Counterfactual analysis are Responsible AI *dashboard* components. The delete-only reduction could only remove item 5, because line 300 asserts Error analysis as scorecard content too — so a partial fix would have left a known-false claim standing while introducing a renumbering artifact (1,2,3,4,6,7). Operator declined the partial fix 2026-08-03 and sent the whole case here. Correct repair spans both locators.",
|
||||
"evidence": "docs/r11-pilot-results.md §9.6; https://learn.microsoft.com/azure/machine-learning/how-to-responsible-ai-scorecard",
|
||||
"anchors": [
|
||||
"4. **Error analysis**: Error rates per cohort, confusion matrices",
|
||||
"| **Risk assessment** | Responsible AI Scorecard: Error analysis, fairness assessment |"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "idx-27",
|
||||
"file": "skills/ms-ai-governance/references/responsible-ai/transparency-documentation-standards.md",
|
||||
"class": "replacement",
|
||||
"status": "open",
|
||||
"raised": "2026-08-03",
|
||||
"summary": "Failed out of O2 on cond 3 (§9.6): the cited source DOES establish the mechanism — faqs-generative-orchestration states 'Makers can require user confirmation before executing tools that modify data' — so deleting the Plugin-actions row would destroy source-confirmed information. The defect is modality, not fabrication: the file presents confirmation prompts as a built-in disclosure, whereas the source makes them maker-configured. The Chat-interface row in the same table is imprecise for the same reason: the FAQ documents a default transparency message ('Just so you are aware, I sometimes use AI to answer your questions.'), not a 'Powered by AI' badge. Repair is a replacement, outside the delete-only envelope.",
|
||||
"evidence": "docs/r11-pilot-results.md §9.6; https://learn.microsoft.com/microsoft-copilot-studio/faqs-generative-orchestration",
|
||||
"anchors": [
|
||||
"| **Plugin actions** | Confirmation prompts før sensitive actions (send email, delete file) |",
|
||||
"| **Chat interface** | \"Powered by AI\" badge i chat window |"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "idx-36",
|
||||
"file": "skills/ms-ai-security/references/ai-security-engineering/ai-threat-modeling-stride.md",
|
||||
"class": "multi-locator",
|
||||
"status": "open",
|
||||
"raised": "2026-08-03",
|
||||
"summary": "Applying idx 36 alone yields a severity table more precise than the prose that cites it, so a companion edit is required for the prose to match the narrowed table. Held back from 957ebef as out of envelope.",
|
||||
"evidence": "docs/r11-pilot-results.md §9.4, §9.5",
|
||||
"anchors": [
|
||||
"Øker severity bar; krever mer robust adversarial defenses"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "idx-18",
|
||||
"file": "skills/ms-ai-engineering/references/rag-architecture/rag-caching-optimization.md",
|
||||
"class": "multi-locator",
|
||||
"status": "open",
|
||||
"raised": "2026-08-03",
|
||||
"summary": "No reduction exists. The surviving claim is a whole titled section on Azure AI Search built-in caching plus a **Verified** row in the verification table, so no deletion confined to a single locator can repair the file. This is the member that most clearly motivated G7.",
|
||||
"evidence": "docs/r11-pilot-results.md §9.3, §9.4",
|
||||
"anchors": [
|
||||
"**Automatic Caching Behavior:**",
|
||||
"| Azure AI Search caching | **Verified** | Microsoft Learn docs (4, 6) |"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
100
scripts/kb-eval/lib/g7-queue.mjs
Normal file
100
scripts/kb-eval/lib/g7-queue.mjs
Normal file
|
|
@ -0,0 +1,100 @@
|
|||
/**
|
||||
* G7 review queue — the named queue into the human review phase.
|
||||
*
|
||||
* G7 is the gap for corrections that are RIGHT but larger than the O2 envelope
|
||||
* (one locator + deletion-only). Measured in R11 §9.6: of the four subtractions
|
||||
* applied in 957ebef, two left a residue. Residues are the normal by-product of
|
||||
* a delete-only envelope, not an exception — and two of the five members are
|
||||
* replacements rather than multi-locator cases, which a deletion-oriented O4
|
||||
* class would not have fixed. Hence a queue (form b), ratified 2026-08-03.
|
||||
*
|
||||
* This module validates the queue. It deliberately does NOT apply anything:
|
||||
* every entry here is by definition outside the machine-appliable envelope.
|
||||
*
|
||||
* Anchors are verbatim strings, never line numbers — line numbers drift, and
|
||||
* §9.4 measured `line` ≠ `real_line` in 9 of 17 records. An open entry whose
|
||||
* anchor no longer occurs is reported as drift rather than quietly passing,
|
||||
* because the whole purpose of the queue is that a defect cannot fall out of
|
||||
* the programme unnoticed.
|
||||
*/
|
||||
|
||||
export const ENTRY_CLASSES = new Set(['multi-locator', 'replacement']);
|
||||
export const ENTRY_STATES = new Set(['open', 'resolved']);
|
||||
|
||||
const REQUIRED = ['id', 'file', 'class', 'status', 'raised', 'summary', 'evidence', 'anchors'];
|
||||
|
||||
/**
|
||||
* @param {unknown} entries queue entries
|
||||
* @param {(path: string) => string} readFile
|
||||
* @returns {{ ok: boolean, findings: Array<{id?: string, kind: string, message: string}> }}
|
||||
*/
|
||||
export function validateQueue(entries, readFile) {
|
||||
const findings = [];
|
||||
|
||||
if (!Array.isArray(entries)) {
|
||||
return { ok: false, findings: [{ kind: 'schema', message: 'queue must be an array of entries' }] };
|
||||
}
|
||||
|
||||
const seen = new Set();
|
||||
|
||||
for (const entry of entries) {
|
||||
const id = typeof entry?.id === 'string' ? entry.id : '<no id>';
|
||||
|
||||
const missing = REQUIRED.filter((f) => entry?.[f] === undefined);
|
||||
if (missing.length > 0) {
|
||||
findings.push({ id, kind: 'schema', message: `missing required field(s): ${missing.join(', ')}` });
|
||||
continue;
|
||||
}
|
||||
|
||||
if (seen.has(entry.id)) {
|
||||
findings.push({ id, kind: 'schema', message: `duplicate id: ${entry.id}` });
|
||||
continue;
|
||||
}
|
||||
seen.add(entry.id);
|
||||
|
||||
if (!ENTRY_CLASSES.has(entry.class)) {
|
||||
findings.push({ id, kind: 'schema', message: `unknown class: ${entry.class}` });
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!ENTRY_STATES.has(entry.status)) {
|
||||
findings.push({ id, kind: 'schema', message: `unknown status: ${entry.status}` });
|
||||
continue;
|
||||
}
|
||||
|
||||
// A resolved entry has to say what closed it. Without that, "resolved" is
|
||||
// indistinguishable from "quietly dropped" — the exact failure G7 exists to
|
||||
// prevent. Resolved entries are exempt from anchor checking, since a real
|
||||
// fix is expected to have changed the text the anchor pointed at.
|
||||
if (entry.status === 'resolved') {
|
||||
if (typeof entry.resolution !== 'string' || entry.resolution.trim() === '') {
|
||||
findings.push({ id, kind: 'schema', message: 'a resolved entry must record a resolution' });
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!Array.isArray(entry.anchors) || entry.anchors.length === 0) {
|
||||
findings.push({ id, kind: 'schema', message: 'an open entry must carry at least one anchor' });
|
||||
continue;
|
||||
}
|
||||
|
||||
let text;
|
||||
try {
|
||||
text = readFile(entry.file);
|
||||
} catch {
|
||||
findings.push({ id, kind: 'file_unreadable', message: `cannot read ${entry.file}` });
|
||||
continue;
|
||||
}
|
||||
|
||||
const gone = entry.anchors.filter((a) => !text.includes(a));
|
||||
if (gone.length > 0) {
|
||||
findings.push({
|
||||
id,
|
||||
kind: 'anchor_drift',
|
||||
message: `anchor no longer occurs in ${entry.file}: ${gone.map((g) => JSON.stringify(g)).join(', ')}`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return { ok: findings.length === 0, findings };
|
||||
}
|
||||
128
tests/kb-eval/test-g7-queue.test.mjs
Normal file
128
tests/kb-eval/test-g7-queue.test.mjs
Normal file
|
|
@ -0,0 +1,128 @@
|
|||
import { test } from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { validateQueue, ENTRY_CLASSES, ENTRY_STATES } from '../../scripts/kb-eval/lib/g7-queue.mjs';
|
||||
|
||||
// A queue entry is the ONLY thing standing between a real defect and silent loss,
|
||||
// so the checks here are deliberately unforgiving about shape and about anchors
|
||||
// that no longer exist in the corpus.
|
||||
|
||||
const okEntry = () => ({
|
||||
id: 'idx-99',
|
||||
file: 'skills/x/references/y.md',
|
||||
class: 'multi-locator',
|
||||
status: 'open',
|
||||
raised: '2026-08-03',
|
||||
summary: 'Something the file asserts in two places.',
|
||||
evidence: 'docs/r11-pilot-results.md §9.6',
|
||||
anchors: ['the exact text'],
|
||||
});
|
||||
|
||||
const stub = (contents) => (p) => {
|
||||
if (!(p in contents)) throw new Error(`ENOENT ${p}`);
|
||||
return contents[p];
|
||||
};
|
||||
|
||||
test('a well-formed open entry whose anchor is present passes', () => {
|
||||
const r = validateQueue([okEntry()], stub({ 'skills/x/references/y.md': 'aaa the exact text bbb' }));
|
||||
assert.equal(r.ok, true);
|
||||
assert.deepEqual(r.findings, []);
|
||||
});
|
||||
|
||||
test('vocabularies are closed', () => {
|
||||
assert.deepEqual([...ENTRY_CLASSES].sort(), ['multi-locator', 'replacement']);
|
||||
assert.deepEqual([...ENTRY_STATES].sort(), ['open', 'resolved']);
|
||||
});
|
||||
|
||||
test('unknown class is rejected', () => {
|
||||
const e = { ...okEntry(), class: 'deletion' };
|
||||
const r = validateQueue([e], stub({ 'skills/x/references/y.md': 'the exact text' }));
|
||||
assert.equal(r.ok, false);
|
||||
assert.match(r.findings[0].message, /class/);
|
||||
});
|
||||
|
||||
test('unknown status is rejected', () => {
|
||||
const e = { ...okEntry(), status: 'wontfix' };
|
||||
const r = validateQueue([e], stub({ 'skills/x/references/y.md': 'the exact text' }));
|
||||
assert.equal(r.ok, false);
|
||||
assert.match(r.findings[0].message, /status/);
|
||||
});
|
||||
|
||||
test('a missing required field is reported by name', () => {
|
||||
const e = okEntry();
|
||||
delete e.evidence;
|
||||
const r = validateQueue([e], stub({ 'skills/x/references/y.md': 'the exact text' }));
|
||||
assert.equal(r.ok, false);
|
||||
assert.match(r.findings[0].message, /evidence/);
|
||||
});
|
||||
|
||||
test('an open entry MUST carry at least one anchor', () => {
|
||||
const e = { ...okEntry(), anchors: [] };
|
||||
const r = validateQueue([e], stub({ 'skills/x/references/y.md': 'the exact text' }));
|
||||
assert.equal(r.ok, false);
|
||||
assert.match(r.findings[0].message, /anchor/i);
|
||||
});
|
||||
|
||||
// The point of the whole mechanism: a defect must not be able to leave the queue
|
||||
// just because someone edited the file around it.
|
||||
test('an open entry whose anchor no longer occurs is a DRIFT finding, not a pass', () => {
|
||||
const r = validateQueue([okEntry()], stub({ 'skills/x/references/y.md': 'nothing matching here' }));
|
||||
assert.equal(r.ok, false);
|
||||
assert.equal(r.findings[0].kind, 'anchor_drift');
|
||||
assert.match(r.findings[0].message, /the exact text/);
|
||||
});
|
||||
|
||||
test('every anchor is checked, not just the first', () => {
|
||||
const e = { ...okEntry(), anchors: ['present', 'absent'] };
|
||||
const r = validateQueue([e], stub({ 'skills/x/references/y.md': 'present only' }));
|
||||
assert.equal(r.ok, false);
|
||||
assert.equal(r.findings.length, 1);
|
||||
assert.match(r.findings[0].message, /absent/);
|
||||
});
|
||||
|
||||
test('a missing file is reported rather than thrown', () => {
|
||||
const r = validateQueue([okEntry()], stub({}));
|
||||
assert.equal(r.ok, false);
|
||||
assert.equal(r.findings[0].kind, 'file_unreadable');
|
||||
});
|
||||
|
||||
test('a resolved entry MUST record how it was resolved', () => {
|
||||
const e = { ...okEntry(), status: 'resolved' };
|
||||
const r = validateQueue([e], stub({ 'skills/x/references/y.md': 'whatever' }));
|
||||
assert.equal(r.ok, false);
|
||||
assert.match(r.findings[0].message, /resolution/);
|
||||
});
|
||||
|
||||
test('a resolved entry is exempt from anchor checking', () => {
|
||||
const e = { ...okEntry(), status: 'resolved', resolution: 'colon -> period, ratified 2026-08-03' };
|
||||
const r = validateQueue([e], stub({ 'skills/x/references/y.md': 'anchor is long gone' }));
|
||||
assert.equal(r.ok, true);
|
||||
});
|
||||
|
||||
test('duplicate ids are rejected', () => {
|
||||
const r = validateQueue([okEntry(), okEntry()], stub({ 'skills/x/references/y.md': 'the exact text' }));
|
||||
assert.equal(r.ok, false);
|
||||
assert.match(r.findings[0].message, /duplicate/i);
|
||||
});
|
||||
|
||||
test('findings from several entries are all reported', () => {
|
||||
const a = { ...okEntry(), id: 'idx-1' };
|
||||
const b = { ...okEntry(), id: 'idx-2', anchors: ['gone'] };
|
||||
const r = validateQueue([a, b], stub({ 'skills/x/references/y.md': 'the exact text' }));
|
||||
assert.equal(r.ok, false);
|
||||
assert.equal(r.findings.length, 1);
|
||||
assert.equal(r.findings[0].id, 'idx-2');
|
||||
});
|
||||
|
||||
test('a non-array queue is rejected without throwing', () => {
|
||||
const r = validateQueue({ entries: [] }, stub({}));
|
||||
assert.equal(r.ok, false);
|
||||
assert.match(r.findings[0].message, /array/i);
|
||||
});
|
||||
|
||||
test('the real queue file validates against the live corpus', async () => {
|
||||
const { readFileSync } = await import('node:fs');
|
||||
const queue = JSON.parse(readFileSync('scripts/kb-eval/data/g7-review-queue.json', 'utf8'));
|
||||
const r = validateQueue(queue.entries, (p) => readFileSync(p, 'utf8'));
|
||||
assert.deepEqual(r.findings, [], `G7 queue drifted: ${JSON.stringify(r.findings, null, 2)}`);
|
||||
assert.equal(r.ok, true);
|
||||
});
|
||||
Loading…
Add table
Add a link
Reference in a new issue