feat(ms-ai-architect): R11 §4b implementert — status-synonymtabellen målt, 8 forslag hånd-dømt [skip-docs]

Implementerer den ratifiserte §4b-tabellen i lib/fix-op.mjs (19 nye tester,
suite 996/996). Alle tre skrankene har egne tester: tabellen er LUKKET, fil-
tokenet må være en KOMPLETT livssyklus-etikett, og verdien som skrives er den
korpus-side ekvivalenten med filas egen markup bevart.

To implementasjonsvalg den ratifiserte teksten lot stå åpne, begge løst mot
fail-closed: status-lokatoren er LINJE-scopet (livssyklus-vokabular gjentas
nedover hver kolonne i en statustabell, så et blokkvindu er tvetydig ved
konstruksjon), og et sitat som hevder to rader aborterer.

MÅLT: 15 pilot / 54 korpus-brede flagg -> 5 og 8 provbare. Alle 8 hånd-dømt
mot kilden (r11-pilot-results.md appendiks B): 5 korrekte, 1 ubevist, 2 GALE.

De tre defektene er én familie: §4b binder tabellen, etikettens fullstendighet
og verdien som skrives — og INGENTING om hvorvidt kilde-frasen refererer til
radens eget subjekt. Samme proveniens-uten-referent-defekt som falsifiserte §4.
Klassen er derfor REVIEW-grade, ikke apply-grade: `status` står bevisst utenfor
o1_recommended, ingen driver applikerer den.

To kandidatvilkår er kostnadsberegnet over de åtte (begge dreper gale forslag
og null korrekte) men IKKE implementert — å utvide en tabell operatøren
ratifiserte som lukket er en operatørbeslutning, slik vilkår 5 var i §4a.

Rettet samtidig 2 NUL-bytes i testfila (pre-eksisterende, fra en tidligere
økt) som gjorde at git behandlet hele fila som binær og blokkerte diff-
gjennomgang før commit.
This commit is contained in:
Kjell Tore Guttormsen 2026-08-03 17:12:47 +02:00
commit e4925c6b28
5 changed files with 330 additions and 10 deletions

View file

@ -4,7 +4,7 @@
the live ledger. No KB file was edited and no ledger record was written.**
Instrument: `scripts/kb-eval/lib/fix-op.mjs` (+ `tests/kb-eval/test-fix-op-classify.test.mjs`,
30 tests) driven by `scripts/kb-eval/classify-fix-ops.mjs`. The classifier **is**
49 tests after §4b) driven by `scripts/kb-eval/classify-fix-ops.mjs`. The classifier **is**
the O1 driver with writes disabled — it constructs the swap and checks the §4
invariant, so measurements 1 and 3 come out of the mechanism that would later
touch the corpus, not out of a proxy heuristic.
@ -32,6 +32,12 @@ other flag needs a human.
| Pilot (`not_grounded` ≥ 7) | 24 | 202 | 2 | 2 |
| Whole `not_grounded` corpus | 218 | 776 | 15 | **9** |
> These are **numeric-path** figures, and they stay that way. §4b (the status
> synonym table) was implemented afterwards and adds a separate class with its own
> hand-verification — see §8. A run today prints O1 = 7 (pilot) and 23 (corpus)
> because the status proposals are included in the total; the numeric line above
> is unchanged and is still what `s4_as_written` compares against.
Reproduce: `node scripts/kb-eval/classify-fix-ops.mjs` (pilot) and
`node scripts/kb-eval/classify-fix-ops.mjs --threshold 1` (corpus). The nine are
enumerated with verdicts in appendix A — that hand-verification is the only thing
@ -178,6 +184,54 @@ nine-item machine assist, and its leverage lies entirely in the O2 decision.
---
## 8. §4b implemented — the status class measured, 2026-08-03
The ratified synonym table (`docs/r11-tiered-fix-design.md` §4b) is implemented in
`fix-op.mjs` and the class is measured. Reproduce:
`node scripts/kb-eval/classify-fix-ops.mjs --threshold 1` (`status_synonym` block).
| Population | STATUS_SYNONYM flags | Proven by §4b | Still aborting |
|---|---|---|---|
| Pilot (`not_grounded` ≥ 7) | 15 | **5** | 10 |
| Whole `not_grounded` corpus | 54 | **8** | 46 |
Why the other 46 abort, corpus-wide — this sub-distribution is the actionable
part, because the top-level `STATUS_SYNONYM` count alone says nothing:
| Reason | N | What it means |
|---|---|---|
| `NO_COMPLETE_FILE_LABEL` | 27 | the file writes the status inside a sentence — `(preview)` in a list item, `**DSPM (preview):**`, `"[Preview]: …"` in a JSON string. Constraint 2 refuses these, correctly. |
| `NO_SOURCE_STATUS` | 15 | the cited quote carries no listed lifecycle phrasing at all — the flag was never a status swap. |
| `SOURCE_STATUS_AMBIGUOUS` | 2 | the quote asserts two different rows (e.g. "…is now generally available. Partner solutions remain in preview."). |
| `FILE_ALREADY_MATCHES` | 2 | file and source agree; the mismatch was in the LLM-extracted claim, not in the corpus. |
**The class is REVIEW-grade, not apply-grade — 5 of 8 correct.** All eight were
hand-judged against the cited source (appendix B). Three defects, all one family:
§4b binds the table, the completeness of the file label and the written value, and
**nothing about whether the source phrasing refers to the row's own subject**.
That is the same provenance-without-referent defect that falsified §4 (§2), now
reproduced in the status class. `status` is therefore deliberately **absent from
`o1_recommended`**: the machine writes nothing, and every proposal reaches a human.
### Two candidate conditions, costed over the eight
Neither is implemented — extending a table the operator ratified as *closed* is an
operator decision, exactly as condition 5 was in §4a. Both are pure gain on this
population (they kill wrong proposals and no correct one), which is the number the
decision needs:
| Candidate | Kills | Correct proposals lost |
|---|---|---|
| **A** — count a bare `GA` in the quote as a GA-row source phrasing, so a quote saying both `GA` and "public preview" becomes ambiguous | 1 (`onelake:198`) | 0 |
| **B** — abort when the quote is a multi-entity enumeration (≥ 2 pipes, or a numbered list) | 2 (`onelake:198`, `owasp:79`) | 0 |
B subsumes A on these eight. Neither catches `security-copilot-integration.md:94`,
where the quote is prose and the `(Preview)` marker simply belongs to a different
agent. A stricter **referent-name** condition (the row's subject must appear in the
quote) would catch it — and would also kill two *correct* proposals (`:83`, `:93`),
where the source names the capability rather than the agent. That trade is real and
is why this is put to the operator rather than shipped.
## Appendix A — the 15 admitted proposals, hand-verified
Every proposal the classifier (§4 + context condition) admitted over the whole
@ -209,3 +263,29 @@ matching prefix (`AI-`, `gpt-`, `Agent `) satisfies the context condition while
the digit is part of a name rather than a quantity. The two unverified are also
`number` and are excluded by the same class rule — verifying them costs a source
fetch each and would move the total to at most 11.
## Appendix B — the 8 §4b status proposals, hand-verified
Every status proposal the classifier admits over the whole `not_grounded`
population, judged against the cited source. Same discipline as appendix A: a
later run that admits a ninth can diff against this list, and "5 of 8" is
otherwise an unreproducible claim.
Reproduce: `node scripts/kb-eval/classify-fix-ops.mjs --threshold 1 --write`, then
read the `proposal.type === 'status'` items in
`scripts/kb-eval/data/r11-pilot-classification.json`.
| # | File:line | Swap | Verdict |
|---|---|---|---|
| 1 | `agent-orchestration/foundry-agent-service-ga.md:68` | `**Preview**``**GA**` | **correct** — quote: "hosted agents are generally available"; the row's subject is *Hosted agents* |
| 2 | `agent-orchestration/foundry-agent-service-ga.md:72` | `**GA**``**Preview**` | **correct** — quote: "Trigger an agent by using Logic Apps (preview)"; the row's subject is the Logic Apps trigger |
| 3 | `ai-security-engineering/security-copilot-integration.md:83` | `Public Preview``GA` | **correct** — quote: "Email and collaboration alert triage capabilities are already generally available (GA)"; the row is the phishing/email triage agent |
| 4 | `ai-security-engineering/security-copilot-integration.md:93` | `GA``Preview` | **correct** — quote is from the agent's own doc page: "This feature is in public preview" |
| 5 | `ai-security-engineering/entra-agent-id-zero-trust.md:439` | `Public Preview``GA` | **correct** — quote: "The Microsoft Entra Agent ID platform is now generally available"; the row's subject is Entra Agent ID (kjerne) |
| 6 | `ai-security-engineering/security-copilot-integration.md:94` | `GA``Preview` | **unproven** — the quote's `(Preview)` marker belongs to *Identity Risk Management Agent*, not to Access Review Agent. The judge's prose `reason` does support preview from a what's-new post, so the outcome is plausibly right; the cited evidence does not establish it. Not applied. |
| 7 | `data-engineering/onelake-data-strategy.md:198` | `GA``Preview` | **wrong** — the quote says `Lakehouse \| Yes \| GA`. "Public preview" in the same quote belongs to *Eventhouse*. Killed by candidate A and B. |
| 8 | `ai-security-engineering/owasp-llm-top10-azure-mitigations.md:79` | `GA``Preview` | **wrong** — the source marks only *Response Completeness* as preview; the row covers the groundedness/completeness pair, so the edit makes the groundedness half false. The correct fix is to split the row (O2/O3). Killed by candidate B. |
**5 correct · 1 unproven · 2 wrong.** All five correct ones carry the source
phrasing on the row's own subject; all three defects are the referent gap
described in §8. No proposal was applied — §4b output is a human review list.

View file

@ -142,8 +142,34 @@ the file does not itself appear verbatim in the quote:
- The written value is the **corpus-side** equivalent with the file's own markup
preserved (`**Preview**``**GA**`), never the English phrase pasted in.
**Not yet implemented.** The classifier still aborts these as `STATUS_SYNONYM`;
the ratification above is the contract a later session builds against.
**IMPLEMENTED 2026-08-03** in `classifyStatusSynonym()` / `fileStatusLabel()` /
`sourceStatusRows()` (`scripts/kb-eval/lib/fix-op.mjs`), 19 tests. Two
implementation decisions the ratified text left open, both resolved towards
failing closed:
- **The status locator is LINE-scoped**, not block-scoped like the numeric one.
Lifecycle vocabulary repeats down every column of a status table, so a block
window is ambiguous by construction; all 15 pilot flags in this class point at
the row that carries the claim.
- **A quote asserting two different rows aborts** (`SOURCE_STATUS_AMBIGUOUS`), and
a row with two corpus-side labels writes the **first** — the least specific one,
so a source saying only "preview" can never produce "Public Preview".
Aborts keep the `STATUS_SYNONYM` code and name their cause in `detail.reason`, so
the §10 abort taxonomy stays comparable across the implementation.
**Measured: 15 pilot / 54 corpus-wide flags → 5 and 8 proven; 5 of the 8 are
correct** (`docs/r11-pilot-results.md` §8 + appendix B). The three defects are one
family: §4b binds the table, the completeness of the file label and the written
value, and **nothing about whether the source phrasing refers to the row's own
subject** — provenance without referent, the same defect that falsified §4. The
class is therefore **review-grade, not apply-grade**: `status` is absent from
`o1_recommended` and no driver applies it.
**Open operator decision — a referent condition for §4b.** Two candidates are
costed over the eight in `r11-pilot-results.md` §8; both kill wrong proposals and
no correct one, and neither is implemented, because extending a table ratified as
*closed* is an operator decision, exactly as condition 5 was in §4a.
## 5. The O2 policy — RATIFIED 2026-08-03, with a remainder check

View file

@ -86,7 +86,21 @@ const r8 = items.filter((i) => i.rule === 'R8');
const LOCATOR_CODES = new Set([ABORT_CODES.LOCATOR_MISS, ABORT_CODES.LOCATOR_AMBIGUOUS]);
const locatorAborts = o3.filter((i) => LOCATOR_CODES.has(i.code)).length;
const s4O1 = s4Only.filter((v) => v.op === 'O1').length;
// §4-as-written is a NUMERIC-path measurement: it is the baseline the context
// condition (§4a) was added against. §4b status swaps do not run through the
// context condition at all, so counting them here would silently inflate the
// baseline and break the comparison with the pilot's hand-verified 6.
const isStatus = (v) => v.op === 'O1' && v.proposal.type === 'status';
const s4O1 = s4Only.filter((v) => v.op === 'O1' && !isStatus(v)).length;
const o1Numeric = o1.filter((i) => i.proposal.type !== 'status').length;
// §4b (ratified 2026-08-03): the STATUS_SYNONYM class, split into what the closed
// synonym table proves and why the remainder still aborts. The abort REASON
// sub-distribution is the actionable part — NO_COMPLETE_FILE_LABEL is a corpus
// shape, SOURCE_STATUS_AMBIGUOUS is a quote shape, FILE_ALREADY_MATCHES means the
// flag was never a status mismatch in the first place.
const statusProposals = items.filter((i) => i.op === 'O1' && i.proposal.type === 'status');
const statusAborts = o3.filter((i) => i.code === ABORT_CODES.STATUS_SYNONYM);
// O1 precision is NOT uniform across token types, and this split is the pilot's
// operational conclusion. Hand-verified over the whole not_grounded population:
@ -117,7 +131,19 @@ const report = {
s4_as_written: {
O1: s4O1,
note:
'What §4 exactly as written would admit. NOT a source of proposals: on the >=7 pilot all 6 were hand-verified and 4 were wrong edits (unit crossing, metric crossing, two mutilated identifiers) — measured precision 2/6. Runs at other thresholds carry no hand-verification.',
'What §4 exactly as written would admit on the NUMERIC path (§4b status swaps excluded — they never run through the context condition). NOT a source of proposals: on the >=7 pilot all 6 were hand-verified and 4 were wrong edits (unit crossing, metric crossing, two mutilated identifiers) — measured precision 2/6. Runs at other thresholds carry no hand-verification.',
},
status_synonym: {
contract: '§4b — the closed synonym table, ratified 2026-08-03',
class_total: statusProposals.length + statusAborts.length,
proven: statusProposals.length,
aborts: tally(statusAborts, (i) => (i.detail && i.detail.reason) || '(unspecified)'),
hand_verified:
THRESHOLD === 7
? 'All 5 hand-judged 2026-08-03 (docs/r11-pilot-results.md appendix B). Four carry the source phrasing on the row\'s OWN subject and are correct. One (security-copilot-integration.md:94) harvests a "(Preview)" marker that belongs to a DIFFERENT agent in an enumerated quote — the same provenance-without-referent defect that falsified §4. Its outcome is plausibly right; its proof is not.'
: 'hand-verification was done on the >=7 pilot only',
applicability:
'REVIEW-GRADE, NOT APPLY-GRADE. status is deliberately absent from o1_recommended: §4b binds the table, the completeness of the file label and the written value, and nothing about whether the source phrasing refers to the row\'s subject. A referent condition is an open operator decision.',
},
o1_by_type: byType,
o1_recommended: {
@ -145,7 +171,11 @@ const handNote =
THRESHOLD === 7
? ' — all 6 hand-verified: 4 are wrong edits (unit crossing, metric crossing, two mutilated identifiers)'
: ' (hand-verification was done on the >=7 pilot only)';
console.log(`\n§4 as written would admit ${s4O1}${handNote}. Context condition removes ${s4O1 - o1.length}.\n`);
console.log(`\n§4 as written would admit ${s4O1} on the numeric path${handNote}. Context condition removes ${s4O1 - o1Numeric}.`);
console.log(
`§4b status class: ${report.status_synonym.class_total} flags -> ${report.status_synonym.proven} proven, ` +
`${JSON.stringify(report.status_synonym.aborts)} — REVIEW-grade, not applied by any driver.\n`,
);
console.log('abort codes:');
for (const [code, n] of Object.entries(byCode).sort((a, b) => b[1] - a[1])) {
console.log(` ${code.padEnd(20)} ${String(n).padStart(4)} ${pct(n)}`);

View file

@ -55,12 +55,195 @@ const TOKEN_PATTERNS = [
// Lifecycle vocabulary. Present in a claim without any numeric token, this is the
// GA/Preview class: the corpus writes `**Preview**` / `**GA**` while the cited
// source writes "generally available". A swap would satisfy §4 literally while
// pasting English prose into a Norwegian table, so the class aborts and is put to
// the operator as a design question (a ratified synonym table, or permanent O3).
// source writes "generally available". A swap satisfies §4 literally while
// pasting English prose into a Norwegian table, which is why the class was put to
// the operator as a design question — answered by the §4b table below.
const STATUS_RE =
/\b(?:GA|generally available|allment tilgjengelig|public preview|private preview|preview|deprecated|utfaset|retired|avviklet)\b/i;
// ------------------------------------------------------- §4b synonym table
//
// RATIFIED 2026-08-03 (docs/r11-tiered-fix-design.md §4b). This is the one place
// where the value written into the file does NOT appear verbatim in the quote —
// §4 condition 1 is structurally unsatisfiable for it, because the corpus writes
// a label and the source writes a phrase. The table is what the operator ratified
// in its place, and it is CLOSED: a pair not listed here aborts, and nothing
// extends it at run time.
//
// `corpus[0]` is the canonical value written back. The order is the table's own:
// the row's least specific label wins, so a source that says only "preview" can
// never produce the more specific "Public Preview" — that would assert something
// the source does not.
export const STATUS_TABLE = [
{ row: 'GA', corpus: ['GA'], source: ['generally available', 'general availability'] },
{ row: 'PREVIEW', corpus: ['Preview', 'Public Preview'], source: ['public preview', 'preview'] },
{ row: 'PRIVATE_PREVIEW', corpus: ['Private Preview'], source: ['private preview'] },
{ row: 'DEPRECATED', corpus: ['Deprecated', 'Utfaset'], source: ['deprecated', 'retired'] },
];
// Longest first, so "private preview" is consumed as its own row before the
// "preview" row can claim the tail of it.
const SOURCE_PHRASES = STATUS_TABLE.flatMap((r) => r.source.map((p) => ({ row: r.row, phrase: p }))).sort(
(a, b) => b.phrase.length - a.phrase.length,
);
const CORPUS_LABELS = new Map(
STATUS_TABLE.flatMap((r) => r.corpus.map((label) => [label.toLowerCase(), r])),
);
// A hyphen counts as a word character HERE, unlike WORD elsewhere in this module:
// `2025-11-15-preview` is an api-version identifier, not a statement that the
// feature is in preview. Treating `-` as a boundary would harvest lifecycle rows
// out of URLs and code samples.
const isPhraseChar = (ch) => ch !== undefined && /[A-Za-z0-9-]/.test(ch);
/**
* Which lifecycle rows does the cited quote assert, per the closed §4b table?
* Matching is case-insensitive, non-overlapping, longest phrase first.
* @returns {string[]} distinct row keys, in table order
*/
export function sourceStatusRows(quote) {
const text = (quote || '').toLowerCase();
const taken = [];
const rows = new Set();
for (const { row, phrase } of SOURCE_PHRASES) {
let from = 0;
for (;;) {
const at = text.indexOf(phrase, from);
if (at === -1) break;
const end = at + phrase.length;
from = end;
if (taken.some(([s, e]) => at < e && end > s)) continue;
if (isPhraseChar(text[at - 1]) || isPhraseChar(text[end])) continue;
taken.push([at, end]);
rows.add(row);
}
}
return STATUS_TABLE.map((r) => r.row).filter((r) => rows.has(r));
}
// Markup wrappers the corpus actually uses around a lifecycle label. The label is
// replaced INSIDE the wrapper, which is how §4b constraint 3 ("the file's own
// markup preserved") is satisfied without any markup handling at write time.
const WRAPPERS = ['**', '__', '*', '_', '`'];
/**
* Is this segment exactly a lifecycle label, once whitespace and one or more
* markup wrappers are peeled off? Returns the label's span in the ORIGINAL line.
*/
function labelInSegment(segment, segStart) {
let text = segment;
let off = 0;
const trim = () => {
const lead = text.length - text.trimStart().length;
off += lead;
text = text.trim();
};
trim();
for (;;) {
const w = WRAPPERS.find((x) => text.length > 2 * x.length && text.startsWith(x) && text.endsWith(x));
if (!w) break;
off += w.length;
text = text.slice(w.length, text.length - w.length);
trim();
}
const rec = CORPUS_LABELS.get(text.toLowerCase());
return rec ? { row: rec.row, label: text, index: segStart + off, length: text.length } : null;
}
// An emphasised run anywhere on the line. `|` is excluded from the inner text so a
// run can never span two table cells.
const EMPHASIS_RE = /(\*\*|__|\*|_|`)([^*_`|]+?)\1/g;
/**
* The single complete lifecycle label on `line`, or null when there is none or
* more than one (§4b constraint 2: a whole table cell or an emphasised token,
* never a substring of a longer sentence).
*
* Deliberately LINE-scoped rather than block-scoped, unlike the numeric locator.
* Lifecycle vocabulary repeats down every column of a status table, so a block
* window is ambiguous by construction and all 15 pilot flags in this class
* point at the row that carries the claim, not at the table header.
*
* @returns {{row: string, label: string, index: number, length: number}|null}
*/
export function fileStatusLabel(line) {
const text = line || '';
const hits = [];
if (text.includes('|')) {
let at = 0;
for (const cell of text.split('|')) {
const hit = labelInSegment(cell, at);
if (hit) hits.push(hit);
at += cell.length + 1;
}
}
EMPHASIS_RE.lastIndex = 0;
let m;
while ((m = EMPHASIS_RE.exec(text)) !== null) {
const hit = labelInSegment(m[0], m.index);
if (hit && !hits.some((h) => h.index === hit.index)) hits.push(hit);
}
return hits.length === 1 ? hits[0] : null;
}
/**
* §4b: classify a status-vocabulary flag against the ratified table.
*
* Every abort keeps the STATUS_SYNONYM code and names its cause in `detail.reason`
* the top-level abort taxonomy is the pilot's measurement #3 and stays
* comparable across the implementation, with the reasons reported as a
* sub-distribution.
*/
function classifyStatusSynonym(flag, lines) {
const detail = (reason, extra = {}) => abort(ABORT_CODES.STATUS_SYNONYM, { detail: { reason, ...extra } });
if (flag.line < 1 || flag.line > lines.length) return detail('LINE_OUT_OF_RANGE', { line: flag.line });
const rows = sourceStatusRows(flag.evidence_quote || '');
if (rows.length === 0) return detail('NO_SOURCE_STATUS');
if (rows.length > 1) return detail('SOURCE_STATUS_AMBIGUOUS', { rows });
const before = lines[flag.line - 1];
const hit = fileStatusLabel(before);
if (!hit) return detail('NO_COMPLETE_FILE_LABEL');
if (hit.row === rows[0]) return detail('FILE_ALREADY_MATCHES', { row: hit.row });
const target = STATUS_TABLE.find((r) => r.row === rows[0]);
const replacement = target.corpus[0];
const after = before.slice(0, hit.index) + replacement + before.slice(hit.index + hit.length);
// §4 conditions 2 and 3 still bind. Condition 1 is replaced by the table: what
// must appear verbatim in the quote is the SOURCE phrasing, not the written value.
const rest =
before.slice(0, hit.index) === after.slice(0, hit.index) &&
before.slice(hit.index + hit.length) === after.slice(hit.index + replacement.length);
const rebuilt = lines.slice();
rebuilt[flag.line - 1] = after;
const changedLines = rebuilt.reduce((n, l, i) => n + (l === lines[i] ? 0 : 1), 0);
if (!rest || changedLines !== 1) {
return abort(ABORT_CODES.INVARIANT_FAIL, { detail: { restIdentical: rest, changedLines } });
}
return {
op: 'O1',
code: PROVEN,
proposal: {
file: flag.file,
line: flag.line,
token: hit.label,
replacement,
type: 'status',
status_row_from: hit.row,
status_row_to: rows[0],
before,
after,
evidence_url: flag.evidence_url,
evidence_quote: flag.evidence_quote,
},
};
}
/**
* Extract swappable value tokens, non-overlapping and priority ordered.
* Status words are NOT value tokens see STATUS_RE.
@ -206,7 +389,8 @@ export function classifyFlag(flag, fileText, opts = {}) {
const tokens = distinct(extractValueTokens(claim));
if (tokens.length === 0) {
return abort(hasStatusWord(claim) ? ABORT_CODES.STATUS_SYNONYM : ABORT_CODES.NO_VALUE_TOKEN);
if (!hasStatusWord(claim)) return abort(ABORT_CODES.NO_VALUE_TOKEN);
return classifyStatusSynonym(flag, lines); // §4b — the ratified table, not a swap of prose
}
if (tokens.length > 1) {
return abort(ABORT_CODES.MULTI_VALUE_TOKEN, { detail: { candidates: tokens.map((t) => t.value) } });