open/ms-ai-architect
1
0
Fork 0
Code Issues Releases Activity

feat(ms-ai-architect): R11 §4b implementert — status-synonymtabellen målt, 8 forslag hånd-dømt [skip-docs]

Browse source 
Implementerer den ratifiserte §4b-tabellen i lib/fix-op.mjs (19 nye tester,
suite 996/996). Alle tre skrankene har egne tester: tabellen er LUKKET, fil-
tokenet må være en KOMPLETT livssyklus-etikett, og verdien som skrives er den
korpus-side ekvivalenten med filas egen markup bevart.

To implementasjonsvalg den ratifiserte teksten lot stå åpne, begge løst mot
fail-closed: status-lokatoren er LINJE-scopet (livssyklus-vokabular gjentas
nedover hver kolonne i en statustabell, så et blokkvindu er tvetydig ved
konstruksjon), og et sitat som hevder to rader aborterer.

MÅLT: 15 pilot / 54 korpus-brede flagg -> 5 og 8 provbare. Alle 8 hånd-dømt
mot kilden (r11-pilot-results.md appendiks B): 5 korrekte, 1 ubevist, 2 GALE.

De tre defektene er én familie: §4b binder tabellen, etikettens fullstendighet
og verdien som skrives — og INGENTING om hvorvidt kilde-frasen refererer til
radens eget subjekt. Samme proveniens-uten-referent-defekt som falsifiserte §4.
Klassen er derfor REVIEW-grade, ikke apply-grade: `status` står bevisst utenfor
o1_recommended, ingen driver applikerer den.

To kandidatvilkår er kostnadsberegnet over de åtte (begge dreper gale forslag
og null korrekte) men IKKE implementert — å utvide en tabell operatøren
ratifiserte som lukket er en operatørbeslutning, slik vilkår 5 var i §4a.

Rettet samtidig 2 NUL-bytes i testfila (pre-eksisterende, fra en tidligere
økt) som gjorde at git behandlet hele fila som binær og blokkerte diff-
gjennomgang før commit.
This commit is contained in:
Kjell Tore Guttormsen 2026-08-03 17:12:47 +02:00
commit e4925c6b28
5 changed files with 330 additions and 10 deletions
Download patch file Download diff file Expand all files Collapse all files

82
docs/r11-pilot-results.md
View file

@ -4,7 +4,7 @@
the live ledger. No KB file was edited and no ledger record was written.**
Instrument: `scripts/kb-eval/lib/fix-op.mjs` (+ `tests/kb-eval/test-fix-op-classify.test.mjs`,
30 tests) driven by `scripts/kb-eval/classify-fix-ops.mjs`. The classifier **is**
49 tests after §4b) driven by `scripts/kb-eval/classify-fix-ops.mjs`. The classifier **is**
the O1 driver with writes disabled — it constructs the swap and checks the §4
invariant, so measurements 1 and 3 come out of the mechanism that would later
touch the corpus, not out of a proxy heuristic.
@ -32,6 +32,12 @@ other flag needs a human.
| Pilot (`not_grounded` ≥ 7) | 24 | 202 | 2 | 2 |
| Whole `not_grounded` corpus | 218 | 776 | 15 | **9** |
> These are **numeric-path** figures, and they stay that way. §4b (the status
> synonym table) was implemented afterwards and adds a separate class with its own
> hand-verification — see §8. A run today prints O1 = 7 (pilot) and 23 (corpus)
> because the status proposals are included in the total; the numeric line above
> is unchanged and is still what `s4_as_written` compares against.
Reproduce: `node scripts/kb-eval/classify-fix-ops.mjs` (pilot) and
`node scripts/kb-eval/classify-fix-ops.mjs --threshold 1` (corpus). The nine are
enumerated with verdicts in appendix A — that hand-verification is the only thing
@ -178,6 +184,54 @@ nine-item machine assist, and its leverage lies entirely in the O2 decision.
---
## 8. §4b implemented — the status class measured, 2026-08-03
The ratified synonym table (`docs/r11-tiered-fix-design.md` §4b) is implemented in
`fix-op.mjs` and the class is measured. Reproduce:
`node scripts/kb-eval/classify-fix-ops.mjs --threshold 1` (`status_synonym` block).
| Population | STATUS_SYNONYM flags | Proven by §4b | Still aborting |
|---|---|---|---|
| Pilot (`not_grounded` ≥ 7) | 15 | **5** | 10 |
| Whole `not_grounded` corpus | 54 | **8** | 46 |
Why the other 46 abort, corpus-wide — this sub-distribution is the actionable
part, because the top-level `STATUS_SYNONYM` count alone says nothing:
| Reason | N | What it means |
|---|---|---|
| `NO_COMPLETE_FILE_LABEL` | 27 | the file writes the status inside a sentence — `(preview)` in a list item, `**DSPM (preview):**`, `"[Preview]: …"` in a JSON string. Constraint 2 refuses these, correctly. |
| `NO_SOURCE_STATUS` | 15 | the cited quote carries no listed lifecycle phrasing at all — the flag was never a status swap. |
| `SOURCE_STATUS_AMBIGUOUS` | 2 | the quote asserts two different rows (e.g. "…is now generally available. Partner solutions remain in preview."). |
| `FILE_ALREADY_MATCHES` | 2 | file and source agree; the mismatch was in the LLM-extracted claim, not in the corpus. |
**The class is REVIEW-grade, not apply-grade — 5 of 8 correct.** All eight were
hand-judged against the cited source (appendix B). Three defects, all one family:
§4b binds the table, the completeness of the file label and the written value, and
**nothing about whether the source phrasing refers to the row's own subject**.
That is the same provenance-without-referent defect that falsified §4 (§2), now
reproduced in the status class. `status` is therefore deliberately **absent from
`o1_recommended`**: the machine writes nothing, and every proposal reaches a human.
### Two candidate conditions, costed over the eight
Neither is implemented — extending a table the operator ratified as *closed* is an
operator decision, exactly as condition 5 was in §4a. Both are pure gain on this
population (they kill wrong proposals and no correct one), which is the number the
decision needs:
| Candidate | Kills | Correct proposals lost |
|---|---|---|
| **A** — count a bare `GA` in the quote as a GA-row source phrasing, so a quote saying both `GA` and "public preview" becomes ambiguous | 1 (`onelake:198`) | 0 |
| **B** — abort when the quote is a multi-entity enumeration (≥ 2 pipes, or a numbered list) | 2 (`onelake:198`, `owasp:79`) | 0 |
B subsumes A on these eight. Neither catches `security-copilot-integration.md:94`,
where the quote is prose and the `(Preview)` marker simply belongs to a different
agent. A stricter **referent-name** condition (the row's subject must appear in the
quote) would catch it — and would also kill two *correct* proposals (`:83`, `:93`),
where the source names the capability rather than the agent. That trade is real and
is why this is put to the operator rather than shipped.
## Appendix A — the 15 admitted proposals, hand-verified
Every proposal the classifier (§4 + context condition) admitted over the whole
@ -209,3 +263,29 @@ matching prefix (`AI-`, `gpt-`, `Agent `) satisfies the context condition while
the digit is part of a name rather than a quantity. The two unverified are also
`number` and are excluded by the same class rule — verifying them costs a source
fetch each and would move the total to at most 11.
## Appendix B — the 8 §4b status proposals, hand-verified
Every status proposal the classifier admits over the whole `not_grounded`
population, judged against the cited source. Same discipline as appendix A: a
later run that admits a ninth can diff against this list, and "5 of 8" is
otherwise an unreproducible claim.
Reproduce: `node scripts/kb-eval/classify-fix-ops.mjs --threshold 1 --write`, then
read the `proposal.type === 'status'` items in
`scripts/kb-eval/data/r11-pilot-classification.json`.
| # | File:line | Swap | Verdict |
|---|---|---|---|
| 1 | `agent-orchestration/foundry-agent-service-ga.md:68` | `**Preview**` → `**GA**` | **correct** — quote: "hosted agents are generally available"; the row's subject is *Hosted agents* |
| 2 | `agent-orchestration/foundry-agent-service-ga.md:72` | `**GA**` → `**Preview**` | **correct** — quote: "Trigger an agent by using Logic Apps (preview)"; the row's subject is the Logic Apps trigger |
| 3 | `ai-security-engineering/security-copilot-integration.md:83` | `Public Preview` → `GA` | **correct** — quote: "Email and collaboration alert triage capabilities are already generally available (GA)"; the row is the phishing/email triage agent |
| 4 | `ai-security-engineering/security-copilot-integration.md:93` | `GA` → `Preview` | **correct** — quote is from the agent's own doc page: "This feature is in public preview" |
| 5 | `ai-security-engineering/entra-agent-id-zero-trust.md:439` | `Public Preview` → `GA` | **correct** — quote: "The Microsoft Entra Agent ID platform is now generally available"; the row's subject is Entra Agent ID (kjerne) |
| 6 | `ai-security-engineering/security-copilot-integration.md:94` | `GA` → `Preview` | **unproven** — the quote's `(Preview)` marker belongs to *Identity Risk Management Agent*, not to Access Review Agent. The judge's prose `reason` does support preview from a what's-new post, so the outcome is plausibly right; the cited evidence does not establish it. Not applied. |
| 7 | `data-engineering/onelake-data-strategy.md:198` | `GA` → `Preview` | **wrong** — the quote says `Lakehouse \| Yes \| GA`. "Public preview" in the same quote belongs to *Eventhouse*. Killed by candidate A and B. |
| 8 | `ai-security-engineering/owasp-llm-top10-azure-mitigations.md:79` | `GA` → `Preview` | **wrong** — the source marks only *Response Completeness* as preview; the row covers the groundedness/completeness pair, so the edit makes the groundedness half false. The correct fix is to split the row (O2/O3). Killed by candidate B. |
**5 correct · 1 unproven · 2 wrong.** All five correct ones carry the source
phrasing on the row's own subject; all three defects are the referent gap
described in §8. No proposal was applied — §4b output is a human review list.

30
docs/r11-tiered-fix-design.md
View file

@ -142,8 +142,34 @@ the file does not itself appear verbatim in the quote:
- The written value is the **corpus-side** equivalent with the file's own markup
preserved (`**Preview**` → `**GA**`), never the English phrase pasted in.
**Not yet implemented.** The classifier still aborts these as `STATUS_SYNONYM`;
the ratification above is the contract a later session builds against.
**IMPLEMENTED 2026-08-03** in `classifyStatusSynonym()` / `fileStatusLabel()` /
`sourceStatusRows()` (`scripts/kb-eval/lib/fix-op.mjs`), 19 tests. Two
implementation decisions the ratified text left open, both resolved towards
failing closed:
- **The status locator is LINE-scoped**, not block-scoped like the numeric one.
Lifecycle vocabulary repeats down every column of a status table, so a block
window is ambiguous by construction; all 15 pilot flags in this class point at
the row that carries the claim.
- **A quote asserting two different rows aborts** (`SOURCE_STATUS_AMBIGUOUS`), and
a row with two corpus-side labels writes the **first** — the least specific one,
so a source saying only "preview" can never produce "Public Preview".
Aborts keep the `STATUS_SYNONYM` code and name their cause in `detail.reason`, so
the §10 abort taxonomy stays comparable across the implementation.
**Measured: 15 pilot / 54 corpus-wide flags → 5 and 8 proven; 5 of the 8 are
correct** (`docs/r11-pilot-results.md` §8 + appendix B). The three defects are one
family: §4b binds the table, the completeness of the file label and the written
value, and **nothing about whether the source phrasing refers to the row's own
subject** — provenance without referent, the same defect that falsified §4. The
class is therefore **review-grade, not apply-grade**: `status` is absent from
`o1_recommended` and no driver applies it.
**Open operator decision — a referent condition for §4b.** Two candidates are
costed over the eight in `r11-pilot-results.md` §8; both kill wrong proposals and
no correct one, and neither is implemented, because extending a table ratified as
*closed* is an operator decision, exactly as condition 5 was in §4a.
## 5. The O2 policy — RATIFIED 2026-08-03, with a remainder check