open/ms-ai-architect
1
0
Fork 0
Code Issues Releases Activity

docs(ms-ai-architect): R11 utførelses-design — tiering på fikse-operasjon over allerede-evidensert flaggpopulasjon [skip-docs]

Browse source
This commit is contained in:
Kjell Tore Guttormsen 2026-08-03 12:28:24 +02:00
commit 9e7638cfa1
1 changed files with 206 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

206
docs/r11-tiered-fix-design.md Normal file
View file

@ -0,0 +1,206 @@
# R11 execution design — tiered fixes over an already-evidenced flag population
**How the R11 fix step consumes the judge-pass flags. Extends
`docs/r11-flag-format-2026-07.md` (the record contract) with the execution
contract: what the fix operation is per flag, what is machine-provable, what
requires human judgement, and what may never be automated.**
Status: design spec. Written 2026-08-03 against a ledger snapshot of **222
records / 902 flags** (`scripts/kb-eval/data/judge-pass-manifest.json`). The
population is **not yet complete** — batch R7.5 stood at 30 of 51 files — so
every count below is a snapshot, not a final figure. The design does not depend
on the exact counts; the classifier re-derives them at run time.
---
## 1. The measurement this design is built on
**All 712 `not_grounded` flags carry a non-empty verbatim `evidence_quote`.**
712 of 712, measured. Every flag also carries the `evidence_url` the judge
actually fetched, the rule that fired, and a one-sentence `reason` stating what
the source said versus what the claim said.
The consequence is the whole point of this document: **R11 does not start from
zero evidence.** The expensive half — locating the authoritative page, reading
it, and extracting the passage that decides the claim — was already paid for by
the judge pass. A fix step designed as "fetch the source, confirm, correct"
re-does work that is already on disk.
What actually remains per flag is narrower:
1. is the quote still current (freshness, §7), and
2. what should the sentence say instead (§3).
## 2. Measured shape of the flag population
| Measurement | Value |
|---|---|
| Flags total | 902 |
| `not_grounded` (fix targets per flag-format spec) | 712 |
| `source_silent` (recorded, not a fix target) | 190 |
| Flags carrying a verbatim `evidence_quote` | **712 / 712** |
| Distinct `evidence_url` across the 712 | **540** |
| Files carrying at least one `not_grounded` flag | 199 |
| Files with 1 flag | 45 |
| Files with 2–6 flags | 133 (489 flags) |
| Files with ≥7 flags | 21 (**178 flags — 25 % of the volume in 10 % of the files**) |
Rule distribution over the 712: R8 339 · R2 176 · (no rule) 72 · R4 41 · R3 40 ·
R7 28 · R1 16.
**540 distinct source URLs for 712 flags** is the number that forecloses the
obvious optimisation: there is no batching win hiding in shared sources. The
largest cluster is 12 flags on one page. This is ~700 separate facts, and the
work is irreducibly per-claim.
## 3. Three fix operations — the partition is by operation, not by rule
The tiering is defined by **what the fix does to the file**, because that is what
determines whether a machine can prove it and whether a human must decide it.
Rule codes are a signal, not the partition.
| Op | Fix operation | Provable? | Who decides |
|---|---|---|---|
| **O1** | **Value swap** — the claim states X, the cited source states Y; replace the token. | **Yes** (§4) | Machine proposes, human reviews a diff list |
| **O2** | **Subtraction** — remove or generalise the specificity the source does not support. Covers: source-silent claims, absent entities (retired SKUs/models/services), and multi-part claims where the failing sub-assertion can be dropped while the grounded part survives. | Partly — the *invariant* is that no new fact is introduced | Human ratifies the policy once (§5), then reviews proposals |
| **O3** | **Rewrite** — the corrected sentence requires a judgement about what to assert. | No | Human, per claim |
**Do not read the §2 rule counts as tier sizes.** They overlap: of the 202 flags
whose claim *and* quote contain a numeric token — the naive O1 signal — **94 are
R8 multi-part claims**, which are not clean swaps. Heuristics over this
population give upper bounds in several directions at once, never a partition.
**The classifier fails closed to O3.** If it cannot prove an item is O1 or O2, the
item is O3 and a human sees it. A misrouted O3 costs one review; a misrouted O1
ships a wrong edit to a public file.
R8 deserves a specific note, because it is the largest rule class and it is *not*
automatically the most expensive one: the judge's `reason` names **which**
sub-assertion failed. Where the grounded part stands on its own, the fix is O2
(drop the unsupported part), not O3 (rewrite the sentence). How much of R8 falls
that way is unknown and is a primary pilot measurement (§10).
## 4. The O1 invariant (what makes a value swap provable)
An O1 proposal is only valid if, after the edit:
1. the new value appears **verbatim inside the cited `evidence_quote`**, and
2. the rest of the line is **byte-identical** to before, and
3. exactly one line in the file changed.
A driver that cannot establish all three for an item **aborts before writing** and
routes the item to O3. This is the same discipline already proven in the
header-backfill drivers (frozen manifest, hard per-file invariant, abort before
write, idempotent re-run, `atomicWriteSync`) — see `scripts/kb-update/backfill-*.mjs`.
This invariant is deliberately stronger than human review at scale. "The new
value occurs verbatim in the quote the judge cited" is a checkable property; an
operator eyeballing edit number 600 is not. Human attention is therefore spent
where it cannot be replaced (O3), not spread thin across all of them.
## 5. The O2 policy (requires operator ratification before use)
O2 fixes by **subtraction**: the unsupported specificity is removed or
generalised rather than replaced with a researched value.
- It **cannot introduce a new error**, because it asserts strictly less than the
file did before.
- It requires **no new fact-finding**, which is what makes it cheap.
- It **reduces information density**. That is the real cost, and it is an
operator decision, not an engineering one.
The position this design recommends: a knowledge base that says less and says
nothing false is worth more than one carrying stale precision. The corpus is
publicly distributed; an incorrect specific number is a worse failure than an
honest general statement.
Ratifying O2 also resolves the standing `source_silent` question as **one class
decision** instead of 190 individual ones. Until it is ratified, every O2
candidate falls to O3.
## 6. What stays human, permanently
- **Never auto-fix.** No fix reaches a file without human confirmation of the
class (O1/O2) or the item (O3). A judge false positive must become a human
review, never silent corruption of a public file.
- **Subagents never write.** Proposal generation is read-only fan-out; all
writes happen in one place (§8).
- **O3 is not a backlog to automate later.** It is the class where the corrected
assertion is a judgement call, and it is the reason the loop reaches ~100 % fix
precision on top of a fallible judge.
## 7. Evidence freshness
Each `evidence_quote` is current as of the judge's fetch date, not the fix date.
Before any file is touched, the evidence base is refreshed by **re-fetching per
distinct URL — 540, not 712** — as read-only fan-out. A refreshed quote that no
longer supports the flag re-routes the item (possibly closing it as no longer an
error). This preserves the verification duty — fresh confirmation before a public
file changes — without paying for 712 separate fetches.
## 8. Concurrency: one writer, many readers
R11 phases that are machine-bound (evidence refresh, proposal generation) may run
across concurrent sessions. The corpus and the ledger may not.
**Single-writer state** — never written by more than one session:
`scripts/kb-eval/data/judge-pass-manifest.json`, the corpus files themselves, and
the repo's state file.
**Protocol:**
- Worker sessions write **only** disjointly-named artefacts to a local, untracked
working directory (one file per unit of work). They do not ingest, stamp, or
commit.
- One integrator session ingests serially, stamps, and commits.
- If two sessions must write corpus files concurrently, shard **by file, never by
claim**, one git worktree per shard. Disjoint file sets merge without conflict;
the ledger is still written only by the integrator, after merge.
Git worktrees share the main `.git` directory, so the Layer B `pre-commit` scan
symlink applies inside every worktree — verified 2026-08-03 by creating a
worktree and resolving `git rev-parse --git-common-dir` plus the hook target.
The security gate does not weaken under sharding.
The reason this protocol is explicit: whole-tree backup/restore in the write
drivers previously caused silent data loss across concurrent sessions (closed in
the write-safety hardening pass — scoped restore + atomic writes). Sharded writes
without a single-writer rule would reintroduce that class through a different door.
## 9. No full re-judge after fixing
A fixed file does **not** require a fresh judge pass over all of its claims.
- Claims already judged `grounded` keep that verdict; the fix did not touch them.
- A fixed claim's evidence is the O1 invariant (§4) or the human confirmation
(§5–6), recorded with the fix.
- The programme's end-proof is a **fresh blind gold sample** measured after the
fixes, not a re-run of the corpus pass.
This is stated explicitly because assuming otherwise would silently add a second
full corpus pass to the plan.
## 10. Pilot and acceptance criterion
Before any scaling, run the classifier and the O1 driver against the **21 files
carrying ≥7 flags (178 flags)** — a quarter of the volume in a tenth of the
files, and the densest available sample.
Measure and record:
1. the actual O1 / O2 / O3 split (this design's central unknown);
2. how much of R8 resolves as O2 rather than O3 (§3);
3. the O1 driver's abort rate — items it could not prove, which must land in O3;
4. review throughput per operation class, measured rather than assumed.
Scale to the remaining files only on measured numbers. If the split is materially
worse than assumed, that is known after one session rather than after ten.
## 11. Out of scope
- **Rebuild instead of repair.** Regenerating flagged files from source rather
than editing them is a settled decision: the corpus is repaired, not rebuilt.
Reopening it is an operator call, not a design choice made here.
- **Auto-fix in any form** (§6).
- **Price and other unsourced claims** beyond the O2 class decision — these
remain operator-gated as a separate matter.