ms-ai-architect/docs/r11-tiered-fix-design.md

R11 execution design — tiered fixes over an already-evidenced flag population

How the R11 fix step consumes the judge-pass flags. Extends docs/r11-flag-format-2026-07.md (the record contract) with the execution contract: what the fix operation is per flag, what is machine-provable, what requires human judgement, and what may never be automated.

Status: design spec. Written 2026-08-03 against a ledger snapshot of 222 records / 902 flags (scripts/kb-eval/data/judge-pass-manifest.json). The population is not yet complete — batch R7.5 stood at 30 of 51 files — so every count below is a snapshot, not a final figure. The design does not depend on the exact counts; the classifier re-derives them at run time.

---

1. The measurement this design is built on

All 712 not_grounded flags carry a non-empty verbatim evidence_quote. 712 of 712, measured. Every flag also carries the evidence_url the judge actually fetched, the rule that fired, and a one-sentence reason stating what the source said versus what the claim said.

The consequence is the whole point of this document: R11 does not start from zero evidence. The expensive half — locating the authoritative page, reading it, and extracting the passage that decides the claim — was already paid for by the judge pass. A fix step designed as "fetch the source, confirm, correct" re-does work that is already on disk.

What actually remains per flag is narrower:

1. is the quote still current (freshness, §7), and
2. what should the sentence say instead (§3).

2. Measured shape of the flag population

Measurement	Value
Flags total	902
not_grounded (fix targets per flag-format spec)	712
source_silent (recorded, not a fix target)	190
Flags carrying a verbatim evidence_quote	712 / 712
Distinct evidence_url across the 712	540
Files carrying at least one not_grounded flag	199
Files with 1 flag	45
Files with 2–6 flags	133 (489 flags)
Files with ≥7 flags	21 (178 flags — 25 % of the volume in 10 % of the files)

Rule distribution over the 712: R8 339 · R2 176 · (no rule) 72 · R4 41 · R3 40 · R7 28 · R1 16.

540 distinct source URLs for 712 flags is the number that forecloses the obvious optimisation: there is no batching win hiding in shared sources. The largest cluster is 12 flags on one page. This is ~700 separate facts, and the work is irreducibly per-claim.

3. Three fix operations — the partition is by operation, not by rule

The tiering is defined by what the fix does to the file, because that is what determines whether a machine can prove it and whether a human must decide it. Rule codes are a signal, not the partition.

Op	Fix operation	Provable?	Who decides
O1	Value swap — the claim states X, the cited source states Y; replace the token.	Yes (§4)	Machine proposes, human reviews a diff list
O2	Subtraction — remove or generalise the specificity the source does not support. Covers: source-silent claims, absent entities (retired SKUs/models/services), and multi-part claims where the failing sub-assertion can be dropped while the grounded part survives.	Partly — the invariant is that no new fact is introduced	Human ratifies the policy once (§5), then reviews proposals
O3	Rewrite — the corrected sentence requires a judgement about what to assert.	No	Human, per claim

Do not read the §2 rule counts as tier sizes. They overlap: of the 202 flags whose claim and quote contain a numeric token — the naive O1 signal — 94 are R8 multi-part claims, which are not clean swaps. Heuristics over this population give upper bounds in several directions at once, never a partition.

The classifier fails closed to O3. If it cannot prove an item is O1 or O2, the item is O3 and a human sees it. A misrouted O3 costs one review; a misrouted O1 ships a wrong edit to a public file.

R8 deserves a specific note, because it is the largest rule class and it is not automatically the most expensive one: the judge's reason names which sub-assertion failed. Where the grounded part stands on its own, the fix is O2 (drop the unsupported part), not O3 (rewrite the sentence). How much of R8 falls that way is unknown and is a primary pilot measurement (§10).

4. The O1 invariant (what makes a value swap provable)

An O1 proposal is only valid if, after the edit:

1. the new value appears verbatim inside the cited evidence_quote, and
2. the rest of the line is byte-identical to before, and
3. exactly one line in the file changed.

A driver that cannot establish all three for an item aborts before writing and routes the item to O3. This is the same discipline already proven in the header-backfill drivers (frozen manifest, hard per-file invariant, abort before write, idempotent re-run, atomicWriteSync) — see scripts/kb-update/backfill-*.mjs.

This invariant is deliberately stronger than human review at scale. "The new value occurs verbatim in the quote the judge cited" is a checkable property; an operator eyeballing edit number 600 is not. Human attention is therefore spent where it cannot be replaced (O3), not spread thin across all of them.

5. The O2 policy (requires operator ratification before use)

O2 fixes by subtraction: the unsupported specificity is removed or generalised rather than replaced with a researched value.

• It cannot introduce a new error, because it asserts strictly less than the file did before.
• It requires no new fact-finding, which is what makes it cheap.
• It reduces information density. That is the real cost, and it is an operator decision, not an engineering one.

The position this design recommends: a knowledge base that says less and says nothing false is worth more than one carrying stale precision. The corpus is publicly distributed; an incorrect specific number is a worse failure than an honest general statement.

Ratifying O2 also resolves the standing source_silent question as one class decision instead of 190 individual ones. Until it is ratified, every O2 candidate falls to O3.

6. What stays human, permanently

• Never auto-fix. No fix reaches a file without human confirmation of the class (O1/O2) or the item (O3). A judge false positive must become a human review, never silent corruption of a public file.
• Subagents never write. Proposal generation is read-only fan-out; all writes happen in one place (§8).
• O3 is not a backlog to automate later. It is the class where the corrected assertion is a judgement call, and it is the reason the loop reaches ~100 % fix precision on top of a fallible judge.

7. Evidence freshness

Each evidence_quote is current as of the judge's fetch date, not the fix date. Before any file is touched, the evidence base is refreshed by re-fetching per distinct URL — 540, not 712 — as read-only fan-out. A refreshed quote that no longer supports the flag re-routes the item (possibly closing it as no longer an error). This preserves the verification duty — fresh confirmation before a public file changes — without paying for 712 separate fetches.

8. Concurrency: one writer, many readers

R11 phases that are machine-bound (evidence refresh, proposal generation) may run across concurrent sessions. The corpus and the ledger may not.

Single-writer state — never written by more than one session: scripts/kb-eval/data/judge-pass-manifest.json, the corpus files themselves, and the repo's state file.

Protocol:

• Worker sessions write only disjointly-named artefacts to a local, untracked working directory (one file per unit of work). They do not ingest, stamp, or commit.
• One integrator session ingests serially, stamps, and commits.
• If two sessions must write corpus files concurrently, shard by file, never by claim, one git worktree per shard. Disjoint file sets merge without conflict; the ledger is still written only by the integrator, after merge.

Git worktrees share the main .git directory, so the Layer B pre-commit scan symlink applies inside every worktree — verified 2026-08-03 by creating a worktree and resolving git rev-parse --git-common-dir plus the hook target. The security gate does not weaken under sharding.

The reason this protocol is explicit: whole-tree backup/restore in the write drivers previously caused silent data loss across concurrent sessions (closed in the write-safety hardening pass — scoped restore + atomic writes). Sharded writes without a single-writer rule would reintroduce that class through a different door.

9. No full re-judge after fixing

A fixed file does not require a fresh judge pass over all of its claims.

• Claims already judged grounded keep that verdict; the fix did not touch them.
• A fixed claim's evidence is the O1 invariant (§4) or the human confirmation (§5–6), recorded with the fix.
• The programme's end-proof is a fresh blind gold sample measured after the fixes, not a re-run of the corpus pass.

This is stated explicitly because assuming otherwise would silently add a second full corpus pass to the plan.

10. Pilot and acceptance criterion

Before any scaling, run the classifier and the O1 driver against the 24 files carrying ≥7 not_grounded flags (202 flags) — a quarter of the volume in a tenth of the files, and the densest available sample.

Recount 2026-08-03 after R7.5 completed (ledger 222 → 243 records). The rule is unchanged — densest ≥7 sample — only the count moved: 21 files / 178 flags was measured against the 222-record ledger, before the last 21 R7.5 files were ingested. Three files entered the sample (semantic-caching-patterns.md, small-language-models-economics.md, vector-storage-cost-optimization.md). The threshold counts not_grounded only, not source_silent; on all-flags it would be 39 files / 352.

Measure and record:

1. the actual O1 / O2 / O3 split (this design's central unknown);
2. how much of R8 resolves as O2 rather than O3 (§3);
3. the O1 driver's abort rate — items it could not prove, which must land in O3;
4. review throughput per operation class, measured rather than assumed.

Scale to the remaining files only on measured numbers. If the split is materially worse than assumed, that is known after one session rather than after ten.

RUN 2026-08-03 — results in docs/r11-pilot-results.md. The split is materially worse than assumed: 9 provable, correct value swaps in the whole 776-flag not_grounded population (1.2 %), all of them api-version bumps. The pilot also falsifies §4 as written — run exactly as specified it admitted 6 swaps of which 4 were wrong edits (unit crossing, metric crossing, two mutilated identifiers), so the invariant is not "stronger than human review at scale". A context-correspondence condition was added; read §4 together with the results doc, not on its own. §5 (O2) and the STATUS_SYNONYM class are the open operator decisions, and they now carry the whole programme's leverage.

11. Out of scope

• Rebuild instead of repair. Regenerating flagged files from source rather than editing them is a settled decision: the corpus is repaired, not rebuilt. Reopening it is an operator call, not a design choice made here.
• Auto-fix in any form (§6).
• Price and other unsourced claims beyond the O2 class decision — these remain operator-gated as a separate matter.