ms-ai-architect/docs/r11-tiered-fix-design.md

R11 execution design — tiered fixes over an already-evidenced flag population

How the R11 fix step consumes the judge-pass flags. Extends docs/r11-flag-format-2026-07.md (the record contract) with the execution contract: what the fix operation is per flag, what is machine-provable, what requires human judgement, and what may never be automated.

Status: design spec. Written 2026-08-03 against a ledger snapshot of 222 records / 902 flags (scripts/kb-eval/data/judge-pass-manifest.json). The population is not yet complete — batch R7.5 stood at 30 of 51 files — so every count below is a snapshot, not a final figure. The design does not depend on the exact counts; the classifier re-derives them at run time.

---

1. The measurement this design is built on

All 712 not_grounded flags carry a non-empty verbatim evidence_quote. 712 of 712, measured. Every flag also carries the evidence_url the judge actually fetched, the rule that fired, and a one-sentence reason stating what the source said versus what the claim said.

The consequence is the whole point of this document: R11 does not start from zero evidence. The expensive half — locating the authoritative page, reading it, and extracting the passage that decides the claim — was already paid for by the judge pass. A fix step designed as "fetch the source, confirm, correct" re-does work that is already on disk.

What actually remains per flag is narrower:

1. is the quote still current (freshness, §7), and
2. what should the sentence say instead (§3).

2. Measured shape of the flag population

Measurement	Value
Flags total	902
not_grounded (fix targets per flag-format spec)	712
source_silent (recorded, not a fix target)	190
Flags carrying a verbatim evidence_quote	712 / 712
Distinct evidence_url across the 712	540
Files carrying at least one not_grounded flag	199
Files with 1 flag	45
Files with 2–6 flags	133 (489 flags)
Files with ≥7 flags	21 (178 flags — 25 % of the volume in 10 % of the files)

Rule distribution over the 712: R8 339 · R2 176 · (no rule) 72 · R4 41 · R3 40 · R7 28 · R1 16.

540 distinct source URLs for 712 flags is the number that forecloses the obvious optimisation: there is no batching win hiding in shared sources. The largest cluster is 12 flags on one page. This is ~700 separate facts, and the work is irreducibly per-claim.

3. Three fix operations — the partition is by operation, not by rule

The tiering is defined by what the fix does to the file, because that is what determines whether a machine can prove it and whether a human must decide it. Rule codes are a signal, not the partition.

Op	Fix operation	Provable?	Who decides
O1	Value swap — the claim states X, the cited source states Y; replace the token.	Yes (§4)	Machine proposes, human reviews a diff list
O2	Subtraction — remove or generalise the specificity the source does not support. Covers: source-silent claims, absent entities (retired SKUs/models/services), and multi-part claims where the failing sub-assertion can be dropped while the grounded part survives.	Partly — the invariant is that no new fact is introduced	Human ratifies the policy once (§5), then reviews proposals
O3	Rewrite — the corrected sentence requires a judgement about what to assert.	No	Human, per claim

Do not read the §2 rule counts as tier sizes. They overlap: of the 202 flags whose claim and quote contain a numeric token — the naive O1 signal — 94 are R8 multi-part claims, which are not clean swaps. Heuristics over this population give upper bounds in several directions at once, never a partition.

The classifier fails closed to O3. If it cannot prove an item is O1 or O2, the item is O3 and a human sees it. A misrouted O3 costs one review; a misrouted O1 ships a wrong edit to a public file.

R8 deserves a specific note, because it is the largest rule class and it is not automatically the most expensive one: the judge's reason names which sub-assertion failed. Where the grounded part stands on its own, the fix is O2 (drop the unsupported part), not O3 (rewrite the sentence). How much of R8 falls that way is unknown and is a primary pilot measurement (§10).

MEASURED 2026-08-03 — docs/r11-pilot-results.md §9. It falls that way for a minority. Over the 46 pilot flags with the O2 shape (R8 ∧ MULTI_PART_CLAIM): 17 O2 candidates, 29 O3, and all 29 are blocked by §5 condition 3 — the source supplies a corrected value, so the fix is a swap or a rewrite and subtraction would destroy true information. Only 2 of the 46 clear both human-judged conditions affirmatively. The paragraph above is not wrong, but the case it describes is the exception in R8, not the rule.

4. The O1 invariant (what makes a value swap provable)

An O1 proposal is only valid if, after the edit:

1. the new value appears verbatim inside the cited evidence_quote, and
2. the rest of the line is byte-identical to before, and
3. exactly one line in the file changed.

A driver that cannot establish all three for an item aborts before writing and routes the item to O3. This is the same discipline already proven in the header-backfill drivers (frozen manifest, hard per-file invariant, abort before write, idempotent re-run, atomicWriteSync) — see scripts/kb-update/backfill-*.mjs.

This invariant is deliberately stronger than human review at scale. FALSIFIED 2026-08-03 — see docs/r11-pilot-results.md §2. Run exactly as written it admitted 6 swaps of which 4 were wrong edits. Conditions 1–3 constrain where the value came from and what the edit looks like, and nothing about whether the two tokens denote the same quantity. Conditions 1–3 are necessary; they are not sufficient.

4a. Condition 4 — context correspondence (added 2026-08-03)

4. the token must sit under the same label, or the same trailing unit, on both sides (contextCorresponds() in scripts/kb-eval/lib/fix-op.mjs).

Deliberately lexical, with no translation table beyond §4b: a swap is therefore provable essentially only where the context is language-neutral — a URL, a code sample, a parameter key.

Condition 5 — the applied class is iso_date only (operator decision, 2026-08-03). Condition 4 is still not sufficient: a matching identifier prefix (AI-, gpt-, Agent ) satisfies it while the digit is part of a name rather than a quantity, which produced AI-900 → AI-901, gpt-4o → gpt-5.1o (twice) and a Java-agent downgrade. Hand-verification over the whole population: iso_date 9/9 correct, number and version 0/6. A driver may apply iso_date proposals and must never apply number or version ones. The classifier keeps reporting all admitted types — that is the measurement — and marks the applicable set as o1_recommended.

4b. The ratified status-synonym table (operator decision, 2026-08-03)

STATUS_SYNONYM — 15 pilot flags, 54 corpus-wide — is the class where the corpus writes **Preview** / **GA** while the source writes "generally available". A narrow, closed equivalence table is ratified:

Corpus-side label	Source-side phrasing (must appear verbatim in evidence_quote)
GA	generally available, general availability
Preview, Public Preview	public preview, preview
Private Preview	private preview
Deprecated, Utfaset	deprecated, retired

Three constraints, because this is the one place where the value written into the file does not itself appear verbatim in the quote:

• The table is closed. Any pair not listed aborts to O3; it is never extended by inference at run time.
• The file-side token must be a complete lifecycle label (a whole table cell or emphasised token), never a substring of a longer sentence.
• The written value is the corpus-side equivalent with the file's own markup preserved (**Preview** → **GA**), never the English phrase pasted in.

IMPLEMENTED 2026-08-03 in classifyStatusSynonym() / fileStatusLabel() / sourceStatusRows() (scripts/kb-eval/lib/fix-op.mjs), 19 tests. Two implementation decisions the ratified text left open, both resolved towards failing closed:

• The status locator is LINE-scoped, not block-scoped like the numeric one. Lifecycle vocabulary repeats down every column of a status table, so a block window is ambiguous by construction; all 15 pilot flags in this class point at the row that carries the claim.
• A quote asserting two different rows aborts (SOURCE_STATUS_AMBIGUOUS), and a row with two corpus-side labels writes the first — the least specific one, so a source saying only "preview" can never produce "Public Preview".

Aborts keep the STATUS_SYNONYM code and name their cause in detail.reason, so the §10 abort taxonomy stays comparable across the implementation.

Measured: 15 pilot / 54 corpus-wide flags → 5 and 8 proven; 5 of the 8 are correct (docs/r11-pilot-results.md §8 + appendix B). The three defects are one family: §4b binds the table, the completeness of the file label and the written value, and nothing about whether the source phrasing refers to the row's own subject — provenance without referent, the same defect that falsified §4. The class is therefore review-grade, not apply-grade: status is absent from o1_recommended and no driver applies it.

Open operator decision — a referent condition for §4b. Two candidates are costed over the eight in r11-pilot-results.md §8; both kill wrong proposals and no correct one, and neither is implemented, because extending a table ratified as closed is an operator decision, exactly as condition 5 was in §4a.

5. The O2 policy — RATIFIED 2026-08-03, with a remainder check

O2 fixes by subtraction: the unsupported specificity is removed or generalised rather than replaced with a researched value.

Ratified by the operator on 2026-08-03, with one condition: the remainder check. Subtraction is not admitted as a blanket rule, because the pilot found two ways it fails (docs/r11-pilot-results.md §5):

• It can leave a misleading remainder. Removing er GA (juni 2025) from a claim about a tool the source calls deprecated leaves that tool standing in a list of available ones. Strictly less asserted, still misleading.
• It can destroy true information. Dropping prebuilt-check from a model list removes a model that exists — its ID is prebuilt-check.us, so the correct fix is a swap.

An O2 proposal is valid only if all three hold, and a human confirms them:

1. the edited sentence asserts strictly less than before;
2. the remainder carries no false or misleading standing implication — read as a reader would read it, not as a logician would;
3. nothing the source confirms is removed. Where the source supports a corrected value, the fix is O1 or O3, never subtraction.

Conditions 2 and 3 require a human to read the remainder. O2 is therefore cheaper than O3 — no fact-finding — but not mechanical, and the §10 throughput assumption should be re-measured against that.

• It requires no new fact-finding, which is what makes it cheap.
• It reduces information density. That is the real cost, and it is an operator decision, not an engineering one.

The position this design recommends: a knowledge base that says less and says nothing false is worth more than one carrying stale precision. The corpus is publicly distributed; an incorrect specific number is a worse failure than an honest general statement.

Ratifying O2 also resolves the standing source_silent question as one class decision instead of 190 individual ones. Until it is ratified, every O2 candidate falls to O3. Ratified — O2 is in use, subject to the remainder check above. Not implemented in the classifier, and now measured to be the right call: the classifier still routes every non-O1 item to O3, because O2 candidacy turns on the judge's prose reason. The prose classification was run separately (docs/r11-pilot-results.md §9) and found that condition 3 forecloses every non-candidate in the class — a mechanical O2 driver would therefore have proposed deletions where the source hands over a corrected value, which is precisely the F2 failure. O2 output is a human review list, like §4b's.

6. What stays human, permanently

• Never auto-fix. No fix reaches a file without human confirmation of the class (O1/O2) or the item (O3). A judge false positive must become a human review, never silent corruption of a public file.
• Subagents never write. Proposal generation is read-only fan-out; all writes happen in one place (§8).
• O3 is not a backlog to automate later. It is the class where the corrected assertion is a judgement call, and it is the reason the loop reaches ~100 % fix precision on top of a fallible judge.

7. Evidence freshness

Each evidence_quote is current as of the judge's fetch date, not the fix date. Before any file is touched, the evidence base is refreshed by re-fetching per distinct URL — 540, not 712 — as read-only fan-out. A refreshed quote that no longer supports the flag re-routes the item (possibly closing it as no longer an error). This preserves the verification duty — fresh confirmation before a public file changes — without paying for 712 separate fetches.

8. Concurrency: one writer, many readers

R11 phases that are machine-bound (evidence refresh, proposal generation) may run across concurrent sessions. The corpus and the ledger may not.

Single-writer state — never written by more than one session: scripts/kb-eval/data/judge-pass-manifest.json, the corpus files themselves, and the repo's state file.

Protocol:

• Worker sessions write only disjointly-named artefacts to a local, untracked working directory (one file per unit of work). They do not ingest, stamp, or commit.
• One integrator session ingests serially, stamps, and commits.
• If two sessions must write corpus files concurrently, shard by file, never by claim, one git worktree per shard. Disjoint file sets merge without conflict; the ledger is still written only by the integrator, after merge.

Git worktrees share the main .git directory, so the Layer B pre-commit scan symlink applies inside every worktree — verified 2026-08-03 by creating a worktree and resolving git rev-parse --git-common-dir plus the hook target. The security gate does not weaken under sharding.

The reason this protocol is explicit: whole-tree backup/restore in the write drivers previously caused silent data loss across concurrent sessions (closed in the write-safety hardening pass — scoped restore + atomic writes). Sharded writes without a single-writer rule would reintroduce that class through a different door.

9. No full re-judge after fixing

A fixed file does not require a fresh judge pass over all of its claims.

• Claims already judged grounded keep that verdict; the fix did not touch them.
• A fixed claim's evidence is the O1 invariant (§4) or the human confirmation (§5–6), recorded with the fix.
• The programme's end-proof is a fresh blind gold sample measured after the fixes, not a re-run of the corpus pass.

This is stated explicitly because assuming otherwise would silently add a second full corpus pass to the plan.

10. Pilot and acceptance criterion

Before any scaling, run the classifier and the O1 driver against the 24 files carrying ≥7 not_grounded flags (202 flags) — a quarter of the volume in a tenth of the files, and the densest available sample.

Recount 2026-08-03 after R7.5 completed (ledger 222 → 243 records). The rule is unchanged — densest ≥7 sample — only the count moved: 21 files / 178 flags was measured against the 222-record ledger, before the last 21 R7.5 files were ingested. Three files entered the sample (semantic-caching-patterns.md, small-language-models-economics.md, vector-storage-cost-optimization.md). The threshold counts not_grounded only, not source_silent; on all-flags it would be 39 files / 352.

Measure and record:

1. the actual O1 / O2 / O3 split (this design's central unknown);
2. how much of R8 resolves as O2 rather than O3 (§3);
3. the O1 driver's abort rate — items it could not prove, which must land in O3;
4. review throughput per operation class, measured rather than assumed.

Scale to the remaining files only on measured numbers. If the split is materially worse than assumed, that is known after one session rather than after ten.

RUN 2026-08-03 — results in docs/r11-pilot-results.md. The split is materially worse than assumed: 9 provable, correct value swaps in the whole 776-flag not_grounded population (1.2 %), all of them api-version bumps. The pilot also falsifies §4 as written — run exactly as specified it admitted 6 swaps of which 4 were wrong edits (unit crossing, metric crossing, two mutilated identifiers), so the invariant is not "stronger than human review at scale". A context-correspondence condition was added; read §4 together with the results doc, not on its own. §5 (O2) and the STATUS_SYNONYM class are the open operator decisions, and they now carry the whole programme's leverage.

11. Out of scope

• Rebuild instead of repair. Regenerating flagged files from source rather than editing them is a settled decision: the corpus is repaired, not rebuilt. Reopening it is an operator call, not a design choice made here.
• Auto-fix in any form (§6).
• Price and other unsourced claims beyond the O2 class decision — these remain operator-gated as a separate matter.