ms-ai-architect/docs/r11-tiered-fix-design.md

# R11 execution design — tiered fixes over an already-evidenced flag population

**How the R11 fix step consumes the judge-pass flags. Extends
`docs/r11-flag-format-2026-07.md` (the record contract) with the execution
contract: what the fix operation is per flag, what is machine-provable, what
requires human judgement, and what may never be automated.**

Status: design spec. Written 2026-08-03 against a ledger snapshot of **222
records / 902 flags** (`scripts/kb-eval/data/judge-pass-manifest.json`). The
population is **not yet complete** — batch R7.5 stood at 30 of 51 files — so
every count below is a snapshot, not a final figure. The design does not depend
on the exact counts; the classifier re-derives them at run time.

---

## 1. The measurement this design is built on

**All 712 `not_grounded` flags carry a non-empty verbatim `evidence_quote`.**
712 of 712, measured. Every flag also carries the `evidence_url` the judge
actually fetched, the rule that fired, and a one-sentence `reason` stating what
the source said versus what the claim said.

The consequence is the whole point of this document: **R11 does not start from
zero evidence.** The expensive half — locating the authoritative page, reading
it, and extracting the passage that decides the claim — was already paid for by
the judge pass. A fix step designed as "fetch the source, confirm, correct"
re-does work that is already on disk.

What actually remains per flag is narrower:

1. is the quote still current (freshness, §7), and
2. what should the sentence say instead (§3).

## 2. Measured shape of the flag population

| Measurement | Value |
|---|---|
| Flags total | 902 |
| `not_grounded` (fix targets per flag-format spec) | 712 |
| `source_silent` (recorded, not a fix target) | 190 |
| Flags carrying a verbatim `evidence_quote` | **712 / 712** |
| Distinct `evidence_url` across the 712 | **540** |
| Files carrying at least one `not_grounded` flag | 199 |
| Files with 1 flag | 45 |
| Files with 2–6 flags | 133 (489 flags) |
| Files with ≥7 flags | 21 (**178 flags — 25 % of the volume in 10 % of the files**) |

Rule distribution over the 712: R8 339 · R2 176 · (no rule) 72 · R4 41 · R3 40 ·
R7 28 · R1 16.

**540 distinct source URLs for 712 flags** is the number that forecloses the
obvious optimisation: there is no batching win hiding in shared sources. The
largest cluster is 12 flags on one page. This is ~700 separate facts, and the
work is irreducibly per-claim.

## 3. Three fix operations — the partition is by operation, not by rule

The tiering is defined by **what the fix does to the file**, because that is what
determines whether a machine can prove it and whether a human must decide it.
Rule codes are a signal, not the partition.

| Op | Fix operation | Provable? | Who decides |
|---|---|---|---|
| **O1** | **Value swap** — the claim states X, the cited source states Y; replace the token. | **Yes** (§4) | Machine proposes, human reviews a diff list |
| **O2** | **Subtraction** — remove or generalise the specificity the source does not support. Covers: source-silent claims, absent entities (retired SKUs/models/services), and multi-part claims where the failing sub-assertion can be dropped while the grounded part survives. | Partly — the *invariant* is that no new fact is introduced | Human ratifies the policy once (§5), then reviews proposals |
| **O3** | **Rewrite** — the corrected sentence requires a judgement about what to assert. | No | Human, per claim |

**Do not read the §2 rule counts as tier sizes.** They overlap: of the 202 flags
whose claim *and* quote contain a numeric token — the naive O1 signal — **94 are
R8 multi-part claims**, which are not clean swaps. Heuristics over this
population give upper bounds in several directions at once, never a partition.

**The classifier fails closed to O3.** If it cannot prove an item is O1 or O2, the
item is O3 and a human sees it. A misrouted O3 costs one review; a misrouted O1
ships a wrong edit to a public file.

R8 deserves a specific note, because it is the largest rule class and it is *not*
automatically the most expensive one: the judge's `reason` names **which**
sub-assertion failed. Where the grounded part stands on its own, the fix is O2
(drop the unsupported part), not O3 (rewrite the sentence). ~~How much of R8 falls
that way is unknown and is a primary pilot measurement (§10).~~

**MEASURED 2026-08-03 — `docs/r11-pilot-results.md` §9. It falls that way for a
minority.** Over the 46 pilot flags with the O2 shape (R8 ∧ `MULTI_PART_CLAIM`):
17 O2 candidates, 29 O3, and **all 29 are blocked by §5 condition 3** — the
source supplies a corrected value, so the fix is a swap or a rewrite and
subtraction would destroy true information. Only 2 of the 46 clear both
human-judged conditions affirmatively. The paragraph above is not wrong, but the
case it describes is the exception in R8, not the rule.

## 4. The O1 invariant (what makes a value swap provable)

An O1 proposal is only valid if, after the edit:

1. the new value appears **verbatim inside the cited `evidence_quote`**, and
2. the rest of the line is **byte-identical** to before, and
3. exactly one line in the file changed.

A driver that cannot establish all three for an item **aborts before writing** and
routes the item to O3. This is the same discipline already proven in the
header-backfill drivers (frozen manifest, hard per-file invariant, abort before
write, idempotent re-run, `atomicWriteSync`) — see `scripts/kb-update/backfill-*.mjs`.

~~This invariant is deliberately stronger than human review at scale.~~
**FALSIFIED 2026-08-03 — see `docs/r11-pilot-results.md` §2.** Run exactly as
written it admitted 6 swaps of which **4 were wrong edits**. Conditions 1–3
constrain where the value *came from* and what the edit *looks like*, and nothing
about whether the two tokens denote the same quantity. Conditions 1–3 are
necessary; they are not sufficient.

### 4a. Condition 4 — context correspondence (added 2026-08-03)

4. the token must sit under **the same label, or the same trailing unit, on both
   sides** (`contextCorresponds()` in `scripts/kb-eval/lib/fix-op.mjs`).

Deliberately lexical, with no translation table beyond §4b: a swap is therefore
provable essentially only where the context is language-neutral — a URL, a code
sample, a parameter key.

**Condition 5 — the applied class is `iso_date` only** (operator decision,
2026-08-03). Condition 4 is still not sufficient: a matching identifier *prefix*
(`AI-`, `gpt-`, `Agent `) satisfies it while the digit is part of a **name**
rather than a quantity, which produced `AI-900` → `AI-901`, `gpt-4o` →
`gpt-5.1o` (twice) and a Java-agent downgrade. Hand-verification over the whole
population: **`iso_date` 9/9 correct, `number` and `version` 0/6**. A driver may
apply `iso_date` proposals and **must never apply `number` or `version` ones**.
The classifier keeps reporting all admitted types — that is the measurement — and
marks the applicable set as `o1_recommended`.

### 4b. The ratified status-synonym table (operator decision, 2026-08-03)

`STATUS_SYNONYM` — 15 pilot flags, **54 corpus-wide** — is the class where the
corpus writes `**Preview**` / `**GA**` while the source writes *"generally
available"*. A narrow, **closed** equivalence table is ratified:

| Corpus-side label | Source-side phrasing (must appear verbatim in `evidence_quote`) |
|---|---|
| `GA` | `generally available`, `general availability` |
| `Preview`, `Public Preview` | `public preview`, `preview` |
| `Private Preview` | `private preview` |
| `Deprecated`, `Utfaset` | `deprecated`, `retired` |

Three constraints, because this is the **one** place where the value written into
the file does not itself appear verbatim in the quote:

- The table is **closed**. Any pair not listed aborts to O3; it is never extended
  by inference at run time.
- The file-side token must be a **complete lifecycle label** (a whole table cell
  or emphasised token), never a substring of a longer sentence.
- The written value is the **corpus-side** equivalent with the file's own markup
  preserved (`**Preview**` → `**GA**`), never the English phrase pasted in.

**IMPLEMENTED 2026-08-03** in `classifyStatusSynonym()` / `fileStatusLabel()` /
`sourceStatusRows()` (`scripts/kb-eval/lib/fix-op.mjs`), 19 tests. Two
implementation decisions the ratified text left open, both resolved towards
failing closed:

- **The status locator is LINE-scoped**, not block-scoped like the numeric one.
  Lifecycle vocabulary repeats down every column of a status table, so a block
  window is ambiguous by construction; all 15 pilot flags in this class point at
  the row that carries the claim.
- **A quote asserting two different rows aborts** (`SOURCE_STATUS_AMBIGUOUS`), and
  a row with two corpus-side labels writes the **first** — the least specific one,
  so a source saying only "preview" can never produce "Public Preview".

Aborts keep the `STATUS_SYNONYM` code and name their cause in `detail.reason`, so
the §10 abort taxonomy stays comparable across the implementation.

**Measured: 15 pilot / 54 corpus-wide flags → 5 and 8 proven; 5 of the 8 are
correct** (`docs/r11-pilot-results.md` §8 + appendix B). The three defects are one
family: §4b binds the table, the completeness of the file label and the written
value, and **nothing about whether the source phrasing refers to the row's own
subject** — provenance without referent, the same defect that falsified §4. The
class is therefore **review-grade, not apply-grade**: `status` is absent from
`o1_recommended` and no driver applies it.

**Open operator decision — a referent condition for §4b.** Two candidates are
costed over the eight in `r11-pilot-results.md` §8; both kill wrong proposals and
no correct one, and neither is implemented, because extending a table ratified as
*closed* is an operator decision, exactly as condition 5 was in §4a.

## 5. The O2 policy — RATIFIED 2026-08-03, with a remainder check

O2 fixes by **subtraction**: the unsupported specificity is removed or
generalised rather than replaced with a researched value.

**Ratified by the operator on 2026-08-03, with one condition: the remainder
check.** Subtraction is not admitted as a blanket rule, because the pilot found
two ways it fails (`docs/r11-pilot-results.md` §5):

- It can leave a **misleading remainder**. Removing `er GA (juni 2025)` from a
  claim about a tool the source calls *deprecated* leaves that tool standing in a
  list of available ones. Strictly less asserted, still misleading.
- It can **destroy true information**. Dropping `prebuilt-check` from a model list
  removes a model that exists — its ID is `prebuilt-check.us`, so the correct fix
  is a swap.

**An O2 proposal is valid only if all three hold, and a human confirms them:**

1. the edited sentence asserts **strictly less** than before;
2. the **remainder carries no false or misleading standing implication** — read as
   a reader would read it, not as a logician would;
3. nothing the source **confirms** is removed. Where the source supports a
   corrected value, the fix is O1 or O3, never subtraction.

Conditions 2 and 3 require a human to read the remainder. O2 is therefore
**cheaper than O3 — no fact-finding — but not mechanical**, and the §10
throughput assumption should be re-measured against that.

- It requires **no new fact-finding**, which is what makes it cheap.
- It **reduces information density**. That is the real cost, and it is an
  operator decision, not an engineering one.

The position this design recommends: a knowledge base that says less and says
nothing false is worth more than one carrying stale precision. The corpus is
publicly distributed; an incorrect specific number is a worse failure than an
honest general statement.

Ratifying O2 also resolves the standing `source_silent` question as **one class
decision** instead of 190 individual ones. ~~Until it is ratified, every O2
candidate falls to O3.~~ Ratified — O2 is in use, subject to the remainder check
above. **Not implemented in the classifier, and now measured to be the right
call:** the classifier still routes every non-O1 item to O3, because O2 candidacy
turns on the judge's prose `reason`. The prose classification was run separately
(`docs/r11-pilot-results.md` §9) and found that condition 3 forecloses **every**
non-candidate in the class — a mechanical O2 driver would therefore have proposed
deletions where the source hands over a corrected value, which is precisely the
F2 failure. O2 output is a human review list, like §4b's.

## 6. What stays human, permanently

- **Never auto-fix.** No fix reaches a file without human confirmation of the
  class (O1/O2) or the item (O3). A judge false positive must become a human
  review, never silent corruption of a public file.
- **Subagents never write.** Proposal generation is read-only fan-out; all
  writes happen in one place (§8).
- **O3 is not a backlog to automate later.** It is the class where the corrected
  assertion is a judgement call, and it is the reason the loop reaches ~100 % fix
  precision on top of a fallible judge.

## 7. Evidence freshness

Each `evidence_quote` is current as of the judge's fetch date, not the fix date.
Before any file is touched, the evidence base is refreshed by **re-fetching per
distinct URL — 540, not 712** — as read-only fan-out. A refreshed quote that no
longer supports the flag re-routes the item (possibly closing it as no longer an
error). This preserves the verification duty — fresh confirmation before a public
file changes — without paying for 712 separate fetches.

## 8. Concurrency: one writer, many readers

R11 phases that are machine-bound (evidence refresh, proposal generation) may run
across concurrent sessions. The corpus and the ledger may not.

**Single-writer state** — never written by more than one session:
`scripts/kb-eval/data/judge-pass-manifest.json`, the corpus files themselves, and
the repo's state file.

**Protocol:**

- Worker sessions write **only** disjointly-named artefacts to a local, untracked
  working directory (one file per unit of work). They do not ingest, stamp, or
  commit.
- One integrator session ingests serially, stamps, and commits.
- If two sessions must write corpus files concurrently, shard **by file, never by
  claim**, one git worktree per shard. Disjoint file sets merge without conflict;
  the ledger is still written only by the integrator, after merge.

Git worktrees share the main `.git` directory, so the Layer B `pre-commit` scan
symlink applies inside every worktree — verified 2026-08-03 by creating a
worktree and resolving `git rev-parse --git-common-dir` plus the hook target.
The security gate does not weaken under sharding.

The reason this protocol is explicit: whole-tree backup/restore in the write
drivers previously caused silent data loss across concurrent sessions (closed in
the write-safety hardening pass — scoped restore + atomic writes). Sharded writes
without a single-writer rule would reintroduce that class through a different door.

## 9. No full re-judge after fixing

A fixed file does **not** require a fresh judge pass over all of its claims.

- Claims already judged `grounded` keep that verdict; the fix did not touch them.
- A fixed claim's evidence is the O1 invariant (§4) or the human confirmation
  (§5–6), recorded with the fix.
- The programme's end-proof is a **fresh blind gold sample** measured after the
  fixes, not a re-run of the corpus pass.

This is stated explicitly because assuming otherwise would silently add a second
full corpus pass to the plan.

## 10. Pilot and acceptance criterion

Before any scaling, run the classifier and the O1 driver against the **24 files
carrying ≥7 `not_grounded` flags (202 flags)** — a quarter of the volume in a
tenth of the files, and the densest available sample.

> Recount 2026-08-03 after R7.5 completed (ledger 222 → 243 records). The rule is
> unchanged — densest ≥7 sample — only the count moved: 21 files / 178 flags was
> measured against the 222-record ledger, before the last 21 R7.5 files were
> ingested. Three files entered the sample (`semantic-caching-patterns.md`,
> `small-language-models-economics.md`, `vector-storage-cost-optimization.md`).
> The threshold counts `not_grounded` only, not `source_silent`; on all-flags it
> would be 39 files / 352.

Measure and record:

1. the actual O1 / O2 / O3 split (this design's central unknown);
2. how much of R8 resolves as O2 rather than O3 (§3);
3. the O1 driver's abort rate — items it could not prove, which must land in O3;
4. review throughput per operation class, measured rather than assumed.

Scale to the remaining files only on measured numbers. If the split is materially
worse than assumed, that is known after one session rather than after ten.

> **RUN 2026-08-03 — results in `docs/r11-pilot-results.md`.** The split is
> materially worse than assumed: **9 provable, correct value swaps in the whole
> 776-flag `not_grounded` population (1.2 %)**, all of them `api-version` bumps.
> The pilot also falsifies §4 as written — run exactly as specified it admitted 6
> swaps of which **4 were wrong edits** (unit crossing, metric crossing, two
> mutilated identifiers), so the invariant is *not* "stronger than human review at
> scale". A context-correspondence condition was added; read §4 together with the
> results doc, not on its own. §5 (O2) and the `STATUS_SYNONYM` class are the open
> operator decisions, and they now carry the whole programme's leverage.

## 11. Out of scope

- **Rebuild instead of repair.** Regenerating flagged files from source rather
  than editing them is a settled decision: the corpus is repaired, not rebuilt.
  Reopening it is an operator call, not a design choice made here.
- **Auto-fix in any form** (§6).
- **Price and other unsourced claims** beyond the O2 class decision — these
  remain operator-gated as a separate matter.