ms-ai-architect/docs/r11-pilot-results.md

# R11 pilot results — measured, 2026-08-03

**The §10 acceptance measurement of `docs/r11-tiered-fix-design.md`, run against
the live ledger. No KB file was edited and no ledger record was written.**

Instrument: `scripts/kb-eval/lib/fix-op.mjs` (+ `tests/kb-eval/test-fix-op-classify.test.mjs`,
49 tests after §4b) driven by `scripts/kb-eval/classify-fix-ops.mjs`. The classifier **is**
the O1 driver with writes disabled — it constructs the swap and checks the §4
invariant, so measurements 1 and 3 come out of the mechanism that would later
touch the corpus, not out of a proxy heuristic.

Artefact: `scripts/kb-eval/data/r11-pilot-classification.json` (untracked,
regenerable; per-flag records so the run can be re-analysed without re-running).
It holds the **pilot** run — `node scripts/kb-eval/classify-fix-ops.mjs --write`.
Every corpus-wide figure below is from `--threshold 1`, and the per-table
reproduce command is stated where it is used.

> **Two different 202s.** This population is 202 flags. §3's "202 flags whose
> claim *and* quote contain a numeric token" is a different 202, measured over
> the full 712-flag population. They are unrelated.

---

## 1. The headline

**Nine provable, correct value swaps exist in the entire 776-flag `not_grounded`
population — 1.2 %.** The machine half of the R11 tiering buys nine edits. Every
other flag needs a human.

| Population | Files | Flags | O1 admitted | O1 hand-verified correct |
|---|---|---|---|---|
| Pilot (`not_grounded` ≥ 7) | 24 | 202 | 2 | 2 |
| Whole `not_grounded` corpus | 218 | 776 | 15 | **9** |

> These are **numeric-path** figures, and they stay that way. §4b (the status
> synonym table) was implemented afterwards and adds a separate class with its own
> hand-verification — see §8. A run today prints O1 = 7 (pilot) and 23 (corpus)
> because the status proposals are included in the total; the numeric line above
> is unchanged and is still what `s4_as_written` compares against.

Reproduce: `node scripts/kb-eval/classify-fix-ops.mjs` (pilot) and
`node scripts/kb-eval/classify-fix-ops.mjs --threshold 1` (corpus). The nine are
enumerated with verdicts in appendix A — that hand-verification is the only thing
separating 9 from 15, so it is recorded rather than left in a session transcript.

This is the answer §10 asked for, and it is materially worse than the design
assumed: *"If the split is materially worse than assumed, that is known after one
session rather than after ten."*

> **The O2 half, measured afterwards (§9), does not rescue the number.** Of the 46
> R8 multi-part claims that have the O2 shape, 17 are candidates and **2 are clean
> subtractions on the available evidence**; all 29 non-candidates fail because the
> source supplies a *corrected value*, which makes them swaps or rewrites. O2's
> value in R11 is triage — it tells a human which 17 to look at first — not
> automation.

## 2. §4 as written is not sufficient — measured, not argued

§4 claims its invariant is *"deliberately stronger than human review at scale."*
It is not. Run exactly as specified over the pilot, it admitted **6 swaps, of
which 4 were wrong** — precision **2/6**:

| Proposed swap | Why it is wrong |
|---|---|
| `30-dagers` → `24-dagers` | **Unit crossing.** The quote says 24 **hours**. |
| `3000 requests/sekund` → `50` | **Metric crossing.** The quote is a *query* throttle per index; the claim is an *indexing* rate per replica. |
| `Microsoft Agent 365` → `Agent 7` | **Identifier mutilated.** The `7` was harvested out of `E7`. |
| `text-embedding-ada-002` → `ada-2` | **Identifier mutilated.** The `2` came from a dimensions column. |

The defect is structural, not incidental. §4 constrains **where the new value
came from** (verbatim in the cited quote) and **what the edit looks like** (one
line, rest byte-identical). It constrains nothing about whether the two tokens
**denote the same quantity**. Same-type-and-provenance is not same-referent.

### The added condition

`contextCorresponds()` requires the token to sit under **the same label or the
same trailing unit on both sides**. It is deliberately lexical, with **no
translation table**: `dokumenter` is not taught to equal `documents`, because a
synonym table introduces a new fact source and is an operator decision, not an
engineering one. Consequence, measured: a swap is provable essentially only where
the context is language-neutral — a URL, a code sample, a parameter key.

## 3. The condition is necessary but still not sufficient

Corpus-wide the context condition cut 38 admissions to 15. Hand-verifying all 15
splits them cleanly by token type:

Reproduce: `node scripts/kb-eval/classify-fix-ops.mjs --threshold 1`
(the persisted artefact is the **pilot** run — the corpus-wide tables in §1 and §3
come from this threshold-1 run). Per-proposal verdicts: appendix A.

| Token type | Proposals | Correct | Wrong | Unverified | Failure mode |
|---|---|---|---|---|---|
| `iso_date` | 9 | **9** | 0 | 0 | — every one is an `api-version=` bump in a URL or code sample |
| `number` | 5 | 0 | 3 | 2 | `AI-900` → `AI-901`, `gpt-4o` → `gpt-5.1o` (×2) |
| `version` | 1 | 0 | 1 | 0 | Java agent `3.7.5` → `3.4.0` — a downgrade |

A matching identifier *prefix* (`AI-`, `gpt-`) satisfies the context condition
while the digit is part of a **name**, not a quantity. **Only `iso_date` survives
hand-verification**, and the report marks it as the sole recommended class
(`o1_recommended`). `number` and `version` proposals must not be applied.

## 4. The four §10 measurements

1. **O1 / O2 / O3 split.** O1 = 2/202 on the pilot (9/776 corpus-wide, safe
   class only). ~~O2 is **undetermined** — it does not exist as a class until §5 is
   ratified, so every non-O1 item is O3 by design.~~ §5 was ratified 2026-08-03,
   and O2 has since been measured over the class where it can exist at all — see
   measurement 2 and §9. It remains **unmeasured outside R8 ∧
   `MULTI_PART_CLAIM`**; the classifier still routes every non-O1 item to O3, so
   O3 ≥ 200/202 stands as the machine's own partition.
2. **How much of R8 resolves as O2.** **MEASURED 2026-08-03 — see §9.** R8 is
   87/202 on the pilot (366/776 corpus-wide) and yields **zero** O1. Of the
   pilot's 87, **46 are structural enumerations** (R8 ∧ `MULTI_PART_CLAIM`) — the
   O2 candidate shape. Which of them subtract cleanly turns on the judge's prose
   `reason`, and no regex reads prose, so this was done by prose classification:
   **17 of 46 (37 %) are O2 candidates, 29 are O3.** All 29 are foreclosed by
   condition 3.
3. **O1 abort rate: 99 % (200/202).** Typed, because "99 %" alone is not
   actionable:

   | Code | Pilot | Class |
   |---|---|---|
   | `MULTI_PART_CLAIM` | 96 (47.5 %) | intrinsic — not a value swap at all |
   | `MULTI_VALUE_TOKEN` | 29 | intrinsic |
   | `NO_VALUE_TOKEN` | 28 | intrinsic — the claim asserts prose |
   | `STATUS_SYNONYM` | 15 | **operator question** (§6.2) |
   | `NOT_VERBATIM` | 15 | intrinsic |
   | `LOCATOR_AMBIGUOUS` | 7 | **fixable engineering gap** |
   | `MULTI_REPLACEMENT` | 6 | intrinsic |
   | `CONTEXT_MISMATCH` | 4 | intrinsic — these are the 4 wrong edits above |

   **Only 7 of 200 aborts (3.5 %) are a fixable engineering gap.** More locator
   engineering cannot move the O1 number materially.
4. **Review throughput per class. NOT MEASURED.** It requires human review
   sessions, which have not happened. Recording it as measured would be false.

## 5. Two further findings

**F1 — subtraction can leave a misleading remainder.** §5 argues O2 *"cannot
introduce a new error, because it asserts strictly less."* True of the sentence,
false of the reader's inference. Real case: *"Deep Research tool (o3-deep-research
+ Bing) er GA (juni 2025)"* where the source says the tool is **deprecated**.
Subtracting `er GA (juni 2025)` leaves the tool standing in a list of available
tools. Strictly less asserted; still misleading. O2 therefore still requires a
human to look at the remainder — cheaper than O3 (no fact-finding) but not
mechanical.

**F2 — subtraction can destroy true information.** Real case: a list of seven
prebuilt model IDs where the judge found six correct and `prebuilt-check` wrong —
the real ID is `prebuilt-check.us`. Subtraction drops a model that **exists**; the
correct fix is a swap. Subtraction is not the safe default everywhere.

**F3 — `disposition` carries zero information.** It is `outdated` on **202 of
202** flags. `docs/r11-flag-format-2026-07.md` specifies `not_grounded →
{outdated, wrong}` with *"the human assigns which at R11"*, but the pass
hard-assigned `outdated`. Do not use it as a classifier signal. Spec/data
divergence, recorded.

**F4 — claims are not file text.** `claim` is an LLM-extracted, translated
restatement: **0 of 202** match their file line verbatim, and 188 share no 40-char
run with it. For table claims, `line` points at the **header**, not the value.
This is why the locator exists at all, and why it searches the enclosing block
rather than the line.

## 6. Operator decisions — ALL THREE RATIFIED 2026-08-03

All three were put to the operator with the recommendations below and **all three
were accepted as recommended**. The contract text now lives in
`docs/r11-tiered-fix-design.md` §4a/§4b/§5; this section records what was asked
and what the answer was.

**None of the three is implemented yet.** The classifier still aborts
`STATUS_SYNONYM` and still routes every non-O1 item to O3. A later session builds
against the ratified contract — it must not assume the code already honours it.

1. **Ratify O2 (§5)?** → **RATIFIED, with the remainder check** (not as a blanket
   rule), exactly as F1/F2 above argued. Contract: design doc §5, three
   conditions, human-confirmed.
2. **Amend §4 with a ratified synonym table?** → **RATIFIED, narrow and closed.**
   Contract: design doc §4b — four label rows, closed table, complete-label-only,
   corpus-side value with the file's own markup preserved. Unlocks up to 54
   corpus-wide flags.
3. **Is O1 worth building at all?** → **KEPT, locked to `iso_date`.** Contract:
   design doc §4a condition 5 — a driver may apply `iso_date` proposals and must
   never apply `number` or `version` ones. Nine edits corpus-wide.

## 7. What this does not change

The design's core reading survives: the expensive half (locating the source,
reading it, extracting the deciding passage) was already paid for by the judge
pass, and §7's freshness re-fetch per distinct URL is untouched. What the pilot
falsifies is the assumption that a meaningful share of that evidence converts into
machine-provable edits. It does not. R11 is a human review programme with a
nine-item machine assist, and its leverage lies entirely in the O2 decision.

---

## 8. §4b implemented — the status class measured, 2026-08-03

The ratified synonym table (`docs/r11-tiered-fix-design.md` §4b) is implemented in
`fix-op.mjs` and the class is measured. Reproduce:
`node scripts/kb-eval/classify-fix-ops.mjs --threshold 1` (`status_synonym` block).

| Population | STATUS_SYNONYM flags | Proven by §4b | Still aborting |
|---|---|---|---|
| Pilot (`not_grounded` ≥ 7) | 15 | **5** | 10 |
| Whole `not_grounded` corpus | 54 | **8** | 46 |

Why the other 46 abort, corpus-wide — this sub-distribution is the actionable
part, because the top-level `STATUS_SYNONYM` count alone says nothing:

| Reason | N | What it means |
|---|---|---|
| `NO_COMPLETE_FILE_LABEL` | 27 | the file writes the status inside a sentence — `(preview)` in a list item, `**DSPM (preview):**`, `"[Preview]: …"` in a JSON string. Constraint 2 refuses these, correctly. |
| `NO_SOURCE_STATUS` | 15 | the cited quote carries no listed lifecycle phrasing at all — the flag was never a status swap. |
| `SOURCE_STATUS_AMBIGUOUS` | 2 | the quote asserts two different rows (e.g. "…is now generally available. Partner solutions remain in preview."). |
| `FILE_ALREADY_MATCHES` | 2 | file and source agree; the mismatch was in the LLM-extracted claim, not in the corpus. |

**The class is REVIEW-grade, not apply-grade — 5 of 8 correct.** All eight were
hand-judged against the cited source (appendix B). Three defects, all one family:
§4b binds the table, the completeness of the file label and the written value, and
**nothing about whether the source phrasing refers to the row's own subject**.
That is the same provenance-without-referent defect that falsified §4 (§2), now
reproduced in the status class. `status` is therefore deliberately **absent from
`o1_recommended`**: the machine writes nothing, and every proposal reaches a human.

### Two candidate conditions, costed over the eight

Neither is implemented — extending a table the operator ratified as *closed* is an
operator decision, exactly as condition 5 was in §4a. Both are pure gain on this
population (they kill wrong proposals and no correct one), which is the number the
decision needs:

| Candidate | Kills | Correct proposals lost |
|---|---|---|
| **A** — count a bare `GA` in the quote as a GA-row source phrasing, so a quote saying both `GA` and "public preview" becomes ambiguous | 1 (`onelake:198`) | 0 |
| **B** — abort when the quote is a multi-entity enumeration (≥ 2 pipes, or a numbered list) | 2 (`onelake:198`, `owasp:79`) | 0 |

B subsumes A on these eight. Neither catches `security-copilot-integration.md:94`,
where the quote is prose and the `(Preview)` marker simply belongs to a different
agent. A stricter **referent-name** condition (the row's subject must appear in the
quote) would catch it — and would also kill two *correct* proposals (`:83`, `:93`),
where the source names the capability rather than the agent. That trade is real and
is why this is put to the operator rather than shipped.

## 9. §10 measurement #2 — how much of R8 resolves as O2, measured 2026-08-03

The last machine-answerable pilot measurement. §4.2 recorded it as *"not
answered, and not answerable by machine"* — true of a regex, not of prose
classification, which is what this ran. Reproduce the verification and the
tally: `node scripts/kb-eval/check-o2-returns.mjs`.

**Population.** The 46 pilot flags that are R8 ∧ `MULTI_PART_CLAIM` — the O2
candidate shape, re-derived from the ledger, not read from a plan
(`classify-fix-ops.mjs --threshold 7`). 17 distinct files.

**Method.** Eight subagents, six items each, classifying against §5's three
conditions. Constraints, all deliberate: read-only (no writes, no commits); **no
web or MCP lookups** — O2 is *defined* by requiring no new fact-finding, so the
`evidence_quote` is the only source evidence a classifier may use; and every
proposal must be obtainable from the file text by **deleting characters only**.
Because `claim` matches its file line verbatim in 0 of 202 cases (F4), each
classifier had to open the actual file and locate the real text rather than edit
the restatement. All 46 located it; `locator_failed` is 0.

| Verdict | N | Share |
|---|---|---|
| **O2 candidate** | **17** | 37 % |
| O3 | 29 | 63 % |

**The result that matters is not the split — it is what blocks the other 29.**

| Blocking condition(s) | N |
|---|---|
| condition 2 + condition 3 | 21 |
| condition 3 alone | 3 |
| all three | 5 |

**All 29 are foreclosed by condition 3: the source supplies a *corrected value*,
so the fix is a swap or a rewrite and subtraction would destroy true
information.** Condition 1 — "asserts strictly less", the one that sounds like
the hard one — blocks only 5, and never alone. This is F2 (§5) reproduced at
scale: `prebuilt-check.us`, `prebuilt-mortgage.us.closingDisclosure`,
`Set-DlpCompliancePolicy`, `jensen_shannon_distance`, F300 = 384 GB, `DurationMs`
/ `ResultSignature`, Claude 4.5 → 4.6. **R8's failing multi-part claims are
predominantly a wrong-value class, not a surplus-specificity class.** The design's
reading of R8 in §3 — *"where the grounded part stands on its own, the fix is O2"*
— holds for a minority of the class.

**Machine verification of the returns (V1/V2/V2b, §9.1).** 46 of 46 pass V1: the
quoted file text occurs verbatim in the named file, æ/ø/å and markup intact.
16 of the 17 O2 proposals are deletion-only; one is flagged
(`feedback-loops-continuous-improvement.md:555`, where `Automatically add` →
`Add` recapitalises rather than merely deletes). That is a text change, not a
subtraction, and it goes to a human as such.

**The 17 are candidates, not admitted edits — and the split inside them is the
honest number:**

| | N |
|---|---|
| conditions 2 **and** 3 both affirmatively `yes` | **2** (idx 8, 17) |
| at least one condition marked `human_must_confirm` | 15 |
| classifier confidence `high` | 1 |

So the machine's own reading is that **2 of 46 (4 %) are clean subtractions on
the evidence available, and 15 more are worth a human's time.** This is a triage,
not a machine assist. It is **not** measurement #4 — review throughput still
requires human review sessions that have not happened (§4.4) — but it is the
input #4 needs: it says how many items enter review and in what state, which is
the half of throughput that does not require a stopwatch. The recurring reason for `human_must_confirm` on condition 3 is
structural and worth recording: the removed material is often **true of something
else** (Purview really does classify data; the Communication Compliance template
really exists; Redis really is in Norway West) — it is merely false *of the
subject the row names*. Deleting it is defensible; relocating it may be better.
That is a judgement about the corpus, not about the source, which is exactly why
§5 put conditions 2 and 3 in human hands.

**Extrapolation, flagged as such.** `MULTI_PART_CLAIM` is 161 corpus-wide under
R8. At the pilot's 37 % that is ~60 O2 candidates and ~6 clean ones. **This is an
extrapolation from one measured sample, not a measurement**, and the pilot was
deliberately drawn from the densest files.

### 9.1 What the machine checks, and what it deliberately does not

`scripts/kb-eval/lib/o2-return-check.mjs` (25 tests). The checks do **not** decide
O2 — conditions 2 and 3 stay human by ratified contract. They bound the two
failure modes a human reviewing 46 proposals cannot catch cheaply:

- **V1** — the quoted file text must occur verbatim in the file. Catches invented
  text and silent æ/ø/å transliteration. Applied to **every** row, not just the
  O2 ones: an O3 verdict resting on invented text is equally wrong, merely wrong
  in the safe direction.
- **V2** — the remainder must be obtainable by deleting characters only.
- **V2b** — word-level and case-sensitive, because V2 alone is too weak: deleting
  a leading word and recapitalising the next passes the character test, since the
  capital already existed inside the deleted word. This check was added *after*
  wave 1 produced exactly that case.
- **V3** — schema completeness and verdict/condition coherence.

The raw returns are committed under `scripts/kb-eval/data/r11-o2-returns/` —
they are evidence, not regenerable output, same discipline as appendices A and B.

### 9.2 The two affirmative candidates, hand-verified — one of them fails

Before putting anything in front of a human ratifier, the two candidates the
classifier marked affirmative on **both** human conditions (idx 8, 17) were
checked by hand, 2026-08-03. The check was not a re-reading of the source: it
was a check of the classifier's *own* cond-2 reasoning, which cites other lines
in the same file as its justification. V1/V2/V2b never touch those citations —
they bound the quoted block and the remainder string, nothing else.

**Anchoring, verified for all 17.** Each `file_text_verbatim` occurs exactly
**once** in its file (17/17), so text-anchored editing is unambiguous. The line
numbers are not: `line` differs from `real_line` in **9 of 17** records. Any
edit must be anchored on the verbatim block, never on the line number.

**idx 17 — holds.** Both cross-references check out: line 164 does carry
`Start med 0.15`, and the policy sample does use `score-threshold="0.15"` —
at line **238**, not 239 as the classifier wrote (an off-by-one in the citation;
the substance stands). The three bands appear nowhere else in the file
(`grep` for the band values and their labels returns only lines 254-256, the
block itself), so deleting them leaves nothing dangling internally. Two cosmetic
residues for the ratifier, both already visible in the record: the heading keeps
a now-trailing colon, and the `**Verified** (Microsoft Learn - Enable semantic
caching for LLM APIs)` stamp on line 258 afterwards stamps only the direction
statement. Neither makes the remainder false.

**idx 8 — not ratifiable as written, under either reading of cond 2 — but the
reason and the remedy differ, and choosing between them is an operator call.**

First, the fork, because everything below depends on it. Cond 2 says the
remainder must not be misleading. **Its scope was never fixed:**

- **Broad reading** — misleading *to a reader of the file*. Then the rest of the
  file is in scope, and a remainder that contradicts a passage seventy lines
  down fails.
- **Narrow reading** — misleading *as a statement of what the source grounds*.
  Then only the edited passage is in scope, and a contradiction elsewhere in the
  file is a **separate ungrounded claim**, to be flagged on its own, not a
  defeater of this subtraction.

**The operator ratified the broad reading, 2026-08-03.** Cond 2 is measured
against the whole file: a remainder that contradicts the file it sits in is
misleading, whatever the source says. The stated ground is that these files are
publicly distributed and read as wholes — a self-contradicting file is a trust
defect regardless of which half is wrong. The narrow reading (cond 2 scoped to
the edited passage, with contradictions elsewhere handled as separate ungrounded
claims) was considered and rejected. Recorded here because the fork was real and
a later run must not silently re-open it.

The classifier justified cond 2 on one of the two sub-deletions and never
checked the other:

- Sub-deletion (b), the `Optional: Application Insights` bullet: **verified.**
  Lines 202-203 do cover Application Insights on their own terms
  (`**Azure Monitor + Application Insights** (Verified)` / "Drift metrics
  emitteres til Application Insights"), so removing the prerequisite bullet
  leaves no false implication.
- Sub-deletion (a), `eller managed compute cluster`: **fails.** The same file
  documents
  `**Managed Compute Cluster** (for store volumer)` as a real compute option at
  lines 263-265, with pricing and a usage recommendation. Deleting the
  alternative from the prerequisites leaves the file asserting serverless Spark
  as the only compute requirement seventy lines above a cost section that prices
  the alternative. That is a misleading remainder under the ratified reading.

The candidate therefore does not go in as written: **(a) must be dropped from
the subtraction**, leaving (b) alone.

There is a further limit the O2 envelope cannot resolve: cond 3 passes for (a)
only because the source does not *mention* managed compute — not-mentioned
passes cond 3 by construction. Whether Azure ML actually permits managed compute
for monitoring is a fact question, and O2 is defined by zero fact-gathering. It
is an operator call, not a gap to be read harder.

A reduced subtraction — sub-deletion (b) alone — would still be deletion-only
and is not defeated by anything found here. But it is a **different remainder
string** than the one V2b attested, so it is not machine-clean until
`check-o2-returns.mjs` is re-run against it. The same rule governs any operator
amendment, including dropping idx 17's trailing colon: **amended remainder →
re-run the check before it counts as verified.**

**The generalisable finding.** The classifier judged cond 2 against the *source*
and against citations it chose itself. It did not systematically judge it against
**the rest of the same file**. Under the ratified reading that is a defect:
internal consistency is a cond-2 dimension the wave prompts never assigned, and
idx 8 — the single `high` confidence record in the set — is the proof that it
bites. The remaining 15 have not had the check. It must be run per candidate
before any of them reaches a ratifier, and its output corrects the existing
cond-2 verdicts rather than merely adding to them; expect it to move some.

**Score after hand-verification: 1 of 46 clears both conditions, not 2.** Only
idx 17 survives. That is the number §10 measurement #2 should be read with — the
classifier's own "2 of 46" counted idx 8 on a cond-2 justification that was
half-unchecked.

### 9.3 The whole-file check run on the remaining 15

Run 2026-08-03, one candidate at a time, over the 15 O2 candidates other than
idx 8 and 17 — ten distinct files. No KB file was edited.

**Method, and its one calibration.** For each record, the deletion segments were
recovered by diffing `file_text_verbatim` against `proposed_remainder` (word-level
LCS), content phrases were extracted from each segment (markdown stripped,
Norwegian and English stopwords dropped, contiguous content runs of 2-3 words kept
as noun-phrase units), and each phrase was matched case-insensitively against every
line of the file *outside* the verbatim block. The extractor was calibrated on
idx 8 before the sweep: it must surface `Managed Compute Cluster` at line 263 from
the deleted `eller managed compute cluster`. It does, as the top-ranked multi-word
hit. Without that calibration a narrower extractor would have returned a clean
bill on idx 8 — and silently on others.

**The grep is necessary and not sufficient.** Two of the findings below have no
lexical overlap with the deleted tokens at all. Idx 14's surviving claim is the
Norwegian `automatisk` restating a deleted English `Automatically`; no token
search finds it. Idx 36's deleted `indiscriminate` appears nowhere else in its
file — the grep returns nothing at all — and the finding at line 310 came from
reading the section, not from a hit. Every candidate was therefore also read in place — the enclosing
section around `real_line`, plus every hit line with context — and asked the
second question the grep cannot: *does the surviving text now claim something
broader or narrower than before, and does anything else in the file depend on the
version that was there?*

**Result: 9 clean · 4 contradicted · 2 operator calls.**

| idx | file:line | outcome | the line that decides it |
|---|---|---|---|
| 7 | `document-intelligence-prebuilt-models.md:79` | clean | `prebuilt-document` occurs only in the deleted row; the one `General` hit (192) is `generalisering`; no count binds the table |
| 9 | `data-drift-monitoring-detection.md:218` | clean | the only other Foundry/RAG reference (320) asserts exactly `groundedness, relevance` — the remainder — and never claims drift detection over grounding data |
| 14 | `feedback-loops-continuous-improvement.md:555` | **contradicted (partial)** | 566: `Reviewed documents automatisk tilgjengelige i "Feedback loop" data source når modellen retraines` |
| 18 | `rag-caching-optimization.md:29` | **contradicted** | 303-318: a whole section `### Azure AI Search - Built-in Caching`, plus 510: `Azure AI Search caching \| **Verified**` |
| 19 | `rag-caching-optimization.md:297` | clean | the deleted bullet is the file's only indexing statement; the code sample's silence about explicit vector indexes predates the deletion |
| 26 | `transparency-documentation-standards.md:117` | **contradicted (partial)** | 300: `\| **Risk assessment** \| Responsible AI Scorecard: Error analysis, fairness assessment \|` |
| 27 | `transparency-documentation-standards.md:426` | **contradicted (partial)** | 216: `- **Copilot Studio**: "Powered by AI" disclosure i chat interface` |
| 28 | `transparency-documentation-standards.md:83` | clean | `Hugging` occurs only in the deleted bullet; no model-card template is claimed anywhere else |
| 31 | `ai-incident-response-procedures.md:139` | clean | `legalHold` occurs only in this block; `enabled` at 148 is Blob versioning; 82's "Legal hold på alle artifacts" is consistent with a tags-only hold |
| 33 | `ai-threat-modeling-stride.md:211` | operator call | 357 restates both deleted capabilities — but about the CAF *document*, not about Defender AISPM |
| 36 | `ai-threat-modeling-stride.md:38` | operator call | 310: `Backdoored models og data poisoning er Critical-severity trusler` — unqualified, while the remainder narrows the register to `(targeted)` |
| 38 | `data-leakage-prevention-ai.md:396` | clean | both deleted policy templates occur only here; the following one-click-policy block names neither |
| 40 | `supply-chain-security-ai-models.md:133` | clean | the CVSS band definition occurs only here; 158's `critical vulnerabilities` is container-image scanning, a different tool |
| 42 | `supply-chain-security-ai-models.md:200` | clean (strengthened) | every other HuggingFace reference (32, 240, 492, 513) treats it as an *unverified* source; the deleted bullet was the outlier |
| 45 | `semantic-caching-patterns.md:436` | clean (strengthened) | `Norway West` appears nowhere else; 303, 451, 488 and 628 are all Norway East |

**The four contradictions are one shape.** In each, an enumerating passage would
lose a member that the file continues to assert elsewhere — a section (18), a
mapping row (26), a bullet in a sibling section (27), a prose restatement (14).
That is idx 8's shape exactly: a list narrowed against a file that documents what
was removed. Idx 18 is the hardest of them, because the surviving claim is not a
stray sentence but a titled section *and* a row in the verification table stamping
it `**Verified**`. No deletion confined to line 29 can fix that file; the correct
edit is larger than the O2 envelope permits.

**Three of the four admit a reduced subtraction**, on the same terms as idx 8:

- **idx 14** — drop the `Automatically`→`Add` half, keep `/ SharePoint` (which is
  the file's only occurrence). This also removes the V2b machine flag, since the
  recapitalisation was the flagged part.
- **idx 26** — drop item 4 (Error analysis), keep item 5 (Counterfactual analysis);
  324, 475 and 693 attach counterfactuals to the dashboard and to GDPR, never to
  the scorecard. The renumbering artifact (`1,2,3,4,6,7`) survives either way.
- **idx 27** — drop the Chat-interface row, keep the Plugin-actions row; `grep -niE
  "confirmation|plugin"` returns line 428 alone.

Every one of these is a **different remainder string** than the one V2b attested,
so none is machine-clean until `check-o2-returns.mjs` is re-run against it. Idx 18
has no reduction: it is a single deletion.

**The two operator calls are a distinct class, and are not being called
contradictions.** In both, the deleted content survives elsewhere in the file
without the remainder becoming false:

- **idx 33** — line 357 does say `AI asset inventory via Azure Resource Graph` and
  `Microsoft Purview Insider Risk Management for prompt-basert data
  exfiltration-deteksjon`, stamped `*(Verified MCP 2026-04)*`. But it says it about
  what the *Cloud Adoption Framework document* now covers, while the deleted
  bullets attributed those capabilities to *Defender for Cloud AISPM*. Different
  subjects, so no contradiction — but the edit's benefit is smaller than it looks,
  because the content it removes stays in the file under another attribution.
- **idx 36** — the remainder narrows the severity register to `Data Poisoning
  (targeted)` while line 310 still justifies a recommendation with the unqualified
  `data poisoning er Critical-severity`. A narrower statement does not contradict a
  broader one; it is subsumed by it. What the edit produces is a file whose
  severity table is more precise than the prose that cites it. Whether that is
  acceptable, or whether 310 needs the same qualifier, is a judgement about the
  file — and a companion edit at 310 exceeds the single-locator O2 envelope.

**What this does to the score — nothing, and that is deliberate.** The sweep
settles the whole-file dimension of cond 2 for all 15: affirmative for 9, negative
for 4, operator for 2. It settles nothing about cond 3, which stands at
`human_must_confirm` for ten of them.

**The verified score remains 1 of 46 (idx 17).** The temptation here is idx 19,
and it should be named rather than acted on. Its cond-2 doubt was itself
whole-file-shaped — the classifier worried that the preceding code sample's
silence about explicit vector indexes might read as "no setup needed" — and its
cond 3 the classifier already marked `yes`. The whole-file check finds no
contradiction: the silence predates the deletion, and the deletion removes an
affirmative false claim. That **narrows** the objection; it does not resolve it.
What is left is an omission judgement — does pre-existing silence mislead a reader
of this file? — and that is the ratifier's call, not the sweep's. Counting idx 19
would mean promoting a `human_must_confirm` to affirmative on the checker's own
authority, which is the one direction this gate exists to prevent. Note the
asymmetry: idx 14, 18 and 26 were also `cond2 = confirm`, and for those the sweep
*confirmed* the doubt. Moving a candidate the other way is not the sweep's to do.

**Idx 19 is therefore the strongest new candidate for ratification** — not a
member of the verified class.

**What the sweep says about the method.** The classifier's cond-2 column was
wrong in one direction only. Of the eleven candidates it marked `cond 2 = yes`,
the whole-file check overturns two (27, 36) and confirms nine. Of the four it
marked `human_must_confirm`, the check clears one (19) and confirms the doubt on
three (14, 18, 26) — in every one of those three the classifier had already
written the contradicting line number into its own evidence field without
treating it as a defeater. The information was in the returns; the contract just
never asked the classifier to act on it. That is a prompt gap, not a model
failure, and it is cheap to close in a later wave: name internal consistency as a
cond-2 dimension and require the citation.

## Appendix A — the 15 admitted proposals, hand-verified

Every proposal the classifier (§4 + context condition) admitted over the whole
`not_grounded` population, with the verdict that produced §3's table. A later run
that admits a 16th can diff against this list; without it, "9 of 15" is an
unreproducible claim.

| # | File:line | Swap | Type | Verdict |
|---|---|---|---|---|
| 1 | `agent-orchestration/agent-evaluation-testing-frameworks.md:56` | `4.1` → `5` (`gpt-4.1-mini` → `gpt-5-mini`) | number | **unverified** — model identifier; the result is a real model name, but not checked against the source. Not applied. |
| 2 | `api-management/logging-analytics-ai-traffic.md:49` | `2023-09-01` → `2025-09-01` | iso_date | **correct** — ARM `loggers@` api-version bump |
| 3 | `azure-ai-services/translator-document-translation.md:162` | `40` → `10` (MB) | number | **unverified** — matched on the unit `MB`, but sync/async limits differ; metric-crossing risk. Not applied. |
| 4 | `monitoring-observability/log-analytics-kql-ai-queries.md:617` | `2025-09-01` → `2026-04-01` | iso_date | **correct** — `api-version=` inside a KQL string literal |
| 5 | `responsible-ai/responsible-ai-training-awareness.md:77` | `900` → `901` (`AI-900` → `AI-901`) | number | **wrong** — certification identifier mutilated |
| 6 | `bcdr/cost-analysis-dr-configurations.md:120` | `4` → `5.1` (`GPT-4o` → `GPT-5.1o`) | number | **wrong** — model identifier mutilated |
| 7 | `bcdr/multi-region-azure-openai-deployment.md:316` | `2024-06-01` → `2024-10-01` | iso_date | **correct** — `api-version=` in a management URL |
| 8 | `ai-security-engineering/ai-prompt-shield-network.md:309` | `2024-09-01` → `2024-09-15` | iso_date | **correct** — Content Safety api-version |
| 9 | `ai-security-engineering/content-safety-filter-calibration.md:277` | `2024-10-01` → `2024-10-21` | iso_date | **correct** — Azure OpenAI api-version in a curl sample |
| 10 | `ai-security-engineering/jailbreak-prevention-production.md:305` | `2024-09-01` → `2024-09-15` | iso_date | **correct** — Content Safety api-version in a curl sample |
| 11 | `cost-optimization/observability-cost-reduction.md:114` | `3.7.5` → `3.4.0` (Java Agent) | version | **wrong** — a downgrade; the quote's version is not the claim's referent |
| 12 | `cost-optimization/vector-storage-cost-optimization.md:266` | `2025-09-01` → `2026-04-01` | iso_date | **correct** — AI Search api-version |
| 13 | `cost-optimization/vector-storage-cost-optimization.md:318` | `2024-02-01` → `2024-10-21` | iso_date | **correct** — embeddings api-version |
| 14 | `performance-scalability/response-chunking-strategies.md:56` | `4` → `5.1` (`gpt-4o` → `gpt-5.1o`) | number | **wrong** — model identifier mutilated |
| 15 | `performance-scalability/token-per-second-optimization.md:295` | `2024-12-01` → `2025-01-01` | iso_date | **correct** — Azure OpenAI api-version |

**9 correct · 4 wrong · 2 unverified.** All nine correct are `iso_date`; every
wrong one is a digit inside a product, model or certification identifier, where a
matching prefix (`AI-`, `gpt-`, `Agent `) satisfies the context condition while
the digit is part of a name rather than a quantity. The two unverified are also
`number` and are excluded by the same class rule — verifying them costs a source
fetch each and would move the total to at most 11.

## Appendix B — the 8 §4b status proposals, hand-verified

Every status proposal the classifier admits over the whole `not_grounded`
population, judged against the cited source. Same discipline as appendix A: a
later run that admits a ninth can diff against this list, and "5 of 8" is
otherwise an unreproducible claim.

Reproduce: `node scripts/kb-eval/classify-fix-ops.mjs --threshold 1 --write`, then
read the `proposal.type === 'status'` items in
`scripts/kb-eval/data/r11-pilot-classification.json`.

| # | File:line | Swap | Verdict |
|---|---|---|---|
| 1 | `agent-orchestration/foundry-agent-service-ga.md:68` | `**Preview**` → `**GA**` | **correct** — quote: "hosted agents are generally available"; the row's subject is *Hosted agents* |
| 2 | `agent-orchestration/foundry-agent-service-ga.md:72` | `**GA**` → `**Preview**` | **correct** — quote: "Trigger an agent by using Logic Apps (preview)"; the row's subject is the Logic Apps trigger |
| 3 | `ai-security-engineering/security-copilot-integration.md:83` | `Public Preview` → `GA` | **correct** — quote: "Email and collaboration alert triage capabilities are already generally available (GA)"; the row is the phishing/email triage agent |
| 4 | `ai-security-engineering/security-copilot-integration.md:93` | `GA` → `Preview` | **correct** — quote is from the agent's own doc page: "This feature is in public preview" |
| 5 | `ai-security-engineering/entra-agent-id-zero-trust.md:439` | `Public Preview` → `GA` | **correct** — quote: "The Microsoft Entra Agent ID platform is now generally available"; the row's subject is Entra Agent ID (kjerne) |
| 6 | `ai-security-engineering/security-copilot-integration.md:94` | `GA` → `Preview` | **unproven** — the quote's `(Preview)` marker belongs to *Identity Risk Management Agent*, not to Access Review Agent. The judge's prose `reason` does support preview from a what's-new post, so the outcome is plausibly right; the cited evidence does not establish it. Not applied. |
| 7 | `data-engineering/onelake-data-strategy.md:198` | `GA` → `Preview` | **wrong** — the quote says `Lakehouse \| Yes \| GA`. "Public preview" in the same quote belongs to *Eventhouse*. Killed by candidate A and B. |
| 8 | `ai-security-engineering/owasp-llm-top10-azure-mitigations.md:79` | `GA` → `Preview` | **wrong** — the source marks only *Response Completeness* as preview; the row covers the groundedness/completeness pair, so the edit makes the groundedness half false. The correct fix is to split the row (O2/O3). Killed by candidate B. |

**5 correct · 1 unproven · 2 wrong.** All five correct ones carry the source
phrasing on the row's own subject; all three defects are the referent gap
described in §8. No proposal was applied — §4b output is a human review list.

## Appendix C — the 17 O2 candidates

Every proposal the prose classification admitted over the 46, with the two
human-judged conditions as the classifier left them. Same discipline as
appendices A and B: without this list, "17 of 46" is an unreproducible claim.
`confirm` = the classifier marked the condition `human_must_confirm`, i.e. it
could not settle it on the evidence available and is handing it over — not a
defect, it is the contract.

The full records, including each proposal's verbatim file text and the exact
remainder, are in `scripts/kb-eval/data/r11-o2-returns/`.

| # | File:line | Failing sub-assertion | Cond 2 | Cond 3 | Confidence |
|---|---|---|---|---|---|
| 7 | `ms-ai-engineering/azure-ai-services/document-intelligence-prebuilt-models.md:79` | The third table row presenting `prebuilt-document` (General Document) as a current basic model — the source states the general document model is no l… | yes | confirm | medium |
| 8 | `ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md:191` | Two sub-assertions: (a) `eller managed compute cluster` as an alternative compute option — the how-to page and the monitor schema require a Spark poo… | yes | yes | high |
| 9 | `ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md:218` | The second sentence, `Støtter også drift detection for grounding data i RAG scenarios.` — the canonical observability page lists only Evaluation, Mon… | yes | confirm | medium |
| 14 | `ms-ai-engineering/mlops-genaiops/feedback-loops-continuous-improvement.md:555` | Two sub-assertions: 'SharePoint' as a feedback storage service, and the word 'Automatically' in 'Automatically add reviewed samples to training set' … | confirm | yes | medium |
| 17 | `ms-ai-engineering/rag-architecture/rag-caching-optimization.md:254` | The three-band rubric (0.1-0.2 strict / 0.3-0.5 balanced / 0.6-0.8 liberal) — undocumented, and the two upper bands contradict the source's warning t… | yes | yes | medium |
| 18 | `ms-ai-engineering/rag-architecture/rag-caching-optimization.md:29` | The list item 'Azure AI Search (built-in caching av search results)' — the source states each query operates on the current index view with no cachin… | confirm | yes | medium |
| 19 | `ms-ai-engineering/rag-architecture/rag-caching-optimization.md:297` | The bullet "Automatic indexing av vectors" — the judge states vector indexes must be declared explicitly in the indexing policy (only at container cr… | confirm | yes | medium |
| 26 | `ms-ai-governance/responsible-ai/transparency-documentation-standards.md:117` | Items 4 (Error analysis) and 5 (Counterfactual analysis) are listed as Responsible AI Scorecard components, but the canonical scorecard segment enume… | confirm | yes | medium |
| 27 | `ms-ai-governance/responsible-ai/transparency-documentation-standards.md:426` | The 'Chat interface' row (a "Powered by AI" badge in the chat window) and the 'Plugin actions' row (confirmation prompts before sensitive actions) ar… | yes | confirm | medium |
| 28 | `ms-ai-governance/responsible-ai/transparency-documentation-standards.md:83` | The second and third bullets — that Hugging Face model cards are synchronised automatically, and that a template exists for generating model cards fo… | yes | confirm | medium |
| 31 | `ms-ai-security/ai-security-engineering/ai-incident-response-procedures.md:139` | The legalHold object is given a field named "enabled"; the Storage API's LegalHold model exposes tags and hasLegalHold, so the literal field name "en… | yes | confirm | medium |
| 33 | `ms-ai-security/ai-security-engineering/ai-threat-modeling-stride.md:211` | Two parts attributed to Defender for Cloud AI Security Posture Management that the AISPM page does not support: the discovery mechanism "(via Azure R… | yes | confirm | medium |
| 36 | `ms-ai-security/ai-security-engineering/ai-threat-modeling-stride.md:38` | The "/indiscriminate" qualifier, which extends the Tampering placement and the Critical severity to indiscriminate data poisoning; the source gives t… | yes | confirm | medium |
| 38 | `ms-ai-security/ai-security-engineering/data-leakage-prevention-ai.md:396` | The listing of 'DSPM for AI - Unethical behavior in AI apps' and 'DSPM for AI - Protect sensitive data from Copilot processing' as Insider Risk Manag… | yes | confirm | medium |
| 40 | `ms-ai-security/ai-security-engineering/supply-chain-security-ai-models.md:133` | The third bullet '**CVE severity mapping**' presented as a category of alert that dependency scanning generates; severity is a property of an alert, … | yes | confirm | medium |
| 42 | `ms-ai-security/ai-security-engineering/supply-chain-security-ai-models.md:200` | The second bullet presenting the HuggingFace Registry as a Microsoft channel for verified models with provenance tracking; the source calls it a comm… | yes | confirm | medium |
| 45 | `ms-ai-security/cost-optimization/semantic-caching-patterns.md:436` | The '/West' half of the region pair, i.e. the standing implication that Azure OpenAI can be deployed in Norway West. | yes | confirm | medium |

**2 affirmative on both conditions (8, 17) · 15 needing a human call.** Item 14
additionally carries a machine flag: its remainder recapitalises rather than
deletes (§9.1, V2b), so it is a text change and must be reviewed as one.

⚠️ **The `cond 2` column above is the classifier's claim, not a verified fact —
and it has now been corrected.** All 17 rows have had the whole-file check (idx 8
and 17 in §9.2, the other 15 in §9.3). Read the column together with §9.3's table,
which overrides it:

- **Contradicted, not ratifiable as written:** idx 8, 14, 18, 26, 27. All but 18
  admit a reduced subtraction; every reduction is a new remainder string and must
  be re-run through `check-o2-returns.mjs` before it counts as verified.
- **Operator call:** idx 33, 36 — deleted content survives elsewhere in the file
  without the remainder becoming false.
- **Clean on the whole-file dimension:** idx 7, 9, 17, 19, 28, 31, 38, 40, 42, 45.
  Cond 3 is still `human_must_confirm` for most of them; clean here means cond 2
  only. **Idx 19 is a narrowing, not a resolution** — its cond 2 stands at
  `human_must_confirm`, with the residual reduced to an omission question (§9.3).

**The verified score remains 1 of 46 — idx 17 alone.** The sweep removed no
member and added none; it corrected four rows and left idx 19 as the strongest
candidate for the next ratification.

Reproduce the tally and the machine checks:
`node scripts/kb-eval/check-o2-returns.mjs`.