docs(ms-ai-architect): R11 §9.2 — idx 8/17 hand-verified; idx 8 cond2 falsifisert [skip-docs]

docs/r11-pilot-results.md

@@ -343,6 +343,68 @@ failure modes a human reviewing 46 proposals cannot catch cheaply:
 The raw returns are committed under `scripts/kb-eval/data/r11-o2-returns/` —
 they are evidence, not regenerable output, same discipline as appendices A and B.
 
+### 9.2 The two affirmative candidates, hand-verified — one of them fails
+
+Before putting anything in front of a human ratifier, the two candidates the
+classifier marked affirmative on **both** human conditions (idx 8, 17) were
+checked by hand, 2026-08-03. The check was not a re-reading of the source: it
+was a check of the classifier's *own* cond-2 reasoning, which cites other lines
+in the same file as its justification. V1/V2/V2b never touch those citations —
+they bound the quoted block and the remainder string, nothing else.
+
+**Anchoring, verified for all 17.** Each `file_text_verbatim` occurs exactly
+**once** in its file (17/17), so text-anchored editing is unambiguous. The line
+numbers are not: `line` differs from `real_line` in **9 of 17** records. Any
+edit must be anchored on the verbatim block, never on the line number.
+
+**idx 17 — holds.** Both cross-references check out: line 164 does carry
+`Start med 0.15`, and the policy sample does use `score-threshold="0.15"` —
+at line **238**, not 239 as the classifier wrote (an off-by-one in the citation;
+the substance stands). The three bands appear nowhere else in the file
+(`grep` for the band values and their labels returns only lines 254-256, the
+block itself), so deleting them leaves nothing dangling internally. Two cosmetic
+residues for the ratifier, both already visible in the record: the heading keeps
+a now-trailing colon, and the `**Verified** (Microsoft Learn - Enable semantic
+caching for LLM APIs)` stamp on line 258 afterwards stamps only the direction
+statement. Neither makes the remainder false.
+
+**idx 8 — cond 2 does not hold, and the candidate is not ratifiable as written.**
+The classifier justified cond 2 on one of the two sub-deletions and never
+checked the other:
+
+- Sub-deletion (b), the `Optional: Application Insights` bullet: **verified.**
+  Lines 202-203 do cover Application Insights on their own terms
+  (`**Azure Monitor + Application Insights** (Verified)` / "Drift metrics
+  emitteres til Application Insights"), so removing the prerequisite bullet
+  leaves no false implication.
+- Sub-deletion (a), `eller managed compute cluster`: **fails.** The same file
+  documents `**Managed Compute Cluster** (for store volumer)` as a real compute
+  option at lines 263-265, with pricing and a usage recommendation. Deleting the
+  alternative from the prerequisites leaves the file asserting serverless Spark
+  as the only compute requirement seventy lines above a cost section that prices
+  the alternative. That is precisely a misleading remainder.
+
+There is a further limit the O2 envelope cannot resolve: cond 3 passes for (a)
+only because the source does not *mention* managed compute — not-mentioned
+passes cond 3 by construction. Whether Azure ML actually permits managed compute
+for monitoring is a fact question, and O2 is defined by zero fact-gathering. It
+is an operator call, not a gap to be read harder.
+
+A reduced subtraction — sub-deletion (b) alone — would still be deletion-only
+and is not defeated by anything found here. But it is a **different remainder
+string** than the one V2b attested, so it is not machine-clean until
+`check-o2-returns.mjs` is re-run against it. The same rule governs any operator
+amendment, including dropping idx 17's trailing colon: **amended remainder →
+re-run the check before it counts as verified.**
+
+**The generalisable finding.** The classifier judged cond 2 against the *source*
+and against citations it chose itself. It did not systematically judge it
+against **the rest of the same file**. Internal consistency is a cond-2
+dimension the wave prompts never assigned, and idx 8 — the single `high`
+confidence record in the set — is the proof that it bites. The remaining 15
+candidates have not had this check. It must be run per candidate before any of
+them reaches a ratifier; expect it to move some of them.
+
 ## Appendix A — the 15 admitted proposals, hand-verified
 
 Every proposal the classifier (§4 + context condition) admitted over the whole
@@ -437,5 +499,12 @@ remainder, are in `scripts/kb-eval/data/r11-o2-returns/`.
 additionally carries a machine flag: its remainder recapitalises rather than
 deletes (§9.1, V2b), so it is a text change and must be reviewed as one.
 
+⚠️ **The `cond 2 = yes` column is the classifier's claim, not a verified fact.**
+Hand-checking the two affirmative rows (§9.2) falsified idx 8: half its cond-2
+justification was never checked and does not hold, because the same file
+documents the deleted alternative in its cost section. Read every `yes` in this
+column as "the classifier believed this", and verify against the rest of the
+file before ratifying. Only **idx 17** has survived that check so far.
+
 Reproduce the tally and the machine checks:
 `node scripts/kb-eval/check-o2-returns.mjs`.