docs(ms-ai-architect): R11 §10 måling #2 ført inn — O2 er triage, ikke automatisering [skip-docs]
§9 + appendiks C i pilot-resultatene, og kontrakten oppdatert der den fortsatt sa at dette var umålt (§3, §5, §4.1, §4.2). Målt over de 46 R8 ∧ MULTI_PART_CLAIM: 17 O2-kandidater, 29 O3. - ALLE 29 felles av betingelse 3 — kilden leverer en korrigert verdi, så fiksen er swap/rewrite og subtraksjon ville ødelagt sann informasjon. Betingelse 1 («strengt mindre») feller bare 5, aldri alene. R8s sviktende multipart-påstander er overveiende en FEILVERDI-klasse, ikke en overflødig-spesifisitet-klasse. Det er F2 reprodusert i skala, og det bekrefter at en mekanisk O2-driver ville vært feil å bygge. - Kun 2 av 46 klarerer begge menneske-dømte betingelser bekreftende; 15 er merket human_must_confirm. Gjentakende grunn: det fjernede er ofte SANT om noe ANNET, bare ikke om radens eget subjekt — flytting kan slå sletting. - Ekstrapolering til korpus (~60 kandidater) er merket som ekstrapolering, ikke måling. Presisering under skriving: triage-fordelingen er ikke måling #4 (review- throughput krever menneskelige review-økter som ikke er kjørt) — den er inputen #4 trenger. Suite 1021/1021.
This commit is contained in:
parent
4a36fd1853
commit
d8ce788709
2 changed files with 174 additions and 12 deletions
|
|
@ -47,6 +47,13 @@ This is the answer §10 asked for, and it is materially worse than the design
|
|||
assumed: *"If the split is materially worse than assumed, that is known after one
|
||||
session rather than after ten."*
|
||||
|
||||
> **The O2 half, measured afterwards (§9), does not rescue the number.** Of the 46
|
||||
> R8 multi-part claims that have the O2 shape, 17 are candidates and **2 are clean
|
||||
> subtractions on the available evidence**; all 29 non-candidates fail because the
|
||||
> source supplies a *corrected value*, which makes them swaps or rewrites. O2's
|
||||
> value in R11 is triage — it tells a human which 17 to look at first — not
|
||||
> automation.
|
||||
|
||||
## 2. §4 as written is not sufficient — measured, not argued
|
||||
|
||||
§4 claims its invariant is *"deliberately stronger than human review at scale."*
|
||||
|
|
@ -97,13 +104,19 @@ hand-verification**, and the report marks it as the sole recommended class
|
|||
## 4. The four §10 measurements
|
||||
|
||||
1. **O1 / O2 / O3 split.** O1 = 2/202 on the pilot (9/776 corpus-wide, safe
|
||||
class only). O2 is **undetermined** — it does not exist as a class until §5 is
|
||||
ratified, so every non-O1 item is O3 by design. O3 ≥ 200/202.
|
||||
2. **How much of R8 resolves as O2.** **Not answered, and not answerable by
|
||||
machine.** R8 is 87/202 on the pilot (366/776 corpus-wide) and yields **zero**
|
||||
O1. Of the pilot's 87, **46 are structural enumerations** — the O2 candidate
|
||||
shape. Which of them subtract cleanly turns on the judge's prose `reason`, and
|
||||
no regex reads prose. This needs prose classification (see §6).
|
||||
class only). ~~O2 is **undetermined** — it does not exist as a class until §5 is
|
||||
ratified, so every non-O1 item is O3 by design.~~ §5 was ratified 2026-08-03,
|
||||
and O2 has since been measured over the class where it can exist at all — see
|
||||
measurement 2 and §9. It remains **unmeasured outside R8 ∧
|
||||
`MULTI_PART_CLAIM`**; the classifier still routes every non-O1 item to O3, so
|
||||
O3 ≥ 200/202 stands as the machine's own partition.
|
||||
2. **How much of R8 resolves as O2.** **MEASURED 2026-08-03 — see §9.** R8 is
|
||||
87/202 on the pilot (366/776 corpus-wide) and yields **zero** O1. Of the
|
||||
pilot's 87, **46 are structural enumerations** (R8 ∧ `MULTI_PART_CLAIM`) — the
|
||||
O2 candidate shape. Which of them subtract cleanly turns on the judge's prose
|
||||
`reason`, and no regex reads prose, so this was done by prose classification:
|
||||
**17 of 46 (37 %) are O2 candidates, 29 are O3.** All 29 are foreclosed by
|
||||
condition 3.
|
||||
3. **O1 abort rate: 99 % (200/202).** Typed, because "99 %" alone is not
|
||||
actionable:
|
||||
|
||||
|
|
@ -232,6 +245,104 @@ quote) would catch it — and would also kill two *correct* proposals (`:83`, `:
|
|||
where the source names the capability rather than the agent. That trade is real and
|
||||
is why this is put to the operator rather than shipped.
|
||||
|
||||
## 9. §10 measurement #2 — how much of R8 resolves as O2, measured 2026-08-03
|
||||
|
||||
The last machine-answerable pilot measurement. §4.2 recorded it as *"not
|
||||
answered, and not answerable by machine"* — true of a regex, not of prose
|
||||
classification, which is what this ran. Reproduce the verification and the
|
||||
tally: `node scripts/kb-eval/check-o2-returns.mjs`.
|
||||
|
||||
**Population.** The 46 pilot flags that are R8 ∧ `MULTI_PART_CLAIM` — the O2
|
||||
candidate shape, re-derived from the ledger, not read from a plan
|
||||
(`classify-fix-ops.mjs --threshold 7`). 17 distinct files.
|
||||
|
||||
**Method.** Eight subagents, six items each, classifying against §5's three
|
||||
conditions. Constraints, all deliberate: read-only (no writes, no commits); **no
|
||||
web or MCP lookups** — O2 is *defined* by requiring no new fact-finding, so the
|
||||
`evidence_quote` is the only source evidence a classifier may use; and every
|
||||
proposal must be obtainable from the file text by **deleting characters only**.
|
||||
Because `claim` matches its file line verbatim in 0 of 202 cases (F4), each
|
||||
classifier had to open the actual file and locate the real text rather than edit
|
||||
the restatement. All 46 located it; `locator_failed` is 0.
|
||||
|
||||
| Verdict | N | Share |
|
||||
|---|---|---|
|
||||
| **O2 candidate** | **17** | 37 % |
|
||||
| O3 | 29 | 63 % |
|
||||
|
||||
**The result that matters is not the split — it is what blocks the other 29.**
|
||||
|
||||
| Blocking condition(s) | N |
|
||||
|---|---|
|
||||
| condition 2 + condition 3 | 21 |
|
||||
| condition 3 alone | 3 |
|
||||
| all three | 5 |
|
||||
|
||||
**All 29 are foreclosed by condition 3: the source supplies a *corrected value*,
|
||||
so the fix is a swap or a rewrite and subtraction would destroy true
|
||||
information.** Condition 1 — "asserts strictly less", the one that sounds like
|
||||
the hard one — blocks only 5, and never alone. This is F2 (§5) reproduced at
|
||||
scale: `prebuilt-check.us`, `prebuilt-mortgage.us.closingDisclosure`,
|
||||
`Set-DlpCompliancePolicy`, `jensen_shannon_distance`, F300 = 384 GB, `DurationMs`
|
||||
/ `ResultSignature`, Claude 4.5 → 4.6. **R8's failing multi-part claims are
|
||||
predominantly a wrong-value class, not a surplus-specificity class.** The design's
|
||||
reading of R8 in §3 — *"where the grounded part stands on its own, the fix is O2"*
|
||||
— holds for a minority of the class.
|
||||
|
||||
**Machine verification of the returns (V1/V2/V2b, §9.1).** 46 of 46 pass V1: the
|
||||
quoted file text occurs verbatim in the named file, æ/ø/å and markup intact.
|
||||
16 of the 17 O2 proposals are deletion-only; one is flagged
|
||||
(`feedback-loops-continuous-improvement.md:555`, where `Automatically add` →
|
||||
`Add` recapitalises rather than merely deletes). That is a text change, not a
|
||||
subtraction, and it goes to a human as such.
|
||||
|
||||
**The 17 are candidates, not admitted edits — and the split inside them is the
|
||||
honest number:**
|
||||
|
||||
| | N |
|
||||
|---|---|
|
||||
| conditions 2 **and** 3 both affirmatively `yes` | **2** (idx 8, 17) |
|
||||
| at least one condition marked `human_must_confirm` | 15 |
|
||||
| classifier confidence `high` | 1 |
|
||||
|
||||
So the machine's own reading is that **2 of 46 (4 %) are clean subtractions on
|
||||
the evidence available, and 15 more are worth a human's time.** This is a triage,
|
||||
not a machine assist. It is **not** measurement #4 — review throughput still
|
||||
requires human review sessions that have not happened (§4.4) — but it is the
|
||||
input #4 needs: it says how many items enter review and in what state, which is
|
||||
the half of throughput that does not require a stopwatch. The recurring reason for `human_must_confirm` on condition 3 is
|
||||
structural and worth recording: the removed material is often **true of something
|
||||
else** (Purview really does classify data; the Communication Compliance template
|
||||
really exists; Redis really is in Norway West) — it is merely false *of the
|
||||
subject the row names*. Deleting it is defensible; relocating it may be better.
|
||||
That is a judgement about the corpus, not about the source, which is exactly why
|
||||
§5 put conditions 2 and 3 in human hands.
|
||||
|
||||
**Extrapolation, flagged as such.** `MULTI_PART_CLAIM` is 161 corpus-wide under
|
||||
R8. At the pilot's 37 % that is ~60 O2 candidates and ~6 clean ones. **This is an
|
||||
extrapolation from one measured sample, not a measurement**, and the pilot was
|
||||
deliberately drawn from the densest files.
|
||||
|
||||
### 9.1 What the machine checks, and what it deliberately does not
|
||||
|
||||
`scripts/kb-eval/lib/o2-return-check.mjs` (25 tests). The checks do **not** decide
|
||||
O2 — conditions 2 and 3 stay human by ratified contract. They bound the two
|
||||
failure modes a human reviewing 46 proposals cannot catch cheaply:
|
||||
|
||||
- **V1** — the quoted file text must occur verbatim in the file. Catches invented
|
||||
text and silent æ/ø/å transliteration. Applied to **every** row, not just the
|
||||
O2 ones: an O3 verdict resting on invented text is equally wrong, merely wrong
|
||||
in the safe direction.
|
||||
- **V2** — the remainder must be obtainable by deleting characters only.
|
||||
- **V2b** — word-level and case-sensitive, because V2 alone is too weak: deleting
|
||||
a leading word and recapitalising the next passes the character test, since the
|
||||
capital already existed inside the deleted word. This check was added *after*
|
||||
wave 1 produced exactly that case.
|
||||
- **V3** — schema completeness and verdict/condition coherence.
|
||||
|
||||
The raw returns are committed under `scripts/kb-eval/data/r11-o2-returns/` —
|
||||
they are evidence, not regenerable output, same discipline as appendices A and B.
|
||||
|
||||
## Appendix A — the 15 admitted proposals, hand-verified
|
||||
|
||||
Every proposal the classifier (§4 + context condition) admitted over the whole
|
||||
|
|
@ -289,3 +400,42 @@ read the `proposal.type === 'status'` items in
|
|||
**5 correct · 1 unproven · 2 wrong.** All five correct ones carry the source
|
||||
phrasing on the row's own subject; all three defects are the referent gap
|
||||
described in §8. No proposal was applied — §4b output is a human review list.
|
||||
|
||||
## Appendix C — the 17 O2 candidates
|
||||
|
||||
Every proposal the prose classification admitted over the 46, with the two
|
||||
human-judged conditions as the classifier left them. Same discipline as
|
||||
appendices A and B: without this list, "17 of 46" is an unreproducible claim.
|
||||
`confirm` = the classifier marked the condition `human_must_confirm`, i.e. it
|
||||
could not settle it on the evidence available and is handing it over — not a
|
||||
defect, it is the contract.
|
||||
|
||||
The full records, including each proposal's verbatim file text and the exact
|
||||
remainder, are in `scripts/kb-eval/data/r11-o2-returns/`.
|
||||
|
||||
| # | File:line | Failing sub-assertion | Cond 2 | Cond 3 | Confidence |
|
||||
|---|---|---|---|---|---|
|
||||
| 7 | `ms-ai-engineering/azure-ai-services/document-intelligence-prebuilt-models.md:79` | The third table row presenting `prebuilt-document` (General Document) as a current basic model — the source states the general document model is no l… | yes | confirm | medium |
|
||||
| 8 | `ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md:191` | Two sub-assertions: (a) `eller managed compute cluster` as an alternative compute option — the how-to page and the monitor schema require a Spark poo… | yes | yes | high |
|
||||
| 9 | `ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md:218` | The second sentence, `Støtter også drift detection for grounding data i RAG scenarios.` — the canonical observability page lists only Evaluation, Mon… | yes | confirm | medium |
|
||||
| 14 | `ms-ai-engineering/mlops-genaiops/feedback-loops-continuous-improvement.md:555` | Two sub-assertions: 'SharePoint' as a feedback storage service, and the word 'Automatically' in 'Automatically add reviewed samples to training set' … | confirm | yes | medium |
|
||||
| 17 | `ms-ai-engineering/rag-architecture/rag-caching-optimization.md:254` | The three-band rubric (0.1-0.2 strict / 0.3-0.5 balanced / 0.6-0.8 liberal) — undocumented, and the two upper bands contradict the source's warning t… | yes | yes | medium |
|
||||
| 18 | `ms-ai-engineering/rag-architecture/rag-caching-optimization.md:29` | The list item 'Azure AI Search (built-in caching av search results)' — the source states each query operates on the current index view with no cachin… | confirm | yes | medium |
|
||||
| 19 | `ms-ai-engineering/rag-architecture/rag-caching-optimization.md:297` | The bullet "Automatic indexing av vectors" — the judge states vector indexes must be declared explicitly in the indexing policy (only at container cr… | confirm | yes | medium |
|
||||
| 26 | `ms-ai-governance/responsible-ai/transparency-documentation-standards.md:117` | Items 4 (Error analysis) and 5 (Counterfactual analysis) are listed as Responsible AI Scorecard components, but the canonical scorecard segment enume… | confirm | yes | medium |
|
||||
| 27 | `ms-ai-governance/responsible-ai/transparency-documentation-standards.md:426` | The 'Chat interface' row (a "Powered by AI" badge in the chat window) and the 'Plugin actions' row (confirmation prompts before sensitive actions) ar… | yes | confirm | medium |
|
||||
| 28 | `ms-ai-governance/responsible-ai/transparency-documentation-standards.md:83` | The second and third bullets — that Hugging Face model cards are synchronised automatically, and that a template exists for generating model cards fo… | yes | confirm | medium |
|
||||
| 31 | `ms-ai-security/ai-security-engineering/ai-incident-response-procedures.md:139` | The legalHold object is given a field named "enabled"; the Storage API's LegalHold model exposes tags and hasLegalHold, so the literal field name "en… | yes | confirm | medium |
|
||||
| 33 | `ms-ai-security/ai-security-engineering/ai-threat-modeling-stride.md:211` | Two parts attributed to Defender for Cloud AI Security Posture Management that the AISPM page does not support: the discovery mechanism "(via Azure R… | yes | confirm | medium |
|
||||
| 36 | `ms-ai-security/ai-security-engineering/ai-threat-modeling-stride.md:38` | The "/indiscriminate" qualifier, which extends the Tampering placement and the Critical severity to indiscriminate data poisoning; the source gives t… | yes | confirm | medium |
|
||||
| 38 | `ms-ai-security/ai-security-engineering/data-leakage-prevention-ai.md:396` | The listing of 'DSPM for AI - Unethical behavior in AI apps' and 'DSPM for AI - Protect sensitive data from Copilot processing' as Insider Risk Manag… | yes | confirm | medium |
|
||||
| 40 | `ms-ai-security/ai-security-engineering/supply-chain-security-ai-models.md:133` | The third bullet '**CVE severity mapping**' presented as a category of alert that dependency scanning generates; severity is a property of an alert, … | yes | confirm | medium |
|
||||
| 42 | `ms-ai-security/ai-security-engineering/supply-chain-security-ai-models.md:200` | The second bullet presenting the HuggingFace Registry as a Microsoft channel for verified models with provenance tracking; the source calls it a comm… | yes | confirm | medium |
|
||||
| 45 | `ms-ai-security/cost-optimization/semantic-caching-patterns.md:436` | The '/West' half of the region pair, i.e. the standing implication that Azure OpenAI can be deployed in Norway West. | yes | confirm | medium |
|
||||
|
||||
**2 affirmative on both conditions (8, 17) · 15 needing a human call.** Item 14
|
||||
additionally carries a machine flag: its remainder recapitalises rather than
|
||||
deletes (§9.1, V2b), so it is a text change and must be reviewed as one.
|
||||
|
||||
Reproduce the tally and the machine checks:
|
||||
`node scripts/kb-eval/check-o2-returns.mjs`.
|
||||
|
|
|
|||
|
|
@ -77,8 +77,16 @@ ships a wrong edit to a public file.
|
|||
R8 deserves a specific note, because it is the largest rule class and it is *not*
|
||||
automatically the most expensive one: the judge's `reason` names **which**
|
||||
sub-assertion failed. Where the grounded part stands on its own, the fix is O2
|
||||
(drop the unsupported part), not O3 (rewrite the sentence). How much of R8 falls
|
||||
that way is unknown and is a primary pilot measurement (§10).
|
||||
(drop the unsupported part), not O3 (rewrite the sentence). ~~How much of R8 falls
|
||||
that way is unknown and is a primary pilot measurement (§10).~~
|
||||
|
||||
**MEASURED 2026-08-03 — `docs/r11-pilot-results.md` §9. It falls that way for a
|
||||
minority.** Over the 46 pilot flags with the O2 shape (R8 ∧ `MULTI_PART_CLAIM`):
|
||||
17 O2 candidates, 29 O3, and **all 29 are blocked by §5 condition 3** — the
|
||||
source supplies a corrected value, so the fix is a swap or a rewrite and
|
||||
subtraction would destroy true information. Only 2 of the 46 clear both
|
||||
human-judged conditions affirmatively. The paragraph above is not wrong, but the
|
||||
case it describes is the exception in R8, not the rule.
|
||||
|
||||
## 4. The O1 invariant (what makes a value swap provable)
|
||||
|
||||
|
|
@ -211,9 +219,13 @@ honest general statement.
|
|||
Ratifying O2 also resolves the standing `source_silent` question as **one class
|
||||
decision** instead of 190 individual ones. ~~Until it is ratified, every O2
|
||||
candidate falls to O3.~~ Ratified — O2 is in use, subject to the remainder check
|
||||
above. **Not yet implemented:** the classifier still routes every non-O1 item to
|
||||
O3, because O2 candidacy turns on the judge's prose `reason` and needs prose
|
||||
classification (`docs/r11-pilot-results.md` §4.2).
|
||||
above. **Not implemented in the classifier, and now measured to be the right
|
||||
call:** the classifier still routes every non-O1 item to O3, because O2 candidacy
|
||||
turns on the judge's prose `reason`. The prose classification was run separately
|
||||
(`docs/r11-pilot-results.md` §9) and found that condition 3 forecloses **every**
|
||||
non-candidate in the class — a mechanical O2 driver would therefore have proposed
|
||||
deletions where the source hands over a corrected value, which is precisely the
|
||||
F2 failure. O2 output is a human review list, like §4b's.
|
||||
|
||||
## 6. What stays human, permanently
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue