Kjell Tore Guttormsen
e86948a71a
Operatøren fulgte alle tre anbefalingene (2026-08-03). Beslutningene styrer all nedstrøms fiksing av et offentlig distribuert korpus, så de føres i den SPOREDE kontrakten, ikke i en økt-transkripsjon. §5 — O2 RATIFISERT, med rest-sjekk (ikke blankt). Tre betingelser, alle menneske-bekreftet: (1) setningen påstår strengt mindre, (2) resten bærer ingen falsk eller misvisende stående implikasjon — lest som en LESER leser den, ikke som en logiker, (3) ingenting kilden BEKREFTER fjernes. Betingelse 2 og 3 krever at et menneske leser resten, så O2 er billigere enn O3 (ingen fakta-innhenting) men ikke mekanisk. §10s throughput-antakelse bør re-måles mot det. §4a — kontekst-korrespondanse som betingelse 4, og betingelse 5: den APPLISERBARE klassen er `iso_date` alene. Hånd-verifisert 9/9 mot 0/6 for number/version. Klassifikatoren fortsetter å rapportere alle typer — det er målingen — og merker den appliserbare mengden som `o1_recommended`. §4b — status-synonymtabellen RATIFISERT, snever og LUKKET. Fire rader. Tre skranker, fordi dette er det ENESTE stedet der verdien som skrives inn i fila ikke selv står ordrett i quoten: tabellen utvides aldri ved slutning i run time, fil-tokenet må være en komplett livssyklus-etikett, og verdien som skrives er den KORPUS-side ekvivalenten med filas egen markup bevart (`**Preview**` → `**GA**`), aldri den engelske frasen limt inn. §4s påstand om at invarianten er «deliberately stronger than human review at scale» er strøket og merket falsifisert, med peker til målingen. INGEN AV DE TRE ER IMPLEMENTERT. Klassifikatoren aborterer fortsatt STATUS_SYNONYM og ruter fortsatt alt ikke-O1 til O3. Begge dokumentene sier det eksplisitt, slik at en senere økt ikke antar at koden allerede følger kontrakten.
211 lines
13 KiB
Markdown
211 lines
13 KiB
Markdown
# R11 pilot results — measured, 2026-08-03
|
||
|
||
**The §10 acceptance measurement of `docs/r11-tiered-fix-design.md`, run against
|
||
the live ledger. No KB file was edited and no ledger record was written.**
|
||
|
||
Instrument: `scripts/kb-eval/lib/fix-op.mjs` (+ `tests/kb-eval/test-fix-op-classify.test.mjs`,
|
||
30 tests) driven by `scripts/kb-eval/classify-fix-ops.mjs`. The classifier **is**
|
||
the O1 driver with writes disabled — it constructs the swap and checks the §4
|
||
invariant, so measurements 1 and 3 come out of the mechanism that would later
|
||
touch the corpus, not out of a proxy heuristic.
|
||
|
||
Artefact: `scripts/kb-eval/data/r11-pilot-classification.json` (untracked,
|
||
regenerable; per-flag records so the run can be re-analysed without re-running).
|
||
It holds the **pilot** run — `node scripts/kb-eval/classify-fix-ops.mjs --write`.
|
||
Every corpus-wide figure below is from `--threshold 1`, and the per-table
|
||
reproduce command is stated where it is used.
|
||
|
||
> **Two different 202s.** This population is 202 flags. §3's "202 flags whose
|
||
> claim *and* quote contain a numeric token" is a different 202, measured over
|
||
> the full 712-flag population. They are unrelated.
|
||
|
||
---
|
||
|
||
## 1. The headline
|
||
|
||
**Nine provable, correct value swaps exist in the entire 776-flag `not_grounded`
|
||
population — 1.2 %.** The machine half of the R11 tiering buys nine edits. Every
|
||
other flag needs a human.
|
||
|
||
| Population | Files | Flags | O1 admitted | O1 hand-verified correct |
|
||
|---|---|---|---|---|
|
||
| Pilot (`not_grounded` ≥ 7) | 24 | 202 | 2 | 2 |
|
||
| Whole `not_grounded` corpus | 218 | 776 | 15 | **9** |
|
||
|
||
Reproduce: `node scripts/kb-eval/classify-fix-ops.mjs` (pilot) and
|
||
`node scripts/kb-eval/classify-fix-ops.mjs --threshold 1` (corpus). The nine are
|
||
enumerated with verdicts in appendix A — that hand-verification is the only thing
|
||
separating 9 from 15, so it is recorded rather than left in a session transcript.
|
||
|
||
This is the answer §10 asked for, and it is materially worse than the design
|
||
assumed: *"If the split is materially worse than assumed, that is known after one
|
||
session rather than after ten."*
|
||
|
||
## 2. §4 as written is not sufficient — measured, not argued
|
||
|
||
§4 claims its invariant is *"deliberately stronger than human review at scale."*
|
||
It is not. Run exactly as specified over the pilot, it admitted **6 swaps, of
|
||
which 4 were wrong** — precision **2/6**:
|
||
|
||
| Proposed swap | Why it is wrong |
|
||
|---|---|
|
||
| `30-dagers` → `24-dagers` | **Unit crossing.** The quote says 24 **hours**. |
|
||
| `3000 requests/sekund` → `50` | **Metric crossing.** The quote is a *query* throttle per index; the claim is an *indexing* rate per replica. |
|
||
| `Microsoft Agent 365` → `Agent 7` | **Identifier mutilated.** The `7` was harvested out of `E7`. |
|
||
| `text-embedding-ada-002` → `ada-2` | **Identifier mutilated.** The `2` came from a dimensions column. |
|
||
|
||
The defect is structural, not incidental. §4 constrains **where the new value
|
||
came from** (verbatim in the cited quote) and **what the edit looks like** (one
|
||
line, rest byte-identical). It constrains nothing about whether the two tokens
|
||
**denote the same quantity**. Same-type-and-provenance is not same-referent.
|
||
|
||
### The added condition
|
||
|
||
`contextCorresponds()` requires the token to sit under **the same label or the
|
||
same trailing unit on both sides**. It is deliberately lexical, with **no
|
||
translation table**: `dokumenter` is not taught to equal `documents`, because a
|
||
synonym table introduces a new fact source and is an operator decision, not an
|
||
engineering one. Consequence, measured: a swap is provable essentially only where
|
||
the context is language-neutral — a URL, a code sample, a parameter key.
|
||
|
||
## 3. The condition is necessary but still not sufficient
|
||
|
||
Corpus-wide the context condition cut 38 admissions to 15. Hand-verifying all 15
|
||
splits them cleanly by token type:
|
||
|
||
Reproduce: `node scripts/kb-eval/classify-fix-ops.mjs --threshold 1`
|
||
(the persisted artefact is the **pilot** run — the corpus-wide tables in §1 and §3
|
||
come from this threshold-1 run). Per-proposal verdicts: appendix A.
|
||
|
||
| Token type | Proposals | Correct | Wrong | Unverified | Failure mode |
|
||
|---|---|---|---|---|---|
|
||
| `iso_date` | 9 | **9** | 0 | 0 | — every one is an `api-version=` bump in a URL or code sample |
|
||
| `number` | 5 | 0 | 3 | 2 | `AI-900` → `AI-901`, `gpt-4o` → `gpt-5.1o` (×2) |
|
||
| `version` | 1 | 0 | 1 | 0 | Java agent `3.7.5` → `3.4.0` — a downgrade |
|
||
|
||
A matching identifier *prefix* (`AI-`, `gpt-`) satisfies the context condition
|
||
while the digit is part of a **name**, not a quantity. **Only `iso_date` survives
|
||
hand-verification**, and the report marks it as the sole recommended class
|
||
(`o1_recommended`). `number` and `version` proposals must not be applied.
|
||
|
||
## 4. The four §10 measurements
|
||
|
||
1. **O1 / O2 / O3 split.** O1 = 2/202 on the pilot (9/776 corpus-wide, safe
|
||
class only). O2 is **undetermined** — it does not exist as a class until §5 is
|
||
ratified, so every non-O1 item is O3 by design. O3 ≥ 200/202.
|
||
2. **How much of R8 resolves as O2.** **Not answered, and not answerable by
|
||
machine.** R8 is 87/202 on the pilot (366/776 corpus-wide) and yields **zero**
|
||
O1. Of the pilot's 87, **46 are structural enumerations** — the O2 candidate
|
||
shape. Which of them subtract cleanly turns on the judge's prose `reason`, and
|
||
no regex reads prose. This needs prose classification (see §6).
|
||
3. **O1 abort rate: 99 % (200/202).** Typed, because "99 %" alone is not
|
||
actionable:
|
||
|
||
| Code | Pilot | Class |
|
||
|---|---|---|
|
||
| `MULTI_PART_CLAIM` | 96 (47.5 %) | intrinsic — not a value swap at all |
|
||
| `MULTI_VALUE_TOKEN` | 29 | intrinsic |
|
||
| `NO_VALUE_TOKEN` | 28 | intrinsic — the claim asserts prose |
|
||
| `STATUS_SYNONYM` | 15 | **operator question** (§6.2) |
|
||
| `NOT_VERBATIM` | 15 | intrinsic |
|
||
| `LOCATOR_AMBIGUOUS` | 7 | **fixable engineering gap** |
|
||
| `MULTI_REPLACEMENT` | 6 | intrinsic |
|
||
| `CONTEXT_MISMATCH` | 4 | intrinsic — these are the 4 wrong edits above |
|
||
|
||
**Only 7 of 200 aborts (3.5 %) are a fixable engineering gap.** More locator
|
||
engineering cannot move the O1 number materially.
|
||
4. **Review throughput per class. NOT MEASURED.** It requires human review
|
||
sessions, which have not happened. Recording it as measured would be false.
|
||
|
||
## 5. Two further findings
|
||
|
||
**F1 — subtraction can leave a misleading remainder.** §5 argues O2 *"cannot
|
||
introduce a new error, because it asserts strictly less."* True of the sentence,
|
||
false of the reader's inference. Real case: *"Deep Research tool (o3-deep-research
|
||
+ Bing) er GA (juni 2025)"* where the source says the tool is **deprecated**.
|
||
Subtracting `er GA (juni 2025)` leaves the tool standing in a list of available
|
||
tools. Strictly less asserted; still misleading. O2 therefore still requires a
|
||
human to look at the remainder — cheaper than O3 (no fact-finding) but not
|
||
mechanical.
|
||
|
||
**F2 — subtraction can destroy true information.** Real case: a list of seven
|
||
prebuilt model IDs where the judge found six correct and `prebuilt-check` wrong —
|
||
the real ID is `prebuilt-check.us`. Subtraction drops a model that **exists**; the
|
||
correct fix is a swap. Subtraction is not the safe default everywhere.
|
||
|
||
**F3 — `disposition` carries zero information.** It is `outdated` on **202 of
|
||
202** flags. `docs/r11-flag-format-2026-07.md` specifies `not_grounded →
|
||
{outdated, wrong}` with *"the human assigns which at R11"*, but the pass
|
||
hard-assigned `outdated`. Do not use it as a classifier signal. Spec/data
|
||
divergence, recorded.
|
||
|
||
**F4 — claims are not file text.** `claim` is an LLM-extracted, translated
|
||
restatement: **0 of 202** match their file line verbatim, and 188 share no 40-char
|
||
run with it. For table claims, `line` points at the **header**, not the value.
|
||
This is why the locator exists at all, and why it searches the enclosing block
|
||
rather than the line.
|
||
|
||
## 6. Operator decisions — ALL THREE RATIFIED 2026-08-03
|
||
|
||
All three were put to the operator with the recommendations below and **all three
|
||
were accepted as recommended**. The contract text now lives in
|
||
`docs/r11-tiered-fix-design.md` §4a/§4b/§5; this section records what was asked
|
||
and what the answer was.
|
||
|
||
**None of the three is implemented yet.** The classifier still aborts
|
||
`STATUS_SYNONYM` and still routes every non-O1 item to O3. A later session builds
|
||
against the ratified contract — it must not assume the code already honours it.
|
||
|
||
1. **Ratify O2 (§5)?** → **RATIFIED, with the remainder check** (not as a blanket
|
||
rule), exactly as F1/F2 above argued. Contract: design doc §5, three
|
||
conditions, human-confirmed.
|
||
2. **Amend §4 with a ratified synonym table?** → **RATIFIED, narrow and closed.**
|
||
Contract: design doc §4b — four label rows, closed table, complete-label-only,
|
||
corpus-side value with the file's own markup preserved. Unlocks up to 54
|
||
corpus-wide flags.
|
||
3. **Is O1 worth building at all?** → **KEPT, locked to `iso_date`.** Contract:
|
||
design doc §4a condition 5 — a driver may apply `iso_date` proposals and must
|
||
never apply `number` or `version` ones. Nine edits corpus-wide.
|
||
|
||
## 7. What this does not change
|
||
|
||
The design's core reading survives: the expensive half (locating the source,
|
||
reading it, extracting the deciding passage) was already paid for by the judge
|
||
pass, and §7's freshness re-fetch per distinct URL is untouched. What the pilot
|
||
falsifies is the assumption that a meaningful share of that evidence converts into
|
||
machine-provable edits. It does not. R11 is a human review programme with a
|
||
nine-item machine assist, and its leverage lies entirely in the O2 decision.
|
||
|
||
---
|
||
|
||
## Appendix A — the 15 admitted proposals, hand-verified
|
||
|
||
Every proposal the classifier (§4 + context condition) admitted over the whole
|
||
`not_grounded` population, with the verdict that produced §3's table. A later run
|
||
that admits a 16th can diff against this list; without it, "9 of 15" is an
|
||
unreproducible claim.
|
||
|
||
| # | File:line | Swap | Type | Verdict |
|
||
|---|---|---|---|---|
|
||
| 1 | `agent-orchestration/agent-evaluation-testing-frameworks.md:56` | `4.1` → `5` (`gpt-4.1-mini` → `gpt-5-mini`) | number | **unverified** — model identifier; the result is a real model name, but not checked against the source. Not applied. |
|
||
| 2 | `api-management/logging-analytics-ai-traffic.md:49` | `2023-09-01` → `2025-09-01` | iso_date | **correct** — ARM `loggers@` api-version bump |
|
||
| 3 | `azure-ai-services/translator-document-translation.md:162` | `40` → `10` (MB) | number | **unverified** — matched on the unit `MB`, but sync/async limits differ; metric-crossing risk. Not applied. |
|
||
| 4 | `monitoring-observability/log-analytics-kql-ai-queries.md:617` | `2025-09-01` → `2026-04-01` | iso_date | **correct** — `api-version=` inside a KQL string literal |
|
||
| 5 | `responsible-ai/responsible-ai-training-awareness.md:77` | `900` → `901` (`AI-900` → `AI-901`) | number | **wrong** — certification identifier mutilated |
|
||
| 6 | `bcdr/cost-analysis-dr-configurations.md:120` | `4` → `5.1` (`GPT-4o` → `GPT-5.1o`) | number | **wrong** — model identifier mutilated |
|
||
| 7 | `bcdr/multi-region-azure-openai-deployment.md:316` | `2024-06-01` → `2024-10-01` | iso_date | **correct** — `api-version=` in a management URL |
|
||
| 8 | `ai-security-engineering/ai-prompt-shield-network.md:309` | `2024-09-01` → `2024-09-15` | iso_date | **correct** — Content Safety api-version |
|
||
| 9 | `ai-security-engineering/content-safety-filter-calibration.md:277` | `2024-10-01` → `2024-10-21` | iso_date | **correct** — Azure OpenAI api-version in a curl sample |
|
||
| 10 | `ai-security-engineering/jailbreak-prevention-production.md:305` | `2024-09-01` → `2024-09-15` | iso_date | **correct** — Content Safety api-version in a curl sample |
|
||
| 11 | `cost-optimization/observability-cost-reduction.md:114` | `3.7.5` → `3.4.0` (Java Agent) | version | **wrong** — a downgrade; the quote's version is not the claim's referent |
|
||
| 12 | `cost-optimization/vector-storage-cost-optimization.md:266` | `2025-09-01` → `2026-04-01` | iso_date | **correct** — AI Search api-version |
|
||
| 13 | `cost-optimization/vector-storage-cost-optimization.md:318` | `2024-02-01` → `2024-10-21` | iso_date | **correct** — embeddings api-version |
|
||
| 14 | `performance-scalability/response-chunking-strategies.md:56` | `4` → `5.1` (`gpt-4o` → `gpt-5.1o`) | number | **wrong** — model identifier mutilated |
|
||
| 15 | `performance-scalability/token-per-second-optimization.md:295` | `2024-12-01` → `2025-01-01` | iso_date | **correct** — Azure OpenAI api-version |
|
||
|
||
**9 correct · 4 wrong · 2 unverified.** All nine correct are `iso_date`; every
|
||
wrong one is a digit inside a product, model or certification identifier, where a
|
||
matching prefix (`AI-`, `gpt-`, `Agent `) satisfies the context condition while
|
||
the digit is part of a name rather than a quantity. The two unverified are also
|
||
`number` and are excluded by the same class rule — verifying them costs a source
|
||
fetch each and would move the total to at most 11.
|