Kjell Tore Guttormsen
b0b5890703
§10-målingen er gjennomført mot live ledger (243 records). Ingen KB-fil er redigert og ingen ledger-record er skrevet — §8s single-writer-state er urørt. Instrumentet ER O1-driveren med writes av (scripts/kb-eval/lib/fix-op.mjs, 30 tester). Måling #1 og #3 kommer dermed ut av mekanismen som senere skal ta på korpuset, ikke ut av en proxy-heuristikk. HOVEDFUNN — §4 som skrevet er utilstrekkelig, målt: Kjørt eksakt som spesifisert slapp den gjennom 6 swaps på piloten, hvorav 4 er GALE editer (presisjon 2/6): - 30-dagers → 24-dagers (enhets-kryssing: kilden sier 24 HOURS) - 3000 req/sek → 50 (metrikk-kryssing: query-throttle vs indexing-rate) - Microsoft Agent 365 → 7 (identifikator lemlestet, «7» høstet fra «E7») - text-embedding-ada-002 → ada-2 (identifikator lemlestet) §4 binder proveniensen til verdien og formen på editen — ingenting om at de to tokenene betegner SAMME STØRRELSE. Påstanden om at invarianten er «deliberately stronger than human review at scale» holder ikke. TILLEGG: contextCorresponds() krever samme label eller samme enhet på begge sider. Bevisst leksikalsk, UTEN oversettelsestabell — «dokumenter» læres ikke å være «documents», fordi en synonymtabell innfører en ny faktakilde og er en operatørbeslutning. Konsekvensen er målt: swap er provbar praktisk talt bare der konteksten er språknøytral (URL, kodeeksempel, parameternøkkel). TALLENE: - Pilot (≥7): 24 filer / 202 flagg → O1 = 2 (1,0 %), O3 = 200 (99,0 %) - Hele korpuset: 218 filer / 776 flagg → 15 sluppet gjennom, 9 korrekte - Kun iso_date (api-version-bump) overlever hånd-verifisering: 9/9. number/version lemlester identifikatorer (AI-900 → AI-901, gpt-4o → gpt-5.1o ×2, Java-agent 3.7.5 → 3.4.0 = nedgradering) og skal IKKE påføres. - Kun 7 av 200 aborter (3,5 %) er en fiksbar engineering-gap. Mer locator- arbeid kan ikke flytte O1-tallet vesentlig. Måling #2 (R8 → O2) er IKKE besvart og kan ikke besvares maskinelt: R8 gir null O1, og hvilke av de 46 enumerasjonene som subtraherer rent avhenger av dommerens PROSA-reason. Måling #4 (review-throughput) er ikke målt — det krever menneskelige review-økter som ikke har skjedd. Begge står som ikke-målt, ikke som antatt. VIDERE FUNN: subtraksjon kan etterlate en misvisende rest (§5 sier den «cannot introduce a new error» — sant om setningen, usant om leserens slutning), og kan ødelegge sann informasjon (prebuilt-check → finnes, heter prebuilt-check.us). `disposition` er `outdated` på 202/202 og bærer null informasjon, i strid med flagg-formatspesifikasjonen. `claim` matcher fillinjen ordrett i 0 av 202. Full oppskrift og åpne operatørbeslutninger: docs/r11-pilot-results.md
224 lines
11 KiB
Markdown
224 lines
11 KiB
Markdown
# R11 execution design — tiered fixes over an already-evidenced flag population
|
||
|
||
**How the R11 fix step consumes the judge-pass flags. Extends
|
||
`docs/r11-flag-format-2026-07.md` (the record contract) with the execution
|
||
contract: what the fix operation is per flag, what is machine-provable, what
|
||
requires human judgement, and what may never be automated.**
|
||
|
||
Status: design spec. Written 2026-08-03 against a ledger snapshot of **222
|
||
records / 902 flags** (`scripts/kb-eval/data/judge-pass-manifest.json`). The
|
||
population is **not yet complete** — batch R7.5 stood at 30 of 51 files — so
|
||
every count below is a snapshot, not a final figure. The design does not depend
|
||
on the exact counts; the classifier re-derives them at run time.
|
||
|
||
---
|
||
|
||
## 1. The measurement this design is built on
|
||
|
||
**All 712 `not_grounded` flags carry a non-empty verbatim `evidence_quote`.**
|
||
712 of 712, measured. Every flag also carries the `evidence_url` the judge
|
||
actually fetched, the rule that fired, and a one-sentence `reason` stating what
|
||
the source said versus what the claim said.
|
||
|
||
The consequence is the whole point of this document: **R11 does not start from
|
||
zero evidence.** The expensive half — locating the authoritative page, reading
|
||
it, and extracting the passage that decides the claim — was already paid for by
|
||
the judge pass. A fix step designed as "fetch the source, confirm, correct"
|
||
re-does work that is already on disk.
|
||
|
||
What actually remains per flag is narrower:
|
||
|
||
1. is the quote still current (freshness, §7), and
|
||
2. what should the sentence say instead (§3).
|
||
|
||
## 2. Measured shape of the flag population
|
||
|
||
| Measurement | Value |
|
||
|---|---|
|
||
| Flags total | 902 |
|
||
| `not_grounded` (fix targets per flag-format spec) | 712 |
|
||
| `source_silent` (recorded, not a fix target) | 190 |
|
||
| Flags carrying a verbatim `evidence_quote` | **712 / 712** |
|
||
| Distinct `evidence_url` across the 712 | **540** |
|
||
| Files carrying at least one `not_grounded` flag | 199 |
|
||
| Files with 1 flag | 45 |
|
||
| Files with 2–6 flags | 133 (489 flags) |
|
||
| Files with ≥7 flags | 21 (**178 flags — 25 % of the volume in 10 % of the files**) |
|
||
|
||
Rule distribution over the 712: R8 339 · R2 176 · (no rule) 72 · R4 41 · R3 40 ·
|
||
R7 28 · R1 16.
|
||
|
||
**540 distinct source URLs for 712 flags** is the number that forecloses the
|
||
obvious optimisation: there is no batching win hiding in shared sources. The
|
||
largest cluster is 12 flags on one page. This is ~700 separate facts, and the
|
||
work is irreducibly per-claim.
|
||
|
||
## 3. Three fix operations — the partition is by operation, not by rule
|
||
|
||
The tiering is defined by **what the fix does to the file**, because that is what
|
||
determines whether a machine can prove it and whether a human must decide it.
|
||
Rule codes are a signal, not the partition.
|
||
|
||
| Op | Fix operation | Provable? | Who decides |
|
||
|---|---|---|---|
|
||
| **O1** | **Value swap** — the claim states X, the cited source states Y; replace the token. | **Yes** (§4) | Machine proposes, human reviews a diff list |
|
||
| **O2** | **Subtraction** — remove or generalise the specificity the source does not support. Covers: source-silent claims, absent entities (retired SKUs/models/services), and multi-part claims where the failing sub-assertion can be dropped while the grounded part survives. | Partly — the *invariant* is that no new fact is introduced | Human ratifies the policy once (§5), then reviews proposals |
|
||
| **O3** | **Rewrite** — the corrected sentence requires a judgement about what to assert. | No | Human, per claim |
|
||
|
||
**Do not read the §2 rule counts as tier sizes.** They overlap: of the 202 flags
|
||
whose claim *and* quote contain a numeric token — the naive O1 signal — **94 are
|
||
R8 multi-part claims**, which are not clean swaps. Heuristics over this
|
||
population give upper bounds in several directions at once, never a partition.
|
||
|
||
**The classifier fails closed to O3.** If it cannot prove an item is O1 or O2, the
|
||
item is O3 and a human sees it. A misrouted O3 costs one review; a misrouted O1
|
||
ships a wrong edit to a public file.
|
||
|
||
R8 deserves a specific note, because it is the largest rule class and it is *not*
|
||
automatically the most expensive one: the judge's `reason` names **which**
|
||
sub-assertion failed. Where the grounded part stands on its own, the fix is O2
|
||
(drop the unsupported part), not O3 (rewrite the sentence). How much of R8 falls
|
||
that way is unknown and is a primary pilot measurement (§10).
|
||
|
||
## 4. The O1 invariant (what makes a value swap provable)
|
||
|
||
An O1 proposal is only valid if, after the edit:
|
||
|
||
1. the new value appears **verbatim inside the cited `evidence_quote`**, and
|
||
2. the rest of the line is **byte-identical** to before, and
|
||
3. exactly one line in the file changed.
|
||
|
||
A driver that cannot establish all three for an item **aborts before writing** and
|
||
routes the item to O3. This is the same discipline already proven in the
|
||
header-backfill drivers (frozen manifest, hard per-file invariant, abort before
|
||
write, idempotent re-run, `atomicWriteSync`) — see `scripts/kb-update/backfill-*.mjs`.
|
||
|
||
This invariant is deliberately stronger than human review at scale. "The new
|
||
value occurs verbatim in the quote the judge cited" is a checkable property; an
|
||
operator eyeballing edit number 600 is not. Human attention is therefore spent
|
||
where it cannot be replaced (O3), not spread thin across all of them.
|
||
|
||
## 5. The O2 policy (requires operator ratification before use)
|
||
|
||
O2 fixes by **subtraction**: the unsupported specificity is removed or
|
||
generalised rather than replaced with a researched value.
|
||
|
||
- It **cannot introduce a new error**, because it asserts strictly less than the
|
||
file did before.
|
||
- It requires **no new fact-finding**, which is what makes it cheap.
|
||
- It **reduces information density**. That is the real cost, and it is an
|
||
operator decision, not an engineering one.
|
||
|
||
The position this design recommends: a knowledge base that says less and says
|
||
nothing false is worth more than one carrying stale precision. The corpus is
|
||
publicly distributed; an incorrect specific number is a worse failure than an
|
||
honest general statement.
|
||
|
||
Ratifying O2 also resolves the standing `source_silent` question as **one class
|
||
decision** instead of 190 individual ones. Until it is ratified, every O2
|
||
candidate falls to O3.
|
||
|
||
## 6. What stays human, permanently
|
||
|
||
- **Never auto-fix.** No fix reaches a file without human confirmation of the
|
||
class (O1/O2) or the item (O3). A judge false positive must become a human
|
||
review, never silent corruption of a public file.
|
||
- **Subagents never write.** Proposal generation is read-only fan-out; all
|
||
writes happen in one place (§8).
|
||
- **O3 is not a backlog to automate later.** It is the class where the corrected
|
||
assertion is a judgement call, and it is the reason the loop reaches ~100 % fix
|
||
precision on top of a fallible judge.
|
||
|
||
## 7. Evidence freshness
|
||
|
||
Each `evidence_quote` is current as of the judge's fetch date, not the fix date.
|
||
Before any file is touched, the evidence base is refreshed by **re-fetching per
|
||
distinct URL — 540, not 712** — as read-only fan-out. A refreshed quote that no
|
||
longer supports the flag re-routes the item (possibly closing it as no longer an
|
||
error). This preserves the verification duty — fresh confirmation before a public
|
||
file changes — without paying for 712 separate fetches.
|
||
|
||
## 8. Concurrency: one writer, many readers
|
||
|
||
R11 phases that are machine-bound (evidence refresh, proposal generation) may run
|
||
across concurrent sessions. The corpus and the ledger may not.
|
||
|
||
**Single-writer state** — never written by more than one session:
|
||
`scripts/kb-eval/data/judge-pass-manifest.json`, the corpus files themselves, and
|
||
the repo's state file.
|
||
|
||
**Protocol:**
|
||
|
||
- Worker sessions write **only** disjointly-named artefacts to a local, untracked
|
||
working directory (one file per unit of work). They do not ingest, stamp, or
|
||
commit.
|
||
- One integrator session ingests serially, stamps, and commits.
|
||
- If two sessions must write corpus files concurrently, shard **by file, never by
|
||
claim**, one git worktree per shard. Disjoint file sets merge without conflict;
|
||
the ledger is still written only by the integrator, after merge.
|
||
|
||
Git worktrees share the main `.git` directory, so the Layer B `pre-commit` scan
|
||
symlink applies inside every worktree — verified 2026-08-03 by creating a
|
||
worktree and resolving `git rev-parse --git-common-dir` plus the hook target.
|
||
The security gate does not weaken under sharding.
|
||
|
||
The reason this protocol is explicit: whole-tree backup/restore in the write
|
||
drivers previously caused silent data loss across concurrent sessions (closed in
|
||
the write-safety hardening pass — scoped restore + atomic writes). Sharded writes
|
||
without a single-writer rule would reintroduce that class through a different door.
|
||
|
||
## 9. No full re-judge after fixing
|
||
|
||
A fixed file does **not** require a fresh judge pass over all of its claims.
|
||
|
||
- Claims already judged `grounded` keep that verdict; the fix did not touch them.
|
||
- A fixed claim's evidence is the O1 invariant (§4) or the human confirmation
|
||
(§5–6), recorded with the fix.
|
||
- The programme's end-proof is a **fresh blind gold sample** measured after the
|
||
fixes, not a re-run of the corpus pass.
|
||
|
||
This is stated explicitly because assuming otherwise would silently add a second
|
||
full corpus pass to the plan.
|
||
|
||
## 10. Pilot and acceptance criterion
|
||
|
||
Before any scaling, run the classifier and the O1 driver against the **24 files
|
||
carrying ≥7 `not_grounded` flags (202 flags)** — a quarter of the volume in a
|
||
tenth of the files, and the densest available sample.
|
||
|
||
> Recount 2026-08-03 after R7.5 completed (ledger 222 → 243 records). The rule is
|
||
> unchanged — densest ≥7 sample — only the count moved: 21 files / 178 flags was
|
||
> measured against the 222-record ledger, before the last 21 R7.5 files were
|
||
> ingested. Three files entered the sample (`semantic-caching-patterns.md`,
|
||
> `small-language-models-economics.md`, `vector-storage-cost-optimization.md`).
|
||
> The threshold counts `not_grounded` only, not `source_silent`; on all-flags it
|
||
> would be 39 files / 352.
|
||
|
||
Measure and record:
|
||
|
||
1. the actual O1 / O2 / O3 split (this design's central unknown);
|
||
2. how much of R8 resolves as O2 rather than O3 (§3);
|
||
3. the O1 driver's abort rate — items it could not prove, which must land in O3;
|
||
4. review throughput per operation class, measured rather than assumed.
|
||
|
||
Scale to the remaining files only on measured numbers. If the split is materially
|
||
worse than assumed, that is known after one session rather than after ten.
|
||
|
||
> **RUN 2026-08-03 — results in `docs/r11-pilot-results.md`.** The split is
|
||
> materially worse than assumed: **9 provable, correct value swaps in the whole
|
||
> 776-flag `not_grounded` population (1.2 %)**, all of them `api-version` bumps.
|
||
> The pilot also falsifies §4 as written — run exactly as specified it admitted 6
|
||
> swaps of which **4 were wrong edits** (unit crossing, metric crossing, two
|
||
> mutilated identifiers), so the invariant is *not* "stronger than human review at
|
||
> scale". A context-correspondence condition was added; read §4 together with the
|
||
> results doc, not on its own. §5 (O2) and the `STATUS_SYNONYM` class are the open
|
||
> operator decisions, and they now carry the whole programme's leverage.
|
||
|
||
## 11. Out of scope
|
||
|
||
- **Rebuild instead of repair.** Regenerating flagged files from source rather
|
||
than editing them is a settled decision: the corpus is repaired, not rebuilt.
|
||
Reopening it is an operator call, not a design choice made here.
|
||
- **Auto-fix in any form** (§6).
|
||
- **Price and other unsourced claims** beyond the O2 class decision — these
|
||
remain operator-gated as a separate matter.
|