Kjell Tore Guttormsen
16ec574687
Operator rule, 2026-09-20, for every repository: our code does not start Claude Code. `tools/okf_adjudicate.py` did, with `subprocess.run` on a hardcoded `/Users/.../claude` -- a path to one binary on one machine, in a public repository. Removed with its test file. References: **17 -> 3**, all three in the CHANGELOG, and that is a deliberate deviation stated in the removal entry itself: two of them describe what the tool did under the versions that shipped it, and a changelog that edits its own past is not a record. The three K3/K4/K5 reports that used it now say so in the past tense, with the removal noted where the tool was named. It is the only such place, measured rather than assumed: `subprocess.*claude| CLAUDE_CLI|claude", *"-` over `src` and `tools` gives **0** here and **3** on `df5a118`. Every other subprocess in `tools/` starts `git`, `pytest` through `sys.executable`, a PDF utility, or the coordination script `okf_watch.py` owns -- none of them a model. `okf build --gate` and the guard are untouched; nothing that judges CONTENT moved. What this removes is a tool that asked a model to judge a proposal, and the only measurement that ever rested on it is a 2026-09-02 baseline, which is reported and dated. 2318 passed, 2 skipped (2334 before, minus this tool's 18 tests, plus 2 new). `ruff check` and `ruff format --check` clean over 159 files, `mypy --strict src/` clean over 25. Wheel built and installed into a clean venv: `okf --help` exits 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Correction to `23ef6bb`: its `9 -> 6` is true of the 74 files the span touched when it was measured, and the span has grown since -- re-measured over the finished span's 79 files, the same pattern is **11 -> 7**, the one extra pair being `tools/okf_adjudicate.py` (gone with this commit) and `tests/test_okf_skill.py`'s own known-positive for the absolute-path check.
137 lines
7.2 KiB
Markdown
137 lines
7.2 KiB
Markdown
# K3, K4 and K5 — the method, committed before any verdict
|
|
|
|
This file declares a method and **nothing else**. It contains no verdict, no
|
|
count and no share. Those live in a separate file, committed after this one.
|
|
|
|
The split is not bookkeeping. A method declared in the same commit as the
|
|
numbers it governs is a method that could have been fitted to them, and no
|
|
amount of prose inside one file can distinguish the two. **The commit order is
|
|
the evidence**: this commit lands first, and that is what makes `n` a
|
|
commitment rather than a description of whatever was convenient to count.
|
|
|
|
## The frame
|
|
|
|
The population is the **39 substantively merged source documents** from the
|
|
2026-09-02 corpus run over `~/corpora/okf-telling-20260829/K2/trinn1`
|
|
(`N = 43`, four coded rejections, zero degenerate merges — see
|
|
`docs/2026-09-02-korpuskjoring-k1-k2.md`). Its format composition is 32 `pdf`,
|
|
5 `docx`, 2 `xlsx`.
|
|
|
|
**Degenerate merges are excluded from the frame** by the rule the harness
|
|
declares: a merge is degenerate when the extracted text is zero characters
|
|
after stripping whitespace. A concept with an empty body cannot carry one unit
|
|
of knowledge, so judging its segmentation would measure extraction failure
|
|
rather than segmentation quality. On this corpus that exclusion removes nothing
|
|
— the degenerate count is 0 — but the rule is stated because it governs the
|
|
frame regardless of what this particular corpus contained.
|
|
|
|
The four coded rejections are outside the frame for the same reason: a file
|
|
that never merged has no segmentation to judge.
|
|
|
|
## n, declared here and not derived later
|
|
|
|
**n = 12.** Stratified across the frame's format classes, proportionally and
|
|
rounded to whole documents: **8 `pdf`, 3 `docx`, 1 `xlsx`**.
|
|
|
|
The draw is deterministic and reproducible without this file naming a single
|
|
document: within each format stratum, order the frame's filenames by the
|
|
hex SHA-256 of the filename and take the first k. Ordering the whole sample by
|
|
the same hash gives the sample's canonical order, used below for the blind
|
|
subset.
|
|
|
|
Document size is spanned by the strata rather than sampled on directly: on this
|
|
corpus format and size are strongly correlated (the `pdf` stratum carries the
|
|
long technical reports, the `docx` stratum the short letters and forms), so a
|
|
stratified draw over format already spans the size range. This is stated as an
|
|
assumption about this corpus, not as a general claim.
|
|
|
|
**Replacement rule.** A drawn file that turns out not to be in the frame — a
|
|
rejection or a degenerate merge that the draw could not see because membership
|
|
is only known after extraction — is replaced by the next file in hash order
|
|
within its stratum. Any replacement that occurs is reported with the numbers.
|
|
|
|
## The four categories, which sum to n
|
|
|
|
Every sampled document's proposed segmentation is placed in exactly one of:
|
|
|
|
- **too coarse** — the proposal leaves distinct units of knowledge fused in one
|
|
segment
|
|
- **too fine** — the proposal splits one unit of knowledge across segments
|
|
- **duplicate** — the proposal emits the same unit of knowledge more than once
|
|
- **correct** — the proposal's boundaries match the document's units of
|
|
knowledge
|
|
|
|
The categories are exclusive and exhaustive by construction: a proposal that is
|
|
wrong in two ways is placed in the category naming its **dominant** error, and
|
|
the tie-break is stated rather than tuned — coarse before fine before
|
|
duplicate. The four counts must sum to `n = 12`.
|
|
|
|
## The adjudicator
|
|
|
|
adjudicator: `claude-opus-5/okf-session-85`
|
|
|
|
An identifier, not a role, because "the operator" or "a reviewer" cannot be
|
|
checked against anything six months from now. This is a **model identity, not a
|
|
human one**, and every figure resting on it inherits that: this is a baseline
|
|
produced by a machine reading proposals, not a human annotation study. Nothing
|
|
here should be read as a human adjudication rate.
|
|
|
|
The judgement was recorded against the proposal with the adjudication tool
|
|
this repository carried at the time (the tool was removed 2026-09-20: nothing in this repository starts another program to judge anything),
|
|
which left the proposal byte-untouched and wrote the verdict as a sibling.
|
|
Its advisory model leg stayed **off**: pre-annotation has been measured lowering
|
|
a good annotator's accuracy, and a leg that cannot be switched off is a leg
|
|
whose value can never be measured.
|
|
|
|
## The blind second-rater protocol
|
|
|
|
**n_blind = 6.** Its own denominator, declared here and not derived from `n`
|
|
afterwards.
|
|
|
|
- **How the subset is drawn:** every second document in the sample's canonical
|
|
hash order — positions 0, 2, 4, 6, 8, 10 of the twelve.
|
|
- **How the second rater judges:** in its own context, with the proposal and
|
|
the source document, the four category definitions and the tie-break rule,
|
|
and **without sight of the first rater's verdicts** or of any count derived
|
|
from them. It returns one category per document and nothing else.
|
|
- **How agreement is computed:** percent agreement — the number of documents
|
|
where both raters chose the same category, over `n_blind`. Reported as a
|
|
fraction with its denominator visible, never as a bare percentage.
|
|
|
|
**Cohen's kappa is deliberately not reported.** With four categories and
|
|
`n_blind = 6`, a kappa estimate is dominated by its own sampling error, and a
|
|
chance-corrected statistic quoted at that denominator would look more rigorous
|
|
than percent agreement while being less informative.
|
|
|
|
**The independence caveat belongs here, not beside the number.** Both raters
|
|
are instances of the same model family. Agreement between them therefore
|
|
overstates independence, probably substantially: shared training produces
|
|
shared reading habits, and two such raters can agree on a mistaken boundary as
|
|
readily as on a correct one. The proposals themselves come from a deterministic
|
|
tool with no model in it, so the pre-annotation effect does not apply to the
|
|
proposal — but it does not rescue the rater independence either.
|
|
|
|
## What this method does not set
|
|
|
|
- **No threshold on K3, and no threshold on K5.** Ratification of a bar is the
|
|
operator's, and setting one inside the same work that produces the first
|
|
measurement would be fitting the bar to the number.
|
|
- **K4 over time is PM-owned and out of scope here.** This work produces one
|
|
baseline. Comparing baselines across runs is a programme-level question that
|
|
the rubric's owner holds, and answering it from inside this repository would
|
|
be answering it in the wrong place.
|
|
|
|
## K4 and K5, as this method will measure them
|
|
|
|
- **K4a** is binary: adjudicate one document, re-run the path, `diff -r` the
|
|
two outputs. Empty or not empty.
|
|
- **K4b** is decisions and minutes per document, with its denominator. If the
|
|
adjudication half is not attempted, K4b is recorded as **`not attempted`** —
|
|
never as `unmet`, which would claim a measurement was made and fell short.
|
|
- **K4c** is one yes/no question — can a consumer see the adjudication state
|
|
and discount on it? — answered by a pointer to the test that pins the
|
|
`adjudication` key. A share of proposed versus adjudicated concepts is a
|
|
distribution fact and is **not** K4c.
|
|
- **K5** is the share of concepts carrying enough of SPEC § 5.1 for a
|
|
falsifiability judgement to rest on, reported together with its known ceiling
|
|
of 2 of 5 keys on a single source.
|