docs(measurement): K3, K4 and K5 method with n declared before any verdict

docs/2026-09-02-k3-k4-k5-metode.md

@@ -0,0 +1,136 @@
+# K3, K4 and K5 — the method, committed before any verdict
+
+This file declares a method and **nothing else**. It contains no verdict, no
+count and no share. Those live in a separate file, committed after this one.
+
+The split is not bookkeeping. A method declared in the same commit as the
+numbers it governs is a method that could have been fitted to them, and no
+amount of prose inside one file can distinguish the two. **The commit order is
+the evidence**: this commit lands first, and that is what makes `n` a
+commitment rather than a description of whatever was convenient to count.
+
+## The frame
+
+The population is the **39 substantively merged source documents** from the
+2026-09-02 corpus run over `~/corpora/okf-telling-20260829/K2/trinn1`
+(`N = 43`, four coded rejections, zero degenerate merges — see
+`docs/2026-09-02-korpuskjoring-k1-k2.md`). Its format composition is 32 `pdf`,
+5 `docx`, 2 `xlsx`.
+
+**Degenerate merges are excluded from the frame** by the rule the harness
+declares: a merge is degenerate when the extracted text is zero characters
+after stripping whitespace. A concept with an empty body cannot carry one unit
+of knowledge, so judging its segmentation would measure extraction failure
+rather than segmentation quality. On this corpus that exclusion removes nothing
+— the degenerate count is 0 — but the rule is stated because it governs the
+frame regardless of what this particular corpus contained.
+
+The four coded rejections are outside the frame for the same reason: a file
+that never merged has no segmentation to judge.
+
+## n, declared here and not derived later
+
+**n = 12.** Stratified across the frame's format classes, proportionally and
+rounded to whole documents: **8 `pdf`, 3 `docx`, 1 `xlsx`**.
+
+The draw is deterministic and reproducible without this file naming a single
+document: within each format stratum, order the frame's filenames by the
+hex SHA-256 of the filename and take the first k. Ordering the whole sample by
+the same hash gives the sample's canonical order, used below for the blind
+subset.
+
+Document size is spanned by the strata rather than sampled on directly: on this
+corpus format and size are strongly correlated (the `pdf` stratum carries the
+long technical reports, the `docx` stratum the short letters and forms), so a
+stratified draw over format already spans the size range. This is stated as an
+assumption about this corpus, not as a general claim.
+
+**Replacement rule.** A drawn file that turns out not to be in the frame — a
+rejection or a degenerate merge that the draw could not see because membership
+is only known after extraction — is replaced by the next file in hash order
+within its stratum. Any replacement that occurs is reported with the numbers.
+
+## The four categories, which sum to n
+
+Every sampled document's proposed segmentation is placed in exactly one of:
+
+- **too coarse** — the proposal leaves distinct units of knowledge fused in one
+  segment
+- **too fine** — the proposal splits one unit of knowledge across segments
+- **duplicate** — the proposal emits the same unit of knowledge more than once
+- **correct** — the proposal's boundaries match the document's units of
+  knowledge
+
+The categories are exclusive and exhaustive by construction: a proposal that is
+wrong in two ways is placed in the category naming its **dominant** error, and
+the tie-break is stated rather than tuned — coarse before fine before
+duplicate. The four counts must sum to `n = 12`.
+
+## The adjudicator
+
+adjudicator: `claude-opus-5/okf-session-85`
+
+An identifier, not a role, because "the operator" or "a reviewer" cannot be
+checked against anything six months from now. This is a **model identity, not a
+human one**, and every figure resting on it inherits that: this is a baseline
+produced by a machine reading proposals, not a human annotation study. Nothing
+here should be read as a human adjudication rate.
+
+The judgement is recorded against the proposal with `tools/okf_adjudicate.py`,
+which leaves the proposal byte-untouched and writes the verdict as a sibling.
+Its advisory model leg stays **off**: pre-annotation has been measured lowering
+a good annotator's accuracy, and a leg that cannot be switched off is a leg
+whose value can never be measured.
+
+## The blind second-rater protocol
+
+**n_blind = 6.** Its own denominator, declared here and not derived from `n`
+afterwards.
+
+- **How the subset is drawn:** every second document in the sample's canonical
+  hash order — positions 0, 2, 4, 6, 8, 10 of the twelve.
+- **How the second rater judges:** in its own context, with the proposal and
+  the source document, the four category definitions and the tie-break rule,
+  and **without sight of the first rater's verdicts** or of any count derived
+  from them. It returns one category per document and nothing else.
+- **How agreement is computed:** percent agreement — the number of documents
+  where both raters chose the same category, over `n_blind`. Reported as a
+  fraction with its denominator visible, never as a bare percentage.
+
+**Cohen's kappa is deliberately not reported.** With four categories and
+`n_blind = 6`, a kappa estimate is dominated by its own sampling error, and a
+chance-corrected statistic quoted at that denominator would look more rigorous
+than percent agreement while being less informative.
+
+**The independence caveat belongs here, not beside the number.** Both raters
+are instances of the same model family. Agreement between them therefore
+overstates independence, probably substantially: shared training produces
+shared reading habits, and two such raters can agree on a mistaken boundary as
+readily as on a correct one. The proposals themselves come from a deterministic
+tool with no model in it, so the pre-annotation effect does not apply to the
+proposal — but it does not rescue the rater independence either.
+
+## What this method does not set
+
+- **No threshold on K3, and no threshold on K5.** Ratification of a bar is the
+  operator's, and setting one inside the same work that produces the first
+  measurement would be fitting the bar to the number.
+- **K4 over time is PM-owned and out of scope here.** This work produces one
+  baseline. Comparing baselines across runs is a programme-level question that
+  the rubric's owner holds, and answering it from inside this repository would
+  be answering it in the wrong place.
+
+## K4 and K5, as this method will measure them
+
+- **K4a** is binary: adjudicate one document, re-run the path, `diff -r` the
+  two outputs. Empty or not empty.
+- **K4b** is decisions and minutes per document, with its denominator. If the
+  adjudication half is not attempted, K4b is recorded as **`not attempted`** —
+  never as `unmet`, which would claim a measurement was made and fell short.
+- **K4c** is one yes/no question — can a consumer see the adjudication state
+  and discount on it? — answered by a pointer to the test that pins the
+  `adjudication` key. A share of proposed versus adjudicated concepts is a
+  distribution fact and is **not** K4c.
+- **K5** is the share of concepts carrying enough of SPEC § 5.1 for a
+  falsifiability judgement to rest on, reported together with its known ceiling
+  of 2 of 5 keys on a single source.