136 lines
7.1 KiB
Markdown
136 lines
7.1 KiB
Markdown
# K3, K4 and K5 — the method, committed before any verdict
|
|
|
|
This file declares a method and **nothing else**. It contains no verdict, no
|
|
count and no share. Those live in a separate file, committed after this one.
|
|
|
|
The split is not bookkeeping. A method declared in the same commit as the
|
|
numbers it governs is a method that could have been fitted to them, and no
|
|
amount of prose inside one file can distinguish the two. **The commit order is
|
|
the evidence**: this commit lands first, and that is what makes `n` a
|
|
commitment rather than a description of whatever was convenient to count.
|
|
|
|
## The frame
|
|
|
|
The population is the **39 substantively merged source documents** from the
|
|
2026-09-02 corpus run over `~/corpora/okf-telling-20260829/K2/trinn1`
|
|
(`N = 43`, four coded rejections, zero degenerate merges — see
|
|
`docs/2026-09-02-korpuskjoring-k1-k2.md`). Its format composition is 32 `pdf`,
|
|
5 `docx`, 2 `xlsx`.
|
|
|
|
**Degenerate merges are excluded from the frame** by the rule the harness
|
|
declares: a merge is degenerate when the extracted text is zero characters
|
|
after stripping whitespace. A concept with an empty body cannot carry one unit
|
|
of knowledge, so judging its segmentation would measure extraction failure
|
|
rather than segmentation quality. On this corpus that exclusion removes nothing
|
|
— the degenerate count is 0 — but the rule is stated because it governs the
|
|
frame regardless of what this particular corpus contained.
|
|
|
|
The four coded rejections are outside the frame for the same reason: a file
|
|
that never merged has no segmentation to judge.
|
|
|
|
## n, declared here and not derived later
|
|
|
|
**n = 12.** Stratified across the frame's format classes, proportionally and
|
|
rounded to whole documents: **8 `pdf`, 3 `docx`, 1 `xlsx`**.
|
|
|
|
The draw is deterministic and reproducible without this file naming a single
|
|
document: within each format stratum, order the frame's filenames by the
|
|
hex SHA-256 of the filename and take the first k. Ordering the whole sample by
|
|
the same hash gives the sample's canonical order, used below for the blind
|
|
subset.
|
|
|
|
Document size is spanned by the strata rather than sampled on directly: on this
|
|
corpus format and size are strongly correlated (the `pdf` stratum carries the
|
|
long technical reports, the `docx` stratum the short letters and forms), so a
|
|
stratified draw over format already spans the size range. This is stated as an
|
|
assumption about this corpus, not as a general claim.
|
|
|
|
**Replacement rule.** A drawn file that turns out not to be in the frame — a
|
|
rejection or a degenerate merge that the draw could not see because membership
|
|
is only known after extraction — is replaced by the next file in hash order
|
|
within its stratum. Any replacement that occurs is reported with the numbers.
|
|
|
|
## The four categories, which sum to n
|
|
|
|
Every sampled document's proposed segmentation is placed in exactly one of:
|
|
|
|
- **too coarse** — the proposal leaves distinct units of knowledge fused in one
|
|
segment
|
|
- **too fine** — the proposal splits one unit of knowledge across segments
|
|
- **duplicate** — the proposal emits the same unit of knowledge more than once
|
|
- **correct** — the proposal's boundaries match the document's units of
|
|
knowledge
|
|
|
|
The categories are exclusive and exhaustive by construction: a proposal that is
|
|
wrong in two ways is placed in the category naming its **dominant** error, and
|
|
the tie-break is stated rather than tuned — coarse before fine before
|
|
duplicate. The four counts must sum to `n = 12`.
|
|
|
|
## The adjudicator
|
|
|
|
adjudicator: `claude-opus-5/okf-session-85`
|
|
|
|
An identifier, not a role, because "the operator" or "a reviewer" cannot be
|
|
checked against anything six months from now. This is a **model identity, not a
|
|
human one**, and every figure resting on it inherits that: this is a baseline
|
|
produced by a machine reading proposals, not a human annotation study. Nothing
|
|
here should be read as a human adjudication rate.
|
|
|
|
The judgement is recorded against the proposal with `tools/okf_adjudicate.py`,
|
|
which leaves the proposal byte-untouched and writes the verdict as a sibling.
|
|
Its advisory model leg stays **off**: pre-annotation has been measured lowering
|
|
a good annotator's accuracy, and a leg that cannot be switched off is a leg
|
|
whose value can never be measured.
|
|
|
|
## The blind second-rater protocol
|
|
|
|
**n_blind = 6.** Its own denominator, declared here and not derived from `n`
|
|
afterwards.
|
|
|
|
- **How the subset is drawn:** every second document in the sample's canonical
|
|
hash order — positions 0, 2, 4, 6, 8, 10 of the twelve.
|
|
- **How the second rater judges:** in its own context, with the proposal and
|
|
the source document, the four category definitions and the tie-break rule,
|
|
and **without sight of the first rater's verdicts** or of any count derived
|
|
from them. It returns one category per document and nothing else.
|
|
- **How agreement is computed:** percent agreement — the number of documents
|
|
where both raters chose the same category, over `n_blind`. Reported as a
|
|
fraction with its denominator visible, never as a bare percentage.
|
|
|
|
**Cohen's kappa is deliberately not reported.** With four categories and
|
|
`n_blind = 6`, a kappa estimate is dominated by its own sampling error, and a
|
|
chance-corrected statistic quoted at that denominator would look more rigorous
|
|
than percent agreement while being less informative.
|
|
|
|
**The independence caveat belongs here, not beside the number.** Both raters
|
|
are instances of the same model family. Agreement between them therefore
|
|
overstates independence, probably substantially: shared training produces
|
|
shared reading habits, and two such raters can agree on a mistaken boundary as
|
|
readily as on a correct one. The proposals themselves come from a deterministic
|
|
tool with no model in it, so the pre-annotation effect does not apply to the
|
|
proposal — but it does not rescue the rater independence either.
|
|
|
|
## What this method does not set
|
|
|
|
- **No threshold on K3, and no threshold on K5.** Ratification of a bar is the
|
|
operator's, and setting one inside the same work that produces the first
|
|
measurement would be fitting the bar to the number.
|
|
- **K4 over time is PM-owned and out of scope here.** This work produces one
|
|
baseline. Comparing baselines across runs is a programme-level question that
|
|
the rubric's owner holds, and answering it from inside this repository would
|
|
be answering it in the wrong place.
|
|
|
|
## K4 and K5, as this method will measure them
|
|
|
|
- **K4a** is binary: adjudicate one document, re-run the path, `diff -r` the
|
|
two outputs. Empty or not empty.
|
|
- **K4b** is decisions and minutes per document, with its denominator. If the
|
|
adjudication half is not attempted, K4b is recorded as **`not attempted`** —
|
|
never as `unmet`, which would claim a measurement was made and fell short.
|
|
- **K4c** is one yes/no question — can a consumer see the adjudication state
|
|
and discount on it? — answered by a pointer to the test that pins the
|
|
`adjudication` key. A share of proposed versus adjudicated concepts is a
|
|
distribution fact and is **not** K4c.
|
|
- **K5** is the share of concepts carrying enough of SPEC § 5.1 for a
|
|
falsifiability judgement to rest on, reported together with its known ceiling
|
|
of 2 of 5 keys on a single source.
|