open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

docs(measurement): K3, K4 and K5 numbers against the committed method

Browse source
This commit is contained in:
Kjell Tore Guttormsen 2026-09-02 15:55:19 +02:00
commit 94f31ac880
1 changed files with 138 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

138
docs/2026-09-02-k3-k4-k5-baseline.md Normal file
View file

@ -0,0 +1,138 @@
# K3, K4 and K5 — the numbers, on the method committed before them
The method is `docs/2026-09-02-k3-k4-k5-metode.md`, committed in the preceding
commit. **It is not revised here.** Where adjudication showed something the
method did not anticipate, it is stated below as a finding; reopening the
method file in this commit would erase the ordering the split exists to prove.
## K3 — the four category counts, n = 12
| category | count |
|------------|-------|
| too coarse | 7 |
| too fine | 3 |
| duplicate | 1 |
| correct | 1 |
Sum: 12, which is the `n` the method committed. No replacement was needed: all
twelve drawn documents were in the frame.
The frame's proposals came from `tools/okf_propose_segments.py` run over the
sample; four of the twelve proposals were **empty** — the tool proposed no
split at all — and an empty proposal means the whole document would become one
concept. All four fell in the first row.
**No threshold is applied.** The method declared none, and none is invented
here.
## Blind second rater and agreement
n_blind = 6, the even positions of the sample's canonical hash order, judged in
a separate context by `claude-opus-5/blind-rater-1` with the proposals, the
source documents, the four category definitions and the tie-break rule, and
without sight of the first rater's verdicts. The first rater's labels were
written to a file before the second rater's were read.
**Agreement: 5/6.**
The single disagreement is worth more than the ratio. On one document the first
rater judged the six proposed section boundaries to match the document's
sections; the second rater judged the same proposal too coarse, and named its
evidence — one numbered chapter has no segment of its own and is absorbed into
the preceding segment's span, and the document's introduction is covered by no
segment at all. That is a specific, checkable claim, and the first rater did
not check at that resolution. **The verdict is not revised** — a first rater
who edits their label after reading the second rater's is not a second rater at
all — but the asymmetry is the finding: the disagreement was resolved by
evidence in one direction only, which suggests the true count of `too coarse`
is a floor rather than an estimate.
The method's independence caveat stands and should be read with the 5/6: both
raters are instances of the same model family, so agreement overstates
independence.
## K4a — determinism, binary
**PASS.** One document was adjudicated with `tools/okf_adjudicate.py`, then run
through the path twice into two bundles, and the two compared:
```
diff -r /tmp/k4a/a /tmp/k4a/b # exit 0, no output
```
Six concepts and an index were written on each run, byte-identical, under
`SEGMENTED_OKF_V0_2` with `bundle_id` supplied by the caller.
## K4b — decisions and minutes per document, with the denominator
Denominator: **12 documents**, carrying **115 proposed entries** between them.
| figure | value |
|-------------------------------|----------|
| documents adjudicated | 12 |
| entry-level decisions | 115 |
| wall clock, whole adjudication| 314 s |
| per document | 26.2 s |
| per entry-level decision | 2.73 s |
**This is not a human throughput figure and must never be quoted as one.** The
adjudicator is the model identity the method named. A human reading 115
proposed segment boundaries against their source documents would not take 26
seconds per document, and nothing here measures that.
### A finding the method did not anticipate
**4 of 12 verdicts could not be recorded at all.** `tools/okf_adjudicate.py`
parses the proposal before writing, and the parser refuses a plan with zero
entries — correctly, because an empty plan replayed on the run path would
persist nothing for a document that was dropped. The consequence is that the
one judgement most worth recording about those four documents — *the proposal
is empty, and that is wrong* — has no artifact to live in. The K3 counts above
include all twelve because a category is a judgement, not a file; the
adjudication record covers 8/12.
This is stated, not fixed: fixing it is a change to the adjudication tool or to
the plan grammar, and neither belongs in a measurement commit.
## K4c — binary, and it is not a share
**Yes.** A consumer can see the adjudication state and discount on it. The
evidence is the test committed in **Step 14**, which pins the `adjudication`
key, its closed value set and its companion keys:
`tests/test_segmented_index.py:338` (an unratified segment carries `proposed`),
`:346` (a ratified one carries adjudicator, time and dwell), `:359` (the state
is projected as an index facet) and `:410` (the older profile writes no
adjudication state at all, so its absence is distinguishable from `proposed`).
The K4a run confirms the same thing on real output: each of its six concepts
carries `adjudication: adjudicated`, `adjudicated_by`, `adjudicated_at` and
`adjudication_dwell_s` in its frontmatter.
**A distribution fact, reported as such and not as K4c:** in the corpus bundle
from the 2026-09-02 run, 0 of 39 concepts carry any adjudication state, because
that run used `STRUCTURED_V1`, which writes none. In the K4a bundle, 6 of 6 do.
Neither number answers K4c's question, which is why the two were previously
confused.
## K5 — SPEC § 5.1 provenance
**0 of 39 concepts** in the corpus bundle carry any SPEC § 5.1 source entry.
The measured keys — `sources`, `sources[].resource`, `sources[].id`,
`sources[].title`, `sources[].author`, `usage_count`, `last_modified` — are all
present on 0 concepts. Door B under `STRUCTURED_V1` records provenance instead
as `source_file` and `source_sha256`, present on 39 of 39; those are this
repository's keys, not § 5.1's, and a consumer reading for § 5.1 finds nothing.
The known ceiling stands and bounds what a different profile could achieve
here: **2 of 5 keys on a single source**. A dropped file yields a resource and a
title; the credibility keys (`author`, `usage_count`, `last_modified`) have no
data behind them on an inbox drop, so no profile can fill them by writing more.
**No threshold is applied to K5.** The method declared none.
## Out of scope, restated
**K4 over time is PM-owned and out of scope here.** This work produces one
baseline. Comparing baselines across runs is a programme-level question the
rubric's owner holds, and answering it from inside this repository would be
answering it in the wrong place.