open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

chore(measure): first-rater verdicts for Arm B, D and E

Browse source 
36 verdicts, one rater identity, one round, all three arms re-rated rather than
carried over. The rows:

  Arm B  8 too coarse / 4 too fine / 0 duplicate / 0 correct
  Arm D  4 / 5 / 0 / 3
  Arm E  4 / 3 / 0 / 5

Against the direction criterion pre-registered in the brief before the row was
seen -- fewer `too fine` WITHOUT more `too coarse` -- `too fine` goes 5 -> 3
and `too coarse` stays 4. Both halves are published. No threshold is set and
none is implied.

Against the ceiling measured in step 9 (3 of 39 documents carry any table row,
so at most 3 of the 12 can move), the row moved 2 of the 3 it could. The third
is position 5, and it is the finding worth more than the two that moved: Arm E
takes it from 21 concepts to 6, which is the biggest single reduction in the
round, and the LABEL DOES NOT CHANGE. Each of its three references is still cut
into a 114-char title concept carrying a heading and no body, plus its table.
Joining the table rows removed most of the fragmentation and left the rest.

Two things this rating is not. It is not independent: this rater had read the
Arm D report's published rows, so Arm B reproducing 8/4/0/0 and Arm D
reproducing 4/5/0/3 is consistency, not confirmation. And positions 2, 6 and 8
are `correct` only under the reading that a top-level chapter is one unit of
knowledge -- the question Arm D surfaced, which is the operator's and is not
decided here.

Sources were verified before rating, not assumed: both archives reproduced
byte-for-byte in step 8.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-07 11:43:11 +02:00
commit d0f2d67a5f
Download patch file Download diff file Expand all files Collapse all files

Diff content is not available