open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

chore(measure): six blind raters, two per arm, n_blind 6 each

Browse source 
36 blind ratings, six k/6 figures, as the order asked -- twice the Arm D
round's protocol, which used one rater per arm. Blindness is structural: six
separate subagent spawns, each given one arm-neutral file and told to read no
other. Rule names were stripped, because a rule:table-grid in the list would
have identified the arm.

The rows:

  Arm B  blind-1a 4/1/0/1   blind-1b 4/1/0/1     agreement 5/6, 5/6
  Arm D  blind-2a 2/1/0/3   blind-2b 3/1/0/2     agreement 6/6, 5/6
  Arm E  blind-3a 2/1/0/3   blind-3b 2/1/0/3     agreement 5/6, 5/6

Within-arm agreement exists for the first time, because there are two raters
per arm: B 6/6, D 5/6, E 6/6. It bounds self-consistency and nothing more --
every rater is an instance of the same model family.

The finding the first-rater row does not contain: `too fine` is 1 of 6 in
EVERY arm, Arm E included. Both Arm E raters kept `too fine` at position 10,
which the first rater moved to `correct`. Their reason changed rather than
disappearing. Under Arm B and D they object that thirteen table rows are
severed from their header; under Arm E, where the table is one concept with its
header, they object that the table is severed from the sentence that introduces
it. Arm E fixed the first complaint and does not touch the second.

And the denominator that governs how much any of this can say: the blind
positions are fixed at 0, 2, 4, 6, 8, 10, and only ONE of Arm E's three movable
documents falls in that set. Positions 5 and 11 were seen by no blind rater. So
"the blind row did not move" is partly a fact about where the protocol looks,
not only about the arm, and the report says so rather than letting the number
stand alone.

One disagreement is a limitation of the material rather than of a rater: at
position 6 the first rater calls Arm B `too coarse` because a chapter is
ABSORBED into a neighbour and does not exist as a concept, which a rater seeing
only the emitted concepts cannot detect.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-07 11:45:59 +02:00
commit fb46424a05
Download patch file Download diff file Expand all files Collapse all files

Diff content is not available