open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(propose): Arm F, one unit fold behind a flag, measured against the operator's worksheet [skip-docs]

Browse source 
Order 20260908T133512Z-139864689-from-.claude. First iteration of the
per-file-type directive (operator 2026-09-08 13:05Z), not the last. No
threshold is set: ratifying a bar is the operator's, and setting one inside the
work that produces the measurement would be fitting the bar to the number.

[skip-docs] covers README.md only, and it follows a precedent re-measured this
round rather than quoted: `grep -c` for outline-run, table-grid, Arm C, Arm D
and Arm E returns 0 in README.md and CHANGELOG.md, while --path-prefix, a real
interface change, has a CHANGELOG entry. The rule is "interface and behaviour
changes yes, arm flags no", and --unit-fold is an arm flag that defaults off.
CLAUDE.md IS updated, because its `okf build` bullet enumerates which arms are
off there and would otherwise become false.

FUNN 1, and step 1 asked for it: the reproduction broke. Arm E on HEAD is
byte-identical to the archive on 31 of 33 plans; the two that differ are 2 of 2
spreadsheets in the corpus. The cause is EXTRACTION, not segmentation --
56ae274 writes a workbook as pipe tables, and the sample's price sheet extracts
to 11 048 characters where the worksheet records 100 694, which is the figure
that commit's own message predicts. The consequence is a segmentation
regression against the reference: K3 position 3 went 3 concepts -> 1 under both
Arm D and Arm E, where the operator wants eleven. The mechanism is the orphan
check dropping the sheet heading once a table opens below it (propose.py:461),
already reported there as a ranking regression. Doors unchanged: 43 .err, 4
FAILED, extractable 39/43.

THE MATCH CRITERION WAS WRITTEN DOWN BEFORE ANY CELL WAS SCORED, and it stalls
at 7/12 on the literal calibration gate after three rounds, each revision
recorded. The five failures are not the criterion's: at every one it agrees
with the operator's own (a), (b) or free text and disagrees only with (c).
Column (c) is a RELATIVE judgement ("closest today"); the four K3 categories
are absolute. The only way to reach 12/12 is to define "correct" as "the
closest arm", which reads (c) back out of itself. The dominance gate, declared
in advance as the second reading, holds at 11/12.

ARM F is one rule with three clauses derived from the operator's three, not
twelve special cases, and it only MERGES or DISCARDS: a run of at least
CONTENTS_RUN same-level page-numbered headings is a contents list and goes; a
heading deeper than the unit level folds into its parent, extending the
parent's span; a table folds back into the shorter heading that introduces it,
keeping the HEADING's name. K3 first rater, n=12: 2 coarse / 5 fine / 0
duplicate / 5 correct -- best of four arms, ceiling was 4, two moved, nothing
regressed anywhere.

THE PAPER MEASUREMENT CAME FIRST AND FALSIFIED THE FIRST VERSION. Clause 2 was
letting rule:outline -- Arm D's RECOVERY of an integer numbering run -- vote on
the unit level, which took K3 positions 1, 7 and 9 to 3, 4 and 7 concepts
instead of 17, 34 and 11. A recovered numbering is a heuristic, not a level a
document declares, and the unit worksheet showed the operator ATX and dotted
headings only. Fixed with its own red test; 11 of 12 predictions correct after.

PER FILE TYPE, which is the directive: docx 3 of 3 (solved on this sample), pdf
2 of 8 (lags, unchanged by Arm F, and the remainder is decomposed per position
rather than left as one number), xlsx 0 of 1 (regressed, see FUNN 1). Outside
the corpus, n=1 each: pptx and odt byte-identical, rtf proposes nothing either
way, txt differs and exposes clause 2's fallback.

CONTENTS_RUN swept 1..5 and off. Distance prefers 1; three ships anyway,
because at 1 the body chapter "... i henhold til TEK 17" is deleted for ending
in a number, and no K3 cell differs between 1 and 4 -- the metric prefers a
value that provably deletes a chapter and cannot see the cost.

Whole corpus, all 43 through arm_run in ascending foreground chunks: 32 plans,
491 entries against Arm E's 679, 14 documents changed, 1 plan disappeared
entirely (three drawing-schedule numbers that clause 1 correctly reads as a
contents run) and that is reported rather than special-cased.

THE okf build MECHANISM IS REPRODUCED AND IT IS NOT DOOR B: cli.py calls the
proposer with no arm flag at all, so the shipped build path is Arm B. On a
tender PDF that means no boundary where Arm D finds nine and the reference says
nine. Largest per-file-type gap this round found; it is a default change and
therefore the operator's.

5 tests red first, 1373 -> 1379. ruff clean, mypy --strict clean on 17 files.
K2 consumer bundle unchanged: 1108 files, digest 9cd74519... with the flag off.
No bundle built, no version bump, no tag, no push.

Report: docs/2026-09-08-k3-arm-f-mot-enhetsarket.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 16:28:41 +02:00
commit dfaf3cc134
4 changed files with 748 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

385
docs/2026-09-08-k3-arm-f-mot-enhetsarket.md Normal file
View file

@ -0,0 +1,385 @@
# K3 against the operator's unit worksheet: Arm F, per file type, 2026-09-08
First iteration, not the last. Order `20260908T133512Z-139864689-from-.claude`,
which refuses a threshold, refuses a push, and adds one directive: the numbers
are broken down **per file type**, because a bundle whose `docx` is right and
whose `pdf` is not has no predictable quality at all.
The reference this round measures against is the operator's own filled unit
worksheet (36 of 36 crosses, 2026-09-08 09:45-10:05). The worksheet is
local-only and gitignored -- it quotes document titles and extracted text --
so nothing below reproduces its content beyond the three rules the operator
wrote in free text, its (a)/(b)/(c) codes, and counts.
## Section 0: what was measured and what was not
**Measured.** The sample draw (12/12). The 36 concept counts behind the
worksheet (36/36). Arm E's byte identity on HEAD over the whole corpus
(31 of 33 plans identical; the two exceptions are diagnosed below). A match
criterion written down before any cell was scored, revised three times, with
both readings of its calibration gate reported. A 12 x 4 cell matrix. One
general splitting rule behind a flag, with a red test per clause and a
known-negative. The rule on all 43 corpus documents. The rule's one number,
swept. The whole-corpus door counts. The `okf build` behaviour on a five-file
folder, per file type, per arm.
**Not measured.** No blind rater (the operator is the reference now, and the
order did not ask for one). No threshold -- ratifying a bar belongs to the
operator, and setting one inside the work that produces the measurement would
be fitting the bar to the number. No bundle was built for Arm F. No consumer
bundle changed: the K2 bundle's locale-pinned digest is unchanged.
## FUNN 1: the reproduction broke, and it is the spreadsheet
The order's step 1 asks for byte-identical Arm E plans on HEAD and says that a
difference is finding number one. There is one.
| control | result |
|---|---|
| sample draw reproduced | **12/12**, in the published order |
| worksheet's concept counts vs the archived plans | **36/36** (12 documents x 3 arms) |
| Arm E on HEAD vs the archive, whole corpus | **31 of 33** plans byte-identical |
| the two that differ | both `.xlsx` -- **2 of 2** spreadsheets in the corpus |
| doors: `.err` files / `FAILED` | **43** / **4**, so extractable **39/43**, unchanged |
The cause is **extraction, not segmentation**. `56ae274` (2026-09-08 10:06,
"write a spreadsheet as pipe tables") changed how a workbook becomes text; the
Arm E archive is from 2026-09-07. The sample's spreadsheet extracts to 11 048
characters on HEAD where the worksheet records 100 694, and 11 048 is the
figure `56ae274`'s own commit message predicts. Its `text_sha256` differs
accordingly.
The consequence is a regression against the reference:
| K3 position 3 (`.xlsx`) | Arm B | Arm D | Arm E |
|---|---|---|---|
| archive, 2026-09-07 (what the operator rated) | 1 | 3 | 3 |
| HEAD, 2026-09-08 | 1 | **1** | **1** |
The operator's answer for that document is (a) `U`, one unit per numbered cost
group -- there are eleven -- so three was already too coarse and one is worse.
The mechanism is named in `56ae274`'s own message: the orphan check drops the
sheet heading once a table block opens two lines below it. It was reported
there as a ranking regression; this round measures it as a **segmentation**
regression, on the only spreadsheet in the K3 sample.
## The reference, recounted here
Counted from the worksheet's 12 checkbox blocks, denominator 12 per question,
independently of the count in the order:
- **(a) unit:** top-level chapter **x8** (positions 1, 2, 5, 6, 7, 8, 9, 10);
numbered subsection **x1** (3); other **x3** (0, 4, 11).
- **(b) table:** a unit WITH its introduction **x3** (3, 5, 10); part of the
section **x3** (7, 8, 9); a unit alone **x1** (11); not applicable **x5**
(0, 1, 2, 4, 6).
- **(c) closest today:** B **x3** (4, 6, 9); D **x4** (1, 2, 3, 8); E **x3**
(5, 10, 11); none **x2** (0, 7).
All three agree with the worksheet's own summary table, read separately. The
operator's three rules, quoted: "hvert h2-kapittel med sine h3" is the unit;
"tabellen med innledningen" is one lookup; "innholdsfortegnelsen er ikke
konsepter".
## The match criterion, and what its calibration measured
The criterion was written to a file **before** any cell was scored. It derives
an expected unit set `F(d)` per document from (a), (b) and the free text, using
the shipped grammar imported rather than copied, and scores each arm
`treffer / for fint / for grovt / duplikat` with K3's tie-break (coarse before
fine before duplicate).
Two readings of the gate were declared in advance, because (c) is a
**relative** judgement ("closest today") while the four categories are
**absolute**:
- **LITERAL** -- the arm named in (c) must be `treffer`.
- **DOMINANCE** -- the arm named in (c) must be the row's minimum-distance arm,
distance `= |U_P - |F(d)|| + T_P + S_P`.
**Three rounds.** Round 1: 6/12 literal. Round 2 fixed a defect in the
criterion -- `_TRAILING_PAGE_NUMBER` reads a table concept titled
`Tabell linje 48` as a contents line, and the unit count was subtracted twice,
which drove it negative; a contents line is now heading-derived only. 7/12.
Round 3 made a contents line one of a RUN of at least three, because a single
body heading ending in a number is not a contents list -- measured: `... i
henhold til TEK 17`. Still **7/12 literal, 11/12 dominance** over the three
arms the operator rated.
**The criterion stalls at 7/12 and the remaining five failures are not the
criterion's.** At each one the criterion agrees with the operator's own (a),
(b) or free text and disagrees only with (c):
| pos | (c) says | criterion says | the operator's own words |
|---|---|---|---|
| 1 | D | D `for fint` | "innholdsfortegnelsen er ikke konsepter" -- D emits 6 of them |
| 3 | D | D `for grovt` | "85 000 tegn i en bolk er for grovt" |
| 5 | E | E `for fint` | (b) = the table is a unit WITH its introduction |
| 9 | B | B `for fint` | "radene er ikke overskrifter"; B keeps a contents line |
| 10 | E | E `for fint` | (b) = a unit WITH its introduction |
The only way to reach 12/12 literally is to define `treffer` as "the closest
arm", which reads column (c) back out of itself and measures nothing. So the
honest report is: **column (c) cannot be converted into an absolute label, and
columns (a)/(b) are the ones an arm can be measured against.** The dominance
gate, which respects that, holds at 11/12 (it fails at position 1, where the
criterion ranks Arm B closer than the Arm D the operator named).
## Arm F: the unit fold
One rule, three clauses, derived from the operator's three rules rather than
from twelve special cases. It **merges and discards only** -- it proposes no
boundary of its own, so every plan it can produce is a subset of Arm E's.
> **Clause 1.** A run of at least `CONTENTS_RUN` consecutive same-level
> headings, each ending in a page number, is a contents list and is discarded.
> A run of siblings, never a single line.
>
> **Clause 2.** The unit level is the shallowest DECLARED heading level
> occurring more than once; anything deeper folds into the preceding candidate
> at or above that level, extending the parent's span rather than deleting the
> child's body.
>
> **Clause 3.** A table folds back into the heading immediately before it when
> that heading's own span is shorter than the table's. The surviving concept
> keeps the HEADING's name.
Flag: `--unit-fold`, off by default, boolean.
### The paper measurement came first, and it falsified the first version
Predicted before the run, on the twelve: 11 of 12 correct in the version that
shipped. The version before it was **wrong on three** -- positions 1, 7 and 9
came out at 3, 4 and 7 concepts instead of 17, 34 and 11. The cause is
measurable and is now a test: clause 2 was letting `rule:outline` -- Arm D's
RECOVERY of an integer numbering run -- vote on the unit level. On every PDF
that has both, the recovered integer level is the shallowest repeated one, so
every dotted heading the operator actually named folded into it. The unit
worksheet showed the operator ATX and dotted headings and nothing else, which
is the same set clause 2 now reads. The one remaining prediction miss is
position 7: 33 predicted, **34** measured.
### The ceiling, stated before the verdict
Arm F and Arm E give a different plan at positions **1, 5, 7, 10**, so **at
most 4 of 12 can move.** Two moved.
### Tests
Four red-first tests, one per clause plus the known-negative, and a fifth for
the round-2 fix. The known-negative is the control the other four rest on: a
document with one heading level and no table comes out of the fold as
**identical objects**, not merely an equal count. `1373 -> 1379` tests, `ruff`
clean, `mypy --strict` clean on 17 files.
### The one number, swept
`CONTENTS_RUN`, on the twelve, as total distance to the reference:
| run | 1 | 2 | 3 | 4 | 5 | off |
|---|---|---|---|---|---|---|
| sum of distances | **24** | 27 | 27 | 27 | 34 | 40 |
Distance prefers 1. **Three is shipped anyway, and the reason is measured
rather than argued:** at `run = 1` the body chapter `Sikkerhet mot
naturpaakjenninger i henhold til TEK 17` is deleted, because its title ends in
a number. No K3 CELL differs between 1 and 4 -- the label is insensitive across
that whole span -- so the distance metric prefers a value that provably deletes
a chapter and cannot see the cost. Two, three and four are indistinguishable on
this corpus; three is the middle of the flat region.
## K3, first rater, n = 12
| arm | too coarse | too fine | duplicate | correct | sum |
|---|---|---|---|---|---|
| Arm B | 5 | 5 | 0 | 2 | 12 |
| Arm D | 2 | 8 | 0 | 2 | 12 |
| Arm E | 2 | 7 | 0 | 3 | 12 |
| **Arm F** | **2** | **5** | **0** | **5** | 12 |
Not comparable with the row in `docs/2026-09-07-k3-arm-e.md`: that row was one
model identity reading proposals against the four category definitions; this
one is a written criterion evaluated against the operator's filled worksheet.
Same categories, different instrument.
### The 12 x 4 matrix
| pos | type | (a)(b)(c) | \|F\| | Arm B | Arm D | Arm E | Arm F |
|---|---|---|---|---|---|---|---|
| 0 | pdf | A - I | undef | coarse | coarse | coarse | coarse |
| 1 | pdf | K - D | 15 | coarse | fine | fine | fine |
| 2 | pdf | K - D | 9 | coarse | **correct** | **correct** | **correct** |
| 3 | xlsx | U M D | 11 | coarse | coarse | coarse | coarse |
| 4 | pdf | A - B | 1 | **correct** | fine | fine | fine |
| 5 | docx | K M E | 3 | fine | fine | fine | **correct** |
| 6 | pdf | K - B | 6 | **correct** | fine | fine | fine |
| 7 | pdf | K D I | 30 | fine | fine | fine | fine |
| 8 | pdf | K D D | 8 | coarse | **correct** | **correct** | **correct** |
| 9 | pdf | K D B | 4 | fine | fine | fine | fine |
| 10 | docx | K M E | 2 | fine | fine | fine | **correct** |
| 11 | docx | A E E | 1 | fine | fine | **correct** | **correct** |
**What moved:** positions 5 and 10, both `too fine -> correct`, both `.docx`,
both the case the operator's (b) names -- a table severed from the sentence
that introduces it. Those are exactly the two complaints the Arm E round's
blind raters kept after Arm E fixed the other one.
**What did not move, and why.** Position 1: the contents run goes (T 6 -> 0)
and the distance falls from 8 to 2, but 17 body concepts against 15 expected is
still `too fine`. Position 7: distance 18 -> 4, still `too fine` at 34 against
30 -- and the operator marked it "none" already. Position 9: one contents line,
not a run of three, so clause 1 does not fire; unchanged. Positions 0 and 7 are
the two where the operator says no arm is closest, and **Arm F says nothing at
position 0** (an unreadable document has no candidate to fold) while it halves
the distance at position 7 without changing the label.
**Nothing regressed.** No cell got worse in any arm.
## Per file type -- the directive of 2026-09-08 13:05Z
| type | n | Arm B | Arm D | Arm E | Arm F |
|---|---|---|---|---|---|
| `pdf` | 8 | 2 correct | 2 | 2 | **2** |
| `docx` | 3 | 0 correct | 0 | 1 | **3** |
| `xlsx` | 1 | 0 correct | 0 | 0 | **0** |
**`docx` is solved on this sample: 3 of 3.** Both clauses that fire there --
the table fold and the contents discard -- are what the operator asked for, and
the third `.docx` was already right under Arm E.
**`pdf` is the file type that lags, 2 of 8, and it did not improve.** Arm F
moves it closer twice (positions 1 and 7) without crossing into `correct`. The
The remainder is not the fold: at position 1 seventeen concepts against fifteen is the
heading rule over-proposing, at position 9 it is a single contents line below
the run threshold, at position 4 the arms split a note the operator wants
whole, and position 0 is an extraction failure (95.1 % `(cid:N)` glyphs) that
no segmentation can repair.
**`xlsx` is 0 of 1 and regressed this week** -- see FUNN 1. Arm F changes
nothing there, because after the new extraction the sheet has one table
candidate and no heading in front of it to fold into.
### File types outside the corpus
`pptx`, `odt`, `rtf` exist only as synthetic fixtures (**n = 1 each**, three in
total), and `txt` as one written for this measurement. Arm F was measured on
them as byte identity:
| type | Arm E | Arm F | |
|---|---|---|---|
| `pptx` | 2 | 2 | byte-identical |
| `odt` | 1 | 1 | byte-identical |
| `rtf` | no boundary | no boundary | both propose nothing |
| `txt` | 2 | 1 | **differs** -- the `##` folds into the `#` |
The `txt` case exposes clause 2's fallback: when NO heading level repeats, the
shallowest level wins and everything folds into the first heading. On a
two-heading document that is what "h2 with its h3" means; on a longer one
nobody has measured it, and `n = 1` is not a measurement.
## Whole-corpus behaviour, 43 documents
`arm_run` in ascending chunks, all in the foreground.
| figure | Arm E | Arm F | denominator |
|---|---|---|---|
| plans written | 33 | **32** | 43 |
| entries | 679 | **491** | delta **-188** |
| documents whose entry count changed | -- | **14** | 33 |
| documents whose plan disappeared | -- | **1** | 33 |
| `.err` files | 43 | 43 | 43 |
| `.err` recording `FAILED` | 4 | 4 | so extractable **39/43** |
| changed, by type | -- | 12 `pdf`, 2 `docx`, 0 `xlsx` | of 27 / 4 / 2 |
**Arm E's whole-corpus entry count is 679 here and 681 in the Arm E round.**
The difference is the two spreadsheets of FUNN 1, not the flag.
**One document lost its plan entirely, and that is clause 1 doing its job
loudly.** Its three candidates were `477 3 025 1`, `750 1` and `300 1` --
numbers from a drawing schedule, all `rule:outline`, all at one level, all
ending in a digit. Clause 1 discards all three and the document falls back to
one flat concept. Three junk concepts and no concept are both wrong; the
document is a drawing set with no recoverable prose either way. It is reported
rather than special-cased.
## The `pdf` mechanism in `okf build`, reproduced
On a five-document folder (2 `pdf`, 2 `docx`, 1 `xlsx`), `okf build` puts both
PDFs in the bundle root as one flat `inbox-*.md` each, while the `docx` and
`xlsx` files get a folder of concepts. Reproduced this round: 31 markdown
files, 5 of 5 merged substantively, 0 rejections.
**It is not a Door B rejection and it is not a missing outline in the file.**
`cli.py`'s `_propose_plans` calls the proposer with **no arm flag at all** --
`outline_run` 0, `table_grid` off, `unit_fold` off -- which is Arm B. Measured
on those five documents, per arm:
| document | B (what `okf build` runs) | D | E | F |
|---|---|---|---|---|
| tender PDF, technical requirements | **no boundary** | 9 | 9 | 9 |
| tender PDF, technical layout | no boundary | no boundary | no boundary | no boundary |
| price sheet `.xlsx` | 1 | 1 | 1 | 1 |
| experience list `.docx` | 21 | 21 | 6 | **3** |
| agreement `.docx` | 2 | 2 | 1 | **1** |
So for the first PDF the shipped build path produces one concept while a flag
that already exists produces nine, and the reference says nine. The second PDF
proposes nothing under any arm -- that one has no recoverable structure at all.
**The build path is a full arm behind the proposer**, and that is the single
largest per-file-type gap this round found.
## Reproducing
```
C=~/corpora/okf-telling-20260829
# The corpus loop is the one published in docs/2026-09-07-k3-arm-e.md
# section Reproducing, unchanged. Ascending chunks, each a foreground call.
for lo_hi in "1 12" "13 20" "21 28" "29 36" "37 43"; do
set -- $lo_hi
arm_run "$C/K2-plans-armE-check-20260908" 2026-09-07T00:00:00Z "$1" "$2" \
--outline-run 3 --table-grid
arm_run "$C/K2-plans-armF-20260908" 2026-09-08T00:00:00Z "$1" "$2" \
--outline-run 3 --table-grid --unit-fold
done
# Assert the counts BEFORE the diff: a diff over two trees where every
# document failed compares nothing and exits 0.
ls "$C"/K2-plans-armE-check-20260908/*.json | wc -l # 33
ls "$C"/K2-plans-armF-20260908/*.json | wc -l # 32
ls "$C"/K2-plans-armF-20260908/*.err | wc -l # 43
grep -l FAILED "$C"/K2-plans-armF-20260908/*.err | wc -l # 4 -> 39/43
# FUNN 1: Arm E on HEAD against the archive. Two files differ, both .xlsx.
diff -rq "$C/K2-plans-armE-20260907" "$C/K2-plans-armE-check-20260908" \
-x '*.err' -x '_index.txt'
# The consumer bundle, unchanged by all of it.
cd "$C" && LC_ALL=C find K2-bundle-20260903 -type f | LC_ALL=C sort \
| xargs shasum -a 256 | shasum -a 256
# 1108 files, 9cd745194346cda0c70eab9c7136fa44506203bbe85bc17d7eff2766c6e9b4d1
```
The suite, after `git add` of the named files: `ruff check src tests tools`
clean, `ruff format --check` clean, `mypy --strict src/` clean on 17 files,
`pytest -q` **1379 passed** (C1 left it at 1373).
## Honesty limits
- **n = 12, one rater, and the reference is one operator.** Every figure above
rests on twelve documents scored by one written criterion against one
person's twelve judgements. Per file type the denominators are 8, 3 and 1,
and a `1/1` is not a rate.
- **The criterion is this session's, not the method's.** It is written down
before the scoring and revised three times with each revision stated, which
bounds fitting but does not remove it.
- **Clause 3's length condition was derived from the two documents it moves.**
Both are `.docx`, both in the sample, and the whole-corpus screen found two
more `.docx` it changes -- that tests generalisation outward, not the
circularity inward.
- **The rule was measured on plans, never on a bundle.** No bundle was built
for Arm F, so nothing here says what its concepts look like after
materialisation.
- **Position 7 is diagnosed and not fixed.** Its remaining `too fine` is the
heading rule proposing 34 units where the operator wants 30.
- **No threshold is set, and none is implied.**