feat(propose): Arm F, one unit fold behind a flag, measured against the operator's worksheet [skip-docs]
Order 20260908T133512Z-139864689-from-.claude. First iteration of the
per-file-type directive (operator 2026-09-08 13:05Z), not the last. No
threshold is set: ratifying a bar is the operator's, and setting one inside the
work that produces the measurement would be fitting the bar to the number.
[skip-docs] covers README.md only, and it follows a precedent re-measured this
round rather than quoted: `grep -c` for outline-run, table-grid, Arm C, Arm D
and Arm E returns 0 in README.md and CHANGELOG.md, while --path-prefix, a real
interface change, has a CHANGELOG entry. The rule is "interface and behaviour
changes yes, arm flags no", and --unit-fold is an arm flag that defaults off.
CLAUDE.md IS updated, because its `okf build` bullet enumerates which arms are
off there and would otherwise become false.
FUNN 1, and step 1 asked for it: the reproduction broke. Arm E on HEAD is
byte-identical to the archive on 31 of 33 plans; the two that differ are 2 of 2
spreadsheets in the corpus. The cause is EXTRACTION, not segmentation --
56ae274 writes a workbook as pipe tables, and the sample's price sheet extracts
to 11 048 characters where the worksheet records 100 694, which is the figure
that commit's own message predicts. The consequence is a segmentation
regression against the reference: K3 position 3 went 3 concepts -> 1 under both
Arm D and Arm E, where the operator wants eleven. The mechanism is the orphan
check dropping the sheet heading once a table opens below it (propose.py:461),
already reported there as a ranking regression. Doors unchanged: 43 .err, 4
FAILED, extractable 39/43.
THE MATCH CRITERION WAS WRITTEN DOWN BEFORE ANY CELL WAS SCORED, and it stalls
at 7/12 on the literal calibration gate after three rounds, each revision
recorded. The five failures are not the criterion's: at every one it agrees
with the operator's own (a), (b) or free text and disagrees only with (c).
Column (c) is a RELATIVE judgement ("closest today"); the four K3 categories
are absolute. The only way to reach 12/12 is to define "correct" as "the
closest arm", which reads (c) back out of itself. The dominance gate, declared
in advance as the second reading, holds at 11/12.
ARM F is one rule with three clauses derived from the operator's three, not
twelve special cases, and it only MERGES or DISCARDS: a run of at least
CONTENTS_RUN same-level page-numbered headings is a contents list and goes; a
heading deeper than the unit level folds into its parent, extending the
parent's span; a table folds back into the shorter heading that introduces it,
keeping the HEADING's name. K3 first rater, n=12: 2 coarse / 5 fine / 0
duplicate / 5 correct -- best of four arms, ceiling was 4, two moved, nothing
regressed anywhere.
THE PAPER MEASUREMENT CAME FIRST AND FALSIFIED THE FIRST VERSION. Clause 2 was
letting rule:outline -- Arm D's RECOVERY of an integer numbering run -- vote on
the unit level, which took K3 positions 1, 7 and 9 to 3, 4 and 7 concepts
instead of 17, 34 and 11. A recovered numbering is a heuristic, not a level a
document declares, and the unit worksheet showed the operator ATX and dotted
headings only. Fixed with its own red test; 11 of 12 predictions correct after.
PER FILE TYPE, which is the directive: docx 3 of 3 (solved on this sample), pdf
2 of 8 (lags, unchanged by Arm F, and the remainder is decomposed per position
rather than left as one number), xlsx 0 of 1 (regressed, see FUNN 1). Outside
the corpus, n=1 each: pptx and odt byte-identical, rtf proposes nothing either
way, txt differs and exposes clause 2's fallback.
CONTENTS_RUN swept 1..5 and off. Distance prefers 1; three ships anyway,
because at 1 the body chapter "... i henhold til TEK 17" is deleted for ending
in a number, and no K3 cell differs between 1 and 4 -- the metric prefers a
value that provably deletes a chapter and cannot see the cost.
Whole corpus, all 43 through arm_run in ascending foreground chunks: 32 plans,
491 entries against Arm E's 679, 14 documents changed, 1 plan disappeared
entirely (three drawing-schedule numbers that clause 1 correctly reads as a
contents run) and that is reported rather than special-cased.
THE okf build MECHANISM IS REPRODUCED AND IT IS NOT DOOR B: cli.py calls the
proposer with no arm flag at all, so the shipped build path is Arm B. On a
tender PDF that means no boundary where Arm D finds nine and the reference says
nine. Largest per-file-type gap this round found; it is a default change and
therefore the operator's.
5 tests red first, 1373 -> 1379. ruff clean, mypy --strict clean on 17 files.
K2 consumer bundle unchanged: 1108 files, digest 9cd74519... with the flag off.
No bundle built, no version bump, no tag, no push.
Report: docs/2026-09-08-k3-arm-f-mot-enhetsarket.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
171798ed32
commit
dfaf3cc134
4 changed files with 748 additions and 6 deletions
|
|
@ -258,8 +258,11 @@ and fixtures, never code.
|
||||||
recursively, and two same-named documents in different folders must not
|
recursively, and two same-named documents in different folders must not
|
||||||
collide); `--ingested-at` and `--proposed-at` default to one shared epoch
|
collide); `--ingested-at` and `--proposed-at` default to one shared epoch
|
||||||
constant rather than the clock, because a wall-clock default takes
|
constant rather than the clock, because a wall-clock default takes
|
||||||
rebuild-equals-incremental away from anyone who omits them. Arm C, Arm D
|
rebuild-equals-incremental away from anyone who omits them. Arm C, Arm D,
|
||||||
and Arm E are off and not exposed here.
|
Arm E and Arm F are off and not exposed here -- which is measured, not
|
||||||
|
incidental: the build path calls the proposer with no arm flag at all, so a
|
||||||
|
tender PDF that Arm D splits into nine concepts lands as one
|
||||||
|
(`docs/2026-09-08-k3-arm-f-mot-enhetsarket.md`, the `okf build` section).
|
||||||
- Consume a bundle: `python3 tools/okf_consume.py <bundle> --question "<q>"
|
- Consume a bundle: `python3 tools/okf_consume.py <bundle> --question "<q>"
|
||||||
[--k N] [--limit N] [--out PATH] [--ref IDENTITY]` — the **pre-pass**
|
[--k N] [--limit N] [--out PATH] [--ref IDENTITY]` — the **pre-pass**
|
||||||
`docs/consumption-contract.md` § 1 defines, and the only reading direction
|
`docs/consumption-contract.md` § 1 defines, and the only reading direction
|
||||||
|
|
|
||||||
385
docs/2026-09-08-k3-arm-f-mot-enhetsarket.md
Normal file
385
docs/2026-09-08-k3-arm-f-mot-enhetsarket.md
Normal file
|
|
@ -0,0 +1,385 @@
|
||||||
|
# K3 against the operator's unit worksheet: Arm F, per file type, 2026-09-08
|
||||||
|
|
||||||
|
First iteration, not the last. Order `20260908T133512Z-139864689-from-.claude`,
|
||||||
|
which refuses a threshold, refuses a push, and adds one directive: the numbers
|
||||||
|
are broken down **per file type**, because a bundle whose `docx` is right and
|
||||||
|
whose `pdf` is not has no predictable quality at all.
|
||||||
|
|
||||||
|
The reference this round measures against is the operator's own filled unit
|
||||||
|
worksheet (36 of 36 crosses, 2026-09-08 09:45-10:05). The worksheet is
|
||||||
|
local-only and gitignored -- it quotes document titles and extracted text --
|
||||||
|
so nothing below reproduces its content beyond the three rules the operator
|
||||||
|
wrote in free text, its (a)/(b)/(c) codes, and counts.
|
||||||
|
|
||||||
|
## Section 0: what was measured and what was not
|
||||||
|
|
||||||
|
**Measured.** The sample draw (12/12). The 36 concept counts behind the
|
||||||
|
worksheet (36/36). Arm E's byte identity on HEAD over the whole corpus
|
||||||
|
(31 of 33 plans identical; the two exceptions are diagnosed below). A match
|
||||||
|
criterion written down before any cell was scored, revised three times, with
|
||||||
|
both readings of its calibration gate reported. A 12 x 4 cell matrix. One
|
||||||
|
general splitting rule behind a flag, with a red test per clause and a
|
||||||
|
known-negative. The rule on all 43 corpus documents. The rule's one number,
|
||||||
|
swept. The whole-corpus door counts. The `okf build` behaviour on a five-file
|
||||||
|
folder, per file type, per arm.
|
||||||
|
|
||||||
|
**Not measured.** No blind rater (the operator is the reference now, and the
|
||||||
|
order did not ask for one). No threshold -- ratifying a bar belongs to the
|
||||||
|
operator, and setting one inside the work that produces the measurement would
|
||||||
|
be fitting the bar to the number. No bundle was built for Arm F. No consumer
|
||||||
|
bundle changed: the K2 bundle's locale-pinned digest is unchanged.
|
||||||
|
|
||||||
|
## FUNN 1: the reproduction broke, and it is the spreadsheet
|
||||||
|
|
||||||
|
The order's step 1 asks for byte-identical Arm E plans on HEAD and says that a
|
||||||
|
difference is finding number one. There is one.
|
||||||
|
|
||||||
|
| control | result |
|
||||||
|
|---|---|
|
||||||
|
| sample draw reproduced | **12/12**, in the published order |
|
||||||
|
| worksheet's concept counts vs the archived plans | **36/36** (12 documents x 3 arms) |
|
||||||
|
| Arm E on HEAD vs the archive, whole corpus | **31 of 33** plans byte-identical |
|
||||||
|
| the two that differ | both `.xlsx` -- **2 of 2** spreadsheets in the corpus |
|
||||||
|
| doors: `.err` files / `FAILED` | **43** / **4**, so extractable **39/43**, unchanged |
|
||||||
|
|
||||||
|
The cause is **extraction, not segmentation**. `56ae274` (2026-09-08 10:06,
|
||||||
|
"write a spreadsheet as pipe tables") changed how a workbook becomes text; the
|
||||||
|
Arm E archive is from 2026-09-07. The sample's spreadsheet extracts to 11 048
|
||||||
|
characters on HEAD where the worksheet records 100 694, and 11 048 is the
|
||||||
|
figure `56ae274`'s own commit message predicts. Its `text_sha256` differs
|
||||||
|
accordingly.
|
||||||
|
|
||||||
|
The consequence is a regression against the reference:
|
||||||
|
|
||||||
|
| K3 position 3 (`.xlsx`) | Arm B | Arm D | Arm E |
|
||||||
|
|---|---|---|---|
|
||||||
|
| archive, 2026-09-07 (what the operator rated) | 1 | 3 | 3 |
|
||||||
|
| HEAD, 2026-09-08 | 1 | **1** | **1** |
|
||||||
|
|
||||||
|
The operator's answer for that document is (a) `U`, one unit per numbered cost
|
||||||
|
group -- there are eleven -- so three was already too coarse and one is worse.
|
||||||
|
The mechanism is named in `56ae274`'s own message: the orphan check drops the
|
||||||
|
sheet heading once a table block opens two lines below it. It was reported
|
||||||
|
there as a ranking regression; this round measures it as a **segmentation**
|
||||||
|
regression, on the only spreadsheet in the K3 sample.
|
||||||
|
|
||||||
|
## The reference, recounted here
|
||||||
|
|
||||||
|
Counted from the worksheet's 12 checkbox blocks, denominator 12 per question,
|
||||||
|
independently of the count in the order:
|
||||||
|
|
||||||
|
- **(a) unit:** top-level chapter **x8** (positions 1, 2, 5, 6, 7, 8, 9, 10);
|
||||||
|
numbered subsection **x1** (3); other **x3** (0, 4, 11).
|
||||||
|
- **(b) table:** a unit WITH its introduction **x3** (3, 5, 10); part of the
|
||||||
|
section **x3** (7, 8, 9); a unit alone **x1** (11); not applicable **x5**
|
||||||
|
(0, 1, 2, 4, 6).
|
||||||
|
- **(c) closest today:** B **x3** (4, 6, 9); D **x4** (1, 2, 3, 8); E **x3**
|
||||||
|
(5, 10, 11); none **x2** (0, 7).
|
||||||
|
|
||||||
|
All three agree with the worksheet's own summary table, read separately. The
|
||||||
|
operator's three rules, quoted: "hvert h2-kapittel med sine h3" is the unit;
|
||||||
|
"tabellen med innledningen" is one lookup; "innholdsfortegnelsen er ikke
|
||||||
|
konsepter".
|
||||||
|
|
||||||
|
## The match criterion, and what its calibration measured
|
||||||
|
|
||||||
|
The criterion was written to a file **before** any cell was scored. It derives
|
||||||
|
an expected unit set `F(d)` per document from (a), (b) and the free text, using
|
||||||
|
the shipped grammar imported rather than copied, and scores each arm
|
||||||
|
`treffer / for fint / for grovt / duplikat` with K3's tie-break (coarse before
|
||||||
|
fine before duplicate).
|
||||||
|
|
||||||
|
Two readings of the gate were declared in advance, because (c) is a
|
||||||
|
**relative** judgement ("closest today") while the four categories are
|
||||||
|
**absolute**:
|
||||||
|
|
||||||
|
- **LITERAL** -- the arm named in (c) must be `treffer`.
|
||||||
|
- **DOMINANCE** -- the arm named in (c) must be the row's minimum-distance arm,
|
||||||
|
distance `= |U_P - |F(d)|| + T_P + S_P`.
|
||||||
|
|
||||||
|
**Three rounds.** Round 1: 6/12 literal. Round 2 fixed a defect in the
|
||||||
|
criterion -- `_TRAILING_PAGE_NUMBER` reads a table concept titled
|
||||||
|
`Tabell linje 48` as a contents line, and the unit count was subtracted twice,
|
||||||
|
which drove it negative; a contents line is now heading-derived only. 7/12.
|
||||||
|
Round 3 made a contents line one of a RUN of at least three, because a single
|
||||||
|
body heading ending in a number is not a contents list -- measured: `... i
|
||||||
|
henhold til TEK 17`. Still **7/12 literal, 11/12 dominance** over the three
|
||||||
|
arms the operator rated.
|
||||||
|
|
||||||
|
**The criterion stalls at 7/12 and the remaining five failures are not the
|
||||||
|
criterion's.** At each one the criterion agrees with the operator's own (a),
|
||||||
|
(b) or free text and disagrees only with (c):
|
||||||
|
|
||||||
|
| pos | (c) says | criterion says | the operator's own words |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 1 | D | D `for fint` | "innholdsfortegnelsen er ikke konsepter" -- D emits 6 of them |
|
||||||
|
| 3 | D | D `for grovt` | "85 000 tegn i en bolk er for grovt" |
|
||||||
|
| 5 | E | E `for fint` | (b) = the table is a unit WITH its introduction |
|
||||||
|
| 9 | B | B `for fint` | "radene er ikke overskrifter"; B keeps a contents line |
|
||||||
|
| 10 | E | E `for fint` | (b) = a unit WITH its introduction |
|
||||||
|
|
||||||
|
The only way to reach 12/12 literally is to define `treffer` as "the closest
|
||||||
|
arm", which reads column (c) back out of itself and measures nothing. So the
|
||||||
|
honest report is: **column (c) cannot be converted into an absolute label, and
|
||||||
|
columns (a)/(b) are the ones an arm can be measured against.** The dominance
|
||||||
|
gate, which respects that, holds at 11/12 (it fails at position 1, where the
|
||||||
|
criterion ranks Arm B closer than the Arm D the operator named).
|
||||||
|
|
||||||
|
## Arm F: the unit fold
|
||||||
|
|
||||||
|
One rule, three clauses, derived from the operator's three rules rather than
|
||||||
|
from twelve special cases. It **merges and discards only** -- it proposes no
|
||||||
|
boundary of its own, so every plan it can produce is a subset of Arm E's.
|
||||||
|
|
||||||
|
> **Clause 1.** A run of at least `CONTENTS_RUN` consecutive same-level
|
||||||
|
> headings, each ending in a page number, is a contents list and is discarded.
|
||||||
|
> A run of siblings, never a single line.
|
||||||
|
>
|
||||||
|
> **Clause 2.** The unit level is the shallowest DECLARED heading level
|
||||||
|
> occurring more than once; anything deeper folds into the preceding candidate
|
||||||
|
> at or above that level, extending the parent's span rather than deleting the
|
||||||
|
> child's body.
|
||||||
|
>
|
||||||
|
> **Clause 3.** A table folds back into the heading immediately before it when
|
||||||
|
> that heading's own span is shorter than the table's. The surviving concept
|
||||||
|
> keeps the HEADING's name.
|
||||||
|
|
||||||
|
Flag: `--unit-fold`, off by default, boolean.
|
||||||
|
|
||||||
|
### The paper measurement came first, and it falsified the first version
|
||||||
|
|
||||||
|
Predicted before the run, on the twelve: 11 of 12 correct in the version that
|
||||||
|
shipped. The version before it was **wrong on three** -- positions 1, 7 and 9
|
||||||
|
came out at 3, 4 and 7 concepts instead of 17, 34 and 11. The cause is
|
||||||
|
measurable and is now a test: clause 2 was letting `rule:outline` -- Arm D's
|
||||||
|
RECOVERY of an integer numbering run -- vote on the unit level. On every PDF
|
||||||
|
that has both, the recovered integer level is the shallowest repeated one, so
|
||||||
|
every dotted heading the operator actually named folded into it. The unit
|
||||||
|
worksheet showed the operator ATX and dotted headings and nothing else, which
|
||||||
|
is the same set clause 2 now reads. The one remaining prediction miss is
|
||||||
|
position 7: 33 predicted, **34** measured.
|
||||||
|
|
||||||
|
### The ceiling, stated before the verdict
|
||||||
|
|
||||||
|
Arm F and Arm E give a different plan at positions **1, 5, 7, 10**, so **at
|
||||||
|
most 4 of 12 can move.** Two moved.
|
||||||
|
|
||||||
|
### Tests
|
||||||
|
|
||||||
|
Four red-first tests, one per clause plus the known-negative, and a fifth for
|
||||||
|
the round-2 fix. The known-negative is the control the other four rest on: a
|
||||||
|
document with one heading level and no table comes out of the fold as
|
||||||
|
**identical objects**, not merely an equal count. `1373 -> 1379` tests, `ruff`
|
||||||
|
clean, `mypy --strict` clean on 17 files.
|
||||||
|
|
||||||
|
### The one number, swept
|
||||||
|
|
||||||
|
`CONTENTS_RUN`, on the twelve, as total distance to the reference:
|
||||||
|
|
||||||
|
| run | 1 | 2 | 3 | 4 | 5 | off |
|
||||||
|
|---|---|---|---|---|---|---|
|
||||||
|
| sum of distances | **24** | 27 | 27 | 27 | 34 | 40 |
|
||||||
|
|
||||||
|
Distance prefers 1. **Three is shipped anyway, and the reason is measured
|
||||||
|
rather than argued:** at `run = 1` the body chapter `Sikkerhet mot
|
||||||
|
naturpaakjenninger i henhold til TEK 17` is deleted, because its title ends in
|
||||||
|
a number. No K3 CELL differs between 1 and 4 -- the label is insensitive across
|
||||||
|
that whole span -- so the distance metric prefers a value that provably deletes
|
||||||
|
a chapter and cannot see the cost. Two, three and four are indistinguishable on
|
||||||
|
this corpus; three is the middle of the flat region.
|
||||||
|
|
||||||
|
## K3, first rater, n = 12
|
||||||
|
|
||||||
|
| arm | too coarse | too fine | duplicate | correct | sum |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| Arm B | 5 | 5 | 0 | 2 | 12 |
|
||||||
|
| Arm D | 2 | 8 | 0 | 2 | 12 |
|
||||||
|
| Arm E | 2 | 7 | 0 | 3 | 12 |
|
||||||
|
| **Arm F** | **2** | **5** | **0** | **5** | 12 |
|
||||||
|
|
||||||
|
Not comparable with the row in `docs/2026-09-07-k3-arm-e.md`: that row was one
|
||||||
|
model identity reading proposals against the four category definitions; this
|
||||||
|
one is a written criterion evaluated against the operator's filled worksheet.
|
||||||
|
Same categories, different instrument.
|
||||||
|
|
||||||
|
### The 12 x 4 matrix
|
||||||
|
|
||||||
|
| pos | type | (a)(b)(c) | \|F\| | Arm B | Arm D | Arm E | Arm F |
|
||||||
|
|---|---|---|---|---|---|---|---|
|
||||||
|
| 0 | pdf | A - I | undef | coarse | coarse | coarse | coarse |
|
||||||
|
| 1 | pdf | K - D | 15 | coarse | fine | fine | fine |
|
||||||
|
| 2 | pdf | K - D | 9 | coarse | **correct** | **correct** | **correct** |
|
||||||
|
| 3 | xlsx | U M D | 11 | coarse | coarse | coarse | coarse |
|
||||||
|
| 4 | pdf | A - B | 1 | **correct** | fine | fine | fine |
|
||||||
|
| 5 | docx | K M E | 3 | fine | fine | fine | **correct** |
|
||||||
|
| 6 | pdf | K - B | 6 | **correct** | fine | fine | fine |
|
||||||
|
| 7 | pdf | K D I | 30 | fine | fine | fine | fine |
|
||||||
|
| 8 | pdf | K D D | 8 | coarse | **correct** | **correct** | **correct** |
|
||||||
|
| 9 | pdf | K D B | 4 | fine | fine | fine | fine |
|
||||||
|
| 10 | docx | K M E | 2 | fine | fine | fine | **correct** |
|
||||||
|
| 11 | docx | A E E | 1 | fine | fine | **correct** | **correct** |
|
||||||
|
|
||||||
|
**What moved:** positions 5 and 10, both `too fine -> correct`, both `.docx`,
|
||||||
|
both the case the operator's (b) names -- a table severed from the sentence
|
||||||
|
that introduces it. Those are exactly the two complaints the Arm E round's
|
||||||
|
blind raters kept after Arm E fixed the other one.
|
||||||
|
|
||||||
|
**What did not move, and why.** Position 1: the contents run goes (T 6 -> 0)
|
||||||
|
and the distance falls from 8 to 2, but 17 body concepts against 15 expected is
|
||||||
|
still `too fine`. Position 7: distance 18 -> 4, still `too fine` at 34 against
|
||||||
|
30 -- and the operator marked it "none" already. Position 9: one contents line,
|
||||||
|
not a run of three, so clause 1 does not fire; unchanged. Positions 0 and 7 are
|
||||||
|
the two where the operator says no arm is closest, and **Arm F says nothing at
|
||||||
|
position 0** (an unreadable document has no candidate to fold) while it halves
|
||||||
|
the distance at position 7 without changing the label.
|
||||||
|
|
||||||
|
**Nothing regressed.** No cell got worse in any arm.
|
||||||
|
|
||||||
|
## Per file type -- the directive of 2026-09-08 13:05Z
|
||||||
|
|
||||||
|
| type | n | Arm B | Arm D | Arm E | Arm F |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| `pdf` | 8 | 2 correct | 2 | 2 | **2** |
|
||||||
|
| `docx` | 3 | 0 correct | 0 | 1 | **3** |
|
||||||
|
| `xlsx` | 1 | 0 correct | 0 | 0 | **0** |
|
||||||
|
|
||||||
|
**`docx` is solved on this sample: 3 of 3.** Both clauses that fire there --
|
||||||
|
the table fold and the contents discard -- are what the operator asked for, and
|
||||||
|
the third `.docx` was already right under Arm E.
|
||||||
|
|
||||||
|
**`pdf` is the file type that lags, 2 of 8, and it did not improve.** Arm F
|
||||||
|
moves it closer twice (positions 1 and 7) without crossing into `correct`. The
|
||||||
|
The remainder is not the fold: at position 1 seventeen concepts against fifteen is the
|
||||||
|
heading rule over-proposing, at position 9 it is a single contents line below
|
||||||
|
the run threshold, at position 4 the arms split a note the operator wants
|
||||||
|
whole, and position 0 is an extraction failure (95.1 % `(cid:N)` glyphs) that
|
||||||
|
no segmentation can repair.
|
||||||
|
|
||||||
|
**`xlsx` is 0 of 1 and regressed this week** -- see FUNN 1. Arm F changes
|
||||||
|
nothing there, because after the new extraction the sheet has one table
|
||||||
|
candidate and no heading in front of it to fold into.
|
||||||
|
|
||||||
|
### File types outside the corpus
|
||||||
|
|
||||||
|
`pptx`, `odt`, `rtf` exist only as synthetic fixtures (**n = 1 each**, three in
|
||||||
|
total), and `txt` as one written for this measurement. Arm F was measured on
|
||||||
|
them as byte identity:
|
||||||
|
|
||||||
|
| type | Arm E | Arm F | |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `pptx` | 2 | 2 | byte-identical |
|
||||||
|
| `odt` | 1 | 1 | byte-identical |
|
||||||
|
| `rtf` | no boundary | no boundary | both propose nothing |
|
||||||
|
| `txt` | 2 | 1 | **differs** -- the `##` folds into the `#` |
|
||||||
|
|
||||||
|
The `txt` case exposes clause 2's fallback: when NO heading level repeats, the
|
||||||
|
shallowest level wins and everything folds into the first heading. On a
|
||||||
|
two-heading document that is what "h2 with its h3" means; on a longer one
|
||||||
|
nobody has measured it, and `n = 1` is not a measurement.
|
||||||
|
|
||||||
|
## Whole-corpus behaviour, 43 documents
|
||||||
|
|
||||||
|
`arm_run` in ascending chunks, all in the foreground.
|
||||||
|
|
||||||
|
| figure | Arm E | Arm F | denominator |
|
||||||
|
|---|---|---|---|
|
||||||
|
| plans written | 33 | **32** | 43 |
|
||||||
|
| entries | 679 | **491** | delta **-188** |
|
||||||
|
| documents whose entry count changed | -- | **14** | 33 |
|
||||||
|
| documents whose plan disappeared | -- | **1** | 33 |
|
||||||
|
| `.err` files | 43 | 43 | 43 |
|
||||||
|
| `.err` recording `FAILED` | 4 | 4 | so extractable **39/43** |
|
||||||
|
| changed, by type | -- | 12 `pdf`, 2 `docx`, 0 `xlsx` | of 27 / 4 / 2 |
|
||||||
|
|
||||||
|
**Arm E's whole-corpus entry count is 679 here and 681 in the Arm E round.**
|
||||||
|
The difference is the two spreadsheets of FUNN 1, not the flag.
|
||||||
|
|
||||||
|
**One document lost its plan entirely, and that is clause 1 doing its job
|
||||||
|
loudly.** Its three candidates were `477 3 025 1`, `750 1` and `300 1` --
|
||||||
|
numbers from a drawing schedule, all `rule:outline`, all at one level, all
|
||||||
|
ending in a digit. Clause 1 discards all three and the document falls back to
|
||||||
|
one flat concept. Three junk concepts and no concept are both wrong; the
|
||||||
|
document is a drawing set with no recoverable prose either way. It is reported
|
||||||
|
rather than special-cased.
|
||||||
|
|
||||||
|
## The `pdf` mechanism in `okf build`, reproduced
|
||||||
|
|
||||||
|
On a five-document folder (2 `pdf`, 2 `docx`, 1 `xlsx`), `okf build` puts both
|
||||||
|
PDFs in the bundle root as one flat `inbox-*.md` each, while the `docx` and
|
||||||
|
`xlsx` files get a folder of concepts. Reproduced this round: 31 markdown
|
||||||
|
files, 5 of 5 merged substantively, 0 rejections.
|
||||||
|
|
||||||
|
**It is not a Door B rejection and it is not a missing outline in the file.**
|
||||||
|
`cli.py`'s `_propose_plans` calls the proposer with **no arm flag at all** --
|
||||||
|
`outline_run` 0, `table_grid` off, `unit_fold` off -- which is Arm B. Measured
|
||||||
|
on those five documents, per arm:
|
||||||
|
|
||||||
|
| document | B (what `okf build` runs) | D | E | F |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| tender PDF, technical requirements | **no boundary** | 9 | 9 | 9 |
|
||||||
|
| tender PDF, technical layout | no boundary | no boundary | no boundary | no boundary |
|
||||||
|
| price sheet `.xlsx` | 1 | 1 | 1 | 1 |
|
||||||
|
| experience list `.docx` | 21 | 21 | 6 | **3** |
|
||||||
|
| agreement `.docx` | 2 | 2 | 1 | **1** |
|
||||||
|
|
||||||
|
So for the first PDF the shipped build path produces one concept while a flag
|
||||||
|
that already exists produces nine, and the reference says nine. The second PDF
|
||||||
|
proposes nothing under any arm -- that one has no recoverable structure at all.
|
||||||
|
**The build path is a full arm behind the proposer**, and that is the single
|
||||||
|
largest per-file-type gap this round found.
|
||||||
|
|
||||||
|
## Reproducing
|
||||||
|
|
||||||
|
```
|
||||||
|
C=~/corpora/okf-telling-20260829
|
||||||
|
|
||||||
|
# The corpus loop is the one published in docs/2026-09-07-k3-arm-e.md
|
||||||
|
# section Reproducing, unchanged. Ascending chunks, each a foreground call.
|
||||||
|
for lo_hi in "1 12" "13 20" "21 28" "29 36" "37 43"; do
|
||||||
|
set -- $lo_hi
|
||||||
|
arm_run "$C/K2-plans-armE-check-20260908" 2026-09-07T00:00:00Z "$1" "$2" \
|
||||||
|
--outline-run 3 --table-grid
|
||||||
|
arm_run "$C/K2-plans-armF-20260908" 2026-09-08T00:00:00Z "$1" "$2" \
|
||||||
|
--outline-run 3 --table-grid --unit-fold
|
||||||
|
done
|
||||||
|
|
||||||
|
# Assert the counts BEFORE the diff: a diff over two trees where every
|
||||||
|
# document failed compares nothing and exits 0.
|
||||||
|
ls "$C"/K2-plans-armE-check-20260908/*.json | wc -l # 33
|
||||||
|
ls "$C"/K2-plans-armF-20260908/*.json | wc -l # 32
|
||||||
|
ls "$C"/K2-plans-armF-20260908/*.err | wc -l # 43
|
||||||
|
grep -l FAILED "$C"/K2-plans-armF-20260908/*.err | wc -l # 4 -> 39/43
|
||||||
|
|
||||||
|
# FUNN 1: Arm E on HEAD against the archive. Two files differ, both .xlsx.
|
||||||
|
diff -rq "$C/K2-plans-armE-20260907" "$C/K2-plans-armE-check-20260908" \
|
||||||
|
-x '*.err' -x '_index.txt'
|
||||||
|
|
||||||
|
# The consumer bundle, unchanged by all of it.
|
||||||
|
cd "$C" && LC_ALL=C find K2-bundle-20260903 -type f | LC_ALL=C sort \
|
||||||
|
| xargs shasum -a 256 | shasum -a 256
|
||||||
|
# 1108 files, 9cd745194346cda0c70eab9c7136fa44506203bbe85bc17d7eff2766c6e9b4d1
|
||||||
|
```
|
||||||
|
|
||||||
|
The suite, after `git add` of the named files: `ruff check src tests tools`
|
||||||
|
clean, `ruff format --check` clean, `mypy --strict src/` clean on 17 files,
|
||||||
|
`pytest -q` **1379 passed** (C1 left it at 1373).
|
||||||
|
|
||||||
|
## Honesty limits
|
||||||
|
|
||||||
|
- **n = 12, one rater, and the reference is one operator.** Every figure above
|
||||||
|
rests on twelve documents scored by one written criterion against one
|
||||||
|
person's twelve judgements. Per file type the denominators are 8, 3 and 1,
|
||||||
|
and a `1/1` is not a rate.
|
||||||
|
- **The criterion is this session's, not the method's.** It is written down
|
||||||
|
before the scoring and revised three times with each revision stated, which
|
||||||
|
bounds fitting but does not remove it.
|
||||||
|
- **Clause 3's length condition was derived from the two documents it moves.**
|
||||||
|
Both are `.docx`, both in the sample, and the whole-corpus screen found two
|
||||||
|
more `.docx` it changes -- that tests generalisation outward, not the
|
||||||
|
circularity inward.
|
||||||
|
- **The rule was measured on plans, never on a bundle.** No bundle was built
|
||||||
|
for Arm F, so nothing here says what its concepts look like after
|
||||||
|
materialisation.
|
||||||
|
- **Position 7 is diagnosed and not fixed.** Its remaining `too fine` is the
|
||||||
|
heading rule proposing 34 units where the operator wants 30.
|
||||||
|
- **No threshold is set, and none is implied.**
|
||||||
|
|
@ -56,7 +56,7 @@ import json
|
||||||
import re
|
import re
|
||||||
import sys
|
import sys
|
||||||
import unicodedata
|
import unicodedata
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass, replace
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
|
|
@ -307,7 +307,11 @@ def outline_runs(
|
||||||
|
|
||||||
|
|
||||||
def find_candidates(
|
def find_candidates(
|
||||||
text: str, *, outline_run: int = 0, table_grid: bool = False
|
text: str,
|
||||||
|
*,
|
||||||
|
outline_run: int = 0,
|
||||||
|
table_grid: bool = False,
|
||||||
|
unit_fold: bool = False,
|
||||||
) -> list[Candidate]:
|
) -> list[Candidate]:
|
||||||
"""Every boundary the mechanical rules propose, in document order.
|
"""Every boundary the mechanical rules propose, in document order.
|
||||||
|
|
||||||
|
|
@ -324,6 +328,11 @@ def find_candidates(
|
||||||
numbered outline contributes boundaries where the integers sustain an
|
numbered outline contributes boundaries where the integers sustain an
|
||||||
ascending run of at least `N`.
|
ascending run of at least `N`.
|
||||||
|
|
||||||
|
`unit_fold` is Arm F's gate and it is OFF at False, where `fold_units` is
|
||||||
|
not called at all. On, the candidate list is folded once, at the end: the
|
||||||
|
arm adds no boundary, so every plan it can produce is a subset of the one
|
||||||
|
the flags below produce.
|
||||||
|
|
||||||
`table_grid` is Arm E's gate and it is OFF at False, where the branch is not
|
`table_grid` is Arm E's gate and it is OFF at False, where the branch is not
|
||||||
even evaluated. On, a pandoc grid-table rule line no longer closes an open
|
even evaluated. On, a pandoc grid-table rule line no longer closes an open
|
||||||
table block, so one grid table proposes one candidate instead of one per row
|
table block, so one grid table proposes one candidate instead of one per row
|
||||||
|
|
@ -496,7 +505,111 @@ def find_candidates(
|
||||||
grid=position_in_list in joined,
|
grid=position_in_list in joined,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
return candidates
|
return fold_units(candidates) if unit_fold else candidates
|
||||||
|
|
||||||
|
|
||||||
|
#: Arm F clause 1. How many CONSECUTIVE same-level page-numbered headings make
|
||||||
|
#: a contents list. Three, and it is swept rather than guessed: at 1 and 2 the
|
||||||
|
#: rule deletes body chapters (a heading like `... i henhold til TEK 17` ends
|
||||||
|
#: in a number and is not a contents line), and the sweep front is published in
|
||||||
|
#: `docs/2026-09-08-k3-arm-f-mot-enhetsarket.md`. Not a CLI knob: a number a
|
||||||
|
#: caller can turn without recording the sweep is a number nobody has measured.
|
||||||
|
CONTENTS_RUN = 3
|
||||||
|
|
||||||
|
|
||||||
|
def fold_units(candidates: list[Candidate]) -> list[Candidate]:
|
||||||
|
"""Arm F: ONE rule, three clauses, and it only MERGES or DISCARDS.
|
||||||
|
|
||||||
|
Derived from the three rules the operator wrote across the K3 unit
|
||||||
|
worksheet (2026-09-08), not from twelve special cases:
|
||||||
|
|
||||||
|
1. "innholdsfortegnelsen er ikke konsepter" -- a RUN of at least
|
||||||
|
`CONTENTS_RUN` consecutive same-level headings each ending in a page
|
||||||
|
number is a contents list and is discarded. A run of siblings, never a
|
||||||
|
single line: one body heading ending in a number is not a contents list.
|
||||||
|
2. "hvert h2-kapittel med sine h3" -- the unit level is the SHALLOWEST
|
||||||
|
heading level occurring more than once; anything deeper folds into the
|
||||||
|
preceding candidate at or above that level, which EXTENDS the parent's
|
||||||
|
span rather than deleting the child's body.
|
||||||
|
3. "tabellen med innledningen" -- a table folds back into the heading
|
||||||
|
immediately before it when that heading's own span is shorter than the
|
||||||
|
table's. The surviving concept keeps the HEADING's name: merging the
|
||||||
|
right bytes under `Tabell linje 48` would produce a concept no reader
|
||||||
|
can look up.
|
||||||
|
|
||||||
|
It proposes no boundary of its own, so it can only ever reduce a plan. A
|
||||||
|
document with one heading level and no table comes out identical.
|
||||||
|
"""
|
||||||
|
if not candidates:
|
||||||
|
return candidates
|
||||||
|
|
||||||
|
# Clause 1.
|
||||||
|
drop: set[int] = set()
|
||||||
|
index = 0
|
||||||
|
while index < len(candidates):
|
||||||
|
candidate = candidates[index]
|
||||||
|
if candidate.rule == RULE_TABLE_BLOCK or not _TRAILING_PAGE_NUMBER.search(candidate.title):
|
||||||
|
index += 1
|
||||||
|
continue
|
||||||
|
end = index
|
||||||
|
while (
|
||||||
|
end < len(candidates)
|
||||||
|
and candidates[end].rule != RULE_TABLE_BLOCK
|
||||||
|
and candidates[end].level == candidate.level
|
||||||
|
and _TRAILING_PAGE_NUMBER.search(candidates[end].title)
|
||||||
|
):
|
||||||
|
end += 1
|
||||||
|
if end - index >= CONTENTS_RUN:
|
||||||
|
drop.update(range(index, end))
|
||||||
|
index = end
|
||||||
|
|
||||||
|
kept = [c for position, c in enumerate(candidates) if position not in drop]
|
||||||
|
if not kept:
|
||||||
|
return kept
|
||||||
|
|
||||||
|
# Clause 2. The unit level is read from DECLARED headings only -- ATX and
|
||||||
|
# dotted-numbered, `RULE_HEADING`. Two exclusions, each measured:
|
||||||
|
#
|
||||||
|
# - a table block carries the sentinel level 9, and letting it vote would
|
||||||
|
# make every document with two tables a two-level document;
|
||||||
|
# - `RULE_OUTLINE` is Arm D's RECOVERY of an integer numbering run, not a
|
||||||
|
# level the document declares. Letting it vote made it the shallowest
|
||||||
|
# repeated level on every PDF that has both, folding the dotted headings
|
||||||
|
# the operator actually named into it: 23 -> 3, 48 -> 4 and 11 -> 7 on
|
||||||
|
# K3 positions 1, 7 and 9. The unit worksheet showed the operator ATX and
|
||||||
|
# dotted headings and nothing else, which is the same set.
|
||||||
|
#
|
||||||
|
# With no declared heading at all the fold has no level to work from, and
|
||||||
|
# `unit_level` is set below every candidate's, so clause 2 is inert and
|
||||||
|
# only clause 3 can fire. That is the honest behaviour: a document whose
|
||||||
|
# structure was recovered rather than declared has no unit level to read.
|
||||||
|
levels = [c.level for c in kept if c.rule == RULE_HEADING]
|
||||||
|
repeated = sorted({level for level in levels if levels.count(level) > 1})
|
||||||
|
if repeated:
|
||||||
|
unit_level = repeated[0]
|
||||||
|
elif levels:
|
||||||
|
unit_level = min(levels)
|
||||||
|
else:
|
||||||
|
unit_level = -1
|
||||||
|
|
||||||
|
folded: list[Candidate] = []
|
||||||
|
for candidate in kept:
|
||||||
|
table = candidate.rule == RULE_TABLE_BLOCK
|
||||||
|
deeper = candidate.rule == RULE_HEADING and candidate.level > unit_level
|
||||||
|
if table and folded:
|
||||||
|
previous = folded[-1]
|
||||||
|
introduces = (
|
||||||
|
previous.rule != RULE_TABLE_BLOCK
|
||||||
|
and previous.end - previous.start < candidate.end - candidate.start
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
introduces = False
|
||||||
|
if (deeper or introduces) and folded:
|
||||||
|
previous = folded[-1]
|
||||||
|
folded[-1] = replace(previous, end=candidate.end)
|
||||||
|
continue
|
||||||
|
folded.append(candidate)
|
||||||
|
return folded
|
||||||
|
|
||||||
|
|
||||||
def _cut_points(text: str, start: int, end: int, cap: int) -> list[int]:
|
def _cut_points(text: str, start: int, end: int, cap: int) -> list[int]:
|
||||||
|
|
@ -646,12 +759,15 @@ def build_plan(
|
||||||
max_segment_chars: int = 0,
|
max_segment_chars: int = 0,
|
||||||
outline_run: int = 0,
|
outline_run: int = 0,
|
||||||
table_grid: bool = False,
|
table_grid: bool = False,
|
||||||
|
unit_fold: bool = False,
|
||||||
) -> dict[str, Any]:
|
) -> dict[str, Any]:
|
||||||
"""The artifact. Every entry PROPOSED, the plan itself never adjudicated."""
|
"""The artifact. Every entry PROPOSED, the plan itself never adjudicated."""
|
||||||
taken: set[str] = set()
|
taken: set[str] = set()
|
||||||
extractor_id = source.suffix.lower().lstrip(".") or "none"
|
extractor_id = source.suffix.lower().lstrip(".") or "none"
|
||||||
entries: list[dict[str, Any]] = []
|
entries: list[dict[str, Any]] = []
|
||||||
candidates = find_candidates(text, outline_run=outline_run, table_grid=table_grid)
|
candidates = find_candidates(
|
||||||
|
text, outline_run=outline_run, table_grid=table_grid, unit_fold=unit_fold
|
||||||
|
)
|
||||||
for candidate in subdivide(text, candidates, max_segment_chars):
|
for candidate in subdivide(text, candidates, max_segment_chars):
|
||||||
entries.append(
|
entries.append(
|
||||||
{
|
{
|
||||||
|
|
@ -720,6 +836,7 @@ def run(
|
||||||
max_segment_chars: int = 0,
|
max_segment_chars: int = 0,
|
||||||
outline_run: int = 0,
|
outline_run: int = 0,
|
||||||
table_grid: bool = False,
|
table_grid: bool = False,
|
||||||
|
unit_fold: bool = False,
|
||||||
) -> int:
|
) -> int:
|
||||||
if max_segment_chars < 0:
|
if max_segment_chars < 0:
|
||||||
raise ProposerError(
|
raise ProposerError(
|
||||||
|
|
@ -771,6 +888,7 @@ def run(
|
||||||
max_segment_chars=max_segment_chars,
|
max_segment_chars=max_segment_chars,
|
||||||
outline_run=outline_run,
|
outline_run=outline_run,
|
||||||
table_grid=table_grid,
|
table_grid=table_grid,
|
||||||
|
unit_fold=unit_fold,
|
||||||
)
|
)
|
||||||
# Nothing to propose is an OUTCOME, and it is not an artifact. An empty
|
# Nothing to propose is an OUTCOME, and it is not an artifact. An empty
|
||||||
# plan cannot be replayed -- `process_inbox` refuses one, because a plan
|
# plan cannot be replayed -- `process_inbox` refuses one, because a plan
|
||||||
|
|
@ -848,6 +966,21 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
|
||||||
"the K3 method file does not name it either"
|
"the K3 method file does not name it either"
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--unit-fold",
|
||||||
|
action="store_true",
|
||||||
|
help=(
|
||||||
|
"Arm F: fold the proposed candidates into the operator's units --"
|
||||||
|
" discard a run of contents-list headings, fold a deeper heading"
|
||||||
|
" into its parent, and fold a table back into the shorter heading"
|
||||||
|
" that introduces it. ONE rule with three clauses, derived from the"
|
||||||
|
" three rules the K3 unit worksheet records (2026-09-08). It adds"
|
||||||
|
" no boundary, so it can only reduce a plan. Absent (the default)"
|
||||||
|
" is OFF and leaves the artifact byte-identical to the arm below"
|
||||||
|
" it. A boolean: the rule's one number, CONTENTS_RUN, is a module"
|
||||||
|
" constant whose sweep is published, not a knob a caller can turn"
|
||||||
|
),
|
||||||
|
)
|
||||||
parser.add_argument(
|
parser.add_argument(
|
||||||
"--table-grid",
|
"--table-grid",
|
||||||
action="store_true",
|
action="store_true",
|
||||||
|
|
@ -884,6 +1017,7 @@ def main(argv: list[str] | None = None) -> int:
|
||||||
max_segment_chars=args.max_segment_chars,
|
max_segment_chars=args.max_segment_chars,
|
||||||
outline_run=args.outline_run,
|
outline_run=args.outline_run,
|
||||||
table_grid=args.table_grid,
|
table_grid=args.table_grid,
|
||||||
|
unit_fold=args.unit_fold,
|
||||||
)
|
)
|
||||||
except ProposerError as exc:
|
except ProposerError as exc:
|
||||||
print(f"{PROPOSER_ID}: FAILED - {exc}", file=sys.stderr)
|
print(f"{PROPOSER_ID}: FAILED - {exc}", file=sys.stderr)
|
||||||
|
|
|
||||||
|
|
@ -1624,3 +1624,223 @@ def test_table_grid_takes_no_argument(tmp_path: Path, capsys: pytest.CaptureFixt
|
||||||
okf_propose_segments.main([str(source), "--out", str(out), "--table-grid", "3"])
|
okf_propose_segments.main([str(source), "--out", str(out), "--table-grid", "3"])
|
||||||
assert exit_info.value.code == 2
|
assert exit_info.value.code == 2
|
||||||
assert "usage:" in capsys.readouterr().err
|
assert "usage:" in capsys.readouterr().err
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Arm F: the unit fold. ONE rule with three clauses, derived from the three
|
||||||
|
# rules the operator wrote in the K3 unit worksheet (2026-09-08). It only ever
|
||||||
|
# MERGES or DISCARDS -- it proposes no boundary of its own -- and it is off
|
||||||
|
# unless a caller asks, so every default stays byte-identical.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# A contents list as the converter actually leaves one: each entry keeps the
|
||||||
|
# dot leaders that carried the page number across the page, so each heading has
|
||||||
|
# a body and the shipped orphan check does NOT already remove it. A fixture
|
||||||
|
# whose contents lines were bodiless would be green before Arm F existed.
|
||||||
|
CONTENTS_THEN_BODY = """# Innhold 1
|
||||||
|
|
||||||
|
Oversikt over kapitlene.
|
||||||
|
|
||||||
|
## Innledning 4
|
||||||
|
|
||||||
|
.................................................
|
||||||
|
|
||||||
|
## Grunnforhold 6
|
||||||
|
|
||||||
|
.................................................
|
||||||
|
|
||||||
|
## Vurdering 9
|
||||||
|
|
||||||
|
.................................................
|
||||||
|
|
||||||
|
## Innledning
|
||||||
|
|
||||||
|
Bakgrunn for arbeidet.
|
||||||
|
|
||||||
|
## Grunnforhold
|
||||||
|
|
||||||
|
Loesmasser og berg.
|
||||||
|
|
||||||
|
## Vurdering
|
||||||
|
|
||||||
|
Konklusjonen staar her.
|
||||||
|
"""
|
||||||
|
|
||||||
|
NESTED_HEADINGS = """## Koordinatsystem
|
||||||
|
|
||||||
|
Prosjektets koordinatsystem er EUREF89.
|
||||||
|
|
||||||
|
### Kartdata og nullpunkt
|
||||||
|
|
||||||
|
Nullpunktet ligger i sydvest.
|
||||||
|
|
||||||
|
## Aksesystem
|
||||||
|
|
||||||
|
Akser nummereres fra A.
|
||||||
|
"""
|
||||||
|
|
||||||
|
INTRO_THEN_TABLE = """## Sjekkliste
|
||||||
|
|
||||||
|
Kryss av for hvert punkt under.
|
||||||
|
|
||||||
|
| Punkt | Svar |
|
||||||
|
|-------|------|
|
||||||
|
| Ett | Ja |
|
||||||
|
| To | Nei |
|
||||||
|
"""
|
||||||
|
|
||||||
|
FLAT_NO_TABLE = """## Om prosjektet
|
||||||
|
|
||||||
|
Prosjektet gjelder en utvidelse.
|
||||||
|
|
||||||
|
## Grunnforhold
|
||||||
|
|
||||||
|
Grunnen er morene.
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_contents_run_is_discarded_and_the_body_survives() -> None:
|
||||||
|
"""Clause 1. The operator's words: "innholdsfortegnelsen er ikke konsepter".
|
||||||
|
|
||||||
|
Both halves over the same text, for the reason the Arm E tests give: the
|
||||||
|
flag-on assertion alone stays green if the DEFAULT moved too.
|
||||||
|
|
||||||
|
A RUN, never a single line. `... i henhold til TEK 17` is a body heading
|
||||||
|
that `_TRAILING_PAGE_NUMBER` reads as a contents line, and discarding it
|
||||||
|
would delete a chapter. Measured on the K3 sample: one such heading at
|
||||||
|
position 1.
|
||||||
|
"""
|
||||||
|
off = okf_propose_segments.find_candidates(CONTENTS_THEN_BODY)
|
||||||
|
assert [c.title for c in off] == [
|
||||||
|
"Innhold 1",
|
||||||
|
"Innledning 4",
|
||||||
|
"Grunnforhold 6",
|
||||||
|
"Vurdering 9",
|
||||||
|
"Innledning",
|
||||||
|
"Grunnforhold",
|
||||||
|
"Vurdering",
|
||||||
|
]
|
||||||
|
on = okf_propose_segments.find_candidates(CONTENTS_THEN_BODY, unit_fold=True)
|
||||||
|
assert [c.title for c in on] == ["Innhold 1", "Innledning", "Grunnforhold", "Vurdering"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_deeper_heading_folds_into_its_parent() -> None:
|
||||||
|
"""Clause 2. The operator's words: "h2-kapittel med sine h3".
|
||||||
|
|
||||||
|
The parent's span must EXTEND over the child, not merely lose it: a fold
|
||||||
|
that dropped the child would delete its body, and a count assertion alone
|
||||||
|
cannot tell the two apart. That is what the `end` assertion pins.
|
||||||
|
"""
|
||||||
|
off = okf_propose_segments.find_candidates(NESTED_HEADINGS)
|
||||||
|
assert [c.title for c in off] == ["Koordinatsystem", "Kartdata og nullpunkt", "Aksesystem"]
|
||||||
|
on = okf_propose_segments.find_candidates(NESTED_HEADINGS, unit_fold=True)
|
||||||
|
assert [c.title for c in on] == ["Koordinatsystem", "Aksesystem"]
|
||||||
|
assert on[0].start == off[0].start
|
||||||
|
assert on[0].end == off[1].end
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_table_folds_back_into_the_heading_that_introduces_it() -> None:
|
||||||
|
"""Clause 3. The operator's words: "tabellen med innledningen".
|
||||||
|
|
||||||
|
Conditioned on the introduction being SHORTER than the table, so a long
|
||||||
|
chapter does not swallow a table that is a unit in its own right. The
|
||||||
|
surviving concept keeps the heading's NAME -- a fold that kept
|
||||||
|
`Tabell linje 5` would merge the right bytes under a name no reader can
|
||||||
|
look up.
|
||||||
|
"""
|
||||||
|
off = okf_propose_segments.find_candidates(INTRO_THEN_TABLE)
|
||||||
|
assert [c.title for c in off] == ["Sjekkliste", "Tabell linje 5"]
|
||||||
|
on = okf_propose_segments.find_candidates(INTRO_THEN_TABLE, unit_fold=True)
|
||||||
|
assert [c.title for c in on] == ["Sjekkliste"]
|
||||||
|
assert on[0].end == off[1].end
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_document_without_a_deeper_heading_or_a_table_is_byte_identical() -> None:
|
||||||
|
"""The known-negative, and it is the control the other three rest on.
|
||||||
|
|
||||||
|
Three clauses that all fire on their own fixture prove each clause runs;
|
||||||
|
they do not prove the rule is SILENT where the operator said nothing. A
|
||||||
|
document with one heading level and no table must come out of the fold
|
||||||
|
unchanged -- identical objects, not merely an equal count.
|
||||||
|
"""
|
||||||
|
off = okf_propose_segments.find_candidates(FLAT_NO_TABLE)
|
||||||
|
on = okf_propose_segments.find_candidates(FLAT_NO_TABLE, unit_fold=True)
|
||||||
|
assert len(off) == 2
|
||||||
|
assert off == on
|
||||||
|
|
||||||
|
|
||||||
|
def test_unit_fold_takes_no_argument(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
|
||||||
|
"""Boolean at the CLI, like Arm E's gate and unlike Arm D's run length.
|
||||||
|
|
||||||
|
The one number in the rule -- how many consecutive page-numbered headings
|
||||||
|
make a contents list -- is a module constant that is swept in the report,
|
||||||
|
not a knob a caller can turn without recording the sweep.
|
||||||
|
"""
|
||||||
|
source = write(tmp_path, FLAT_NO_TABLE, "noarg-f.md")
|
||||||
|
out = tmp_path / "noarg-f.json"
|
||||||
|
with pytest.raises(SystemExit) as exit_info:
|
||||||
|
okf_propose_segments.main([str(source), "--out", str(out), "--unit-fold", "3"])
|
||||||
|
assert exit_info.value.code == 2
|
||||||
|
assert "usage:" in capsys.readouterr().err
|
||||||
|
|
||||||
|
|
||||||
|
OUTLINE_PLUS_DOTTED = """1 Innledning
|
||||||
|
|
||||||
|
Bakgrunn.
|
||||||
|
|
||||||
|
1.1 Formaalet med planen
|
||||||
|
|
||||||
|
Formaalet er beskrevet her.
|
||||||
|
|
||||||
|
1.2 Orientering om prosjektet
|
||||||
|
|
||||||
|
Prosjektet gjelder en utvidelse.
|
||||||
|
|
||||||
|
2 Grunnlag
|
||||||
|
|
||||||
|
Grunnlaget er dette.
|
||||||
|
|
||||||
|
2.1 Loesmasser
|
||||||
|
|
||||||
|
Morene over berg.
|
||||||
|
|
||||||
|
2.2 Berggrunn
|
||||||
|
|
||||||
|
Gneis.
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_recovered_outline_level_does_not_decide_the_unit() -> None:
|
||||||
|
"""Round 2 of Arm F's clause 2, and the reason it exists is a measurement.
|
||||||
|
|
||||||
|
Arm D recovers a document's integer numbering (`1 Innledning`) as a level-1
|
||||||
|
candidate. Letting that level vote makes it the shallowest repeated level,
|
||||||
|
so every dotted `1.1` heading folds into it -- and on the K3 sample that
|
||||||
|
took position 1 from 23 concepts to 3, position 7 from 48 to 4 and position
|
||||||
|
9 from 11 to 7, all in the direction the operator did NOT choose. The unit
|
||||||
|
worksheet showed the operator ATX and dotted-numbered headings only, and
|
||||||
|
those are the levels a document DECLARES. A recovered numbering run is a
|
||||||
|
heuristic, so it does not decide what a unit is.
|
||||||
|
|
||||||
|
Asserted as titles rather than as a count: a fold that kept the right
|
||||||
|
number of concepts by discarding the dotted headings instead of the
|
||||||
|
integer ones would pass a count assertion and be exactly backwards.
|
||||||
|
"""
|
||||||
|
off = okf_propose_segments.find_candidates(OUTLINE_PLUS_DOTTED, outline_run=2)
|
||||||
|
assert [c.title for c in off] == [
|
||||||
|
"Innledning",
|
||||||
|
"Formaalet med planen",
|
||||||
|
"Orientering om prosjektet",
|
||||||
|
"Grunnlag",
|
||||||
|
"Loesmasser",
|
||||||
|
"Berggrunn",
|
||||||
|
]
|
||||||
|
on = okf_propose_segments.find_candidates(OUTLINE_PLUS_DOTTED, outline_run=2, unit_fold=True)
|
||||||
|
assert [c.title for c in on] == [
|
||||||
|
"Innledning",
|
||||||
|
"Formaalet med planen",
|
||||||
|
"Orientering om prosjektet",
|
||||||
|
"Grunnlag",
|
||||||
|
"Loesmasser",
|
||||||
|
"Berggrunn",
|
||||||
|
]
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue