open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(propose,extract,cli): a title that ends in a number, and a converter's own anchor in a concept id

Browse source 
Round 9: the four rests in STATE's NESTE that needed no operator decision.

CLAUSE 1 CLASSIFIED BY THE NUMBER, NOT THE TITLE. `_TRAILING_PAGE_NUMBER`
admitted a candidate into a contents run by asking whether the title ended in
an integer -- a question about the number. A drawing's dimension chain, a
schematic's labels, a door schedule, a coordinate column and a soil-layer
table all end in integers and name nothing. Measured over the 43-document
corpus: 68 candidates discarded over 11 of 39 readable documents, of which
19 over 5 documents are data rows.

That corrects round 8's own decomposition. Its "four misclassified numeric
tables and seven real contents listings" needs each document on one side, and
two of the eleven are both. Read across all 68 titles rather than the
three-title sample: 5 documents carry a data row, 8 carry a real entry.

`--contents-name` requires a NAME to survive stripping the page number. The
threshold is SWEPT, not chosen, and collapses at both ends: at an alphabetic
run of 1 a door schedule keeps a stray `V` and 13 of 19 are rescued; at 3 the
two-letter section name `VA` stops being a name, falls out of run membership,
and takes `RIB`, `MMI` and `Tittelfelt` below `CONTENTS_RUN` with it -- one
acronym costing four REAL entries. At 2: 16 of 19 rescued, 0 of 49 regressed.
The three not rescued carry a real word and are named rather than rounded off.

THE CONVERTER'S ANCHOR WAS IN THE CONCEPT ID. Pandoc writes a sheet as
`## <name> {#sheet-N}` and a titled slide as `## <title> {#slide-N}`. Because
a filename is reduced FROM the title, the anchor reached both. Operator
authorised the strip 2026-09-09 after the exposure was counted: 2 of 810
concepts on the previous default bundle, 2 of 1108 on Arm B, 1 of 26 on the
operator's folder. Two ids renamed, one of which `portfolio-optimiser` has
cited in writing; both are in the report so that message can be sent.

One rule in one function, read by BOTH title-forming sites -- a rule in only
one would leave the id and the title naming the same concept differently. The
known-negative is the point: `Mal for {kundenavn}` is a title an author wrote.

odt/rtf/pptx MEASURED END TO END FOR THE FIRST TIME, on hand-built documents,
because the corpus denominator is genuinely zero (86 files: 66 pdf, 10 docx,
4 xlsx, 2 zip, 2 smc, 2 doc). `_EVIDENCE` gains a third class rather than
stretching an existing one: `constructed` means the row has met a document,
but not one anyone wrote for their own purposes. odt 1 of 1 declared headings;
pptx 2 of 2 on a deck that declares slide titles and 0 of 2 on one that does
not -- round 7's reading of pptx was a fixture property, not the format; rtf
0 segments, because the container has no heading style and the author's title
is bold text. rtf is the one open finding.

ACCEPTANCE, all four. The 12-position reference is label-identical in BOTH
readings (pdf 7/8, docx 3/3, xlsx 0/1 or 1/1, sheet 10/12 or 11/12). One K2
bundle carrying both changes: 453 concepts / 865 md, hit@8 [1,1,1,1,1,None]
on it AND on Arm B, with the known-negative still reproducing on the new
bytes. `okf project` byte-equal to `okf build`, `diff -r` empty. Consumer
cost is a re-run: 436/832 -> 453/865, digest 21af4a1aa98315cf.

Three published numbers corrected: README's 596 tests (1515), README's "15
concepts out" for `okf project` (that was the O6 defect; it is 26), and O6's
print-mode method, which does not reproduce without --allowedTools.

Report: docs/2026-09-09-k3-runde9-restene.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-09 21:45:08 +02:00
commit 6ff18fd703
15 changed files with 964 additions and 67 deletions
Download patch file Download diff file Expand all files Collapse all files

327
docs/2026-09-09-k3-runde9-restene.md Normal file
View file

@ -0,0 +1,327 @@
# K3 round 9: the rests that did not need the operator
Four things stood in `STATE.md`'s NESTE that no operator decision blocked:
clause 1 misclassifying numeric tables, the `{#…}` converter attribute leaking
into concept titles, the first end-to-end measurement of `odt`/`rtf`/`pptx`,
and the prefix overmatch. This round is those four, plus two documentation
findings PM's re-measurement of O6 produced.
Baseline: HEAD `fbad5b0`, tree clean, tag `v0.7.0` on `1260fac` (two commits
back, deliberately — O5/O6's install measurement is of those bytes), remote
`main` at `b01492b`, so five commits unpushed plus the tag. All four
reproduced before any code was touched.
**Every fraction below carries its denominator. Where a published number did
not reproduce, this report says so before it says anything else.**
## 0 What did not reproduce
**Round 8's decomposition of clause 1 splits the eleven documents into "four
misclassified numeric tables and seven real contents listings". Two of the
eleven are BOTH.** Re-measured with round 8's own instrument
(`/tmp/k3r8/clause1.py`, unchanged), the reach reproduces exactly — **68
candidates over 11 of 39 readable documents** — but reading all 68 dropped
titles rather than the three-title sample changes the split:
| | documents | candidates |
|---|---|---|
| carry at least one misclassified data row | **5** of 11 | **19** of 68 |
| carry at least one real contents entry | **8** of 11 | **49** of 68 |
Documents 21 and 22 appear in both rows. The 4 + 7 = 11 reading is only
available if each document is assigned to one side, and two of them cannot be.
The four purely-misclassified documents are a drawing's dimension chain
(`477 3 025 1`), a P&ID's schematic labels (`40.23`), a door schedule
(`3000 V 1`) and a borehole log's coordinate column (`619649.097`); document 22
additionally carries three soil-layer rows (`- 8.225m Fjell 41`) inside an
otherwise real contents listing.
**The README published 596 tests, measured 2026-08-21 and never updated.** The
suite is 1515. A count is a measurement with a date on it.
**The README published "5 documents in, 15 concepts out" for `okf project`.**
15 was the O6 defect, not the result; the same command writes 26. Corrected.
## 1 Clause 1: a title that ends in a number is not thereby a contents entry
`_TRAILING_PAGE_NUMBER` admits a candidate into a contents run by asking
whether the title ENDS in an integer. That is a question about the number, not
about the title. A drawing's dimension chain, a schematic's labels, a door
schedule and a coordinate column all end in integers and name nothing.
The damage was already gone: round 8's `--close-span-gaps` carries a removed
mark's text on the mark above, so the corpus coverage gap is 0 either way
(re-measured: **0 characters of 1 739 511**, both before and after this round).
What remained was classification.
### 1.1 The rule, and where the threshold sits
A title is admitted only if a NAME survives stripping the page number. What
counts as a name is **swept, not chosen** — an alphabetic run of at least *n*
characters, over the 43-document corpus:
| n | candidates dropped | documents | data rows rescued | REAL entries wrongly rescued |
|---|---|---|---|---|
| — (shipped) | 68 | 11 | 0 of 19 | 0 of 49 |
| 1 | 55 | 9 | **13** of 19 | 0 of 49 |
| **2** | **52** | **8** | **16** of 19 | **0** of 49 |
| 3 | 48 | 8 | 16 of 19 | **4** of 49 |
It collapses at both ends. At n = 1 a door schedule keeps a stray single letter
(`3000 V`) and stays misclassified. At n = 3 the two-letter section name `VA`
stops being a name, falls out of run membership, and the run around it drops
below `CONTENTS_RUN` — so `RIB`, `MMI` and `Tittelfelt` are emitted as concepts
too. One two-letter acronym costs four real entries. **n = 2.**
**The three it does not rescue are named rather than rounded off.** Document
22's `- 8.225m Fjell 41` rows carry a real word, so this predicate cannot
separate them from a contents entry. 16 of 19, not 19 of 19.
### 1.2 What it costs
Corpus, extracted text, default flags: **429 → 447 candidates**, documents with
a plan **32 → 33** (one document had all three of its candidates discarded and
so had no plan at all), characters in no segment **0 → 0**.
On the operator's five-document folder the flag changes **nothing** — `diff -r`
between a build with `--no-contents-name` and one without is empty. Those five
documents contain no run of data rows. A flag whose measured effect on one
folder is zero is still the right default; it is measured where the shape
occurs.
### 1.3 The acceptance criteria, all four
**(i) The red test is green on both sides.** A run of five coordinate rows is
no longer discarded; a contents listing whose shortest entry is the two-letter
`VA` still is. `tests/test_contents_name_part.py`.
**(ii) The twelve-position reference loses no cell, in EITHER reading.**
Scored with round 8's instrument imported unchanged (which imports round 5's
criterion unchanged), round 8's shipped default against round 9's:
| reading | `pdf` | `docx` | `xlsx` | sheet | labels identical |
|---|---|---|---|---|---|
| fasit as authored, `|F|`[3] = 11 | 7 of 8 | 3 of 3 | 0 of 1 | 10 of 12 | **yes** |
| `xlsx` fasit read as 12 (PM's working answer) | 7 of 8 | 3 of 3 | 1 of 1 | 11 of 12 | **yes** |
Not one of the 12 positions changes label. Position 0 stays `for grovt` with
an undefined `|F|`, which is the operator decision this round does not touch.
**(iii) No fasit row loses rank 1, on either bundle.** One K2 bundle built
carrying both of this round's shipped changes
(`--bundle-id k2-trinn1-20260903 --okf-version 0.2`, default flags, default
epoch stamp), 825 s wall time, `merged + coded rejections = 43`, 39 of 43
substantive, 4 coded rejections (1 `extractor_empty_pdf`, 3
`extractor_unknown`):
| bundle | concepts | md | `tie_shared_rank` ON | OFF (known-negative) |
|---|---|---|---|---|
| round 9 (new default) | **453** | 865 | **`[1,1,1,1,1,None]`** | `[2,1,1,1,1,–]` |
| previous default `…-20260911` | 436 | 832 | `[1,1,1,1,1,None]` | `[2,1,1,1,1,–]` |
| Arm B `…-20260903` | 629 | 1108 | `[1,1,1,1,1,None]` | `[1,1,1,1,1,–]` |
hit@8 is **5 of 6** on all three. The known-negative still reproduces on the
NEW bytes, so the pin continues to name its own cause rather than being green
for an unstated reason.
**(iv) `okf project` is still byte-equal to `okf build`.** `diff -r` over the
operator's five-document folder after every change: **0 differences**. The new
flag's `cli.build` signature default and its argparse default are both `True`,
which is the check O6's defect made mandatory.
### 1.4 What it costs a consumer
A re-run, and it is not small. The reference corpus goes 629 concepts / 1108
files (2026-09-03) → 492 / 944 → 425 / 810 → 436 / 832 → **453 / 865**.
Digest, from inside the bundle, `find . -type f -print0 | sort -z | xargs -0
shasum -a 256 | shasum -a 256`: **`21af4a1aa98315cf…`** (the previous default
was `8c93e5e3222577a2…`). On the operator's five-document folder the count is
unchanged at 26 concepts / 52 md.
The pinned artefact is `~/corpora/okf-telling-20260829/K2-bundle-default-20260912`.
It was **copied** from the build scratch rather than rebuilt a second time: the
pin's own comment records that two independent builds differ in nothing,
including `log.md`, which carries the corpus path and never the bundle's own.
`--contents-name`, ON by default since 2026-09-09, opt-out `--no-contents-name`.
## 2 The converter attribute in concept titles
Pandoc writes a spreadsheet's sheet as `## <name> {#sheet-<n>}` and a slide
with a title placeholder as `## <title> {#slide-<n>}`. The brace block is
pandoc's own anchor. It reached the artifact twice: into `title`, and from
there into the concept id, because a filename is reduced FROM the title.
**Exposure, measured 2026-09-09 with a known-positive control run first:**
| bundle / folder | concepts | concepts carrying the attribute |
|---|---|---|
| `K2-bundle-default-20260911` | 436 | **2** |
| `K2-bundle-20260903` (Arm B) | 629 | **2** |
| `~/okf-test/dokumenter` | 26 | **1** |
The "2 of 52" figure carried into this round counted FILES containing `{#`
anywhere, one of which was an `index.md`. At concept level it is 1 of 26.
**The renamed ids, before and after, in full:**
| before | after |
|---|---|
| `del-ii-bilag-7-prisskjema/prissammenstilling-sheet-1` | `del-ii-bilag-7-prisskjema/prissammenstilling` |
| `del-ii-bilag-0-dokumentliste-del-ii/ark1-sheet-1` | `del-ii-bilag-0-dokumentliste-del-ii/ark1` |
**The first of those two is the id `portfolio-optimiser` has cited in
writing.** This round does not send that message: the rename is reported here
so PM can.
The strip lives in one function (`extract.strip_converter_attribute`) read by
both title-forming sites — `propose` names a segment from an ATX heading,
`structure` derives a document title from its leading heading — because a rule
in only one of them would leave the id and the title naming the same concept
differently.
**Provenance holds, measured rather than assumed.** The renamed concept keeps
`source_file`, `source_sha256`, `sources`, `source_sheet: Prissammenstilling`
(the extractor's unit table still reads the anchor, which is why the strip is
applied to a title downstream and never to the extracted text), `references`,
and the parent index entry points at the new filename. `okf check`:
**conformant, 15 rules over 8 excerpts and 18 withheld entries, 0 findings.**
The known-negative is the point: `Mal for {kundenavn}` and
`Feltet {"id": 4} i nyttelasten` are titles an author wrote and are untouched.
## 3 `odt`, `rtf` and `pptx`, measured end to end for the first time
The corpus denominator is genuinely zero, re-counted: `K2/trinn1` holds 86
files — **66 `pdf`, 10 `docx`, 4 `xlsx`, 2 `zip`, 2 `smc`, 2 `doc`** — and not
one `odt`, `rtf` or `pptx`. `K1` is empty. So these rows are measured on
hand-built documents, and the report says so in every row.
| type | declared headings recovered | concepts | chars in no segment | N |
|---|---|---|---|---|
| `odt` | **1 of 1** | 1 | 0 of 1366 | 1 document |
| `pptx` (deck declaring slide titles) | **2 of 2** | 2 | 0 | 1 deck |
| `pptx` (deck declaring none) | **0 of 2** | 2 | 0 of 1285 | 1 deck |
| `rtf` | 0 of **0 declared** | **0** | 1368 of 1368 | 1 document |
End to end through Door B, all three containers, **0 of 3 rejected**.
**`pptx` is not broken, and round 7's reading of it was a fixture property.**
The committed fixture's title shapes carry no `<p:ph type="title"/>`
placeholder, so the converter has no title to name a slide with and writes
`Slide 1` / `Slide 2`. Given a deck that declares one, it writes the authored
title plus `{#slide-N}` — and § 2's strip removes the anchor, so the concept is
named by the title the author wrote. Pinned in
`tests/test_converter_attribute_titles.py`, with the anchor's presence asserted
first so the test cannot be green over an empty set.
**`rtf` is the one open finding.** The container has no heading style; the
author's title is bold text. The proposer proposes nothing, and the document
reaches the bundle inbox as one whole concept — content preserved, structure
zero. That is not data loss and it is not segmentation either.
`extract._EVIDENCE` gains a third class rather than stretching an existing one.
`measured` means real corpus files and a hand-counted fasit; `unmeasured` means
no document has ever been through the row; **`constructed`** means the row has
met a document end to end, but not one anyone wrote for their own purposes. All
three office rows move `unmeasured` → `constructed`. The suite asserts the new
class, and asserts the `unmeasured` set is now empty rather than dropping it.
## 4 The prefix overmatch, re-measured
Round 5's `df 159 of 270` was measured on the N500 bundle and has not been
re-measured since. It is not reproduced here — that bundle is a different one —
but the mechanism is, on both K2 bundles, **with a known-positive control run
first because a query that cannot find would report zero either way**:
| bundle | N | `df(undersjøisk)` prefix rule | `df(undersjøisk)` equality | control `df(under)` |
|---|---|---|---|---|
| `K2-bundle-default-20260911` | 436 | **171** | 0 | 79 |
| `K2-bundle-20260903` | 629 | **174** | 0 | 76 |
A token that occurs **zero** times in either bundle matches 39.2 % and 27.7 %
of their concepts through `MIN_SHARED_PREFIX = 4`. The first attempt at this
measurement returned 0 for the control as well, which is how the broken query
was caught rather than published.
Not fixed this round, and not because it is small: the acceptance bar is zero
regression on hit@8 across six rows and two bundles plus the twelve-position
reference under both readings, and `MIN_SHARED_PREFIX` has a known-positive in
the suite that Norwegian compounding is the reason for. It stands, with this
number.
## 5 Two documentation findings from PM's O6 re-measurement
**(a) The print-mode method as published does not reproduce.** O6 § 6 gave the
method as `cd <project> && claude -p "<the question>"`. Run that way — including
with `--permission-mode acceptEdits` — the `okf consume` call is refused, the
model answers without opening the bundle, and a hypothesis run marks both
premises `undecidable-from-bundle`. The four runs reported in fact used
`claude -p --allowedTools=Bash,Read,Grep,Glob "<question>"` (plus `Write,Edit`
for the document row). The numbers are unaffected; the recipe for getting them
was wrong. Corrected in § 6 of that report and added to the README as its own
short paragraph for agents running non-interactively.
**(b) Green lint hangs on the lockfile.** `uv sync --frozen` resolves ruff
0.15.22 and the tree is clean. A loose install resolves ruff 0.16.6, under
which the same tree reports 144 lint errors and 2 files that would be
reformatted — new rules, not new defects. Not upgraded here. One sentence added
to the README's developer section. **One thing to add to PM's finding:**
`uv sync --frozen` alone also drops the `[extract]` extra, after which
`mypy src` fails with `Cannot find implementation or library stub for module
named "pdfplumber"`. The green command is
`uv sync --frozen --extra extract`.
## 6 A finding reported, not fixed: the `tools/okf_consume.py` shim
`vegnormal-okf` reported that the v0.7.0 shim broke a caller using
`importlib.util.spec_from_file_location`. Reproduced here, with numbers: such a
caller holds a module object carrying **4** public names against the
`sys.modules` entry's **90**, and `build_payload` is absent. `sys.modules[__name__]
= _impl` replaces the registry entry; it cannot reach a module object the
caller already holds.
A backward-compatible repair is one line placed before the alias:
```python
globals().update({k: v for k, v in vars(_impl).items() if not k.startswith("__")})
```
Verified to restore all 90 names on the caller-held object. **The dunder filter
is load-bearing**: an unfiltered `vars(_impl)` overwrites `__name__` with
`llm_ingestion_okf.consume` before the next line reads it, aliasing the module
under the wrong key. Note the repair restores attribute ACCESS and not
patch-through — a caller monkeypatching the copy patches a binding the
implementation does not read, which is the reason the alias exists.
**Not applied.** This round's order fences `tools/okf_consume*.py` as run, not
edited. The mechanism and the line are here so a session whose order authorises
it can apply them in one step.
## Honesty limits
- **The twelve-position reference is n = 12, of which 8 are `pdf`, 3 `docx`
and 1 `xlsx`.** Every per-type fraction is a fraction of 8, 3 or 1. A single
`xlsx` position means that row moves between 0/1 and 1/1 and nothing else.
- **Both sheet readings are reported, never one.** The sheet is 11 of 12 under
`|F|` = 12 and 10 of 12 under `|F|` = 11; the difference is entirely the
`xlsx` cell.
- **The retrieval side is n = 6 questions, one rater, one fasit**, and row 6
misses on every bundle measured so far. `hit@8 5 of 6` is that.
- **"General" rests on ONE corpus of 43 documents**, plus the operator's five.
`K2/trinn2` is byte-identical to `K2/trinn1` and `K1` is empty, so the 86
files are 43 documents.
- **Each new file-type row rests on ONE hand-built document** (`pptx` on two
decks). They are constructed documents, not corpus files, and the evidence
class says so.
- **`delivered` is not `answered`.** Every count here is of concepts and
excerpts a payload carries, never of correct answers.
- **Position 0 is blocked on an operator decision, not on code.** So are the
`xlsx` fasit's `|F|` = 12, the `pptx` fasit, and whether routing between
bundles may rest on `--id`. None of the four moved this round.
- **The renamed concept ids break a citation `portfolio-optimiser` made in
writing.** Both ids are in § 2 so that message can be sent; it was not sent
from here.
- **The threshold sweep in § 1.1 is over one corpus.** n = 2 is where it
collapses on these 43 documents; a corpus with one-letter section names would
put it elsewhere.

16
docs/2026-09-09-o6-forsteskjerm-og-skillmodi.md
View file

@ -233,9 +233,19 @@ was run, because the measurement is of the tag.
## 6 The three end-to-end runs (C1's instrument)
Skills installed in a project directory outside this repository;
`cd <project> && claude -p "<the question>"`; **one run per row, no re-run for a
better answer**. Sources grep-verified against the bundle tree with the
denominator stated.
`cd <project> && claude -p --allowedTools=Bash,Read,Grep,Glob "<the question>"`
(row 3, which produces a document, additionally `Write,Edit`); **one run per
row, no re-run for a better answer**. Sources grep-verified against the bundle
tree with the denominator stated.
**CORRECTED 2026-09-09 (round 9).** This section first stated the method as
`cd <project> && claude -p "<the question>"`, without the tool list. Re-run
that way -- including with `--permission-mode acceptEdits` -- the `okf consume`
call is refused, the model answers without ever opening the bundle, and a
hypothesis run marks both premises `undecidable-from-bundle`. All four runs
reported below in fact used the explicit `--allowedTools` list above; the
method line omitted it, so as written the section did not reproduce. The
NUMBERS are unaffected -- what was wrong was the recipe for getting them.
| # | Mode | Asked | Sources verified | Denominator |
|---|---|---|---|---|