llm-ingestion-okf/docs/2026-09-11-k3-runde13-xml-sts.md

# K3 round 13: the file that IS the product had no reader

**Date:** 2026-09-11 · **Commits:** `3f8f03c` (red), `525d68e`, `324cd0b` (red),
`5bb3b04` (red), `567a63d`, `fc23844` (red), `f280b14` · **Base:** `e1f4faa`

## 1. What was measured, and against what

A zip a publisher's own viewer delivers as "Html" holds 0 html, 1 xml and 109
images. `okf build` on it gave **110 of 110 unreadable, 0 plans, exit 2**, with
`no extractor is registered for file extension '.xml'` -- reproduced here
before any code was touched. The conservation identity `merged + coded
rejections == N` was never written at all: the run aborts earlier on `FAILED -
no segmentation plans`, so the hole was not even visible as a count. After the
reader, that same run writes `merged + coded rejections = 110; N = 110`.

The one xml file is the whole product: R761 Prosesskoden:2025, the document
round 12 met as a 701-page PDF, in NISO-STS form. Measured here with
`xml.etree.ElementTree`, reproducing SK1 exactly:

| row | measured |
|---|---|
| `<sec>` total | 7 715 |
| `<sec>` with `<title>` | **2 761** |
| `<sec>` with `<label>` and no `<title>` | 4 954 |
| `<table-wrap>` | 10 |
| `<p>` | 12 528 |
| root element | `<standard>` |
| `<!DOCTYPE` / `<!ENTITY` | 0 / 0 |
| `<sec>`-nesting depth over the 2 761 titled | 28 · 118 · 500 · 1 141 · 868 · 97 · 9 |

That last row is **row for row identical** to the fasit built from the same
file, so the ceiling here is structural rather than computed: **2 761 of
2 761**, against the PDF arm's 2 759 of 2 761.

### Two premises in the material do not hold, and one instrument is uncorrected

- **`<title>` does not carry the number.** SK2's report and `hitk_sk2.py`'s
  docstring say "STS glues the token into `<title>`". Measured: **2 of 2 761**
  `<title>` strings begin with a digit. `<label>` carries `2.1`, `<title>`
  carries `Hovedprosesser`; the fasit's `"2.1Hovedprosesser"` is SK2's own
  fasit-BUILDER joining them and setting `label: null`. The fasit is right, the
  account of where the number comes from is not.
- **One titled section has no label:** `Forord` (`sec-type="foreword"`, depth 1).
- **The shipped instrument carries the round-12 correction in NO block.** The
  order stated that its headline, depth and false-positive rows use the
  corrected key. Measured against `experiments/sk2_grenser.py` as it stands:
  line 86 is `cnorms = ckeys` and the `literal` set is used only on the
  `[ARTEFAKT]` line. The correction is absent everywhere, not just in the
  K3-9/DELMENGDE/FORSIDE blocks. A scratch COPY was corrected in every block
  (headline, depth, false positives, DELMENGDE, FORSIDE, K3-9), and the
  haystack was made a parameter; both forms are reported side by side below.
  The correction is proposed to `vegnormal-okf`, never applied there.

## 2. Step 0: a bookmark that shared a line left no trace

Round 12's arm lost one `/Outlines` node without counting it: **2 763 nodes in,
2 762 marks out, `unresolved` = 0**. Reproduced exactly. The lost node is
`SVV - Forside`, sharing line 0 with the tree's own root node `R761
Prosesskoden`. The cause is structural: marks are collected in a dict keyed on
the destination line index, so `setdefault` discards the second in silence.

**The rule was chosen by measurement.** Keeping both as sibling marks was run
through `find_candidates` on R761's own text: the candidate list goes 2 762 ->
2 763 and the FIRST candidate closes with an **empty span (0, 0)**, which the
orphan check deletes without a word. Keeping both loses the same node one step
later and adds a front-matter title that is in no fasit. So: first in tree
order wins, the loser is COUNTED, and the identity holds -- **2 763 = 2 762 + 0
+ 1** on R761 with the shipped function.

Known-negatives: `outlined-krav.pdf` and `outline-broken-dest.pdf` unchanged in
outcome and reporting `collided == 0`; `~/okf-test/dokumenter` byte-identical
against a frozen `e1f4faa` export; the pinned K2 bundle's concept count and
per-row ranks unchanged.

## 3. The reader

`.xml` is a **core** type beside `.html`, not an `[extract]` one: the parser is
stdlib, and putting it behind the extra would make a pure-stdlib file type
binary-dependent. `defusedxml` and `lxml` occur **0 times** in `uv.lock` before
and after.

The output grammar is MARKDOWN -- the same markdown the office rows and round
11's HTML row reach the proposer through -- and **`propose.py` was not
touched**. `<sec>` with a `<title>` becomes one ATX line carrying `<label>` +
space + `<title>` at its own nesting depth; `<sec>` with only a `<label>`
becomes a body line with the label in front; `<table-wrap>` becomes its label
plus one markdown table through this package's own `render_table`.

Two choices are measurements rather than preferences:

- **Inline by allowlist, block by default** -- the inverse of the HTML reader,
  because block-versus-inline is a property of HTML and XML has no such
  universal. The allowlist is load-bearing: that document carries 1 701
  `<italic>` and 1 396 `<bold>` inside its prose, and breaking on them would
  shred every paragraph.
- **The ATX ceiling is 6 and STS nesting reaches 7**, so the depth is CLIPPED,
  not dropped: 9 of 2 761 sit at depth 7 and `#######` matches nothing. A
  clipped heading still sets its boundary and states its nesting one level too
  shallow; a dropped one loses the section. The fasit's depth row is the
  source's own, so the clipping does not show there -- it shows in frontmatter
  nesting. `tests/test_frontmatter_nesting.py` is unchanged and green.

**A DTD is refused unparsed**, with its own code. Measured on this interpreter
(3.14.0, `pyexpat.version_info` 2.7.3): an external SYSTEM entity is refused by
the stdlib and never fetched, but the amplification limit that stops a
billion-laughs comes from **libexpat >= 2.4.0 and not from Python** -- five
levels still expand -- while `pyproject.toml` requires only `>=3.10` and no
lockfile pins an interpreter. `XMLParser` exposes no `.parser` attribute on the
C accelerator, so the handler route is not portable. Refusing every DTD is a
guarantee about the code instead of about the machine, and it costs nothing
here: 0 of 1 file carries one.

`.xml` never routes through the converter, which is a second parser that would
never see that refusal. The existing fence test says so in one more line and
`test_every_office_row_names_its_reader` is untouched.

## 4. The gate, in both instrument forms

The extractor reaches the ceiling. **The build does not**, and the whole
distance is two proposer rules.

| row | measured | denominator |
|---|---|---|
| ATX heading lines emitted | **2 761** | 2 761 |
| text preserved, non-whitespace | **1 283 395 = 1 283 395, ratio 1.000000** | exact, not a percentage |
| provenance: unit table vs `extract_text` | starts == line offsets, numbers 1..N | 15 271 lines |
| concepts, shipped defaults | **23** | -- |
| boundaries, shipped defaults | **15 of 2 761 (0.5 %)** corrected · 0 of 2 761 pair-only | 2 761 |
| concepts, `--no-unit-fold` | **2 089** | -- |
| boundaries, `--no-unit-fold` | **2 065 of 2 761 (74.8 %)** corrected · 28 of 2 761 pair-only | 2 761 |
| by depth, `--no-unit-fold`, corrected | 15/28 · 98/118 · 430/500 · 935/1141 · 554/868 · 30/97 · 3/9 | 2 761 |
| by depth, pair-only | 0/28 · 2/118 · 5/500 · 10/1141 · 8/868 · 3/97 · 0/9 | 2 761 |
| false positives | **10 of 2 089 (0.5 %)** corrected · 2 061 of 2 089 pair-only | 2 089 |
| lettered points that became concepts | **0 of 4 954** | 4 954 |
| tables as table blocks | **10 of 10** (separator lines emitted) | 10 |
| unnumbered titles | 10 of 22 corrected · 0 of 22 pair-only | 22 |
| directories with two concept files | 14 numbered + the document folder | 2 089 |

**The pair-only column is an instrument artefact, not a segmentation result**,
and the two are separated rather than mixed. The pair key looks up `(concept
directory, normalised residual title)`; this bundle's concept TITLES carry the
numbering token, because `propose._ATX` keeps the whole title and fills
`number` in addition. The corrected form -- accept a match when the concept's
own normalised title equals the fasit's `norm` -- is the one that measures
segmentation here. A row the instrument's key cannot reach is not a boundary
okf failed to set.

**The 10 false positives are the ten table blocks**, named `Tabell linje N`.
The instrument's table regex reports 12 rather than 10 because two data rows
are followed by a row whose cells contain only dashes, which its pattern reads
as a separator; counting the separator lines this reader actually emits gives
**10 of 10**. SK2 measured **0 of 10** on the PDF path, and the PDF bundle
built here reproduces that: 0 table blocks over 0 files.

### The measurement that fells the first attempt, and the second attempt

**Attempt 1 -- the pure `_HTMLTextExtractor` pattern, shipped defaults: 23
concepts, 15 of 2 761.** Two mechanisms, both in `propose`, both decomposed
with a denominator and a control:

1. **The orphan check removes 710 of 2 761 (25.7 %).** Verified as the SOLE
   mechanism rather than assumed: **710 of 710** removed headings are
   immediately followed by another heading line, and **0 of 2 051** delivered
   ones are. This is not the reader losing anything -- 2 735 of the 2 761
   titled sections carry no prose of their own in the SOURCE, and the 710 are
   exactly those whose children are themselves titled sections. `propose.py`'s
   own comment names this outcome: the bookmark arm is exempt because the check
   "judges whether a GUESS was a heading -- a question a publisher's own tree
   has already answered, and one that deletes every container section if
   asked". An STS `<title>` is that same declaration; it just arrives with
   `rule:heading`, indistinguishable from a heading recovered from a converted
   `docx`.
2. **Arm F (`--unit-fold`, a build default since 2026-09-08) removes 2 066
   more, 2 089 -> 23.** Swept one flag at a time: every other default leaves the
   count at 23 or 24, and `unit_fold=False` alone restores 2 089. On a process
   code whose headings are almost all `14.121 Langsgående sikring T1 oppgjort
   etter lengde`, Arm F reads the whole document as a unit sheet. That is the
   arm doing what it was built for, on a document where it is wrong.

**Attempt 2 -- `--no-unit-fold`: 2 089 concepts, 2 065 of 2 761 (74.8 %).** The
gate of 2 761 is NOT reached, and the remaining 696 are the orphan bucket, each
one named in the instrument's output rather than left as a remainder.

The PDF arm reaches 2 762 because `find_candidates` **early-returns** when
`outline_marks` are given: the bookmark route bypasses the orphan check AND Arm
F together. The closing move is therefore to let a declared XML structure take
the route declared PDF structure already takes. That is a `propose.py` change
this round was fenced out of, and it is in DU SKAL BESVARE with the number that
decides it.

## 5. Consumption, and the cross-arm number

hit@k over SK2's six questions, fasit validated as present in the bundle FIRST
(**7 of 7**, on both bundles, independent of key form):

| | hit@1 | hit@8 | hit@50 | KP | KN |
|---|---|---|---|---|---|
| XML bundle (`--no-unit-fold`) | **3/6** | **5/6** | **6/6** | **rank 1** | no rank |
| PDF arm (`--pdf-outline`), same machine | 3/6 | 5/6 | 6/6 | rank 13 | no rank |
| SK2's arm-off reference | 1/6 | 2/6 | 4/6 | -- | -- |

No row is worse. Per row at k=50: S1 4 against 3, S2/S3/S4 1 against 1, S5 11
against 11, S6 3 against 3. **KP goes from rank 13 to rank 1** -- the one row
round 12 left open on the ranking side is closed on this channel, and it is
delivered at the default `k` as well. KN (`prosess 97.4`, fabricated) still
gets no rank on either bundle.

**Cross-arm:** **2 022 concept ids are shared -- 96.8 % of the XML bundle and
73.2 % of the PDF arm's.** Only in XML: 67. Only in PDF: 740, which is the
orphan bucket plus front matter. A named differing pair:
`1-bruksomr-der-for-prosesskoden` (XML) against `1/bruksomr-der-for-prosesskoden`
(PDF) -- the same section, because a bare `1` is not split by `_NUMBERED`
(which requires a dot) while the bookmark arm's `_split_outline_title` does
split it. Normalised TITLES intersect on only 38, for that same reason: the ids
agree and the titles do not.

## 6. Cost, and it is machine-relative

Both arms measured in the SAME session on the same machine, `/usr/bin/time -l`,
whose maximum-resident figure is in **BSD BYTES** -- converted here and said so.

| build | wall | peak RSS | concepts |
|---|---|---|---|
| XML, shipped defaults | 2.10 s | 107 180 032 B = **102.2 MiB** | 23 |
| XML, `--no-unit-fold` | 14.93 s | 114 040 832 B = **108.8 MiB** | 2 089 |
| PDF arm, `--pdf-outline` | 181.52 s | 3 408 691 200 B = **3 251.1 MiB** | 2 762 |

**Direction and order of magnitude, not a number against a number:** the XML
route is about **12x faster and about 30x smaller in peak memory** at a
comparable concept count. Absolute values are not comparable across reports --
round 12's child measured 183.31 s / 3 251 MiB and PM measured 145 s / 3 347
MiB on the same commit; this machine reproduces the child's figures.

## 7. Nothing else moved, and it was measured

- `~/okf-test/dokumenter` **byte-identical** against a frozen `e1f4faa` export,
  `diff -r` empty, 52 md / 26 concepts. That folder holds **0 `.xml` of 5
  files**, so it is a real known-negative and not merely an unchanged number.
- The K2 corpus holds **0 `.xml` of 86 files**. The pinned bundle's
  `EXPECTED_CONCEPTS` and `EXPECTED_RANKS` are unchanged (7 passed), including
  its `--no-tie-shared-rank` known-negative.
- `okf project` still byte-equal to `okf build`, `diff -r` empty.
- `okf skill` + `okf consume` + `okf check --skill … --payload …`:
  **conformant, 15 rules over 8 excerpts and 2 081 withheld entries, 0
  findings**.
- Suite **1567 passed, 1 skipped** (1554 before); `ruff check` and
  `ruff format --check` clean over 116 files; `mypy src` clean over 21 files --
  all four after `git add` and after commit.
- `_EVIDENCE` gains a `.xml` row at **`measured`**, asserted by the suite. The
  class was chosen against the definitions above the table rather than
  assumed: `measured` is "real CORPUS files and a hand-counted fasit -- someone
  wrote the document for their own purposes and we counted what we got", and
  this is a publisher's own published document with the publisher's own
  structure as the fasit, written long before any lookup of ours. It lands in
  the same place as `.html`, whose limit already reduces its 828 files to one
  product from one publisher; the file count is smaller and the fasit is
  stronger, being explicit rather than a generator's cut.

## Honesty limits

- **The denominator is ONE file, from ONE publisher, in ONE schema.** `.xml` as
  a file type is far wider than NISO-STS. 828 HTML files and eight corpus PDFs
  are different denominators and not comparable ones.
- **The fasit is the source's own structure**, so "correct boundary" here means
  what the publisher declared, not what is useful to a reader.
- **The denominator 2 761 is CHOSEN.** The file holds 7 715 `<sec>`; 4 954 of
  them are lettered points inside a process description. Whether those should
  ever be concepts is a different question, answered here as "no" and measured
  at 0 of 4 954.
- **The ceiling is structural, not recovered.** This route reads the
  publisher's claim about its own structure directly, where the PDF arm had to
  bridge from (page, y) to a line index. A high percentage here is therefore
  **not** comparable to an equally high percentage there.
- **A STS document with wrong or stale `<sec>` nesting carries that error
  straight into the segmentation**, and one document says nothing about how
  common that is.
- **The generic fallback is measured on fixtures only**, never on a corpus.
  That half of the row is `constructed` whatever the STS half earned.
- **SK1's and SK2's figures are `vegnormal-okf`'s own documentation** except
  where re-measured here and stated as such.
- **The unit worksheet is n = 12 positions** (8 `pdf`, 3 `docx`, 1 `xlsx`) and
  holds no `xml`, so this row is measured outside it, as `md`/`odt`/`pptx`/
  `html` are.
- **"Delivered" is not "answered."** hit@k says a gold document reached the
  payload, not that the payload answers the question.