llm-ingestion-okf/docs/2026-09-19-regnskapsgaten-rest-og-normaliseringsdoren.md

The accounting gate's remainder, and one normalisation door

llm-ingestion-okf, 2026-09-19. Order 20260918T110211Z-9941518407, raised from PM's re-measurement of ed8d9d7 (verdict: PARTIAL). Six holes, each with a red test written in its own commit, plus the operator's decision of 2026-09-18 about U+00AD.

Nothing is tagged, pushed or released here.

1. What the round closes

#	Hole	Closed by	Red first
H1	a partly refused corpus is "clean"	Unit.refused + Accounting.refused	5d1060d -> d27ca50
H2	one of B-1's six refusals held by no test	mutant X2 + a two-sided test	a59898e -> a94b0c5
H3	math/figure_caption do not fire on a real delivery	_local on the JSON tag + a figcaption clause	37599a3 -> d31ab73
H4	the mutation harness is not a gate	verdict(survived, errors)	6abe4b4 -> f5b263f
H5	the skipped-row guard is unreachable	it measures the DEFAULT sources	253c4fb -> 8b4f409
H6	N200 contributes one blank red	measures_no_class in row 6	e6d9f16 -> 280d6bd
—	R761 refused whole over 71 soft hyphens	extract.normalise_extracted	68cfa90 -> eebaf53

2. The normalisation door (operator decision 2026-09-18)

llm-ingestion-guard 1.4.0 keeps U+00AD in _ZERO_WIDTH_CPS beside U+200B, U+200C, U+200D and U+FEFF, and output:zero-width-present is an any-tier carrier: fail_secure at every trust level, with no sanitisation and no exception. R761 Prosesskoden:2025 carries 71 U+00AD and 0 of the four real zero-width characters (U+2060 also 0). All 71 are Norwegian hyphenation points inside words -- ar[SHY]beider, bitu[SHY]men, asfalt[SHY]betong -- so a 701-page process code was unreadable for the whole chain over typography. The verdict was formally right and materially a false positive.

Of the three ways out, the operator chose one door in okf, before the guard, that removes the character and BOOKS the number. The other two were felled: weakening the guard's rule is the security repository's call and takes a screen away from every consumer; delivering a cleaned corpus fixes one document and no other.

Where it is. extract.normalise_extracted(text) -> (text, removed), applied once at the end of extract_document, so both entry points and both sides of a plan's text_sha256 see one string. _pdf_units measures its page offsets through the same door: that table is CHARACTER offsets rebuilt from the pages while the text it indexes comes back normalised, and a table built against the other rendering names the wrong page with full confidence. Removing a character removes no newline, so every line-indexed rule downstream -- the proposer's grammars, xml_outline, pdf_outline's per-page line check -- is unmoved by construction.

What it says. ExtractedDocument.soft_hyphens carries the count out, InboxResult.normalised carries it per document, and account_run reads it off the RUN rather than recounting the source -- a second count would be a second reader. It reaches the accounting JSON as normalised_soft_hyphen at both levels and log.md as a **Normalisation** bullet.

What it does not touch. The guard. The other four characters -- they carry no typographic job in running text, so removing one would be a decision about what the guard screens for, taken in the wrong repository. U+00A0 NBSP, which is not in the guard's set at all (6 633 of them in R761).

Measured, on R761's own delivery, through the DEFAULT gate:

exit 0, wall 25.2 s
run normalised_soft_hyphen: 71
refused: 0  unaccounted: 0  double_booked: 0
  doc 14ce59dc-....xml  status persisted  normalised_soft_hyphen 71
LOG: * **Accounting**: 1 document(s) and 109 other file(s); 31096 elements
     found in the sources; 0 unaccounted, 0 double-booked; 0 of 1 document(s)
     refused whole.
LOG: * **Normalisation**: 71 soft hyphen(s) (U+00AD) removed from 1 of 1
     document(s) before the persist gate. No other character is touched.

Reach, measured rather than argued. 0 of the 78 readable documents of the K2 reference corpus carry U+00AD or any of the five other characters (the 8 unreadable ones raise before extraction, so the denominator is 78 of 86); 0 in the pinned K2 bundle's concept bodies; U+00AD is 0 of the 230 readable tracked files across tests/fixtures, examples, skills, docs, src, README and CHANGELOG. The door cannot have moved a byte anyone here has measured.

Corrected 2026-09-19. That last clause first read "0 across" for all SIX characters, and that is false: 2 of the 230 carry U+200B -- tests/fixtures/accounting/rejected/avvist.html and its inventory twin, this repo's own known-negative for a carrier the door refuses to remove. For U+00AD, the character the door actually removes, the figure is 0, and that is the one the reach argument needs.

Known-negative, in the suite. A source carrying U+200B is still fail_secure, still exits 1, and the character is still in the extracted text. Text preservation is stated as an EXACT invariant and never a share: the extracted text of the hyphenated source EQUALS the extracted text of the same source written without the hyphens.

3. H1 -- a document refused whole is never clean

A refused document books every element as a coded rejection, so u = 0 and d = 0 and the unit balances by construction; refused_whole asks its question only for a corpus that persisted NOTHING. PM measured the ordinary case: one refused source beside one accepted one gave clean = 4 of 4, refused_whole = None, okf build exit 0 and three elements gone unseen.

Unit.refused is a fifth column and the only one that is not a defect in the report. Row 3's reason carries N element(s) lost with R of D document(s) refused whole, and each unclean unit's detail line carries refused= with the document's own rejection code in the note.

On the build side, Accounting.refused goes into the JSON and into log.md. The exit code does not move: it belongs to the whole run, and a corpus holding one unreadable file among many is ordinary. The order allowed either, and this is the half that does not break every build that passes today.

test_a_corpus_refused_whole_under_the_default_gate_is_red kept its point and lost its premise: it now asserts u = 0 and d = 0 rather than a clean unit.

4. H3 -- the role map reads the publisher's own tags

Two independent causes, one line each.

• count_sts_xml takes every tag through _local, which strips a Clark-notation namespace AND a prefix. count_sts_json compared the raw string, so mml:math reached tag == "math" on nothing: 74 formulas in N200, 2 811 characters, counted as 0.
• The publisher's JSON writes a figure's caption as figcaption under the graphic, not as the fig/caption NISO-STS writes: 49 in N200, counted as 0.

The figcaption clause is in the JSON map ALONE. The two maps are written twice on purpose so row 5 can see a hole in either, and the XML delivery has never been measured carrying one.

No other count moves, measured role by role over the eight JSON deliveries on this machine and the three committed twins:

delivery	before -> after
N100-2022	figure_caption 0 -> 123
N100-2023	figure_caption 0 -> 127
N101-2025	figure_caption 0 -> 62
N200-2024	figure_caption 0 -> 49, math 0 -> 74
N303-2026	figure_caption 0 -> 43, math 0 -> 3
N500-2024	figure_caption 0 -> 27, math 0 -> 88
N601-2022	no change
R761-2025	no change (row 5's pair)
the two existing twins	no change

The known-negative for stripping a prefix is N101's 568 tbx: tags: not one of their local names is a role here, and it is in the suite.

5. H2, H4, H5, H6

H2. PM's X2 mutant -- delete the refusal of a report that declares a document rejected while the bundle holds a concept from it -- is now in tools/okf_gate_mutants.py. Reproduced before the test: 98 passed, rc 0, the pattern applied exactly once. The test drives it from both sides, so a gate refusing EVERY rejected document would not pass either.

H4. verdict(survived, errors): 2 when a mutant could not be applied (it was never measured, and that outranks everything), 1 when one survived, 0 otherwise. The run ended 2 if errors else 0, so killed 0 of 1 was an exit 0.

The harness caught this round's own change. Its first run reported killed 34 of 35 and ERROR: M21 ... pattern occurs 0 times -- NOT MEASURED: H1 had rewritten the clean property M21 mutates. M21 is repaired, and X3 -- "a document refused whole is clean again" -- is added beside it, because M21 removes unverified, invalid AND refused at once and would be killed by any one of the three. Final run: killed 36 of 36, 0 survived, 0 errors, exit 0.

H5. The skipped-row guard measures real_corpora(R761_DEFAULT, N200_DEFAULT). Row 6 is SKIPPED exactly when the corpora the ARGUMENTS name are absent, so asking the arguments made the branch unreachable. Two tests, so a guard that fires on everything fails too: with the default source present and the arguments pointing elsewhere the run exits 1 and names the row; with no corpus on the machine it exits 0.

H6. measures_no_class(units) returns a sentence when EVERY document of a corpus has no declared fate, and row 6 prints it beside that corpus's tally. Reproduced 2026-09-19: okf build on N200 proposes 0 plans, prints FAILED - no segmentation plans and exits 2 BEFORE the accounting door -- no accounting file is written at all. What it would take is a capability and not a threshold: .json is read as generic JSON, and the publisher's STS delivery in that form would have to reach the markdown grammar _extract_xml writes for the XML one. standardContent occurs 0 times in src/ and 4 in the witness, with the query validated against that known positive.

6. The gate, after

python3 tools/okf_accounting_gate.py -> exit 1, GATE RED: rows 2, 3, 6.

row	k of M	status	moved this round
1 file types with a fasit fixture	13 of 13	GREEN	no
2 source inventory before build	8 of 13	RED	no
3 accounting after build	17 of 22	RED	no
4 a rejected document is reported honestly	1 of 1	GREEN	no
5 two witnesses agree	4 of 4	GREEN	no
6 real corpora	110 of 111	RED	the reason did

Rows 2 and 3 are the reference corpus's own losses, which the operator decided on 2026-09-17 stay red until they are fixed.

Row 6 keeps its count and changes its cause. R761 now passes the DEFAULT gate: exit 0, 1 of 1 document(s) persisted, 50 asset file(s); 31 081 carried element(s) found in the bundle, 0 claimed and not found, u = 0, d = 0 under both gates. The only refused whole left is N200, and that is the build exiting 2 on 0 plans -- H6's blank, now stated in the row.

7. Limits of this round

• Rows 2 and 3 are untouched. Nothing here fixes a loss in the reference corpus; the round closes the instrument's holes and one refusal.
• The 43-document reference corpus was not rebuilt. The exposure argument for the normalisation door is a census over the corpus's extracted text (0 of 78) and over the pinned bundle's bytes (0), not a rebuild.
• Red first is per commit and per hole, not per assertion. H2's red is the mutant surviving, not a failing test: the check it protects already worked, and a test for working code cannot be red.
• test_r761s_own_seventy_one_soft_hyphens... skips where the corpus is absent. The committed twin fixture is what holds the rule elsewhere.
• The witness is still a second implementation of the same definitions. A definition wrong for a format is wrong on both sides at once, and the gate says so in its own output.