docs: the normalisation door, the refused column and the role map

README gains the partial-refusal paragraph and the soft-hyphen door
with its measured reach; CHANGELOG gets an Unreleased section over
this round; CLAUDE.md's open commitment "R761 is refused whole by
guard 1.4.0 ... that is the security repo's question" is replaced by
what was decided and built, with the alternatives that were felled and
the numbers behind each rule.

No version bump and no tag: `v0.10.1` is not tagged yet and releasing
is the operator's.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-19 06:13:06 +02:00
commit be5fe2baa5
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
3 changed files with 140 additions and 2 deletions

View file

@ -1034,8 +1034,70 @@ and fixtures, never code.
column, never also `extractor_unknown`. The files come from
`ExtractedDocument.files` (what the resolver actually resolved), never
from byte equality, which would credit R761's 7 unpointed duplicates.
- **R761 is refused whole by guard 1.4.0 because of its 71 U+00AD soft
hyphens.** That is the security repo's question, and it has been asked.
- **R761 WAS refused whole by guard 1.4.0 because of its 71 U+00AD soft
hyphens, and since 2026-09-19 okf removes them itself.** Operator decision
2026-09-18, the chosen alternative verbatim: **one documented
normalisation door BEFORE the guard**, `extract.normalise_extracted`,
removing U+00AD and BOOKING the count as `normalised_soft_hyphen` per
document and per run, in the accounting JSON and in a `**Normalisation**`
bullet in `log.md`. The guard is untouched -- the two alternatives
(weaken the rule, ship a cleaned corpus) were felled as a weaker screen
for every consumer and a fix for one document. `output:zero-width-present`
is an ANY-TIER carrier, so the verdict was formally right at every trust
level and materially a false positive: the 71 are Norwegian hyphenation
points inside words (`ar[SHY]beider`, `bitu[SHY]men`), and the four real
zero-width characters (U+200B, U+200C, U+200D, U+FEFF) plus U+2060 are
**0 of 0** in both deliveries. Those four are NOT removed and U+00A0 NBSP
(6 633 in R761) is not either -- it is not in the guard's set at all.
Applied ONCE, at the end of `extract_document`, so `propose` and Door B
cannot disagree about the text a plan's `text_sha256` indexes; `_pdf_units`
measures its page offsets through the same door, because that table is
CHARACTER offsets rebuilt from the pages while the text it indexes comes
back normalised. Removing a character removes no newline, so every
line-indexed rule downstream is unmoved by construction. **Reach measured
rather than argued: 0 of the 78** readable documents of the K2 reference
corpus carry any of the six characters, 0 in the pinned bundle's concept
bodies and 0 across `tests/fixtures`, `examples`, `skills`, `docs`, `src`,
README and CHANGELOG -- the door cannot have moved a byte measured here.
- **A DOCUMENT REFUSED WHOLE IS NEVER CLEAN, AND A PARTIAL REFUSAL IS NEVER
SILENT (H1, 2026-09-19).** `refused_whole` asked its question only when a
corpus persisted NOTHING, and a refused document's own unit balances by
construction -- every element booked as a coded rejection is u = 0 and
d = 0. Measured by PM: one refused source beside one accepted one gave
`clean = 4 of 4`, `refused_whole = None`, `okf build` exit 0 and three
elements gone unseen, which is the ORDINARY case on a heterogeneous
corpus. `Unit.refused` is a fifth column and the only one that is not a
defect in the report; the build writes `refused` into the accounting and
`R of D document(s) refused whole` into `log.md`. **The exit code does not
move**: it belongs to the whole run, and a corpus holding one unreadable
file among many is ordinary -- so the order's other half, state it in the
accounting, is the one taken.
- **THE JSON ROLE MAP READS THE PUBLISHER'S OWN TAGS (H3, 2026-09-19).**
`count_sts_json` compared the RAW tag string where `count_sts_xml` has
always used `_local`, so N200 Vegbygging:2024's **74 `mml:math`** (2 811
characters) counted as 0; and the publisher's JSON writes a figure's
caption as `figcaption` under the `graphic`, not as the `fig/caption`
NISO-STS writes, so **49 `figcaption`** counted as 0. The `figcaption`
clause is in the JSON map ALONE -- the two maps are written twice on
purpose so row 5 can see a hole in either. NOT ONE OTHER COUNT MOVES,
measured role by role over the eight JSON deliveries on this machine and
the three committed twins: `figure_caption` 0 -> 123/127/62/49/43/27 and
`math` 0 -> 74/3/88, R761 and N601 unchanged, the two existing twins
unchanged. The known-negative for stripping a prefix is N101's **574
`tbx:`** tags, not one of whose local names is a role.
- **The mutation harness is a GATE since 2026-09-19 (H4)**: a surviving
mutant exits 1, where `2 if errors else 0` made `killed 0 of 1` an exit 0.
PM's **X2** -- "a report may declare a document rejected while the bundle
holds it" -- is in the set and killed by a test driven from both sides;
it survived 98 green tests when it was written. The skipped-row guard
measures the DEFAULT sources and not the arguments (H5: row 6 is SKIPPED
exactly when the arguments' corpora are absent, so the branch was
unreachable), and row 6 now SAYS when a corpus measures no element class
at all (H6: on N200 the build proposes 0 plans and exits 2 before the
accounting door, so 16 549 elements land with no declared fate -- a
finding about the run that read as a finding about the build; what it
would take is an STS reader for the JSON delivery, and `standardContent`
occurs 0 times in `src/`).
- **Operator 2026-09-17:** the door stays OPT-IN until the reference
corpus's losses are fixed, and of the three proposed exceptions only the
PDF one (heading, paragraph, table) is APPROVED. It moves no number --