docs: the normalisation door, the refused column and the role map

README gains the partial-refusal paragraph and the soft-hyphen door
with its measured reach; CHANGELOG gets an Unreleased section over
this round; CLAUDE.md's open commitment "R761 is refused whole by
guard 1.4.0 ... that is the security repo's question" is replaced by
what was decided and built, with the alternatives that were felled and
the numbers behind each rule.

No version bump and no tag: `v0.10.1` is not tagged yet and releasing
is the operator's.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-19 06:13:06 +02:00
commit be5fe2baa5
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
3 changed files with 140 additions and 2 deletions

View file

@ -434,6 +434,32 @@ declares no heading, paragraph or table, so no witness can count them. An
image in a workbook, or in md, txt, csv, json, odt or rtf, stays unaccounted
and therefore stays red.
A run that refused a document whole says so in both places: the accounting
carries `refused` and each document's own `status`, and `log.md` carries
`R of D document(s) refused whole`. The exit code does not move for it — it
belongs to the whole run, and a corpus holding one unreadable file among many
is ordinary — so the count is what keeps a partial refusal from being silent.
The judge treats such a document as never clean, with its elements in their own
`refused` column: every one of them is booked honestly as a coded rejection, so
u and d both stay 0 and nothing else could see the loss.
**The soft hyphen is removed before the persist gate, and counted** (operator
decision 2026-09-18). U+00AD is in `llm-ingestion-guard`'s zero-width set, and
`output:zero-width-present` is an any-tier carrier: a document carrying one is
`fail_secure` at every trust level. Measured on R761 Prosesskoden:2025 — 71
U+00AD, and 0 of U+200B, U+200C, U+200D, U+FEFF and U+2060 — those 71 are
Norwegian hyphenation points inside words (`ar[SHY]beider`, `bitu[SHY]men`),
so a 701-page process code was unreadable for the whole chain over typography.
`extract.normalise_extracted` removes that one character from every extracted
text and reports the count as `normalised_soft_hyphen`, per document and for
the run, in the accounting JSON and in a `**Normalisation**` bullet in
`log.md`. The guard is not touched and the other four characters are not
touched: they carry no typographic job in running text, so removing one would
be a decision about what the guard screens for, taken in the wrong repository.
U+00A0 NBSP is not in the guard's set and is not touched either. Reach,
measured 2026-09-19: **0 of the 78** readable documents of the reference
corpus carry any of the six characters, so no bundle measured here moves.
Two things hold with or without the flag:
- `okf build` exits 1 when it extracted at least one document and persisted