docs: the normalisation door, the refused column and the role map
README gains the partial-refusal paragraph and the soft-hyphen door with its measured reach; CHANGELOG gets an Unreleased section over this round; CLAUDE.md's open commitment "R761 is refused whole by guard 1.4.0 ... that is the security repo's question" is replaced by what was decided and built, with the alternatives that were felled and the numbers behind each rule. No version bump and no tag: `v0.10.1` is not tagged yet and releasing is the operator's. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
eebaf534fa
commit
be5fe2baa5
3 changed files with 140 additions and 2 deletions
26
README.md
26
README.md
|
|
@ -434,6 +434,32 @@ declares no heading, paragraph or table, so no witness can count them. An
|
|||
image in a workbook, or in md, txt, csv, json, odt or rtf, stays unaccounted
|
||||
and therefore stays red.
|
||||
|
||||
A run that refused a document whole says so in both places: the accounting
|
||||
carries `refused` and each document's own `status`, and `log.md` carries
|
||||
`R of D document(s) refused whole`. The exit code does not move for it — it
|
||||
belongs to the whole run, and a corpus holding one unreadable file among many
|
||||
is ordinary — so the count is what keeps a partial refusal from being silent.
|
||||
The judge treats such a document as never clean, with its elements in their own
|
||||
`refused` column: every one of them is booked honestly as a coded rejection, so
|
||||
u and d both stay 0 and nothing else could see the loss.
|
||||
|
||||
**The soft hyphen is removed before the persist gate, and counted** (operator
|
||||
decision 2026-09-18). U+00AD is in `llm-ingestion-guard`'s zero-width set, and
|
||||
`output:zero-width-present` is an any-tier carrier: a document carrying one is
|
||||
`fail_secure` at every trust level. Measured on R761 Prosesskoden:2025 — 71
|
||||
U+00AD, and 0 of U+200B, U+200C, U+200D, U+FEFF and U+2060 — those 71 are
|
||||
Norwegian hyphenation points inside words (`ar[SHY]beider`, `bitu[SHY]men`),
|
||||
so a 701-page process code was unreadable for the whole chain over typography.
|
||||
`extract.normalise_extracted` removes that one character from every extracted
|
||||
text and reports the count as `normalised_soft_hyphen`, per document and for
|
||||
the run, in the accounting JSON and in a `**Normalisation**` bullet in
|
||||
`log.md`. The guard is not touched and the other four characters are not
|
||||
touched: they carry no typographic job in running text, so removing one would
|
||||
be a decision about what the guard screens for, taken in the wrong repository.
|
||||
U+00A0 NBSP is not in the guard's set and is not touched either. Reach,
|
||||
measured 2026-09-19: **0 of the 78** readable documents of the reference
|
||||
corpus carry any of the six characters, so no bundle measured here moves.
|
||||
|
||||
Two things hold with or without the flag:
|
||||
|
||||
- `okf build` exits 1 when it extracted at least one document and persisted
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue