docs: the normalisation door, the refused column and the role map
README gains the partial-refusal paragraph and the soft-hyphen door with its measured reach; CHANGELOG gets an Unreleased section over this round; CLAUDE.md's open commitment "R761 is refused whole by guard 1.4.0 ... that is the security repo's question" is replaced by what was decided and built, with the alternatives that were felled and the numbers behind each rule. No version bump and no tag: `v0.10.1` is not tagged yet and releasing is the operator's. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
eebaf534fa
commit
be5fe2baa5
3 changed files with 140 additions and 2 deletions
66
CLAUDE.md
66
CLAUDE.md
|
|
@ -1034,8 +1034,70 @@ and fixtures, never code.
|
|||
column, never also `extractor_unknown`. The files come from
|
||||
`ExtractedDocument.files` (what the resolver actually resolved), never
|
||||
from byte equality, which would credit R761's 7 unpointed duplicates.
|
||||
- **R761 is refused whole by guard 1.4.0 because of its 71 U+00AD soft
|
||||
hyphens.** That is the security repo's question, and it has been asked.
|
||||
- **R761 WAS refused whole by guard 1.4.0 because of its 71 U+00AD soft
|
||||
hyphens, and since 2026-09-19 okf removes them itself.** Operator decision
|
||||
2026-09-18, the chosen alternative verbatim: **one documented
|
||||
normalisation door BEFORE the guard**, `extract.normalise_extracted`,
|
||||
removing U+00AD and BOOKING the count as `normalised_soft_hyphen` per
|
||||
document and per run, in the accounting JSON and in a `**Normalisation**`
|
||||
bullet in `log.md`. The guard is untouched -- the two alternatives
|
||||
(weaken the rule, ship a cleaned corpus) were felled as a weaker screen
|
||||
for every consumer and a fix for one document. `output:zero-width-present`
|
||||
is an ANY-TIER carrier, so the verdict was formally right at every trust
|
||||
level and materially a false positive: the 71 are Norwegian hyphenation
|
||||
points inside words (`ar[SHY]beider`, `bitu[SHY]men`), and the four real
|
||||
zero-width characters (U+200B, U+200C, U+200D, U+FEFF) plus U+2060 are
|
||||
**0 of 0** in both deliveries. Those four are NOT removed and U+00A0 NBSP
|
||||
(6 633 in R761) is not either -- it is not in the guard's set at all.
|
||||
Applied ONCE, at the end of `extract_document`, so `propose` and Door B
|
||||
cannot disagree about the text a plan's `text_sha256` indexes; `_pdf_units`
|
||||
measures its page offsets through the same door, because that table is
|
||||
CHARACTER offsets rebuilt from the pages while the text it indexes comes
|
||||
back normalised. Removing a character removes no newline, so every
|
||||
line-indexed rule downstream is unmoved by construction. **Reach measured
|
||||
rather than argued: 0 of the 78** readable documents of the K2 reference
|
||||
corpus carry any of the six characters, 0 in the pinned bundle's concept
|
||||
bodies and 0 across `tests/fixtures`, `examples`, `skills`, `docs`, `src`,
|
||||
README and CHANGELOG -- the door cannot have moved a byte measured here.
|
||||
- **A DOCUMENT REFUSED WHOLE IS NEVER CLEAN, AND A PARTIAL REFUSAL IS NEVER
|
||||
SILENT (H1, 2026-09-19).** `refused_whole` asked its question only when a
|
||||
corpus persisted NOTHING, and a refused document's own unit balances by
|
||||
construction -- every element booked as a coded rejection is u = 0 and
|
||||
d = 0. Measured by PM: one refused source beside one accepted one gave
|
||||
`clean = 4 of 4`, `refused_whole = None`, `okf build` exit 0 and three
|
||||
elements gone unseen, which is the ORDINARY case on a heterogeneous
|
||||
corpus. `Unit.refused` is a fifth column and the only one that is not a
|
||||
defect in the report; the build writes `refused` into the accounting and
|
||||
`R of D document(s) refused whole` into `log.md`. **The exit code does not
|
||||
move**: it belongs to the whole run, and a corpus holding one unreadable
|
||||
file among many is ordinary -- so the order's other half, state it in the
|
||||
accounting, is the one taken.
|
||||
- **THE JSON ROLE MAP READS THE PUBLISHER'S OWN TAGS (H3, 2026-09-19).**
|
||||
`count_sts_json` compared the RAW tag string where `count_sts_xml` has
|
||||
always used `_local`, so N200 Vegbygging:2024's **74 `mml:math`** (2 811
|
||||
characters) counted as 0; and the publisher's JSON writes a figure's
|
||||
caption as `figcaption` under the `graphic`, not as the `fig/caption`
|
||||
NISO-STS writes, so **49 `figcaption`** counted as 0. The `figcaption`
|
||||
clause is in the JSON map ALONE -- the two maps are written twice on
|
||||
purpose so row 5 can see a hole in either. NOT ONE OTHER COUNT MOVES,
|
||||
measured role by role over the eight JSON deliveries on this machine and
|
||||
the three committed twins: `figure_caption` 0 -> 123/127/62/49/43/27 and
|
||||
`math` 0 -> 74/3/88, R761 and N601 unchanged, the two existing twins
|
||||
unchanged. The known-negative for stripping a prefix is N101's **574
|
||||
`tbx:`** tags, not one of whose local names is a role.
|
||||
- **The mutation harness is a GATE since 2026-09-19 (H4)**: a surviving
|
||||
mutant exits 1, where `2 if errors else 0` made `killed 0 of 1` an exit 0.
|
||||
PM's **X2** -- "a report may declare a document rejected while the bundle
|
||||
holds it" -- is in the set and killed by a test driven from both sides;
|
||||
it survived 98 green tests when it was written. The skipped-row guard
|
||||
measures the DEFAULT sources and not the arguments (H5: row 6 is SKIPPED
|
||||
exactly when the arguments' corpora are absent, so the branch was
|
||||
unreachable), and row 6 now SAYS when a corpus measures no element class
|
||||
at all (H6: on N200 the build proposes 0 plans and exits 2 before the
|
||||
accounting door, so 16 549 elements land with no declared fate -- a
|
||||
finding about the run that read as a finding about the build; what it
|
||||
would take is an STS reader for the JSON delivery, and `standardContent`
|
||||
occurs 0 times in `src/`).
|
||||
- **Operator 2026-09-17:** the door stays OPT-IN until the reference
|
||||
corpus's losses are fixed, and of the three proposed exceptions only the
|
||||
PDF one (heading, paragraph, table) is APPROVED. It moves no number --
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue