open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

fix(structure): a derived reference needs a cue, not just a number shape

Browse source 
STRUCTURED_V1 derived `references` from every number-shaped token in a body.
A consumer measured 12 false references out of 12 on their corpus, ten of them
version numbers in titles read as document numbers. Measured again here before
changing anything, on two corpora, through `derive_document_structure` itself:

  corpus A  a consumer's normative bundles   2 561 docs -> 2 838 subjects
  corpus B  this repository's own docs/         28 docs ->   559 subjects

The reported class reproduces, and two larger ones they did not report turn up:
702 of corpus A's subjects (24.7 %) are hexadecimal fragments of UUIDs read out
of `id="..."` attributes in embedded MathML, and corpus B lifts tokens out of
escape sequences in quoted source (`\n60` -> `N60`), licence identifiers
(`AGPL-3.0` -> `3.0`) and package pins.

A derived subject is now a bundle-local link target, or a number immediately
preceded by a cue from a closed set. Positive rather than a blacklist because
the data forces it: `V221` is a genuine document number in corpus A and
`V0.3.0` is a software version in corpus B, and they are the same token shape,
so only the words in front of them can tell them apart. The cue matches at a
word boundary (a Norwegian compound ending in `-klasse` otherwise satisfies the
cue `se`, which admitted 86 class designations) and the window is NFC-normalised
so a cue survives a decomposed filesystem. Fragment-only and brace-carrying link
targets go too: neither can name a concept, so neither is a pending pointer.

After: 2 838 -> 1 279 (A) and 559 -> 72 (B). Hand-classified against the
occurrence that actually passed the gate: 30 of 30 sampled genuine on A, 60 of
60 on B. Residual known falses: 9 of B's 72, all illustrative link targets in
prose about link syntax.

The prefix-resolution rule stays open, per the order's condition: of 2 589
documents, 2 562 carry a number and 0 of those are dotted, so unique-prefix
match has no data here to be defended against.

Cost stated rather than hidden: a corpus phrasing cross-references outside this
vocabulary derives nothing and must declare `references`. A missing reference is
visible to the reader; a false one is not.

Record: docs/plan/references-cue-rule.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ATyA5Lx53N1rKzwMnuMoem
This commit is contained in:
Kjell Tore Guttormsen 2026-08-29 09:26:57 +02:00
commit 2827be0ece
4 changed files with 277 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

7
docs/plan/structure-derivation.md
View file

@ -1,5 +1,10 @@
# Structure derivation: numbering, hierarchy, cross-references, supersession
> **Superseded in one place, 2026-08-29 (order `…507632913`).** What makes a
> body mention a `references` subject is now the cue rule measured in
> `docs/plan/references-cue-rule.md`, not "every number mention". Everything
> else below stands.
Answers order `20260826T124341Z-181848459` (from `.claude`, 2026-08-26). The
premise it waited on — the Door B / Door A capability gap — was closed in
`dc9ea59`: the answer was title derivation, not a door change and not a profile
@ -53,7 +58,7 @@ nothing derived can still use everything declared.
| | leading number token of filename, then of title | **derived** |
| `parent` | arithmetic on `number` (drop last dotted part) | inherits `number`'s |
| `references` | producer's `references` key | declared |
| | bundle-local link targets + number mentions in the body | **derived** |
| | bundle-local link targets + cued number mentions in the body | **derived** |
| `supersedes` | producer's `supersedes` key | declared |
| | same number + ordered `version` | **derived** |
| `version`, `status`, `date` | producer only | declared |