open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

fix(structure): a derived reference needs a cue, not just a number shape

Browse source 
STRUCTURED_V1 derived `references` from every number-shaped token in a body.
A consumer measured 12 false references out of 12 on their corpus, ten of them
version numbers in titles read as document numbers. Measured again here before
changing anything, on two corpora, through `derive_document_structure` itself:

  corpus A  a consumer's normative bundles   2 561 docs -> 2 838 subjects
  corpus B  this repository's own docs/         28 docs ->   559 subjects

The reported class reproduces, and two larger ones they did not report turn up:
702 of corpus A's subjects (24.7 %) are hexadecimal fragments of UUIDs read out
of `id="..."` attributes in embedded MathML, and corpus B lifts tokens out of
escape sequences in quoted source (`\n60` -> `N60`), licence identifiers
(`AGPL-3.0` -> `3.0`) and package pins.

A derived subject is now a bundle-local link target, or a number immediately
preceded by a cue from a closed set. Positive rather than a blacklist because
the data forces it: `V221` is a genuine document number in corpus A and
`V0.3.0` is a software version in corpus B, and they are the same token shape,
so only the words in front of them can tell them apart. The cue matches at a
word boundary (a Norwegian compound ending in `-klasse` otherwise satisfies the
cue `se`, which admitted 86 class designations) and the window is NFC-normalised
so a cue survives a decomposed filesystem. Fragment-only and brace-carrying link
targets go too: neither can name a concept, so neither is a pending pointer.

After: 2 838 -> 1 279 (A) and 559 -> 72 (B). Hand-classified against the
occurrence that actually passed the gate: 30 of 30 sampled genuine on A, 60 of
60 on B. Residual known falses: 9 of B's 72, all illustrative link targets in
prose about link syntax.

The prefix-resolution rule stays open, per the order's condition: of 2 589
documents, 2 562 carry a number and 0 of those are dotted, so unique-prefix
match has no data here to be defended against.

Cost stated rather than hidden: a corpus phrasing cross-references outside this
vocabulary derives nothing and must declare `references`. A missing reference is
visible to the reader; a false one is not.

Record: docs/plan/references-cue-rule.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ATyA5Lx53N1rKzwMnuMoem
This commit is contained in:
Kjell Tore Guttormsen 2026-08-29 09:26:57 +02:00
commit 2827be0ece
4 changed files with 277 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

120
docs/plan/references-cue-rule.md Normal file
View file

@ -0,0 +1,120 @@
# What a derived `references` subject is
Record of the decision taken 2026-08-29 on order `…507632913`, and of the
measurement it rests on. Supersedes the reference half of
`docs/plan/structure-derivation.md`; nothing else in that document moves.
## The defect
`STRUCTURED_V1` derived `references` by taking every number-shaped token in a
document body. A consumer measured it on their own corpus and reported 12 false
references out of 12, ten of them version numbers carried in titles and read as
document numbers.
The order was not to inherit that figure. It was measured again here, on two
corpora this repository can reach, before anything was changed.
## Measured before the change
Both figures come from `derive_document_structure` itself, run over every
unique document (deduplicated by sha256) and counting the subjects it returns
with `references` marked derived.
| corpus | documents | documents with derived `references` | subjects |
|---|---|---|---|
| A — a consumer's normative bundle set, six bundles, Norwegian | 2 561 | 1 010 | 2 838 |
| B — this repository's own `docs/`, English and Norwegian technical prose | 28 | 27 | 559 |
What those subjects were, classified against the position the scanner actually
lifted each one from:
- **Corpus A.** 702 of 2 838 (24.7 %) were hexadecimal fragments of UUIDs read
out of `id="…"` attribute values inside embedded HTML and MathML. The rest of
the false population was unit symbols (`kN/m2` reduced to `M2`), table row
and class labels, and formula numbers.
- **Corpus B.** Dominated by software versions (`v0.3.0`, `3.14.0`, `0.11.10`,
`1.28.2`), a licence identifier (`AGPL-3.0` → `3.0`), a container format
version (`%PDF-1.6` → `1.6`), a hash algorithm name (`SHA256`), and tokens
lifted out of two-character escape sequences inside quoted source (`\n60` →
`N60`).
The consumer's reported class reproduces. Two classes they did not report —
identifier fragments inside markup, and escape-sequence artefacts — are larger.
## The decision
**A derived reference subject is one of exactly two things: a bundle-local
markdown link target, or a number immediately preceded by a reference cue from
a closed set.** Everything else number-shaped in body prose is not a reference.
The rule is positive rather than a blacklist, and that is forced by the data,
not chosen for taste: corpus A proves `V221` and `V240` are genuine document
numbers, corpus B proves `V0.3.0` is a software version, and the two are the
same token shape. Nothing structural separates them. Only the words in front of
them do, so only a positive cue can be defended.
The cue must match at a word boundary. Without that guard a Norwegian compound
ending in `-klasse` satisfies the cue `se`, which admitted 86 class
designations on corpus A. The window is normalised to NFC first, so a cue
carrying a Norwegian letter matches whether the file arrived composed or
decomposed.
Two link targets are also excluded, on the same ground rather than a new one: a
fragment-only target (`#anchor`) points inside the document it appears in, and
a target carrying a brace is a template placeholder from prose *about* links.
Neither can name a concept — `reduce_to_id_grammar` emits lowercase
alphanumerics and hyphens — so neither is a pointer awaiting resolution. It is
a claim that cannot come true.
## Measured after the change
Same script, same corpora, `HEAD` against the working tree.
| corpus | subjects before | subjects after | dropped |
|---|---|---|---|
| A | 2 838 | 1 279 | 1 559 (54.9 %) |
| B | 559 | 72 | 487 (87.1 %) |
Precision was checked by hand against the occurrence that actually passed the
gate, not the subject's first occurrence in the file — reading the first one
mis-attributes context and was a real error in an earlier pass of this
measurement.
- Corpus A: 30 subjects drawn at random from 1 279. **30 of 30 genuine.**
- Corpus B: all 60 cue-gated subjects. **60 of 60 genuine.**
**Residual known false positives: 9 of corpus B's 72**, all in the link
channel, all illustrative targets in prose about link syntax (`link`, `target`,
`/sti`, `subdir/`). They are indistinguishable from real relative paths without
guessing at path semantics, so they stay. Corpus A's link channel is empty, so
this residue is not visible there at all.
## The cost, stated rather than hidden
A corpus whose cross-references are phrased outside this vocabulary — a third
language, a house style — derives **nothing**, and must declare `references`
itself. That is the intended failure: a missing reference is visible to the
reader, a false one is not.
Two recall losses inside the corpora measured, both accepted: a reference whose
cue is a preposition (`i henhold til …` where the noun is absent) and one
written as a prefixed compound (`N-V240`) are dropped.
## What was deliberately not done
**The prefix-resolution rule (`N130` in a body not finding a document numbered
`N130.2`) stays open.** The order allowed it in the same pass only if the data
bore it. It does not: of 2 589 documents across both corpora, 2 562 carry a
number and **0** of those numbers are dotted, so a unique-prefix match has
nothing here to be defended against. Closing two things badly is worse than
closing one well.
## Adjacent defect found, not fixed — needs its own order
The line-oriented frontmatter parser lets a **nested key overwrite a top-level
key of the same name**. A `sources:` block whose entry carries its own `title:`
replaces the document's `title`, and the `number` derived from that title
follows it. This is the same family as the known block-list finding but strictly
worse: that one drops a value silently, this one *substitutes* one. It changes
`title`, `number` and `parent`, so it is outside this order's scope, and it is
why corpus A's numbers above are what they are.

7
docs/plan/structure-derivation.md
View file

@ -1,5 +1,10 @@
# Structure derivation: numbering, hierarchy, cross-references, supersession
> **Superseded in one place, 2026-08-29 (order `…507632913`).** What makes a
> body mention a `references` subject is now the cue rule measured in
> `docs/plan/references-cue-rule.md`, not "every number mention". Everything
> else below stands.
Answers order `20260826T124341Z-181848459` (from `.claude`, 2026-08-26). The
premise it waited on — the Door B / Door A capability gap — was closed in
`dc9ea59`: the answer was title derivation, not a door change and not a profile
@ -53,7 +58,7 @@ nothing derived can still use everything declared.
| | leading number token of filename, then of title | **derived** |
| `parent` | arithmetic on `number` (drop last dotted part) | inherits `number`'s |
| `references` | producer's `references` key | declared |
| | bundle-local link targets + number mentions in the body | **derived** |
| | bundle-local link targets + cued number mentions in the body | **derived** |
| `supersedes` | producer's `supersedes` key | declared |
| | same number + ordered `version` | **derived** |
| `version`, `status`, `date` | producer only | declared |

69
src/llm_ingestion_okf/structure.py
View file

@ -53,6 +53,43 @@ _NUMBER_ANYWHERE = re.compile(rf"(?<![\w.])({_NUMBER})")
# this library's job.
_LINK = re.compile(r"\[[^\]]*\]\(([^)\s]+)\)")
# What makes a number token in running prose a POINTER rather than a quantity.
#
# The rule is positive — a closed set of cues — and not a blacklist, because a
# blacklist cannot work: measured 2026-08-29, `V221` and `V240` are genuine
# document numbers in one real corpus and `v0.3.0` is a software version in
# another, and the two are the same token shape. Nothing structural separates
# them; only the words in front of them do. Scanning every number instead
# produced, on those two corpora, hexadecimal fragments of UUIDs lifted from
# inside markup attributes, unit symbols (`kN/m2` -> `M2`), table row labels,
# formula numbers, licence identifiers (`AGPL-3.0` -> `3.0`) and package pins.
# Each of those stands in the index as a relation this library asserted, under
# the producer's name. A reference the reader never sees is a smaller harm
# than one the reader cannot tell from a fact.
#
# The consequence is stated rather than hidden: a corpus whose cross-references
# are phrased outside this vocabulary — another language, a house style — gets
# NOTHING derived, and must declare `references` itself. Silence is the honest
# failure; a guess dressed as a relation is not.
#
# The leading guard is a word boundary that also covers the Norwegian letters
# `\w` handles but `[a-z]` does not. Without it `belysningsklasse C5` ends in
# the cue `se`, which admitted 86 class designations on the corpus measured.
_CUE = re.compile(
r"(?:(?<![^\W\d_])(?:"
r"kapittel|kapitlene|kapitler|avsnitt|punkt|pkt|vedlegg|tabell|tabellen|"
r"figur|figuren|krav|h\u00e5ndbok|jf|iht|nr|se|ogs\u00e5|henhold til|"
r"chapter|section|clause|appendix|annex|paragraph|table|figure|requirement|"
r"handbook|cf|see|also"
r")\.?|\u00a7+)\s*$",
re.IGNORECASE,
)
# How far back a cue is looked for. A cue sits immediately before its number,
# so the window only has to be long enough for the longest cue plus the
# whitespace and punctuation that may follow it.
_CUE_WINDOW = 24
# The fields this module can infer. Named as a constant because `derived` is a
# contract with the consumer, not an implementation detail.
DERIVABLE_FIELDS = frozenset({"title", "number", "parent", "references"})
@ -146,11 +183,19 @@ def _parent_of(number: str) -> str | None:
def _scan_references(body: str, offset: int, own_number: str | None) -> tuple[str, ...]:
"""Bundle-local reference subjects, ordered by first appearance.
Two kinds of pointer, and nothing else. A markdown link is an explicit one
the producer authored, so it is taken as written. A number in running prose
is taken only when a :data:`_CUE` stands immediately in front of it —
everything else number-shaped is a version, a measurement, a label or an
identifier fragment, and emitting it puts a relation this library invented
into the index under the producer's name.
Link targets are collected first and their spans masked with spaces before
the number scan runs, so a link to `n500.md` yields the link target once
rather than the target plus a phantom `N500` read out of the URL. Masking
with spaces rather than deleting keeps every later offset aligned, which is
what makes "first appearance" a property of the original text.
what makes "first appearance" a property of the original text — and it is
also what keeps a cue from being read across a link it does not precede.
"""
found: list[tuple[int, str]] = []
masked = list(body)
@ -159,12 +204,28 @@ def _scan_references(body: str, offset: int, own_number: str | None) -> tuple[st
start, end = match.span(1)
for position in range(start, end):
masked[position] = " "
if target.startswith(("http://", "https://", "//", "mailto:")):
# A fragment-only target points inside THIS document, and a target
# carrying a brace is a template placeholder from prose ABOUT links
# (`reduce_to_id_grammar` cannot emit a brace). Neither can ever
# resolve to another concept, so carrying them states a relation that
# cannot exist rather than one not dropped yet.
if target.startswith(("http://", "https://", "//", "mailto:", "#")):
continue
if "{" in target or "}" in target:
continue
found.append((offset + start, target))
for match in _NUMBER_ANYWHERE.finditer("".join(masked)):
scanned = "".join(masked)
for match in _NUMBER_ANYWHERE.finditer(scanned):
number = _normalize_number(match.group(1))
if number != own_number:
if number == own_number:
continue
# NFC first: a cue carrying a Norwegian letter arrives decomposed from
# a macOS filesystem, and a cue that matches only one of the two forms
# is a rule that holds or not depending on where the file was written.
window = unicodedata.normalize(
"NFC", scanned[max(0, match.start() - _CUE_WINDOW) : match.start()]
)
if _CUE.search(window):
found.append((offset + match.start(), number))
ordered: list[str] = []

87
tests/test_structure.py
View file

@ -168,7 +168,9 @@ def test_a_document_never_references_itself() -> None:
def test_references_are_deduplicated_and_ordered_by_first_appearance() -> None:
doc = derive("N300 then N200 then N300 again.\n", "n500-x.md")
# Each mention carries a cue, because a number without one is no longer a
# reference at all. What this test pins is the ordering and the dedup.
doc = derive("se N300 then se N200 then se N300 again.\n", "n500-x.md")
assert doc.references == ("N300", "N200")
@ -370,3 +372,86 @@ def test_edges_are_deterministically_ordered() -> None:
doc("inbox-a", "See N300.\n", "n100-a.md"),
)
assert list(bundle.edges) == sorted(bundle.edges, key=lambda e: (e.source, e.kind, e.subject))
# --- what a derived reference IS ------------------------------------------
#
# Measured 2026-08-29 against two real corpora, because the previous rule --
# "every number-shaped token in the body" -- was reported producing 12 false
# references out of 12 on a third. Denominators, before the rule below:
#
# corpus A 2 561 unique normative concept documents -> 2 838 subjects, of
# which 702 (24.7 %) were hexadecimal fragments of UUIDs lifted out
# of HTML/MathML attribute values, plus unit symbols (`kN/m2` ->
# `M2`), table row labels and formula numbers;
# corpus B 28 of this repository's own documents -> 559 subjects, dominated
# by software versions (`v0.3.0`, `3.14.0`, `0.11.10`), a licence
# id (`AGPL-3.0` -> `3.0`), a container version (`%PDF-1.6` ->
# `1.6`), a hash algorithm name (`SHA256`), and tokens lifted out
# of escape sequences in quoted source (`\n60` -> `N60`).
#
# The version class cannot be excluded structurally: corpus A proves `V221` and
# `V240` are genuine document numbers and corpus B proves `V0.3.0` is a
# software version, and they are the same token shape. So no blacklist can
# work, and a positive cue is the only rule that can be defended against data.
# A missing reference is visible to the reader; a false one is not.
def test_a_package_version_is_not_a_reference() -> None:
doc = derive("Built on pdfplumber 0.11.10 and Python 3.14.0, licence AGPL-3.0.\n", "n500-x.md")
assert doc.references == ()
def test_a_v_prefixed_version_is_not_a_reference() -> None:
# The reported failure class: a version string carried in a title, read as
# a document number. `V221` in corpus A is a real document; `v0.5.0` here
# is not, and only the cue in front of it can tell them apart.
doc = derive("The classifier lags v0.3.0 and the gap is wider now (v0.5.0a2).\n", "n500-x.md")
assert doc.references == ()
def test_a_number_inside_markup_is_not_a_reference() -> None:
doc = derive('<math id="id-647a093b-a70f-4e38-d277-b139040226a4">x</math>\n', "n500-x.md")
assert doc.references == ()
def test_a_measurement_is_not_a_reference() -> None:
doc = derive("Compressive strength at least 500 kN/m2, used only below 2.5 m.\n", "n500-x.md")
assert doc.references == ()
def test_a_cue_word_makes_a_number_a_reference() -> None:
# KNOWN-POSITIVE. Without it the two tests above pass for the wrong reason:
# a rule that finds nothing at all also finds nothing false.
doc = derive("Laid to depth as se kapittel 2.6.2.2 and tabell 4.8.4-3 give.\n", "n500-x.md")
assert doc.references == ("2.6.2.2", "4.8.4")
assert "references" in doc.derived
def test_a_section_sign_is_a_cue() -> None:
doc = derive("Emitted as one block, in §10.2's own listing order.\n", "n500-x.md")
assert doc.references == ("10.2",)
def test_a_cue_only_counts_at_a_word_boundary() -> None:
# A Norwegian compound ending in `-klasse` ends in the cue `se`. Matching
# an unanchored cue admitted 86 such class designations on corpus A, none
# of which name a document.
doc = derive("Se også R210; lysklasse C5 gjelder.\n", "n500-x.md")
assert doc.references == ("R210",)
def test_a_fragment_only_link_target_is_not_a_reference() -> None:
# An anchor points inside THIS document. It can never resolve to another
# concept, so emitting it states a relation that cannot exist.
doc = derive("See [table pairing](#table-pairing).\n", "n500-x.md")
assert doc.references == ()
def test_a_template_placeholder_link_target_is_not_a_reference() -> None:
# Documentation about link SYNTAX carries example targets. A brace cannot
# occur in a concept name (`reduce_to_id_grammar` emits lowercase
# alphanumerics and hyphens), so such a target can never resolve — and an
# edge that can never resolve is a claim, not a pending pointer.
doc = derive("Managed lines read `- [title](ingest-{id}.md)`.\n", "n500-x.md")
assert doc.references == ()