open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(identity): an STS document's doc-number names its directory and its title the address

Browse source 
K3-19 a. `extract.declared_identity` reads what a NISO-STS document states
about itself -- exactly one <std-ident> (<doc-number>, <year>) and exactly one
<title-wrap> (<full>, else <main>) -- and returns None for every other row,
for XML that is not STS, for an unparseable file, and for a document that
states neither. A value stated more than once is not read: an adopted
standard carries one <std-ident> per issuing body, and picking one is a guess.

`okf build` names a document's directory from its <doc-number> through the id
grammar, replacing only the file's stem. A declared name another document in
the run also claims falls back to the file name for both, said on stderr: the
existing collision gate would refuse both with "rename one", and a name read
from inside a document is not one a rename can change.

`sources[0].title` becomes <doc-number> + <year>, then the <title-wrap>
title, then the file name -- the first that survives the gate and can be
written into the flow mapping verbatim. Measured on R761, <full> carries a
comma, which ends a flow mapping, so it is never the title there; it is never
cleaned up either. `resource` stays the inbox-relative file.

Every other row, and every profile without an address, is untouched: the
identity is asked for only where `sources` is written.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-11 03:08:30 +02:00
commit ee8d5b5776
6 changed files with 230 additions and 13 deletions
Download patch file Download diff file Expand all files Collapse all files

12
CLAUDE.md
View file

@ -120,7 +120,17 @@ one boundary rule:
reachable without inventing an id; round 13's 14 such directories were false
positives of the TEXT route reading the document's own contents listing and
are gone. Report:
`docs/2026-09-10-k3-runde14-deklarert-struktur-tar-ruten.md`. The registries are COUPLED: a row in
`docs/2026-09-10-k3-runde14-deklarert-struktur-tar-ruten.md`.
**Since K3-19 an STS document's own identity names its directory**
(`extract.declared_identity`, read by `cli._document_prefixes` and the door):
the directory was the delivery file's stem, a UUID occurring **0 times** in
the document, while its one `<std-ident>` carried `<doc-number>`. Only the
stem is replaced, and a declared name two documents in one run claim is
used by NEITHER -- the `slug_owners` gate would refuse both with "rename
one", which a name read from inside a document cannot obey. The `sources`
title is `<doc-number>` + `<year>`, then `<title-wrap>`, then the file name:
R761's `<full>` carries a COMMA, a flow terminator, so it is never written
and never cleaned up. The registries are COUPLED: a row in
`_CORE_EXTRACTORS` and not in `segmentation._STDLIB_EXTRACTOR_IDS` refuses
every proposal for the type, two layers away from the extractor.
`pdf`/`docx`/`xlsx` only via

7
README.md
View file

@ -626,6 +626,13 @@ bundle:
`<table-wrap>` becomes one markdown table. Any other XML keeps its text in
document order and gets no invented structure. XML carrying a
`<!DOCTYPE` is refused unparsed.
A NISO-STS document that **states who it is** names its own directory:
`okf build` takes the directory from the document's one `<std-ident>`
`<doc-number>` (reduced to the id grammar) instead of the file's stem, and
the `sources` title from `<doc-number>` + `<year>`, then `<title-wrap>`,
then the file name, whichever is the first that can be written verbatim.
Stated more than once, or claimed by a second document in the same run, a
declared name is not used and the file name stays.
The drop directory is walked **recursively**, in sorted relative-path order:
a file at any depth is ingested and records its path relative to the inbox
root as its `source_file`, while dot-directories and a bundle directory

60
src/llm_ingestion_okf/cli.py
View file

@ -70,12 +70,15 @@ from __future__ import annotations
import argparse
import sys
import tempfile
from collections.abc import Sequence
from functools import partial
from pathlib import Path
from .corpus import LOG_NAME, CorpusReport, load_plans, measure
from .errors import IngestError
from .extract import declared_identity
from .inbox import walk_inbox
from .materialize import reduce_to_id_grammar
from .profiles import SEGMENTED_OKF_V0_2, STRUCTURED_V1, BundleProfile
from .propose import ProposerError, heading_reserve_applies
from .propose import run as propose_run
@ -248,6 +251,60 @@ DEFAULT_PDF_OUTLINE = False
DEFAULT_STAMP = "1970-01-01T00:00:00Z"
def _document_prefixes(inbox: Path, walked: Sequence[Path]) -> dict[Path, str]:
"""Each document's directory: the name it declares, else its file name.
MEASURED: a NISO-STS delivery landed every one of its 2 761 concepts under
a directory named for the delivery path's file name, a UUID occurring 0
times in the document, while the document's own `<doc-number>` said what it
was. Only the file's STEM is replaced; the folders above it are the
operator's arrangement and stay.
A declared name another document in this run also claims -- by declaring
it, or by its file name reducing to it -- is not taken by either, and both
keep their file name. The gate Door B already has would refuse both and
tell the operator to rename one, and a name read from inside a document is
not one a rename can change. Said on stderr rather than silently, because a
directory that stays a UUID is otherwise indistinguishable from this rule
never having run.
"""
def scope(prefix: str) -> str:
return "/".join(reduce_to_id_grammar(part) for part in prefix.split("/"))
named = {source: source.relative_to(inbox).with_suffix("").as_posix() for source in walked}
declared: dict[Path, tuple[str, str]] = {}
for source in walked:
try:
identity = declared_identity(source.name, source.read_bytes())
except OSError:
# The proposer reads the same file next and reports it per file.
continue
if identity is None or identity.doc_number is None:
continue
slug = reduce_to_id_grammar(identity.doc_number)
if slug:
parent = source.relative_to(inbox).parent
declared[source] = ((parent / slug).as_posix(), identity.doc_number)
claims: dict[str, set[Path]] = {}
for source in walked:
claims.setdefault(scope(named[source]), set()).add(source)
for source, (prefix, _) in declared.items():
claims.setdefault(scope(prefix), set()).add(source)
prefixes = dict(named)
for source, (prefix, doc_number) in declared.items():
others = sorted(named[other] for other in claims[scope(prefix)] if other != source)
if others:
print(
f"{CLI_ID}: {named[source]}: <doc-number> {doc_number!r} names {prefix!r}, "
f"which {', '.join(others)} also claims; both keep their file name",
file=sys.stderr,
)
continue
prefixes[source] = prefix
return prefixes
def _propose_plans(
inbox: Path,
bundle: Path,
@ -284,6 +341,7 @@ def _propose_plans(
corpora the two-script path completes.
"""
walked, _ = walk_inbox(inbox, exclude=bundle)
prefixes = _document_prefixes(inbox, walked)
written = nothing = failed = 0
for position, source in enumerate(walked, start=1):
relative = source.relative_to(inbox)
@ -293,7 +351,7 @@ def _propose_plans(
plans_dir / f"{position:02d}.json",
okf_type=okf_type,
proposed_at=proposed_at,
path_prefix=relative.with_suffix("").as_posix(),
path_prefix=prefixes[source],
outline_run=outline_run,
table_grid=table_grid,
unit_fold=unit_fold,

82
src/llm_ingestion_okf/extract.py
View file

@ -610,6 +610,13 @@ def _xml_document(data: bytes) -> tuple[str, tuple[OutlineMark, ...]]:
document type declaration costs nothing here (0 of 1 file carries one) and
holds on every interpreter.
"""
root = _parse_xml(data)
reader = _XmlTextExtractor(sts=_is_sts(root))
return reader.text(root), tuple(reader.marks)
def _parse_xml(data: bytes) -> Element:
"""The one parse, with the DTD refusal in front of it (see `_xml_document`)."""
text = decode_text(data)
prologue = text[: text.find("<", text.find("<") + 1) + 1] if "<" in text else text
if "<!DOCTYPE" in prologue or "<!DOCTYPE" in text[:4096]:
@ -620,14 +627,81 @@ def _xml_document(data: bytes) -> tuple[str, tuple[OutlineMark, ...]]:
code="extractor_xml_doctype",
)
try:
root = ElementTree.fromstring(text)
return ElementTree.fromstring(text)
except ElementTree.ParseError as exc:
raise ExtractionError(
f"the XML parser failed on this file: {exc}", code="extractor_xml_parse_error"
) from exc
sts = _local_name(root.tag) == _STS_ROOT or next(root.iter("sec"), None) is not None
reader = _XmlTextExtractor(sts=sts)
return reader.text(root), tuple(reader.marks)
def _is_sts(root: Element) -> bool:
"""The NAMED schema test: a `<standard>` root, or any `<sec>`."""
return _local_name(root.tag) == _STS_ROOT or next(root.iter("sec"), None) is not None
@dataclass(frozen=True)
class DeclaredIdentity:
"""What a document states about itself, read from its own elements.
Each field is `None` when the document does not state it, and ALSO when it
states it more than once: an adopted standard carries one `<std-ident>` per
body that issued it, and taking the first would be a guess dressed as a
reading. The caller falls back to the file name for whatever is `None`.
"""
doc_number: str | None
year: str | None
title: str | None
def declared_identity(name: str, data: bytes) -> DeclaredIdentity | None:
"""`xml`: the identity a NISO-STS document declares, or `None`.
MEASURED ON THE ONE STS DOCUMENT THIS ROW HAS: exactly one `<std-ident>`
(`<doc-number>R761 Prosesskoden</doc-number>` beside `<year>2025</year>`)
and one `<title-wrap>` whose `<full>` is the document's title -- while the
file carrying it was named for a delivery path, a UUID occurring 0 times in
the document. `<doc-type>` is read by nobody: it said `Innledning` there,
which is the name of a chapter and not a kind of document.
`None` for every other row and for XML that is not STS: a declaration is a
property of a schema, and a text that merely LOOKS like one declares
nothing. An unparseable file is `None` too, never an exception -- extracting
the same bytes refuses it with its own code, and an identity is not the
place a document is refused.
"""
if Path(name).suffix.lower() != ".xml":
return None
try:
root = _parse_xml(data)
except ExtractionError:
return None
if not _is_sts(root):
return None
declared = [
(_child_text(element, "doc-number"), _child_text(element, "year"))
for element in root.iter()
if _local_name(element.tag) == "std-ident"
]
declared = [pair for pair in declared if pair[0]]
doc_number, year = declared[0] if len(declared) == 1 else (None, None)
wraps = [element for element in root.iter() if _local_name(element.tag) == "title-wrap"]
title = (
(_child_text(wraps[0], "full") or _child_text(wraps[0], "main"))
if len(wraps) == 1
else None
)
if doc_number is None and title is None:
return None
return DeclaredIdentity(doc_number=doc_number, year=year, title=title)
def _child_text(element: Element, name: str) -> str | None:
"""A direct child's whole text, whitespace collapsed; `None` when absent or empty."""
for child in element:
if _local_name(child.tag) == name:
return " ".join("".join(child.itertext()).split()) or None
return None
def _extract_xml(data: bytes) -> str:

79
src/llm_ingestion_okf/inbox.py
View file

@ -27,7 +27,7 @@ from dataclasses import dataclass, replace
from pathlib import Path, PurePosixPath
from .errors import IngestError, MaterializationError, SegmentationError, SourceError
from .extract import SourceUnits, extract_text, source_units
from .extract import DeclaredIdentity, SourceUnits, declared_identity, extract_text, source_units
from .materialize import (
_render_root_frontmatter,
check_filename_length,
@ -113,6 +113,7 @@ def render_inbox_concept(
bundle_id: str | None = None,
units: SourceUnits | None = None,
span: tuple[int, int] | None = None,
source_title: str | None = None,
) -> str:
"""Frame extracted text as an inbox concept file with its provenance layer.
@ -127,6 +128,9 @@ def render_inbox_concept(
the text arriving here is the SANITIZED text and its length is not
necessarily the extracted text's.
`source_title` is what the document calls itself, for the `sources`
entry's `title`; `None` keeps the file name there, as before it existed.
`segment` and `bundle_id` carry the 1-to-N identity layer and are read ONLY
when the profile declares the segmentation capability. A concept the plan
does not cover keeps today's rule verbatim, and the four shipped profiles
@ -231,6 +235,7 @@ def render_inbox_concept(
source_file=source_file,
units=units,
span=located,
title=source_title,
)
)
return f"---\n{profile.frontmatter.emit(frontmatter)}\n---\n\n{_normalize_body(text)}"
@ -249,6 +254,7 @@ def _provenance_frontmatter(
source_file: str,
units: SourceUnits | None,
span: tuple[int, int] | None,
title: str | None = None,
) -> dict[str, str]:
"""The address, and the locator when one is available.
@ -265,11 +271,14 @@ def _provenance_frontmatter(
"than mangled",
code="inbox_source_file_unaddressable",
)
values = {
policy.sources_key: (
f"[{{ resource: {source_file}, title: {PurePosixPath(source_file).name} }}]"
if title is not None and not _flow_expressible(title):
raise MaterializationError(
f"source title {title!r} cannot be written into the `sources` flow "
"mapping verbatim; refused rather than mangled",
code="inbox_source_title_unaddressable",
)
}
shown = title if title is not None else PurePosixPath(source_file).name
values = {policy.sources_key: f"[{{ resource: {source_file}, title: {shown} }}]"}
if units is None or span is None:
return values
first, last = units.covering(*span)
@ -288,6 +297,54 @@ def _provenance_frontmatter(
return values
def _flow_expressible(value: str) -> bool:
"""Whether `value` survives as a plain scalar inside a flow mapping."""
return bool(value) and not any(char in value for char in f"{_FLOW_TERMINATORS}\n\r")
def _screened(gate: Gate, value: str | None) -> str | None:
"""A value read from the DOCUMENT and persisted outside its screened body.
The body goes through the gate before anything is written; a frontmatter
value taken from the same bytes would otherwise be the one route around
it. Kept only on the gate's non-blocking floor, as the SANITIZED text, and
dropped rather than refused otherwise: the body carrying the same words is
judged on its own, and a document is never lost over an optional key.
"""
if value is None:
return None
decision = gate(value)
if decision.disposition != _DISPOSITION_PERSIST:
return None
return " ".join(decision.sanitized_text.split()) or None
def _declared_sources_title(identity: DeclaredIdentity | None, gate: Gate) -> str | None:
"""The `sources` title a document declares, or `None` for the file name.
`<doc-number>` + `<year>` first, then the `<title-wrap>` title: measured on
the one STS document this row has, the `<full>` title carries a COMMA,
which ends a flow mapping, and the guard refuses the quoted scalar that
could have carried it. A declared value that cannot be written verbatim
falls to the next layer -- never cleaned up, because a title with its comma
removed is a title the document does not carry.
"""
if identity is None:
return None
candidates: list[str] = []
if identity.doc_number is not None:
candidates.append(
f"{identity.doc_number} {identity.year}" if identity.year else identity.doc_number
)
if identity.title is not None:
candidates.append(identity.title)
for candidate in candidates:
kept = _screened(gate, candidate)
if kept is not None and _flow_expressible(kept):
return kept
return None
# --- the guard seam -------------------------------------------------------
# The guard's non-blocking floor. `Disposition` is a `str, Enum` in
@ -620,6 +677,7 @@ def _render_segments(
bundle_id: str,
source_file: str,
units: SourceUnits | None,
source_title: str | None = None,
) -> BlockedFile | None:
"""Render every segment, or refuse the WHOLE document.
@ -686,6 +744,7 @@ def _render_segments(
segment=entry,
bundle_id=bundle_id,
units=units,
source_title=source_title,
),
decision.reasons,
)
@ -996,6 +1055,14 @@ def process_inbox(
if profile.provenance is not None
else None
)
# What the document says it is, for the address's title. Asked only
# where an address is written, so the four profiles without one do
# not parse anything they would never emit.
source_title = (
_declared_sources_title(declared_identity(source_name(path), source_bytes), gate)
if profile.provenance is not None
else None
)
covering = _plan_covering(plans, source_bytes)
if covering is not None:
blocked = _render_segments(
@ -1009,6 +1076,7 @@ def process_inbox(
bundle_id=(root_frontmatter_values or {})[_bundle_id_key(profile)],
source_file=source_name(path),
units=units,
source_title=source_title,
)
if blocked is not None:
if blocked.disposition == _DISPOSITION_QUARANTINE:
@ -1061,6 +1129,7 @@ def process_inbox(
# gate that removed a character would shift every
# unit boundary after it.
span=(0, len(text)),
source_title=source_title,
),
decision.reasons,
)

3
tests/test_document_identity.py
View file

@ -113,8 +113,7 @@ def test_two_doc_numbers_are_no_doc_number() -> None:
"""
data = IDENTITY.read_bytes().replace(
b"</std-doc-meta>",
b"<std-ident><doc-number>NS 9000</doc-number><year>2020</year></std-ident>"
b"</std-doc-meta>",
b"<std-ident><doc-number>NS 9000</doc-number><year>2020</year></std-ident></std-doc-meta>",
)
identity = extract.declared_identity(IDENTITY.name, data)