open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

feat(description): an STS section's description is its own first spec point

Browse source 
K3-19 c. The NISO-STS reader records, per titled <sec>, the FIRST <p> of its
FIRST direct-child <sec sec-type="spec"> as `OutlineMark.description`. The
plan entry carries it (`description`, only where the source has one, so every
other row's plan keeps its bytes), `parse_segmentation_plan` refuses an empty,
multi-line or non-string value, and the door writes it as the concept's
`description` after the gate has seen it: it is document text persisted
outside the body the gate screens, so it is kept only on the non-blocking
floor and as the sanitized text.

SPEC SS 4.1 makes `description` RECOMMENDED and sets no length, in SS 4.1,
SS 8 or SS 11. The limit is ours and structural -- one paragraph, whole --
because a cut inside it writes a sentence the source never wrote. Measured on
R761: 2 026 of 2 761 titled sections carry a direct-child spec point; the
first <p> runs 17 / 109 / 273 / 521 / 942 characters (min / median / p90 /
p99 / max). A section with none gets no key; nothing is derived from the
title. A stated `--frontmatter description=...` replaces it.

The extracted text does not move: the description is read beside it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-11 03:22:44 +02:00
commit de7849e35b
6 changed files with 96 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

12
src/llm_ingestion_okf/inbox.py
View file

@ -213,6 +213,10 @@ def render_inbox_concept(
frontmatter[policy.offset_key] = _render_flow_list([str(offset) for offset in segment.span])
if segment.parent_id is not None:
frontmatter["parent"] = segment.parent_id
if segment.description is not None:
# The SOURCE's words, carried by the plan. Absent is the source
# saying nothing -- never a summary derived from the title.
frontmatter["description"] = segment.description
if policy.adjudication_key is not None:
# The per-entry verdict IS the discriminator. A plan-level
# `adjudicated: true` with no record for THIS entry leaves it
@ -834,7 +838,13 @@ def _render_segments(
ingested_at=entry.ingested_at,
profile=profile,
structure=structure,
segment=entry,
# The description is document text persisted OUTSIDE the
# body this gate just screened, so it is screened too.
segment=(
replace(entry, description=_screened(gate, entry.description))
if entry.description is not None
else entry
),
bundle_id=bundle_id,
units=units,
source_title=source_title,