feat(propose): --shell-parent points a heading-only section at the ancestor holding its text
A plan entry whose span holds its heading alone gets `parent_id` naming the nearest preceding entry at a smaller level whose own span holds text, passing over an ancestor that is empty too; the door writes the existing `parent:` key. Nothing is copied and no boundary moves. The rule reads the plan's level and order, never the row. Off by default: `okf consume` reads no `parent` key, so no payload ranks differently, while the flag moves the bytes of every bundle holding a heading-only section. Measured before building on one 2 761-concept process code: 710 concepts are heading-only; the level route names the ancestor the document's own <sec> nesting names on 708 of 710 (two sit at depth 7, clipped to 6, and point one level too high), where reading section numbers gets 686 (`12` begins with `1`); 35 have no ancestor holding text and get none. The red test expected only concept files to move. The index is a projection of the frontmatter and shows the key as a facet, so the test now holds both; the facet renders a segment id as unresolved (`p1?`), because `structure` reads `parent` as a document number -- named in README, CLAUDE.md and the CHANGELOG, not repaired here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
f7fd0d4a43
commit
edd3b70a90
6 changed files with 140 additions and 4 deletions
18
CHANGELOG.md
18
CHANGELOG.md
|
|
@ -7,6 +7,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||||
|
|
||||||
## [Unreleased]
|
## [Unreleased]
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- **`okf build --shell-parent` (K3-20), off by default.** A concept whose body
|
||||||
|
is its heading alone gets `parent:` naming the `segment_id` of the nearest
|
||||||
|
ancestor that holds text: the nearest preceding plan entry at a smaller
|
||||||
|
level, passing over an ancestor that is empty too. Nothing is copied and no
|
||||||
|
boundary moves. It reads the plan's level and order, never the row. Measured
|
||||||
|
on one process code, 710 of 2 761 concepts are heading-only; the route names
|
||||||
|
the ancestor the document's own nesting names on 708 of them (two sit at
|
||||||
|
depth 7, which a markdown heading clips to 6), where reading section numbers
|
||||||
|
gets 686.
|
||||||
|
- **Off, by measurement:** `okf consume` reads no `parent` key, so no payload
|
||||||
|
ranks differently, while the flag moves the bytes of every bundle holding
|
||||||
|
a heading-only section.
|
||||||
|
- **Known cost:** the index projects `parent` as a document NUMBER, so a
|
||||||
|
segment id always renders unresolved there (`parent: p1?`). The same key
|
||||||
|
already carries both meanings for an adjudicator's declared parent.
|
||||||
|
|
||||||
### Fixed
|
### Fixed
|
||||||
|
|
||||||
- **A directory every concept id shares no longer ranks the concepts
|
- **A directory every concept id shares no longer ranks the concepts
|
||||||
|
|
|
||||||
15
CLAUDE.md
15
CLAUDE.md
|
|
@ -596,6 +596,21 @@ and fixtures, never code.
|
||||||
which would make that door claim a Door B file) is refused before anything
|
which would make that door claim a Door B file) is refused before anything
|
||||||
is read. `okf project` does not take it -- it owns no flag that moves a
|
is read. `okf project` does not take it -- it owns no flag that moves a
|
||||||
bundle's bytes.
|
bundle's bytes.
|
||||||
|
- **`--shell-parent` (K3-20) is OFF and is not a segmentation flag either**:
|
||||||
|
a plan entry whose span holds its heading alone gets `parent_id` naming the
|
||||||
|
nearest PRECEDING entry at a smaller level whose own span holds text
|
||||||
|
(`propose._link_shells`), passing over an empty ancestor; the door writes
|
||||||
|
the existing `parent:` key. Nothing is copied -- a consumer's own build of
|
||||||
|
the same standard copied the inherited text in and took hit@1 6/6 -> 2/6.
|
||||||
|
The route reads the PLAN, never the row: on R761 it names the ancestor the
|
||||||
|
`<sec>` nesting names on **708 of 710** shells (the 2 misses sit at depth 7,
|
||||||
|
clipped to 6), where reading section numbers gets 686 (`12` begins with
|
||||||
|
`1`). 35 of 710 have no ancestor holding text and get none. Off because
|
||||||
|
`okf consume` reads no `parent` key. **The index renders every such parent
|
||||||
|
UNRESOLVED** (`parent: p1?`): `structure` reads `parent` as a document
|
||||||
|
NUMBER, and a segment id answers to none -- the one key with two meanings
|
||||||
|
`inbox.py` already names for a declared parent. Report:
|
||||||
|
`docs/2026-09-11-k3-runde20-delt-katalog-og-arvet-kontekst.md`.
|
||||||
- **A TWELFTH flag, `--pdf-outline`, is OFF** (round 12, 2026-09-10) and it is
|
- **A TWELFTH flag, `--pdf-outline`, is OFF** (round 12, 2026-09-10) and it is
|
||||||
the only one here that does not read the extracted text at all: it cuts a PDF
|
the only one here that does not read the extracted text at all: it cuts a PDF
|
||||||
at the boundaries its own `/Outlines` bookmark tree declares. It is NOT Arm D
|
at the boundaries its own `/Outlines` bookmark tree declares. It is NOT Arm D
|
||||||
|
|
|
||||||
15
README.md
15
README.md
|
|
@ -153,6 +153,21 @@ whole. It adds any key and replaces only `sources` and `description`, the two
|
||||||
with a layer below them (what the document declares, else the file name);
|
with a layer below them (what the document declares, else the file name);
|
||||||
every other key the door writes itself is refused before anything is read.
|
every other key the door writes itself is refused before anything is read.
|
||||||
|
|
||||||
|
`--shell-parent` (**off**; opt out explicitly with `--no-shell-parent`) gives a
|
||||||
|
concept whose body is its heading alone a `parent:` naming the `segment_id` of
|
||||||
|
the nearest ancestor that holds text — the nearest preceding plan entry at a
|
||||||
|
smaller level, passing over an ancestor that is empty too. It copies no text
|
||||||
|
and moves no boundary. It exists for a document that states its points once and
|
||||||
|
lets every nested section inherit them: measured on one process code, **710 of
|
||||||
|
2 761** concepts are heading-only, and the plan's level and order name the same
|
||||||
|
ancestor as the document's own nesting on **708 of 710** (the two others sit at
|
||||||
|
depth 7, which a markdown heading clips to 6, and point one level too high). It
|
||||||
|
is off because `okf consume` does not read `parent`, so no payload ranks
|
||||||
|
differently with it, while it moves the bytes of every bundle holding a
|
||||||
|
heading-only section. One cost is known and not repaired: the index projects
|
||||||
|
`parent` as a document number, so a segment id always renders there as
|
||||||
|
unresolved (`parent: p1?`), even though the concept it names is in the bundle.
|
||||||
|
|
||||||
### The segmentation flags
|
### The segmentation flags
|
||||||
|
|
||||||
Nine rules are reachable from `okf build`, and since 2026-09-11 **all nine
|
Nine rules are reachable from `okf build`, and since 2026-09-11 **all nine
|
||||||
|
|
|
||||||
|
|
@ -172,6 +172,13 @@ DEFAULT_FIRST_SPAN_FROM_ZERO = True
|
||||||
#: rule closes each span against the next SURVIVOR. It adds no boundary, so the
|
#: rule closes each span against the next SURVIVOR. It adds no boundary, so the
|
||||||
#: concept count is unchanged and the cost to a consumer is a re-run.
|
#: concept count is unchanged and the cost to a consumer is a re-run.
|
||||||
DEFAULT_CLOSE_SPAN_GAPS = True
|
DEFAULT_CLOSE_SPAN_GAPS = True
|
||||||
|
#: Round 20 (K3-20). A section whose body is its heading alone gets `parent:`
|
||||||
|
#: naming the nearest ancestor that holds text (`propose._link_shells`). NOT a
|
||||||
|
#: segmentation rule: it adds no boundary and copies no text, and the concept
|
||||||
|
#: count is unchanged. OFF, and the default is a measurement of what it buys:
|
||||||
|
#: `okf consume` reads no `parent` key, so no payload ranks differently with
|
||||||
|
#: it, while it moves the bytes of every bundle holding a heading-only section.
|
||||||
|
DEFAULT_SHELL_PARENT = False
|
||||||
#: Round 9. Clause 1 asked whether a title ENDED in a number, which is a
|
#: Round 9. Clause 1 asked whether a title ENDED in a number, which is a
|
||||||
#: question about the number rather than about the title: a drawing's dimension
|
#: question about the number rather than about the title: a drawing's dimension
|
||||||
#: chain, a P&ID's schematic labels, a door schedule and a borehole log's
|
#: chain, a P&ID's schematic labels, a door schedule and a borehole log's
|
||||||
|
|
@ -353,6 +360,7 @@ def _propose_plans(
|
||||||
pdf_headings_reserve: bool = False,
|
pdf_headings_reserve: bool = False,
|
||||||
ocr: bool = False,
|
ocr: bool = False,
|
||||||
pdf_outline: bool = DEFAULT_PDF_OUTLINE,
|
pdf_outline: bool = DEFAULT_PDF_OUTLINE,
|
||||||
|
shell_parent: bool = DEFAULT_SHELL_PARENT,
|
||||||
) -> tuple[int, int, int]:
|
) -> tuple[int, int, int]:
|
||||||
"""Propose a plan per dropped file. Returns (written, nothing, failed).
|
"""Propose a plan per dropped file. Returns (written, nothing, failed).
|
||||||
|
|
||||||
|
|
@ -393,6 +401,7 @@ def _propose_plans(
|
||||||
pdf_headings_reserve=pdf_headings_reserve,
|
pdf_headings_reserve=pdf_headings_reserve,
|
||||||
ocr=ocr,
|
ocr=ocr,
|
||||||
pdf_outline=pdf_outline,
|
pdf_outline=pdf_outline,
|
||||||
|
shell_parent=shell_parent,
|
||||||
)
|
)
|
||||||
except ProposerError as exc:
|
except ProposerError as exc:
|
||||||
print(f"{CLI_ID}: {relative.as_posix()}: {exc}", file=sys.stderr)
|
print(f"{CLI_ID}: {relative.as_posix()}: {exc}", file=sys.stderr)
|
||||||
|
|
@ -431,6 +440,7 @@ def build(
|
||||||
pdf_headings_reserve: bool = DEFAULT_PDF_HEADINGS_RESERVE,
|
pdf_headings_reserve: bool = DEFAULT_PDF_HEADINGS_RESERVE,
|
||||||
ocr: bool = DEFAULT_OCR,
|
ocr: bool = DEFAULT_OCR,
|
||||||
pdf_outline: bool = DEFAULT_PDF_OUTLINE,
|
pdf_outline: bool = DEFAULT_PDF_OUTLINE,
|
||||||
|
shell_parent: bool = DEFAULT_SHELL_PARENT,
|
||||||
frontmatter: Mapping[str, str] | None = None,
|
frontmatter: Mapping[str, str] | None = None,
|
||||||
) -> CorpusReport:
|
) -> CorpusReport:
|
||||||
"""Folder in, bundle out. The whole command, minus argument parsing.
|
"""Folder in, bundle out. The whole command, minus argument parsing.
|
||||||
|
|
@ -511,6 +521,7 @@ def build(
|
||||||
pdf_headings_reserve=pdf_headings_reserve,
|
pdf_headings_reserve=pdf_headings_reserve,
|
||||||
ocr=ocr,
|
ocr=ocr,
|
||||||
pdf_outline=pdf_outline,
|
pdf_outline=pdf_outline,
|
||||||
|
shell_parent=shell_parent,
|
||||||
)
|
)
|
||||||
print(
|
print(
|
||||||
f"{CLI_ID}: proposed {written} plan(s); {nothing} document(s) with no boundary; "
|
f"{CLI_ID}: proposed {written} plan(s); {nothing} document(s) with no boundary; "
|
||||||
|
|
@ -830,6 +841,26 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
|
||||||
"pre-2026-09-11 spans byte for byte"
|
"pre-2026-09-11 spans byte for byte"
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
build_parser.add_argument(
|
||||||
|
"--shell-parent",
|
||||||
|
action="store_true",
|
||||||
|
default=DEFAULT_SHELL_PARENT,
|
||||||
|
help=(
|
||||||
|
"Give a concept whose body is its heading alone a `parent:` naming "
|
||||||
|
"the nearest ancestor that holds text, by the plan's level and "
|
||||||
|
"order, passing over an empty ancestor. Nothing is copied and no "
|
||||||
|
"boundary moves. Measured on one standard: 710 of 2 761 concepts "
|
||||||
|
"are heading-only, and the route names the ancestor its own "
|
||||||
|
"nesting names on 708 of them. OFF: `okf consume` does not read "
|
||||||
|
"the key"
|
||||||
|
),
|
||||||
|
)
|
||||||
|
build_parser.add_argument(
|
||||||
|
"--no-shell-parent",
|
||||||
|
action="store_false",
|
||||||
|
dest="shell_parent",
|
||||||
|
help="The explicit form of the default",
|
||||||
|
)
|
||||||
build_parser.add_argument(
|
build_parser.add_argument(
|
||||||
"--contents-name",
|
"--contents-name",
|
||||||
action="store_true",
|
action="store_true",
|
||||||
|
|
@ -982,6 +1013,7 @@ def main(argv: list[str] | None = None) -> int:
|
||||||
pdf_headings_reserve=args.pdf_headings == "font-reserve",
|
pdf_headings_reserve=args.pdf_headings == "font-reserve",
|
||||||
ocr=args.ocr,
|
ocr=args.ocr,
|
||||||
pdf_outline=args.pdf_outline,
|
pdf_outline=args.pdf_outline,
|
||||||
|
shell_parent=args.shell_parent,
|
||||||
frontmatter=frontmatter_from_flags(args.frontmatter or ()),
|
frontmatter=frontmatter_from_flags(args.frontmatter or ()),
|
||||||
)
|
)
|
||||||
except (IngestError, OSError, ValueError) as exc:
|
except (IngestError, OSError, ValueError) as exc:
|
||||||
|
|
|
||||||
|
|
@ -1461,6 +1461,49 @@ def _segment_path(candidate: Candidate, taken: set[str], prefix: str = "") -> st
|
||||||
return path
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
def _link_shells(entries: list[dict[str, Any]], levels: Sequence[int], text: str) -> None:
|
||||||
|
"""Give each HEADING-ONLY entry the nearest ancestor holding text as `parent_id`.
|
||||||
|
|
||||||
|
A process code states its lettered points once, on the section that owns
|
||||||
|
them, and every section nested below inherits them. Built faithfully, the
|
||||||
|
nested section is a concept whose body is its heading line and nothing
|
||||||
|
else -- measured on one 2 761-concept standard, **710** of them -- and the
|
||||||
|
bundle's directory tree is two levels deep, so the path does not name the
|
||||||
|
parent either. `parent_id` is the plan's existing relation: validated
|
||||||
|
against the plan's own ids and written by the door as `parent:`, naming the
|
||||||
|
ancestor's `segment_id`, which every concept of the run carries.
|
||||||
|
|
||||||
|
**An ancestor is the nearest PRECEDING entry at a smaller level**, and an
|
||||||
|
ancestor whose own span is empty too is passed over, so the pointer lands
|
||||||
|
on text in one step. A shell with no such ancestor gets none. Measured
|
||||||
|
against the standard's own `<sec>` nesting, this route names the same
|
||||||
|
ancestor on **708 of 710** shells; the two it misses sit at depth 7, which
|
||||||
|
ATX clips to 6, and point one level too high -- still an ancestor. Reading
|
||||||
|
the section NUMBER instead agreed on 686 of 710: `12` begins with `1`, and
|
||||||
|
an unnumbered document has no number to read at all.
|
||||||
|
|
||||||
|
Nothing is copied. Inheriting the text itself was measured by a consumer
|
||||||
|
through its own build of the same standard: every shell filled, and hit@1
|
||||||
|
over six scored questions fell from 6 of 6 to 2 of 6 as the inherited text
|
||||||
|
grew the excerpts past the budget.
|
||||||
|
"""
|
||||||
|
bodied = [
|
||||||
|
any(line.strip() and not line.startswith("#") for line in text[start:end].split("\n"))
|
||||||
|
for start, end in (entry["span"] for entry in entries)
|
||||||
|
]
|
||||||
|
for index, entry in enumerate(entries):
|
||||||
|
if bodied[index]:
|
||||||
|
continue
|
||||||
|
bound = levels[index]
|
||||||
|
for earlier in range(index - 1, -1, -1):
|
||||||
|
if levels[earlier] >= bound:
|
||||||
|
continue
|
||||||
|
if bodied[earlier]:
|
||||||
|
entry["parent_id"] = entries[earlier]["segment_id"]
|
||||||
|
break
|
||||||
|
bound = levels[earlier]
|
||||||
|
|
||||||
|
|
||||||
def build_plan(
|
def build_plan(
|
||||||
source: Path,
|
source: Path,
|
||||||
text: str,
|
text: str,
|
||||||
|
|
@ -1482,6 +1525,7 @@ def build_plan(
|
||||||
contents_name: bool = False,
|
contents_name: bool = False,
|
||||||
bold_title: bool = False,
|
bold_title: bool = False,
|
||||||
outline_marks: Sequence[OutlineMark] | None = None,
|
outline_marks: Sequence[OutlineMark] | None = None,
|
||||||
|
shell_parent: bool = False,
|
||||||
) -> dict[str, Any]:
|
) -> dict[str, Any]:
|
||||||
"""The artifact. Every entry PROPOSED, the plan itself never adjudicated."""
|
"""The artifact. Every entry PROPOSED, the plan itself never adjudicated."""
|
||||||
taken: set[str] = set()
|
taken: set[str] = set()
|
||||||
|
|
@ -1530,7 +1574,9 @@ def build_plan(
|
||||||
starts.append(offsets[mark.line])
|
starts.append(offsets[mark.line])
|
||||||
notes.append(mark.description)
|
notes.append(mark.description)
|
||||||
cursor = 0
|
cursor = 0
|
||||||
|
levels: list[int] = []
|
||||||
for candidate in subdivide(text, candidates, max_segment_chars):
|
for candidate in subdivide(text, candidates, max_segment_chars):
|
||||||
|
levels.append(candidate.level)
|
||||||
entries.append(
|
entries.append(
|
||||||
{
|
{
|
||||||
"segment_id": f"p{len(entries) + 1}",
|
"segment_id": f"p{len(entries) + 1}",
|
||||||
|
|
@ -1572,6 +1618,10 @@ def build_plan(
|
||||||
description = notes[cursor]
|
description = notes[cursor]
|
||||||
if description is not None:
|
if description is not None:
|
||||||
entries[-1]["description"] = description
|
entries[-1]["description"] = description
|
||||||
|
# AFTER every entry exists, and only then: an ancestor is read off the
|
||||||
|
# final plan's level and order, so no rule above can move it afterwards.
|
||||||
|
if shell_parent:
|
||||||
|
_link_shells(entries, levels, text)
|
||||||
return {
|
return {
|
||||||
"version": "1",
|
"version": "1",
|
||||||
"source_sha256": hashlib.sha256(source_bytes).hexdigest(),
|
"source_sha256": hashlib.sha256(source_bytes).hexdigest(),
|
||||||
|
|
@ -1617,6 +1667,7 @@ def run(
|
||||||
pdf_headings_reserve: bool = False,
|
pdf_headings_reserve: bool = False,
|
||||||
ocr: bool = False,
|
ocr: bool = False,
|
||||||
pdf_outline: bool = False,
|
pdf_outline: bool = False,
|
||||||
|
shell_parent: bool = False,
|
||||||
) -> int:
|
) -> int:
|
||||||
if max_segment_chars < 0:
|
if max_segment_chars < 0:
|
||||||
raise ProposerError(
|
raise ProposerError(
|
||||||
|
|
@ -1705,6 +1756,7 @@ def run(
|
||||||
contents_name=contents_name,
|
contents_name=contents_name,
|
||||||
bold_title=bold_title,
|
bold_title=bold_title,
|
||||||
outline_marks=marks,
|
outline_marks=marks,
|
||||||
|
shell_parent=shell_parent,
|
||||||
)
|
)
|
||||||
# Nothing to propose is an OUTCOME, and it is not an artifact. An empty
|
# Nothing to propose is an OUTCOME, and it is not an artifact. An empty
|
||||||
# plan cannot be replayed -- `process_inbox` refuses one, because a plan
|
# plan cannot be replayed -- `process_inbox` refuses one, because a plan
|
||||||
|
|
|
||||||
|
|
@ -23,6 +23,7 @@ sentence from any source.
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
from llm_ingestion_okf import cli, extract, propose
|
from llm_ingestion_okf import cli, extract, propose
|
||||||
|
|
@ -167,13 +168,16 @@ def test_a_build_without_the_flag_is_the_build_with_the_opt_out(tmp_path: Path)
|
||||||
assert files == sorted(p.relative_to(opted_out) for p in opted_out.rglob("*") if p.is_file())
|
assert files == sorted(p.relative_to(opted_out) for p in opted_out.rglob("*") if p.is_file())
|
||||||
for relative in files:
|
for relative in files:
|
||||||
assert (default / relative).read_bytes() == (opted_out / relative).read_bytes()
|
assert (default / relative).read_bytes() == (opted_out / relative).read_bytes()
|
||||||
# The flag adds exactly one `parent:` line to each shell with an ancestor.
|
# The flag adds one `parent:` line to each shell with an ancestor, and the
|
||||||
|
# index -- a projection of the frontmatter -- shows the same key as a facet
|
||||||
|
# on that shell's entry. Nothing else moves.
|
||||||
added = 0
|
added = 0
|
||||||
for relative in files:
|
for relative in files:
|
||||||
before = (default / relative).read_text(encoding="utf-8").split("\n")
|
before = (default / relative).read_text(encoding="utf-8").split("\n")
|
||||||
after = (flagged / relative).read_text(encoding="utf-8").split("\n")
|
after = (flagged / relative).read_text(encoding="utf-8").split("\n")
|
||||||
extra = [line for line in after if line not in before]
|
if relative.name == "index.md":
|
||||||
assert all(line.startswith("parent: ") for line in extra)
|
assert [re.sub(r"parent: p\d+\?; ", "", line) for line in after] == before
|
||||||
|
continue
|
||||||
assert [line for line in after if not line.startswith("parent: ")] == before
|
assert [line for line in after if not line.startswith("parent: ")] == before
|
||||||
added += len(extra)
|
added += sum(1 for line in after if line.startswith("parent: "))
|
||||||
assert added == sum(1 for ancestor in EXPECTED.values() if ancestor is not None)
|
assert added == sum(1 for ancestor in EXPECTED.values() if ancestor is not None)
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue