open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

fix(frontmatter): a nested key must not substitute for a top-level one

Browse source 
The line-oriented frontmatter grammar exists in three copies, each with the
duplication documented at its site: `materialize` reads a path, `structure`
needs a character offset, `profiles` returns body lines. All three keyed on
`key.strip()`, which discards the indentation that is the only thing telling
a nested key from a top-level one. An indented `title:` under a `sources:`
block therefore landed in the same flat namespace as the document's own
`title:` and, arriving later, won.

The failure is substitution, not omission. A dropped value is visible to
whoever reads the concept; a substituted one is not -- the document carries a
title that looks entirely right and belongs to something else. Because
`number` derives from `title` and `parent` derives from `number`, one
substitution walks the hierarchy. Measured, not inferred: a document titled
`N100.2` with a nested source titled `N200.7` came back as N200.7 with parent
N200 instead of N100.2 with parent N100.

Measured incidence across the two corpora, denominators stated:
`_okf-canonical` @ ad30107, 54 documents with parsable frontmatter, 49 carry
a nested key colliding with a top-level name (90.7%); `_okf-upstream` @
9a15b13, 66 documents, 58 collide (87.9%). The colliding key is `title`, and
often `resource` with it -- in `acme_retail/tables/orders.md` the concept's
own BigQuery resource pointer was replaced by a nested one. This is a fix
that clears observed damage, not a hardening without a witness.

The fix refuses indented lines; it does not read them. Block form stays
unreadable -- `sources` and `verified` still come back empty -- so D4's
flow-form emission rule is untouched and the structured reader is still D1b.
Two characterization tests that pinned the old behaviour now pin the new: the
block-list family still DROPS its value, and only the key-space pollution is
gone. That family is not otherwise addressed here.

Test first, red before the code was touched, with known-positive controls for
all three parsers so that a parser returning nothing could not pass.

Order: 20260830T000740Z-4733930312-from-.claude

Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
Kjell Tore Guttormsen 2026-08-31 23:31:53 +02:00
commit 2337a328d9
6 changed files with 239 additions and 20 deletions
Download patch file Download diff file Expand all files Collapse all files

27
tests/test_okf_v0_2_characterization.py
View file

@ -65,15 +65,23 @@ def test_an_inline_flow_mapping_survives_the_scalar_parser_verbatim(tmp_path: Pa
assert parse_frontmatter(path) == values
def test_a_block_list_pollutes_the_scalar_parsers_key_space(tmp_path: Path) -> None:
def test_a_block_list_is_dropped_without_polluting_the_key_space(tmp_path: Path) -> None:
"""The measured reason `sources` is emitted as an inline flow sequence.
Upstream's canonical `sources` is a block list of multi-key mappings. Read
through this line-oriented parser, each item line becomes a KEY: the list
disappears and `- id` / `resource` appear as frontmatter keys that no
document declared. `_is_ingest_owned` reads through this same parser, so
emitting the block form would have forced the gate and the parser to be
hardened in one step.
Upstream's canonical `sources` is a block list of multi-key mappings, and
this line-oriented parser still cannot READ one: the list value comes back
empty. That is unchanged, and it remains the whole reason this library
emits the flow form -- a value it can write is a value it can read back.
What changed (2026-08-31, order `...4733930312`) is the second, quieter
half. The item lines no longer become KEYS. `- id` and `resource` are
indented, so they belong to the block above them and are refused rather
than flattened into the document's namespace. The distinction matters
because `_is_ingest_owned` reads through this same parser: a fabricated
top-level key is a fact about the document that no document declared.
Refusing is not parsing. The block form stays unreadable; it is now
unreadable LOUDLY rather than by substitution. Reading it is D1b.
"""
path = tmp_path / "concept.md"
path.write_bytes(
@ -84,8 +92,9 @@ def test_a_block_list_pollutes_the_scalar_parsers_key_space(tmp_path: Path) -> N
parsed = parse_frontmatter(path)
assert parsed["sources"] == ""
assert parsed["- id"] == "margin-standard"
assert parsed["resource"] == "policies/margin-standard.md"
assert "- id" not in parsed
assert "resource" not in parsed
assert set(parsed) == {"type", "sources"}
def _stamped_concept(generated: str) -> bytes: