open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

fix(profiles,materialize,structure,consume): a block sources sequence is decoded, not skipped

Browse source 
One grammar, four call sites. `read_block_mappings` moves out of
`consume.read_sources` -- where it was written and measured -- into
`profiles`, the module both the flat readers and `consume` already import,
and the three copies of the line-oriented frontmatter grammar now decode a
block sequence for the keys `STRUCTURED_BLOCK_KEYS` names. Two copies of a
block grammar would be two answers to one question.

The value TYPE was the real choice and it was measured, not argued.
`parse_frontmatter` is public API (`okf.parse_frontmatter`) returning
`dict[str, str]`, and a list of mappings is not a `str`. Widening the return
type to `str | list[dict[str, str]]` costs 15 `mypy --strict` errors across
four of the five modules that touch the reader, plus a signature every
caller outside this repository would have to follow. Rendering the entries
back into the flow form those same readers already round-trip costs 0. The
rendering is a READING projection and says so: it is not a claim that the
value is writable -- `yaml_flow_plain` still refuses a `?` and the guard
still refuses a quote inside a flow mapping, which is why the producer
writes block in the first place.

`STRUCTURED_BLOCK_KEYS` is one key wide. `sources` is the key `read_sources`
already knows how to read; a fixture in this tree carries a block
`verified:` that still reads as an empty value, and a test pins that state
so the next widening is a decision rather than a side effect.

Nothing nested reaches the document's namespace: the entries land inside
their own value, and the K3-20 substitution guarantee is asserted per reader
copy.

Three tests that pinned the old behaviour are rewritten to what is now true,
none weakened on its other half: the block round trip in
`test_multi_source_provenance` (the evidence behind `_render_sources`'
reason 1), the v0.2 characterization (whose key-space assertion is the half
that must never weaken), and K3-22's shipped-file known-positive, where the
one difference is counted and pinned at 1.

Suite 1807 passed / 1 skipped, rc 0, 94 s -- 1782/1 before plus 25 new.
ruff clean, `mypy --strict` clean over 21 files, `uv.lock` untouched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-12 16:42:22 +02:00
commit 28f9a4b540
8 changed files with 211 additions and 65 deletions
Download patch file Download diff file Expand all files Collapse all files

30
tests/test_okf_v0_2_characterization.py
View file

@ -66,22 +66,22 @@ def test_an_inline_flow_mapping_survives_the_scalar_parser_verbatim(tmp_path: Pa
def test_a_block_list_is_dropped_without_polluting_the_key_space(tmp_path: Path) -> None:
"""The measured reason `sources` is emitted as an inline flow sequence.
"""Upstream's canonical `sources` is a block list of multi-key mappings,
and this line-oriented parser now READS one -- without letting a single
entry key into the document's namespace.
Upstream's canonical `sources` is a block list of multi-key mappings, and
this line-oriented parser still cannot READ one: the list value comes back
empty. That is unchanged, and it remains the whole reason this library
emits the flow form -- a value it can write is a value it can read back.
Two halves, taken in two different rounds. The quieter one (2026-08-31,
order `...4733930312`) is unchanged and is what this test still guards:
`- id` and `resource` are indented, belong to the block above them, and
must never be flattened into the document's keys, because
`_is_ingest_owned` reads through this same parser and a fabricated
top-level key is a fact no document declared.
What changed (2026-08-31, order `...4733930312`) is the second, quieter
half. The item lines no longer become KEYS. `- id` and `resource` are
indented, so they belong to the block above them and are refused rather
than flattened into the document's namespace. The distinction matters
because `_is_ingest_owned` reads through this same parser: a fabricated
top-level key is a fact about the document that no document declared.
Refusing is not parsing. The block form stays unreadable; it is now
unreadable LOUDLY rather than by substitution. Reading it is D1b.
The louder one moved 2026-09-12 (K3-24). The value no longer comes back
empty: a `sources:` block sequence is decoded into the flow form this
library's readers round-trip, measured against PyYAML and the pinned
guard. `set(parsed)` below is the assertion that carries the first half,
and it is the one that must never weaken.
"""
path = tmp_path / "concept.md"
path.write_bytes(
@ -91,7 +91,7 @@ def test_a_block_list_is_dropped_without_polluting_the_key_space(tmp_path: Path)
parsed = parse_frontmatter(path)
assert parsed["sources"] == ""
assert parsed["sources"] == "[{ id: margin-standard, resource: policies/margin-standard.md }]"
assert "- id" not in parsed
assert "resource" not in parsed
assert set(parsed) == {"type", "sources"}