fix(profiles,materialize,structure,consume): a block sources sequence is decoded, not skipped

One grammar, four call sites. `read_block_mappings` moves out of
`consume.read_sources` -- where it was written and measured -- into
`profiles`, the module both the flat readers and `consume` already import,
and the three copies of the line-oriented frontmatter grammar now decode a
block sequence for the keys `STRUCTURED_BLOCK_KEYS` names. Two copies of a
block grammar would be two answers to one question.

The value TYPE was the real choice and it was measured, not argued.
`parse_frontmatter` is public API (`okf.parse_frontmatter`) returning
`dict[str, str]`, and a list of mappings is not a `str`. Widening the return
type to `str | list[dict[str, str]]` costs 15 `mypy --strict` errors across
four of the five modules that touch the reader, plus a signature every
caller outside this repository would have to follow. Rendering the entries
back into the flow form those same readers already round-trip costs 0. The
rendering is a READING projection and says so: it is not a claim that the
value is writable -- `yaml_flow_plain` still refuses a `?` and the guard
still refuses a quote inside a flow mapping, which is why the producer
writes block in the first place.

`STRUCTURED_BLOCK_KEYS` is one key wide. `sources` is the key `read_sources`
already knows how to read; a fixture in this tree carries a block
`verified:` that still reads as an empty value, and a test pins that state
so the next widening is a decision rather than a side effect.

Nothing nested reaches the document's namespace: the entries land inside
their own value, and the K3-20 substitution guarantee is asserted per reader
copy.

Three tests that pinned the old behaviour are rewritten to what is now true,
none weakened on its other half: the block round trip in
`test_multi_source_provenance` (the evidence behind `_render_sources`'
reason 1), the v0.2 characterization (whose key-space assertion is the half
that must never weaken), and K3-22's shipped-file known-positive, where the
one difference is counted and pinned at 1.

Suite 1807 passed / 1 skipped, rc 0, 94 s -- 1782/1 before plus 25 new.
ruff clean, `mypy --strict` clean over 21 files, `uv.lock` untouched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-12 16:42:22 +02:00
commit 28f9a4b540
8 changed files with 211 additions and 65 deletions

View file

@ -43,7 +43,7 @@ from llm_ingestion_okf.errors import MaterializationError
from llm_ingestion_okf.inbox import render_inbox_concept
from llm_ingestion_okf.manifest import FileSource
from llm_ingestion_okf.materialize import _render_sources, parse_frontmatter
from llm_ingestion_okf.profiles import DEFAULT, SEGMENTED_OKF_V0_2
from llm_ingestion_okf.profiles import DEFAULT, SEGMENTED_OKF_V0_2, STRUCTURED_BLOCK_KEYS
from llm_ingestion_okf.profiles import _split_frontmatter as _profiles_split
from llm_ingestion_okf.structure import _split_frontmatter as _structure_split
@ -275,7 +275,14 @@ def _strip_only(path: Path) -> dict[str, str]:
def test_every_shipped_frontmatter_reads_exactly_as_before() -> None:
"""Known-positive: the unquoting must be a no-op on every file this
repository ships, because none of them carries a `"`-wrapped value."""
repository ships, because none of them carries a `"`-wrapped value.
K3-24 added ONE difference on purpose and it is counted rather than waved
through: a `sources:` block sequence is now decoded where the pre-K3-22
reader left the key empty. Every other key on every shipped file must
still read identically, and the count is pinned so a wider change cannot
hide inside this exemption.
"""
paths = [
path
for root in (PROJECT_ROOT / "tests" / "fixtures", PROJECT_ROOT / "examples")
@ -283,8 +290,16 @@ def test_every_shipped_frontmatter_reads_exactly_as_before() -> None:
if path.read_text(encoding="utf-8").startswith("---\n")
]
assert len(paths) >= 26
decoded = 0
for path in paths:
assert parse_frontmatter(path) == _strip_only(path), path
before, now = _strip_only(path), parse_frontmatter(path)
assert set(before) == set(now), path
for key in before:
if key in STRUCTURED_BLOCK_KEYS and before[key] == "" and now[key] != "":
decoded += 1
continue
assert before[key] == now[key], (path, key)
assert decoded == 1
# --- the rules, validated against the reader --------------------------------