fix(profiles,materialize,structure,consume): a block sources sequence is decoded, not skipped

One grammar, four call sites. `read_block_mappings` moves out of
`consume.read_sources` -- where it was written and measured -- into
`profiles`, the module both the flat readers and `consume` already import,
and the three copies of the line-oriented frontmatter grammar now decode a
block sequence for the keys `STRUCTURED_BLOCK_KEYS` names. Two copies of a
block grammar would be two answers to one question.

The value TYPE was the real choice and it was measured, not argued.
`parse_frontmatter` is public API (`okf.parse_frontmatter`) returning
`dict[str, str]`, and a list of mappings is not a `str`. Widening the return
type to `str | list[dict[str, str]]` costs 15 `mypy --strict` errors across
four of the five modules that touch the reader, plus a signature every
caller outside this repository would have to follow. Rendering the entries
back into the flow form those same readers already round-trip costs 0. The
rendering is a READING projection and says so: it is not a claim that the
value is writable -- `yaml_flow_plain` still refuses a `?` and the guard
still refuses a quote inside a flow mapping, which is why the producer
writes block in the first place.

`STRUCTURED_BLOCK_KEYS` is one key wide. `sources` is the key `read_sources`
already knows how to read; a fixture in this tree carries a block
`verified:` that still reads as an empty value, and a test pins that state
so the next widening is a decision rather than a side effect.

Nothing nested reaches the document's namespace: the entries land inside
their own value, and the K3-20 substitution guarantee is asserted per reader
copy.

Three tests that pinned the old behaviour are rewritten to what is now true,
none weakened on its other half: the block round trip in
`test_multi_source_provenance` (the evidence behind `_render_sources`'
reason 1), the v0.2 characterization (whose key-space assertion is the half
that must never weaken), and K3-22's shipped-file known-positive, where the
one difference is counted and pinned at 1.

Suite 1807 passed / 1 skipped, rc 0, 94 s -- 1782/1 before plus 25 new.
ruff clean, `mypy --strict` clean over 21 files, `uv.lock` untouched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

src/llm_ingestion_okf/consume.py

@@ -65,6 +65,7 @@ from .profiles import (
     RESERVED_OKF_TYPE,
     SEGMENTED_OKF_V0_2,
     BundleProfile,
+    read_block_mappings,
     unquote_scalar,
 )
 
@@ -608,28 +609,15 @@ def read_sources(lines: Sequence[str]) -> tuple[tuple[Mapping[str, str], ...], b
             if flow is None:
                 return (), True
             return tuple(flow), True
-        entries: list[dict[str, str]] = []
+        # The same grammar the flat readers decode with (K3-24). It lives in
-        for nested in lines[position + 1 :]:
+        # `profiles` because `materialize` imports that module and this one
-            if not nested.strip():
+        # imports `materialize`: one loop, or the two readers would answer the
-                continue
+        # same question differently. An indented line before any `- ` opens no
-            if nested[:1] not in (" ", "\t"):
+        # entry and comes back as `None`, refused rather than folded into one.
-                break
+        entries = read_block_mappings(lines, position)
-            item = nested.strip()
+        if entries is None:
-            if item.startswith("- "):
-                entries.append({})
-                item = item[2:].strip()
-            elif not entries:
-                # An indented line before any `- ` opens no entry. Refused
-                # rather than folded into one, which would invent an entry the
-                # document does not have.
-                return (), True
-            key, separator, raw = item.partition(":")
-            if not separator:
-                return (), True
-            entries[-1][key.strip()] = unquote_scalar(raw.strip())
-        if not entries:
             return (), True
-        return tuple(entries), True
+        return entries, True
     return (), False
 
 

src/llm_ingestion_okf/materialize.py

@@ -29,7 +29,14 @@ from .manifest import (
     generated_filename,
     load_manifest_bytes,
 )
-from .profiles import DEFAULT, BundleProfile, unquote_scalar, yaml_flow_plain
+from .profiles import (
+    DEFAULT,
+    STRUCTURED_BLOCK_KEYS,
+    BundleProfile,
+    block_mapping_value,
+    unquote_scalar,
+    yaml_flow_plain,
+)
 from .render import render_fenced_block, render_table
 
 _LOGGER = logging.getLogger(__name__)
@@ -121,7 +128,7 @@ def parse_frontmatter(path: Path) -> dict[str, str]:
     if not lines or lines[0].strip() != "---":
         return {}
     frontmatter: dict[str, str] = {}
-    for line in lines[1:]:
+    for position, line in enumerate(lines[1:], start=1):
         if line.strip() == "---":
             break
         # An INDENTED key belongs to the block above it, not to the document.
@@ -129,16 +136,27 @@ def parse_frontmatter(path: Path) -> dict[str, str]:
         # as the top-level keys and, arriving later, SUBSTITUTE for one of them
         # -- a `sources:` entry's own `title:` silently becoming the document's,
         # carrying `number` and `parent` with it. Skipping is deliberately not
-        # parsing: the nested value is not read, only refused. The structured
+        # parsing: the nested value is not read, only refused. That refusal is
-        # reader is D1b.
+        # unchanged, and `STRUCTURED_BLOCK_KEYS` does not weaken it: a decoded
+        # block lands INSIDE its own value, never in this namespace. The
+        # structured reader is still D1b.
         if line[:1] in (" ", "\t"):
             continue
         key, sep, value = line.partition(":")
         if sep:
             # A `"`-wrapped value is how the emitter writes a scalar a YAML
             # reader would refuse plain (K3-22); read back as that reader
-            # would. A `'`-wrapped one is returned as it stands.
+            # would. A `'`-wrapped one is returned as it stands. A named key
-            frontmatter[key.strip()] = unquote_scalar(value.strip())
+            # whose value is a block sequence is DECODED rather than left
+            # empty (K3-24) -- an empty value is an address disappearing with
+            # nothing raised.
+            name, raw = key.strip(), value.strip()
+            rendered = (
+                block_mapping_value(lines, position)
+                if not raw and name in STRUCTURED_BLOCK_KEYS
+                else None
+            )
+            frontmatter[name] = unquote_scalar(raw) if rendered is None else rendered
     return frontmatter
 
 

src/llm_ingestion_okf/profiles.py

@@ -195,6 +195,89 @@ def block_scalar(value: str) -> str:
     return value if yaml_block_plain(value) else quote_scalar(value)
 
 
+#: The frontmatter keys whose BLOCK form the line-oriented grammar decodes
+#: rather than skips (K3-24). One key wide on purpose: `sources` is the key
+#: `consume.read_sources` already knows how to read, so decoding it here adds
+#: no second grammar to disagree with the first. Widening this set changes
+#: what every flat reader reports for keys no measurement covers -- a fixture
+#: in this tree carries a block `verified:` that still reads as empty, and a
+#: test pins that state so the next widening is a decision rather than a
+#: side effect.
+STRUCTURED_BLOCK_KEYS = frozenset({"sources"})
+
+#: A leaf carrying one of these has no plain form inside a flow mapping THIS
+#: library's own readers parse back: a comma or a brace would re-split the
+#: mapping, a leading `"` would open a quoted scalar. A `?` is absent on
+#: purpose -- it is what `yaml_flow_plain` refuses for PyYAML, and refusing it
+#: here would refuse exactly the address this decoding exists to carry.
+_FLOW_RENDER_UNSAFE = frozenset(",[]{}")
+
+
+def read_block_mappings(
+    lines: Sequence[str], position: int
+) -> tuple[Mapping[str, str], ...] | None:
+    """The block sequence of mappings opened at `lines[position]`, or `None`.
+
+    `None` is "this reader cannot decode it", never "there is nothing here":
+    an indented line before any `- ` opens no entry and is refused rather than
+    folded into one, which would invent an entry the document does not have.
+
+    One grammar, four call sites: the three copies of the line-oriented
+    frontmatter reader and `consume.read_sources`, which is where this loop
+    was written and measured. Two copies of a block grammar would be two
+    answers to one question.
+    """
+    entries: list[dict[str, str]] = []
+    for nested in lines[position + 1 :]:
+        if not nested.strip():
+            continue
+        if nested[:1] not in (" ", "\t"):
+            break
+        item = nested.strip()
+        if item.startswith("- "):
+            entries.append({})
+            item = item[2:].strip()
+        elif not entries:
+            return None
+        key, separator, raw = item.partition(":")
+        if not separator:
+            return None
+        entries[-1][key.strip()] = unquote_scalar(raw.strip())
+    if not entries:
+        return None
+    return tuple(entries)
+
+
+def render_flow_mappings(entries: Sequence[Mapping[str, str]]) -> str:
+    """`entries` as the flow sequence the flat readers already round-trip.
+
+    A READING projection, not an emission: the flat grammar's value type is
+    `str`, and the flow form is the one string shape this library's own
+    readers decode back into the same entries. It is deliberately NOT a claim
+    that the rendering is writable -- `yaml_flow_plain` still refuses a
+    `?` and the guard still refuses a quote inside a flow mapping, so the
+    emission rule is untouched and a value rendered here may have no writable
+    flow form at all. That is the whole reason the producer writes block.
+    """
+    items = []
+    for entry in entries:
+        pairs = ", ".join(f"{key}: {_flow_leaf(value)}" for key, value in entry.items())
+        items.append("{ " + pairs + " }" if pairs else "{}")
+    return "[" + ", ".join(items) + "]"
+
+
+def _flow_leaf(value: str) -> str:
+    if not value or value[0] == '"' or any(char in value for char in _FLOW_RENDER_UNSAFE):
+        return quote_scalar(value)
+    return value
+
+
+def block_mapping_value(lines: Sequence[str], position: int) -> str | None:
+    """The flow rendering of the block sequence at `position`, or `None`."""
+    entries = read_block_mappings(lines, position)
+    return None if entries is None else render_flow_mappings(entries)
+
+
 @dataclass(frozen=True)
 class TypeRejection:
     """Why a profile refuses an `okf_type`, for the door to frame and raise.
@@ -546,7 +629,18 @@ def _split_frontmatter(text: str) -> tuple[dict[str, str], list[str]]:
             continue
         key, sep, value = line.partition(":")
         if sep:
-            head[key.strip()] = unquote_scalar(value.strip())
+            name, raw = key.strip(), value.strip()
+            # A `sources:` block sequence is the one nested shape this grammar
+            # DECODES instead of skipping: the key is present with an empty
+            # value otherwise, which is an address disappearing rather than an
+            # error anyone can catch (K3-24). Nothing nested reaches the
+            # document's namespace -- the entries land inside the value.
+            rendered = (
+                block_mapping_value(lines, offset)
+                if not raw and name in STRUCTURED_BLOCK_KEYS
+                else None
+            )
+            head[name] = unquote_scalar(raw) if rendered is None else rendered
     return head, []
 
 

src/llm_ingestion_okf/structure.py

@@ -35,7 +35,7 @@ from dataclasses import dataclass, field
 from pathlib import Path
 
 from .extract import strip_converter_attribute
-from .profiles import unquote_scalar
+from .profiles import STRUCTURED_BLOCK_KEYS, block_mapping_value, unquote_scalar
 
 # A document number is either an alpha-prefixed identifier (`N500`, `V720`,
 # `R610.4`) or a dotted numeric section (`4.2.1`). A BARE integer is
@@ -141,7 +141,7 @@ def _split_frontmatter(text: str) -> tuple[dict[str, str], int]:
         return {}, 0
     declared: dict[str, str] = {}
     offset = len(lines[0])
-    for line in lines[1:]:
+    for position, line in enumerate(lines[1:], start=1):
         offset += len(line)
         if line.strip() == "---":
             return declared, offset
@@ -150,13 +150,24 @@ def _split_frontmatter(text: str) -> tuple[dict[str, str], int]:
         # as the top-level keys and, arriving later, SUBSTITUTE for one of them
         # -- a `sources:` entry's own `title:` silently becoming the document's,
         # carrying `number` and `parent` with it. Skipping is deliberately not
-        # parsing: the nested value is not read, only refused. The structured
+        # parsing: the nested value is not read, only refused. That refusal is
-        # reader is D1b.
+        # unchanged by `STRUCTURED_BLOCK_KEYS`: a decoded block lands INSIDE
+        # its own value. The structured reader is still D1b.
         if line[:1] in (" ", "\t"):
             continue
         key, sep, value = line.partition(":")
         if sep:
-            declared[key.strip()] = _unquote(value.strip())
+            name, raw = key.strip(), value.strip()
+            # A block `sources:` is decoded rather than left empty (K3-24).
+            # The entries keep `unquote_scalar`'s rule, which is the rule the
+            # entries were written and read under, rather than this module's
+            # older `'`-stripping one -- one grammar for the block form.
+            rendered = (
+                block_mapping_value(lines, position)
+                if not raw and name in STRUCTURED_BLOCK_KEYS
+                else None
+            )
+            declared[name] = _unquote(raw) if rendered is None else rendered
     # An unterminated block is not frontmatter; the whole text is body.
     return {}, 0
 

tests/test_block_sources_flat_readers.py

@@ -26,6 +26,7 @@ from llm_ingestion_guard import okf as guard_okf
 
 from llm_ingestion_okf.consume import _frontmatter_lines, _parse_flow_mappings, read_sources
 from llm_ingestion_okf.materialize import parse_frontmatter
+from llm_ingestion_okf.profiles import STRUCTURED_BLOCK_KEYS
 from llm_ingestion_okf.profiles import _split_frontmatter as _profiles_split
 from llm_ingestion_okf.structure import _split_frontmatter as _structure_split
 
@@ -278,7 +279,7 @@ def test_every_fixture_frontmatter_keeps_every_value_a_reference_reader_finds()
     cannot turn this green over an empty set."""
     paths = _fixture_frontmatters()
     assert len(paths) == 12
-    carrying_block_sources = 0
+    carrying_sources_list = 0
     for path in paths:
         text = path.read_text(encoding="utf-8")
         flat = parse_frontmatter(path)
@@ -290,13 +291,15 @@ def test_every_fixture_frontmatter_keeps_every_value_a_reference_reader_finds()
         for key, value in reference.items():
             if value in (None, "", [], {}):
                 continue
-            if isinstance(value, (list, dict)) and key != "sources":
+            if isinstance(value, (list, dict)) and key not in STRUCTURED_BLOCK_KEYS:
                 continue
             assert flat.get(key, "") != "", f"{path.name}: {key} lost"
         if isinstance(reference.get("sources"), list):
-            carrying_block_sources += 1
+            carrying_sources_list += 1
             assert _parse_flow_mappings(flat["sources"]) == reference["sources"]
-    assert carrying_block_sources == 1
+    # Two of the twelve: one block form and one flow form, both read by
+    # PyYAML as a list and both required to decode to the same entries here.
+    assert carrying_sources_list == 2
 
 
 def test_a_block_key_outside_the_named_set_is_still_empty() -> None:
@@ -308,6 +311,7 @@ def test_a_block_key_outside_the_named_set_is_still_empty() -> None:
     measurement covers. A fixture in this tree carries a block `verified:` for
     exactly this reason, and it still reads as an empty value. Whoever widens
     the set will see this test, which is the point of pinning it."""
+    assert STRUCTURED_BLOCK_KEYS == frozenset({"sources"})
     path = FIXTURES / "consume-bundle" / "dyp" / "nivaa" / "blokkform-verifisert.md"
     reference = yaml.safe_load(path.read_text(encoding="utf-8").split("---\n")[1])
     assert isinstance(reference["verified"], list)

tests/test_multi_source_provenance.py

@@ -30,6 +30,7 @@ from pathlib import Path
 
 import pytest
 
+from llm_ingestion_okf.consume import _frontmatter_lines, _parse_flow_mappings, read_sources
 from llm_ingestion_okf.errors import MaterializationError
 from llm_ingestion_okf.manifest import FileSource, HttpSource, Source, SqlSource
 from llm_ingestion_okf.materialize import _render_sources, parse_frontmatter
@@ -100,11 +101,19 @@ def test_two_sources_round_trip_through_our_own_parser(tmp_path: Path) -> None:
     assert sorted(frontmatter) == ["generated", "sources", "title", "type"]
 
 
-def test_the_block_form_round_trips_to_nothing(tmp_path: Path) -> None:
+def test_the_block_form_round_trips_through_the_flat_reader(tmp_path: Path) -> None:
-    """The NEGATIVE CONTROL, and the measured reason the block form is not
+    """Two entries go in and BOTH come back (K3-24).
-    emitted. Two entries go in; an empty string comes back, and no error is
+
-    raised anywhere. This test must stay green: it is the evidence, not a
+    This test carried the opposite assertion until 2026-09-12, and it was the
-    regression guard."""
+    evidence behind the first of the three reasons `_render_sources` gives for
+    not emitting the block form: a block list round-tripped to an EMPTY value
+    with every entry gone, silently. That reason is now false, and the test
+    says what is true instead of standing as a justification nothing measures.
+
+    It is still a control and not a regression guard: what it pins is that
+    the flat reader and `read_sources` return the same entries from the same
+    bytes. The emission rule did not move with it -- reasons 2 and 3 are
+    separate measurements and live in the docstring they belong to."""
     path = tmp_path / "concept.md"
     path.write_text(
         "---\n"
@@ -120,9 +129,16 @@ def test_the_block_form_round_trips_to_nothing(tmp_path: Path) -> None:
     )
 
     frontmatter = parse_frontmatter(path)
+    entries, present = read_sources(_frontmatter_lines(path))
 
-    assert frontmatter["sources"] == ""
+    assert present
-    assert "golden-db" not in "".join(frontmatter.values())
+    assert _parse_flow_mappings(frontmatter["sources"]) == [
+        {"id": "golden-catalogue", "resource": "fixture"},
+        {"id": "golden-db", "resource": "OKF_GOLDEN_SQL_DB"},
+    ]
+    assert _parse_flow_mappings(frontmatter["sources"]) == [dict(entry) for entry in entries]
+    # The quieter half, unchanged: an entry's keys stay inside the value.
+    assert set(frontmatter) == {"type", "sources"}
 
 
 # --- the refusal applies to every entry, not only the first -----------------

tests/test_okf_v0_2_characterization.py

@@ -66,22 +66,22 @@ def test_an_inline_flow_mapping_survives_the_scalar_parser_verbatim(tmp_path: Pa
 
 
 def test_a_block_list_is_dropped_without_polluting_the_key_space(tmp_path: Path) -> None:
-    """The measured reason `sources` is emitted as an inline flow sequence.
+    """Upstream's canonical `sources` is a block list of multi-key mappings,
+    and this line-oriented parser now READS one -- without letting a single
+    entry key into the document's namespace.
 
-    Upstream's canonical `sources` is a block list of multi-key mappings, and
+    Two halves, taken in two different rounds. The quieter one (2026-08-31,
-    this line-oriented parser still cannot READ one: the list value comes back
+    order `...4733930312`) is unchanged and is what this test still guards:
-    empty. That is unchanged, and it remains the whole reason this library
+    `- id` and `resource` are indented, belong to the block above them, and
-    emits the flow form -- a value it can write is a value it can read back.
+    must never be flattened into the document's keys, because
+    `_is_ingest_owned` reads through this same parser and a fabricated
+    top-level key is a fact no document declared.
 
-    What changed (2026-08-31, order `...4733930312`) is the second, quieter
+    The louder one moved 2026-09-12 (K3-24). The value no longer comes back
-    half. The item lines no longer become KEYS. `- id` and `resource` are
+    empty: a `sources:` block sequence is decoded into the flow form this
-    indented, so they belong to the block above them and are refused rather
+    library's readers round-trip, measured against PyYAML and the pinned
-    than flattened into the document's namespace. The distinction matters
+    guard. `set(parsed)` below is the assertion that carries the first half,
-    because `_is_ingest_owned` reads through this same parser: a fabricated
+    and it is the one that must never weaken.
-    top-level key is a fact about the document that no document declared.
-
-    Refusing is not parsing. The block form stays unreadable; it is now
-    unreadable LOUDLY rather than by substitution. Reading it is D1b.
     """
     path = tmp_path / "concept.md"
     path.write_bytes(
@@ -91,7 +91,7 @@ def test_a_block_list_is_dropped_without_polluting_the_key_space(tmp_path: Path)
 
     parsed = parse_frontmatter(path)
 
-    assert parsed["sources"] == ""
+    assert parsed["sources"] == "[{ id: margin-standard, resource: policies/margin-standard.md }]"
     assert "- id" not in parsed
     assert "resource" not in parsed
     assert set(parsed) == {"type", "sources"}

tests/test_yaml_frontmatter.py

@@ -43,7 +43,7 @@ from llm_ingestion_okf.errors import MaterializationError
 from llm_ingestion_okf.inbox import render_inbox_concept
 from llm_ingestion_okf.manifest import FileSource
 from llm_ingestion_okf.materialize import _render_sources, parse_frontmatter
-from llm_ingestion_okf.profiles import DEFAULT, SEGMENTED_OKF_V0_2
+from llm_ingestion_okf.profiles import DEFAULT, SEGMENTED_OKF_V0_2, STRUCTURED_BLOCK_KEYS
 from llm_ingestion_okf.profiles import _split_frontmatter as _profiles_split
 from llm_ingestion_okf.structure import _split_frontmatter as _structure_split
 
@@ -275,7 +275,14 @@ def _strip_only(path: Path) -> dict[str, str]:
 
 def test_every_shipped_frontmatter_reads_exactly_as_before() -> None:
     """Known-positive: the unquoting must be a no-op on every file this
-    repository ships, because none of them carries a `"`-wrapped value."""
+    repository ships, because none of them carries a `"`-wrapped value.
+
+    K3-24 added ONE difference on purpose and it is counted rather than waved
+    through: a `sources:` block sequence is now decoded where the pre-K3-22
+    reader left the key empty. Every other key on every shipped file must
+    still read identically, and the count is pinned so a wider change cannot
+    hide inside this exemption.
+    """
     paths = [
         path
         for root in (PROJECT_ROOT / "tests" / "fixtures", PROJECT_ROOT / "examples")
@@ -283,8 +290,16 @@ def test_every_shipped_frontmatter_reads_exactly_as_before() -> None:
         if path.read_text(encoding="utf-8").startswith("---\n")
     ]
     assert len(paths) >= 26
+    decoded = 0
     for path in paths:
-        assert parse_frontmatter(path) == _strip_only(path), path
+        before, now = _strip_only(path), parse_frontmatter(path)
+        assert set(before) == set(now), path
+        for key in before:
+            if key in STRUCTURED_BLOCK_KEYS and before[key] == "" and now[key] != "":
+                decoded += 1
+                continue
+            assert before[key] == now[key], (path, key)
+    assert decoded == 1
 
 
 # --- the rules, validated against the reader --------------------------------