open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

fix(profiles,materialize,structure,consume): a block sources sequence is decoded, not skipped

Browse source 
One grammar, four call sites. `read_block_mappings` moves out of
`consume.read_sources` -- where it was written and measured -- into
`profiles`, the module both the flat readers and `consume` already import,
and the three copies of the line-oriented frontmatter grammar now decode a
block sequence for the keys `STRUCTURED_BLOCK_KEYS` names. Two copies of a
block grammar would be two answers to one question.

The value TYPE was the real choice and it was measured, not argued.
`parse_frontmatter` is public API (`okf.parse_frontmatter`) returning
`dict[str, str]`, and a list of mappings is not a `str`. Widening the return
type to `str | list[dict[str, str]]` costs 15 `mypy --strict` errors across
four of the five modules that touch the reader, plus a signature every
caller outside this repository would have to follow. Rendering the entries
back into the flow form those same readers already round-trip costs 0. The
rendering is a READING projection and says so: it is not a claim that the
value is writable -- `yaml_flow_plain` still refuses a `?` and the guard
still refuses a quote inside a flow mapping, which is why the producer
writes block in the first place.

`STRUCTURED_BLOCK_KEYS` is one key wide. `sources` is the key `read_sources`
already knows how to read; a fixture in this tree carries a block
`verified:` that still reads as an empty value, and a test pins that state
so the next widening is a decision rather than a side effect.

Nothing nested reaches the document's namespace: the entries land inside
their own value, and the K3-20 substitution guarantee is asserted per reader
copy.

Three tests that pinned the old behaviour are rewritten to what is now true,
none weakened on its other half: the block round trip in
`test_multi_source_provenance` (the evidence behind `_render_sources`'
reason 1), the v0.2 characterization (whose key-space assertion is the half
that must never weaken), and K3-22's shipped-file known-positive, where the
one difference is counted and pinned at 1.

Suite 1807 passed / 1 skipped, rc 0, 94 s -- 1782/1 before plus 25 new.
ruff clean, `mypy --strict` clean over 21 files, `uv.lock` untouched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-12 16:42:22 +02:00
commit 28f9a4b540
8 changed files with 211 additions and 65 deletions
Download patch file Download diff file Expand all files Collapse all files

30
src/llm_ingestion_okf/consume.py
View file

@ -65,6 +65,7 @@ from .profiles import (
RESERVED_OKF_TYPE,
SEGMENTED_OKF_V0_2,
BundleProfile,
read_block_mappings,
unquote_scalar,
)
@ -608,28 +609,15 @@ def read_sources(lines: Sequence[str]) -> tuple[tuple[Mapping[str, str], ...], b
if flow is None:
return (), True
return tuple(flow), True
entries: list[dict[str, str]] = []
for nested in lines[position + 1 :]:
if not nested.strip():
continue
if nested[:1] not in (" ", "\t"):
break
item = nested.strip()
if item.startswith("- "):
entries.append({})
item = item[2:].strip()
elif not entries:
# An indented line before any `- ` opens no entry. Refused
# rather than folded into one, which would invent an entry the
# document does not have.
return (), True
key, separator, raw = item.partition(":")
if not separator:
return (), True
entries[-1][key.strip()] = unquote_scalar(raw.strip())
if not entries:
# The same grammar the flat readers decode with (K3-24). It lives in
# `profiles` because `materialize` imports that module and this one
# imports `materialize`: one loop, or the two readers would answer the
# same question differently. An indented line before any `- ` opens no
# entry and comes back as `None`, refused rather than folded into one.
entries = read_block_mappings(lines, position)
if entries is None:
return (), True
return tuple(entries), True
return entries, True
return (), False

30
src/llm_ingestion_okf/materialize.py
View file

@ -29,7 +29,14 @@ from .manifest import (
generated_filename,
load_manifest_bytes,
)
from .profiles import DEFAULT, BundleProfile, unquote_scalar, yaml_flow_plain
from .profiles import (
DEFAULT,
STRUCTURED_BLOCK_KEYS,
BundleProfile,
block_mapping_value,
unquote_scalar,
yaml_flow_plain,
)
from .render import render_fenced_block, render_table
_LOGGER = logging.getLogger(__name__)
@ -121,7 +128,7 @@ def parse_frontmatter(path: Path) -> dict[str, str]:
if not lines or lines[0].strip() != "---":
return {}
frontmatter: dict[str, str] = {}
for line in lines[1:]:
for position, line in enumerate(lines[1:], start=1):
if line.strip() == "---":
break
# An INDENTED key belongs to the block above it, not to the document.
@ -129,16 +136,27 @@ def parse_frontmatter(path: Path) -> dict[str, str]:
# as the top-level keys and, arriving later, SUBSTITUTE for one of them
# -- a `sources:` entry's own `title:` silently becoming the document's,
# carrying `number` and `parent` with it. Skipping is deliberately not
# parsing: the nested value is not read, only refused. The structured
# reader is D1b.
# parsing: the nested value is not read, only refused. That refusal is
# unchanged, and `STRUCTURED_BLOCK_KEYS` does not weaken it: a decoded
# block lands INSIDE its own value, never in this namespace. The
# structured reader is still D1b.
if line[:1] in (" ", "\t"):
continue
key, sep, value = line.partition(":")
if sep:
# A `"`-wrapped value is how the emitter writes a scalar a YAML
# reader would refuse plain (K3-22); read back as that reader
# would. A `'`-wrapped one is returned as it stands.
frontmatter[key.strip()] = unquote_scalar(value.strip())
# would. A `'`-wrapped one is returned as it stands. A named key
# whose value is a block sequence is DECODED rather than left
# empty (K3-24) -- an empty value is an address disappearing with
# nothing raised.
name, raw = key.strip(), value.strip()
rendered = (
block_mapping_value(lines, position)
if not raw and name in STRUCTURED_BLOCK_KEYS
else None
)
frontmatter[name] = unquote_scalar(raw) if rendered is None else rendered
return frontmatter

96
src/llm_ingestion_okf/profiles.py
View file

@ -195,6 +195,89 @@ def block_scalar(value: str) -> str:
return value if yaml_block_plain(value) else quote_scalar(value)
#: The frontmatter keys whose BLOCK form the line-oriented grammar decodes
#: rather than skips (K3-24). One key wide on purpose: `sources` is the key
#: `consume.read_sources` already knows how to read, so decoding it here adds
#: no second grammar to disagree with the first. Widening this set changes
#: what every flat reader reports for keys no measurement covers -- a fixture
#: in this tree carries a block `verified:` that still reads as empty, and a
#: test pins that state so the next widening is a decision rather than a
#: side effect.
STRUCTURED_BLOCK_KEYS = frozenset({"sources"})
#: A leaf carrying one of these has no plain form inside a flow mapping THIS
#: library's own readers parse back: a comma or a brace would re-split the
#: mapping, a leading `"` would open a quoted scalar. A `?` is absent on
#: purpose -- it is what `yaml_flow_plain` refuses for PyYAML, and refusing it
#: here would refuse exactly the address this decoding exists to carry.
_FLOW_RENDER_UNSAFE = frozenset(",[]{}")
def read_block_mappings(
lines: Sequence[str], position: int
) -> tuple[Mapping[str, str], ...] | None:
"""The block sequence of mappings opened at `lines[position]`, or `None`.
`None` is "this reader cannot decode it", never "there is nothing here":
an indented line before any `- ` opens no entry and is refused rather than
folded into one, which would invent an entry the document does not have.
One grammar, four call sites: the three copies of the line-oriented
frontmatter reader and `consume.read_sources`, which is where this loop
was written and measured. Two copies of a block grammar would be two
answers to one question.
"""
entries: list[dict[str, str]] = []
for nested in lines[position + 1 :]:
if not nested.strip():
continue
if nested[:1] not in (" ", "\t"):
break
item = nested.strip()
if item.startswith("- "):
entries.append({})
item = item[2:].strip()
elif not entries:
return None
key, separator, raw = item.partition(":")
if not separator:
return None
entries[-1][key.strip()] = unquote_scalar(raw.strip())
if not entries:
return None
return tuple(entries)
def render_flow_mappings(entries: Sequence[Mapping[str, str]]) -> str:
"""`entries` as the flow sequence the flat readers already round-trip.
A READING projection, not an emission: the flat grammar's value type is
`str`, and the flow form is the one string shape this library's own
readers decode back into the same entries. It is deliberately NOT a claim
that the rendering is writable -- `yaml_flow_plain` still refuses a
`?` and the guard still refuses a quote inside a flow mapping, so the
emission rule is untouched and a value rendered here may have no writable
flow form at all. That is the whole reason the producer writes block.
"""
items = []
for entry in entries:
pairs = ", ".join(f"{key}: {_flow_leaf(value)}" for key, value in entry.items())
items.append("{ " + pairs + " }" if pairs else "{}")
return "[" + ", ".join(items) + "]"
def _flow_leaf(value: str) -> str:
if not value or value[0] == '"' or any(char in value for char in _FLOW_RENDER_UNSAFE):
return quote_scalar(value)
return value
def block_mapping_value(lines: Sequence[str], position: int) -> str | None:
"""The flow rendering of the block sequence at `position`, or `None`."""
entries = read_block_mappings(lines, position)
return None if entries is None else render_flow_mappings(entries)
@dataclass(frozen=True)
class TypeRejection:
"""Why a profile refuses an `okf_type`, for the door to frame and raise.
@ -546,7 +629,18 @@ def _split_frontmatter(text: str) -> tuple[dict[str, str], list[str]]:
continue
key, sep, value = line.partition(":")
if sep:
head[key.strip()] = unquote_scalar(value.strip())
name, raw = key.strip(), value.strip()
# A `sources:` block sequence is the one nested shape this grammar
# DECODES instead of skipping: the key is present with an empty
# value otherwise, which is an address disappearing rather than an
# error anyone can catch (K3-24). Nothing nested reaches the
# document's namespace -- the entries land inside the value.
rendered = (
block_mapping_value(lines, offset)
if not raw and name in STRUCTURED_BLOCK_KEYS
else None
)
head[name] = unquote_scalar(raw) if rendered is None else rendered
return head, []

21
src/llm_ingestion_okf/structure.py
View file

@ -35,7 +35,7 @@ from dataclasses import dataclass, field
from pathlib import Path
from .extract import strip_converter_attribute
from .profiles import unquote_scalar
from .profiles import STRUCTURED_BLOCK_KEYS, block_mapping_value, unquote_scalar
# A document number is either an alpha-prefixed identifier (`N500`, `V720`,
# `R610.4`) or a dotted numeric section (`4.2.1`). A BARE integer is
@ -141,7 +141,7 @@ def _split_frontmatter(text: str) -> tuple[dict[str, str], int]:
return {}, 0
declared: dict[str, str] = {}
offset = len(lines[0])
for line in lines[1:]:
for position, line in enumerate(lines[1:], start=1):
offset += len(line)
if line.strip() == "---":
return declared, offset
@ -150,13 +150,24 @@ def _split_frontmatter(text: str) -> tuple[dict[str, str], int]:
# as the top-level keys and, arriving later, SUBSTITUTE for one of them
# -- a `sources:` entry's own `title:` silently becoming the document's,
# carrying `number` and `parent` with it. Skipping is deliberately not
# parsing: the nested value is not read, only refused. The structured
# reader is D1b.
# parsing: the nested value is not read, only refused. That refusal is
# unchanged by `STRUCTURED_BLOCK_KEYS`: a decoded block lands INSIDE
# its own value. The structured reader is still D1b.
if line[:1] in (" ", "\t"):
continue
key, sep, value = line.partition(":")
if sep:
declared[key.strip()] = _unquote(value.strip())
name, raw = key.strip(), value.strip()
# A block `sources:` is decoded rather than left empty (K3-24).
# The entries keep `unquote_scalar`'s rule, which is the rule the
# entries were written and read under, rather than this module's
# older `'`-stripping one -- one grammar for the block form.
rendered = (
block_mapping_value(lines, position)
if not raw and name in STRUCTURED_BLOCK_KEYS
else None
)
declared[name] = _unquote(raw) if rendered is None else rendered
# An unterminated block is not frontmatter; the whole text is body.
return {}, 0