open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

fix(frontmatter): write a value a YAML reader reads back, and read both forms

Browse source 
K3-22. SPEC SS 11 point 1: "Every non-reserved `.md` file in the tree
contains a parseable YAML frontmatter block." Measured with PyYAML 6.0.3,
okf's own default K2 bundle failed safe_load on 41 of 455 blocks and the
R761 build on 1 of 2 763, every one a block scalar written verbatim.

Block (the profile emitter, every key): a value the K3-19 rule refuses as
plain is written double-quoted, `\` and `"` escaped; every other value keeps
its bytes, and a flow collection or an empty value is written as it stands.
The rule, now `profiles.yaml_block_plain`, agrees with PyYAML on every
top-level value in eleven measured trees (0 refused that it reads verbatim,
0 kept that it does not). Double, never single: 0 values in those trees are
`"`-wrapped and 11 193 are `'`-wrapped.

Flow (`sources`, Door A and Door B, and a run-stated flow value): the pinned
guard refuses ANY quote in a flow mapping (1.3.0, measured), so a leaf PyYAML
needs quoted has no form both read. `yaml_flow_plain` refuses it instead:
`,[]{}`, `?`, a quote, ": ", " #", a trailing `:`, a leading indicator -- a
leading `-` before a non-space excepted, which both readers take. The file
name is checked too, because it is the entry's `title` when the document
declares none. Existing codes: inbox_source_file_unaddressable,
inbox_source_title_unaddressable, source_reference_unquotable,
run_frontmatter_invalid.

Readers: parse_frontmatter, profiles' and structure's copies, and both
read_sources branches unquote a `"`-wrapped value (`\"` and `\\` decoded,
nothing else); `'`-wrapped values are untouched, and structure keeps the
single-quote rule it already had. The flow-mapping split is quote-aware, so
`{ title: "a, b" }` is one pair. The generated SKILL.md header goes through
the same block rule.

TWO K3-19 TESTS MOVED, deliberately: test_run_frontmatter built with
`sources=[{ resource: ...?languageCode=nb, ... }]`, the exact form PyYAML
refused on 2 761 of 2 761 frontmatters of K3-19's flagged build. The two
build tests now write an address without `?`; the flag-grammar test keeps
the `?` address (it only splits), and a new test holds that the build
refuses it with exit 2 and writes nothing.

1753 passed, 1 skipped (OKF_HTML_CORPUS, known). No golden moved.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-11 11:09:49 +02:00
commit ed0418f228
7 changed files with 288 additions and 79 deletions
Download patch file Download diff file Expand all files Collapse all files

88
src/llm_ingestion_okf/inbox.py
View file

@ -38,7 +38,16 @@ from .materialize import (
validate_ingested_at,
write_bytes,
)
from .profiles import DEFAULT, BundleProfile, IndexEntry, ProvenancePolicy
from .profiles import (
DEFAULT,
BundleProfile,
IndexEntry,
ProvenancePolicy,
yaml_block_plain,
yaml_flow_collection,
yaml_flow_collection_plain,
yaml_flow_plain,
)
from .segmentation import (
SegmentationPlan,
SegmentEntry,
@ -213,7 +222,7 @@ def render_inbox_concept(
frontmatter[policy.offset_key] = _render_flow_list([str(offset) for offset in segment.span])
if segment.parent_id is not None:
frontmatter["parent"] = segment.parent_id
if segment.description is not None and _yaml_plain(segment.description):
if segment.description is not None and yaml_block_plain(segment.description):
# The SOURCE's words, carried by the plan, and written only where
# they read back verbatim. Absent is the source saying nothing, or
# saying it in a form this line cannot carry -- never a summary
@ -313,9 +322,12 @@ def validate_concept_frontmatter(
SPEC SS 4.1 lets a producer add any key and SS 11 forbids a consumer to
reject one, so the limits here are this package's own and each is a
reader it has to survive: the value is written verbatim on ONE line
because every reader here is line-oriented, which rules out a line break
and -- since `parse_frontmatter` strips -- surrounding whitespace.
reader it has to survive: the value is written on ONE line because every
reader here is line-oriented, which rules out a line break and -- since
`parse_frontmatter` strips -- surrounding whitespace. A scalar goes out
plain where a YAML reader returns it verbatim and double-quoted otherwise;
a flow collection goes out as given, so every leaf in it must be one both a
YAML reader and the guard read back (K3-22).
"""
written = _door_keys(profile)
for key, value in values.items():
@ -336,16 +348,17 @@ def validate_concept_frontmatter(
f"surrounding whitespace, got {value!r}",
code="run_frontmatter_invalid",
)
if yaml_flow_collection(value) and not yaml_flow_collection_plain(value):
raise MaterializationError(
f"frontmatter value for {key!r} is a flow collection a YAML reader and "
"the guard would not both read back as written: a leaf carrying `?`, a "
"quote, ': ', ' #', a trailing `:` or a leading indicator has no flow "
f"form both accept, got {value!r}",
code="run_frontmatter_invalid",
)
return dict(values)
# The characters that would end a YAML flow mapping early, so a path carrying
# one would produce a `sources` list that parses as something other than what
# was written. The guard refuses a quoted scalar inside a flow mapping (1.3.0,
# measured), so escaping is not on the table -- validation is.
_FLOW_TERMINATORS = ",{}[]"
def _provenance_frontmatter(
policy: ProvenancePolicy,
*,
@ -360,22 +373,26 @@ def _provenance_frontmatter(
"which document", the locator answers "where in it", and a consumer is owed
the first even when the second cannot be computed.
"""
bad = [char for char in _FLOW_TERMINATORS if char in source_file]
if bad:
# Validation, never quoting: plain is the one form of a flow-mapping leaf
# that a YAML reader and the guard both read back verbatim -- the guard
# refuses any quote in a flow mapping (1.3.0, measured) -- so a value it
# cannot carry is refused rather than mangled (K3-22). The file name is
# checked too: it is the entry's `title` when the document declares none.
shown = title if title is not None else PurePosixPath(source_file).name
if not yaml_flow_plain(source_file) or (title is None and not yaml_flow_plain(shown)):
raise MaterializationError(
f"source_file {source_file!r} contains {bad[0]!r}, which would end the "
"`sources` flow mapping early; this profile writes an address a "
"consumer can follow, and a path it cannot express is refused rather "
"than mangled",
f"source_file {source_file!r} has no plain form in the `sources` flow "
"mapping that both a YAML reader and the guard read back verbatim; this "
"profile writes an address a consumer can follow, and a path it cannot "
"express is refused rather than mangled",
code="inbox_source_file_unaddressable",
)
if title is not None and not _flow_expressible(title):
if title is not None and not yaml_flow_plain(title):
raise MaterializationError(
f"source title {title!r} cannot be written into the `sources` flow "
"mapping verbatim; refused rather than mangled",
code="inbox_source_title_unaddressable",
)
shown = title if title is not None else PurePosixPath(source_file).name
values = {policy.sources_key: f"[{{ resource: {source_file}, title: {shown} }}]"}
if units is None or span is None:
return values
@ -395,35 +412,6 @@ def _provenance_frontmatter(
return values
def _flow_expressible(value: str) -> bool:
"""Whether `value` survives as a plain scalar inside a flow mapping."""
return bool(value) and not any(char in value for char in f"{_FLOW_TERMINATORS}\n\r")
# What a YAML reader takes as syntax at the START of a plain scalar.
_YAML_INDICATORS = frozenset("-?:,[]{}#&*!|>'\"%@`")
def _yaml_plain(value: str) -> bool:
"""Whether `value` reads back verbatim as a plain scalar in a block mapping.
MEASURED ON R761: 217 of 2 024 first spec points carry `": "`, and PyYAML's
`safe_load` refused exactly those 217 concepts' frontmatter. Decided by
rule rather than by a parser, because this package's one runtime
dependency is the guard -- and over those 2 024 values the rule and PyYAML
agree on every one: 217 refused, 0 refused that PyYAML reads, 0 kept that
it does not.
"""
return (
bool(value)
and value[0] not in _YAML_INDICATORS
and ": " not in value
and " #" not in value
and not value.endswith(":")
and not any(char in value for char in "\t\n\r")
)
def _screened(gate: Gate, value: str | None) -> str | None:
"""A value read from the DOCUMENT and persisted outside its screened body.
@ -462,7 +450,7 @@ def _declared_sources_title(identity: DeclaredIdentity | None, gate: Gate) -> st
candidates.append(identity.title)
for candidate in candidates:
kept = _screened(gate, candidate)
if kept is not None and _flow_expressible(kept):
if kept is not None and yaml_flow_plain(kept):
return kept
return None