fix(frontmatter): write a value a YAML reader reads back, and read both forms

K3-22. SPEC SS 11 point 1: "Every non-reserved `.md` file in the tree
contains a parseable YAML frontmatter block." Measured with PyYAML 6.0.3,
okf's own default K2 bundle failed safe_load on 41 of 455 blocks and the
R761 build on 1 of 2 763, every one a block scalar written verbatim.

Block (the profile emitter, every key): a value the K3-19 rule refuses as
plain is written double-quoted, `\` and `"` escaped; every other value keeps
its bytes, and a flow collection or an empty value is written as it stands.
The rule, now `profiles.yaml_block_plain`, agrees with PyYAML on every
top-level value in eleven measured trees (0 refused that it reads verbatim,
0 kept that it does not). Double, never single: 0 values in those trees are
`"`-wrapped and 11 193 are `'`-wrapped.

Flow (`sources`, Door A and Door B, and a run-stated flow value): the pinned
guard refuses ANY quote in a flow mapping (1.3.0, measured), so a leaf PyYAML
needs quoted has no form both read. `yaml_flow_plain` refuses it instead:
`,[]{}`, `?`, a quote, ": ", " #", a trailing `:`, a leading indicator -- a
leading `-` before a non-space excepted, which both readers take. The file
name is checked too, because it is the entry's `title` when the document
declares none. Existing codes: inbox_source_file_unaddressable,
inbox_source_title_unaddressable, source_reference_unquotable,
run_frontmatter_invalid.

Readers: parse_frontmatter, profiles' and structure's copies, and both
read_sources branches unquote a `"`-wrapped value (`\"` and `\\` decoded,
nothing else); `'`-wrapped values are untouched, and structure keeps the
single-quote rule it already had. The flow-mapping split is quote-aware, so
`{ title: "a, b" }` is one pair. The generated SKILL.md header goes through
the same block rule.

TWO K3-19 TESTS MOVED, deliberately: test_run_frontmatter built with
`sources=[{ resource: ...?languageCode=nb, ... }]`, the exact form PyYAML
refused on 2 761 of 2 761 frontmatters of K3-19's flagged build. The two
build tests now write an address without `?`; the flag-grammar test keeps
the `?` address (it only splits), and a new test holds that the build
refuses it with exit 2 and writes nothing.

1753 passed, 1 skipped (OKF_HTML_CORPUS, known). No golden moved.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-11 11:09:49 +02:00
commit ed0418f228
7 changed files with 288 additions and 79 deletions

View file

@ -42,6 +42,159 @@ RESERVED_OKF_TYPE = "verdict"
_TIMESTAMP_FALLBACK_PAIR = frozenset({"timestamp", "generated"})
# --- YAML scalar forms (K3-22) -----------------------------------------------
#
# SPEC SS 11 point 1 requires "a parseable YAML frontmatter block" and SS 4
# names no YAML version and no subset, so the reader a consumer has decides;
# PyYAML is the common one. The forms are decided by RULES here and never by a
# parser -- this package's one runtime dependency is the guard -- and the rules
# are validated against PyYAML in `tests/test_yaml_frontmatter.py`, both error
# directions counted.
# What a YAML reader takes as syntax at the START of a plain scalar.
_YAML_INDICATORS = frozenset("-?:,[]{}#&*!|>'\"%@`")
# What a plain scalar INSIDE a flow mapping may not carry, for one of the two
# readers a `sources` entry has to survive. `,[]{}` end it for both. `?` ends
# it for PyYAML, whose scanner stops a flow plain scalar there, so a URL with a
# query string fails `safe_load`. A quote anywhere is refused by the pinned
# guard, which admits no quoted leaf in a flow mapping (1.3.0, measured). So
# quoting cannot rescue a flow value: plain fails one reader, quoted the other.
_FLOW_UNSAFE = frozenset(",[]{}?'\"")
# A key inside a flow mapping, as the guard's `_KEY_RE` and our readers take it.
_FLOW_KEY = re.compile(r"^[A-Za-z_][A-Za-z0-9_-]*$")
def yaml_block_plain(value: str) -> bool:
"""Whether `value` reads back verbatim as a plain scalar in a block mapping.
MEASURED ON R761 (K3-19): 217 of 2 024 first spec points carry `": "`, and
PyYAML's `safe_load` refused exactly those 217 concepts' frontmatter. K3-22
measured the same rule over every top-level value in eleven trees (the K2
default bundle, the R761 HTML and XML builds, a five-document project, this
repository's examples and fixtures, and five consumer bundles): 0 refused
that PyYAML returns verbatim, 0 kept that it does not. `" #"` is here
although PyYAML does not refuse it: `title: Kap #3` loads, as `Kap`.
"""
return (
bool(value)
and value[0] not in _YAML_INDICATORS
and ": " not in value
and " #" not in value
and not value.endswith(":")
and not any(char in value for char in "\t\n\r")
)
def yaml_flow_plain(value: str) -> bool:
"""Whether `value` reads back verbatim as a plain scalar inside a flow
mapping, for PyYAML AND the pinned guard.
The block rule plus `_FLOW_UNSAFE`, with one exception the block rule does
not make: a leading `-` followed by a non-space is an ordinary character in
a flow mapping for both readers (`del/-utkast.pdf`), and refusing it would
refuse an address both of them read. The block rule keeps refusing it
because it also decides which spec points are written at all (K3-19), and
in a block mapping a refusal only costs a pair of quotes.
"""
if not value or any(char in value for char in _FLOW_UNSAFE):
return False
rest = value[1:] if value[0] == "-" and value[1:2] not in ("", " ", "\t") else value
return yaml_block_plain(rest)
def yaml_flow_collection(value: str) -> bool:
"""Whether `value` is SHAPED as a flow collection (`[...]` or `{...}`).
Structure a producer built -- `source_offset: [0, 4]`, `generated: { by: x,
at: y }` -- is written as it stands; measured over the same eleven trees,
only structural keys carry this shape and no `title` does.
"""
return (value[:1], value[-1:]) in (("[", "]"), ("{", "}"))
def yaml_flow_collection_plain(value: str) -> bool:
"""Whether a flow collection parses, for PyYAML and the guard, into what
was written: one `{ key: leaf, ... }` mapping, or a `[...]` sequence of
such mappings or of scalars, every leaf `yaml_flow_plain`."""
if value[:1] == "{" and value[-1:] == "}":
return _flow_mapping_plain(value)
if not (value[:1] == "[" and value[-1:] == "]"):
return False
items: list[str] = []
depth = 0
current: list[str] = []
for char in value[1:-1]:
depth += {"{": 1, "}": -1}.get(char, 0)
if depth not in (0, 1):
return False
if char == "," and depth == 0:
items.append("".join(current).strip())
current = []
else:
current.append(char)
items.append("".join(current).strip())
if depth != 0:
return False
return all(
_flow_mapping_plain(item) if item[:1] == "{" else yaml_flow_plain(item) for item in items
)
def _flow_mapping_plain(item: str) -> bool:
if not (item[:1] == "{" and item[-1:] == "}"):
return False
inner = item[1:-1].strip()
if not inner:
return False
for entry in inner.split(","):
key, separator, leaf = entry.strip().partition(": ")
if not separator or not _FLOW_KEY.match(key) or not yaml_flow_plain(leaf.strip()):
return False
return True
def quote_scalar(value: str) -> str:
"""`value` as a double-quoted YAML scalar: `\\` and `"` escaped, nothing else.
Double and never single: over every bundle measured, 0 values carry a
surrounding `"` pair and 11 193 a surrounding `'` pair, so a reader that
unquotes `"` changes the meaning of no value already written.
"""
return '"' + value.replace("\\", "\\\\").replace('"', '\\"') + '"'
def unquote_scalar(value: str) -> str:
"""A surrounding `"` pair removed and `\\"` / `\\\\` decoded; else `value`.
The two escapes `quote_scalar` writes are the two decoded. Any other
backslash sequence (`\\n`, `\\t`, `\\x..`, `\\u....`) is kept as written:
a YAML reader would decode it and this reader does not claim to. A
single-quoted value is returned as it stands -- `'1'` stays `'1'`.
"""
if len(value) < 2 or value[0] != '"' or value[-1] != '"':
return value
inner = value[1:-1]
out: list[str] = []
index = 0
while index < len(inner):
char = inner[index]
if char == "\\" and inner[index + 1 : index + 2] in ('"', "\\"):
out.append(inner[index + 1])
index += 2
continue
out.append(char)
index += 1
return "".join(out)
def block_scalar(value: str) -> str:
"""`value` as written after `key: ` in a block mapping: plain where a YAML
reader returns it verbatim, double-quoted otherwise."""
return value if yaml_block_plain(value) else quote_scalar(value)
@dataclass(frozen=True)
class TypeRejection:
"""Why a profile refuses an `okf_type`, for the door to frame and raise.
@ -258,11 +411,18 @@ class FrontmatterSchema:
than trusting insertion order is what makes a regeneration over the
same data byte-identical.
A scalar a YAML reader would refuse or misread as plain is written
double-quoted (K3-22); every other value keeps its bytes. A flow
collection is written as it stands -- its leaves are validated where it
is built -- and so is an empty value.
Returns the lines only the caller owns the `---` fences.
"""
named = [key for key in self.order if key in values]
tail = sorted(key for key in values if key not in self.order)
return "\n".join(f"{key}: {self._render(key, values[key])}" for key in [*named, *tail])
return "\n".join(
f"{key}: {_emitted(self._render(key, values[key]))}" for key in [*named, *tail]
)
def _render(self, key: str, value: str) -> str:
# §5 mandates whitespace-run collapse for `source_query` only
@ -274,6 +434,12 @@ class FrontmatterSchema:
return " ".join(value.split()) if key in self.collapsed_keys else value
def _emitted(value: str) -> str:
if not value or yaml_flow_collection(value):
return value
return block_scalar(value)
# The v0.1 ingest stamp. A literal rather than a configurable value: it is what
# every bundle this library has already written carries, and recognising it is
# what keeps those bundles re-runnable under a later profile.
@ -380,7 +546,7 @@ def _split_frontmatter(text: str) -> tuple[dict[str, str], list[str]]:
continue
key, sep, value = line.partition(":")
if sep:
head[key.strip()] = value.strip()
head[key.strip()] = unquote_scalar(value.strip())
return head, []