open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

fix(frontmatter): write a value a YAML reader reads back, and read both forms

Browse source 
K3-22. SPEC SS 11 point 1: "Every non-reserved `.md` file in the tree
contains a parseable YAML frontmatter block." Measured with PyYAML 6.0.3,
okf's own default K2 bundle failed safe_load on 41 of 455 blocks and the
R761 build on 1 of 2 763, every one a block scalar written verbatim.

Block (the profile emitter, every key): a value the K3-19 rule refuses as
plain is written double-quoted, `\` and `"` escaped; every other value keeps
its bytes, and a flow collection or an empty value is written as it stands.
The rule, now `profiles.yaml_block_plain`, agrees with PyYAML on every
top-level value in eleven measured trees (0 refused that it reads verbatim,
0 kept that it does not). Double, never single: 0 values in those trees are
`"`-wrapped and 11 193 are `'`-wrapped.

Flow (`sources`, Door A and Door B, and a run-stated flow value): the pinned
guard refuses ANY quote in a flow mapping (1.3.0, measured), so a leaf PyYAML
needs quoted has no form both read. `yaml_flow_plain` refuses it instead:
`,[]{}`, `?`, a quote, ": ", " #", a trailing `:`, a leading indicator -- a
leading `-` before a non-space excepted, which both readers take. The file
name is checked too, because it is the entry's `title` when the document
declares none. Existing codes: inbox_source_file_unaddressable,
inbox_source_title_unaddressable, source_reference_unquotable,
run_frontmatter_invalid.

Readers: parse_frontmatter, profiles' and structure's copies, and both
read_sources branches unquote a `"`-wrapped value (`\"` and `\\` decoded,
nothing else); `'`-wrapped values are untouched, and structure keeps the
single-quote rule it already had. The flow-mapping split is quote-aware, so
`{ title: "a, b" }` is one pair. The generated SKILL.md header goes through
the same block rule.

TWO K3-19 TESTS MOVED, deliberately: test_run_frontmatter built with
`sources=[{ resource: ...?languageCode=nb, ... }]`, the exact form PyYAML
refused on 2 761 of 2 761 frontmatters of K3-19's flagged build. The two
build tests now write an address without `?`; the flag-grammar test keeps
the `?` address (it only splits), and a new test holds that the build
refuses it with exit 2 and writes nothing.

1753 passed, 1 skipped (OKF_HTML_CORPUS, known). No golden moved.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-11 11:09:49 +02:00
commit ed0418f228
7 changed files with 288 additions and 79 deletions
Download patch file Download diff file Expand all files Collapse all files

9
src/llm_ingestion_okf/structure.py
View file

@ -35,6 +35,7 @@ from dataclasses import dataclass, field
from pathlib import Path
from .extract import strip_converter_attribute
from .profiles import unquote_scalar
# A document number is either an alpha-prefixed identifier (`N500`, `V720`,
# `R610.4`) or a dotted numeric section (`4.2.1`). A BARE integer is
@ -100,8 +101,12 @@ DERIVABLE_FIELDS = frozenset({"title", "number", "parent", "references"})
def _unquote(value: str) -> str:
# A producer quotes a scalar to keep YAML from retyping it (`version:
# '2021'` is a string, not an integer). The quotes are the encoding, not
# the value, and carrying them through would put them in the index.
if len(value) >= 2 and value[0] == value[-1] and value[0] in "\"'":
# the value, and carrying them through would put them in the index. A
# `"`-wrapped value is decoded the way the emitter wrote it (K3-22); a
# `'`-wrapped one keeps this module's older rule, unchanged.
if len(value) >= 2 and value[0] == value[-1] == '"':
return unquote_scalar(value)
if len(value) >= 2 and value[0] == value[-1] == "'":
return value[1:-1]
return value