open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

fix(materialize): emit frontmatter and index values verbatim

Browse source 
a7e1bc8 unified title rendering at both sites and routed the open question
to commons, the spec owner: does §5's whitespace collapse apply to `title`,
or only to `source_query`? The ruling is source_query only. §5 mandates the
collapse for that one field — where a multi-line SQL SELECT must render on
one line — while every other value is validated single-line at manifest load
and emitted as-is: validation, not repair.

_single_line is renamed _collapse_whitespace and now runs for source_query
alone. Frontmatter values, the index link label, and the labels_by_target
map all pass through verbatim. A title carrying an internal whitespace run
(or any other operator-supplied bytes) now survives byte-for-byte at both
the frontmatter and the index-label site, instead of being silently altered.

No golden bytes change: no title in the shipped fixtures or either consumer
carries a collapsible whitespace run.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HBbjgS5A55RVavoyjJC4FX
This commit is contained in:
Kjell Tore Guttormsen 2026-07-23 06:41:27 +02:00
commit c011534e9f
2 changed files with 46 additions and 11 deletions
Download patch file Download diff file Expand all files Collapse all files

25
src/llm_ingestion_okf/materialize.py
View file

@ -49,18 +49,22 @@ class IngestResult:
stamp: str
def _single_line(value: str) -> str:
# §5: frontmatter values are single-lined — whitespace runs, including
# newlines, collapse to one space — because the format is line-oriented
# and parsing stops at the first `---` line. §4 makes the extraction
# title BOTH the `title` frontmatter and the index link label, so the
# same rule has to reach both sites or the one value renders two ways.
def _collapse_whitespace(value: str) -> str:
# §5 mandates whitespace-run collapse for ONE field only: `source_query`
# (ingest-spec.md:140-141), where a legitimately multi-line SQL SELECT
# must render on one line. Every other value is validated single-line at
# manifest load and emitted verbatim — validation, not repair — so
# operator-supplied bytes (e.g. a title's internal double space) survive.
return " ".join(value.split())
def _render_frontmatter(frontmatter: dict[str, str]) -> str:
# Line-oriented `key: value`, insertion order.
return "\n".join(f"{key}: {_single_line(value)}" for key, value in frontmatter.items())
# Line-oriented `key: value`, insertion order. Only `source_query` is
# collapsed; all other values pass through verbatim.
return "\n".join(
f"{key}: {_collapse_whitespace(value) if key == 'source_query' else value}"
for key, value in frontmatter.items()
)
def _parse_frontmatter(path: Path) -> dict[str, str]:
@ -148,7 +152,7 @@ def _link_in_index(bundle_dir: Path, target_name: str, label: str) -> None:
if f"]({target_name})" in body:
return
prefix = body if body.endswith("\n") else body + "\n"
index_path.write_bytes(f"{prefix}- [{_single_line(label)}]({target_name})\n".encode())
index_path.write_bytes(f"{prefix}- [{label}]({target_name})\n".encode())
def materialize_bundle(
@ -272,8 +276,7 @@ def materialize_bundle(
# bundle_summary as its body; links are appended in extraction order.
index_path = bundle / _INDEX_NAME
labels_by_target = {
generated_filename(extraction.id): _single_line(extraction.title)
for extraction in manifest.extractions
generated_filename(extraction.id): extraction.title for extraction in manifest.extractions
}
if not index_path.is_file():
_write_bytes(bundle, _INDEX_NAME, manifest.bundle_summary + "\n")