feat(materialize): sources takes a list and renders N flow mappings

PM decision B6 asked for a list-taking _render_sources so a concept can record
more than one source, and prescribed the block list as the emitted form. The
list is delivered; the block form is not.

Three measurements, not an argument. Our own parse_frontmatter skips indented
lines, so a block list round-trips to an empty value with every entry silently
gone -- and _is_ingest_owned reads through that same parser. The consumer B6
was written for accepts the multi-entry flow sequence and classifies a block
sequence as unreadable provenance, so block would hand it exactly the state it
cannot read. And B6's own acceptance test asks for a round trip through this
parser, which no block form can pass.

A single source renders byte-identically, so all six goldens are unmoved. The
unquotable-value gate now runs on every entry, not just the first. New code
sources_empty refuses an empty list.

1023 -> 1034 tests, including the negative control that pins the block form's
silent data loss.
This commit is contained in:
Kjell Tore Guttormsen 2026-09-02 16:12:19 +02:00
commit 16eeeb007e
4 changed files with 240 additions and 32 deletions

View file

@ -9,6 +9,33 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added ### Added
- **A concept can now record more than one source.** `sources` renders a flow
sequence of N flow mappings on one line, so a v0.2 profile can express
multi-source provenance instead of the single entry that was the measured
ceiling on SPEC 5.1 coverage. **A single source is byte-identical to before**,
so every golden is unmoved and no existing bundle changes.
The form is flow, not the block list PM decision B6 prescribed, and the
reason is measured: this library's frontmatter parser is line-oriented and
skips indented lines, so a block list round-trips to an EMPTY value with every
entry silently gone -- and the consumer the decision was written for accepts
the multi-entry flow sequence while classifying a block sequence as unreadable
provenance. Emitting block would have produced records neither side can read.
A negative-control test pins the block form's data loss so the reason stays
measurable rather than remembered.
New error code `sources_empty`: an empty list is refused, because
`sources: []` reads as a measured absence when it is the absence of a
measurement.
- **The consumption contract, stated normatively** in
`docs/consumption-contract.md`, with a copyable skill template
(`skills/okf-consume-template/`) and a checker (`tools/okf_contract_check.py`)
that reads its mechanically checkable half: payload shape, source marking per
excerpt, the closed `adjudication` and `trust_tier` state sets, denominator
identity, and the budget gate with its validated known-positive. The checker
ships outside `src/`, so no consumer's install surface changes.
- **Five office formats through a vendored converter**, behind the same - **Five office formats through a vendored converter**, behind the same
optional `[extract]` extra: `docx`, `xlsx`, `pptx`, `odt`, `rtf`. The optional `[extract]` extra: `docx`, `xlsx`, `pptx`, `odt`, `rtf`. The
converter binary travels inside the wheel and is resolved by path rather than converter binary travels inside the wheel and is resolved by path rather than

View file

@ -129,6 +129,8 @@ class MaterializationError(IngestError):
contains a character that would restructure the `sources` flow mapping contains a character that would restructure the `sources` flow mapping
(Door A, v0.2 profiles); refused rather than emitted, because the (Door A, v0.2 profiles); refused rather than emitted, because the
resulting document parses cleanly into a record no one wrote resulting document parses cleanly into a record no one wrote
- `sources_empty` a `sources` list with no entries; `sources: []` reads
as a measured absence when it is the absence of a measurement
- `inbox_slug_empty` a dropped file's name reduces to an empty slug - `inbox_slug_empty` a dropped file's name reduces to an empty slug
under the id grammar (Door B; never an invented fallback name) under the id grammar (Door B; never an invented fallback name)
- `inbox_slug_too_long` the generated inbox filename would exceed the - `inbox_slug_too_long` the generated inbox filename would exceed the

View file

@ -13,7 +13,7 @@ import hashlib
import logging import logging
import re import re
import unicodedata import unicodedata
from collections.abc import Mapping from collections.abc import Mapping, Sequence
from dataclasses import dataclass from dataclasses import dataclass
from pathlib import Path from pathlib import Path
@ -190,26 +190,52 @@ def _source_locator(source: Source) -> str:
return source.base_url return source.base_url
def _render_sources(source: Source) -> str: def _render_sources(sources: Sequence[Source]) -> str:
"""§5 `sources` as an inline flow sequence of one flow mapping. """§5 `sources` as an inline flow sequence of N flow mappings.
Two keys, not upstream's five: a manifest source has no `author`, no Two keys per entry, not upstream's five: a manifest source has no `author`,
`last_modified`, and no bundle-internal `resource` in upstream's sense, and no `last_modified`, and no bundle-internal `resource` in upstream's sense,
inventing them would be writing fields with no reader. and inventing them would be writing fields with no reader.
The flow form rather than upstream's block list, measured and chosen: a **A LIST, and the flow form PM decision B6 delivered, its mechanism not.**
block list read back through this library's line-oriented parser turns each B6 asked for a list-taking renderer and prescribed the BLOCK list as the
item line into a KEY nobody wrote and `_is_ingest_owned` reads through emitted form. The list is here; the block form is not, and the reason is
that same parser. The flow form also satisfies commons' §5 "all values MUST measured rather than argued:
be single-line", and §11 requires parseable YAML rather than block YAML.
Refusing an unquotable locator is the point of the check rather than a - `parse_frontmatter` is line-oriented and skips indented lines, so a block
nicety: `[{ id: x, resource: data, backup }]` is not a parse ERROR, it is a list round-trips to an EMPTY value with every entry gone, silently. We
mapping with a `backup` key nobody wrote. A silently wrong provenance record would be writing provenance we cannot read back, and `_is_ingest_owned`
is worse than a refused run, and repairing the value by quoting it would reads through that same parser.
change bytes the operator supplied. Validation, not repair the same - The consumer B6 was written for accepts `[{ k: v }, { k: v }]` plural
posture as the filename-length gate. and classifies a block sequence as unreadable provenance. Block would hand
it exactly the state it reports as unreadable.
- B6's own acceptance test asks for a round trip through this parser. No
block form can pass it.
The flow form also satisfies commons' §5 "all values MUST be single-line",
and §11 requires parseable YAML rather than block YAML. Reading block needs
the structured reader (D1b); until then the constraint binds what we write.
A single source renders byte-identically to the one-entry form that shipped
before this took a list, which is what keeps all six goldens unmoved.
Refusing an unquotable value is the point of the check rather than a nicety:
`[{ id: x, resource: data, backup }]` is not a parse ERROR, it is a mapping
with a `backup` key nobody wrote. A silently wrong provenance record is
worse than a refused run, and repairing the value by quoting it would change
bytes the operator supplied. Validation, not repair the same posture as
the filename-length gate. Every entry is checked, not only the first: a gate
that reads the head of a list is a gate the second entry walks past.
""" """
if not sources:
raise MaterializationError(
"a `sources` list must name at least one source — `sources: []` "
"reads as a measured absence when it is the absence of a "
"measurement",
code="sources_empty",
)
entries = []
for source in sources:
locator = _source_locator(source) locator = _source_locator(source)
for label, value in (("id", source.id), ("resource", locator)): for label, value in (("id", source.id), ("resource", locator)):
if _FLOW_UNSAFE_RE.search(value): if _FLOW_UNSAFE_RE.search(value):
@ -220,7 +246,8 @@ def _render_sources(source: Source) -> str:
"record that parses cleanly into something no one wrote", "record that parses cleanly into something no one wrote",
code="source_reference_unquotable", code="source_reference_unquotable",
) )
return f"[{{ id: {source.id}, resource: {locator} }}]" entries.append(f"{{ id: {source.id}, resource: {locator} }}")
return f"[{', '.join(entries)}]"
def _render_concept_file( def _render_concept_file(
@ -248,7 +275,7 @@ def _render_concept_file(
# profiles would append `sources` to every v0.1 bundle — additivity is a # profiles would append `sources` to every v0.1 bundle — additivity is a
# property of what is constructed here, not of the emitter. # property of what is constructed here, not of the emitter.
if "sources" in profile.frontmatter.order: if "sources" in profile.frontmatter.order:
frontmatter["sources"] = _render_sources(manifest.source) frontmatter["sources"] = _render_sources([manifest.source])
return f"---\n{profile.frontmatter.emit(frontmatter)}\n---\n\n{body}" return f"---\n{profile.frontmatter.emit(frontmatter)}\n---\n\n{body}"

View file

@ -0,0 +1,152 @@
"""SPEC 5.1 `sources` with more than one entry, and the form it is emitted in.
PM decision B6 asked for a list-taking `_render_sources` so a concept can
record more than one source. It also prescribed the BLOCK list as the emitted
form. The list is delivered; the block form is not, and this file carries the
measurement rather than the argument.
Three facts, each pinned by a test below:
- **Our own parser loses a block list entirely.** `parse_frontmatter` is
line-oriented and skips indented lines, so `sources:` followed by ` - id: a`
round-trips to an EMPTY value with every entry gone -- silently. A provenance
record we cannot read back is worse than one we never wrote.
- **The consumer's decoder reads the flow sequence and refuses the block one.**
`portfolio-optimiser`'s `decode_flow_value` accepts `[{ k: v }, { k: v }]` --
plural -- and classifies a block sequence as `UnreadableProvenance`. Emitting
block would hand the consumer that asked for multi-source exactly the state it
reports as unreadable.
- **The order's own acceptance test settles it.** It asks for a round trip
through our `parse_frontmatter` equivalent. No block form can pass that.
So the goal is delivered in the form that reaches a reader: one flow sequence of
N flow mappings, on one line. A single source stays byte-identical, which is
what keeps all six goldens unmoved.
"""
from __future__ import annotations
from pathlib import Path
import pytest
from llm_ingestion_okf.errors import MaterializationError
from llm_ingestion_okf.manifest import FileSource, HttpSource, Source, SqlSource
from llm_ingestion_okf.materialize import _render_sources, parse_frontmatter
CATALOGUE = FileSource(id="golden-catalogue", root="fixture")
DATABASE = SqlSource(id="golden-db", connection_ref="OKF_GOLDEN_SQL_DB")
API = HttpSource(id="golden-api", base_url="https://golden.example.test")
def _document(sources: str) -> str:
return (
"---\n"
"type: dataset\n"
"title: Orders\n"
"generated: { by: process:okf-ingest, at: 2026-01-01T00:00:00Z }\n"
f"sources: {sources}\n"
"---\n"
"\nBody.\n"
)
# --- the rendered form ------------------------------------------------------
def test_one_source_renders_the_byte_identical_single_form() -> None:
"""The additivity arm. Every golden in the suite reads this line."""
assert _render_sources([CATALOGUE]) == "[{ id: golden-catalogue, resource: fixture }]"
def test_two_sources_render_as_one_flow_sequence_on_one_line() -> None:
assert _render_sources([CATALOGUE, DATABASE]) == (
"[{ id: golden-catalogue, resource: fixture }, "
"{ id: golden-db, resource: OKF_GOLDEN_SQL_DB }]"
)
assert "\n" not in _render_sources([CATALOGUE, DATABASE])
def test_sources_keep_the_order_they_were_given() -> None:
"""No sort. The order a manifest names its sources in is the manifest's
statement, and nothing here can recover it once reordered."""
forward = _render_sources([CATALOGUE, DATABASE, API])
reverse = _render_sources([API, DATABASE, CATALOGUE])
assert forward.index("golden-catalogue") < forward.index("golden-api")
assert reverse.index("golden-api") < reverse.index("golden-catalogue")
def test_an_empty_source_list_is_refused() -> None:
"""`sources: []` is a provenance record naming no source: it reads as a
measured absence when it is the absence of a measurement."""
with pytest.raises(MaterializationError) as excinfo:
_render_sources([])
assert excinfo.value.code == "sources_empty"
# --- the round trip, which is the acceptance test ---------------------------
def test_two_sources_round_trip_through_our_own_parser(tmp_path: Path) -> None:
rendered = _render_sources([CATALOGUE, DATABASE])
path = tmp_path / "concept.md"
path.write_text(_document(rendered), encoding="utf-8")
frontmatter = parse_frontmatter(path)
assert frontmatter["sources"] == rendered
assert sorted(frontmatter) == ["generated", "sources", "title", "type"]
def test_the_block_form_round_trips_to_nothing(tmp_path: Path) -> None:
"""The NEGATIVE CONTROL, and the measured reason the block form is not
emitted. Two entries go in; an empty string comes back, and no error is
raised anywhere. This test must stay green: it is the evidence, not a
regression guard."""
path = tmp_path / "concept.md"
path.write_text(
"---\n"
"type: dataset\n"
"sources:\n"
" - id: golden-catalogue\n"
" resource: fixture\n"
" - id: golden-db\n"
" resource: OKF_GOLDEN_SQL_DB\n"
"---\n"
"\nBody.\n",
encoding="utf-8",
)
frontmatter = parse_frontmatter(path)
assert frontmatter["sources"] == ""
assert "golden-db" not in "".join(frontmatter.values())
# --- the refusal applies to every entry, not only the first -----------------
@pytest.mark.parametrize("position", [0, 1, 2], ids=["first", "middle", "last"])
def test_an_unquotable_locator_is_refused_in_any_position(position: int) -> None:
"""A gate that only reads the head of a list is a gate the second entry
walks past."""
sources: list[Source] = [CATALOGUE, DATABASE, API]
sources[position] = FileSource(id="broken", root="data, backup")
with pytest.raises(MaterializationError) as excinfo:
_render_sources(sources)
assert excinfo.value.code == "source_reference_unquotable"
@pytest.mark.parametrize("position", [0, 1], ids=["first", "last"])
def test_an_unquotable_id_is_refused_in_any_position(position: int) -> None:
sources: list[Source] = [CATALOGUE, DATABASE]
sources[position] = FileSource(id="a{1}", root="fixture")
with pytest.raises(MaterializationError) as excinfo:
_render_sources(sources)
assert excinfo.value.code == "source_reference_unquotable"