open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(materialize): sources takes a list and renders N flow mappings

Browse source 
PM decision B6 asked for a list-taking _render_sources so a concept can record
more than one source, and prescribed the block list as the emitted form. The
list is delivered; the block form is not.

Three measurements, not an argument. Our own parse_frontmatter skips indented
lines, so a block list round-trips to an empty value with every entry silently
gone -- and _is_ingest_owned reads through that same parser. The consumer B6
was written for accepts the multi-entry flow sequence and classifies a block
sequence as unreadable provenance, so block would hand it exactly the state it
cannot read. And B6's own acceptance test asks for a round trip through this
parser, which no block form can pass.

A single source renders byte-identically, so all six goldens are unmoved. The
unquotable-value gate now runs on every entry, not just the first. New code
sources_empty refuses an empty list.

1023 -> 1034 tests, including the negative control that pins the block form's
silent data loss.
This commit is contained in:
Kjell Tore Guttormsen 2026-09-02 16:12:19 +02:00
commit 16eeeb007e
4 changed files with 240 additions and 32 deletions
Download patch file Download diff file Expand all files Collapse all files

27
CHANGELOG.md
View file

@ -9,6 +9,33 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
- **A concept can now record more than one source.** `sources` renders a flow
sequence of N flow mappings on one line, so a v0.2 profile can express
multi-source provenance instead of the single entry that was the measured
ceiling on SPEC 5.1 coverage. **A single source is byte-identical to before**,
so every golden is unmoved and no existing bundle changes.
The form is flow, not the block list PM decision B6 prescribed, and the
reason is measured: this library's frontmatter parser is line-oriented and
skips indented lines, so a block list round-trips to an EMPTY value with every
entry silently gone -- and the consumer the decision was written for accepts
the multi-entry flow sequence while classifying a block sequence as unreadable
provenance. Emitting block would have produced records neither side can read.
A negative-control test pins the block form's data loss so the reason stays
measurable rather than remembered.
New error code `sources_empty`: an empty list is refused, because
`sources: []` reads as a measured absence when it is the absence of a
measurement.
- **The consumption contract, stated normatively** in
`docs/consumption-contract.md`, with a copyable skill template
(`skills/okf-consume-template/`) and a checker (`tools/okf_contract_check.py`)
that reads its mechanically checkable half: payload shape, source marking per
excerpt, the closed `adjudication` and `trust_tier` state sets, denominator
identity, and the budget gate with its validated known-positive. The checker
ships outside `src/`, so no consumer's install surface changes.
- **Five office formats through a vendored converter**, behind the same
optional `[extract]` extra: `docx`, `xlsx`, `pptx`, `odt`, `rtf`. The
converter binary travels inside the wheel and is resolved by path rather than

2
src/llm_ingestion_okf/errors.py
View file

@ -129,6 +129,8 @@ class MaterializationError(IngestError):
contains a character that would restructure the `sources` flow mapping
(Door A, v0.2 profiles); refused rather than emitted, because the
resulting document parses cleanly into a record no one wrote
- `sources_empty` — a `sources` list with no entries; `sources: []` reads
as a measured absence when it is the absence of a measurement
- `inbox_slug_empty` — a dropped file's name reduces to an empty slug
under the id grammar (Door B; never an invented fallback name)
- `inbox_slug_too_long` — the generated inbox filename would exceed the

85
src/llm_ingestion_okf/materialize.py
View file

@ -13,7 +13,7 @@ import hashlib
import logging
import re
import unicodedata
from collections.abc import Mapping
from collections.abc import Mapping, Sequence
from dataclasses import dataclass
from pathlib import Path
@ -190,37 +190,64 @@ def _source_locator(source: Source) -> str:
return source.base_url
def _render_sources(source: Source) -> str:
"""§5 `sources` as an inline flow sequence of one flow mapping.
def _render_sources(sources: Sequence[Source]) -> str:
"""§5 `sources` as an inline flow sequence of N flow mappings.
Two keys, not upstream's five: a manifest source has no `author`, no
`last_modified`, and no bundle-internal `resource` in upstream's sense, and
inventing them would be writing fields with no reader.
Two keys per entry, not upstream's five: a manifest source has no `author`,
no `last_modified`, and no bundle-internal `resource` in upstream's sense,
and inventing them would be writing fields with no reader.
The flow form rather than upstream's block list, measured and chosen: a
block list read back through this library's line-oriented parser turns each
item line into a KEY nobody wrote — and `_is_ingest_owned` reads through
that same parser. The flow form also satisfies commons' §5 "all values MUST
be single-line", and §11 requires parseable YAML rather than block YAML.
**A LIST, and the flow form — PM decision B6 delivered, its mechanism not.**
B6 asked for a list-taking renderer and prescribed the BLOCK list as the
emitted form. The list is here; the block form is not, and the reason is
measured rather than argued:
Refusing an unquotable locator is the point of the check rather than a
nicety: `[{ id: x, resource: data, backup }]` is not a parse ERROR, it is a
mapping with a `backup` key nobody wrote. A silently wrong provenance record
is worse than a refused run, and repairing the value by quoting it would
change bytes the operator supplied. Validation, not repair — the same
posture as the filename-length gate.
- `parse_frontmatter` is line-oriented and skips indented lines, so a block
list round-trips to an EMPTY value with every entry gone, silently. We
would be writing provenance we cannot read back, and `_is_ingest_owned`
reads through that same parser.
- The consumer B6 was written for accepts `[{ k: v }, { k: v }]` — plural —
and classifies a block sequence as unreadable provenance. Block would hand
it exactly the state it reports as unreadable.
- B6's own acceptance test asks for a round trip through this parser. No
block form can pass it.
The flow form also satisfies commons' §5 "all values MUST be single-line",
and §11 requires parseable YAML rather than block YAML. Reading block needs
the structured reader (D1b); until then the constraint binds what we write.
A single source renders byte-identically to the one-entry form that shipped
before this took a list, which is what keeps all six goldens unmoved.
Refusing an unquotable value is the point of the check rather than a nicety:
`[{ id: x, resource: data, backup }]` is not a parse ERROR, it is a mapping
with a `backup` key nobody wrote. A silently wrong provenance record is
worse than a refused run, and repairing the value by quoting it would change
bytes the operator supplied. Validation, not repair — the same posture as
the filename-length gate. Every entry is checked, not only the first: a gate
that reads the head of a list is a gate the second entry walks past.
"""
locator = _source_locator(source)
for label, value in (("id", source.id), ("resource", locator)):
if _FLOW_UNSAFE_RE.search(value):
raise MaterializationError(
f"the source {label} {value!r} contains a character that would "
"restructure the `sources` flow mapping (one of `,[]{}` or a "
"colon followed by whitespace) — refusing to emit a provenance "
"record that parses cleanly into something no one wrote",
code="source_reference_unquotable",
)
return f"[{{ id: {source.id}, resource: {locator} }}]"
if not sources:
raise MaterializationError(
"a `sources` list must name at least one source — `sources: []` "
"reads as a measured absence when it is the absence of a "
"measurement",
code="sources_empty",
)
entries = []
for source in sources:
locator = _source_locator(source)
for label, value in (("id", source.id), ("resource", locator)):
if _FLOW_UNSAFE_RE.search(value):
raise MaterializationError(
f"the source {label} {value!r} contains a character that would "
"restructure the `sources` flow mapping (one of `,[]{}` or a "
"colon followed by whitespace) — refusing to emit a provenance "
"record that parses cleanly into something no one wrote",
code="source_reference_unquotable",
)
entries.append(f"{{ id: {source.id}, resource: {locator} }}")
return f"[{', '.join(entries)}]"
def _render_concept_file(
@ -248,7 +275,7 @@ def _render_concept_file(
# profiles would append `sources` to every v0.1 bundle — additivity is a
# property of what is constructed here, not of the emitter.
if "sources" in profile.frontmatter.order:
frontmatter["sources"] = _render_sources(manifest.source)
frontmatter["sources"] = _render_sources([manifest.source])
return f"---\n{profile.frontmatter.emit(frontmatter)}\n---\n\n{body}"

152
tests/test_multi_source_provenance.py Normal file
View file

@ -0,0 +1,152 @@
"""SPEC 5.1 `sources` with more than one entry, and the form it is emitted in.
PM decision B6 asked for a list-taking `_render_sources` so a concept can
record more than one source. It also prescribed the BLOCK list as the emitted
form. The list is delivered; the block form is not, and this file carries the
measurement rather than the argument.
Three facts, each pinned by a test below:
- **Our own parser loses a block list entirely.** `parse_frontmatter` is
line-oriented and skips indented lines, so `sources:` followed by ` - id: a`
round-trips to an EMPTY value with every entry gone -- silently. A provenance
record we cannot read back is worse than one we never wrote.
- **The consumer's decoder reads the flow sequence and refuses the block one.**
`portfolio-optimiser`'s `decode_flow_value` accepts `[{ k: v }, { k: v }]` --
plural -- and classifies a block sequence as `UnreadableProvenance`. Emitting
block would hand the consumer that asked for multi-source exactly the state it
reports as unreadable.
- **The order's own acceptance test settles it.** It asks for a round trip
through our `parse_frontmatter` equivalent. No block form can pass that.
So the goal is delivered in the form that reaches a reader: one flow sequence of
N flow mappings, on one line. A single source stays byte-identical, which is
what keeps all six goldens unmoved.
"""
from __future__ import annotations
from pathlib import Path
import pytest
from llm_ingestion_okf.errors import MaterializationError
from llm_ingestion_okf.manifest import FileSource, HttpSource, Source, SqlSource
from llm_ingestion_okf.materialize import _render_sources, parse_frontmatter
CATALOGUE = FileSource(id="golden-catalogue", root="fixture")
DATABASE = SqlSource(id="golden-db", connection_ref="OKF_GOLDEN_SQL_DB")
API = HttpSource(id="golden-api", base_url="https://golden.example.test")
def _document(sources: str) -> str:
return (
"---\n"
"type: dataset\n"
"title: Orders\n"
"generated: { by: process:okf-ingest, at: 2026-01-01T00:00:00Z }\n"
f"sources: {sources}\n"
"---\n"
"\nBody.\n"
)
# --- the rendered form ------------------------------------------------------
def test_one_source_renders_the_byte_identical_single_form() -> None:
"""The additivity arm. Every golden in the suite reads this line."""
assert _render_sources([CATALOGUE]) == "[{ id: golden-catalogue, resource: fixture }]"
def test_two_sources_render_as_one_flow_sequence_on_one_line() -> None:
assert _render_sources([CATALOGUE, DATABASE]) == (
"[{ id: golden-catalogue, resource: fixture }, "
"{ id: golden-db, resource: OKF_GOLDEN_SQL_DB }]"
)
assert "\n" not in _render_sources([CATALOGUE, DATABASE])
def test_sources_keep_the_order_they_were_given() -> None:
"""No sort. The order a manifest names its sources in is the manifest's
statement, and nothing here can recover it once reordered."""
forward = _render_sources([CATALOGUE, DATABASE, API])
reverse = _render_sources([API, DATABASE, CATALOGUE])
assert forward.index("golden-catalogue") < forward.index("golden-api")
assert reverse.index("golden-api") < reverse.index("golden-catalogue")
def test_an_empty_source_list_is_refused() -> None:
"""`sources: []` is a provenance record naming no source: it reads as a
measured absence when it is the absence of a measurement."""
with pytest.raises(MaterializationError) as excinfo:
_render_sources([])
assert excinfo.value.code == "sources_empty"
# --- the round trip, which is the acceptance test ---------------------------
def test_two_sources_round_trip_through_our_own_parser(tmp_path: Path) -> None:
rendered = _render_sources([CATALOGUE, DATABASE])
path = tmp_path / "concept.md"
path.write_text(_document(rendered), encoding="utf-8")
frontmatter = parse_frontmatter(path)
assert frontmatter["sources"] == rendered
assert sorted(frontmatter) == ["generated", "sources", "title", "type"]
def test_the_block_form_round_trips_to_nothing(tmp_path: Path) -> None:
"""The NEGATIVE CONTROL, and the measured reason the block form is not
emitted. Two entries go in; an empty string comes back, and no error is
raised anywhere. This test must stay green: it is the evidence, not a
regression guard."""
path = tmp_path / "concept.md"
path.write_text(
"---\n"
"type: dataset\n"
"sources:\n"
" - id: golden-catalogue\n"
" resource: fixture\n"
" - id: golden-db\n"
" resource: OKF_GOLDEN_SQL_DB\n"
"---\n"
"\nBody.\n",
encoding="utf-8",
)
frontmatter = parse_frontmatter(path)
assert frontmatter["sources"] == ""
assert "golden-db" not in "".join(frontmatter.values())
# --- the refusal applies to every entry, not only the first -----------------
@pytest.mark.parametrize("position", [0, 1, 2], ids=["first", "middle", "last"])
def test_an_unquotable_locator_is_refused_in_any_position(position: int) -> None:
"""A gate that only reads the head of a list is a gate the second entry
walks past."""
sources: list[Source] = [CATALOGUE, DATABASE, API]
sources[position] = FileSource(id="broken", root="data, backup")
with pytest.raises(MaterializationError) as excinfo:
_render_sources(sources)
assert excinfo.value.code == "source_reference_unquotable"
@pytest.mark.parametrize("position", [0, 1], ids=["first", "last"])
def test_an_unquotable_id_is_refused_in_any_position(position: int) -> None:
sources: list[Source] = [CATALOGUE, DATABASE]
sources[position] = FileSource(id="a{1}", root="fixture")
with pytest.raises(MaterializationError) as excinfo:
_render_sources(sources)
assert excinfo.value.code == "source_reference_unquotable"