feat(materialize): sources takes a list and renders N flow mappings
PM decision B6 asked for a list-taking _render_sources so a concept can record more than one source, and prescribed the block list as the emitted form. The list is delivered; the block form is not. Three measurements, not an argument. Our own parse_frontmatter skips indented lines, so a block list round-trips to an empty value with every entry silently gone -- and _is_ingest_owned reads through that same parser. The consumer B6 was written for accepts the multi-entry flow sequence and classifies a block sequence as unreadable provenance, so block would hand it exactly the state it cannot read. And B6's own acceptance test asks for a round trip through this parser, which no block form can pass. A single source renders byte-identically, so all six goldens are unmoved. The unquotable-value gate now runs on every entry, not just the first. New code sources_empty refuses an empty list. 1023 -> 1034 tests, including the negative control that pins the block form's silent data loss.
This commit is contained in:
parent
bc0b4130f1
commit
16eeeb007e
4 changed files with 240 additions and 32 deletions
27
CHANGELOG.md
27
CHANGELOG.md
|
|
@ -9,6 +9,33 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
|
|
||||||
|
- **A concept can now record more than one source.** `sources` renders a flow
|
||||||
|
sequence of N flow mappings on one line, so a v0.2 profile can express
|
||||||
|
multi-source provenance instead of the single entry that was the measured
|
||||||
|
ceiling on SPEC 5.1 coverage. **A single source is byte-identical to before**,
|
||||||
|
so every golden is unmoved and no existing bundle changes.
|
||||||
|
|
||||||
|
The form is flow, not the block list PM decision B6 prescribed, and the
|
||||||
|
reason is measured: this library's frontmatter parser is line-oriented and
|
||||||
|
skips indented lines, so a block list round-trips to an EMPTY value with every
|
||||||
|
entry silently gone -- and the consumer the decision was written for accepts
|
||||||
|
the multi-entry flow sequence while classifying a block sequence as unreadable
|
||||||
|
provenance. Emitting block would have produced records neither side can read.
|
||||||
|
A negative-control test pins the block form's data loss so the reason stays
|
||||||
|
measurable rather than remembered.
|
||||||
|
|
||||||
|
New error code `sources_empty`: an empty list is refused, because
|
||||||
|
`sources: []` reads as a measured absence when it is the absence of a
|
||||||
|
measurement.
|
||||||
|
|
||||||
|
- **The consumption contract, stated normatively** in
|
||||||
|
`docs/consumption-contract.md`, with a copyable skill template
|
||||||
|
(`skills/okf-consume-template/`) and a checker (`tools/okf_contract_check.py`)
|
||||||
|
that reads its mechanically checkable half: payload shape, source marking per
|
||||||
|
excerpt, the closed `adjudication` and `trust_tier` state sets, denominator
|
||||||
|
identity, and the budget gate with its validated known-positive. The checker
|
||||||
|
ships outside `src/`, so no consumer's install surface changes.
|
||||||
|
|
||||||
- **Five office formats through a vendored converter**, behind the same
|
- **Five office formats through a vendored converter**, behind the same
|
||||||
optional `[extract]` extra: `docx`, `xlsx`, `pptx`, `odt`, `rtf`. The
|
optional `[extract]` extra: `docx`, `xlsx`, `pptx`, `odt`, `rtf`. The
|
||||||
converter binary travels inside the wheel and is resolved by path rather than
|
converter binary travels inside the wheel and is resolved by path rather than
|
||||||
|
|
|
||||||
|
|
@ -129,6 +129,8 @@ class MaterializationError(IngestError):
|
||||||
contains a character that would restructure the `sources` flow mapping
|
contains a character that would restructure the `sources` flow mapping
|
||||||
(Door A, v0.2 profiles); refused rather than emitted, because the
|
(Door A, v0.2 profiles); refused rather than emitted, because the
|
||||||
resulting document parses cleanly into a record no one wrote
|
resulting document parses cleanly into a record no one wrote
|
||||||
|
- `sources_empty` — a `sources` list with no entries; `sources: []` reads
|
||||||
|
as a measured absence when it is the absence of a measurement
|
||||||
- `inbox_slug_empty` — a dropped file's name reduces to an empty slug
|
- `inbox_slug_empty` — a dropped file's name reduces to an empty slug
|
||||||
under the id grammar (Door B; never an invented fallback name)
|
under the id grammar (Door B; never an invented fallback name)
|
||||||
- `inbox_slug_too_long` — the generated inbox filename would exceed the
|
- `inbox_slug_too_long` — the generated inbox filename would exceed the
|
||||||
|
|
|
||||||
|
|
@ -13,7 +13,7 @@ import hashlib
|
||||||
import logging
|
import logging
|
||||||
import re
|
import re
|
||||||
import unicodedata
|
import unicodedata
|
||||||
from collections.abc import Mapping
|
from collections.abc import Mapping, Sequence
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
@ -190,26 +190,52 @@ def _source_locator(source: Source) -> str:
|
||||||
return source.base_url
|
return source.base_url
|
||||||
|
|
||||||
|
|
||||||
def _render_sources(source: Source) -> str:
|
def _render_sources(sources: Sequence[Source]) -> str:
|
||||||
"""§5 `sources` as an inline flow sequence of one flow mapping.
|
"""§5 `sources` as an inline flow sequence of N flow mappings.
|
||||||
|
|
||||||
Two keys, not upstream's five: a manifest source has no `author`, no
|
Two keys per entry, not upstream's five: a manifest source has no `author`,
|
||||||
`last_modified`, and no bundle-internal `resource` in upstream's sense, and
|
no `last_modified`, and no bundle-internal `resource` in upstream's sense,
|
||||||
inventing them would be writing fields with no reader.
|
and inventing them would be writing fields with no reader.
|
||||||
|
|
||||||
The flow form rather than upstream's block list, measured and chosen: a
|
**A LIST, and the flow form — PM decision B6 delivered, its mechanism not.**
|
||||||
block list read back through this library's line-oriented parser turns each
|
B6 asked for a list-taking renderer and prescribed the BLOCK list as the
|
||||||
item line into a KEY nobody wrote — and `_is_ingest_owned` reads through
|
emitted form. The list is here; the block form is not, and the reason is
|
||||||
that same parser. The flow form also satisfies commons' §5 "all values MUST
|
measured rather than argued:
|
||||||
be single-line", and §11 requires parseable YAML rather than block YAML.
|
|
||||||
|
|
||||||
Refusing an unquotable locator is the point of the check rather than a
|
- `parse_frontmatter` is line-oriented and skips indented lines, so a block
|
||||||
nicety: `[{ id: x, resource: data, backup }]` is not a parse ERROR, it is a
|
list round-trips to an EMPTY value with every entry gone, silently. We
|
||||||
mapping with a `backup` key nobody wrote. A silently wrong provenance record
|
would be writing provenance we cannot read back, and `_is_ingest_owned`
|
||||||
is worse than a refused run, and repairing the value by quoting it would
|
reads through that same parser.
|
||||||
change bytes the operator supplied. Validation, not repair — the same
|
- The consumer B6 was written for accepts `[{ k: v }, { k: v }]` — plural —
|
||||||
posture as the filename-length gate.
|
and classifies a block sequence as unreadable provenance. Block would hand
|
||||||
|
it exactly the state it reports as unreadable.
|
||||||
|
- B6's own acceptance test asks for a round trip through this parser. No
|
||||||
|
block form can pass it.
|
||||||
|
|
||||||
|
The flow form also satisfies commons' §5 "all values MUST be single-line",
|
||||||
|
and §11 requires parseable YAML rather than block YAML. Reading block needs
|
||||||
|
the structured reader (D1b); until then the constraint binds what we write.
|
||||||
|
|
||||||
|
A single source renders byte-identically to the one-entry form that shipped
|
||||||
|
before this took a list, which is what keeps all six goldens unmoved.
|
||||||
|
|
||||||
|
Refusing an unquotable value is the point of the check rather than a nicety:
|
||||||
|
`[{ id: x, resource: data, backup }]` is not a parse ERROR, it is a mapping
|
||||||
|
with a `backup` key nobody wrote. A silently wrong provenance record is
|
||||||
|
worse than a refused run, and repairing the value by quoting it would change
|
||||||
|
bytes the operator supplied. Validation, not repair — the same posture as
|
||||||
|
the filename-length gate. Every entry is checked, not only the first: a gate
|
||||||
|
that reads the head of a list is a gate the second entry walks past.
|
||||||
"""
|
"""
|
||||||
|
if not sources:
|
||||||
|
raise MaterializationError(
|
||||||
|
"a `sources` list must name at least one source — `sources: []` "
|
||||||
|
"reads as a measured absence when it is the absence of a "
|
||||||
|
"measurement",
|
||||||
|
code="sources_empty",
|
||||||
|
)
|
||||||
|
entries = []
|
||||||
|
for source in sources:
|
||||||
locator = _source_locator(source)
|
locator = _source_locator(source)
|
||||||
for label, value in (("id", source.id), ("resource", locator)):
|
for label, value in (("id", source.id), ("resource", locator)):
|
||||||
if _FLOW_UNSAFE_RE.search(value):
|
if _FLOW_UNSAFE_RE.search(value):
|
||||||
|
|
@ -220,7 +246,8 @@ def _render_sources(source: Source) -> str:
|
||||||
"record that parses cleanly into something no one wrote",
|
"record that parses cleanly into something no one wrote",
|
||||||
code="source_reference_unquotable",
|
code="source_reference_unquotable",
|
||||||
)
|
)
|
||||||
return f"[{{ id: {source.id}, resource: {locator} }}]"
|
entries.append(f"{{ id: {source.id}, resource: {locator} }}")
|
||||||
|
return f"[{', '.join(entries)}]"
|
||||||
|
|
||||||
|
|
||||||
def _render_concept_file(
|
def _render_concept_file(
|
||||||
|
|
@ -248,7 +275,7 @@ def _render_concept_file(
|
||||||
# profiles would append `sources` to every v0.1 bundle — additivity is a
|
# profiles would append `sources` to every v0.1 bundle — additivity is a
|
||||||
# property of what is constructed here, not of the emitter.
|
# property of what is constructed here, not of the emitter.
|
||||||
if "sources" in profile.frontmatter.order:
|
if "sources" in profile.frontmatter.order:
|
||||||
frontmatter["sources"] = _render_sources(manifest.source)
|
frontmatter["sources"] = _render_sources([manifest.source])
|
||||||
return f"---\n{profile.frontmatter.emit(frontmatter)}\n---\n\n{body}"
|
return f"---\n{profile.frontmatter.emit(frontmatter)}\n---\n\n{body}"
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
152
tests/test_multi_source_provenance.py
Normal file
152
tests/test_multi_source_provenance.py
Normal file
|
|
@ -0,0 +1,152 @@
|
||||||
|
"""SPEC 5.1 `sources` with more than one entry, and the form it is emitted in.
|
||||||
|
|
||||||
|
PM decision B6 asked for a list-taking `_render_sources` so a concept can
|
||||||
|
record more than one source. It also prescribed the BLOCK list as the emitted
|
||||||
|
form. The list is delivered; the block form is not, and this file carries the
|
||||||
|
measurement rather than the argument.
|
||||||
|
|
||||||
|
Three facts, each pinned by a test below:
|
||||||
|
|
||||||
|
- **Our own parser loses a block list entirely.** `parse_frontmatter` is
|
||||||
|
line-oriented and skips indented lines, so `sources:` followed by ` - id: a`
|
||||||
|
round-trips to an EMPTY value with every entry gone -- silently. A provenance
|
||||||
|
record we cannot read back is worse than one we never wrote.
|
||||||
|
- **The consumer's decoder reads the flow sequence and refuses the block one.**
|
||||||
|
`portfolio-optimiser`'s `decode_flow_value` accepts `[{ k: v }, { k: v }]` --
|
||||||
|
plural -- and classifies a block sequence as `UnreadableProvenance`. Emitting
|
||||||
|
block would hand the consumer that asked for multi-source exactly the state it
|
||||||
|
reports as unreadable.
|
||||||
|
- **The order's own acceptance test settles it.** It asks for a round trip
|
||||||
|
through our `parse_frontmatter` equivalent. No block form can pass that.
|
||||||
|
|
||||||
|
So the goal is delivered in the form that reaches a reader: one flow sequence of
|
||||||
|
N flow mappings, on one line. A single source stays byte-identical, which is
|
||||||
|
what keeps all six goldens unmoved.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from llm_ingestion_okf.errors import MaterializationError
|
||||||
|
from llm_ingestion_okf.manifest import FileSource, HttpSource, Source, SqlSource
|
||||||
|
from llm_ingestion_okf.materialize import _render_sources, parse_frontmatter
|
||||||
|
|
||||||
|
CATALOGUE = FileSource(id="golden-catalogue", root="fixture")
|
||||||
|
DATABASE = SqlSource(id="golden-db", connection_ref="OKF_GOLDEN_SQL_DB")
|
||||||
|
API = HttpSource(id="golden-api", base_url="https://golden.example.test")
|
||||||
|
|
||||||
|
|
||||||
|
def _document(sources: str) -> str:
|
||||||
|
return (
|
||||||
|
"---\n"
|
||||||
|
"type: dataset\n"
|
||||||
|
"title: Orders\n"
|
||||||
|
"generated: { by: process:okf-ingest, at: 2026-01-01T00:00:00Z }\n"
|
||||||
|
f"sources: {sources}\n"
|
||||||
|
"---\n"
|
||||||
|
"\nBody.\n"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# --- the rendered form ------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_one_source_renders_the_byte_identical_single_form() -> None:
|
||||||
|
"""The additivity arm. Every golden in the suite reads this line."""
|
||||||
|
assert _render_sources([CATALOGUE]) == "[{ id: golden-catalogue, resource: fixture }]"
|
||||||
|
|
||||||
|
|
||||||
|
def test_two_sources_render_as_one_flow_sequence_on_one_line() -> None:
|
||||||
|
assert _render_sources([CATALOGUE, DATABASE]) == (
|
||||||
|
"[{ id: golden-catalogue, resource: fixture }, "
|
||||||
|
"{ id: golden-db, resource: OKF_GOLDEN_SQL_DB }]"
|
||||||
|
)
|
||||||
|
assert "\n" not in _render_sources([CATALOGUE, DATABASE])
|
||||||
|
|
||||||
|
|
||||||
|
def test_sources_keep_the_order_they_were_given() -> None:
|
||||||
|
"""No sort. The order a manifest names its sources in is the manifest's
|
||||||
|
statement, and nothing here can recover it once reordered."""
|
||||||
|
forward = _render_sources([CATALOGUE, DATABASE, API])
|
||||||
|
reverse = _render_sources([API, DATABASE, CATALOGUE])
|
||||||
|
|
||||||
|
assert forward.index("golden-catalogue") < forward.index("golden-api")
|
||||||
|
assert reverse.index("golden-api") < reverse.index("golden-catalogue")
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_empty_source_list_is_refused() -> None:
|
||||||
|
"""`sources: []` is a provenance record naming no source: it reads as a
|
||||||
|
measured absence when it is the absence of a measurement."""
|
||||||
|
with pytest.raises(MaterializationError) as excinfo:
|
||||||
|
_render_sources([])
|
||||||
|
|
||||||
|
assert excinfo.value.code == "sources_empty"
|
||||||
|
|
||||||
|
|
||||||
|
# --- the round trip, which is the acceptance test ---------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_two_sources_round_trip_through_our_own_parser(tmp_path: Path) -> None:
|
||||||
|
rendered = _render_sources([CATALOGUE, DATABASE])
|
||||||
|
path = tmp_path / "concept.md"
|
||||||
|
path.write_text(_document(rendered), encoding="utf-8")
|
||||||
|
|
||||||
|
frontmatter = parse_frontmatter(path)
|
||||||
|
|
||||||
|
assert frontmatter["sources"] == rendered
|
||||||
|
assert sorted(frontmatter) == ["generated", "sources", "title", "type"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_block_form_round_trips_to_nothing(tmp_path: Path) -> None:
|
||||||
|
"""The NEGATIVE CONTROL, and the measured reason the block form is not
|
||||||
|
emitted. Two entries go in; an empty string comes back, and no error is
|
||||||
|
raised anywhere. This test must stay green: it is the evidence, not a
|
||||||
|
regression guard."""
|
||||||
|
path = tmp_path / "concept.md"
|
||||||
|
path.write_text(
|
||||||
|
"---\n"
|
||||||
|
"type: dataset\n"
|
||||||
|
"sources:\n"
|
||||||
|
" - id: golden-catalogue\n"
|
||||||
|
" resource: fixture\n"
|
||||||
|
" - id: golden-db\n"
|
||||||
|
" resource: OKF_GOLDEN_SQL_DB\n"
|
||||||
|
"---\n"
|
||||||
|
"\nBody.\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
|
||||||
|
frontmatter = parse_frontmatter(path)
|
||||||
|
|
||||||
|
assert frontmatter["sources"] == ""
|
||||||
|
assert "golden-db" not in "".join(frontmatter.values())
|
||||||
|
|
||||||
|
|
||||||
|
# --- the refusal applies to every entry, not only the first -----------------
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("position", [0, 1, 2], ids=["first", "middle", "last"])
|
||||||
|
def test_an_unquotable_locator_is_refused_in_any_position(position: int) -> None:
|
||||||
|
"""A gate that only reads the head of a list is a gate the second entry
|
||||||
|
walks past."""
|
||||||
|
sources: list[Source] = [CATALOGUE, DATABASE, API]
|
||||||
|
sources[position] = FileSource(id="broken", root="data, backup")
|
||||||
|
|
||||||
|
with pytest.raises(MaterializationError) as excinfo:
|
||||||
|
_render_sources(sources)
|
||||||
|
|
||||||
|
assert excinfo.value.code == "source_reference_unquotable"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("position", [0, 1], ids=["first", "last"])
|
||||||
|
def test_an_unquotable_id_is_refused_in_any_position(position: int) -> None:
|
||||||
|
sources: list[Source] = [CATALOGUE, DATABASE]
|
||||||
|
sources[position] = FileSource(id="a{1}", root="fixture")
|
||||||
|
|
||||||
|
with pytest.raises(MaterializationError) as excinfo:
|
||||||
|
_render_sources(sources)
|
||||||
|
|
||||||
|
assert excinfo.value.code == "source_reference_unquotable"
|
||||||
Loading…
Add table
Add a link
Reference in a new issue