A concept named its source file by basename and, when segmented, carried a
`source_offset` into the text THIS LIBRARY extracted. Following that pointer
needed the corpus directory, the extractor and its exact transitive version --
none of which the bundle carries. Hand-walked on a real K2 concept: six steps,
four of them requiring knowledge from outside the bundle, to learn that a
requirement sits on pages 12-13 of a 20-page document.
The address is spec's: `sources: [{ resource, title }]`, where `resource` is
the dropped file's inbox-relative path (SPEC v0.2 5.1:303-306 -- "an absolute
URL, a bundle-relative path, or a path into a `references/` subdirectory").
The locator is ours, and it has to be: 5.1 has no field for a place within a
resource, and the pinned guard (1.3.0) rejects every route to putting one
inside a `sources` entry -- a non-allowlisted key by name, a nested flow list
as "scalar leaves only", and quoting as an unsupported form. So the locator is
top-level keys shaped like `source_offset`, and a path carrying a flow
terminator is refused fail-fast rather than mangled.
The unit table is built AT EXTRACTION, where the extracted text and the
original's structure are known to agree: pdf -> `source_pages` from
pdfplumber's own page numbers (a page that yielded no text does not renumber
the ones after it), xlsx -> `source_sheet` + `source_rows`, everything else ->
`source_lines`. `source_offset` stays.
Two measurements changed the design before it shipped. A `paragraphs` key for
docx would name a number the document does not have: `<w:p>` counts of
108/27/65/176/57 against converted-markdown lines of 75/33/67/144/63, not one
pair agreeing -- so the key is `source_lines` and says what it indexes. And an
empty spreadsheet row renders exactly like a table separator: the content-based
rule ate 8 empty rows on the K2 price sheet and reported its last row as 92
against a workbook that says 100. The separator is now found by position, and
`tomrad.xlsx` keeps that red.
One profile moves. `provenance` is a policy object, `None` everywhere but
`SEGMENTED_OKF_V0_2`; the other five shipped profiles are byte-identical.
K2 rebuilt from a frozen src copy: 629 concepts, 1108 files, name set identical,
0 ids moved, 479 files byte-identical, 629 changed and 0 lines removed anywhere.
629/629 now carry an address and a locator. New ref
`sha256-tree:665563a2f74423fcbcc8e4f0b0954ee73b73985ac0418de4f6987bd162a1f7c8`;
`2f82fcfe...` is stale. The pre-pass payload does not grow by one byte
(209 092 B before and after, 18 changed lines: the ref and eight per-concept
digests) -- because an excerpt carries the body, not the frontmatter, which is
also why the consumer still cannot cite "file X page 12" from a payload alone.
Report: docs/2026-09-08-proveniens-k2.md. 1339 tests, ruff and mypy clean.
Co-Authored-By: Claude <claude-opus-5>
243 lines
10 KiB
Python
243 lines
10 KiB
Python
"""The bundle contract as configuration (Phase 3, step 1).
|
|
|
|
What Phases 1 and 2 hard-coded about a valid bundle — the index name and its
|
|
managed-link shape, the three filename namespaces, the reserved layer, and the
|
|
frontmatter key order — becomes a profile object here. `DEFAULT` must express
|
|
exactly the existing contract: the Phase 1/2 suite and the golden fixtures are
|
|
the byte-level proof (assumption C1), and these tests pin the profile's own
|
|
behavior.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import dataclasses
|
|
|
|
import pytest
|
|
|
|
from llm_ingestion_okf.profiles import (
|
|
DEFAULT,
|
|
BundleProfile,
|
|
FrontmatterSchema,
|
|
TypePolicy,
|
|
)
|
|
|
|
|
|
def test_the_profile_and_every_policy_on_it_are_frozen() -> None:
|
|
"""A profile is a value, not a mutable registry.
|
|
|
|
A consumer holds DEFAULT while a run is in flight; a profile that could be
|
|
mutated mid-run would make the contract a moving target and the golden
|
|
determinism claim unprovable.
|
|
"""
|
|
for value in (DEFAULT, DEFAULT.types, DEFAULT.frontmatter, DEFAULT.paths, DEFAULT.index):
|
|
with pytest.raises(dataclasses.FrozenInstanceError):
|
|
value.name = "mutated" # type: ignore[misc]
|
|
|
|
|
|
def test_default_profile_pins_the_phase_1_2_names() -> None:
|
|
"""The names the two doors already write, now stated in one place."""
|
|
assert DEFAULT.index.name == "index.md"
|
|
assert DEFAULT.paths.concept_suffix == ".md"
|
|
assert DEFAULT.paths.ingest_prefix == "ingest-"
|
|
assert DEFAULT.paths.inbox_prefix == "inbox-"
|
|
assert DEFAULT.paths.import_prefix == "import-"
|
|
|
|
|
|
# --- the reserved layer (assumption C3) -----------------------------------
|
|
|
|
|
|
@pytest.mark.parametrize("reserved", ["verdict", "Verdict", "VERDICT"])
|
|
def test_no_type_policy_can_admit_the_reserved_verdict_layer(reserved: str) -> None:
|
|
"""C3: the verdict reservation is a spec invariant, not profile config.
|
|
|
|
Refused at CONSTRUCTION, not at use: a profile that admits `verdict` must
|
|
not exist at all, or the promotion gate stops being the only path into that
|
|
layer the moment someone passes the profile to a door.
|
|
"""
|
|
with pytest.raises(ValueError, match="verdict"):
|
|
TypePolicy(allowed=frozenset({"Concept", reserved}))
|
|
|
|
|
|
def test_a_closed_type_policy_without_the_reserved_layer_constructs() -> None:
|
|
policy = TypePolicy(allowed=frozenset({"Concept", "Guide", "Reference", "Release"}))
|
|
assert policy.rejection("Concept") is None
|
|
|
|
|
|
def test_default_type_policy_is_open_apart_from_the_reserved_layer() -> None:
|
|
"""Phase 1/2 accept any okf_type but `verdict`; DEFAULT must not narrow that."""
|
|
assert DEFAULT.types.allowed is None
|
|
for admitted in ("Concept", "note", "whatever-the-operator-wants"):
|
|
assert DEFAULT.types.rejection(admitted) is None
|
|
for reserved in ("verdict", "Verdict", "VERDICT"):
|
|
assert DEFAULT.types.rejection(reserved) is not None
|
|
|
|
|
|
def test_the_reserved_rejection_reproduces_both_doors_message_and_code() -> None:
|
|
"""The doors keep their own typed errors; the policy supplies the wording.
|
|
|
|
Door A raises ManifestError and Door B MaterializationError for the same
|
|
refusal, so the policy cannot raise — it hands back the reason and the
|
|
stable code, and each door frames it. These are the exact strings the
|
|
Phase 1/2 tests already assert on.
|
|
"""
|
|
rejection = DEFAULT.types.rejection("verdict")
|
|
assert rejection is not None
|
|
assert rejection.code == "okf_type_reserved"
|
|
assert f"okf_type {rejection.reason}" == "okf_type must not be 'verdict' (reserved layer)"
|
|
|
|
|
|
def test_a_closed_type_policy_rejects_an_off_enum_type_distinctly() -> None:
|
|
"""An off-enum refusal is not the reserved-layer refusal wearing its code."""
|
|
policy = TypePolicy(allowed=frozenset({"Concept", "Release"}))
|
|
rejection = policy.rejection("Guide")
|
|
assert rejection is not None
|
|
assert rejection.code == "okf_type_not_allowed"
|
|
assert "Concept, Release" in rejection.reason
|
|
|
|
|
|
# --- index policy ---------------------------------------------------------
|
|
|
|
|
|
def test_index_link_template_and_pattern_describe_the_same_shape() -> None:
|
|
"""Two fields that must agree, proven to agree rather than trusted to.
|
|
|
|
Rendering and parsing managed index links are separate operations (§6
|
|
appends, and maintenance rewrites), so the profile carries both a template
|
|
and a pattern. A round-trip is what keeps them from drifting apart.
|
|
"""
|
|
for label, target in [("Title", "ingest-x.md"), ("", "inbox-y.md")]:
|
|
line = DEFAULT.index.render_link(label, target)
|
|
match = DEFAULT.index.link_pattern.match(line)
|
|
assert match is not None, f"pattern does not match its own template output: {line!r}"
|
|
assert match.group("label") == label
|
|
assert match.group("target") == target
|
|
|
|
|
|
def test_default_index_link_is_the_phase_1_shape() -> None:
|
|
assert DEFAULT.index.render_link("Sales", "ingest-sales.md") == "- [Sales](ingest-sales.md)"
|
|
|
|
|
|
def test_index_link_pattern_is_anchored_to_the_whole_line() -> None:
|
|
"""§6 removal keys on the exact shape, never a bare substring — a curated
|
|
line that merely mentions a link must survive verbatim."""
|
|
assert DEFAULT.index.link_pattern.match("see - [Sales](ingest-sales.md) below") is None
|
|
|
|
|
|
# --- frontmatter emission -------------------------------------------------
|
|
|
|
|
|
def test_emission_is_the_ordered_prefix_then_the_sorted_tail() -> None:
|
|
"""Commons decision D1, carried over intact.
|
|
|
|
Keys named in the profile's order come first in that order; anything else
|
|
follows sorted. That is what makes a regeneration over the same data
|
|
byte-identical, which is what the proving consumer's hash registry needs.
|
|
"""
|
|
schema = FrontmatterSchema(order=("type", "title"))
|
|
emitted = schema.emit({"zeta": "3", "title": "T", "alpha": "1", "type": "Concept"})
|
|
assert emitted == "type: Concept\ntitle: T\nalpha: 1\nzeta: 3"
|
|
|
|
|
|
def test_emission_skips_ordered_keys_that_are_absent() -> None:
|
|
schema = FrontmatterSchema(order=("type", "title", "generated"))
|
|
assert schema.emit({"generated": "true", "type": "Concept"}) == "type: Concept\ngenerated: true"
|
|
|
|
|
|
def test_default_emission_reproduces_door_a_key_order() -> None:
|
|
"""Byte-identity with `_render_concept_file`'s §5 frontmatter."""
|
|
emitted = DEFAULT.frontmatter.emit(
|
|
{
|
|
"type": "Concept",
|
|
"title": "Sales",
|
|
"source_system": "crm",
|
|
"source_query": "sales.csv",
|
|
"ingested_at": "2026-07-03T12:00:00Z",
|
|
"ingest_manifest": "m@0123456789abcdef",
|
|
"generated": "true",
|
|
}
|
|
)
|
|
assert emitted.splitlines() == [
|
|
"type: Concept",
|
|
"title: Sales",
|
|
"source_system: crm",
|
|
"source_query: sales.csv",
|
|
"ingested_at: 2026-07-03T12:00:00Z",
|
|
"ingest_manifest: m@0123456789abcdef",
|
|
"generated: true",
|
|
]
|
|
|
|
|
|
def test_default_emission_reproduces_door_b_key_order() -> None:
|
|
"""Byte-identity with `render_inbox_concept`'s provenance layer.
|
|
|
|
Door B's six keys are a different subset of the same schema, and the
|
|
canonical order must reproduce BOTH doors — a single order that reordered
|
|
either one would change bytes already frozen in the fixtures.
|
|
"""
|
|
emitted = DEFAULT.frontmatter.emit(
|
|
{
|
|
"type": "Concept",
|
|
"title": "Notes",
|
|
"source_file": "notes.md",
|
|
"source_sha256": "abc",
|
|
"ingested_at": "2026-07-03T12:00:00Z",
|
|
"generated": "true",
|
|
}
|
|
)
|
|
assert emitted.splitlines() == [
|
|
"type: Concept",
|
|
"title: Notes",
|
|
"source_file: notes.md",
|
|
"source_sha256: abc",
|
|
"ingested_at: 2026-07-03T12:00:00Z",
|
|
"generated: true",
|
|
]
|
|
|
|
|
|
def test_source_query_is_the_only_collapsed_key() -> None:
|
|
"""§5 collapses whitespace runs for `source_query` alone (a multi-line
|
|
SELECT must render on one line); every other value is validated single-line
|
|
at load and emitted verbatim, so an operator's internal double space
|
|
survives."""
|
|
assert DEFAULT.frontmatter.collapsed_keys == frozenset({"source_query"})
|
|
emitted = DEFAULT.frontmatter.emit(
|
|
{"title": "two spaces", "source_query": "SELECT a\n FROM t"}
|
|
)
|
|
assert emitted == "title: two spaces\nsource_query: SELECT a FROM t"
|
|
|
|
|
|
def test_a_profile_is_assembled_from_its_policies() -> None:
|
|
"""The profile is its policies and nothing else — no behavior branches
|
|
outside what the object expresses.
|
|
|
|
`ownership` joined the original four at D2, and it is the same rule rather
|
|
than an exception to it: the ingest stamp differs per profile, so the
|
|
alternative was a version branch inside the collision gate. A policy on the
|
|
object is what keeps the emitter and the predicate changeable only together.
|
|
"""
|
|
assert {field.name for field in dataclasses.fields(BundleProfile)} == {
|
|
"types",
|
|
"frontmatter",
|
|
"paths",
|
|
"index",
|
|
"ownership",
|
|
# SEGMENTED_V1's capability, registered here for the same reason
|
|
# `ownership` was at D2: the alternative to a policy on the object is a
|
|
# profile branch somewhere else, which is exactly what this assertion
|
|
# exists to forbid. Defaulted to None, so the four shipped profiles
|
|
# construct unchanged and their bytes do not move.
|
|
"segmentation",
|
|
# Arm E's capability, on the same terms. Updated DELIBERATELY: this
|
|
# assertion pins the exact field set precisely so a field cannot arrive
|
|
# without someone deciding it should, and the red run it produced is
|
|
# the mechanism working rather than a regression. `None` means the
|
|
# profile does not have the capability; every shipped profile still
|
|
# constructs unchanged and no golden moved.
|
|
"renderers",
|
|
# O3's capability, and the same deliberate update again: a concept that
|
|
# points back at the document it came from is a contract question, so
|
|
# the alternative to a policy on the object is a door branching on which
|
|
# profile it was handed. Set on `SEGMENTED_OKF_V0_2` alone; `None`
|
|
# everywhere else, so five shipped profiles construct unchanged.
|
|
"provenance",
|
|
}
|