feat(okf-v0.2): D5 — the v0.2 golden fixture, with okf_version in root frontmatter
Placement settled by catalog's own reading of upstream at the pinned commit
3fcbb9f: SS8:509-510 and SS12:773-775 both put `okf_version` in a bundle-root
`index.md` frontmatter block, and SS12 calls it the only place frontmatter is
permitted in an index. Catalog's spec says the opposite about the same file;
that divergence is theirs against upstream, and we conform to upstream.
The value never touches a profile. `OKF_V0_2.index.root_frontmatter` names the
key; the caller supplies the value through a new keyword-only
`root_frontmatter_values` mapping. That keeps V4/V-A5 intact - `okf_version`'s
value tracks the upstream Google version and belongs to catalog (E1), so a
constant here would claim a decision we do not own and would have to be chased
on every upstream release. In the fixture the value is fixture DATA
(`okf-version.txt`), not a literal in our source.
Ordering comes from the policy, not the caller's mapping: a dict preserves
insertion order, so two callers passing the same keys would otherwise emit
different bytes. A key the policy does not name is refused fail-fast, before
any disk mutation. Omitting the argument emits no block at all - SS12 is a MAY
and none of upstream's four reference bundles declares the key.
The block is written only when the index is CREATED, so a re-run into an
existing bundle stays byte-identical (A-E5).
Raw-byte assertions rather than parsed ones, on the committed fixture as well
as on fresh runs: catalog measured that a quoted value fails their shape regex
with exit 1 and that a BOM hides the marker while still exiting 0.
`yaml.safe_load` returns "0.2" either way and strips a BOM first, so a parsed
assertion masks exactly those two defects. Asserting the frozen fixture catches
what a self-comparison cannot - regenerating from a broken emitter moves both
sides together.
A-E6 is now placement-explicit (promised catalog in 99cf987), and separates the
two byte properties: BOM-free is a property of the file, unquoted is a property
of CATALOG'S GATE and not of OKF v0.2 - upstream's own SS12 example is quoted,
so their gate rejects the spec's canonical form.
README gains the upstream-version section it was missing; CLAUDE.md gains the
mechanism behind "no profile hard-codes an upstream version": a profile names a
key, a caller owns its value.
550 -> 559 tests. test_profile_threading's `OKF_V0_2.index is DEFAULT.index`
assertion is replaced rather than deleted: object identity was a proxy for "the
shipped profiles differ in no NAME-bearing field", which is what makes the
synthetic test profile necessary, so the guard now asserts that directly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dgkSPjkLpACjMayd9R5jx
This commit is contained in:
parent
ed08ac15e9
commit
2504011010
14 changed files with 470 additions and 34 deletions
|
|
@ -95,7 +95,12 @@ latest version supported as *stable*, so flipping that alias is the GA event, no
|
|||
a merge side effect.
|
||||
|
||||
Two invariants fall out: no profile hard-codes an upstream version, and no bundle
|
||||
declares a version its shape has not earned. Where upstream itself defers a
|
||||
declares a version its shape has not earned. The first has a mechanism, not just
|
||||
an intention: **a profile names a key, a caller owns its value.** `okf_version`
|
||||
is declared through `materialize_bundle(..., root_frontmatter_values=...)`
|
||||
because its value tracks the upstream Google version and belongs to catalog
|
||||
(decision E1) — a constant here would claim a decision we do not own, and would
|
||||
be the one thing to chase on every upstream release. Where upstream itself defers a
|
||||
contract — v0.2's attestation receipt and verdict wire formats — the format is
|
||||
supported and the unspecified runtime is not; it re-enters scope when upstream
|
||||
specifies it. Because "always latest" decays silently, the release checklist
|
||||
|
|
|
|||
42
README.md
42
README.md
|
|
@ -104,6 +104,48 @@ verification criteria:
|
|||
contract and fixture suite, not code —
|
||||
[plan](docs/plan/phase-4-node-half.md).
|
||||
|
||||
## Upstream OKF versions
|
||||
|
||||
The library targets the current latest version of Google's OKF. Support is
|
||||
**additive** — a new upstream version arrives as a new profile, never as a
|
||||
migration of an existing one — so upgrading the library does not change the
|
||||
bytes an existing profile emits.
|
||||
|
||||
| Profile | Contract | Status |
|
||||
|---|---|---|
|
||||
| `DEFAULT` | commons' ingest-spec §5 layer (OKF v0.1 semantics) | stable |
|
||||
| `STRICT_V1` | a consumer's ratified v0.1 contract | stable |
|
||||
| `OKF_V0_2` | OKF v0.2 | **provisional**, pre-release only |
|
||||
| `OKF_LATEST` | alias for the latest version supported as *stable* | currently `DEFAULT` |
|
||||
|
||||
`OKF_V0_2` ships first as a pre-release to a named pilot set and may change on
|
||||
their feedback without a deprecation cycle. Pin the versioned constant rather
|
||||
than `OKF_LATEST` unless you have explicitly opted into tracking; `OKF_LATEST`
|
||||
moves at general availability, which is a deliberate release event rather than
|
||||
a side effect of an upgrade.
|
||||
|
||||
Selecting a profile is keyword-only, so existing call sites are unaffected:
|
||||
|
||||
```python
|
||||
materialize_bundle(manifest, bundle_dir, ingested_at, profile=OKF_V0_2)
|
||||
```
|
||||
|
||||
A bundle may declare the version it targets. OKF v0.2 §12 makes this a MAY, and
|
||||
puts the declaration in the bundle-root `index.md`'s frontmatter block. The
|
||||
profile names the key; the **caller supplies the value**, because that value
|
||||
tracks the upstream version and is not this library's to decide:
|
||||
|
||||
```python
|
||||
materialize_bundle(
|
||||
manifest, bundle_dir, ingested_at,
|
||||
profile=OKF_V0_2,
|
||||
root_frontmatter_values={"okf_version": "0.2"},
|
||||
)
|
||||
```
|
||||
|
||||
Omit the argument and no frontmatter block is written. Offering a key the
|
||||
profile does not name is refused before anything is written to disk.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Verdict/feedback machinery from the method specification (stays in the
|
||||
|
|
|
|||
|
|
@ -752,24 +752,45 @@ output is the baseline. Materialize it twice with the *same* explicit
|
|||
undecided: whether a `DEFAULT`-written bundle may be re-run **in place** under
|
||||
`OKF_V0_2` — i.e. whether the predicate accepts both the v0.1 stamp and the
|
||||
v0.2 form. That is a D2 requirement question standing with the operator.
|
||||
- **A-E6** The root `index.md` declares `okf_version: 0.2` — **unquoted**, and
|
||||
the file carries **no UTF-8 BOM**. Both are load-bearing, not stylistic:
|
||||
catalog measured that a quoted value fails their shape regex `exit 1`, and
|
||||
that a BOM makes the marker invisible to them while still exiting `0`. The
|
||||
emitter renders values verbatim, so this is a property of the value we hand
|
||||
it. Verifying the raw bytes here is part of the expectation.
|
||||
- **A-E6** The root `index.md` declares `okf_version: 0.2` in a **frontmatter
|
||||
block opening the file** — the first bytes are `---\nokf_version: 0.2\n---\n\n`
|
||||
— with the value **unquoted** and **no UTF-8 BOM**.
|
||||
|
||||
**What A-E6 does not yet say is WHERE**, and as of 2026-07-26 that silence is
|
||||
load-bearing rather than cosmetic. Upstream §12 puts `okf_version` in the root
|
||||
`index.md`'s frontmatter block; catalog's own spec (`:55`, `:95`) says an
|
||||
`index.md` has no frontmatter and carries the marker as a body line. A bundle
|
||||
conforming to one is non-conformant against the other. Catalog verifies against
|
||||
upstream before touching their §6, and D5 is not frozen until they have. Their
|
||||
gate cannot settle it either way — the shape regex is unanchored and matches in
|
||||
both places — so a green TEST B measures the value, never the placement. A-E6
|
||||
becomes placement-explicit before the pilot receives it, whichever way the
|
||||
choice lands; an expectation that does not state what it expects is the defect
|
||||
class this very expectation is named after.
|
||||
**PLACEMENT — settled 2026-07-31, frontmatter.** Catalog read upstream
|
||||
themselves at the pinned commit `3fcbb9f` and reported §8:509-510 ("Index files
|
||||
contain no frontmatter, with one exception: a bundle-root `index.md` MAY carry
|
||||
an `okf_version` key") and §12:773-775 (declared "in a bundle-root `index.md`
|
||||
frontmatter block (the only place frontmatter is permitted in an `index.md`)").
|
||||
Their own spec (`:55`, `:95`) says the opposite about the same file; the
|
||||
divergence is real, it is **theirs against upstream**, and its resolution is
|
||||
their operator's. We conform to upstream. D5 is frozen on that.
|
||||
|
||||
**The two byte properties are expectations about DIFFERENT things, and A-E6
|
||||
must not conflate them** — doing so would be this expectation's own defect
|
||||
class:
|
||||
|
||||
- **Unquoted** is a property of **catalog's gate**, NOT of OKF v0.2. Their
|
||||
shape regex `/^\d+(\.\d+)*$/` reads the quotes as part of the value and
|
||||
exits 1. Upstream's §12:773 is the only line in the whole spec showing the
|
||||
key *with* a value, and it shows it **quoted** — so their gate rejects
|
||||
upstream's own canonical example. That is catalog's defect to resolve, not
|
||||
ours to work around, but while it stands, an unquoted value is what passes.
|
||||
We emit unquoted, and A-E6 expects unquoted **of catalog's gate**.
|
||||
- **BOM-free** is a property of the file we emit, full stop. A BOM makes the
|
||||
marker invisible to their gate *while still exiting 0* — a failure that
|
||||
reports success.
|
||||
|
||||
Both are asserted on **raw bytes**, never on a parsed value: `yaml.safe_load`
|
||||
returns `"0.2"` whether or not it was quoted and strips a BOM before any caller
|
||||
sees it, so a parsed assertion masks precisely these two defects.
|
||||
|
||||
**A green TEST B does not confirm the placement, and a red one does not
|
||||
identify it.** The shape regex is unanchored, so it matches in the frontmatter
|
||||
block and in a body line alike — green measures the value, never the position.
|
||||
Catalog sharpened the other direction on 2026-07-31: moving the marker into
|
||||
frontmatter *with* quoting, as §12 literally shows it, also yields FAIL. So a
|
||||
red run cannot distinguish "wrong quoting" from "wrong placement". Read TEST B
|
||||
as a statement about the value's shape only, in both directions.
|
||||
|
||||
**What would surprise us — report immediately:** any diff outside the frontmatter
|
||||
block (the profile seam leaks); `at` differing from the `ingested_at` they passed
|
||||
|
|
@ -1096,7 +1117,17 @@ everything after. **Done 2026-07-26** (`1215f98`, `7bc366b`).
|
|||
6. **D5** — the v0.2 golden fixture, generated by the code. `okf_version: 0.2`
|
||||
is declared only here, once 1–5 are green (conform first, claim after).
|
||||
**Unquoted** — the quoted form written here previously contradicted A-E6 and
|
||||
is exactly what catalog measured as `exit 1`.
|
||||
is exactly what catalog measured as `exit 1`. **Done 2026-07-31** (559
|
||||
tests): `examples/ingest-golden-okf-v0-2/`, emitted by the code and frozen
|
||||
byte-exact, with the marker in the root `index.md`'s frontmatter block per
|
||||
upstream §8/§12. The value reaches the emitter as
|
||||
`materialize_bundle(..., root_frontmatter_values={"okf_version": ...})` —
|
||||
keyword-only, defaulting to none, so every existing call site is untouched.
|
||||
The profile names the key (`OKF_V0_2.index.root_frontmatter`) and never
|
||||
carries the value, which is V4/V-A5 held intact: the value is catalog's (E1),
|
||||
and in the fixture it is fixture DATA (`okf-version.txt`), not a constant in
|
||||
our source. Offering a key the policy does not name is refused fail-fast
|
||||
before any disk mutation.
|
||||
7. **P1** — read-only sweep over real pilot bundles. Cheap, zero-risk, and the
|
||||
first point where a finding can come from outside our own assumptions.
|
||||
8. **Pre-release tag `v0.5.0a1`**, then **P2** (producer) and **P3** (gate).
|
||||
|
|
|
|||
6
examples/ingest-golden-okf-v0-2/expected-bundle/index.md
Normal file
6
examples/ingest-golden-okf-v0-2/expected-bundle/index.md
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
okf_version: 0.2
|
||||
---
|
||||
|
||||
Golden OKF v0.2 bundle: regional sales extracted from a CSV source.
|
||||
- [Regional Sales](ingest-sales.md)
|
||||
|
|
@ -0,0 +1,16 @@
|
|||
---
|
||||
type: dataset
|
||||
title: Regional Sales
|
||||
source_system: golden-v0-2-sales
|
||||
source_query: sales.csv
|
||||
ingested_at: 2026-07-16T12:00:00Z
|
||||
ingest_manifest: manifest@a83c86e8f847eca2
|
||||
generated: { by: process:llm-ingestion-okf, at: 2026-07-16T12:00:00Z }
|
||||
sources: [{ id: golden-v0-2-sales, resource: fixture }]
|
||||
---
|
||||
|
||||
| region | units | revenue |
|
||||
| --- | --- | --- |
|
||||
| nord | 412 | 88120 |
|
||||
| vest | 297 | 61540 |
|
||||
| sor | 158 | 33010 |
|
||||
4
examples/ingest-golden-okf-v0-2/fixture/sales.csv
Normal file
4
examples/ingest-golden-okf-v0-2/fixture/sales.csv
Normal file
|
|
@ -0,0 +1,4 @@
|
|||
region,units,revenue
|
||||
nord,412,88120
|
||||
vest,297,61540
|
||||
sor,158,33010
|
||||
|
1
examples/ingest-golden-okf-v0-2/ingested-at.txt
Normal file
1
examples/ingest-golden-okf-v0-2/ingested-at.txt
Normal file
|
|
@ -0,0 +1 @@
|
|||
2026-07-16T12:00:00Z
|
||||
18
examples/ingest-golden-okf-v0-2/manifest.json
Normal file
18
examples/ingest-golden-okf-v0-2/manifest.json
Normal file
|
|
@ -0,0 +1,18 @@
|
|||
{
|
||||
"manifest_version": 1,
|
||||
"source": {
|
||||
"type": "file",
|
||||
"id": "golden-v0-2-sales",
|
||||
"root": "fixture"
|
||||
},
|
||||
"bundle_summary": "Golden OKF v0.2 bundle: regional sales extracted from a CSV source.",
|
||||
"extractions": [
|
||||
{
|
||||
"id": "sales",
|
||||
"title": "Regional Sales",
|
||||
"query": "sales.csv",
|
||||
"okf_type": "dataset",
|
||||
"max_rows": 10
|
||||
}
|
||||
]
|
||||
}
|
||||
1
examples/ingest-golden-okf-v0-2/okf-version.txt
Normal file
1
examples/ingest-golden-okf-v0-2/okf-version.txt
Normal file
|
|
@ -0,0 +1 @@
|
|||
0.2
|
||||
|
|
@ -13,6 +13,7 @@ import hashlib
|
|||
import logging
|
||||
import re
|
||||
import unicodedata
|
||||
from collections.abc import Mapping
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
|
|
@ -307,6 +308,36 @@ def link_in_index(
|
|||
index_path.write_bytes(f"{prefix}{line}\n".encode())
|
||||
|
||||
|
||||
def _render_root_frontmatter(values: Mapping[str, str], *, profile: BundleProfile) -> str:
|
||||
"""The root index's frontmatter block (§8, §12), or "" when nothing is
|
||||
declared.
|
||||
|
||||
The policy names the keys and fixes their order; the caller supplies the
|
||||
values. Ordering by the POLICY rather than by the mapping is what keeps two
|
||||
callers passing the same keys from emitting different bytes — a dict
|
||||
preserves insertion order, and a golden fixture would then depend on the
|
||||
order a caller happened to build its argument in.
|
||||
|
||||
Values are written verbatim. `okf_version` must reach catalog's shape gate
|
||||
unquoted, so nothing here may add quoting; the golden fixture asserts that
|
||||
on raw bytes.
|
||||
"""
|
||||
unknown = sorted(set(values) - set(profile.index.root_frontmatter))
|
||||
if unknown:
|
||||
raise MaterializationError(
|
||||
f"root frontmatter key(s) {', '.join(repr(key) for key in unknown)} are not "
|
||||
f"named by the {profile.index.name} policy, which pins "
|
||||
f"{profile.index.root_frontmatter or '()'} — writing an unnamed key would "
|
||||
"put a value in a file no reader of this contract looks at",
|
||||
code="index_root_frontmatter_unexpected",
|
||||
)
|
||||
declared = [key for key in profile.index.root_frontmatter if key in values]
|
||||
if not declared:
|
||||
return ""
|
||||
lines = "".join(f"{key}: {values[key]}\n" for key in declared)
|
||||
return f"---\n{lines}---\n\n"
|
||||
|
||||
|
||||
def materialize_bundle(
|
||||
manifest_path: Path,
|
||||
bundle_dir: Path,
|
||||
|
|
@ -315,6 +346,7 @@ def materialize_bundle(
|
|||
allow_network: bool = False,
|
||||
http_get: HttpGet | None = None,
|
||||
profile: BundleProfile = DEFAULT,
|
||||
root_frontmatter_values: Mapping[str, str] | None = None,
|
||||
) -> IngestResult:
|
||||
"""Materialize a manifest's extractions into an OKF bundle (§5).
|
||||
|
||||
|
|
@ -335,10 +367,25 @@ def materialize_bundle(
|
|||
gate recognises, the concept filenames, and the index; it does NOT reach
|
||||
manifest type validation, which runs against `DEFAULT` (the two policies
|
||||
compare equal today). `STRICT_V1` is not supported here: its index policy
|
||||
sets `per_directory`, `entries_match_directory` and `root_frontmatter`,
|
||||
none of which this materializer honours.
|
||||
sets `per_directory` and `entries_match_directory`, neither of which this
|
||||
materializer honours.
|
||||
|
||||
`root_frontmatter_values` supplies the values for the keys the profile's
|
||||
index policy names — `okf_version` under `OKF_V0_2` (§8, §12). The split is
|
||||
deliberate: the profile names the key, the caller owns the value, because
|
||||
`okf_version`'s value tracks the upstream Google version and belongs to
|
||||
catalog (E1). Offering a key the policy does not name is refused fail-fast,
|
||||
before any disk mutation. Omitting the argument emits no block at all — §12
|
||||
is a MAY, and none of upstream's reference bundles declares it.
|
||||
|
||||
The block is written only when the index is CREATED. A re-run into an
|
||||
existing bundle leaves it untouched, which is what makes the second run
|
||||
byte-identical to the first (A-E5).
|
||||
"""
|
||||
validate_ingested_at(ingested_at)
|
||||
# Before any source access or disk mutation: a caller error here must not
|
||||
# leave a partially written bundle behind.
|
||||
root_frontmatter = _render_root_frontmatter(root_frontmatter_values or {}, profile=profile)
|
||||
manifest_file = Path(manifest_path)
|
||||
try:
|
||||
raw = manifest_file.read_bytes()
|
||||
|
|
@ -440,7 +487,7 @@ def materialize_bundle(
|
|||
for extraction in manifest.extractions
|
||||
}
|
||||
if not index_path.is_file():
|
||||
write_bytes(bundle, profile.index.name, manifest.bundle_summary + "\n")
|
||||
write_bytes(bundle, profile.index.name, root_frontmatter + manifest.bundle_summary + "\n")
|
||||
else:
|
||||
# Links whose target is an ingest-owned file removed this run MUST be
|
||||
# removed; all other links — curated and promoted — are preserved.
|
||||
|
|
|
|||
|
|
@ -22,7 +22,7 @@ from __future__ import annotations
|
|||
|
||||
import re
|
||||
from collections.abc import Collection, Mapping, Sequence
|
||||
from dataclasses import dataclass, field
|
||||
from dataclasses import dataclass, field, replace
|
||||
|
||||
# The one layer no profile may admit (ingest-spec §3): the promotion gate is
|
||||
# the only path into it. Compared case-insensitively, as both doors already do.
|
||||
|
|
@ -682,13 +682,25 @@ _OKF_V0_2_KEY_ORDER = (
|
|||
# unknown `type` value or on unknown additional keys, so an allowlist or a key
|
||||
# pattern here would put the profile in violation of the version it is named
|
||||
# for. `type` is required and is the only one (§4, §11).
|
||||
# - **It does not declare `okf_version`.** The index policy is DEFAULT's, which
|
||||
# binds `index.md` to the bundle root alone — upstream's shape (§8), and none
|
||||
# of upstream's four reference bundles declares the version at all (§12 makes
|
||||
# it a MAY). Declaring it is D5's, once, in the fixture: the value belongs to
|
||||
# catalog (E1), and WHERE it goes is open between upstream's root-index
|
||||
# frontmatter block and catalog's body-line convention. A profile that pinned
|
||||
# one of those today would be pinning the wrong one half the time.
|
||||
# - **It NAMES `okf_version` but never carries its value.** The value tracks the
|
||||
# upstream Google version and belongs to catalog (decision E1), so a constant
|
||||
# here would be this repo claiming a decision it does not own — and the one
|
||||
# that would have to be chased on every upstream release. The caller supplies
|
||||
# it (`materialize_bundle(..., root_frontmatter_values=...)`); this policy
|
||||
# fixes only the key and its position.
|
||||
#
|
||||
# WHERE it goes was open until 2026-07-31 between upstream's root-index
|
||||
# frontmatter block and catalog's body-line convention. Catalog verified
|
||||
# upstream themselves at the pinned commit `3fcbb9f` and reported §8:509-510
|
||||
# ("Index files contain no frontmatter, with one exception: a bundle-root
|
||||
# `index.md` MAY carry an `okf_version` key") and §12:773-775 ("in a
|
||||
# bundle-root `index.md` frontmatter block (the only place frontmatter is
|
||||
# permitted in an `index.md`)"). Frontmatter it is; their own spec diverges
|
||||
# from upstream here, and that divergence is theirs to resolve.
|
||||
#
|
||||
# Declaring it stays a MAY: none of upstream's four reference bundles carries
|
||||
# the key at all (catalog grepped `okf/bundles` and `okf/samples`: zero hits),
|
||||
# so omitting `root_frontmatter_values` emits no block.
|
||||
#
|
||||
# **Measured limitation (guard 0.2.0, 2026-07-26):** a bundle emitted under this
|
||||
# profile cannot be read back through a guard-gated import. The guard's T2
|
||||
|
|
@ -706,7 +718,10 @@ OKF_V0_2 = BundleProfile(
|
|||
required=frozenset({"type"}),
|
||||
),
|
||||
paths=DEFAULT.paths,
|
||||
index=DEFAULT.index,
|
||||
# DEFAULT's index in every respect but one: the root MAY carry `okf_version`
|
||||
# (§8, §12). Built with `replace` rather than restated so a later change to
|
||||
# the shared shape cannot drift between the two.
|
||||
index=replace(DEFAULT.index, root_frontmatter=("okf_version",)),
|
||||
ownership=OwnershipPolicy(actor="process:llm-ingestion-okf"),
|
||||
)
|
||||
|
||||
|
|
|
|||
|
|
@ -14,9 +14,11 @@ from pathlib import Path
|
|||
import pytest
|
||||
|
||||
from llm_ingestion_okf.materialize import materialize_bundle
|
||||
from llm_ingestion_okf.profiles import OKF_V0_2
|
||||
|
||||
EXAMPLES = Path(__file__).parent.parent / "examples"
|
||||
INGESTED_AT_NAME = "ingested-at.txt"
|
||||
OKF_VERSION_NAME = "okf-version.txt"
|
||||
|
||||
|
||||
def fixture_backed_get(fixture_dir: Path) -> object:
|
||||
|
|
@ -42,11 +44,34 @@ def materialize_case(case_dir: Path, out_dir: Path, monkeypatch: pytest.MonkeyPa
|
|||
allow_network=True,
|
||||
http_get=fixture_backed_get(case_dir / "fixture"), # type: ignore[arg-type]
|
||||
)
|
||||
elif case_dir.name.endswith("okf-v0-2"):
|
||||
# The declared version is fixture DATA, read from the case, never a
|
||||
# constant in this file. Its value tracks the upstream Google version
|
||||
# and belongs to catalog (E1); a literal here would be this repo
|
||||
# claiming a decision it does not own, and the golden would then have
|
||||
# to be chased on every upstream release.
|
||||
materialize_bundle(
|
||||
case_dir / "manifest.json",
|
||||
out_dir,
|
||||
ingested_at,
|
||||
profile=OKF_V0_2,
|
||||
root_frontmatter_values={
|
||||
"okf_version": (case_dir / OKF_VERSION_NAME).read_text(encoding="utf-8").strip()
|
||||
},
|
||||
)
|
||||
else:
|
||||
materialize_bundle(case_dir / "manifest.json", out_dir, ingested_at)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("case", ["ingest-golden-file", "ingest-golden-sql", "ingest-golden-http"])
|
||||
@pytest.mark.parametrize(
|
||||
"case",
|
||||
[
|
||||
"ingest-golden-file",
|
||||
"ingest-golden-sql",
|
||||
"ingest-golden-http",
|
||||
"ingest-golden-okf-v0-2",
|
||||
],
|
||||
)
|
||||
def test_golden_case_byte_for_byte(
|
||||
case: str, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
|
|
|
|||
|
|
@ -214,4 +214,14 @@ def test_synthetic_profile_is_not_a_shipped_profile() -> None:
|
|||
assert _SYNTHETIC.paths is not DEFAULT.paths
|
||||
assert _SYNTHETIC.index is not DEFAULT.index
|
||||
assert OKF_V0_2.paths is DEFAULT.paths
|
||||
assert OKF_V0_2.index is DEFAULT.index
|
||||
# `OKF_V0_2.index` stopped being DEFAULT's object at D5, which named
|
||||
# `okf_version` in its `root_frontmatter`. That is NOT the distinctness this
|
||||
# guard is about: every field that carries a NAME — the index filename and
|
||||
# the concept filenames — is still shared, so a threading bug that passed
|
||||
# the wrong profile would still emit identical bytes under either. Object
|
||||
# identity was only ever a proxy for that; assert the thing itself, or this
|
||||
# guard would go green on a change that leaves the instrument blind.
|
||||
assert OKF_V0_2.index.name == DEFAULT.index.name
|
||||
assert OKF_V0_2.index.link_template == DEFAULT.index.link_template
|
||||
assert _SYNTHETIC.index.name != DEFAULT.index.name
|
||||
assert _SYNTHETIC.paths.concept_suffix != DEFAULT.paths.concept_suffix
|
||||
|
|
|
|||
215
tests/test_root_frontmatter_emission.py
Normal file
215
tests/test_root_frontmatter_emission.py
Normal file
|
|
@ -0,0 +1,215 @@
|
|||
"""D5's emission half: the root `index.md` frontmatter block.
|
||||
|
||||
Upstream §8 binds `index.md` to "no frontmatter, with one exception: a
|
||||
bundle-root `index.md` MAY carry an `okf_version` key (§12)", and §12 puts the
|
||||
declaration "in a bundle-root `index.md` frontmatter block (the only place
|
||||
frontmatter is permitted in an `index.md`)". Catalog verified both quotations
|
||||
against the pinned commit `3fcbb9f` on 2026-07-31 and reported the same
|
||||
reading, so placement is settled: frontmatter, not a body line.
|
||||
|
||||
The value never comes from a profile. V4/V-A5 gives `okf_version`'s *value* to
|
||||
catalog (decision E1) and leaves this library the narrower obligation — name
|
||||
the key, express any value. So the profile pins `root_frontmatter` and the
|
||||
caller supplies the mapping.
|
||||
|
||||
Two assertions here read RAW BYTES rather than a parsed value, deliberately.
|
||||
Catalog measured that a quoted value fails their shape regex with `exit 1`, and
|
||||
that a UTF-8 BOM makes the marker invisible to them while still exiting `0`. A
|
||||
parsed assertion masks exactly those two defects: `yaml.safe_load` returns the
|
||||
string `"0.2"` whether or not it was quoted, and strips a BOM before the
|
||||
caller ever sees it.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from dataclasses import replace
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from llm_ingestion_okf.errors import MaterializationError
|
||||
from llm_ingestion_okf.materialize import materialize_bundle
|
||||
from llm_ingestion_okf.profiles import DEFAULT, OKF_V0_2
|
||||
|
||||
INGESTED_AT = "2026-07-16T12:00:00Z"
|
||||
|
||||
|
||||
def build_case(tmp_path: Path) -> Path:
|
||||
"""A minimal file-source manifest; the frontmatter block is what is under
|
||||
test, so the extraction is kept as small as it can be."""
|
||||
case = tmp_path / "case"
|
||||
fixture = case / "fixture"
|
||||
fixture.mkdir(parents=True)
|
||||
(fixture / "rows.csv").write_text("id,label\n1,alpha\n", encoding="utf-8")
|
||||
manifest = {
|
||||
"manifest_version": 1,
|
||||
"source": {"type": "file", "id": "root-fm", "root": "fixture"},
|
||||
"bundle_summary": "Root frontmatter case.",
|
||||
"extractions": [
|
||||
{
|
||||
"id": "rows",
|
||||
"title": "Rows",
|
||||
"query": "rows.csv",
|
||||
"okf_type": "dataset",
|
||||
"max_rows": 10,
|
||||
}
|
||||
],
|
||||
}
|
||||
(case / "manifest.json").write_text(json.dumps(manifest), encoding="utf-8")
|
||||
return case
|
||||
|
||||
|
||||
def test_root_index_carries_the_declared_key(tmp_path: Path) -> None:
|
||||
"""The block opens the file and the body follows it."""
|
||||
case = build_case(tmp_path)
|
||||
bundle = tmp_path / "bundle"
|
||||
|
||||
materialize_bundle(
|
||||
case / "manifest.json",
|
||||
bundle,
|
||||
INGESTED_AT,
|
||||
profile=OKF_V0_2,
|
||||
root_frontmatter_values={"okf_version": "0.2"},
|
||||
)
|
||||
|
||||
assert (bundle / "index.md").read_bytes() == (
|
||||
b"---\nokf_version: 0.2\n---\n\nRoot frontmatter case.\n- [Rows](ingest-rows.md)\n"
|
||||
)
|
||||
|
||||
|
||||
def test_the_value_is_unquoted_in_the_raw_bytes(tmp_path: Path) -> None:
|
||||
"""Catalog's shape regex `/^\\d+(\\.\\d+)*$/` sees the quotes as part of the
|
||||
value and exits 1. Asserted on bytes: a parsed check passes either way."""
|
||||
case = build_case(tmp_path)
|
||||
bundle = tmp_path / "bundle"
|
||||
|
||||
materialize_bundle(
|
||||
case / "manifest.json",
|
||||
bundle,
|
||||
INGESTED_AT,
|
||||
profile=OKF_V0_2,
|
||||
root_frontmatter_values={"okf_version": "0.2"},
|
||||
)
|
||||
|
||||
raw = (bundle / "index.md").read_bytes()
|
||||
assert b"okf_version: 0.2\n" in raw
|
||||
assert b'okf_version: "0.2"' not in raw
|
||||
assert b"okf_version: '0.2'" not in raw
|
||||
|
||||
|
||||
def test_the_file_carries_no_utf8_bom(tmp_path: Path) -> None:
|
||||
"""A BOM leaves the marker invisible to catalog's gate while still exiting
|
||||
0 — the failure that reports success, so it gets its own byte assertion."""
|
||||
case = build_case(tmp_path)
|
||||
bundle = tmp_path / "bundle"
|
||||
|
||||
materialize_bundle(
|
||||
case / "manifest.json",
|
||||
bundle,
|
||||
INGESTED_AT,
|
||||
profile=OKF_V0_2,
|
||||
root_frontmatter_values={"okf_version": "0.2"},
|
||||
)
|
||||
|
||||
raw = (bundle / "index.md").read_bytes()
|
||||
assert not raw.startswith(b"\xef\xbb\xbf")
|
||||
assert raw.startswith(b"---\n")
|
||||
|
||||
|
||||
def test_a_key_the_policy_does_not_name_is_refused(tmp_path: Path) -> None:
|
||||
"""`DEFAULT` names no root-frontmatter key, so offering one is a caller
|
||||
error rather than something to write. Fail-fast: refused BEFORE any disk
|
||||
mutation, or a rejected run would still leave a half-written bundle."""
|
||||
case = build_case(tmp_path)
|
||||
bundle = tmp_path / "bundle"
|
||||
|
||||
with pytest.raises(MaterializationError) as excinfo:
|
||||
materialize_bundle(
|
||||
case / "manifest.json",
|
||||
bundle,
|
||||
INGESTED_AT,
|
||||
profile=DEFAULT,
|
||||
root_frontmatter_values={"okf_version": "0.2"},
|
||||
)
|
||||
|
||||
assert excinfo.value.code == "index_root_frontmatter_unexpected"
|
||||
assert not bundle.exists()
|
||||
|
||||
|
||||
def test_omitting_the_values_emits_no_frontmatter(tmp_path: Path) -> None:
|
||||
"""§12 is a MAY, and none of upstream's four reference bundles declares the
|
||||
key at all (catalog grepped `okf/bundles` and `okf/samples`: zero hits). A
|
||||
profile that names the key must therefore still emit a bundle without it."""
|
||||
case = build_case(tmp_path)
|
||||
bundle = tmp_path / "bundle"
|
||||
|
||||
materialize_bundle(case / "manifest.json", bundle, INGESTED_AT, profile=OKF_V0_2)
|
||||
|
||||
raw = (bundle / "index.md").read_bytes()
|
||||
assert not raw.startswith(b"---")
|
||||
assert b"okf_version" not in raw
|
||||
|
||||
|
||||
def test_key_order_follows_the_policy_not_the_mapping(tmp_path: Path) -> None:
|
||||
"""The policy pins the order; a caller's dict ordering must not reach the
|
||||
file, or two callers passing the same keys would emit different bytes."""
|
||||
case = build_case(tmp_path)
|
||||
bundle = tmp_path / "bundle"
|
||||
profile = replace(
|
||||
OKF_V0_2,
|
||||
index=replace(OKF_V0_2.index, root_frontmatter=("okf_version", "bundle_profile")),
|
||||
)
|
||||
|
||||
materialize_bundle(
|
||||
case / "manifest.json",
|
||||
bundle,
|
||||
INGESTED_AT,
|
||||
profile=profile,
|
||||
root_frontmatter_values={"bundle_profile": "okf-v0-2", "okf_version": "0.2"},
|
||||
)
|
||||
|
||||
raw = (bundle / "index.md").read_bytes()
|
||||
assert raw.startswith(b"---\nokf_version: 0.2\nbundle_profile: okf-v0-2\n---\n\n")
|
||||
|
||||
|
||||
def test_the_frozen_golden_declares_the_version_gate_safely() -> None:
|
||||
"""D5's raw-byte guard, asserted on the COMMITTED fixture rather than on a
|
||||
fresh run.
|
||||
|
||||
`test_golden.py` already compares a run against these bytes, but that only
|
||||
proves the code agrees with itself: regenerate the fixture from a quoted or
|
||||
BOM-carrying emitter and both sides move together, silently. This asserts
|
||||
the shape of what we froze and hand to catalog's gate — the one thing a
|
||||
self-comparison cannot catch.
|
||||
"""
|
||||
raw = (
|
||||
Path(__file__).parent.parent
|
||||
/ "examples"
|
||||
/ "ingest-golden-okf-v0-2"
|
||||
/ "expected-bundle"
|
||||
/ "index.md"
|
||||
).read_bytes()
|
||||
|
||||
assert raw.startswith(b"---\nokf_version: 0.2\n---\n\n")
|
||||
assert not raw.startswith(b"\xef\xbb\xbf") # a BOM exits 0 while hiding the marker
|
||||
assert b'"' not in raw.split(b"---\n", 2)[1] # a quoted value exits 1
|
||||
assert b"\r\n" not in raw # LF-only, like every other emitted file
|
||||
|
||||
|
||||
def test_reingest_in_place_is_byte_identical(tmp_path: Path) -> None:
|
||||
"""A-E5. The second run goes into the FIRST run's directory — the shape the
|
||||
pilot runs — so the frontmatter block must survive §6 index maintenance
|
||||
rather than being appended a second time or dropped."""
|
||||
case = build_case(tmp_path)
|
||||
bundle = tmp_path / "bundle"
|
||||
call = dict(
|
||||
profile=OKF_V0_2,
|
||||
root_frontmatter_values={"okf_version": "0.2"},
|
||||
)
|
||||
|
||||
materialize_bundle(case / "manifest.json", bundle, INGESTED_AT, **call) # type: ignore[arg-type]
|
||||
first = (bundle / "index.md").read_bytes()
|
||||
materialize_bundle(case / "manifest.json", bundle, INGESTED_AT, **call) # type: ignore[arg-type]
|
||||
|
||||
assert (bundle / "index.md").read_bytes() == first
|
||||
Loading…
Add table
Add a link
Reference in a new issue