open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

Compare commits

base: open:v0.5.0a1
Branches Tags
open:main
open:v0.5.0a2
open:v0.5.0a1
..
compare: open:v0.4.0
Branches Tags
open:main
open:v0.5.0a2
open:v0.5.0a1

No commits in common. "v0.5.0a1" and "v0.4.0" have entirely different histories.
v0.5.0a1 ... v0.4.0

33 changed files with 127 additions and 5450 deletions
Download patch file Download diff file Expand all files Collapse all files

69
CHANGELOG.md
View file

@ -5,63 +5,6 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [0.5.0a1] — 2026-07-31
**Pre-release. PROVISIONAL surface.** Shipped to a named pilot set —
`portfolio-optimiser-claude`, the plugin marketplace catalog, and
`claude-code-llm-wiki` — so that real data can find what fixtures cannot. The
v0.2 surface may change on their feedback **without a deprecation cycle**.
Saying so up front is what buys the freedom to act on the feedback; discovering
it later is what would make a pilot a de-facto release. Do not pin this tag
outside the pilot set.
Support for a new upstream version is **additive — a new profile, never a
migration**. `OKF_LATEST` still points at `DEFAULT`; flipping it is the GA
event, not a side effect of this tag.
### Added
- **`OKF_V0_2` profile.** Google OKF v0.2 as a profile alongside `DEFAULT` and
`STRICT_V1`. It closes nothing: §14 forbids a conformant consumer to reject on
an unknown `type` value or on unknown additional keys, so `type` is the only
required key. It NAMES `okf_version` but never carries its value — that value
tracks the upstream Google version and belongs to catalog (decision E1), so
the caller supplies it via `materialize_bundle(..., root_frontmatter_values=…)`
and the profile fixes only the key and its position.
- **`root_frontmatter_values`** on `materialize_bundle`, keyword-only: the
mechanism behind "a profile names a key, a caller owns its value".
- **`sources` emitted as an inline flow sequence**, populated from the manifest.
- **The v0.2 golden fixture**, compared byte-for-byte like the others.
### Changed
- **`profile` on `materialize_bundle` is keyword-only**, so existing positional
call sites stay source-compatible.
- **`_is_ingest_owned` is profile-aware.** Ownership is a policy on the profile,
not a literal; recognition stays one-way.
### Fixed
- **`root_frontmatter` permitted a key without requiring it — the two had been
the same profile field.** Upstream says MAY where the field said MUST, so a
root index that legally omitted an optional key was rejected. `root_frontmatter`
now permits and orders; the new `root_frontmatter_required` requires.
`STRICT_V1` keeps requiring its three (unchanged for its consumer), `OKF_V0_2`
requires none, `DEFAULT` is untouched. Measured against the eight root indexes
available locally: 7 of 8 failed before, 0 of 8 after.
### Notes
- **`DEFAULT` and `STRICT_V1` are byte-stable across this release.** The emit
path is byte-identical; the golden suite is the check.
- **Known, deliberately unfixed:** `TypePolicy`'s closed branch does not strip
quotes from a declared type. All three call sites pass `DEFAULT.types`, and
both `DEFAULT` and `OKF_V0_2` set `allowed=None`, so the branch is
unreachable in shipped code — it fires only for a caller constructing
`STRICT_V1` and calling `rejection` directly. Fixing it would repair a write
path that never sees quotes, and would fix the meaning of a quote character
without a value model that can express one, in a line-oriented format.
## [0.4.0] — 2026-07-25
Phase 2. The bundle inbox (Door B) and external-bundle import (Door C) ship,
@ -147,17 +90,6 @@ and with them this library's first — and only ever — runtime dependency.
- Phase 2's binary extraction is **not** in this release: the `[extract]`
extra is declared but empty, and `pdf`/`docx`/`xlsx` therefore fail fast.
That is the one outstanding item from the phase, and it ships separately.
- **The ownership change above is an extension point, not a spec fix.** Filed
under "Fixed" because it stops silent data loss, but the spec owner
(`portfolio-optimiser-commons`) has since pointed out that removing *every*
ingest-stamped file is verbatim what ingest-spec v1 §5 mandates: v1 assumes
one manifest per bundle and defers multiple manifests feeding one bundle as
a named extension point. So this release implements that extension point and
outruns the frozen text rather than conforming to it. The mechanic itself is
not at risk — matching by manifest stem follows from the spec's own
`{stem}@{h}` stamp definition, since `{h}` changes on every content edit —
but the surrounding prose is queued for amendment in commons and is not
ratified. Treat multi-manifest bundles as ahead of the spec until it is.
- Exception `__cause__` preservation is now pinned by a conformance suite, one
test per fail-fast wrap site, alongside the existing `.code` suite.
@ -244,7 +176,6 @@ Phase 1 (Door A) implemented against the normative `ingest-spec.md` owned by
- The Door A public surface: `materialize_bundle` plus the typed error hierarchy
rooted in `IngestError`.
[0.5.0a1]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.4.0...v0.5.0a1
[0.4.0]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.3.2...v0.4.0
[0.3.2]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.3.1...v0.3.2
[0.3.1]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.3.0...v0.3.1

75
CLAUDE.md
View file

@ -49,11 +49,8 @@ are the library baseline. First consumer: `portfolio-optimiser-claude`.
2. **Phase 2 — Doors B/C (Python).** Bundle inbox and external-bundle
import, guard-gated.
3. **Phase 3 — Configurable bundle contract.** Types, layers, frontmatter
sets, index shape, and reserved-file policy become config instead of
constants; proving consumer is `claude-code-llm-wiki` (`strict-v1`
profile). Two consumers hold opposite postures on whether an index is
authored or directory-derived, so neither is a library invariant and
nothing here enumerates a directory unless the profile says derived.
sets, and reserved-file policy become config instead of constants;
proving consumer is `claude-code-llm-wiki` (`strict-v1` profile).
4. **Phase 4 — Node half (`node/`).** Zero-dependency Node/ESM package
(importable *and* CLI-invokable, vendorable per plugin — matching the
marketplace precedent) for the second-brain world: bundle check, index
@ -61,68 +58,8 @@ are the library baseline. First consumer: `portfolio-optimiser-claude`.
(docx/pdf/eml/html → md). Covers okr, linkedin-studio, ms-ai-architect,
and the marketplace catalog.
5. **Phase 5 — MCP as a way to populate a bundle.** Committed by the operator
2026-07-27; the design is open, the scope is not. Plan:
`docs/plan/mcp-bundle-population.md`. The fork that must be settled before
any code — whether we are the MCP **server** (an agent calls our doors as
tools) or an MCP **client** (a manifest source type that pulls from someone
else's server) — decides both the owner and the dependency question, and the
two are not variants of one feature.
The two halves share the OKF contract and fixture suite, **not code**.
### Upstream version policy (standing, non-negotiable)
**The library always supports the current latest version of Google OKF.** Set by
the operator 2026-07-26. Phases 1–3 were built against v0.1; v0.2 shipped
2026-07-25, so v0.2 support is committed work — not contingent on a consumer
asking for it. Plan: `docs/plan/okf-v0.2-alignment.md`.
Support is **additive, expressed as a new profile**, never a migration of
existing ones. This is what makes the policy sustainable instead of a recurring
crisis, and it is bounded by three facts that do not yield to it:
- `DEFAULT` states commons' ingest-spec §5 layer — its `generated` shape is
commons' call, raised there, never patched locally.
- `STRICT_V1` mirrors the proving consumer's ratified contract — changing another
repo's contract from here violates O2.
- `okf_version`'s *value* belongs to catalog (decision E1).
**Rollout is pilot-first.** A new upstream version reaches a small pilot set on a
pre-release tag and is revised on their feedback before general availability —
consumers testing real data find what fixtures cannot. `OKF_LATEST` means the
latest version supported as *stable*, so flipping that alias is the GA event, not
a merge side effect.
Two invariants fall out: no profile hard-codes an upstream version, and no bundle
declares a version its shape has not earned. The first has a mechanism, not just
an intention: **a profile names a key, a caller owns its value.** `okf_version`
is declared through `materialize_bundle(..., root_frontmatter_values=...)`
because its value tracks the upstream Google version and belongs to catalog
(decision E1) — a constant here would claim a decision we do not own, and would
be the one thing to chase on every upstream release. Where upstream itself defers a
contract — v0.2's attestation receipt and verdict wire formats — the format is
supported and the unspecified runtime is not; it re-enters scope when upstream
specifies it. Because "always latest" decays silently, the release checklist
carries an upstream-version re-check.
**Every upstream release runs `docs/upstream-okf-upgrade-runbook.md`.** Pin the
commit, enumerate the whole `okf/` tree, **read the shipped example bundles and not
only `SPEC.md`**, classify the diff, measure our exposure and each consumer's, plan
additively, pilot before GA, then inform every OKF-consuming repo. The runbook is
not optional and not a summary of good intentions: each step names the concrete
failure it prevents, and all of them are failures that happened during v0.1 → v0.2.
**This repo is a black box for its consumers.** The target cost of an upstream
release to a consuming repo is **a re-run, nothing more**: support is additive
(a new profile, never a migration), existing profiles stay byte-stable, new public
parameters are keyword-only with defaults so positional call sites stay
source-compatible, and consumer golden fixtures must not churn. The boundary is
stated every time rather than glossed — the library absorbs *shape* changes, not
upstream changes to content a consumer authored (v0.2's `timestamp` and
`# Citations` supersessions). For that class the deliverable is a measured exposure
report per consumer, sent before they ask.
Phase 4 preconditions (coordination, not unilateral moves):
- Lifts okr's reference implementations (`okf-check.mjs`, `okf-index.mjs`,
innboks libs) in agreement with okr and the marketplace catalog; the
@ -182,14 +119,6 @@ and fixtures, never code.
and `docs/oppstartsprompt.md` are LOCAL-ONLY (gitignored) — never commit
session state or internal briefs. No secrets, sober English prose, no
marketing language.
- **Consumer content stays at form level in public files.** Some consumers we
read are private (`claude-code-llm-wiki` is, pending an Anthropic ToS
assessment). Key names, counts, gate names, profile fields and contract shapes
are publishable; page bodies, full title or path lists from a consumer's
bundle, and Anthropic-derived prose are not. Findings about a private
consumer's data go back to them through coord, never as a file here. This
costs nothing — every question this library asks of a consumer is about shapes
and key sets — and it is not reversible once pushed.
- After `git commit`: push to Forgejo (`git push origin`) immediately.
Never GitHub.

69
README.md
View file

@ -93,9 +93,9 @@ verification criteria:
[plan](docs/plan/phase-1-door-a.md).
2. Bundle inbox and external-bundle import (Python), guard-gated —
[plan](docs/plan/phase-2-doors-b-c.md).
3. Configurable bundle contract (types, layers, frontmatter sets, index
shape, and reserved-file policy as configuration), enabling stricter
bundle profiles such as `strict-v1` —
3. Configurable bundle contract (types, layers, frontmatter sets, and
reserved-file policy as configuration), enabling stricter bundle
profiles such as `strict-v1` —
[plan](docs/plan/phase-3-configurable-contract.md).
4. A `node/` half: a zero-dependency Node/ESM package (importable and
CLI-invokable, vendored per consumer) providing bundle checking, index
@ -104,48 +104,6 @@ verification criteria:
contract and fixture suite, not code —
[plan](docs/plan/phase-4-node-half.md).
## Upstream OKF versions
The library targets the current latest version of Google's OKF. Support is
**additive** — a new upstream version arrives as a new profile, never as a
migration of an existing one — so upgrading the library does not change the
bytes an existing profile emits.
| Profile | Contract | Status |
|---|---|---|
| `DEFAULT` | commons' ingest-spec §5 layer (OKF v0.1 semantics) | stable |
| `STRICT_V1` | a consumer's ratified v0.1 contract | stable |
| `OKF_V0_2` | OKF v0.2 | **provisional**, pre-release only |
| `OKF_LATEST` | alias for the latest version supported as *stable* | currently `DEFAULT` |
`OKF_V0_2` ships first as a pre-release to a named pilot set and may change on
their feedback without a deprecation cycle. Pin the versioned constant rather
than `OKF_LATEST` unless you have explicitly opted into tracking; `OKF_LATEST`
moves at general availability, which is a deliberate release event rather than
a side effect of an upgrade.
Selecting a profile is keyword-only, so existing call sites are unaffected:
```python
materialize_bundle(manifest, bundle_dir, ingested_at, profile=OKF_V0_2)
```
A bundle may declare the version it targets. OKF v0.2 §12 makes this a MAY, and
puts the declaration in the bundle-root `index.md`'s frontmatter block. The
profile names the key; the **caller supplies the value**, because that value
tracks the upstream version and is not this library's to decide:
```python
materialize_bundle(
manifest, bundle_dir, ingested_at,
profile=OKF_V0_2,
root_frontmatter_values={"okf_version": "0.2"},
)
```
Omit the argument and no frontmatter block is written. Offering a key the
profile does not name is refused before anything is written to disk.
## Non-goals
- Verdict/feedback machinery from the method specification (stays in the
@ -157,29 +115,18 @@ profile does not name is refused before anything is written to disk.
## Requirements
Python 3.10+, and exactly one runtime dependency — the security boundary,
`llm-ingestion-guard>=0.2,<0.3`. Everything else is stdlib.
That guard is not on a package index yet, so **with pip, install it first** —
otherwise installing this package fails with `No matching distribution found
for llm-ingestion-guard`:
`llm-ingestion-guard>=0.2,<0.3`. Everything else is stdlib. Until that
package is published to an index, install it from its tag:
```
pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v0.2.0"
pip install "llm-ingestion-okf @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v0.4.0"
```
With uv, one command is enough — `uv pip install "llm-ingestion-okf @ git+…@v0.4.0"`
resolves the guard from the tag on its own, because uv reads the
`[tool.uv.sources]` entry in this project's `pyproject.toml` when it builds
from the source tree. Both paths were measured on 2026-07-25.
A git URL is a PEP 508 direct reference and pins one exact tag, so it is an
install-time *channel*, not the pin: the range above stays the declared
dependency — the built wheel carries `Requires-Dist:
llm-ingestion-guard<0.3,>=0.2` — and resolves normally once the package index
exists. The optional `[extract]` extra (pdf/docx/xlsx parsers) is not
populated yet. The planned Node half targets Node/ESM with zero npm
dependencies.
dependency and resolves normally once the package index exists. The optional
`[extract]` extra (pdf/docx/xlsx parsers) is not populated yet. The planned
Node half targets Node/ESM with zero npm dependencies.
## License

312
docs/phase-3-split-table.md
View file

@ -1,312 +0,0 @@
# Phase 3 split table — the proving consumer's 25 gates
Status: **reviewed by the proving consumer and by the operator (2026-07-25);
the port may begin.** Phase 3's plan requires this mapping before any porting.
Nothing in the consumer's repository is modified by this repository.
The proving consumer (`claude-code-llm-wiki`) answered the same questions from
their own code on 2026-07-25, crossing with this draft rather than responding
to it. Two independent readings of the same source agreed on every substantive
point: the required-key set as their code spells it, three reserved basenames,
`summary` as the only nullable key, the index structure, and the closed sets.
Their answer added four refinements, each folded in below and marked. Their
column assignments match this table's, with one wording difference resolved in
gate 5's row. A later round answered the four open questions outright — see
"Answered by the proving consumer" — and moved the required set from what their
code enforces to what their operator ratified.
Read from the consumer's working tree on 2026-07-25:
`tools/wiki_ingest/validate.py` and `tools/wiki_ingest/bundle.py` at commit
`9ee5a8e` (repository HEAD `20bcdd6`). Their `validate.py` is the single
entrypoint for every per-run gate, so this table is exhaustive over the gate
set as it stands at that commit — assumption C2 is satisfied only as long as
that remains true, and a gate added there after this date is not in this table.
## The three columns
| Column | Meaning | Goes where |
|---|---|---|
| Profile | Structural bundle contract: types, frontmatter keys, paths, reserved files, index shape | This phase, `BundleProfile` |
| Guard | Content safety: carriers, secrets, injection, active content | `llm-ingestion-guard` calls — never reimplemented here |
| Consumer | Domain logic: source pipelines, LLM contract, promotion, run accounting | Stays in the consumer's repository |
## Stage A — detection/acquisition
| # | Gate | Default class | Column | Note |
|---|---|---|---|---|
| 1 | `INGEST_LEDGER_DIVERGENCE` | ALERT (halts) | Consumer | Upstream re-poll reconciliation; no bundle structure involved. |
| 2 | `INGEST_CHANGELOG_PARSE` | ERROR | Consumer | Source-format parsing for one specific upstream. |
| 3 | `INGEST_EXPECTED_SET` | ALERT | Consumer | npm/changelog reconciliation against known deltas. |
| 4 | `INGEST_DOCS_CORPUS_SANITY` | ERROR | Consumer | Corpus size/shape thresholds for one named corpus. Its second half — "a concept must not derive from the changelog page" — is a layer-provenance rule expressed as a URL test; see open question Q3. |
| 5 | `INGEST_INPUT_INJECTION_SCAN` | WARN (ERROR when verbatim) | Guard | Carrier scan plus semantic lexicon on fetched input. Guard *column* means the functionality is the guard's and is never reimplemented — not that the call site moves here. The consumer keeps the call, as they state, since it sits in their acquisition stage. The disposition-aware escalation is Consumer policy over a Guard verdict; see Q1. |
## Stage B — LLM-output contract
| # | Gate | Default class | Column | Note |
|---|---|---|---|---|
| 6 | `LLM_STOP_REASON` | ERROR | Consumer | |
| 7 | `LLM_SCHEMA` | ERROR | Consumer | Frozen run-record schema plus summary length caps. |
| 8 | `LLM_OUTPUT_LINT` | ERROR | mixed | Decomposed below. |
| 9 | `LLM_GROUNDING_ANCHOR` | ERROR | Consumer | Anchor resolution against their concepts layer. |
| 10 | `LLM_SOURCE_TRACE` | ERROR | Consumer | Verbatim-quote and dropped-citation detectors. |
| 11 | `LLM_SEAM_LINKS` | ERROR | Consumer | Proposed links restricted to existing concept IDs. |
| 12 | `LLM_BUDGET` | ERROR | Consumer | Per-run call/token/cost caps. |
## Stage C — bundle conformance and publish
| # | Gate | Default class | Column | Note |
|---|---|---|---|---|
| 13 | `BUNDLE_FRONTMATTER_PARSEABLE` | ERROR | Profile | Every non-reserved `.md` opens with a complete fence. |
| 14 | `BUNDLE_TYPE_ENUM` | ERROR | Profile | Closed enum `Concept`/`Guide`/`Reference`/`Release`. |
| 15 | `BUNDLE_RESERVED_FILES` | ERROR | Profile | `log.md` required-absent; `index.md` structure. Three reserved basenames, not two — see "What STRICT_V1 must express". |
| 16 | `BUNDLE_YAML_STRICT` | ERROR | Profile | Key allowlist, required keys, and the StrictYAML scalar subset. |
| 17 | `BUNDLE_LAYER_PATH_TYPE` | ERROR | Profile | Path prefix ⇄ `layer` ⇄ `source_tier` ⇄ `type` agreement. |
| 18 | `BUNDLE_LINKS_INTERNAL` | ERROR | Profile | Every relative cross-link resolves inside the bundle. Consumer's caveat, accepted: link resolution assumes their layer layout, so it generalises only because `path_policy` owns the layout too. |
| 19 | `BUNDLE_LINKS_EXTERNAL` | WARN (never blocks) | Consumer | Liveness polling; needs network state this library does not collect. |
| 20 | `BUNDLE_URL_ALLOWLIST` | ERROR | mixed | Decomposed below. |
| 21 | `BUNDLE_INDEX_COMPLETE` | ERROR | Profile | Per-directory index exactness plus root-index metadata keys. |
| 22 | `BUNDLE_HASH_REGISTRY` | ERROR | Consumer | Their page-hash snapshotter must cover the concepts layer exactly. |
| 23 | `BUNDLE_SANITIZATION` | ERROR | Guard | Carrier-freedom of published text. |
| 24 | `BUNDLE_BLAST_RADIUS` | ERROR | Consumer | Per-version diff caps from their run manifest. |
| 25 | `BUNDLE_SECRET_SCAN` | ERROR | Guard | Credential patterns over bundle text and run artifacts. |
## The two mixed gates, decomposed
The plan asks for exactly one column per gate. Two gates do not honour that as
written, because each bundles checks belonging to different columns. Splitting
them is the honest mapping; the alternative — assigning the whole gate to its
dominant column — would quietly move the residue to the wrong side.
**Gate 8, `LLM_OUTPUT_LINT`** runs `_lint_text` over enrichment summaries and
verbatim release bodies:
| Check | Column |
|---|---|
| Off-allowlist URL | Consumer (their editorial pinning; see gate 20) |
| HTML tag in prose | Guard — active content is squarely the guard's territory at v0.3.0 |
| base64-like blob | Guard |
| Credential pattern | Guard |
| Consumer-directed imperative | Guard — this is the injection posture, expressed as their house rule |
| Injection lexicon and hex blob | Guard |
**Gate 20, `BUNDLE_URL_ALLOWLIST`** bundles two unrelated rules:
| Check | Column |
|---|---|
| URL host/prefix on the pinned allowlist | Consumer |
| Domain is new versus the previous bundle state | Consumer (drift detection over their own retained state) |
Neither half is Profile: a URL allowlist is editorial policy about which
sources a bundle may cite, not a statement about bundle structure. It is listed
as mixed only because the checks have different natures, and both land in the
same column — noted rather than merged, so a later reviewer can see it was
considered.
## What STRICT_V1 must express
The phase-3 plan sketched this profile before the validator was read. Two of
its claims did not survive contact and are corrected here; the plan file
carries the same corrections. One of the two was then corrected a second time
by the consumer's operator — the sequence is recorded rather than smoothed
over, because the end state is deliberately neither what the plan drafted nor
what their code says.
1. **Required keys are eight — settled by the consumer, not read off their
code.** Their `REQUIRED_KEYS` constant is four (`type`, `title`,
`description`, `timestamp`), and this table first corrected the plan down to
it. Asked whether four was the contract or an artefact of unfinished triage
(Q2), they measured the whole bundle and their operator ratified eight:
`type`, `title`, `description`, `timestamp`, `layer`, `source_tier`,
`source_url`, `source_sha`, present on 522/522 documents. The four-key
constant is their *emit-path* minimum, not their contract.
Two consequences are carried deliberately:
- **STRICT_V1 is stricter than the consumer's own validator.** Should they
ever ship a document class that legitimately lacks `source_sha` — a
hand-written Guide, say — this profile rejects what their validator
passes. The agreed resolution is to bump the profile, not to bend the
bundle, and they send the requirement before shipping such a class rather
than after.
- **The allowlist remains a different thing from the required set.**
`FRONTMATTER_KEY_ORDER` is eleven keys — the eight above plus `version`,
`date`, `summary` — and doubles as canonical emission order. `version` and
`date` are layer-specific (present on 351/522) and `summary` on 349/522,
so all three stay optional-when-absent and are validated against closed
sets **only when present**; a profile that required them would reject
pages the consumer ships today. So `frontmatter_schema` needs three
fields, not one: required, allowlisted, and ordered.
2. **Three reserved basenames, not two.** `index.md`, `log.md`, and
`README.md`. README is bundle documentation, reserved like `index.md`: it
carries no frontmatter, is skipped by the per-document gates, is never
listed in a generated index, and is still subject to the link, URL,
sanitization and secret gates. A two-value reserved-file policy
(required-present / required-absent) cannot express it; the third value is
"present-and-exempt-from-document-gates".
Beyond the frontmatter set, `BundleProfile` must carry:
- **Type policy:** closed enum, four values.
- **Nullable keys:** `summary` alone may be `null` — the single deliberate
escape from strings-only frontmatter.
- **Value shapes:** a string scalar, **or a non-empty list of strings**. This
is the same block-list shape Door C already refuses to round-trip through
this library's line-oriented parser, which is why an imported concept is
written verbatim. A profile that assumes scalars-only rejects the consumer's
own `Release` pages, and a parser that assumes scalars-only would silently
destroy them.
- **Key-name grammar:** `^[a-z_]+$`.
- **Scalar encoding:** the StrictYAML subset (JSON-double-quoted scalars), so
a regeneration over the same data is byte-identical.
- **Emission order — carry this over intact:** the ordered prefix above,
followed by any remaining keys **sorted**. This is the door-A model settled
by commons decision D1, and it is what makes regeneration byte-identical.
A profile that normalises key order any other way breaks the consumer's
`BUNDLE_HASH_REGISTRY` gate, which is what made that decision matter.
- **Layer/path policy:** `concepts/` ⇄ layer `concepts`, source_tier
`official-docs`, any type except `Release`; `releases/` ⇄ layer `releases`,
source_tier `changelog-derived`, type `Release` exactly. Their constants
spell these `bundle/concepts` and `bundle/releases` because their validator
walks a repository whose bundle lives in `bundle/`. That prefix is where
*they* keep the bundle, not part of the contract — see "No placement field"
below.
- **Index policy**, which differs from this library's default in four ways:
one index per *directory* rather than one per bundle; entries formatted
`* [Title](link) - description` rather than `- [label](target)`; index
contents must match the directory exactly, both directions; and the root
index carries frontmatter pinning `okf_version`, `bundle_profile`,
`okf_spec_commit`.
## No placement field — a field the profile must NOT grow
The catalog reported on 2026-07-25 (their commit `3ccb5f6`,
`docs/okf-second-brain/spec.md` §1 and §8) that bundle placement is
**unmandated**, measured against four consumers rather than taken on report:
exactly one installs a bundle anywhere; for the other three the bundle
directory is a caller-supplied argument per run, with no default, no discovery
and no `$HOME` expansion. Discovery, where it happens, keys on the §3 root
marker (`index.md` plus `okf_version`), never on a path.
Three consequences for `BundleProfile`, recorded here because this phase's
non-goal is precisely "do not make the profile too narrow for the Node-world
conventions in phase 4":
- No root-location or install-path field. There is no path form to express.
- "No installation root" is a conforming consumer shape, so a profile that
assumes every consumer has one would be too narrow in exactly the warned way.
- Installing and writing are separate axes: a consumer may write into a bundle
it never placed, so write policy must not be tied to a placement concept.
Every path in this document is therefore bundle-relative, and the bundle
directory stays what it is today in both doors — an argument.
The index-policy divergence is the largest single item. Under the default
profile the observable output must stay byte-identical (assumption C1, proven
by the golden suite), so `index_policy` has to express both shapes rather than
generalising one into the other.
## Answered by the proving consumer
All four questions were answered on 2026-07-25, each measured or read out of
their code that day rather than recalled. **No row in the mapping moved**; the
reasons under three of the four are sharper than the ones this table drafted.
- **Q1 — gate tiers stay Consumer/Guard.** This repository's position was
accepted, and the escalation turns out not to be a document-class rule at
all: gate 5 escalates WARN to ERROR when the per-version *record* carries
`disposition == "verbatim"` (`validate.py:528-540`), i.e. when the LLM
rewrite layer was skipped and the input lint is the only filter left. That is
run state, not bundle shape — a profile could not see it even if it wanted
to. A profile describes structure; it does not set security tiers.
- **Q2 — eight, ratified by their operator.** Folded into "What STRICT_V1 must
express" above, together with the consequence both sides accepted.
- **Q3 — Consumer, for a sharper reason than "URL predicate".** The check is
`is_changelog_url()` (`docs_corpus.py:181-184`): it lowercases the last URL
segment and tests membership in `("changelog", "release-notes")` — a
vendor-specific slug list for one upstream. Two details worth carrying: the
gate is **not** scoped to `concepts/` at all (it walks every non-reserved
document across both layers, `validate.py:480-492`, despite a rationale
string that says "concepts layer"), and it spares their own release pages
only because their `source_url` ends in `CHANGELOG.md`, whose last segment is
`changelog.md` rather than `changelog`. That is a coupling, not a design. A
structural profile could express "layer X may not carry source_tier Y"; it
could not express this.
- **Q4 — both stay Consumer.** Gate 19 sits in their `NEVER_BLOCKING_GATES`
(`validate.py:86-88`) by construction, precisely because liveness is a flake
magnet: a profile that owned it would inherit a gate that can never block.
Gate 22 is their provenance model, the one commons decision D1 settled in
their favour, and it is why the ordered-prefix emission matters. The profile
gets no notion of external state from either.
Two confirmations arrived unasked and are recorded because later work leans on
them:
- **Assumption C2 holds.** `validate.py` is still the single entrypoint for
every per-run gate, and the registry at lines 50-78 is the whole set — 5
`INGEST_*`, 7 `LLM_*`, 13 `BUNDLE_*` = 25. A change reaches this repository
as a message before it can be read out of their code.
- **The index shape is not negotiable downward**, which makes "express both
shapes rather than generalise one into the other" the right call: one index
per directory, exactly one `# ` heading, entries `* [Title](link) -
description` and nothing else, directory and index matching in both
directions, and a root index carrying `okf_version` / `bundle_profile` /
`okf_spec_commit` frontmatter in that key order.
## The index policy, and the conflict that shaped it
The convention owner reported on 2026-07-25 that the second-brain spec's §3
mandates an `index.md` at every directory level, that `IndexPolicy` as it then
stood could not express that, and — the load-bearing part — that an index is an
**authored** count of a directory's children and never a filesystem lookup. On
their reading, an index reader or validator that enumerates a directory to
build or check an index has implemented the wrong contract, with a silent
failure mode: code written to the wrong reading passes every test one would
think to write.
That was checked against the proving consumer before it was adopted, and the
two consumers turn out to be **directly opposed on exactly this point**:
- `validate.py:1081-1120`, gate `BUNDLE_INDEX_COMPLETE` (severity ERROR),
builds its expected set by enumerating the directory and fails unless the
index matches it exactly in both directions.
- `bundle.py:498-567` writes every index from what a tree walk finds, and its
own docstring calls `index.md` machine-generated.
So one consumer mandates the derivation the other calls the wrong contract.
Neither is incoherent inside its own spec, and this repository is not the venue
to adjudicate between them. What it settles is narrower and sufficient:
**authored-versus-derived cannot be a library invariant in either direction.**
It is a policy field (`entries_match_directory`), as are per-directory scope,
the heading requirement, whether prose is admitted, and the root-index key set.
Three consequences that outlive this phase:
1. **Nothing in this library enumerates a directory.** The caller supplies the
listing; `IndexPolicy.violations` refuses one when the profile's index is
authored, and refuses to run without one when it is derived. The
convention owner's construction rule is enforced at the call site rather
than documented and hoped for.
2. **Root and nested indexes are asymmetric**, confirmed independently in both
consumers: the root carries frontmatter (`okf_version`/`bundle_profile`/
`okf_spec_commit` for the wiki, `okf_version`/`okf_layout` for the catalog)
and nested indexes carry none. Different key sets, same shape — which is
why it is a profile field and not a constant.
3. **A per-entry description is template-level, not a separate requirement.**
The wiki requires `* [Title](link) - description`; the catalog requires
prose in the index and no per-entry description. A profile that wants none
simply does not name `{description}` in its template.
`DEFAULT` keeps upstream's root-only index and judges nothing, for the same
measured reason it carries no required frontmatter key set: upstream OKF binds
`index.md` to the bundle root alone, so a default demanding one per level would
declare upstream-conforming bundles invalid.
## What this table does not do
It does not port anything. Per the plan, porting starts only after this
mapping is reviewed, and the Guard column must contain no rows implemented in
this repository — the phase-2 grep gate over guard functionality still applies
unchanged.
Both reviews are in as of 2026-07-25 — the proving consumer's (above) and the
operator's — so the port may begin. The Guard-column constraint is unaffected
by either.

76
docs/plan/execution-order.md
View file

@ -8,14 +8,12 @@ own doc:
- `docs/plan/phase-1-door-a.md` — DONE, shipped at v0.3.2.
- `docs/plan/phase-2-doors-b-c.md`
- `docs/plan/phase-3-configurable-contract.md`
- `docs/plan/okf-v0.2-alignment.md` — a policy track, not a phase; slots between
Phases 3 and 4 for the reason given below.
- `docs/plan/phase-4-node-half.md`
Headline order: **Stage 0 (guard gate) → Phase 2 → Phase 3 → OKF v0.2 alignment
decisions → Phase 4 code**, with Phase 2 split guard-independent-first and
Phase 4's *coordination* started in parallel from the beginning. The quality
argument for this order is below — it is not merely the roadmap numbering.
Headline order: **Stage 0 (guard gate) → Phase 2 → Phase 3 → Phase 4 code**,
with Phase 2 split guard-independent-first and Phase 4's *coordination* started
in parallel from the beginning. The quality argument for this order is below —
it is not merely the roadmap numbering.
## Stage 0 — Guard readiness (gate; do first, cheap)
@ -35,36 +33,6 @@ persist-gate design must know now, not at the integration step.
Outcome recorded in STATE before Phase 2 code starts.
### Stage 0 revisited — measuring guard 0.3.1 before widening the pin
The pin held is `>=0.2,<0.3`, which can only ever resolve `v0.2.0`. The guard repo
gates its own 1.0 on our fixture suite being measured against `v0.3.1`, so this
measurement is owed outward and is that repo's only named blocker. Order is
*measure first, widen after*: widening the range before measuring would ship a
window admitting a version the fixtures never ran against.
Procedure — do not vary it, each step exists because of a specific failure mode:
1. Scratch venv **outside** `.venv`: install the package `--no-deps`, then the
guard from the `v0.3.1` tag. Do not touch `pyproject.toml`.
2. **Assert the resolved version** via `importlib.metadata` *before* running
anything, and carry it in the reported result. An unmodified tree resolves
`0.2.0` and runs green — a true claim about 0.2.0 published under a heading
that says 0.3.1. This step is what makes the result honest.
3. Run `tests/test_guard_adapter.py` (19 tests, real scanner) and the full suite.
Watch percent-escapes specifically: the slugger builds filenames from titles,
and the guard has recorded that as its most likely false positive. Report raw
URLs, not counts.
4. **Send the result whichever way it goes**, and state explicitly that 19
fixture tests are not a corpus — their gate is "does 0.3.1 change our
verdicts", not "how often does the false positive fire".
5. Green → one release: `>=0.2,<0.4`, uv tag `v0.3.1`, CLAUDE.md and README
updated together. Red → no bump, and the finding goes back to the guard.
Until this lands, consumers of v0.4.0 resolve guard 0.2.0 (`pyproject.toml:25`).
That is a silent under-defend: no build fails, so nothing surfaces it. The
release in step 5 is what closes it.
## Phase 2 — Doors B/C (split: guard-independent first)
Internal order follows the phase-2 doc's TDD steps, deliberately:
@ -100,38 +68,6 @@ policy differs per consumer and becomes configurable in Phase 3." This is where
the contract becomes explicit and frozen — the precondition for a clean Node
port.
## OKF v0.2 support — between 3 and 4, and it is real code
Everything through Phase 3 targets OKF v0.1. Upstream published v0.2 on
2026-07-25, and the standing operator policy is that the library always supports
the current latest OKF version (see `CLAUDE.md`). So this is a committed
implementation track, not a decision round:
`docs/plan/okf-v0.2-alignment.md`.
**Why it must land before Phase 4 code.** Phase 4 ports a *frozen, explicit*
contract and uses the shared fixture suite as the cross-runtime conformance
oracle. Freezing that contract before v0.2 is supported means freezing a v0.1
shape into two runtimes instead of one, and the fixture suite would then certify
the drift rather than catch it. Worse, the Node half would need the same v0.2
work done twice, in two languages, against a contract that had already been
declared final. The expensive version of this mistake is discovered after the
Node half exists.
**Why it is not a phase of its own.** It adds no new door and no new stage — it
adds a *profile*, which is precisely what Phase 3 built the seam for. Support is
additive: `DEFAULT` (commons') and `STRICT_V1` (the wiki's) keep their contracts
untouched, and v0.2 arrives as a third profile plus an `OKF_LATEST` alias. This
is the first time the Phase 3 abstraction is forced by something outside this
repo rather than by a second consumer, which is the better test of it.
**Order inside the track.** Characterization and audit first — they add tests and
change no behavior. Then the frontmatter model that can carry block lists
(`sources`, multi-verifier `verified`), then the profile, then the golden
fixtures that make the support claim testable. The `okf_version: 0.2`
declaration comes last — **unquoted**, per A-E6 — once the shape has earned it. Commons' answer on
`DEFAULT`'s `generated` runs in parallel and no longer blocks: the additive
design took it off the critical path.
## Phase 4 — Node half (two clocks)
**Coordination clock — start early, parallel to Stages 0–3, but only the
@ -181,8 +117,6 @@ These are the objective checks that a stage is done and the next may begin:
second-brain spec).
4. **Phase 4 code start:** each shape-independent coordination step has a
recorded sign-off; the contract-shape agreement is signed off *after* the
Phase 3 split-table, not before; and the OKF v0.2 open questions V1–V5 are
each answered or explicitly deferred *in writing*, so the frozen contract
states its relation to upstream on purpose rather than by omission.
Phase 3 split-table, not before.
5. **Throughout:** `mypy --strict src/`, `ruff check .`, `ruff format --check .`
clean; boundary grep-gate empty.

107
docs/plan/mcp-bundle-population.md
View file

@ -1,107 +0,0 @@
# Phase 5 — MCP as a way to populate a bundle
**Status: scoped, not designed.** Committed by the operator 2026-07-27. This
document records the commitment, the fork it turns on, and the constraints any
design has to satisfy. It deliberately does not choose an implementation:
the fork below decides who owns the work, and choosing it wrongly is expensive
in a way writing it later is not.
## The fork — settle this first
"Support MCP to populate a bundle" names two different pieces of work. They
share a protocol and nothing else.
**(A) We are the MCP server.** An agent connects to us and calls tools —
`add_concept`, `list_bundle`, `generate_index` — and the bundle fills up through
our doors. The natural home is **Door B**, which already answers "here is
content, make it OKF": MCP would be a protocol surface over an existing door
rather than a new ingestion path.
**(B) We are an MCP client.** A manifest names an MCP server as a source, the
same way it names `file`/`sql`/`http` today, and Door A pulls concepts from it.
**Default reading, for confirmation rather than assumption:** the phrase "to
populate a bundle" points at (A). Recorded as a reading, not a decision — the
operator has not been asked which one, and the two have different owners.
## Why the fork decides the owner
(B) adds a **source type**, and the source type set is normative in commons'
`ingest-spec.md`. This repo implements that spec and does not edit it. So (B)
cannot start here: it starts as a request to commons, and it lands here only
after they have specified it. (A) adds no source type and no spec surface — the
concepts arrive through Door B exactly as dropped files do — so (A) is ours to
design.
That asymmetry is the whole reason this document exists before any code.
## Constraints any design inherits
None of these are negotiable by this feature, and each one has already refused
something:
1. **Exactly one runtime dependency, ever** (`llm-ingestion-guard`), enforced by
a packaging test. An MCP SDK would be the second. Either the protocol is
hand-rolled over stdlib JSON-RPC on stdio, or it lives behind an optional
extra the way binary parsers do in `[extract]` — never in the core install.
2. **No model calls anywhere in the run path.** MCP is model-facing, and that is
fine in exactly one direction: under (A) the model is the *caller* and we
stay deterministic tools. A design where this library calls a model to decide
what to write is out of scope, and would be a different product.
3. **The persist gate still applies.** Content arriving over MCP is untrusted by
construction. It goes through Door B's injected guard gate like every other
dropped file; MCP is a transport, never a reason to bypass a gate. Security
stays the guard's, as always.
4. **Network access is an explicit per-run opt-in.** A server listening on a
socket, or a client dialing one, is network. The existing fail-fast refusal
is the baseline, not something to reinvent.
5. **Determinism.** `ingested_at` is a required argument with no wall-clock
default. A protocol handler does not get to invent one because a call
arrived "now".
## Open questions
- **Q1 — (A) or (B), or both in order?** Blocks everything else.
- **Q2 — If (A): what is the tool surface?** Door B's public surface is
`process_inbox` over a directory. An MCP tool taking one concept at a time is
a different granularity, and the collision gate and index generation are
currently per-run rather than per-concept. This is the real design work.
**Prior art, measured 2026-07-27 at the pinned upstream commit.** Upstream has
already built this surface in ADK-specific form, and it is worth reading before
we draw our own. `sources/base.py` (38 lines, stdlib only) defines a `Source`
ABC; `tools/source_tools.py` wraps its methods as model-callable tools, which
`agent.py` registers. Three things in it are design signals rather than
incidental:
- **`list_concepts` and `read_concept` are abstract; `sample_rows` is optional
with a `None` default; `find` ships a default implementation.** So the
surface's obligations are *enumerate* and *read*, and sampling is advisory.
A tool surface that made sampling mandatory would be claiming more than the
problem needs.
- **It is per-CONCEPT** (`read_concept(ref)`), which is the granularity this
question is about. That an independent implementation landed on per-concept
is evidence that the per-run shape — Door B's collision gate and index
generation — is the side that has to give, not the tool surface.
- **What not to copy.** `read_concept` returns `dict[str, Any]`, untyped
because a model interprets it; we render bytes and cannot. And `find` is a
linear scan over `list_concepts()`, which assumes total enumeration is cheap
— colliding directly with this library's rule that nothing enumerates unless
a profile says derived.
Read, not vendored. The point is the shape of the question they answered.
- **Q3 — Transport.** stdio is the vendorable, zero-dependency answer and
matches how the marketplace consumes things. HTTP would be a second network
surface with its own gate question.
- **Q4 — Which half?** Phase 4 puts a Node/ESM half next to the Python one. MCP
servers in the plugin world are overwhelmingly Node. If (A) lands in Python
only, the second-brain consumers cannot use it; if it lands in both, the
"share contract and fixtures, not code" rule applies to the tool surface too.
- **Q5 — Does an MCP-populated bundle differ observably** from one Door B wrote
from files? It must not, or the transport has leaked into the artifact.
## Non-goals
- Reimplementing any security check the guard owns.
- Calling a model from this library, under any transport.
- Editing commons' source-type set locally to make (B) possible sooner.

1364
docs/plan/okf-v0.2-alignment.md
View file

File diff suppressed because it is too large Load diff

52
docs/plan/phase-3-configurable-contract.md
View file

@ -18,40 +18,21 @@ the golden suite is the regression harness for that claim.
profiles are constructed in code by the consumer):
- `type_policy`: closed enum vs. open set; the reserved `verdict` exclusion
stays unconditional (it is a spec invariant, not profile config).
- `frontmatter_schema`: three distinct fields, not one — the *required* key
set, the *allowlisted* key set, and the canonical emission *order*.
STRICT_V1 requires eight keys inside an eleven-key allowlist (see
`docs/phase-3-split-table.md`), so a single ordered required/optional pair
cannot express it. Also carries nullable keys and the scalar encoding
subset. Phase 1's seven-key layer and Phase 2's inbox layer are
expressions of this schema.
- `frontmatter_schema`: ordered required keys + optional keys, per concept
type where needed (Phase 1's seven-key layer and Phase 2's inbox layer are
expressions of this schema).
- `path_policy`: layer/path prefixes, filename namespaces, reserved filename
patterns (`index.md`, `promoted-verdict-*`, …).
- `reserved_files`: per-file policy, THREE values not two — required,
required-absent, and present-but-exempt-from-document-gates (the proving
consumer's `README.md`, which carries no frontmatter and is never indexed
yet is still content-scanned).
- `index_policy`: managed-link format and per-level index expectations. This
is the widest gap between the two profiles: one index per bundle versus
one per directory, `- [label](target)` versus
`* [Title](link) - description`, append-if-absent versus exact match in
both directions, and a root index with or without frontmatter.
- `reserved_files`: per-file required / optional / required-absent policy
(e.g. a profile may demand that `log.md` is absent).
- `index_policy`: managed-link format and per-level index expectations.
2. **Built-in profiles:**
- `DEFAULT` (implicit): exactly the ingest-spec v1 + Phase 2 contract.
- `STRICT_V1`: the `claude-code-llm-wiki` contract — closed type enum
(`Concept`/`Guide`/`Reference`/`Release`), eight required frontmatter keys
(`Concept`/`Guide`/`Reference`/`Release`), extended required frontmatter
(`type`, `title`, `description`, `timestamp`, `layer`, `source_tier`,
`source_url`, `source_sha`) inside an eleven-key allowlist that doubles as
emission order, layer/path/type/source_tier agreement rules, `log.md`
required-absent, `README.md` reserved-exempt, per-directory indexes, and
root-index metadata (`okf_version`, `bundle_profile`, `okf_spec_commit`).
This line was drafted as eight, corrected to four against the consumer's
`REQUIRED_KEYS` constant on 2026-07-25, and settled at eight by their
operator the same day: the four-key constant is their emit-path minimum,
not their contract, and all eight are present on 522/522 of their
documents. STRICT_V1 is therefore deliberately stricter than the
consumer's own validator; the split table records the agreed consequence.
`source_url`, `source_sha`), layer path prefixes, `log.md` required-absent,
root-index metadata.
3. **Plumbing change:** materialization, inbox, import, and index generation take
an optional `profile` argument defaulting to `DEFAULT`. No behavior branches
outside what the profile object expresses.
@ -60,10 +41,7 @@ the golden suite is the regression harness for that claim.
The wiki's existing validator mixes three kinds of gates. First implementation
step is a written mapping of every gate to exactly one column — the mapping is
reviewed with the operator before any porting. **Drafted 2026-07-25 in
`docs/phase-3-split-table.md`**, over all 25 gates, and sent to the proving
consumer for review; two gates turned out to bundle checks from different
columns and are decomposed there rather than forced into one.
reviewed with the operator before any porting:
| Column | Meaning | Goes where |
|---|---|---|
@ -91,9 +69,9 @@ columns and are decomposed there rather than forced into one.
| # | Assumption | Test |
|---|---|---|
| C1 | Default-profile refactor is behavior-neutral | Phase 1 golden suite byte-identical; full Phase 1/2 test suite green, unmodified |
| C2 | `strict-v1` requirements are fully enumerable from the wiki's validator | The split table is exhaustive: every wiki gate appears in exactly one column; reviewed with the operator. Drafted 2026-07-25 against `validate.py`/`bundle.py` at their `9ee5a8e`, which is the single gate entrypoint — the enumeration holds only against that commit, so re-check before porting. Two gates required decomposition rather than a single column |
| C2 | `strict-v1` requirements are fully enumerable from the wiki's validator | The split table is exhaustive: every wiki gate appears in exactly one column; reviewed with the operator |
| C3 | Verdict reservation must survive every profile | Named test: no profile construction can permit `type: verdict` |
| C4 | Profiles-in-code (no config files) suffice for the proving consumer | CONFIRMED with the operator 2026-07-25 at phase start: profiles are constructed in code, config-file loading stays an extension point |
| C4 | Profiles-in-code (no config files) suffice for the proving consumer | Confirmed with the operator at phase start; config-file loading is an extension point |
## Non-goals
@ -104,12 +82,6 @@ columns and are decomposed there rather than forced into one.
to block it (checked against the catalog's spec during the split-table step).
- Any change to commons' ingest-spec; if profile work reveals spec friction, it
is raised in commons, not patched locally.
- Aligning the contract with upstream OKF v0.2. This phase targets v0.1, and the
profile object is what makes the upstream version a *policy* rather than a
constant: no profile may hard-code an `okf_version` value, and the
frontmatter-set fields must be able to express either the legacy `timestamp`
or v0.2's `generated`. The decisions themselves live in
`docs/plan/okf-v0.2-alignment.md` and are sequenced after this phase.
## Verification

35
docs/plan/phase-4-node-half.md
View file

@ -46,40 +46,7 @@ code.
parity.
3. **linkedin-studio stays plugin-local.** Its `ingest/published/`
provenance-record grammar has a different lifecycle by design and is NOT
normalized into this library. Explicit non-goal, agreed up front, and
confirmed by them on 2026-07-25 as their own fixed decision rather than
something conceded to us.
Two requirements they named on 2026-07-25, before the scope freeze, which
it must therefore answer explicitly rather than pass over. Both are about
not breaking what they have, not about building toward them:
- **`write_index` must have a do-nothing mode.** They run a fourth index
mode the spec does not describe: create-if-absent, no managed lines,
the whole file hand-editable. Their scaffold never opens an existing
index, so it cannot normalize one. The requirement is that this
library's writer match that — against an existing index carrying no
managed lines, write nothing, and do not normalize the frame "while it
is open anyway". This is the writer-side sibling of the Phase 3
decision that nothing enumerates a directory unless the profile says
the index is derived. `write_index` is deliverable 3, not deliverable
4's `write`; that was ambiguous in this plan and is now stated.
- **An absent vocabulary means "do not snap", never "snap to default".**
The reference checker collapses any non-matching type to a
`DEFAULT_TYPE`, which would erase their Profile / Operations /
PendingDiff distinction with zero byte change. Vocabulary is
per-bundle, never spec-global. This is already the Python half's shape
— `TypePolicy(allowed=None)` is the default and rewrites nothing — so
the Node half inherits a design rather than adopting a constraint, and
the collapse is a known deviation to resolve with the convention owner
under step 1 rather than lift.
They have offered a minimal tolerance fixture pair (a `brain/` hub whose
`index.md` carries `okf_version` as markdown text, and an
`ingest/published/` record: YAML-free five-line header terminated by
`\n---\n`, no trailing newline). Take the bytes rather than the
description — it is held as a tolerance case, never as a format this
library emits.
normalized into this library. Explicit non-goal, agreed up front.
4. **Persist gate = guard-as-contract.** There is no Node guard today. The Node
half documents and enforces the guard's *contract* at its persist seam
(fail-closed hook point, no content-safety logic of its own) so a future Node

179
docs/upstream-okf-upgrade-runbook.md
View file

@ -1,179 +0,0 @@
# Upstream OKF upgrade — runbook
Standing policy (operator, 2026-07-26): **this library always supports the current
latest version of Google OKF.** This document is the procedure that policy runs on.
It exists because a standing policy without a procedure is an intention, and because
the v0.1 → v0.2 round produced five findings that a spec reading alone had settled
wrongly — each step below names the concrete failure it prevents.
Written after the v0.2 round. Every prevented-failure note is something that
actually happened, not a hypothetical.
## The goal that shapes the procedure: this repo is a black box
An upstream release is our problem, not our consumers'. The target cost to a repo
that consumes OKF through this library is **a re-run, and nothing else.**
That is a design commitment with teeth, not an aspiration:
- **Support is additive — a new profile, never a migration.** Existing profiles stay
byte-stable. A consumer who does not opt in sees no change at all.
- **New public parameters are keyword-only with defaults.** A consumer's existing
positional call sites stay source-compatible across an upgrade. This is why
`profile` is specified as `*, profile: BundleProfile = DEFAULT` — **decided, not
yet shipped**: measured 2026-07-26, no door takes the argument at all. It turns
additivity
from something a consumer has to measure into a property of the signature.
- **Consumer golden fixtures must not churn.** If an upgrade rewrites bytes in a
consumer's frozen fixtures, the upgrade is wrong, not the fixture.
**The honest boundary — state it every time, never let the black box be oversold.**
The library absorbs *shape* changes. It cannot absorb upstream changes to content the
consumer authored. v0.2 superseded `timestamp` with `generated.at` and the body
`# Citations` list with `sources`; no library change makes a consumer's existing
`timestamp` field become something else. For that class the deliverable is not
absorption but a **measured exposure report, per consumer, before they ask** — which
is what step 5 produces.
## Trigger
"Always latest" decays silently: nothing fails when upstream ships and we do not
notice. So the re-check is an item on the release checklist — run it at every
release of this library, and record the result **even when unchanged**, because an
unrecorded check is indistinguishable from a skipped one.
Check `GoogleCloudPlatform/knowledge-catalog`, path `okf/`. A version bump appears
as a commit against `okf/SPEC.md` §12 and, in the v0.2 round, as an explicit
migration commit (`okf: migrate format and tooling to Open Knowledge Format v0.2`).
## Step 1 — Pin before reading anything
Record the exact commit SHA and read everything at it.
*Prevents:* reading a moving branch and treating the result as fact. In the v0.2
round both this repo and `portfolio-optimiser-commons` independently read `main`,
which felt like two confirmations and was one unstable reference read twice. Worse,
`SPEC.md` was edited **after** the v0.2 migration commit — so "the v0.2 commit" and
"the current spec text" were different objects, and either alone would have been a
partial answer.
## Step 2 — Enumerate the whole `okf/` tree, not just `SPEC.md`
List every file and directory at the pinned commit before deciding what to read.
*Prevents:* assuming the directory you know about is the one that matters. In the
v0.2 round `okf/` held `SPEC.md`, `README.md`, `pyproject.toml`, `src/`, `tests/`,
`samples/` **and** `bundles/`. The last one held the actual v0.2 example bundles and
was not on anyone's list until the tree was enumerated.
## Step 3 — Read the shipped examples, not only the normative text
This is the step the v0.2 round skipped, and it is the reason this document exists.
**A spec says what is permitted. An example shows what upstream actually emits.**
Those differ, and when we are about to freeze a fixture and hand it to consumers,
the second one is what we are being measured against.
Two traps, both hit in the v0.2 round:
- **A directory called `samples/` was not samples of the format.**
`okf/samples/ga4_merch_store/` is a runner config (`README.md` + `seeds.txt`) that
drives the reference agent to *generate* a bundle. It says nothing about output
shape. The real examples were in `okf/bundles/` — four of them.
- **Fetch tooling that converts to markdown will strip or reformat frontmatter.**
Byte-level questions need byte-level fetches. Check for a BOM, check the trailing
newline, check whether a value is quoted. Those three are exactly the properties
that break downstream gates and exactly the ones a summarizing fetch destroys.
What reading the four v0.2 bundles produced, none of it derivable from `SPEC.md`:
| Finding | Consequence |
|---|---|
| No reference bundle declares `okf_version` at all; root `index.md` has no frontmatter | §12 is a MAY and upstream declines it. Our stricter posture became a *decision* instead of an assumption |
| `generated` is written for **human-authored** content (`by: human:…`) | It can never be an ownership or machine-generated predicate. Governs the collision gate |
| Real frontmatter is multi-line block YAML: block lists of multi-key mappings, nested mappings, flow sequences, booleans, dates | Re-sized the emitter work. It had been scoped against a list of strings |
| §7's canonical tool actor is `<producer>/<version>`, and upstream uses it | Counter-evidence to a recommendation we had already sent to another repo |
## Step 4 — Produce the diff, classified
From the spec's own "Changes from vN" section **plus** the example evidence,
classify every change as:
- **Breaking** — name it, and measure consumer exposure in step 5.
- **Additive** — new optional fields, new conventional headings.
- **Deferred upstream** — a format specified but its runtime protocol left to a
future revision. The format is supportable; an unspecified runtime is not
something to build against. Record it as out of scope *on upstream's deferral*,
not on our preference.
- **Observed but undocumented** — present in an official bundle, not found in the
spec sections read. Record as unresolved. Do not silently promote it to either
category. (v0.2 round: a `not:` family in `metrics/gross-margin.md`.)
## Step 5 — Measure our exposure and each consumer's, in that order
Ours: grep the profiles, the fixtures, and the emitter for every field the diff
touches. Cite file:line. Never reason about a key set — measure it.
Theirs: for each breaking change, state what would have to be true for it to reach
them, then **ask them to measure it against their real data** rather than telling
them whether they are affected. In the v0.2 round both `portfolio-optimiser-claude`
and `linkedin-studio` came back with measured zero exposure and found the framing
useful precisely because it was falsifiable.
## Step 6 — Plan additively, then pilot before general availability
The plan is a new profile, sequenced deliverables, and stated invariants. Rollout is
pilot-first: a pre-release tag to a small pilot set chosen for signal, revised on
their feedback, then GA. Flipping the `OKF_LATEST` alias **is** the GA event, not a
merge side effect.
State each pilot test's expected results **before** the run, numbered, plus what
would surprise us. An expectation we get wrong is a better result than a clean run;
only a stated expectation can be falsified by someone else's run.
**Name fixtures by repo and path, never by path alone.** In the v0.2 round both this
repo and `portfolio-optimiser-claude` had a directory named
`examples/ingest-golden-file/` holding *different* fixtures with zero content
overlap. A pilot baseline was agreed by name, and the name pointed at two files.
## Step 7 — Inform every OKF-consuming repo
All of them, not only the pilots, and not only when something breaks.
Send a **bounded loop of directed messages, not a broadcast** — a broadcast is
inherited by every future repo, which is the wrong lifetime for a
version-specific notice.
Each message carries:
1. The version, the pinned commit, and where to read it.
2. The breaking changes by name, with the shape of the exposure — so they can
measure themselves rather than take our word.
3. **The black-box promise and its boundary**: their cost is a re-run; existing
profiles are byte-stable; new parameters are keyword-only. And the part we
cannot absorb, said plainly.
4. What we want back, if anything, and what happens if they say nothing.
Consumers to cover (2026-07): `portfolio-optimiser-commons` (spec authorship),
`portfolio-optimiser`, `portfolio-optimiser-claude`, `claude-code-llm-wiki`,
`catalog`, `okr`, `linkedin-studio`, `ms-ai-architect`. Re-derive this list each
round rather than trusting it — it is a premise like any other.
## Step 8 — Close the loop in the repo
The plan doc records the pinned commit, the classified diff, the measured exposure,
and every correction a consumer sent back. Corrections stay visible: a premise that
survived verification is only known to have survived if the check is recorded, and a
claim we withdrew is only safely withdrawn if the withdrawal is written where the
claim was.
## Invariants this procedure protects
- No profile hard-codes an upstream version.
- No bundle declares a version its shape has not earned. The spec may permit
declaring conformance without a checkpoint; we decline. *Conform first, claim
after.*
- `DEFAULT` states commons' ingest-spec §5 layer — raised there, never patched here.
- `STRICT_V1` mirrors a consumer's ratified contract — never changed from here.
- The value of `okf_version` belongs to the catalog.
- Security stays the guard's domain in every version.

6
examples/ingest-golden-okf-v0-2/expected-bundle/index.md
View file

@ -1,6 +0,0 @@
---
okf_version: 0.2
---
Golden OKF v0.2 bundle: regional sales extracted from a CSV source.
- [Regional Sales](ingest-sales.md)

16
examples/ingest-golden-okf-v0-2/expected-bundle/ingest-sales.md
View file

@ -1,16 +0,0 @@
---
type: dataset
title: Regional Sales
source_system: golden-v0-2-sales
source_query: sales.csv
ingested_at: 2026-07-16T12:00:00Z
ingest_manifest: manifest@a83c86e8f847eca2
generated: { by: process:llm-ingestion-okf, at: 2026-07-16T12:00:00Z }
sources: [{ id: golden-v0-2-sales, resource: fixture }]
---
| region | units | revenue |
| --- | --- | --- |
| nord | 412 | 88120 |
| vest | 297 | 61540 |
| sor | 158 | 33010 |

4
examples/ingest-golden-okf-v0-2/fixture/sales.csv
View file

@ -1,4 +0,0 @@
region,units,revenue
nord,412,88120
vest,297,61540
sor,158,33010
1 region units revenue
2 nord 412 88120
3 vest 297 61540
4 sor 158 33010

1
examples/ingest-golden-okf-v0-2/ingested-at.txt
View file

@ -1 +0,0 @@
2026-07-16T12:00:00Z

18
examples/ingest-golden-okf-v0-2/manifest.json
View file

@ -1,18 +0,0 @@
{
"manifest_version": 1,
"source": {
"type": "file",
"id": "golden-v0-2-sales",
"root": "fixture"
},
"bundle_summary": "Golden OKF v0.2 bundle: regional sales extracted from a CSV source.",
"extractions": [
{
"id": "sales",
"title": "Regional Sales",
"query": "sales.csv",
"okf_type": "dataset",
"max_rows": 10
}
]
}

1
examples/ingest-golden-okf-v0-2/okf-version.txt
View file

@ -1 +0,0 @@
0.2

15
pyproject.toml
View file

@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "llm-ingestion-okf"
version = "0.5.0a1"
version = "0.4.0"
description = "Shared OKF (Open Knowledge Format) ingestion library: spec-based connectors, bundle inbox, and external-bundle import, with security delegated to llm-ingestion-guard."
readme = "README.md"
license = "MIT"
@ -50,14 +50,9 @@ python_version = "3.10"
module = ["llm_ingestion_guard", "llm_ingestion_guard.*"]
ignore_missing_imports = true
# Install CHANNEL for the guard, which is not on a package index yet. It is
# uv-specific, and it reaches further than a dev-only setting: measured
# 2026-07-25, a consumer installing this package from git WITH UV picks the
# guard up from this tag automatically, because uv reads this file when it
# builds from the source tree. pip does not read it — it resolves
# [project.dependencies] alone and fails with "No matching distribution found
# for llm-ingestion-guard" until the guard is installed from its own tag first
# (README). Either way the range above stays the pin: the built wheel carries
# `Requires-Dist: llm-ingestion-guard<0.3,>=0.2`, verified against the wheel.
# Development-time install CHANNEL for the guard, which is not on a package
# index yet. It resolves `uv lock`/`uv sync` against the tag; it is uv-specific
# and never reaches consumers — the built wheel carries the range from
# [project.dependencies] above, which is the pin.
[tool.uv.sources]
llm-ingestion-guard = { git = "https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git", tag = "v0.2.0" }

2
src/llm_ingestion_okf/__init__.py
View file

@ -73,7 +73,7 @@ from .manifest import (
)
from .materialize import IngestResult, materialize_bundle
__version__ = "0.5.0a1"
__version__ = "0.4.0"
__all__ = [
"BlockedFile",

4
src/llm_ingestion_okf/errors.py
View file

@ -96,10 +96,6 @@ class MaterializationError(IngestError):
- `ingested_at_invalid` — ingested_at is not ISO-8601 UTC with a Z suffix
- `collision_unstamped` — the §3 collision gate: a generated name is
occupied by a file without the ingest stamp
- `source_reference_unquotable` — a manifest source's id or locator
contains a character that would restructure the `sources` flow mapping
(Door A, v0.2 profiles); refused rather than emitted, because the
resulting document parses cleanly into a record no one wrote
- `inbox_slug_empty` — a dropped file's name reduces to an empty slug
under the id grammar (Door B; never an invented fallback name)
- `inbox_slug_too_long` — the generated inbox filename would exceed the

21
src/llm_ingestion_okf/importer.py
View file

@ -34,18 +34,20 @@ from typing import Protocol
from .errors import IngestError, MaterializationError, SourceError
from .extract import decode_text
from .materialize import (
INDEX_NAME,
check_filename_length,
link_in_index,
reduce_to_id_grammar,
validate_ingested_at,
write_bytes,
)
from .profiles import DEFAULT
# An OKF concept is a `.md` document by definition — the guard's path gate
# rejects anything else outright — so nothing else in the source tree is a
# concept, and nothing else is this door's to merge. The suffix and the
# `import-` namespace are the profile's (`DEFAULT.paths`).
# concept, and nothing else is this door's to merge.
_CONCEPT_SUFFIX = ".md"
_FILENAME_PREFIX = "import-"
# The guard's non-blocking floor and its review queue, by VALUE (`Disposition`
# is a `str, Enum`, so the value is the stable thing to compare against across
@ -174,7 +176,7 @@ def import_slug(concept_path: str) -> str:
collapse them onto one filename. A path that reduces to nothing fails fast
rather than being given an invented name.
"""
concept_id = concept_path[: -len(DEFAULT.paths.concept_suffix)]
concept_id = concept_path[: -len(_CONCEPT_SUFFIX)]
slug = reduce_to_id_grammar(concept_id)
if not slug:
raise MaterializationError(
@ -193,8 +195,7 @@ def import_filename(slug: str) -> str:
grammar admits.
"""
return check_filename_length(
f"{DEFAULT.paths.import_prefix}{slug}{DEFAULT.paths.concept_suffix}",
code="import_path_too_long",
f"{_FILENAME_PREFIX}{slug}{_CONCEPT_SUFFIX}", code="import_path_too_long"
)
@ -205,7 +206,7 @@ def _index_label(concept_path: str) -> str:
does not. Fail-fast, never repair — the same rule Door A applies to a
manifest title and Door B to a dropped filename.
"""
label = concept_path[: -len(DEFAULT.paths.concept_suffix)]
label = concept_path[: -len(_CONCEPT_SUFFIX)]
if any(char in label for char in "\n\r[]"):
raise MaterializationError(
f"concept path {concept_path!r} contains '[' or ']', which would break "
@ -228,7 +229,7 @@ def _read_bundle(source: Path) -> tuple[dict[str, str], list[FailedConcept]]:
# glob: glob case-sensitivity follows the FILESYSTEM, so `NOTE.MD`
# would be a concept on APFS and not one on ext4 — the same bundle
# importing differently per platform. The guard folds case here too.
if not path.is_file() or path.suffix.lower() != DEFAULT.paths.concept_suffix:
if not path.is_file() or path.suffix.lower() != _CONCEPT_SUFFIX:
continue
concept_path = path.relative_to(source).as_posix()
try:
@ -408,9 +409,9 @@ def import_bundle(
# §6 index — the last disk mutation, and only when something merged.
if merged:
index_path = bundle / DEFAULT.index.name
index_path = bundle / INDEX_NAME
if not index_path.is_file():
write_bytes(bundle, DEFAULT.index.name, "")
write_bytes(bundle, INDEX_NAME, "")
for entry in merged:
link_in_index(bundle, entry.path.name, _index_label(entry.concept_path))

42
src/llm_ingestion_okf/inbox.py
View file

@ -28,6 +28,7 @@ from pathlib import Path
from .errors import IngestError, MaterializationError, SourceError
from .extract import extract_text
from .materialize import (
INDEX_NAME,
check_filename_length,
link_in_index,
parse_frontmatter,
@ -35,7 +36,11 @@ from .materialize import (
validate_ingested_at,
write_bytes,
)
from .profiles import DEFAULT
_RESERVED_OKF_TYPE = "verdict"
_FILENAME_PREFIX = "inbox-"
_FILENAME_SUFFIX = ".md"
def inbox_slug(source_filename: str) -> str:
@ -68,8 +73,7 @@ def inbox_filename(slug: str) -> str:
not give it.
"""
return check_filename_length(
f"{DEFAULT.paths.inbox_prefix}{slug}{DEFAULT.paths.concept_suffix}",
code="inbox_slug_too_long",
f"{_FILENAME_PREFIX}{slug}{_FILENAME_SUFFIX}", code="inbox_slug_too_long"
)
@ -100,11 +104,12 @@ def render_inbox_concept(
validate_ingested_at(ingested_at)
# The verdict layer is RESERVED: the promotion gate is the only path into
# it, at this door exactly as at Door A's manifest validation — the same
# profile decides, each door raises its own typed error.
rejection = DEFAULT.types.rejection(okf_type)
if rejection is not None:
raise MaterializationError(f"okf_type {rejection.reason}", code=rejection.code)
# it, at this door exactly as at Door A's manifest validation.
if okf_type.lower() == _RESERVED_OKF_TYPE:
raise MaterializationError(
f"okf_type must not be {_RESERVED_OKF_TYPE!r} (reserved layer)",
code="okf_type_reserved",
)
# The title is rendered verbatim into `- [title](target)` and into
# line-oriented frontmatter — met by fail-fast validation, never repair.
if any(char in title for char in "\n\r[]"):
@ -126,7 +131,8 @@ def render_inbox_concept(
"ingested_at": ingested_at,
"generated": "true",
}
return f"---\n{DEFAULT.frontmatter.emit(frontmatter)}\n---\n\n{_normalize_body(text)}"
rendered = "\n".join(f"{key}: {value}" for key, value in frontmatter.items())
return f"---\n{rendered}\n---\n\n{_normalize_body(text)}"
# --- the guard seam -------------------------------------------------------
@ -242,9 +248,11 @@ def process_inbox(
a reserved `okf_type`, and a missing inbox directory.
"""
validate_ingested_at(ingested_at)
run_rejection = DEFAULT.types.rejection(okf_type)
if run_rejection is not None:
raise MaterializationError(f"okf_type {run_rejection.reason}", code=run_rejection.code)
if okf_type.lower() == _RESERVED_OKF_TYPE:
raise MaterializationError(
f"okf_type must not be {_RESERVED_OKF_TYPE!r} (reserved layer)",
code="okf_type_reserved",
)
inbox = Path(inbox_dir)
if not inbox.is_dir():
raise SourceError(f"inbox directory does not exist: {inbox}", code="source_root_missing")
@ -291,11 +299,7 @@ def process_inbox(
# pre-existing curated content by a later file's check.
bundle = Path(bundle_dir)
pre_existing = (
{
path.name
for path in bundle.glob(f"*{DEFAULT.paths.concept_suffix}")
if path.name != DEFAULT.index.name
}
{path.name for path in bundle.glob("*.md") if path.name != INDEX_NAME}
if bundle.is_dir()
else set()
)
@ -365,9 +369,9 @@ def process_inbox(
# §6 index — the last disk mutation, and only when something was written.
if persisted:
index_path = bundle / DEFAULT.index.name
index_path = bundle / INDEX_NAME
if not index_path.is_file():
write_bytes(bundle, DEFAULT.index.name, "")
write_bytes(bundle, INDEX_NAME, "")
for entry in persisted:
link_in_index(
bundle, entry.path.name, unicodedata.normalize("NFC", Path(entry.source_file).stem)

22
src/llm_ingestion_okf/manifest.py
View file

@ -15,10 +15,11 @@ from pathlib import Path
from typing import Any, Union
from .errors import ManifestError
from .profiles import DEFAULT, BundleProfile
_ID_PATTERN = re.compile(r"[a-z0-9][a-z0-9-]*\Z")
_RESERVED_OKF_TYPE = "verdict"
@dataclass(frozen=True)
class FileSource:
@ -59,17 +60,13 @@ class Manifest:
extractions: tuple[Extraction, ...]
def generated_filename(extraction_id: str, *, profile: BundleProfile = DEFAULT) -> str:
def generated_filename(extraction_id: str) -> str:
"""The concept filename for an extraction (spec §5).
The `ingest-` prefix keeps the namespace disjoint from `index.md` and
`promoted-verdict-*` (spec §3) for every id the §4 grammar admits.
The prefix and suffix are the profile's, because the ownership scan globs
on the same two values: a name built from one profile and scanned for under
another is a file the library cannot recognise as its own.
"""
return f"{profile.paths.ingest_prefix}{extraction_id}{profile.paths.concept_suffix}"
return f"ingest-{extraction_id}.md"
def load_manifest(path: Path) -> Manifest:
@ -196,12 +193,11 @@ def _validate_extraction(data: object, index: int) -> Extraction:
okf_type = _require_str(obj["okf_type"], f"{label}.okf_type")
# The verdict layer is RESERVED (spec §3): the promotion gate is the only
# path into it — enforced here, fail-fast, before any source call. The
# profile decides which types a bundle admits; the door raises its own
# typed error, since Door B refuses the same type as a MaterializationError.
rejection = DEFAULT.types.rejection(okf_type)
if rejection is not None:
raise ManifestError(f"{label}.okf_type {rejection.reason}", code=rejection.code)
# path into it — enforced here, fail-fast, before any source call.
if okf_type.lower() == _RESERVED_OKF_TYPE:
raise ManifestError(
f"{label}.okf_type must not be 'verdict' (reserved layer)", code="okf_type_reserved"
)
max_rows = obj["max_rows"]
if not _is_int(max_rows) or max_rows < 1:

229
src/llm_ingestion_okf/materialize.py
View file

@ -13,7 +13,6 @@ import hashlib
import logging
import re
import unicodedata
from collections.abc import Mapping
from dataclasses import dataclass
from pathlib import Path
@ -24,18 +23,23 @@ from .manifest import (
FileSource,
HttpSource,
Manifest,
Source,
SqlSource,
generated_filename,
load_manifest_bytes,
)
from .profiles import DEFAULT, BundleProfile
from .render import render_fenced_block, render_table
_LOGGER = logging.getLogger(__name__)
INDEX_NAME = "index.md"
_INGESTED_AT_RE = re.compile(r"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z$")
# One managed index line: `- [<label>](<target>)`. Anchored full-line —
# removal keys on this exact shape for specific ingest targets, never a bare
# substring (a promoted verdict's line has the same shape but a non-ingest
# target; curated prose mentioning a target inline does not match).
_MANAGED_LINE_RE = re.compile(r"^- \[(?P<label>[^\]]*)\]\((?P<target>[^)]+)\)$")
@dataclass(frozen=True)
class IngestResult:
@ -116,6 +120,24 @@ def check_filename_length(name: str, *, code: str) -> str:
return name
def _collapse_whitespace(value: str) -> str:
# §5 mandates whitespace-run collapse for ONE field only: `source_query`
# (ingest-spec.md:140-141), where a legitimately multi-line SQL SELECT
# must render on one line. Every other value is validated single-line at
# manifest load and emitted verbatim — validation, not repair — so
# operator-supplied bytes (e.g. a title's internal double space) survive.
return " ".join(value.split())
def _render_frontmatter(frontmatter: dict[str, str]) -> str:
# Line-oriented `key: value`, insertion order. Only `source_query` is
# collapsed; all other values pass through verbatim.
return "\n".join(
f"{key}: {_collapse_whitespace(value) if key == 'source_query' else value}"
for key, value in frontmatter.items()
)
def parse_frontmatter(path: Path) -> dict[str, str]:
lines = path.read_text(encoding="utf-8").splitlines()
if not lines or lines[0].strip() != "---":
@ -130,18 +152,10 @@ def parse_frontmatter(path: Path) -> dict[str, str]:
return frontmatter
def _is_ingest_owned(path: Path, manifest_stem: str, *, profile: BundleProfile = DEFAULT) -> bool:
# §3/§5 ownership: the profile's ingest stamp AND an `ingest_manifest`
# reference. Promoted verdict files carry neither key, so they can never
# classify as ingest-owned.
#
# The stamp is the PROFILE's because it differs per profile (v0.1 writes
# `generated: true`, v0.2 a `{ by: ..., at: ... }` mapping) — and because the
# emitter and this predicate are coupled through that value. Changing the
# emitted form without the predicate is what makes the library stop
# recognising its own output, firing the collision gate on the files its own
# previous run wrote. `OwnershipPolicy` is where the two meet, so they can
# only be changed together.
def _is_ingest_owned(path: Path, manifest_stem: str) -> bool:
# §3/§5 ownership: the ingest stamp is `generated: true` AND an
# `ingest_manifest` reference. Promoted verdict files carry neither key,
# so they can never classify as ingest-owned.
#
# §10.2 per-manifest ownership: a file is THIS manifest's to replace only
# when the reference names it by stem. The stamp is `{stem}@{sha256[:16]}`;
@ -151,8 +165,7 @@ def _is_ingest_owned(path: Path, manifest_stem: str, *, profile: BundleProfile =
# keeps its own. rsplit strips the trailing `@{sha}`, so a stem that itself
# contains `@` still compares correctly.
frontmatter = parse_frontmatter(path)
generated = frontmatter.get("generated")
if generated is None or not profile.ownership.owns(generated):
if frontmatter.get("generated") != "true":
return False
reference = frontmatter.get("ingest_manifest")
if reference is None:
@ -160,70 +173,10 @@ def _is_ingest_owned(path: Path, manifest_stem: str, *, profile: BundleProfile =
return reference.rsplit("@", 1)[0] == manifest_stem
# The characters that terminate or restructure a YAML flow mapping. `:\s`
# catches a colon that would open a nested key; a colon inside `https://host`
# does not, and stays a plain scalar.
_FLOW_UNSAFE_RE = re.compile(r"[,\[\]{}]|:\s")
def _source_locator(source: Source) -> str:
"""Where a manifest source points, per source type.
A filesystem root, the NAME of the environment variable holding the DSN, or
the base URL. `credential_ref` is not a locator and is never returned here:
a credential reference has no reader in a bundle.
"""
if isinstance(source, FileSource):
return source.root
if isinstance(source, SqlSource):
return source.connection_ref
return source.base_url
def _render_sources(source: Source) -> str:
"""§5 `sources` as an inline flow sequence of one flow mapping.
Two keys, not upstream's five: a manifest source has no `author`, no
`last_modified`, and no bundle-internal `resource` in upstream's sense, and
inventing them would be writing fields with no reader.
The flow form rather than upstream's block list, measured and chosen: a
block list read back through this library's line-oriented parser turns each
item line into a KEY nobody wrote — and `_is_ingest_owned` reads through
that same parser. The flow form also satisfies commons' §5 "all values MUST
be single-line", and §11 requires parseable YAML rather than block YAML.
Refusing an unquotable locator is the point of the check rather than a
nicety: `[{ id: x, resource: data, backup }]` is not a parse ERROR, it is a
mapping with a `backup` key nobody wrote. A silently wrong provenance record
is worse than a refused run, and repairing the value by quoting it would
change bytes the operator supplied. Validation, not repair — the same
posture as the filename-length gate.
"""
locator = _source_locator(source)
for label, value in (("id", source.id), ("resource", locator)):
if _FLOW_UNSAFE_RE.search(value):
raise MaterializationError(
f"the source {label} {value!r} contains a character that would "
"restructure the `sources` flow mapping (one of `,[]{}` or a "
"colon followed by whitespace) — refusing to emit a provenance "
"record that parses cleanly into something no one wrote",
code="source_reference_unquotable",
)
return f"[{{ id: {source.id}, resource: {locator} }}]"
def _render_concept_file(
manifest: Manifest,
extraction: Extraction,
body: str,
*,
ingested_at: str,
stamp: str,
profile: BundleProfile = DEFAULT,
manifest: Manifest, extraction: Extraction, body: str, *, ingested_at: str, stamp: str
) -> str:
# §5 frontmatter: exactly these keys. The order and the `source_query`
# whitespace collapse are the profile's — DEFAULT states the §5 layer.
# §5 frontmatter: exactly these keys, in exactly this order.
frontmatter = {
"type": extraction.okf_type,
"title": extraction.title,
@ -231,15 +184,9 @@ def _render_concept_file(
"source_query": extraction.query,
"ingested_at": ingested_at,
"ingest_manifest": stamp,
"generated": profile.ownership.stamp(ingested_at),
"generated": "true",
}
# Written only by a profile that NAMES it. `emit` sorts an unnamed key into
# the tail rather than dropping it, so building one mapping for both
# profiles would append `sources` to every v0.1 bundle — additivity is a
# property of what is constructed here, not of the emitter.
if "sources" in profile.frontmatter.order:
frontmatter["sources"] = _render_sources(manifest.source)
return f"---\n{profile.frontmatter.emit(frontmatter)}\n---\n\n{body}"
return f"---\n{_render_frontmatter(frontmatter)}\n---\n\n{body}"
def write_bytes(bundle_dir: Path, name: str, content: str) -> Path:
@ -252,11 +199,7 @@ def write_bytes(bundle_dir: Path, name: str, content: str) -> Path:
def _update_index_lines(
index_path: Path,
removed_targets: set[str],
labels_by_target: dict[str, str],
*,
profile: BundleProfile = DEFAULT,
index_path: Path, removed_targets: set[str], labels_by_target: dict[str, str]
) -> None:
"""§6 maintenance on an EXISTING index: drop managed lines whose target is
an ingest file removed in this run; refresh in place a managed label that
@ -270,7 +213,7 @@ def _update_index_lines(
for line in lines:
content = line.rstrip("\r\n")
ending = line[len(content) :]
match = profile.index.link_pattern.match(content)
match = _MANAGED_LINE_RE.match(content)
if match is not None:
target = match.group("target")
if target in removed_targets:
@ -278,25 +221,17 @@ def _update_index_lines(
continue
new_label = labels_by_target.get(target)
if new_label is not None and match.group("label") != new_label:
line = profile.index.render_link(new_label, target) + ending
line = f"- [{new_label}]({target})" + ending
changed = True
updated.append(line)
if changed:
index_path.write_bytes("".join(updated).encode("utf-8"))
def link_in_index(
bundle_dir: Path, target_name: str, label: str, *, profile: BundleProfile = DEFAULT
) -> None:
def link_in_index(bundle_dir: Path, target_name: str, label: str) -> None:
# §6: idempotent by target — a link whose target is already present in
# the index is never added twice.
#
# `profile` is keyword-only with a default because this function is public
# and called from all three doors (A here, B in inbox.py, C in importer.py).
# Doors B and C keep the default, which is the behaviour they already had;
# which profile THEY should own is a separate question, and answering it by
# changing this signature would have decided it silently.
index_path = safe_resolve(bundle_dir, profile.index.name)
index_path = safe_resolve(bundle_dir, INDEX_NAME)
body = index_path.read_bytes().decode("utf-8")
if f"]({target_name})" in body:
return
@ -304,38 +239,7 @@ def link_in_index(
# bundle_summary first, but Door B has no summary to invent, so its index
# starts empty and must not open with a blank line.
prefix = body if (body == "" or body.endswith("\n")) else body + "\n"
line = profile.index.render_link(label, target_name)
index_path.write_bytes(f"{prefix}{line}\n".encode())
def _render_root_frontmatter(values: Mapping[str, str], *, profile: BundleProfile) -> str:
"""The root index's frontmatter block (§8, §12), or "" when nothing is
declared.
The policy names the keys and fixes their order; the caller supplies the
values. Ordering by the POLICY rather than by the mapping is what keeps two
callers passing the same keys from emitting different bytes — a dict
preserves insertion order, and a golden fixture would then depend on the
order a caller happened to build its argument in.
Values are written verbatim. `okf_version` must reach catalog's shape gate
unquoted, so nothing here may add quoting; the golden fixture asserts that
on raw bytes.
"""
unknown = sorted(set(values) - set(profile.index.root_frontmatter))
if unknown:
raise MaterializationError(
f"root frontmatter key(s) {', '.join(repr(key) for key in unknown)} are not "
f"named by the {profile.index.name} policy, which pins "
f"{profile.index.root_frontmatter or '()'} — writing an unnamed key would "
"put a value in a file no reader of this contract looks at",
code="index_root_frontmatter_unexpected",
)
declared = [key for key in profile.index.root_frontmatter if key in values]
if not declared:
return ""
lines = "".join(f"{key}: {values[key]}\n" for key in declared)
return f"---\n{lines}---\n\n"
index_path.write_bytes(f"{prefix}- [{label}]({target_name})\n".encode())
def materialize_bundle(
@ -345,8 +249,6 @@ def materialize_bundle(
*,
allow_network: bool = False,
http_get: HttpGet | None = None,
profile: BundleProfile = DEFAULT,
root_frontmatter_values: Mapping[str, str] | None = None,
) -> IngestResult:
"""Materialize a manifest's extractions into an OKF bundle (§5).
@ -358,34 +260,8 @@ def materialize_bundle(
seam (default urllib_get, the only socket path; ignored for `file`/`sql`)
so tests run socket-free (§11). Source calls are logged per §8 (which
source, when, row count) — never cell contents, never secrets.
`profile` selects the bundle contract: `DEFAULT` (commons' ingest-spec §5)
or `OKF_V0_2`. It is keyword-only behind the `*` the signature already
carried, so every three-positional call site stays source-compatible —
support for a new upstream version is additive, never a migration. The
profile governs the emitted frontmatter, the ownership stamp the collision
gate recognises, the concept filenames, and the index; it does NOT reach
manifest type validation, which runs against `DEFAULT` (the two policies
compare equal today). `STRICT_V1` is not supported here: its index policy
sets `per_directory` and `entries_match_directory`, neither of which this
materializer honours.
`root_frontmatter_values` supplies the values for the keys the profile's
index policy names — `okf_version` under `OKF_V0_2` (§8, §12). The split is
deliberate: the profile names the key, the caller owns the value, because
`okf_version`'s value tracks the upstream Google version and belongs to
catalog (E1). Offering a key the policy does not name is refused fail-fast,
before any disk mutation. Omitting the argument emits no block at all — §12
is a MAY, and none of upstream's reference bundles declares it.
The block is written only when the index is CREATED. A re-run into an
existing bundle leaves it untouched, which is what makes the second run
byte-identical to the first (A-E5).
"""
validate_ingested_at(ingested_at)
# Before any source access or disk mutation: a caller error here must not
# leave a partially written bundle behind.
root_frontmatter = _render_root_frontmatter(root_frontmatter_values or {}, profile=profile)
manifest_file = Path(manifest_path)
try:
raw = manifest_file.read_bytes()
@ -447,9 +323,9 @@ def materialize_bundle(
"source call: source=%s ingested_at=%s rows=%d", source.id, ingested_at, row_count
)
content = _render_concept_file(
manifest, extraction, body, ingested_at=ingested_at, stamp=stamp, profile=profile
manifest, extraction, body, ingested_at=ingested_at, stamp=stamp
)
staged.append((generated_filename(extraction.id, profile=profile), content))
staged.append((generated_filename(extraction.id), content))
# Disk phase.
bundle = Path(bundle_dir)
@ -460,9 +336,8 @@ def materialize_bundle(
# ingest stamp are ours to replace.
owned = {
path.name
for path in sorted(bundle.glob(f"*{profile.paths.concept_suffix}"))
if path.name != profile.index.name
and _is_ingest_owned(path, manifest_file.stem, profile=profile)
for path in sorted(bundle.glob("*.md"))
if path.name != INDEX_NAME and _is_ingest_owned(path, manifest_file.stem)
}
# §3 collision gate — BEFORE any mutation: a staged filename occupied by
# a file WITHOUT the stamp is curated content; never overwrite it.
@ -481,22 +356,16 @@ def materialize_bundle(
# §6 index generation — the last disk mutation. A fresh index gets
# bundle_summary as its body; links are appended in extraction order.
index_path = bundle / profile.index.name
index_path = bundle / INDEX_NAME
labels_by_target = {
generated_filename(extraction.id, profile=profile): extraction.title
for extraction in manifest.extractions
generated_filename(extraction.id): extraction.title for extraction in manifest.extractions
}
if not index_path.is_file():
write_bytes(bundle, profile.index.name, root_frontmatter + manifest.bundle_summary + "\n")
write_bytes(bundle, INDEX_NAME, manifest.bundle_summary + "\n")
else:
# Links whose target is an ingest-owned file removed this run MUST be
# removed; all other links — curated and promoted — are preserved.
_update_index_lines(index_path, owned - staged_names, labels_by_target, profile=profile)
_update_index_lines(index_path, owned - staged_names, labels_by_target)
for extraction in manifest.extractions:
link_in_index(
bundle,
generated_filename(extraction.id, profile=profile),
extraction.title,
profile=profile,
)
link_in_index(bundle, generated_filename(extraction.id), extraction.title)
return IngestResult(written=written, stamp=stamp)

764
src/llm_ingestion_okf/profiles.py
View file

@ -1,764 +0,0 @@
"""The bundle contract as configuration (Phase 3).
What a valid bundle looks like — which concept types exist, which frontmatter
keys are emitted and in which order, which filename namespaces the doors own,
and what an index line looks like — is a profile, not a set of constants
scattered across the doors. `DEFAULT` is exactly the ingest-spec v1 + Phase 2
contract, so nothing observable changes for a caller that never mentions a
profile; the golden fixtures are the byte-level proof.
Two things deliberately do NOT live here. Security is the guard's, always: no
disposition, origin or channel vocabulary belongs on a profile. And the
reserved `verdict` layer is a spec invariant rather than profile config — it is
refused at construction, so a profile admitting it cannot be built, let alone
passed to a door. The `timestamp`/`generated` pair is refused the same way, and
for the same reason: both are things a profile must not be able to express.
Profiles are constructed in code. Config-file loading and inheritance chains
are extension points, not v1 (settled with the operator at phase start).
"""
from __future__ import annotations
import re
from collections.abc import Collection, Mapping, Sequence
from dataclasses import dataclass, field, replace
# The one layer no profile may admit (ingest-spec §3): the promotion gate is
# the only path into it. Compared case-insensitively, as both doors already do.
RESERVED_OKF_TYPE = "verdict"
# The one key PAIR no profile may name (OKF §13.1): `timestamp` is readable as
# a legacy stand-in for `generated.at` only while `generated` is ABSENT, so a
# schema able to name both can describe a document that has neither. Stated as
# key names rather than as a judgement on values because every `generated` a
# schema can express today is a scalar (`_is_legal_value`) and therefore
# malformed as a v0.2 mapping — naming both IS the hazard here. When a value
# model can express a well-formed `generated`, this narrows with it.
# "Naming" spans EVERY field that puts a key in the schema's namespace — `order`,
# `required`, `allowed` and `nullable` alike. A field left out of that union is a
# hole in the gate, not a narrower gate: the namespace is open by default, so a
# key named only by `nullable` is admitted just as surely as an emitted one.
_TIMESTAMP_FALLBACK_PAIR = frozenset({"timestamp", "generated"})
@dataclass(frozen=True)
class TypeRejection:
"""Why a profile refuses an `okf_type`, for the door to frame and raise.
The policy does not raise: Door A refuses with `ManifestError` and Door B
with `MaterializationError`, so the refusal has to be reported rather than
thrown. `reason` completes the sentence "<label>okf_type ..." and `code` is
the stable `IngestError.code` the caller asserts on.
"""
reason: str
code: str
@dataclass(frozen=True)
class TypePolicy:
"""Which `okf_type` values a bundle admits.
`allowed` is `None` for an open set — Phase 1/2 accept any type the
operator names — or a closed enum. Either way the reserved layer is
excluded, and a closed set that names it fails at construction.
"""
allowed: frozenset[str] | None = None
def __post_init__(self) -> None:
if self.allowed is None:
return
named = sorted(value for value in self.allowed if value.lower() == RESERVED_OKF_TYPE)
if named:
raise ValueError(
f"a profile must not admit the reserved {RESERVED_OKF_TYPE!r} layer "
f"(got {', '.join(repr(value) for value in named)}) — the promotion "
"gate is the only path into it (ingest-spec §3)"
)
def rejection(self, okf_type: str) -> TypeRejection | None:
"""The refusal for `okf_type`, or `None` when the profile admits it."""
if okf_type.lower() == RESERVED_OKF_TYPE:
return TypeRejection(
reason=f"must not be {RESERVED_OKF_TYPE!r} (reserved layer)",
code="okf_type_reserved",
)
if self.allowed is not None and okf_type not in self.allowed:
return TypeRejection(
reason=f"must be one of {', '.join(sorted(self.allowed))}, got {okf_type!r}",
code="okf_type_not_allowed",
)
return None
@dataclass(frozen=True)
class SchemaViolation:
"""One way a document's frontmatter departs from the schema.
Reported rather than raised, like `TypeRejection`: a document can depart in
several ways at once, and the caller decides whether that is a refusal, a
report line, or a migration to-do. `code` is the stable machine-readable
part; `reason` completes the sentence "<key> ...".
"""
key: str
reason: str
code: str
def _is_legal_value(value: object) -> bool:
"""A string scalar, or a non-empty list of strings — nothing else.
The list shape is real, not hypothetical: the proving consumer's `Release`
pages carry block lists. A schema that assumed scalars-only would reject
pages they ship today.
"""
if isinstance(value, str):
return True
if isinstance(value, list):
return bool(value) and all(isinstance(item, str) for item in value)
return False
@dataclass(frozen=True)
class FrontmatterSchema:
"""The frontmatter key namespace, what it admits, and how it is emitted.
`order` is the canonical emission order and `collapsed_keys` names the keys
whose whitespace runs collapse to single spaces on the way out. The DEFAULT
order spans both doors' key sets: Door A emits seven of these keys and Door
B six, and each door's subset comes out in exactly the order it wrote by
hand in Phases 1 and 2.
The judging fields — `required`, `allowed`, `nullable`, `key_pattern` — are
read by `violations`. `allowed` is `None` for an open namespace, which is
what DEFAULT keeps: Door A emits seven keys, Door B six, and Door C writes
an imported concept verbatim with whatever the sender wrote, so a closed
namespace or a required set on DEFAULT would declare invalid the very
bundles this library produces.
"""
order: tuple[str, ...]
collapsed_keys: frozenset[str] = field(default_factory=frozenset)
required: frozenset[str] = field(default_factory=frozenset)
allowed: frozenset[str] | None = None
nullable: frozenset[str] = field(default_factory=frozenset)
key_pattern: re.Pattern[str] | None = None
def __post_init__(self) -> None:
named = set(self.order) | set(self.required) | set(self.nullable)
if self.allowed is not None:
named |= set(self.allowed)
if _TIMESTAMP_FALLBACK_PAIR <= named:
raise ValueError(
"a profile must not name both 'timestamp' and 'generated' (OKF "
"§13.1 grants the timestamp fallback only while `generated` is "
"ABSENT, so a schema naming both can describe a document with "
"neither a valid `generated.at` nor an eligible fallback)"
)
if self.allowed is None:
return
for label, keys in (("required", self.required), ("nullable", self.nullable)):
stray = sorted(keys - self.allowed)
if stray:
raise ValueError(
f"{label} keys must be inside the allowlist "
f"(got {', '.join(repr(key) for key in stray)}) — a schema that "
"demands a key it also forbids can never be satisfied"
)
def violations(self, values: Mapping[str, object]) -> tuple[SchemaViolation, ...]:
"""Every way `values` departs from this schema, deterministically ordered.
Sorted by `(key, code)` rather than reported in mapping order: two runs
over the same document must produce the same report, and mapping order
is an accident of how the document was parsed.
Takes an already-parsed mapping. Parsing strict frontmatter is a
separate concern — this library's line-oriented parser cannot represent
the block lists and nulls a strict schema admits, which is why Door C
writes imported concepts verbatim rather than round-tripping them.
"""
found: list[SchemaViolation] = []
for key in self.required - set(values):
found.append(SchemaViolation(key, "is required and absent", "frontmatter_key_missing"))
for key, value in values.items():
if self.key_pattern is not None and not self.key_pattern.fullmatch(key):
found.append(
SchemaViolation(
key,
f"is not a legal key name (must match {self.key_pattern.pattern})",
"frontmatter_key_malformed",
)
)
continue
if self.allowed is not None and key not in self.allowed:
found.append(
SchemaViolation(
key, "is not on the key allowlist", "frontmatter_key_not_allowed"
)
)
continue
if value is None:
if key not in self.nullable:
found.append(SchemaViolation(key, "must not be null", "frontmatter_value_null"))
continue
if not _is_legal_value(value):
found.append(
SchemaViolation(
key,
"must be a string or a non-empty list of strings",
"frontmatter_value_shape",
)
)
return tuple(sorted(found, key=lambda violation: (violation.key, violation.code)))
def emit(self, values: Mapping[str, str]) -> str:
"""Render `values` as line-oriented `key: value`, one line per key.
Commons decision D1: the keys named in `order` come first, in that
order, followed by any remaining keys SORTED. Ordering the tail rather
than trusting insertion order is what makes a regeneration over the
same data byte-identical.
Returns the lines only — the caller owns the `---` fences.
"""
named = [key for key in self.order if key in values]
tail = sorted(key for key in values if key not in self.order)
return "\n".join(f"{key}: {self._render(key, values[key])}" for key in [*named, *tail])
def _render(self, key: str, value: str) -> str:
# §5 mandates whitespace-run collapse for `source_query` only
# (ingest-spec.md:140-141), where a legitimately multi-line SQL SELECT
# must render on one line. Every other value is validated single-line
# at load and emitted verbatim — validation, not repair — so
# operator-supplied bytes (e.g. a title's internal double space)
# survive.
return " ".join(value.split()) if key in self.collapsed_keys else value
# The v0.1 ingest stamp. A literal rather than a configurable value: it is what
# every bundle this library has already written carries, and recognising it is
# what keeps those bundles re-runnable under a later profile.
_V0_1_STAMP = "true"
@dataclass(frozen=True)
class OwnershipPolicy:
"""The `generated` value this profile writes, and the values it owns back.
Ownership is the §3 collision gate's question — may this run replace the
file already sitting at a generated name? — and the answer is a profile's,
because the stamp differs per profile. v0.1 writes the literal `true`; v0.2
writes `generated: { by: <actor>, at: <ingested_at> }` (§5), where the actor
takes §7's `process:<id>` form.
`actor` is `None` for the v0.1 stamp. Where it is set it carries no version,
deliberately: the value sits inside a byte-compared golden, so a producer
version there would fire golden regression on every release without any
contract having changed, and would make a shared cross-implementation
fixture impossible by construction (plan V1(d), operator 2026-07-27).
Recognition is ONE-WAY, and both directions are decisions rather than
accidents. A v0.2 profile owns the v0.1 stamp as well, so a bundle written
under `DEFAULT` re-runs IN PLACE — the black-box promise is that an upstream
release costs a consumer a re-run and nothing more. The reverse is refused:
`DEFAULT` meeting a v0.2 file fails the run rather than replacing a file
whose shape it does not read (V-A3).
The v0.2 test is a PREFIX rather than an equality, because the value carries
`ingested_at` and therefore differs on every run by design. It works because
`parse_frontmatter` returns the whole flow mapping as one opaque string
(V-A2) — no structure this library cannot yet read is parsed here.
"""
actor: str | None = None
def stamp(self, ingested_at: str) -> str:
"""The `generated` value a run at `ingested_at` writes."""
if self.actor is None:
return _V0_1_STAMP
return f"{{ by: {self.actor}, at: {ingested_at} }}"
def owns(self, value: str) -> bool:
"""Whether a `generated` value read back marks this library's output."""
if value == _V0_1_STAMP:
return True
if self.actor is None:
return False
return value.startswith(f"{{ by: {self.actor},")
@dataclass(frozen=True)
class PathPolicy:
"""The filename namespaces the three doors write into.
Each prefix keeps its door's generated names disjoint from `index.md`, from
the other doors, and from `promoted-verdict-*`, for every id the grammar
admits. The id grammar itself stays in `materialize`: it is a pattern the
slugger derives a separator class from, not a name to configure.
"""
concept_suffix: str
ingest_prefix: str
inbox_prefix: str
import_prefix: str
def _split_frontmatter(text: str) -> tuple[dict[str, str], list[str]]:
"""The leading `---` block as ordered keys, and the body lines after it.
Line-oriented, the same shape `materialize.parse_frontmatter` reads —
duplicated rather than imported because `materialize` imports this module,
and because that one takes a path where an index reader has only text.
"""
lines = text.splitlines()
if not lines or lines[0].strip() != "---":
return {}, lines
head: dict[str, str] = {}
for offset, line in enumerate(lines[1:], start=1):
if line.strip() == "---":
return head, lines[offset + 1 :]
key, sep, value = line.partition(":")
if sep:
head[key.strip()] = value.strip()
return head, []
@dataclass(frozen=True)
class IndexViolation:
"""One way an index file departs from the policy.
Reported rather than raised, like `SchemaViolation`. `subject` is whatever
the finding is about — a line, a link target, or a frontmatter key — and
`code` is the stable machine-readable part.
"""
subject: str
reason: str
code: str
@dataclass(frozen=True)
class IndexPolicy:
"""The index file, the shape of its entries, and where indexes must exist.
`link_template` renders a managed line and `link_pattern` recognises one.
Both are carried because §6 does both — append on write, rewrite on
maintenance — and a round-trip test is what keeps the pair honest. The
pattern is anchored to the whole line by construction: removal keys on this
exact shape, never a bare substring, so curated prose that mentions a
target inline survives verbatim.
The judging fields are read by `violations` and `required_indexes`, and
every one of them is off under DEFAULT. They exist because two consumers
disagree about this file in ways no single shape can hold:
- `per_directory` — an index at every level, or only at the bundle root.
Upstream OKF binds `index.md` to the root ALONE, so a default demanding
one per level would declare upstream-conforming bundles invalid. Policy,
never an OKF rule.
- `heading_required` / `allows_prose` — the wiki's index is a generated
heading plus entries and nothing else; the catalog's *requires*
progressive-disclosure prose. Opposite requirements, both expressible.
- `entries_match_directory` — whether the index must match the directory
exactly, in both directions. The wiki enforces this at ERROR
(`BUNDLE_INDEX_COMPLETE`); the catalog holds that an index is an
AUTHORED count of a directory's children and that a validator
enumerating the directory has implemented the wrong contract. Neither
posture is baked in, and this library never enumerates anything: the
caller supplies the listing, and a policy that does not judge it refuses
to be handed one at all — so code written to the wrong contract fails at
the call rather than passing every test one would think to write.
- `root_frontmatter` — the ordered keys the ROOT index may carry, where
nested indexes carry none. Confirmed independently in both consumers, so
the asymmetry is shape rather than one repo's preference. Naming a key
here PERMITS it and fixes its position; it does not demand it.
- `root_frontmatter_required` — the subset that must actually be present.
Separate from the tuple above because the two are different claims, and
collapsing them was P1-F1: `OKF_V0_2` names `okf_version` to fix its
position, but upstream §8/§12 grant it as a MAY, and judging the pair as
one field failed 14 of 17 real bundles — upstream's own four included —
each with exactly this one violation, while our emitter treated the same
key as optional. A profile whose consumer really does demand the keys
says so here, which is what `STRICT_V1` does.
"""
name: str
link_template: str
link_pattern: re.Pattern[str]
per_directory: bool = False
heading_required: bool = False
allows_prose: bool = True
entries_match_directory: bool = False
root_frontmatter: tuple[str, ...] = ()
root_frontmatter_required: frozenset[str] = field(default_factory=frozenset)
def __post_init__(self) -> None:
stray = sorted(self.root_frontmatter_required - set(self.root_frontmatter))
if stray:
raise ValueError(
f"required root keys must be named in the ordered set "
f"(got {', '.join(repr(key) for key in stray)}) — a key demanded "
"but never named could not be judged for position"
)
if self.requires_description and "description" not in self.link_pattern.groupindex:
raise ValueError(
"link_template names {description} but link_pattern has no "
"'description' group — the pair would round-trip in one "
"direction only"
)
@property
def requires_description(self) -> bool:
"""Whether an entry carries a description alongside label and target."""
return "{description}" in self.link_template
def render_link(self, label: str, target: str, description: str | None = None) -> str:
if self.requires_description and description is None:
raise ValueError(
"this index policy's entries carry a description; rendering "
"without one emits a half-written entry that parses as nothing"
)
if not self.requires_description and description is not None:
raise ValueError(
"this index policy's entries carry no description; the value "
"offered would be dropped silently"
)
return self.link_template.format(label=label, target=target, description=description)
def required_indexes(self, directories: Sequence[str]) -> tuple[str, ...]:
"""The index paths this policy requires, given the caller's directories.
Bundle-relative, with `""` for the root. The directories are an
argument precisely so that nothing here reaches the filesystem.
"""
if not self.per_directory:
return (self.name,)
return tuple(sorted(f"{d}/{self.name}" if d else self.name for d in directories))
def violations(
self,
text: str,
*,
is_root: bool = False,
expected_targets: Collection[str] | None = None,
) -> tuple[IndexViolation, ...]:
"""Every way `text` departs from this policy, deterministically ordered.
`expected_targets` is the directory's contents as the CALLER sees them,
required exactly when `entries_match_directory` is set and refused
otherwise. Sorted by `(subject, code)` so two runs over the same file
agree.
"""
if self.entries_match_directory and expected_targets is None:
raise ValueError(
"this index policy judges the index against the directory, so "
"a listing is required — skipping the check silently would "
"pass a gate whose input never arrived"
)
if not self.entries_match_directory and expected_targets is not None:
raise ValueError(
"this index policy's index is authored rather than derived "
"from the directory, so a listing has nothing to judge"
)
found: list[IndexViolation] = []
head, body = _split_frontmatter(text)
found.extend(self._frontmatter_violations(head, is_root=is_root))
headings: list[str] = []
listed: set[str] = set()
for line in body:
stripped = line.strip()
if not stripped:
continue
if line.startswith("# "):
headings.append(line)
continue
match = self.link_pattern.match(line)
if match is not None:
listed.add(match.group("target"))
continue
if not self.allows_prose:
found.append(
IndexViolation(
stripped,
"is neither the heading nor an entry",
"index_prose_not_allowed",
)
)
if self.heading_required:
if not headings:
found.append(
IndexViolation("", "the index carries no `# ` heading", "index_heading_missing")
)
found.extend(
IndexViolation(extra, "is a second `# ` heading", "index_heading_extra")
for extra in headings[1:]
)
if expected_targets is not None:
expected = set(expected_targets)
found.extend(
IndexViolation(
target, "is in the directory but not in the index", "index_entry_missing"
)
for target in expected - listed
)
found.extend(
IndexViolation(
target, "is in the index but not in the directory", "index_entry_unexpected"
)
for target in listed - expected
)
return tuple(sorted(found, key=lambda violation: (violation.subject, violation.code)))
def _frontmatter_violations(
self, head: Mapping[str, str], *, is_root: bool
) -> list[IndexViolation]:
if not self.root_frontmatter:
return []
if not is_root:
# One finding about the block, not one per key: a nested index
# carrying frontmatter is a single structural fact.
first = next(iter(head), None)
if first is None:
return []
return [
IndexViolation(
first,
"is frontmatter on a nested index, which carries none",
"index_root_frontmatter_unexpected",
)
]
found = [
IndexViolation(key, "is pinned on the root index and absent", "index_root_key_missing")
for key in self.root_frontmatter
if key in self.root_frontmatter_required and key not in head
]
present = [key for key in head if key in self.root_frontmatter]
declared = [key for key in self.root_frontmatter if key in head]
if present != declared:
found.append(
IndexViolation(
present[0],
f"breaks the pinned key order {', '.join(self.root_frontmatter)}",
"index_root_key_order",
)
)
return found
@dataclass(frozen=True)
class BundleProfile:
"""One bundle contract: types, frontmatter, filenames, index."""
types: TypePolicy
frontmatter: FrontmatterSchema
paths: PathPolicy
index: IndexPolicy
ownership: OwnershipPolicy = field(default_factory=OwnershipPolicy)
# The ingest-spec v1 + Phase 2 contract, unchanged. Every value here was a
# constant in `manifest`, `materialize`, `inbox` or `importer` before this
# module existed; the golden suite is what proves the move changed no bytes.
DEFAULT = BundleProfile(
types=TypePolicy(allowed=None),
frontmatter=FrontmatterSchema(
# Door A's seven keys and Door B's six, merged into one order that
# contains both as subsequences — neither door's output moves.
order=(
"type",
"title",
"source_system",
"source_query",
"source_file",
"source_sha256",
"ingested_at",
"ingest_manifest",
"generated",
),
collapsed_keys=frozenset({"source_query"}),
),
paths=PathPolicy(
concept_suffix=".md",
ingest_prefix="ingest-",
inbox_prefix="inbox-",
import_prefix="import-",
),
index=IndexPolicy(
name="index.md",
link_template="- [{label}]({target})",
link_pattern=re.compile(r"^- \[(?P<label>[^\]]*)\]\((?P<target>[^)]+)\)$"),
),
)
# `FRONTMATTER_KEY_ORDER` in the proving consumer's `bundle.py`: an eleven-key
# allowlist that doubles as canonical emission order. The first eight are the
# required set their operator ratified on 2026-07-25 (measured present on
# 522/522 documents); the last three are layer-specific and stay optional.
# Deriving both sets from this one tuple is what keeps them from drifting apart
# — their BUNDLE_HASH_REGISTRY gate depends on the order.
_STRICT_V1_KEY_ORDER = (
"type",
"title",
"description",
"timestamp",
"layer",
"source_tier",
"source_url",
"source_sha",
"version",
"date",
"summary",
)
# The `claude-code-llm-wiki` contract, from `docs/phase-3-split-table.md`.
#
# Deliberately stricter than the consumer's own validator: their REQUIRED_KEYS
# constant is the first four keys, which is their emit-path minimum rather than
# their contract. If they ever ship a document class that legitimately lacks
# `source_sha`, the agreed resolution is to bump this profile — they send the
# requirement before shipping the class, not after.
#
# `paths` is DEFAULT's and that is the whole truth for it: the filename
# namespaces are an ingest-spec invariant about what the doors write, not a
# consumer preference.
#
# The index shape is read out of their code rather than recalled:
# `bundle.py:498-524` renders `* [Title](target) - description` with
# subdirectories linking to their own index; `bundle.py:527-567` walks every
# level, writing frontmatter at the bundle root only and a bare `# ` heading
# below it; `validate.py:1081-1120` (gate BUNDLE_INDEX_COMPLETE, ERROR) demands
# the index and the directory match exactly in both directions.
STRICT_V1 = BundleProfile(
types=TypePolicy(allowed=frozenset({"Concept", "Guide", "Reference", "Release"})),
frontmatter=FrontmatterSchema(
order=_STRICT_V1_KEY_ORDER,
required=frozenset(_STRICT_V1_KEY_ORDER[:8]),
allowed=frozenset(_STRICT_V1_KEY_ORDER),
nullable=frozenset({"summary"}),
key_pattern=re.compile(r"^[a-z_]+$"),
),
paths=DEFAULT.paths,
index=IndexPolicy(
name="index.md",
link_template="* [{label}]({target}) - {description}",
link_pattern=re.compile(
r"^\* \[(?P<label>[^\]]*)\]\((?P<target>[^)\s]+)\) - (?P<description>.+)$"
),
per_directory=True,
heading_required=True,
allows_prose=False,
entries_match_directory=True,
root_frontmatter=("okf_version", "bundle_profile", "okf_spec_commit"),
# The proving consumer demands all three, not merely permits them: their
# root index carries exactly these keys in exactly this order on every
# bundle measured (`c5141f8`). Stated explicitly so that separating
# "permitted" from "required" costs them nothing.
root_frontmatter_required=frozenset({"okf_version", "bundle_profile", "okf_spec_commit"}),
),
)
# OKF v0.2, as an ADDITIVE profile: `DEFAULT` states commons' ingest-spec §5
# layer and keeps stating it, so nothing here migrates anything. The key order
# is DEFAULT's followed by the §5 families v0.2 adds, which is also the order
# upstream's own reference bundles emit them in (`generated` before `sources`).
#
# Naming a family is not writing it. `verified`, `status` and `stale_after` are
# expressible so a caller can emit them in canonical order and so the schema can
# judge a document that carries them; Door A writes none of them, because a
# field with no reader is not written. Their structured v0.2 values — block
# lists of mappings — are beyond this library's value model until D1's reader
# lands; that is a reader gap, not an ordering one.
_OKF_V0_2_KEY_ORDER = (
*DEFAULT.frontmatter.order,
"sources",
"verified",
"status",
"stale_after",
)
# PROVISIONAL. Shipped first as a pre-release (`v0.5.0a1`) to a named pilot set
# — `portfolio-optimiser-claude`, the plugin marketplace catalog, and
# `claude-code-llm-wiki` — and this surface may change on their feedback without
# a deprecation cycle. Saying so is what buys the freedom to act on the
# feedback; discovering it later is what would make the pilot a de-facto
# release. The versioned constants are the stable binding.
#
# Two things this profile deliberately does NOT do:
#
# - **It closes nothing.** §14 forbids a conformant consumer to reject on an
# unknown `type` value or on unknown additional keys, so an allowlist or a key
# pattern here would put the profile in violation of the version it is named
# for. `type` is required and is the only one (§4, §11).
# - **It NAMES `okf_version` but never carries its value.** The value tracks the
# upstream Google version and belongs to catalog (decision E1), so a constant
# here would be this repo claiming a decision it does not own — and the one
# that would have to be chased on every upstream release. The caller supplies
# it (`materialize_bundle(..., root_frontmatter_values=...)`); this policy
# fixes only the key and its position.
#
# WHERE it goes was open until 2026-07-31 between upstream's root-index
# frontmatter block and catalog's body-line convention. Catalog verified
# upstream themselves at the pinned commit `3fcbb9f` and reported §8:509-510
# ("Index files contain no frontmatter, with one exception: a bundle-root
# `index.md` MAY carry an `okf_version` key") and §12:773-775 ("in a
# bundle-root `index.md` frontmatter block (the only place frontmatter is
# permitted in an `index.md`)"). Frontmatter it is; their own spec diverges
# from upstream here, and that divergence is theirs to resolve.
#
# Declaring it stays a MAY: none of upstream's four reference bundles carries
# the key at all (catalog grepped `okf/bundles` and `okf/samples`: zero hits),
# so omitting `root_frontmatter_values` emits no block. The JUDGE says the same
# thing since P1-F1 — the key is named here and left out of
# `root_frontmatter_required` — because for one release it did not, and a
# bundle exercising the MAY was reported as violating by the very profile that
# had emitted it correctly.
#
# **Measured limitation (guard 0.2.0, 2026-07-26):** a bundle emitted under this
# profile cannot be read back through a guard-gated import. The guard's T2
# frontmatter grammar admits scalars and flat lists of strings, and refuses every
# route to a mapping — flow on the disallowed-indicator set, block on the
# nested-mapping check, dotted keys on the key pattern. So `generated` as the
# mapping v0.2 specifies has no expressible form through that gate at all. This
# binds what can be IMPORTED (Door C), never what we emit: Door B's
# `screen_output` does not run that parser.
OKF_V0_2 = BundleProfile(
types=TypePolicy(allowed=None),
frontmatter=FrontmatterSchema(
order=_OKF_V0_2_KEY_ORDER,
collapsed_keys=DEFAULT.frontmatter.collapsed_keys,
required=frozenset({"type"}),
),
paths=DEFAULT.paths,
# DEFAULT's index in every respect but one: the root MAY carry `okf_version`
# (§8, §12). Built with `replace` rather than restated so a later change to
# the shared shape cannot drift between the two.
index=replace(DEFAULT.index, root_frontmatter=("okf_version",)),
ownership=OwnershipPolicy(actor="process:llm-ingestion-okf"),
)
# "The latest version supported as STABLE", not the latest present in this
# module. It therefore keeps v0.1 semantics for as long as v0.2 is provisional,
# and flipping it is the GA event — one auditable action rather than a side
# effect of a merge.
#
# The tradeoff is stated rather than hidden: an alias that moves means a consumer
# bound to it inherits upstream's breaking changes on a library upgrade. The
# versioned constants are the stable binding and are what a consumer should pin;
# this is for callers who have explicitly opted into tracking.
OKF_LATEST = DEFAULT

27
tests/test_golden.py
View file

@ -14,11 +14,9 @@ from pathlib import Path
import pytest
from llm_ingestion_okf.materialize import materialize_bundle
from llm_ingestion_okf.profiles import OKF_V0_2
EXAMPLES = Path(__file__).parent.parent / "examples"
INGESTED_AT_NAME = "ingested-at.txt"
OKF_VERSION_NAME = "okf-version.txt"
def fixture_backed_get(fixture_dir: Path) -> object:
@ -44,34 +42,11 @@ def materialize_case(case_dir: Path, out_dir: Path, monkeypatch: pytest.MonkeyPa
allow_network=True,
http_get=fixture_backed_get(case_dir / "fixture"), # type: ignore[arg-type]
)
elif case_dir.name.endswith("okf-v0-2"):
# The declared version is fixture DATA, read from the case, never a
# constant in this file. Its value tracks the upstream Google version
# and belongs to catalog (E1); a literal here would be this repo
# claiming a decision it does not own, and the golden would then have
# to be chased on every upstream release.
materialize_bundle(
case_dir / "manifest.json",
out_dir,
ingested_at,
profile=OKF_V0_2,
root_frontmatter_values={
"okf_version": (case_dir / OKF_VERSION_NAME).read_text(encoding="utf-8").strip()
},
)
else:
materialize_bundle(case_dir / "manifest.json", out_dir, ingested_at)
@pytest.mark.parametrize(
"case",
[
"ingest-golden-file",
"ingest-golden-sql",
"ingest-golden-http",
"ingest-golden-okf-v0-2",
],
)
@pytest.mark.parametrize("case", ["ingest-golden-file", "ingest-golden-sql", "ingest-golden-http"])
def test_golden_case_byte_for_byte(
case: str, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:

333
tests/test_index_policy.py
View file

@ -1,333 +0,0 @@
"""The index policy as configuration, and the reader that judges it (Phase 3).
Three index shapes have to be expressible without either becoming the default:
- **DEFAULT** — the ingest-spec shape. One index at the bundle root, entries
`- [label](target)`, curated prose alongside them, and no relationship
between the index and the directory it sits in.
- **STRICT_V1** — `claude-code-llm-wiki`, read out of their code rather than
recalled: an index at every level (`bundle.py:527-567`), exactly one `# `
heading, entries `* [Title](link) - description` and nothing else, the root
index carrying `okf_version`/`bundle_profile`/`okf_spec_commit` while nested
indexes carry no frontmatter at all, and gate `BUNDLE_INDEX_COMPLETE`
(`validate.py:1081-1120`, severity ERROR) demanding the index and the
directory match exactly in both directions.
- **The second-brain shape** — not built here, but it must not be excluded.
The catalog reported on 2026-07-25 that their index is an *authored* count
of a directory's children and never a filesystem lookup, that a reader or
validator enumerating the directory has implemented the wrong contract, and
that they require prose in the index but no per-entry description.
The two consumers are directly opposed on exactly one point: the wiki mandates
the directory-derived match the catalog calls the wrong contract. That is what
settles the design. Every one of these differences is a policy field, none is
a library invariant, and `violations` never touches the filesystem — a caller
supplies the listing, and a profile whose index is authored refuses to be
handed one at all.
"""
from __future__ import annotations
import re
import pytest
from llm_ingestion_okf.profiles import DEFAULT, OKF_V0_2, STRICT_V1, IndexPolicy
# --- DEFAULT keeps the Phase 1/2 shape (assumption C1) --------------------
def test_default_index_policy_judges_nothing() -> None:
"""Every new field is off under DEFAULT, so the reader has nothing to say.
Door A emits an index with a curated summary above managed links, Door B
and Door C append to one that may be empty. A judging default would
declare invalid the bundles this library itself produces — the same
reasoning that kept a required key set off DEFAULT's frontmatter schema.
"""
text = "# Sales bundle\n\nCurated prose the operator wrote.\n\n- [Sales](ingest-sales.md)\n"
assert DEFAULT.index.violations(text) == ()
assert DEFAULT.index.violations(text, is_root=True) == ()
def test_default_index_is_root_only_because_upstream_binds_it_there() -> None:
"""Upstream OKF binds `index.md` to the bundle root alone.
A nested directory without its own index is not an error there, which is
why per-directory indexing is policy rather than an OKF rule: a DEFAULT
demanding one per level would declare upstream-conforming bundles invalid.
"""
assert DEFAULT.index.per_directory is False
assert DEFAULT.index.required_indexes(["", "concepts", "concepts/hooks"]) == ("index.md",)
def test_default_index_link_is_unchanged_and_needs_no_description() -> None:
assert DEFAULT.index.requires_description is False
assert DEFAULT.index.render_link("Sales", "ingest-sales.md") == "- [Sales](ingest-sales.md)"
# --- the description on an entry ------------------------------------------
def test_render_link_carries_a_description_when_the_template_names_one() -> None:
assert STRICT_V1.index.requires_description is True
line = STRICT_V1.index.render_link("Hooks", "hooks.md", "How hooks fire.")
assert line == "* [Hooks](hooks.md) - How hooks fire."
def test_render_link_refuses_to_emit_a_half_written_entry() -> None:
"""A template naming `{description}` and a caller omitting it must not
silently render `* [T](t) - `, which parses as nothing and reads as an
entry."""
with pytest.raises(ValueError, match="description"):
STRICT_V1.index.render_link("Hooks", "hooks.md")
def test_a_description_offered_to_a_template_without_one_is_refused() -> None:
"""The mirror case: silently dropping it would lose operator bytes."""
with pytest.raises(ValueError, match="description"):
DEFAULT.index.render_link("Sales", "ingest-sales.md", "dropped on the floor")
def test_strict_v1_link_template_and_pattern_describe_the_same_shape() -> None:
"""Rendering and recognising are separate operations; this is the pair's
round-trip, the same test DEFAULT's shape already carries."""
for label, target, description in (
("Hooks", "hooks.md", "How hooks fire."),
("Concepts", "concepts/index.md", "Machine-generated index of concepts."),
("", "a.md", "-"),
):
line = STRICT_V1.index.render_link(label, target, description)
match = STRICT_V1.index.link_pattern.match(line)
assert match is not None
assert match.group("label") == label
assert match.group("target") == target
assert match.group("description") == description
def test_strict_v1_link_pattern_is_anchored_to_the_whole_line() -> None:
assert STRICT_V1.index.link_pattern.match("see * [Hooks](hooks.md) - x below") is None
# --- per-directory indexes ------------------------------------------------
def test_per_directory_policy_names_one_index_per_level() -> None:
"""The caller supplies the directories; the policy never enumerates them.
This is the catalog's construction rule honoured at the type level: the
library cannot list a directory because it is never given one to list.
"""
assert STRICT_V1.index.per_directory is True
assert STRICT_V1.index.required_indexes(["", "concepts", "concepts/hooks"]) == (
"concepts/hooks/index.md",
"concepts/index.md",
"index.md",
)
# --- the reader: headings -------------------------------------------------
ROOT_HEAD = "---\nokf_version: 0.1\nbundle_profile: strict-v1\nokf_spec_commit: d44368c\n---\n"
def test_strict_index_wants_exactly_one_h1() -> None:
body = "\n* [Hooks](hooks.md) - How hooks fire.\n"
missing = STRICT_V1.index.violations(
ROOT_HEAD + body, is_root=True, expected_targets={"hooks.md"}
)
assert [violation.code for violation in missing] == ["index_heading_missing"]
two = ROOT_HEAD + "\n# Wiki\n\n# Again\n" + body
extra = STRICT_V1.index.violations(two, is_root=True, expected_targets={"hooks.md"})
assert [violation.code for violation in extra] == ["index_heading_extra"]
assert extra[0].subject == "# Again"
def test_a_second_level_heading_is_prose_not_a_heading() -> None:
"""Only `# ` is the heading; `## ` is body text the strict shape forbids."""
text = ROOT_HEAD + "\n# Wiki\n\n## Section\n"
codes = [v.code for v in STRICT_V1.index.violations(text, is_root=True, expected_targets=set())]
assert codes == ["index_prose_not_allowed"]
# --- the reader: prose ----------------------------------------------------
def test_strict_index_admits_nothing_but_the_heading_and_the_entries() -> None:
text = ROOT_HEAD + "\n# Wiki\n\nA sentence the generator would never write.\n"
found = STRICT_V1.index.violations(text, is_root=True, expected_targets=set())
assert [v.code for v in found] == ["index_prose_not_allowed"]
assert found[0].subject == "A sentence the generator would never write."
def test_prose_is_the_catalog_shape_and_stays_expressible() -> None:
"""Their index requires progressive-disclosure prose and no per-entry
description — the opposite of the wiki on both counts, and both have to be
constructible."""
second_brain = IndexPolicy(
name="index.md",
link_template="- [{label}]({target})",
link_pattern=DEFAULT.index.link_pattern,
per_directory=True,
heading_required=True,
allows_prose=True,
root_frontmatter=("okf_version", "okf_layout"),
)
text = "# Notes\n\nWhat lives here and why.\n\n- [A note](a.md)\n"
assert second_brain.violations(text) == ()
assert second_brain.entries_match_directory is False
# --- the reader: directory match, both directions -------------------------
def test_the_match_is_bidirectional_and_names_each_side() -> None:
text = ROOT_HEAD + "\n# Wiki\n\n* [Hooks](hooks.md) - How hooks fire.\n"
found = STRICT_V1.index.violations(
text, is_root=True, expected_targets={"skills.md", "concepts/index.md"}
)
assert [(v.subject, v.code) for v in found] == [
("concepts/index.md", "index_entry_missing"),
("hooks.md", "index_entry_unexpected"),
("skills.md", "index_entry_missing"),
]
def test_a_derived_index_cannot_be_judged_without_the_listing() -> None:
"""Skipping the check silently is the failure mode that matters: a gate
that quietly passes when its input is absent is worse than one that
raises."""
text = ROOT_HEAD + "\n# Wiki\n"
with pytest.raises(ValueError, match="listing"):
STRICT_V1.index.violations(text, is_root=True)
def test_an_authored_index_refuses_a_listing() -> None:
"""The catalog's rule, enforced rather than documented: a profile whose
index is authored cannot be asked to check it against the directory, so
code written to the wrong contract fails at the call instead of passing
every test anyone would think to write."""
with pytest.raises(ValueError, match="authored"):
DEFAULT.index.violations("- [Sales](ingest-sales.md)\n", expected_targets={"a.md"})
# --- the reader: root vs nested frontmatter -------------------------------
def test_the_root_index_pins_its_keys_in_order() -> None:
text = ROOT_HEAD + "\n# Wiki\n"
assert STRICT_V1.index.violations(text, is_root=True, expected_targets=set()) == ()
def test_a_root_index_missing_a_pinned_key_is_reported_per_key() -> None:
text = "---\nokf_version: 0.1\n---\n\n# Wiki\n"
found = STRICT_V1.index.violations(text, is_root=True, expected_targets=set())
assert [(v.subject, v.code) for v in found] == [
("bundle_profile", "index_root_key_missing"),
("okf_spec_commit", "index_root_key_missing"),
]
def test_the_pinned_keys_are_ordered_not_merely_present() -> None:
"""Their `ROOT_INDEX_KEY_ORDER` is what makes a regeneration byte-identical,
so an out-of-order root index is a finding even with every key present."""
text = "---\nbundle_profile: strict-v1\nokf_version: 0.1\nokf_spec_commit: d44368c\n---\n\n# Wiki\n"
found = STRICT_V1.index.violations(text, is_root=True, expected_targets=set())
assert [v.code for v in found] == ["index_root_key_order"]
# --- P1-F1: permitted-and-ordered is not the same claim as required -------
def test_a_permitted_root_key_may_be_omitted_without_a_finding() -> None:
"""P1-F1, measured 2026-07-31: `OKF_V0_2` names `okf_version` because
upstream §8/§12 say a bundle-root index MAY carry it — a permission, not a
demand. Judging the key as required made 14 of the 17 bundles swept fail
with exactly this one violation, upstream's own four reference bundles
among them, while D5's emitter treats the same key as optional (omitting
`root_frontmatter_values` writes no block). We emitted a MAY correctly and
graded it a MUST."""
text = "# Bundle\n\n- [Sales](ingest-sales.md)\n"
assert OKF_V0_2.index.violations(text, is_root=True) == ()
def test_the_permitted_key_is_still_ordered_and_still_root_only() -> None:
"""Separating the two meanings must not cost the checks that were right:
a nested index carrying the key is still a finding, and a root index that
carries it is still judged for order."""
nested = OKF_V0_2.index.violations("---\nokf_version: 0.2\n---\n\n# Concepts\n")
assert [(v.subject, v.code) for v in nested] == [
("okf_version", "index_root_frontmatter_unexpected")
]
root = OKF_V0_2.index.violations("---\nokf_version: 0.2\n---\n\n# Bundle\n", is_root=True)
assert root == ()
def test_a_required_root_key_is_still_reported_when_absent() -> None:
"""`STRICT_V1` mirrors the proving consumer's ratified contract, which
demands all three (they emit all three — measured across their bundle at
`c5141f8`). The fix separates the meanings; it does not relax them."""
found = STRICT_V1.index.violations(
"---\nokf_version: 0.1\n---\n\n# Wiki\n", is_root=True, expected_targets=set()
)
assert [(v.subject, v.code) for v in found] == [
("bundle_profile", "index_root_key_missing"),
("okf_spec_commit", "index_root_key_missing"),
]
def test_a_required_key_outside_the_ordered_set_fails_at_construction() -> None:
"""A key demanded but never named in the ordered tuple could never be
judged for position — the same contradiction `FrontmatterSchema` refuses
when `required` strays outside `allowed`."""
with pytest.raises(ValueError, match="ordered"):
IndexPolicy(
name="index.md",
link_template="- [{label}]({target})",
link_pattern=DEFAULT.index.link_pattern,
root_frontmatter=("okf_version",),
root_frontmatter_required=frozenset({"bundle_profile"}),
)
def test_a_nested_index_carries_no_frontmatter_at_all() -> None:
"""Confirmed independently in two consumers: the wiki writes frontmatter
only at the bundle root (`bundle.py:544-557`), and the catalog reports
their `okf_version`/`okf_layout` markers as root-exclusive. The asymmetry
is shape, not one repo's preference."""
found = STRICT_V1.index.violations(ROOT_HEAD + "\n# Concepts\n", expected_targets=set())
assert [(v.subject, v.code) for v in found] == [
("okf_version", "index_root_frontmatter_unexpected")
]
def test_a_nested_index_without_frontmatter_is_clean() -> None:
assert STRICT_V1.index.violations("# Concepts\n", expected_targets=set()) == ()
# --- determinism ----------------------------------------------------------
def test_violations_are_sorted_so_two_runs_agree() -> None:
text = "---\nokf_spec_commit: d\nokf_version: 0.1\n---\n\nz prose\na prose\n"
found = STRICT_V1.index.violations(text, is_root=True, expected_targets={"m.md"})
assert list(found) == sorted(found, key=lambda v: (v.subject, v.code))
assert found == STRICT_V1.index.violations(text, is_root=True, expected_targets={"m.md"})
def test_index_policy_stays_frozen() -> None:
with pytest.raises(Exception):
STRICT_V1.index.per_directory = False # type: ignore[misc]
def test_the_pattern_group_names_the_template_needs_are_present() -> None:
"""A template naming `{description}` whose pattern cannot recover one is a
pair that round-trips in only one direction."""
with pytest.raises(ValueError, match="description"):
IndexPolicy(
name="index.md",
link_template="* [{label}]({target}) - {description}",
link_pattern=re.compile(r"^\* \[(?P<label>[^\]]*)\]\((?P<target>[^)]+)\)$"),
)

343
tests/test_okf_v0_2_characterization.py
View file

@ -1,343 +0,0 @@
"""The safety net under OKF v0.2 support (plan steps 1-3).
Every test here characterizes behavior this library ALREADY has, before any
v0.2 code exists. Nothing is changed by them; that is the point. They are
written first because the one part of the v0.2 work we called risky —
`_is_ingest_owned`, the pre-mutation collision gate — is only safe to touch
with its current guarantees pinned.
Three groups, one per plan step:
1. **Characterization.** An inline flow mapping survives the scalar
`parse_frontmatter` verbatim (V-A2), a block list does not, and a v0.2
`generated` mapping can never make a file ingest-owned (V-A3). The last one
converts today's fail-safe from an accident into a documented guarantee:
a v0.2 file at a target name is refused, never overwritten.
2. **Door C consumer tolerance** (D3). OKF §14 forbids a consumer to reject on
five things; Door C is a consumer in that sense. One case per MUST NOT, plus
the §5.2 bare-`verified` mapping.
3. **No profile hard-codes an upstream version** (V4/V-A5). The `okf_version`
VALUE belongs to catalog (decision E1); a profile names the key and expresses
any value, which is what keeps "always the latest OKF version" from turning
into a constant this repo has to chase.
"""
from __future__ import annotations
import dataclasses
import json
import re
from collections.abc import Iterator
from pathlib import Path
from typing import Any
import pytest
from llm_ingestion_okf.errors import MaterializationError
from llm_ingestion_okf.materialize import (
_is_ingest_owned,
materialize_bundle,
parse_frontmatter,
)
from llm_ingestion_okf.profiles import DEFAULT, STRICT_V1, BundleProfile, FrontmatterSchema
from test_import_flow import StubImportGate, place, run
INGESTED_AT = "2026-07-25T12:00:00Z"
# The v0.2 `generated` shape, as an inline flow mapping — the form A-E3 pins
# for the coming profile, and the form the §5 single-line MUST admits.
V0_2_GENERATED = "{ by: llm-ingestion-okf/0.4.0, at: 2026-07-25T12:00:00Z }"
# --- step 1: what the scalar parser already does --------------------------
def test_an_inline_flow_mapping_survives_the_scalar_parser_verbatim(tmp_path: Path) -> None:
"""V-A2. `parse_frontmatter` splits on the FIRST colon, so a flow mapping —
colons and all — comes back as one opaque string. Byte-exact both ways:
what the emitter wrote is what the parser returns.
"""
values = {"type": "dataset", "title": "Orders", "generated": V0_2_GENERATED}
path = tmp_path / "concept.md"
path.write_bytes(f"---\n{DEFAULT.frontmatter.emit(values)}\n---\n\nBody.\n".encode())
assert path.read_bytes().decode("utf-8").splitlines()[3] == f"generated: {V0_2_GENERATED}"
assert parse_frontmatter(path) == values
def test_a_block_list_pollutes_the_scalar_parsers_key_space(tmp_path: Path) -> None:
"""The measured reason `sources` is emitted as an inline flow sequence.
Upstream's canonical `sources` is a block list of multi-key mappings. Read
through this line-oriented parser, each item line becomes a KEY: the list
disappears and `- id` / `resource` appear as frontmatter keys that no
document declared. `_is_ingest_owned` reads through this same parser, so
emitting the block form would have forced the gate and the parser to be
hardened in one step.
"""
path = tmp_path / "concept.md"
path.write_bytes(
b"---\ntype: dataset\nsources:\n - id: margin-standard\n"
b" resource: policies/margin-standard.md\n---\n\nBody.\n"
)
parsed = parse_frontmatter(path)
assert parsed["sources"] == ""
assert parsed["- id"] == "margin-standard"
assert parsed["resource"] == "policies/margin-standard.md"
def _stamped_concept(generated: str) -> bytes:
return (
"---\ntype: dataset\ntitle: Orders\n"
"ingest_manifest: manifest@0123456789abcdef\n"
f"generated: {generated}\n---\n\nBody.\n"
).encode()
def test_a_v0_1_generated_true_is_ingest_owned(tmp_path: Path) -> None:
"""The control for the test below: the stamp as v0.1 writes it DOES own."""
path = tmp_path / "ingest-orders.md"
path.write_bytes(_stamped_concept("true"))
assert _is_ingest_owned(path, "manifest") is True
def test_a_v0_2_generated_mapping_is_never_ingest_owned(tmp_path: Path) -> None:
"""V-A3. `generated` as a v0.2 mapping is not the string `true`, so the
predicate returns False and the file is not ours to replace.
The manifest reference is readable and matches — the refusal comes from
`generated` alone. This is also why `generated` can never become the
ownership predicate on its own (U3): upstream writes it for HAND-AUTHORED
files, so presence proves nothing about who generated what.
"""
path = tmp_path / "ingest-orders.md"
path.write_bytes(_stamped_concept(V0_2_GENERATED))
assert parse_frontmatter(path)["ingest_manifest"] == "manifest@0123456789abcdef"
assert _is_ingest_owned(path, "manifest") is False
def test_a_v0_2_file_at_a_target_name_is_refused_not_overwritten(tmp_path: Path) -> None:
"""The fail-safe, end to end: not-owned means the §3 collision gate refuses
the run before any mutation. A v0.2 shape arriving in a bundle Door A also
writes into can cost a run; it can never cost data.
"""
src = tmp_path / "src"
src.mkdir()
manifest_path = src / "manifest.json"
manifest_path.write_text(
json.dumps(
{
"manifest_version": 1,
"source": {"type": "file", "id": "catalogue-1", "root": "data"},
"bundle_summary": "A test bundle.",
"extractions": [
{
"id": "orders",
"title": "Orders",
"query": "orders.csv",
"okf_type": "dataset",
"max_rows": 100,
}
],
}
),
encoding="utf-8",
)
(src / "data").mkdir()
(src / "data" / "orders.csv").write_text("a,b\n1,x\n", encoding="utf-8", newline="")
bundle = tmp_path / "bundle"
materialize_bundle(manifest_path, bundle, INGESTED_AT)
concept = bundle / "ingest-orders.md"
v0_2_bytes = concept.read_bytes().replace(
b"generated: true", f"generated: {V0_2_GENERATED}".encode()
)
concept.write_bytes(v0_2_bytes)
with pytest.raises(MaterializationError) as excinfo:
materialize_bundle(manifest_path, bundle, INGESTED_AT)
assert excinfo.value.code == "collision_unstamped"
assert concept.read_bytes() == v0_2_bytes
# --- step 2: Door C against the §14 consumer tolerance rules --------------
TOLERANCE_CASES = [
pytest.param(
"---\ntype: Concept\n---\n\nOnly a type.\n",
id="missing-optional-frontmatter",
),
pytest.param(
"---\ntype: Attested Computation\ntitle: Gross margin\n---\n\nA v0.2 type.\n",
id="unknown-type-value",
),
pytest.param(
"---\ntype: dataset\nstale_after: 2027-01-01\nnot_a_key_we_know: x\n---\n\nBody.\n",
id="unknown-additional-keys",
),
pytest.param(
"---\ntype: dataset\n---\n\nSee [the other one](does-not-exist.md).\n",
id="broken-cross-link",
),
pytest.param(
"---\ntype: dataset\nverified: { by: human:jsmith@acme, at: 2026-07-01T09:00:00Z }\n"
"---\n\nBody.\n",
id="bare-verified-mapping",
),
]
@pytest.mark.parametrize("document", TOLERANCE_CASES)
def test_door_c_merges_what_a_consumer_must_not_reject(tmp_path: Path, document: str) -> None:
"""D3/§14. Each case is one thing a conformant consumer MUST NOT reject.
Door C's tolerance is structural rather than lenient: it writes the guard's
bytes verbatim and never parses the sender's frontmatter, so there is no
place for a shape judgement to be made. That is what these cases pin — a
future reader (D1b) that starts judging frontmatter at this door would
break them, which is exactly when we want to hear about it.
The bare-`verified` case is the §5.2 MUST in its Door C form: the mapping is
persisted unmodified. Coercing it to a one-element list is the READER's
obligation and lands with D1b; nothing here reads it.
"""
place(tmp_path / "source", "notes/tolerated.md", document)
result, bundle = run(tmp_path, StubImportGate(), ingested_at=INGESTED_AT)
assert result.failed == ()
assert [entry.concept_path for entry in result.merged] == ["notes/tolerated.md"]
assert (bundle / "import-notes-tolerated.md").read_bytes() == document.encode("utf-8")
def test_door_c_does_not_require_the_source_bundle_to_carry_an_index(tmp_path: Path) -> None:
"""The fifth MUST NOT: a missing `index.md` is not grounds for rejection.
Door C generates the target index itself, so the sender's bundle need not
carry one — asserted explicitly here because every other test in the suite
omits the source index incidentally rather than as a stated rule.
"""
document = "---\ntype: dataset\n---\n\nNo index anywhere in the source.\n"
place(tmp_path / "source", "tables/users.md", document)
assert not (tmp_path / "source" / "index.md").exists()
result, bundle = run(tmp_path, StubImportGate(), ingested_at=INGESTED_AT)
assert result.failed == ()
assert [entry.concept_path for entry in result.merged] == ["tables/users.md"]
assert (bundle / "index.md").read_text(encoding="utf-8") == (
"- [tables/users](import-tables-users.md)\n"
)
# --- step 3: no profile hard-codes an upstream version --------------------
# A version LITERAL — `0.1`, `0.2`, `2026.7.3`. Compiled patterns are walked
# past deliberately: a regex is a shape, and a form gate for version values is
# the opposite of hard-coding one.
_VERSION_LITERAL_RE = re.compile(r"^\d+(\.\d+)+$")
def _strings_in(value: Any) -> Iterator[str]:
"""Every string reachable from a profile, through dataclasses and
collections alike. Recursive so that a version added to a nested policy —
the only place it could plausibly be added — is still found."""
if isinstance(value, str):
yield value
elif dataclasses.is_dataclass(value) and not isinstance(value, type):
for field in dataclasses.fields(value):
yield from _strings_in(getattr(value, field.name))
elif isinstance(value, (tuple, list, set, frozenset)):
for item in value:
yield from _strings_in(item)
@pytest.mark.parametrize("profile", [DEFAULT, STRICT_V1], ids=["DEFAULT", "STRICT_V1"])
def test_no_profile_hard_codes_an_upstream_version(profile: BundleProfile) -> None:
"""V4/V-A5. A profile names `okf_version` as a key and never carries its
value. The value tracks the upstream Google version and belongs to catalog
(decision E1), so a constant here would be this repo claiming a decision it
does not own — and the one that would have to be chased on every upstream
release.
"""
hard_coded = sorted({s for s in _strings_in(profile) if _VERSION_LITERAL_RE.match(s)})
assert hard_coded == []
def test_the_key_name_is_what_a_profile_pins(tmp_path: Path) -> None:
"""The other half of V4: naming the key is allowed and is what STRICT_V1
does. Without this, the test above would also pass on a profile that had
stopped requiring the declaration at all."""
assert "okf_version" in STRICT_V1.index.root_frontmatter
@pytest.mark.parametrize(
"schema_kwargs",
[
pytest.param({"order": ("timestamp", "generated")}, id="both-emitted"),
pytest.param(
{"order": ("timestamp",), "allowed": frozenset({"timestamp", "generated"})},
id="one-emitted-one-admitted",
),
pytest.param(
{"order": (), "required": frozenset({"timestamp", "generated"})},
id="both-required",
),
pytest.param(
{"order": ("timestamp", "title"), "nullable": frozenset({"generated"})},
id="one-emitted-one-nullable",
),
],
)
def test_a_schema_naming_both_timestamp_and_generated_cannot_be_built(
schema_kwargs: dict[str, Any],
) -> None:
"""V-A7. OKF §13.1 grants the `timestamp` fallback only while `generated` is
ABSENT, so a schema able to name both can describe a document with neither a
valid `generated.at` nor an eligible fallback. Refused at construction, the
same shape as the reserved `verdict` layer: a profile that could reach the
combination cannot be built, let alone handed to a door.
Every judging field names keys, `nullable` included. An open namespace
already admits `generated`, so a schema that also gives it a null rule has
named it exactly as surely as one that emits it — the last case is the
construction the proving consumer found the day the gate was written.
"""
with pytest.raises(ValueError, match="timestamp"):
FrontmatterSchema(**schema_kwargs)
def test_the_two_shipped_profiles_sit_on_opposite_sides_of_the_fallback() -> None:
"""Why the gate above costs nothing: neither shipped profile is near the
combination, and they are not near it in opposite directions. `DEFAULT`
emits `generated` and no `timestamp`; `STRICT_V1` emits `timestamp` and no
`generated`, which is precisely what puts the wiki on the §13.1 legacy path
rather than in a defect.
"""
assert "generated" in DEFAULT.frontmatter.order
assert "timestamp" not in DEFAULT.frontmatter.order
assert "timestamp" in STRICT_V1.frontmatter.order
assert "generated" not in STRICT_V1.frontmatter.order
assert STRICT_V1.frontmatter.allowed is not None
assert "generated" not in STRICT_V1.frontmatter.allowed
@pytest.mark.parametrize("declared", ["0.1", "0.2", "1.0", "2026.7.3"])
def test_the_root_frontmatter_policy_expresses_any_version_value(declared: str) -> None:
"""V4. The policy judges presence and ORDER, never the value — so the day
upstream ships v0.3, no index policy in this library has to change.
"""
text = (
f"---\nokf_version: {declared}\nbundle_profile: strict-v1\n"
"okf_spec_commit: 0123456789abcdef\n---\n\n# Bundle\n\n* [A](a.md) - a\n"
)
violations = STRICT_V1.index.violations(text, is_root=True, expected_targets={"a.md"})
assert violations == ()

339
tests/test_okf_v0_2_profile.py
View file

@ -1,339 +0,0 @@
"""D2 — the `OKF_V0_2` profile, profile-aware ownership, `sources` derivation.
Three things, in the order the risk runs:
1. **The profile constant.** Names the §5 families, requires only `type` (§4:
"the only always-required key"), and closes nothing — §14 forbids a consumer
to reject on unknown types or unknown keys, so an allowlist here would put
the profile in violation of the spec it is named after.
2. **Ownership.** `_is_ingest_owned` is the pre-mutation collision gate, so it
is the one piece of this work whose defects are expensive and quiet. The v0.2
stamp is a flow mapping carrying `ingested_at`, so equality is impossible by
construction and the predicate is a prefix test. The operator decided the
direction: `OKF_V0_2` accepts BOTH stamps, so a bundle written under `DEFAULT`
re-runs IN PLACE; `DEFAULT` does not accept the v0.2 form, so the recognition
is one-way and the fail-safe from V-A3 is preserved.
3. **`sources`.** Derived from the manifest's source as an inline flow sequence
(requirement 1's measured form), with the locator taken verbatim per source
type: `root`, `connection_ref`, `base_url`. `credential_ref` is never a
locator and never reaches frontmatter.
`generated.by` is `process:llm-ingestion-okf` — option (d) from the plan's V1,
chosen by the operator 2026-07-27 over the `<producer>/<version>` form A-E3 was
written against. The value carries no version, which is what keeps a byte-
compared fixture stable across releases. **A-E3 is now stale in the pilot
specification and owes po-claude a correction.**
"""
from __future__ import annotations
import re
from pathlib import Path
import pytest
from llm_ingestion_okf.errors import MaterializationError
from llm_ingestion_okf.manifest import (
Extraction,
FileSource,
HttpSource,
Manifest,
Source,
SqlSource,
)
from llm_ingestion_okf.materialize import _is_ingest_owned, _render_concept_file
from llm_ingestion_okf.profiles import DEFAULT, OKF_LATEST, OKF_V0_2, STRICT_V1, BundleProfile
INGESTED_AT = "2026-07-25T12:00:00Z"
STAMP = "manifest@0123456789abcdef"
V0_2_STAMP = f"{{ by: process:llm-ingestion-okf, at: {INGESTED_AT} }}"
def _manifest(source: Source) -> Manifest:
return Manifest(
manifest_version=1,
source=source,
bundle_summary="A test bundle.",
extractions=(
Extraction(
id="orders", title="Orders", query="orders.csv", okf_type="dataset", max_rows=10
),
),
)
def _concept(source: Source, *, profile: BundleProfile = OKF_V0_2) -> str:
manifest = _manifest(source)
return _render_concept_file(
manifest,
manifest.extractions[0],
"Body.\n",
ingested_at=INGESTED_AT,
stamp=STAMP,
profile=profile,
)
def _stamped_concept(generated: str) -> bytes:
return (
"---\ntype: dataset\ntitle: Orders\n"
f"ingest_manifest: {STAMP}\n"
f"generated: {generated}\n---\n\nBody.\n"
).encode()
# --- the profile constant -------------------------------------------------
def test_the_v0_2_profile_names_the_five_families_it_must_express() -> None:
"""C-E3's list, as emission order rather than as prose. Naming a key is what
lets a caller emit it in canonical order; judging a structured value is D1's
reader and deliberately absent here.
"""
order = OKF_V0_2.frontmatter.order
for key in ("generated", "sources", "verified", "status", "stale_after"):
assert key in order
assert "timestamp" not in order # §13.1: the fallback is for v0.1 documents
assert order.index("generated") < order.index("sources")
def test_the_v0_2_profile_requires_only_type() -> None:
"""§4/§11: `type` is the only always-required key, and a concept carrying
just `type` is fully conformant."""
assert OKF_V0_2.frontmatter.required == frozenset({"type"})
def test_the_v0_2_profile_closes_neither_the_type_set_nor_the_key_namespace() -> None:
"""§14's MUST NOTs, expressed as configuration: a consumer must not reject
on an unknown `type` value or on unknown additional keys. A profile that
closed either would be non-conformant against the version it is named for.
"""
assert OKF_V0_2.types.allowed is None
assert OKF_V0_2.frontmatter.allowed is None
assert OKF_V0_2.frontmatter.key_pattern is None
def test_the_v0_2_profile_keeps_the_source_query_collapse_and_the_default_paths() -> None:
"""The parts that are commons' §5 rather than upstream's: the doors write
the same filenames whatever the OKF version, and a multi-line SELECT still
renders on one line."""
assert OKF_V0_2.frontmatter.collapsed_keys == frozenset({"source_query"})
assert OKF_V0_2.paths == DEFAULT.paths
def test_okf_latest_still_means_v0_1_until_ga() -> None:
"""`OKF_LATEST` is "the latest version supported as STABLE", not the latest
present in the tree. Flipping it is the GA event — a single auditable
action rather than a side effect of this merge — so during the pilot a
consumer bound to the alias keeps v0.1 semantics.
"""
assert OKF_LATEST is DEFAULT
# --- ownership ------------------------------------------------------------
def test_the_default_profile_still_stamps_the_v0_1_literal() -> None:
assert DEFAULT.ownership.stamp(INGESTED_AT) == "true"
def test_the_v0_2_stamp_is_a_flow_mapping_whose_at_is_the_argument_exactly() -> None:
"""A-E3, corrected: the actor is the §7 `process:<id>` form and carries no
version. `at` binds to the already-validated `ingested_at` argument, so no
wall-clock is introduced and two runs with the same argument agree.
"""
assert OKF_V0_2.ownership.stamp(INGESTED_AT) == V0_2_STAMP
assert "llm-ingestion-okf/" not in OKF_V0_2.ownership.stamp(INGESTED_AT)
def test_the_v0_2_predicate_accepts_the_stamp_its_own_profile_writes(tmp_path: Path) -> None:
"""A-E5's second run: a v0.2 bundle re-runs into the directory it wrote."""
path = tmp_path / "ingest-orders.md"
path.write_bytes(_stamped_concept(V0_2_STAMP))
assert _is_ingest_owned(path, "manifest", profile=OKF_V0_2) is True
def test_the_v0_2_predicate_also_accepts_the_v0_1_stamp(tmp_path: Path) -> None:
"""The operator's decision (2026-07-27): a `DEFAULT`-written bundle is
re-runnable IN PLACE under `OKF_V0_2`. Without this the profile switch would
fire `collision_unstamped` on the library's own previous output, and the
black-box promise — an upstream release costs a consumer a re-run, nothing
more — would not hold across the profile change.
"""
path = tmp_path / "ingest-orders.md"
path.write_bytes(_stamped_concept("true"))
assert _is_ingest_owned(path, "manifest", profile=OKF_V0_2) is True
def test_the_default_predicate_still_refuses_the_v0_2_stamp(tmp_path: Path) -> None:
"""Recognition is one-way, which is what preserves V-A3. `DEFAULT` meeting a
v0.2 file refuses the run rather than replacing the file."""
path = tmp_path / "ingest-orders.md"
path.write_bytes(_stamped_concept(V0_2_STAMP))
assert _is_ingest_owned(path, "manifest") is False
assert _is_ingest_owned(path, "manifest", profile=DEFAULT) is False
def test_a_foreign_v0_2_actor_is_never_ingest_owned(tmp_path: Path) -> None:
"""U3 as a test: `generated` is attribution, and upstream writes it for
HAND-AUTHORED files. Presence proves nothing — the actor has to be ours, or
the collision gate would claim a human's file that merely carries the key.
"""
path = tmp_path / "ingest-orders.md"
path.write_bytes(_stamped_concept(f"{{ by: human:jsmith@acme, at: {INGESTED_AT} }}"))
assert _is_ingest_owned(path, "manifest", profile=OKF_V0_2) is False
def test_the_v0_2_stamp_still_has_to_name_this_manifest(tmp_path: Path) -> None:
"""§10.2 per-manifest ownership is unchanged by the profile: a sibling
manifest sharing the bundle keeps its own files."""
path = tmp_path / "ingest-orders.md"
path.write_bytes(_stamped_concept(V0_2_STAMP))
assert _is_ingest_owned(path, "other-manifest", profile=OKF_V0_2) is False
# --- `sources` derivation -------------------------------------------------
SOURCE_CASES = [
pytest.param(FileSource(id="golden-catalogue", root="fixture"), "fixture", id="file"),
pytest.param(
SqlSource(id="golden-db", connection_ref="OKF_GOLDEN_SQL_DB"),
"OKF_GOLDEN_SQL_DB",
id="sql",
),
pytest.param(
HttpSource(id="golden-api", base_url="https://golden.example.test"),
"https://golden.example.test",
id="http",
),
]
@pytest.mark.parametrize("source,resource", SOURCE_CASES)
def test_sources_is_an_inline_flow_sequence_carrying_the_locator(
source: Source, resource: str
) -> None:
"""A-E4. One entry, `id` from the manifest source and `resource` its locator
verbatim — the filesystem root, the env-var NAME holding the DSN, or the base
URL. Upstream's `resource` is a bundle-internal path; ours is not, and
inventing the `author`/`last_modified` upstream also carries would be writing
fields with no reader.
"""
line = f"sources: [{{ id: {source.id}, resource: {resource} }}]"
assert line in _concept(source).splitlines()
def test_sources_follows_generated_in_the_emitted_block() -> None:
"""The order the schema declares is the order that reaches disk."""
lines = _concept(FileSource(id="golden-catalogue", root="fixture")).splitlines()
assert [line.split(":")[0] for line in lines[1:9]] == [
"type",
"title",
"source_system",
"source_query",
"ingested_at",
"ingest_manifest",
"generated",
"sources",
]
def test_the_default_profile_emits_neither_sources_nor_a_v0_2_generated() -> None:
"""V-A6 at the unit level: the seven-key v0.1 block, byte for byte. The
golden suite proves the same thing end to end; this one localizes a failure
to the renderer rather than to a fixture diff.
"""
rendered = _concept(FileSource(id="golden-catalogue", root="fixture"), profile=DEFAULT)
assert rendered == (
"---\n"
"type: dataset\n"
"title: Orders\n"
"source_system: golden-catalogue\n"
"source_query: orders.csv\n"
f"ingested_at: {INGESTED_AT}\n"
f"ingest_manifest: {STAMP}\n"
"generated: true\n"
"---\n"
"\n"
"Body.\n"
)
def test_the_http_credential_reference_never_reaches_the_frontmatter() -> None:
"""A credential reference is not a locator. The convention forbids the
credential itself in frontmatter; the env-var NAME holding it has no reader
in a bundle either, so it is not written.
"""
source = HttpSource(
id="golden-api",
base_url="https://golden.example.test",
credential_ref="OKF_GOLDEN_API_TOKEN",
)
assert "OKF_GOLDEN_API_TOKEN" not in _concept(source)
@pytest.mark.parametrize(
"root",
["data, backup", "data{1}", "data[1]", "key: value"],
ids=["comma", "braces", "brackets", "colon-space"],
)
def test_a_locator_that_would_restructure_the_flow_mapping_is_refused(root: str) -> None:
"""Fail-fast rather than emit a silently WRONG provenance record.
These characters terminate or restructure a YAML flow mapping, and the
failure is quiet rather than loud. Measured with PyYAML 2026-07-27 rather
than argued: `[{ id: a, resource: data, backup }]` raises nothing and yields
`[{'id': 'a', 'resource': 'data', 'backup': None}]` — a clean parse into a
record no one wrote. Refusing is the same posture as the filename-length
gate: validation, not repair.
The same run confirms what IS emitted parses as intended — `generated` to a
two-key mapping, `sources` to a one-element list of one — including a base
URL, whose colons stay inside a plain scalar in flow context.
"""
with pytest.raises(MaterializationError) as excinfo:
_concept(FileSource(id="golden-catalogue", root=root))
assert excinfo.value.code == "source_reference_unquotable"
def test_the_default_profile_is_unaffected_by_an_unquotable_locator() -> None:
"""The refusal is a property of the v0.2 emission, not a new gate on v0.1:
`DEFAULT` never writes the locator, so it has nothing to refuse. A shared
refusal would have broken additivity — a manifest that ran yesterday must
still run."""
rendered = _concept(FileSource(id="golden-catalogue", root="data, backup"), profile=DEFAULT)
assert "data, backup" not in rendered
# --- V-A5 extended to the new profile -------------------------------------
_VERSION_LITERAL_RE = re.compile(r"^\d+(\.\d+)+$")
@pytest.mark.parametrize(
"profile", [DEFAULT, STRICT_V1, OKF_V0_2], ids=["DEFAULT", "STRICT_V1", "OKF_V0_2"]
)
def test_no_profile_hard_codes_an_upstream_version(profile: BundleProfile) -> None:
"""V-A5, restated over the profile the name of which is the one place a
version literal would look natural. `OKF_V0_2` is a NAME; the value belongs
to catalog (E1) and is declared at D5, in the fixture, once.
Kept alongside the characterization suite's copy rather than replacing it:
that one pins the two shipped profiles at the point they were measured.
"""
from test_okf_v0_2_characterization import _strings_in
assert sorted({s for s in _strings_in(profile) if _VERSION_LITERAL_RE.match(s)}) == []

15
tests/test_packaging.py
View file

@ -32,18 +32,3 @@ def test_the_only_runtime_dependency_is_the_security_boundary() -> None:
tomllib = pytest.importorskip("tomllib") # stdlib from 3.11; the pin holds on 3.10 too
pyproject = tomllib.loads((PROJECT_ROOT / "pyproject.toml").read_text(encoding="utf-8"))
assert pyproject["project"]["dependencies"] == ["llm-ingestion-guard>=0.2,<0.3"]
def test_the_declared_version_agrees_with_the_packaged_one() -> None:
"""The two places a version is written must not drift apart.
The install channel is a direct git reference, so a consumer pins a TAG
while pip records `project.version`. Nothing in the run path reads
`__version__` — which is exactly why a stale one survives a green suite,
and why a consumer installing at a pre-release tag can end up with a
package that reports the previous release. This is the only machine check
on that; the tag name itself remains a human step.
"""
tomllib = pytest.importorskip("tomllib")
pyproject = tomllib.loads((PROJECT_ROOT / "pyproject.toml").read_text(encoding="utf-8"))
assert llm_ingestion_okf.__version__ == pyproject["project"]["version"]

224
tests/test_profile.py
View file

@ -1,224 +0,0 @@
"""The bundle contract as configuration (Phase 3, step 1).
What Phases 1 and 2 hard-coded about a valid bundle — the index name and its
managed-link shape, the three filename namespaces, the reserved layer, and the
frontmatter key order — becomes a profile object here. `DEFAULT` must express
exactly the existing contract: the Phase 1/2 suite and the golden fixtures are
the byte-level proof (assumption C1), and these tests pin the profile's own
behavior.
"""
from __future__ import annotations
import dataclasses
import pytest
from llm_ingestion_okf.profiles import (
DEFAULT,
BundleProfile,
FrontmatterSchema,
TypePolicy,
)
def test_the_profile_and_every_policy_on_it_are_frozen() -> None:
"""A profile is a value, not a mutable registry.
A consumer holds DEFAULT while a run is in flight; a profile that could be
mutated mid-run would make the contract a moving target and the golden
determinism claim unprovable.
"""
for value in (DEFAULT, DEFAULT.types, DEFAULT.frontmatter, DEFAULT.paths, DEFAULT.index):
with pytest.raises(dataclasses.FrozenInstanceError):
value.name = "mutated" # type: ignore[misc]
def test_default_profile_pins_the_phase_1_2_names() -> None:
"""The names the two doors already write, now stated in one place."""
assert DEFAULT.index.name == "index.md"
assert DEFAULT.paths.concept_suffix == ".md"
assert DEFAULT.paths.ingest_prefix == "ingest-"
assert DEFAULT.paths.inbox_prefix == "inbox-"
assert DEFAULT.paths.import_prefix == "import-"
# --- the reserved layer (assumption C3) -----------------------------------
@pytest.mark.parametrize("reserved", ["verdict", "Verdict", "VERDICT"])
def test_no_type_policy_can_admit_the_reserved_verdict_layer(reserved: str) -> None:
"""C3: the verdict reservation is a spec invariant, not profile config.
Refused at CONSTRUCTION, not at use: a profile that admits `verdict` must
not exist at all, or the promotion gate stops being the only path into that
layer the moment someone passes the profile to a door.
"""
with pytest.raises(ValueError, match="verdict"):
TypePolicy(allowed=frozenset({"Concept", reserved}))
def test_a_closed_type_policy_without_the_reserved_layer_constructs() -> None:
policy = TypePolicy(allowed=frozenset({"Concept", "Guide", "Reference", "Release"}))
assert policy.rejection("Concept") is None
def test_default_type_policy_is_open_apart_from_the_reserved_layer() -> None:
"""Phase 1/2 accept any okf_type but `verdict`; DEFAULT must not narrow that."""
assert DEFAULT.types.allowed is None
for admitted in ("Concept", "note", "whatever-the-operator-wants"):
assert DEFAULT.types.rejection(admitted) is None
for reserved in ("verdict", "Verdict", "VERDICT"):
assert DEFAULT.types.rejection(reserved) is not None
def test_the_reserved_rejection_reproduces_both_doors_message_and_code() -> None:
"""The doors keep their own typed errors; the policy supplies the wording.
Door A raises ManifestError and Door B MaterializationError for the same
refusal, so the policy cannot raise — it hands back the reason and the
stable code, and each door frames it. These are the exact strings the
Phase 1/2 tests already assert on.
"""
rejection = DEFAULT.types.rejection("verdict")
assert rejection is not None
assert rejection.code == "okf_type_reserved"
assert f"okf_type {rejection.reason}" == "okf_type must not be 'verdict' (reserved layer)"
def test_a_closed_type_policy_rejects_an_off_enum_type_distinctly() -> None:
"""An off-enum refusal is not the reserved-layer refusal wearing its code."""
policy = TypePolicy(allowed=frozenset({"Concept", "Release"}))
rejection = policy.rejection("Guide")
assert rejection is not None
assert rejection.code == "okf_type_not_allowed"
assert "Concept, Release" in rejection.reason
# --- index policy ---------------------------------------------------------
def test_index_link_template_and_pattern_describe_the_same_shape() -> None:
"""Two fields that must agree, proven to agree rather than trusted to.
Rendering and parsing managed index links are separate operations (§6
appends, and maintenance rewrites), so the profile carries both a template
and a pattern. A round-trip is what keeps them from drifting apart.
"""
for label, target in [("Title", "ingest-x.md"), ("", "inbox-y.md")]:
line = DEFAULT.index.render_link(label, target)
match = DEFAULT.index.link_pattern.match(line)
assert match is not None, f"pattern does not match its own template output: {line!r}"
assert match.group("label") == label
assert match.group("target") == target
def test_default_index_link_is_the_phase_1_shape() -> None:
assert DEFAULT.index.render_link("Sales", "ingest-sales.md") == "- [Sales](ingest-sales.md)"
def test_index_link_pattern_is_anchored_to_the_whole_line() -> None:
"""§6 removal keys on the exact shape, never a bare substring — a curated
line that merely mentions a link must survive verbatim."""
assert DEFAULT.index.link_pattern.match("see - [Sales](ingest-sales.md) below") is None
# --- frontmatter emission -------------------------------------------------
def test_emission_is_the_ordered_prefix_then_the_sorted_tail() -> None:
"""Commons decision D1, carried over intact.
Keys named in the profile's order come first in that order; anything else
follows sorted. That is what makes a regeneration over the same data
byte-identical, which is what the proving consumer's hash registry needs.
"""
schema = FrontmatterSchema(order=("type", "title"))
emitted = schema.emit({"zeta": "3", "title": "T", "alpha": "1", "type": "Concept"})
assert emitted == "type: Concept\ntitle: T\nalpha: 1\nzeta: 3"
def test_emission_skips_ordered_keys_that_are_absent() -> None:
schema = FrontmatterSchema(order=("type", "title", "generated"))
assert schema.emit({"generated": "true", "type": "Concept"}) == "type: Concept\ngenerated: true"
def test_default_emission_reproduces_door_a_key_order() -> None:
"""Byte-identity with `_render_concept_file`'s §5 frontmatter."""
emitted = DEFAULT.frontmatter.emit(
{
"type": "Concept",
"title": "Sales",
"source_system": "crm",
"source_query": "sales.csv",
"ingested_at": "2026-07-03T12:00:00Z",
"ingest_manifest": "m@0123456789abcdef",
"generated": "true",
}
)
assert emitted.splitlines() == [
"type: Concept",
"title: Sales",
"source_system: crm",
"source_query: sales.csv",
"ingested_at: 2026-07-03T12:00:00Z",
"ingest_manifest: m@0123456789abcdef",
"generated: true",
]
def test_default_emission_reproduces_door_b_key_order() -> None:
"""Byte-identity with `render_inbox_concept`'s provenance layer.
Door B's six keys are a different subset of the same schema, and the
canonical order must reproduce BOTH doors — a single order that reordered
either one would change bytes already frozen in the fixtures.
"""
emitted = DEFAULT.frontmatter.emit(
{
"type": "Concept",
"title": "Notes",
"source_file": "notes.md",
"source_sha256": "abc",
"ingested_at": "2026-07-03T12:00:00Z",
"generated": "true",
}
)
assert emitted.splitlines() == [
"type: Concept",
"title: Notes",
"source_file: notes.md",
"source_sha256: abc",
"ingested_at: 2026-07-03T12:00:00Z",
"generated: true",
]
def test_source_query_is_the_only_collapsed_key() -> None:
"""§5 collapses whitespace runs for `source_query` alone (a multi-line
SELECT must render on one line); every other value is validated single-line
at load and emitted verbatim, so an operator's internal double space
survives."""
assert DEFAULT.frontmatter.collapsed_keys == frozenset({"source_query"})
emitted = DEFAULT.frontmatter.emit(
{"title": "two spaces", "source_query": "SELECT a\n FROM t"}
)
assert emitted == "title: two spaces\nsource_query: SELECT a FROM t"
def test_a_profile_is_assembled_from_its_policies() -> None:
"""The profile is its policies and nothing else — no behavior branches
outside what the object expresses.
`ownership` joined the original four at D2, and it is the same rule rather
than an exception to it: the ingest stamp differs per profile, so the
alternative was a version branch inside the collision gate. A policy on the
object is what keeps the emitter and the predicate changeable only together.
"""
assert {field.name for field in dataclasses.fields(BundleProfile)} == {
"types",
"frontmatter",
"paths",
"index",
"ownership",
}

227
tests/test_profile_threading.py
View file

@ -1,227 +0,0 @@
"""Requirement 6 — `profile` threaded through `materialize_bundle`'s disk phase.
`OKF_V0_2` landed in D2, but nothing outside the library could ask for it: the
content phase already accepted `profile`, while `materialize_bundle` neither
took one nor passed one down. The parameter is keyword-only behind the `*` the
signature already carried, so every three-positional call site stays
source-compatible — which is what the consumer who asked for it needs.
Two of these tests would be untestable by construction if written only against
the shipped profiles: `OKF_V0_2.paths is DEFAULT.paths` and
`.index is DEFAULT.index`, so no assertion can tell `profile.index.name` apart
from `DEFAULT.index.name` when they are the same object. That identity is what
makes the change byte-neutral, and it is also what would let six of the nine
threading sites stay hard-coded with every shipped-profile test still green.
`_SYNTHETIC` exists to close exactly that gap: it renames the index and the
concept files, so a site still reading `DEFAULT` writes to the wrong path and
fails. It is a test instrument, never a supported profile.
What this does NOT claim: the materializer honours `profile.types` (validation
still runs against `DEFAULT` in `manifest.py`; both policies compare equal
today, so nothing is hidden — but the parameter does not reach it), nor
`IndexPolicy`'s judging fields (`per_directory`, `entries_match_directory`,
`root_frontmatter`). `STRICT_V1` sets all three, so passing it here produces a
bundle that does not meet its own profile. Supported here: `DEFAULT` and
`OKF_V0_2`.
"""
from __future__ import annotations
import json
from dataclasses import replace
from pathlib import Path
from typing import Any
import pytest
from llm_ingestion_okf.materialize import materialize_bundle
from llm_ingestion_okf.profiles import DEFAULT, OKF_V0_2, BundleProfile
INGESTED_AT = "2026-07-16T12:00:00Z"
# Renames both halves the disk phase reads: a threading site left on `DEFAULT`
# writes `ingest-orders.md` / `index.md` instead, and the assertions below say
# so by name rather than by a byte diff that cannot point at the cause.
_SYNTHETIC = replace(
OKF_V0_2,
paths=replace(DEFAULT.paths, ingest_prefix="x-", concept_suffix=".mdx"),
index=replace(DEFAULT.index, name="contents.md"),
)
def _manifest_data() -> dict[str, Any]:
return {
"manifest_version": 1,
"source": {"type": "file", "id": "catalogue-1", "root": "data"},
"bundle_summary": "A test bundle.",
"extractions": [
{
"id": "orders",
"title": "Orders",
"query": "orders.csv",
"okf_type": "dataset",
"max_rows": 100,
}
],
}
@pytest.fixture
def file_setup(tmp_path: Path) -> tuple[Path, Path]:
"""A manifest + CSV catalogue in tmp/src, and an empty bundle target path."""
src = tmp_path / "src"
src.mkdir(parents=True)
manifest_path = src / "manifest.json"
manifest_path.write_text(json.dumps(_manifest_data()), encoding="utf-8")
(src / "data").mkdir()
(src / "data" / "orders.csv").write_text("a,b\n1,x\n2,y\n", encoding="utf-8", newline="")
return manifest_path, tmp_path / "bundle"
# --- the profile reaches the content phase ---
def test_v0_2_profile_reaches_the_emitted_frontmatter(file_setup: tuple[Path, Path]) -> None:
"""`_render_concept_file` took `profile` since D2; the call site did not pass
one, so `OKF_V0_2` was unreachable through the only public door.
"""
manifest_path, bundle = file_setup
materialize_bundle(manifest_path, bundle, INGESTED_AT, profile=OKF_V0_2)
text = (bundle / "ingest-orders.md").read_text(encoding="utf-8")
assert f"generated: {{ by: process:llm-ingestion-okf, at: {INGESTED_AT} }}" in text
assert "sources: [{ id: catalogue-1, resource: data }]" in text
assert "generated: true" not in text
def test_default_run_is_unchanged_by_the_new_parameter(file_setup: tuple[Path, Path]) -> None:
"""Byte-neutrality at the signature: passing `DEFAULT` explicitly and passing
nothing must produce the same bundle, or the default is not the default.
"""
manifest_path, bundle = file_setup
materialize_bundle(manifest_path, bundle, INGESTED_AT)
implicit = (bundle / "ingest-orders.md").read_bytes()
index_implicit = (bundle / "index.md").read_bytes()
explicit_dir = bundle.parent / "bundle-explicit"
materialize_bundle(manifest_path, explicit_dir, INGESTED_AT, profile=DEFAULT)
assert (explicit_dir / "ingest-orders.md").read_bytes() == implicit
assert (explicit_dir / "index.md").read_bytes() == index_implicit
assert b"generated: true" in implicit
def test_profile_is_keyword_only(file_setup: tuple[Path, Path]) -> None:
"""The parameter sits behind the `*` the signature already had. A positional
fourth argument must not be accepted, or a later reordering would silently
rebind existing call sites.
"""
manifest_path, bundle = file_setup
with pytest.raises(TypeError):
materialize_bundle(manifest_path, bundle, INGESTED_AT, DEFAULT) # type: ignore[misc]
# --- A-E5: two runs into the SAME directory ---
def test_second_v0_2_run_into_the_same_directory_succeeds(
file_setup: tuple[Path, Path],
) -> None:
"""A-E5, end to end. The emitter and the ownership predicate are coupled
through the stamp value; `OKF_V0_2` changes that value. If the predicate is
not given the same profile the emitter used, the library stops recognising
its own output and the §3 collision gate fires `collision_unstamped` on the
files the previous run wrote. One run compared byte for byte cannot see it.
"""
manifest_path, bundle = file_setup
materialize_bundle(manifest_path, bundle, INGESTED_AT, profile=OKF_V0_2)
first = (bundle / "ingest-orders.md").read_bytes()
first_index = (bundle / "index.md").read_bytes()
materialize_bundle(manifest_path, bundle, INGESTED_AT, profile=OKF_V0_2)
assert (bundle / "ingest-orders.md").read_bytes() == first
assert (bundle / "index.md").read_bytes() == first_index
# --- the paths/index half of the profile reaches disk ---
def test_profile_paths_and_index_name_reach_disk(file_setup: tuple[Path, Path]) -> None:
"""Six of the nine threading sites read `paths`/`index`, which are the same
objects on every shipped profile. `_SYNTHETIC` renames both so the sites are
provable rather than merely regression-covered.
"""
manifest_path, bundle = file_setup
result = materialize_bundle(manifest_path, bundle, INGESTED_AT, profile=_SYNTHETIC)
assert (bundle / "x-orders.mdx").is_file()
assert not (bundle / "ingest-orders.md").exists()
assert (bundle / "contents.md").is_file()
assert not (bundle / "index.md").exists()
assert [path.name for path in result.written] == ["x-orders.mdx"]
assert "- [Orders](x-orders.mdx)" in (bundle / "contents.md").read_text(encoding="utf-8")
def test_second_synthetic_run_replaces_rather_than_colliding(
file_setup: tuple[Path, Path],
) -> None:
"""The §3 ownership scan globs `*{concept_suffix}` and excludes `index.name`.
Left on `DEFAULT`, the glob finds nothing under a renamed suffix, `owned` is
empty, and the gate refuses to overwrite the file this same code just wrote.
The index must also not gain a second link for the same target.
"""
manifest_path, bundle = file_setup
materialize_bundle(manifest_path, bundle, INGESTED_AT, profile=_SYNTHETIC)
first = (bundle / "x-orders.mdx").read_bytes()
materialize_bundle(manifest_path, bundle, INGESTED_AT, profile=_SYNTHETIC)
assert (bundle / "x-orders.mdx").read_bytes() == first
index_text = (bundle / "contents.md").read_text(encoding="utf-8")
assert index_text.count("- [Orders](x-orders.mdx)") == 1
def test_removed_extraction_is_unlinked_under_a_renamed_index(
file_setup: tuple[Path, Path], tmp_path: Path
) -> None:
"""`_update_index_lines` runs only on an EXISTING index, so it is reached
solely by a second run — and only its `link_pattern`/`render_link` are
profile-owned. Dropping an extraction is what forces the removal branch.
"""
manifest_path, bundle = file_setup
materialize_bundle(manifest_path, bundle, INGESTED_AT, profile=_SYNTHETIC)
data = _manifest_data()
data["extractions"][0]["id"] = "invoices"
data["extractions"][0]["title"] = "Invoices"
manifest_path.write_text(json.dumps(data), encoding="utf-8")
(tmp_path / "src" / "data" / "invoices.csv").write_text(
"a,b\n1,x\n", encoding="utf-8", newline=""
)
materialize_bundle(manifest_path, bundle, INGESTED_AT, profile=_SYNTHETIC)
index_text = (bundle / "contents.md").read_text(encoding="utf-8")
assert "- [Invoices](x-invoices.mdx)" in index_text
assert "x-orders.mdx" not in index_text
assert not (bundle / "x-orders.mdx").exists()
def test_synthetic_profile_is_not_a_shipped_profile() -> None:
"""Guards the instrument: if a future profile ever shares these overrides,
the two tests above stop proving anything and would go quietly green.
"""
assert isinstance(_SYNTHETIC, BundleProfile)
assert _SYNTHETIC.paths is not DEFAULT.paths
assert _SYNTHETIC.index is not DEFAULT.index
assert OKF_V0_2.paths is DEFAULT.paths
# `OKF_V0_2.index` stopped being DEFAULT's object at D5, which named
# `okf_version` in its `root_frontmatter`. That is NOT the distinctness this
# guard is about: every field that carries a NAME — the index filename and
# the concept filenames — is still shared, so a threading bug that passed
# the wrong profile would still emit identical bytes under either. Object
# identity was only ever a proxy for that; assert the thing itself, or this
# guard would go green on a change that leaves the instrument blind.
assert OKF_V0_2.index.name == DEFAULT.index.name
assert OKF_V0_2.index.link_template == DEFAULT.index.link_template
assert _SYNTHETIC.index.name != DEFAULT.index.name
assert _SYNTHETIC.paths.concept_suffix != DEFAULT.paths.concept_suffix

215
tests/test_root_frontmatter_emission.py
View file

@ -1,215 +0,0 @@
"""D5's emission half: the root `index.md` frontmatter block.
Upstream §8 binds `index.md` to "no frontmatter, with one exception: a
bundle-root `index.md` MAY carry an `okf_version` key (§12)", and §12 puts the
declaration "in a bundle-root `index.md` frontmatter block (the only place
frontmatter is permitted in an `index.md`)". Catalog verified both quotations
against the pinned commit `3fcbb9f` on 2026-07-31 and reported the same
reading, so placement is settled: frontmatter, not a body line.
The value never comes from a profile. V4/V-A5 gives `okf_version`'s *value* to
catalog (decision E1) and leaves this library the narrower obligation — name
the key, express any value. So the profile pins `root_frontmatter` and the
caller supplies the mapping.
Two assertions here read RAW BYTES rather than a parsed value, deliberately.
Catalog measured that a quoted value fails their shape regex with `exit 1`, and
that a UTF-8 BOM makes the marker invisible to them while still exiting `0`. A
parsed assertion masks exactly those two defects: `yaml.safe_load` returns the
string `"0.2"` whether or not it was quoted, and strips a BOM before the
caller ever sees it.
"""
from __future__ import annotations
import json
from dataclasses import replace
from pathlib import Path
import pytest
from llm_ingestion_okf.errors import MaterializationError
from llm_ingestion_okf.materialize import materialize_bundle
from llm_ingestion_okf.profiles import DEFAULT, OKF_V0_2
INGESTED_AT = "2026-07-16T12:00:00Z"
def build_case(tmp_path: Path) -> Path:
"""A minimal file-source manifest; the frontmatter block is what is under
test, so the extraction is kept as small as it can be."""
case = tmp_path / "case"
fixture = case / "fixture"
fixture.mkdir(parents=True)
(fixture / "rows.csv").write_text("id,label\n1,alpha\n", encoding="utf-8")
manifest = {
"manifest_version": 1,
"source": {"type": "file", "id": "root-fm", "root": "fixture"},
"bundle_summary": "Root frontmatter case.",
"extractions": [
{
"id": "rows",
"title": "Rows",
"query": "rows.csv",
"okf_type": "dataset",
"max_rows": 10,
}
],
}
(case / "manifest.json").write_text(json.dumps(manifest), encoding="utf-8")
return case
def test_root_index_carries_the_declared_key(tmp_path: Path) -> None:
"""The block opens the file and the body follows it."""
case = build_case(tmp_path)
bundle = tmp_path / "bundle"
materialize_bundle(
case / "manifest.json",
bundle,
INGESTED_AT,
profile=OKF_V0_2,
root_frontmatter_values={"okf_version": "0.2"},
)
assert (bundle / "index.md").read_bytes() == (
b"---\nokf_version: 0.2\n---\n\nRoot frontmatter case.\n- [Rows](ingest-rows.md)\n"
)
def test_the_value_is_unquoted_in_the_raw_bytes(tmp_path: Path) -> None:
"""Catalog's shape regex `/^\\d+(\\.\\d+)*$/` sees the quotes as part of the
value and exits 1. Asserted on bytes: a parsed check passes either way."""
case = build_case(tmp_path)
bundle = tmp_path / "bundle"
materialize_bundle(
case / "manifest.json",
bundle,
INGESTED_AT,
profile=OKF_V0_2,
root_frontmatter_values={"okf_version": "0.2"},
)
raw = (bundle / "index.md").read_bytes()
assert b"okf_version: 0.2\n" in raw
assert b'okf_version: "0.2"' not in raw
assert b"okf_version: '0.2'" not in raw
def test_the_file_carries_no_utf8_bom(tmp_path: Path) -> None:
"""A BOM leaves the marker invisible to catalog's gate while still exiting
0 — the failure that reports success, so it gets its own byte assertion."""
case = build_case(tmp_path)
bundle = tmp_path / "bundle"
materialize_bundle(
case / "manifest.json",
bundle,
INGESTED_AT,
profile=OKF_V0_2,
root_frontmatter_values={"okf_version": "0.2"},
)
raw = (bundle / "index.md").read_bytes()
assert not raw.startswith(b"\xef\xbb\xbf")
assert raw.startswith(b"---\n")
def test_a_key_the_policy_does_not_name_is_refused(tmp_path: Path) -> None:
"""`DEFAULT` names no root-frontmatter key, so offering one is a caller
error rather than something to write. Fail-fast: refused BEFORE any disk
mutation, or a rejected run would still leave a half-written bundle."""
case = build_case(tmp_path)
bundle = tmp_path / "bundle"
with pytest.raises(MaterializationError) as excinfo:
materialize_bundle(
case / "manifest.json",
bundle,
INGESTED_AT,
profile=DEFAULT,
root_frontmatter_values={"okf_version": "0.2"},
)
assert excinfo.value.code == "index_root_frontmatter_unexpected"
assert not bundle.exists()
def test_omitting_the_values_emits_no_frontmatter(tmp_path: Path) -> None:
"""§12 is a MAY, and none of upstream's four reference bundles declares the
key at all (catalog grepped `okf/bundles` and `okf/samples`: zero hits). A
profile that names the key must therefore still emit a bundle without it."""
case = build_case(tmp_path)
bundle = tmp_path / "bundle"
materialize_bundle(case / "manifest.json", bundle, INGESTED_AT, profile=OKF_V0_2)
raw = (bundle / "index.md").read_bytes()
assert not raw.startswith(b"---")
assert b"okf_version" not in raw
def test_key_order_follows_the_policy_not_the_mapping(tmp_path: Path) -> None:
"""The policy pins the order; a caller's dict ordering must not reach the
file, or two callers passing the same keys would emit different bytes."""
case = build_case(tmp_path)
bundle = tmp_path / "bundle"
profile = replace(
OKF_V0_2,
index=replace(OKF_V0_2.index, root_frontmatter=("okf_version", "bundle_profile")),
)
materialize_bundle(
case / "manifest.json",
bundle,
INGESTED_AT,
profile=profile,
root_frontmatter_values={"bundle_profile": "okf-v0-2", "okf_version": "0.2"},
)
raw = (bundle / "index.md").read_bytes()
assert raw.startswith(b"---\nokf_version: 0.2\nbundle_profile: okf-v0-2\n---\n\n")
def test_the_frozen_golden_declares_the_version_gate_safely() -> None:
"""D5's raw-byte guard, asserted on the COMMITTED fixture rather than on a
fresh run.
`test_golden.py` already compares a run against these bytes, but that only
proves the code agrees with itself: regenerate the fixture from a quoted or
BOM-carrying emitter and both sides move together, silently. This asserts
the shape of what we froze and hand to catalog's gate — the one thing a
self-comparison cannot catch.
"""
raw = (
Path(__file__).parent.parent
/ "examples"
/ "ingest-golden-okf-v0-2"
/ "expected-bundle"
/ "index.md"
).read_bytes()
assert raw.startswith(b"---\nokf_version: 0.2\n---\n\n")
assert not raw.startswith(b"\xef\xbb\xbf") # a BOM exits 0 while hiding the marker
assert b'"' not in raw.split(b"---\n", 2)[1] # a quoted value exits 1
assert b"\r\n" not in raw # LF-only, like every other emitted file
def test_reingest_in_place_is_byte_identical(tmp_path: Path) -> None:
"""A-E5. The second run goes into the FIRST run's directory — the shape the
pilot runs — so the frontmatter block must survive §6 index maintenance
rather than being appended a second time or dropped."""
case = build_case(tmp_path)
bundle = tmp_path / "bundle"
call = dict(
profile=OKF_V0_2,
root_frontmatter_values={"okf_version": "0.2"},
)
materialize_bundle(case / "manifest.json", bundle, INGESTED_AT, **call) # type: ignore[arg-type]
first = (bundle / "index.md").read_bytes()
materialize_bundle(case / "manifest.json", bundle, INGESTED_AT, **call) # type: ignore[arg-type]
assert (bundle / "index.md").read_bytes() == first

371
tests/test_strict_v1.py
View file

@ -1,371 +0,0 @@
"""The proving consumer's contract as a profile (Phase 3, step 3).
`STRICT_V1` is the `claude-code-llm-wiki` bundle contract expressed in the same
object `DEFAULT` uses. Everything pinned here was measured in their repository
on 2026-07-25 and settled in `docs/phase-3-split-table.md`; the eight-key
required set is their operator's ratification, deliberately stricter than the
four-key constant their own validator enforces.
These tests are also the reader that earns the new schema fields. A profile
field no code reads is a claim no test covers, so `required`, `allowed`,
`nullable` and `key_pattern` land together with `FrontmatterSchema.violations`,
which is what reads them.
What is NOT here, and why: `violations` takes an already-parsed mapping. This
library's line-oriented parser cannot represent the block lists and nulls the
consumer's frontmatter admits — that is precisely why Door C writes imported
concepts verbatim — so parsing strict-v1 frontmatter is a separate concern from
judging it, and lands separately.
"""
from __future__ import annotations
import pytest
from llm_ingestion_okf.profiles import (
DEFAULT,
STRICT_V1,
FrontmatterSchema,
)
# The eight keys the consumer's operator ratified on 2026-07-25, measured
# present on 522/522 of their documents.
RATIFIED_REQUIRED = frozenset(
{
"type",
"title",
"description",
"timestamp",
"layer",
"source_tier",
"source_url",
"source_sha",
}
)
# `FRONTMATTER_KEY_ORDER` in their `bundle.py`: the eight above plus three
# layer-specific keys, doubling as canonical emission order.
RATIFIED_ORDER = (
"type",
"title",
"description",
"timestamp",
"layer",
"source_tier",
"source_url",
"source_sha",
"version",
"date",
"summary",
)
def wiki_document() -> dict[str, object]:
"""A conforming `concepts/` page, every allowlisted key present."""
return {
"type": "Concept",
"title": "Hooks",
"description": "Event-driven automation in Claude Code.",
"timestamp": "2026-07-25T12:00:00Z",
"layer": "concepts",
"source_tier": "official-docs",
"source_url": "https://docs.claude.com/en/docs/claude-code/hooks",
"source_sha": "0" * 64,
"version": "2.1.183",
"date": "2026-07-25",
"summary": "What hooks are and when they fire.",
}
def codes(schema: FrontmatterSchema, values: dict[str, object]) -> list[str]:
return [violation.code for violation in schema.violations(values)]
# --- the profile itself ---------------------------------------------------
def test_strict_v1_closes_the_type_enum_to_the_four_document_classes() -> None:
"""Their four classes exactly — an open set would admit anything."""
assert STRICT_V1.types.allowed == frozenset({"Concept", "Guide", "Reference", "Release"})
def test_strict_v1_requires_the_eight_ratified_keys() -> None:
"""Eight, not the four their `REQUIRED_KEYS` constant spells.
Their constant is the emit-path minimum; the contract their operator
ratified is all eight. This is the one place the profile is deliberately
stricter than the consumer's own validator, and the agreed resolution if a
document class ever legitimately lacks `source_sha` is to bump the profile,
not to bend the bundle.
"""
assert STRICT_V1.frontmatter.required == RATIFIED_REQUIRED
def test_strict_v1_allowlist_is_eleven_keys_and_doubles_as_emission_order() -> None:
"""The allowlist and the order are one list, not two that can drift apart.
Their `BUNDLE_HASH_REGISTRY` gate depends on regeneration being
byte-identical, which is what the fixed order buys.
"""
assert STRICT_V1.frontmatter.order == RATIFIED_ORDER
assert STRICT_V1.frontmatter.allowed == frozenset(RATIFIED_ORDER)
def test_the_three_layer_specific_keys_are_optional_not_required() -> None:
"""`version`/`date` sit on 351/522 documents and `summary` on 349/522.
Requiring them would reject pages the consumer ships today.
"""
allowed = STRICT_V1.frontmatter.allowed
assert allowed is not None
for key in ("version", "date", "summary"):
assert key in allowed
assert key not in STRICT_V1.frontmatter.required
def test_summary_is_the_only_nullable_key() -> None:
"""The single deliberate escape from strings-only frontmatter."""
assert STRICT_V1.frontmatter.nullable == frozenset({"summary"})
# --- the reader: a conforming document -----------------------------------
def test_a_conforming_wiki_document_has_no_violations() -> None:
assert STRICT_V1.frontmatter.violations(wiki_document()) == ()
def test_the_optional_keys_may_simply_be_absent() -> None:
"""A `concepts/` page carries neither `version` nor `date`."""
document = wiki_document()
for key in ("version", "date", "summary"):
del document[key]
assert STRICT_V1.frontmatter.violations(document) == ()
# --- the reader: cross-profile rejection ---------------------------------
def test_a_door_a_ingest_document_is_rejected_under_strict_v1() -> None:
"""The plan's cross-profile test, frontmatter half.
A bundle this library generates under `DEFAULT` carries the ingest-spec
layer, which shares only `type` and `title` with the consumer's contract:
six required keys are missing and five of its own keys are off-allowlist.
"""
door_a = {
"type": "dataset",
"title": "Orders",
"source_system": "golden-catalogue",
"source_query": "orders.csv",
"ingested_at": "2026-07-16T12:00:00Z",
"ingest_manifest": "manifest@37674ac20059e788",
"generated": "true",
}
violations = STRICT_V1.frontmatter.violations(door_a)
assert violations != ()
missing = {v.key for v in violations if v.code == "frontmatter_key_missing"}
assert missing == RATIFIED_REQUIRED - {"type", "title"}
off_allowlist = {v.key for v in violations if v.code == "frontmatter_key_not_allowed"}
assert off_allowlist == {
"source_system",
"source_query",
"ingested_at",
"ingest_manifest",
"generated",
}
def test_default_judges_nothing_about_keys_so_both_doors_stay_valid() -> None:
"""`DEFAULT` must NOT grow a required set — measured, not assumed.
Door A emits seven keys, Door B six, and Door C writes an imported concept
verbatim with whatever frontmatter the sender wrote. A required set on
`DEFAULT` would therefore declare invalid the very bundles this library
produces. `DEFAULT` keeps an open namespace; the rejection of a wiki-shaped
bundle under `DEFAULT` comes from the index and path shape instead.
"""
assert DEFAULT.frontmatter.required == frozenset()
assert DEFAULT.frontmatter.allowed is None
assert DEFAULT.frontmatter.violations(wiki_document()) == ()
assert DEFAULT.frontmatter.violations({"type": "dataset", "title": "Orders"}) == ()
# --- the reader: each check ----------------------------------------------
def test_a_missing_required_key_is_named_individually() -> None:
document = wiki_document()
del document["source_sha"]
violations = STRICT_V1.frontmatter.violations(document)
assert [(v.key, v.code) for v in violations] == [("source_sha", "frontmatter_key_missing")]
def test_a_key_outside_the_allowlist_is_refused() -> None:
"""A closed allowlist is what makes the contract enumerable at all."""
document = wiki_document()
document["author"] = "someone"
assert codes(STRICT_V1.frontmatter, document) == ["frontmatter_key_not_allowed"]
def test_a_key_name_outside_the_grammar_is_refused() -> None:
"""`^[a-z_]+$`: a key that reaches the allowlist check malformed would
otherwise be reported only as 'not allowed', which hides the real defect."""
schema = FrontmatterSchema(order=("type",), key_pattern=STRICT_V1.frontmatter.key_pattern)
violations = schema.violations({"Source-URL": "x"})
assert [(v.key, v.code) for v in violations] == [("Source-URL", "frontmatter_key_malformed")]
def test_summary_may_be_null_and_no_other_key_may() -> None:
document = wiki_document()
document["summary"] = None
assert STRICT_V1.frontmatter.violations(document) == ()
document["description"] = None
assert [(v.key, v.code) for v in STRICT_V1.frontmatter.violations(document)] == [
("description", "frontmatter_value_null")
]
def test_a_non_empty_list_of_strings_is_a_legal_value() -> None:
"""The block-list shape their `Release` pages carry.
A profile that assumed scalars-only would reject the consumer's own pages —
and a parser that assumed scalars-only would silently destroy them, which
is why Door C writes imported concepts verbatim.
"""
document = wiki_document()
document["source_url"] = ["https://a.example/one", "https://a.example/two"]
assert STRICT_V1.frontmatter.violations(document) == ()
@pytest.mark.parametrize(
"value",
[[], ["ok", 3], {"nested": "map"}, 7, True],
ids=["empty-list", "list-with-non-string", "mapping", "int", "bool"],
)
def test_a_value_outside_the_scalar_or_string_list_shape_is_refused(value: object) -> None:
document = wiki_document()
document["title"] = value
assert [(v.key, v.code) for v in STRICT_V1.frontmatter.violations(document)] == [
("title", "frontmatter_value_shape")
]
# --- determinism and construction-time invariants -------------------------
def test_violations_come_back_in_a_deterministic_order() -> None:
"""Two runs over the same document must report identically ordered.
Determinism is bit-exact in this library; a violation list that depended on
mapping insertion order would make a report irreproducible.
"""
document = wiki_document()
del document["source_sha"]
del document["layer"]
document["zzz"] = "off-allowlist"
document["aaa"] = "off-allowlist"
reported = [(v.key, v.code) for v in STRICT_V1.frontmatter.violations(document)]
assert reported == sorted(reported)
reversed_document = dict(reversed(list(document.items())))
assert [
(v.key, v.code) for v in STRICT_V1.frontmatter.violations(reversed_document)
] == reported
def test_a_required_key_outside_the_allowlist_cannot_be_constructed() -> None:
"""An unsatisfiable schema is a defect in the profile, not in a document."""
with pytest.raises(ValueError, match="required"):
FrontmatterSchema(
order=("type", "title"),
required=frozenset({"type", "absent_from_allowlist"}),
allowed=frozenset({"type", "title"}),
)
def test_a_nullable_key_outside_the_allowlist_cannot_be_constructed() -> None:
with pytest.raises(ValueError, match="nullable"):
FrontmatterSchema(
order=("type",),
allowed=frozenset({"type"}),
nullable=frozenset({"summary"}),
)
def test_strict_v1_path_namespaces_are_the_spec_namespaces() -> None:
"""The door filename prefixes are an ingest-spec invariant, not a preference.
They describe what Doors A/B/C write, so they are the same under any
profile; the consumer's own files are written by their pipeline, not by a
door, and are unaffected either way.
"""
assert STRICT_V1.paths == DEFAULT.paths
def test_strict_v1_index_policy_is_the_consumers_shape() -> None:
"""The gap this test used to pin is closed; it now states the shape.
Every value is read out of their code rather than recalled — the entry
template and the root/nested frontmatter split from `bundle.py:498-567`,
the bidirectional match from gate `BUNDLE_INDEX_COMPLETE`
(`validate.py:1081-1120`, severity ERROR). The reader that judges all of
it lives on `IndexPolicy.violations`, so no field here is written without
something that reads it.
"""
index = STRICT_V1.index
assert index != DEFAULT.index
assert index.name == DEFAULT.index.name
assert index.render_link("Hooks", "hooks.md", "How hooks fire.") == (
"* [Hooks](hooks.md) - How hooks fire."
)
assert index.per_directory is True
assert index.heading_required is True
assert index.allows_prose is False
assert index.entries_match_directory is True
assert index.root_frontmatter == ("okf_version", "bundle_profile", "okf_spec_commit")
def test_a_conforming_wiki_root_index_passes_and_a_default_one_does_not() -> None:
"""The plan's second cross-profile test: a bundle valid under one profile
is refused by the other. It could not come from frontmatter — DEFAULT has
no required key set, by measurement — so it comes from index shape.
"""
conforming = (
"---\nokf_version: 0.1\nbundle_profile: strict-v1\n"
"okf_spec_commit: d44368c\n---\n\n# Wiki\n\n"
"* [Concepts](concepts/index.md) - Machine-generated index of concepts.\n"
)
assert (
STRICT_V1.index.violations(conforming, is_root=True, expected_targets={"concepts/index.md"})
== ()
)
default_shaped = "# Wiki\n\nCurated prose.\n\n- [Concepts](concepts/index.md)\n"
codes = {
violation.code
for violation in STRICT_V1.index.violations(
default_shaped, is_root=True, expected_targets={"concepts/index.md"}
)
}
assert codes == {
"index_root_key_missing",
"index_prose_not_allowed",
"index_entry_missing",
}
# ...and the reverse direction: DEFAULT judges neither of them, because a
# judging default would condemn the bundles this library itself writes.
assert DEFAULT.index.violations(conforming, is_root=True) == ()
assert DEFAULT.index.violations(default_shaped, is_root=True) == ()
def test_strict_v1_emission_is_the_ordered_prefix_then_the_sorted_tail() -> None:
"""Commons decision D1, unchanged by the stricter namespace."""
emitted = STRICT_V1.frontmatter.emit(
{"title": "Hooks", "type": "Concept", "description": "d", "timestamp": "t"}
)
assert emitted == "type: Concept\ntitle: Hooks\ndescription: d\ntimestamp: t"