open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

Compare commits

base: open:v0.4.0
Branches Tags
open:main
open:v0.5.0a2
open:v0.5.0a1
...
compare: open:v0.5.0a1
Branches Tags
open:main
open:v0.5.0a2
open:v0.5.0a1

42 commits
v0.4.0 ... v0.5.0a1

Author SHA1 Message Date
Kjell Tore Guttormsen
af6e6afe36 chore(release): v0.5.0a1 — version sync, and the entry that makes the pilot provisional 
The tag was specified to land on 128109b, where the package still declared
0.4.0 in both places a version is written. The pilots install FROM the tag, so
they would have pinned a pre-release that reports the previous release.

Two requirements the "minimally requires" list did not cover, both stated
elsewhere in the plan:

- The CHANGELOG entry is load-bearing, not ceremony: the plan makes the
  provisional status real by having the profile docstring AND the entry both
  say the surface may change without a deprecation cycle, and name the pilot
  repos. The docstring had it (profiles.py:694); the CHANGELOG stopped at
  0.4.0.
- pyproject.toml and __init__.py both said 0.4.0.

Nothing in the run path reads __version__, which is why a stale one survives a
green suite. test_the_declared_version_agrees_with_the_packaged_one is the only
machine check on it — written failing first, against a one-sided bump. The tag
NAME remains a human step; no test can catch that one.

README keeps pointing at v0.4.0 on purpose: a pre-release for three named
pilots is not the general install line, and OKF_LATEST does not move until GA.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVTup4v7tW9QiVyBENk2LV
 2026-07-31 18:26:48 +02:00
Kjell Tore Guttormsen
128109bef0 docs(okf-v0.2): the F1 disposition, and A-E6's rationale after catalog 1ca27f6 
Three corrections to the alignment plan, two of them because a fact a peer
repo gave us stopped being true.

**P1 dispositions.** F1 is fixed before the tag; the other five are not. The
line is whether a finding bites in code the pilots will pin. F1 did. F2/F3
do not: measured, all three `TypePolicy.rejection` call sites hard-code
`DEFAULT.types`, and both `DEFAULT` and `OKF_V0_2` set `allowed=None`, so the
closed-set branch is unreachable in shipped code and fires only for a caller
constructing `STRICT_V1` directly — which is what the sweep did. Dequoting in
`TypePolicy` would repair a write path that never sees a quote, and would fix
the meaning of a quote without a value model able to express one. F2/F3/F4
are one root cause and go to D1b; F5/F6 stay P4's.

**A-E6's reason did not survive, though the expectation did.** It justified
"unquoted" as a property of catalog's gate — their shape regex read quotes as
part of the value and so rejected upstream's own §12:773 example. Catalog
resolved that in `1ca27f6`: from spec 0.3 the value is unquoted before the
form check and both forms pass. The expectation stands on a different ground
now, supplied by commons the same day: `method-spec.md:90` parses frontmatter
line-oriented, so a quote is a character in the value rather than syntax a
parser removes, and upstream emits the same way. Recorded as a supersession
rather than swapped silently — reading a stale rationale as current is this
expectation's own defect class.

**TEST B must name the reader revision.** `okf-check.mjs` changed in the same
commit, and the reader's placement-blindness is now a deliberate choice
rather than an unanchored regex. A report that does not say which revision it
ran is not reproducible.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JqCmfJ2ukpFXeFjfab8wvy
 2026-07-31 18:16:30 +02:00
Kjell Tore Guttormsen
2c01e58259 fix(profiles): P1-F1 — a permitted root key is not a required one 
`IndexPolicy.root_frontmatter` carried two claims at once: which keys the
root index may carry and in what order, and which it must carry. The judge
read it as the second, so `OKF_V0_2` — which names `okf_version` only to fix
its position, upstream granting it as a MAY (§8:509-510, §12:773-775) —
reported every bundle exercising that MAY as violating. Measured in P1 over
17 bundles: 14 failed with exactly this one violation, upstream's own four
reference bundles among them, while D5's emitter treated the same key as
optional. We emitted a MAY correctly and graded it a MUST.

`root_frontmatter` now permits and orders; `root_frontmatter_required`
demands. A required key outside the ordered set fails at construction, the
same contradiction `FrontmatterSchema` already refuses when `required`
strays outside `allowed`.

- `OKF_V0_2` requires none — upstream's MAY, stated on the judge side too.
- `STRICT_V1` requires all three: the proving consumer's root index carries
  exactly those keys in that order on every bundle measured (`c5141f8`), so
  separating the meanings costs them nothing.
- `DEFAULT` names no root keys and is untouched.

Re-measured over the eight bundle-root indexes reachable locally (our four
goldens, upstream's four): 7 of 8 failing under the old semantics, 0 of 8
after. Emit path byte-identical — the golden suite fails otherwise.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JqCmfJ2ukpFXeFjfab8wvy
 2026-07-31 18:16:13 +02:00
Kjell Tore Guttormsen
a8ceea9667 docs(okf-v0.2): P1 — the read-only sweep, and the six findings it produced 
17 bundles, 602 documents, five corpora: our goldens, portfolio-optimiser-claude,
catalog's green parity corpus, the wiki's bundle, and upstream's own four
reference bundles at the pinned commit. Read-only throughout; nothing was written
into a consumer tree and no consumer ran anything.

Every finding is on the JUDGE path, so none of them blocks v0.5.0a1, P2 or P3 —
both remaining pilot tests are emit-side. They are P4's work.

The two that matter most needed real data to surface:

- root_frontmatter conflates "permitted and ordered" with "required": 14 of 17
  bundles fail with exactly one violation each, including all four of upstream's
  own reference bundles, purely for exercising a MAY the emitter treats as
  optional. We emit a MAY correctly and grade it as a MUST.
- TypePolicy.allowed cannot judge a consumer that quotes its scalars: 0/526 wiki
  documents pass, 526/526 pass dequoted. The suite cannot see it because its
  tests hand-build the mapping and never parse a file.

Also confirmed against real third-party v0.2 data: V-A3 holds — no upstream
`generated` value reads as owned, so a real v0.2 bundle cannot talk the collision
gate into overwriting.

Test C: C-E1 differed (the quoting gap), C-E2/C-E3/C-E4 as expected. The wiki's
copy went out through coord per their condition, never as a file here.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0186K8oLEnaFfXZ2dWGyHzrB
 2026-07-31 17:45:54 +02:00
Kjell Tore Guttormsen
2504011010 feat(okf-v0.2): D5 — the v0.2 golden fixture, with okf_version in root frontmatter 
Placement settled by catalog's own reading of upstream at the pinned commit
3fcbb9f: SS8:509-510 and SS12:773-775 both put `okf_version` in a bundle-root
`index.md` frontmatter block, and SS12 calls it the only place frontmatter is
permitted in an index. Catalog's spec says the opposite about the same file;
that divergence is theirs against upstream, and we conform to upstream.

The value never touches a profile. `OKF_V0_2.index.root_frontmatter` names the
key; the caller supplies the value through a new keyword-only
`root_frontmatter_values` mapping. That keeps V4/V-A5 intact - `okf_version`'s
value tracks the upstream Google version and belongs to catalog (E1), so a
constant here would claim a decision we do not own and would have to be chased
on every upstream release. In the fixture the value is fixture DATA
(`okf-version.txt`), not a literal in our source.

Ordering comes from the policy, not the caller's mapping: a dict preserves
insertion order, so two callers passing the same keys would otherwise emit
different bytes. A key the policy does not name is refused fail-fast, before
any disk mutation. Omitting the argument emits no block at all - SS12 is a MAY
and none of upstream's four reference bundles declares the key.

The block is written only when the index is CREATED, so a re-run into an
existing bundle stays byte-identical (A-E5).

Raw-byte assertions rather than parsed ones, on the committed fixture as well
as on fresh runs: catalog measured that a quoted value fails their shape regex
with exit 1 and that a BOM hides the marker while still exiting 0.
`yaml.safe_load` returns "0.2" either way and strips a BOM first, so a parsed
assertion masks exactly those two defects. Asserting the frozen fixture catches
what a self-comparison cannot - regenerating from a broken emitter moves both
sides together.

A-E6 is now placement-explicit (promised catalog in 99cf987), and separates the
two byte properties: BOM-free is a property of the file, unquoted is a property
of CATALOG'S GATE and not of OKF v0.2 - upstream's own SS12 example is quoted,
so their gate rejects the spec's canonical form.

README gains the upstream-version section it was missing; CLAUDE.md gains the
mechanism behind "no profile hard-codes an upstream version": a profile names a
key, a caller owns its value.

550 -> 559 tests. test_profile_threading's `OKF_V0_2.index is DEFAULT.index`
assertion is replaced rather than deleted: object identity was a proxy for "the
shipped profiles differ in no NAME-bearing field", which is what makes the
synthetic test profile necessary, so the guard now asserts that directly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dgkSPjkLpACjMayd9R5jx
 2026-07-31 17:27:48 +02:00
Kjell Tore Guttormsen
ed08ac15e9 feat(profiles): materialize_bundle takes a keyword-only profile (req 6) 
`OKF_V0_2` landed in D2 but was unreachable from outside: no door took a
profile. This threads one through, keyword-only behind the `*` the signature
already carried, so every three-positional call site stays source-compatible —
which is what po-claude asked for, and what makes additivity a property of the
signature rather than something a consumer measures.

Nine sites, not the ~6 STATE claimed. The load-bearing one is the call at
materialize.py:378: the CONTENT phase has accepted `profile` since D2, but the
call site never passed one, so A-E3/A-E4/A-E5 were all unreachable. The other
eight are the disk phase (ownership glob, index name, index maintenance,
concept filenames) plus `generated_filename` in manifest.py.

`link_in_index` is public and called from all three doors, so it gets
`*, profile=DEFAULT` rather than having the lookup moved to the call site:
doors B and C keep exactly the behaviour they had, and which profile THEY own
stays an open question instead of being decided silently by a signature change.

Byte-neutrality is proven, not asserted: `OKF_V0_2.paths is DEFAULT.paths` and
`.index is DEFAULT.index`, and the golden suite is green. That identity is also
why six of the nine sites cannot be proven reachable by any shipped-profile
test — no assertion distinguishes two names for one object. A synthetic
test-only profile renaming the index and the concept files closes that gap, so
a site left on `DEFAULT` fails by name rather than passing quietly.

Scope stated rather than glossed: the profile does NOT reach manifest type
validation (`manifest.py:198` still reads `DEFAULT.types`; measured equal to
`OKF_V0_2.types`, so nothing is hidden today), and `STRICT_V1` is not supported
here — its index policy sets three judging fields the materializer does not
honour. Both are named in the docstring.

No `okf_version` anywhere: that lands once, at D5, when the §12 placement
question closes.

550 tests pass (was 542).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tf2BbC8uSRVU4ApQ9NL7QR
 2026-07-31 15:37:17 +02:00
Kjell Tore Guttormsen
ddc8f76446 docs(mcp): Q2 gains upstream's tool surface as measured prior art 
sources/base.py plus tools/source_tools.py is the same problem phase 5 option
(A) contemplates, already solved in ADK-specific form: a data system exposed as
model-callable tools. Recorded under Q2 because that question calls the tool
surface the real design work, and this is a worked example of it.

Three signals, each with the reason it is a signal: enumerate and read are
abstract while sample_rows defaults to None, so sampling is advisory; the
surface is per-concept, which is evidence that Door B's per-run collision gate
and index generation are the side that has to give; and two things not to copy
— an untyped dict[str, Any] return that only a model can interpret, and a find()
that linear-scans list_concepts() and so assumes total enumeration is cheap,
against this library's rule that nothing enumerates unless a profile says
derived.

Read at the pinned commit, not vendored.
 2026-07-27 20:16:47 +02:00
Kjell Tore Guttormsen
ff9f2f82ca docs(roadmap): phase 5 — MCP as a way to populate a bundle 
Committed by the operator; the design is open and the scope is not. Recorded
in the roadmap with a scoping plan rather than in the v0.2 alignment doc, which
is about OKF v0.2 conformance and is the wrong home.

The plan settles nothing on purpose. It names the fork everything turns on —
whether we are the MCP server (an agent calls our doors, Door B's territory,
ours to design) or an MCP client (a manifest source type, which is normative in
commons' ingest-spec and therefore starts as a request to them, not as code
here). The two share a protocol and nothing else, including their owner.

Constraints written down because each has already refused something: the
one-runtime-dependency rule (an MCP SDK would be the second — stdlib JSON-RPC
or an optional extra), no model calls in the run path (the model may CALL us,
never the reverse), the guard persist gate (MCP is transport, not a bypass),
the network opt-in, and the required ingested_at.
 2026-07-27 20:12:58 +02:00
Kjell Tore Guttormsen
0a7a77d5a8 docs(okf-v0.2): U7 resolved — the 'not:' family is an example, not a field 
Zero occurrences in the v0.2 SPEC.md, zero in the v0.1 SPEC.md at the parent of
the migration commit, no reference in upstream's reference_agent source, and one
file in the whole bundle set. An undocumented hand-authored key, legal under
§14's unknown-keys tolerance — a demonstration of the open namespace rather than
a field to support.

The same sweep settles '# Cited by': body prose in two policy files, absent from
the spec, and independently DERIVED by the visualizer (viz.html:56) from sources.
Both fit the upstream pattern already visible in credibility scores and trust
tiers — OKF stores signals, consumers derive views.

Answering this needed the v0.1 spec text, upstream's source, and a whole-tree
grep at once, which is what the local pinned clone made possible. That is U8's
lesson applied rather than restated.
 2026-07-27 16:32:40 +02:00
Kjell Tore Guttormsen
e5e574df86 docs(okf-v0.2): U8 — sources replaces the citations LIST, not the attribution 
Per-claim attribution moves to a markdown footnote keyed to a sources[].id
(SPEC §4.2 :224-225, §5.1 :347-360), and the shipped acme_retail bundle carries
it inside a schema table. Our supersession row stated the destination and not
the mechanism, so a consumer following it would migrate the list and drop the
claim->source binding.

Measurable at the pinned commit the whole time; surfaced by Google's launch post
2026-07-27. U1-U7 measured frontmatter in all four reference bundles and walked
past the bodies. V-A1's discharge is narrowed rather than reworded: the pin
holds, the reading's depth did not.

Consequence outside the doc: the v0.2 exposure report sent to the wiki states
the incomplete half under the heading that exists to size their cost. Correction
owed, same class as A-E3.
 2026-07-27 16:24:56 +02:00
Kjell Tore Guttormsen
aac765fa5a docs(okf-v0.2): A-E3 no longer outlives the decision it encoded 
The operator chose V1(d) for OKF_V0_2, so the pilot expectation A-E3 was
written against — a producer-and-version actor — is stale. Corrected in place,
with the reason, before po-claude receives the tag rather than after they run.

V1's own section records what the choice does and does not settle: DEFAULT's
generated is commons' §5 layer and stays in their queue as item 8. TDD step 5
is marked done at c90171d.
 2026-07-27 09:12:07 +02:00
Kjell Tore Guttormsen
c90171dad0 feat(okf-v0.2): D2 — the profile, the ownership policy, and sources 
The v0.2 profile lands additively: DEFAULT keeps stating commons' §5 layer
byte for byte (the golden suite is the proof, not the claim), and OKF_V0_2
adds the families v0.2 introduces on top of DEFAULT's key order.

Two questions the plan left open were the operator's, and both were decided
before code rather than discovered during it:

- `generated.by` is `process:llm-ingestion-okf` — plan V1's option (d), the
  §7 process actor form. The value carries no version, which is what keeps a
  byte-compared fixture stable across releases and leaves a shared
  cross-implementation fixture possible. A-E3 was written against the
  `<producer>/<version>` form and is now STALE in the pilot specification; the
  correction is owed to portfolio-optimiser-claude before they run.
- `sources[].resource` is the manifest source's locator verbatim: the file
  root, the sql connection_ref (an env-var NAME, never its value), or the http
  base_url. `credential_ref` is not a locator and is never emitted.

Ownership becomes a policy on the profile rather than a literal in the gate.
The emitter and `_is_ingest_owned` are coupled through the stamp value, so
OwnershipPolicy is where they meet and can only change together. Recognition
is one-way by decision: OKF_V0_2 owns the v0.1 stamp too, so a DEFAULT-written
bundle re-runs IN PLACE (operator, 2026-07-27), while DEFAULT still refuses a
v0.2 file rather than replacing it — V-A3's fail-safe is preserved.

An unquotable locator is refused rather than emitted. Measured with PyYAML
rather than reasoned: `[{ id: a, resource: data, backup }]` raises nothing and
parses to a mapping with a `backup` key nobody wrote, so the failure mode is a
silently wrong provenance record. Validation, not repair.

Deliberately NOT here: the `okf_version` declaration. §12 makes it a MAY and
none of upstream's four reference bundles exercises it; WHERE it goes is open
between upstream's root-index frontmatter and catalog's body-line convention,
and catalog verifies against upstream first. It is declared once, at D5.

V-A5 is extended to the new profile — the one whose NAME is the place a
version literal would look natural, and it carries none.

542 tests, mypy --strict clean.
 2026-07-27 09:11:12 +02:00
Kjell Tore Guttormsen
99cf98749f docs(okf-v0.2): A-E6 does not say WHERE, and that silence is now load-bearing 
Upstream §12 puts okf_version in the root index.md's frontmatter;
catalog's own spec says an index.md has no frontmatter and carries the
marker as a body line. A bundle conforming to one is non-conformant
against the other, so the placement A-E6 leaves unstated is no longer
cosmetic. Their gate cannot settle it: the shape regex is unanchored and
matches in both places, so a green TEST B measures the value and never
the placement. D5 stays unfrozen until catalog has verified against
upstream; A-E6 becomes placement-explicit before the pilot receives it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
 2026-07-26 21:38:46 +02:00
Kjell Tore Guttormsen
b1a92c940d docs(okf-v0.2): the guard measurement removes the form question, and A-E5 grows the second run 
Two corrections, both measured against guard 0.2.0 today rather than
reasoned.

Requirement 1: the T2 parser refuses every route to a mapping — the flow
form on the disallowed-indicator set, the block form on the nested-mapping
check, dotted keys on the key pattern. A v0.2 `generated` mapping has no
expressible form through Door C at all, so the chosen flow form is not
worse than the alternatives; there are none. The decision stands on its
original grounds. What changes is a claim: no round-trip through a
guard-gated import may be asserted. Door B never parses frontmatter, so
the grammar binds import, not emission.

A-E5: po-claude located a hole in A-E1. The emitter's `generated: true`
and the ownership predicate are coupled through that literal, and
OKF_V0_2 changes exactly it — one run compared byte for byte cannot see
the predicate stall, since that needs two runs into the same directory.
A-E5 now names the same-directory second run and expects it to succeed.
The separate in-place DEFAULT-to-v0.2 question is marked undecided rather
than quietly turned into an expectation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
 2026-07-26 21:34:19 +02:00
Kjell Tore Guttormsen
a00355533e fix(okf-v0.2): V-A7 counted every naming field but nullable 
The gate refuses a schema that names both `timestamp` and `generated`,
because OKF §13.1 grants the legacy fallback only while `generated` is
ABSENT. `__post_init__` built the named set from `order | required |
allowed` and left `nullable` out, so

    FrontmatterSchema(order=("timestamp", "title"),
                      nullable=frozenset({"generated"}))

constructed, and `violations()` on a document carrying both returned ().
The namespace is open by default, so a key named only by `nullable` is
admitted exactly as surely as an emitted one — the omission was a hole in
the gate, not a narrower gate.

Found by the proving consumer the same day the gate was written; their
construction is the red test case.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
 2026-07-26 21:28:13 +02:00
Kjell Tore Guttormsen
d77224d4ed docs(runbook): the profile parameter is decided, not shipped 
Caught while verifying facts for a consumer notice: the runbook stated in the
past tense that `profile` landed as a keyword-only argument with a DEFAULT
default. Measured -- no door takes the argument at all. The decision is real and
stays; the claim that it is in the code was about to be repeated to a consumer
as a property they could build on.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
 2026-07-26 20:18:38 +02:00
Kjell Tore Guttormsen
c85347e8a5 docs(okf-v0.2): steps 1-3 are done, and step 3 was not behavior-neutral 
The plan's own line said steps 1-3 change no behavior. V-A7's construction gate
made that untrue, and a stale premise in a plan is the class of thing this repo
has already paid for twice. Recorded where it was wrong, with the operator
decision that put the gate before D2 rather than at it, and with Door C's six
tolerance cases measured rather than expected.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
 2026-07-26 20:10:37 +02:00
Kjell Tore Guttormsen
7bc366bc1a feat(okf-v0.2): V-A7 -- a profile cannot name both timestamp and generated 
Plan step 3's remaining half, and the one place steps 1-3 add behavior rather
than only characterizing it. Operator decision this session: build the gate NOW
rather than at D2, so the multi-file D2 work is developed with the invariant
already in place.

OKF section 13.1 reads `timestamp` as a legacy stand-in for `generated.at`
only while `generated` is ABSENT. A schema able to name both can therefore
describe a document with neither a valid `generated.at` nor an eligible
fallback. Refused at construction, the same shape as the reserved `verdict`
layer -- and one level below the sketch that was approved, on
`FrontmatterSchema` rather than `BundleProfile`: the schema is the policy that
names frontmatter keys, it is the exact C3 analog, and a profile must build its
schema first, so the stricter placement subsumes the other.

Stated as key names rather than as a judgement on values, deliberately: every
`generated` a schema can express today is a scalar (`_is_legal_value`) and so
malformed as a v0.2 mapping, which makes naming both exactly the hazard. It
narrows when a value model can express a well-formed `generated` -- recorded at
the constant rather than left for a reader to reconstruct.

Behavior-neutral for everything shipped, and not by assertion: DEFAULT emits
`generated` and no `timestamp`, STRICT_V1 the reverse (which is what puts the
wiki on the legacy path rather than in a defect), and the byte-exact golden
suite is green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
 2026-07-26 20:08:51 +02:00
Kjell Tore Guttormsen
1215f985ae test(okf-v0.2): steps 1-3 -- the safety net before the gate is touched 
Plan steps 1-3: characterization only, no production code, so the one part of
the v0.2 work we called risky -- `_is_ingest_owned`, the pre-mutation collision
gate -- is pinned before it moves.

Step 1. An inline flow mapping round-trips through the scalar
`parse_frontmatter` verbatim (V-A2); a block list does not, and the measured
key-space pollution (`- id`, `resource` arriving as frontmatter keys) is what
requirement 1 chose the flow form over. A v0.2 `generated` mapping never makes
a file ingest-owned (V-A3), with the v0.1 `generated: true` control alongside
so the refusal is attributable to `generated` and not to an unreadable
manifest reference -- and end to end, such a file is refused with
`collision_unstamped` rather than overwritten. Today's fail-safe becomes a
documented guarantee.

Step 2. Door C against the five OKF section 14 consumer MUST NOTs plus the
section 5.2 bare-`verified` mapping. Its tolerance is structural rather than
lenient: the door writes the guard's bytes verbatim and never parses the
sender's frontmatter, so a D1b reader that starts judging shape at this door
breaks these cases -- which is when we want to hear about it.

Step 3. No profile hard-codes an upstream version (V4/V-A5), walked
recursively through dataclasses and collections and falsified against a
planted literal; the key name is what a profile pins, and the root-frontmatter
policy accepts any value. The `okf_version` value is catalog's (E1), so a
constant here would be both a decision we do not own and the thing that would
have to be chased on every upstream release.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
 2026-07-26 20:06:08 +02:00
Kjell Tore Guttormsen
2d0328aaa7 docs(okf-v0.2): requirement 1 re-sized -- the emitter does not change at all 
U4 escalated tag requirement 1: real v0.2 frontmatter is block YAML, and "emit
accepts a block-list value" had been scoped against a list of strings when
upstream's canonical sources is a list of five-key mappings. Re-sized against
the code rather than against its tests, and the answer runs the other way.

Measured: emit (profiles.py:197) takes Mapping[str, str] and renders one
"key: value" line per key through _render (:211), verbatim but for
source_query's whitespace collapse; _is_legal_value (:97) admits a string or a
non-empty list of strings, never mappings; the only production call sites are
materialize.py:166 and inbox.py:129, both passing dict[str, str].

The earlier "discharged" note read the validator correctly and drew too much
from it. But the gap it left is not U4's either: A-E4 owes at least a resource,
not upstream's five keys, and a manifest source has no author, no last_modified,
and no bundle-internal resource in upstream's sense. A field with no reader is
not written. So the question was never how large a YAML emitter we need -- it
was which form sources takes.

Chosen: sources: [{ id: <id>, resource: <ref> }], the same inline flow form A-E3
already chose for generated. It keeps parse_frontmatter line-oriented, which
matters because a block list pollutes that parser with false keys (- id,
resource) -- the failure mode documented for po-claude -- and _is_ingest_owned,
the one piece of this tag we called risky, reads THROUGH that parser. The block
form would force the collision gate and the parser to be hardened in the same
session. It also satisfies commons' section 5 single-line MUST, so the DEFAULT
seam U4 opened stays closed, and section 11 asks for parseable YAML, not block
YAML. The deviation from upstream's emitted shape is real and is recorded as a
decision, like U1, not as compliance we may claim.

Consequences: requirement 1 is smaller than both earlier sizings -- zero emitter
work -- and D1a drops off the critical path to v0.5.0a1. A-E4 is corrected from
"block list" to the flow form before the pilot receives it; a test spec that
outlives its decision is the A-E6 defect class.

Swept the same defect class while here: the TDD order and three other passages
wrote okf_version: "0.2" QUOTED, contradicting A-E6, which is precisely the form
catalog measured as exit 1. All five now unquoted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
 2026-07-26 19:57:29 +02:00
Kjell Tore Guttormsen
bec5608106 docs(okf-v0.2): V6 -- Door C imports the pointer, never the code it points at 
Measured, after asserting the opposite to the security repo an hour earlier: no
door in this library can create, import, or persist an executable file. Door C
skips every suffix that is not .md (importer.py:226-231); Door B has no extractor
for .py in either the core or the optional set (extract.py:113,131-142); Door A
writes only ingest-*.md. We told the guard a third-party attester .py could reach
a consumer's tree through our door. It cannot. Reasoned, not measured, corrected
the same hour.

The real exposure is the inverse. Door C writes a merged concept verbatim,
frontmatter included, and under v0.2 that frontmatter can carry
executor.resource and attester.resource. So we import the reference to executable
code while never importing the code. Either the pointer dangles -- noisy, safe --
or it resolves to something else already in, or later added to, the destination
tree, and the imported concept references code it never shipped. That second one
is a substituted reference, not a broken one, and it looks valid.

Also recorded: the guard is a PERSIST gate, and executable code carries its risk
at EXECUTION. A .py entirely safe to store may be dangerous to run, so no persist
gate covers the class section 10 introduces, and the execution boundary has no
owner in this stack -- upstream deferred the attester ABI and sandboxing, and we
declared runtime out of scope on that same deferral. Written down because "the
guard covers it" is a reasonable conclusion to draw and a hard one to catch later.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
 2026-07-26 19:41:20 +02:00
Kjell Tore Guttormsen
b541e794a9 docs(okf-v0.2): GA is not done when the code ships, but when consumers know 
Operator directive: when the v0.2 implementation is ready, every OKF-consuming
repo is told through the coord mailbox what to do or test. That was a step title
in the runbook and is now a table with content -- nine repos, each with the
action asked of them and, more importantly, what happens if they do nothing.

For most the honest answer is "nothing changes", and saying that plainly is more
useful than an open invitation to review. That is the black-box commitment made
checkable rather than asserted.

Two new GA exit criteria: every repo in the table has been sent its message and
the table was re-derived rather than reused; and the security repo's section 10
boundary questions are either answered or their being open is recorded as an
accepted, stated limit. The second matters because Door C is the path an
external bundle's executable attester would take into a consumer's tree.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
 2026-07-26 19:34:18 +02:00
Kjell Tore Guttormsen
99491fa277 docs(okf-v0.2): D6's checklist item points at the runbook 
The verification criterion said the release checklist carries an
upstream-version re-check. It now names the procedure that check runs, which is
the part that makes it more than a reminder.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
 2026-07-26 19:19:46 +02:00
Kjell Tore Guttormsen
499e96ed92 docs: every upstream OKF release runs a runbook, and this repo is a black box 
The standing always-latest policy had no procedure, and v0.1 -> v0.2 showed what
that costs: the spec was read, the shipped example bundles were not, and five
findings a spec reading had settled wrongly surfaced only after someone asked
whether we had studied the examples. One of them re-sized the release itself.

docs/upstream-okf-upgrade-runbook.md is that procedure. Every step names the
concrete failure it prevents, and each is a failure that actually happened:

- Pin before reading. Two repos read main independently, which felt like two
  confirmations and was one unstable reference read twice -- and SPEC.md was
  edited after the migration commit, so neither alone was the whole answer.
- Enumerate the whole okf/ tree. bundles/ held the real examples and was on
  nobody's list until the tree was listed.
- Read the shipped examples, not only the normative text. A directory named
  samples/ turned out to be a runner config, not samples of the format. And
  markdown-converting fetch tooling strips exactly the byte-level properties
  (BOM, trailing newline, quoting) that break downstream gates.
- Name fixtures by repo AND path. Two repos had examples/ingest-golden-file/
  holding different fixtures with zero content overlap, and a pilot baseline was
  agreed by a name that pointed at two files.

The black-box commitment is now an invariant in CLAUDE.md rather than an
emergent property: a consumer's cost for an upstream release is a re-run.
Additive profiles, byte-stable existing profiles, keyword-only new parameters so
positional call sites survive, no churn in consumer fixtures. The boundary is
stated rather than glossed -- shape changes are absorbed, upstream changes to
consumer-authored content are not, and those get a measured exposure report per
consumer instead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
 2026-07-26 19:18:43 +02:00
Kjell Tore Guttormsen
4669f32b47 docs(okf-v0.2): the pilot baseline was ambiguous and we asserted it was shared 
Two corrections to claims this repo made, both measured rather than argued.

1. `examples/ingest-golden-file/` names TWO different fixtures. Ours holds
   ingest-orders.md + ingest-products.md; po-claude's holds ingest-costs.md +
   ingest-edge.md; the manifests differ at line 3 char 41; content-file overlap
   is zero and the only shared filename, index.md, is byte-different. We agreed
   a pilot baseline by name, and the name pointed at two files. A-E1 is the stop
   condition, so a mismatched baseline would have surfaced as a profile defect.
   The pilot runs THEIR fixture -- A-E1 asserts byte-identity with what their
   pinned version produces, which is a claim about their baseline, not ours --
   and the report must state repo and path.

2. "The shared fixture, ratified by two independent implementations" was false.
   We took it from an incoming message and passed it on the same day without
   measuring, while holding a standing rule that incoming messages are premises,
   not facts, and applying that rule only to spec readings. Commons caught it.

The :29 argument in V1 survives in commons' stronger form: the clause requires
reproducing "the shared golden extractions", the definite article presumes one
set where there are two, so it has no referent today. (b) therefore does not
break a shared fixture we have -- it cements that one can never arise, since a
producer-bearing field inside the byte-compared prefix makes convergence
impossible by construction. The :280 half stays observable and fires on our next
release regardless.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
 2026-07-26 19:13:38 +02:00
Kjell Tore Guttormsen
9bffc92abc docs(okf-v0.2): read upstream's reference bundles, not just its spec 
V-A1 is discharged: everything is now read at pinned commit 3fcbb9f, not off
main. v0.2 itself landed in 780fe9d; SPEC.md was edited after it, so "the v0.2
commit" and "the current spec text" are different objects.

Session 19 read the spec. It never read the four bundles the tooling migrated
to v0.2, and reading them corrected things a spec reading had settled wrongly.
okf/samples/ga4_merch_store is not a bundle -- it is a runner config that
generates one, so it says nothing about output shape.

- U1 No reference bundle declares okf_version. All four root index.md files
  have no frontmatter at all. A-E6 is therefore a stricter posture than
  upstream's own implementation -- defensible, but a decision rather than an
  assumption. po-claude's "deviation" #2 was conformance. catalog's gate reports
  MISSING on every official Google bundle.
- U2 Root index.md is the ONLY index permitted to carry frontmatter (12).
  Constrains the fixture and the index policy.
- U3 `generated` is used for HUMAN-authored content upstream, so it cannot be a
  machine-generated or ownership predicate. Empirical confirmation of commons'
  axis finding, stronger than their spec-text argument.
- U4 Real v0.2 frontmatter is multi-line block YAML: flow sequences, block lists
  of flow mappings, nested mappings, block lists of multi-key mappings, booleans,
  dates. Requirement 1 was scoped against a list of strings; the canonical shape
  is a list of five-key mappings. The one-session sizing must be redone.
- U5 7's actor convention gives `<producer>/<version>` for tools, and upstream
  uses it. That is real evidence for (b), against the (d) we recommended to
  commons today. Sent to them as a correction.
- U6 Conformance and the consumer-tolerance MUSTs quoted verbatim, including the
  bare-`verified`-as-one-element-list requirement on D1's reader.
- U7 An undocumented `not:` family observed in a bundle, unresolved, recorded so
  it is not mistaken for checked.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
 2026-07-26 19:09:03 +02:00
Kjell Tore Guttormsen
d65231c3ce docs(okf-v0.2): V1 moves off our own recommendation, on commons' finding 
We carried (b) -- generated.by naming this library and its version. Commons
found it collides with ingest-spec's conformance clause: :29 requires a
conformant implementation to reproduce the shared goldens byte for byte, and
`generated` sits inside the mandatory ordered prefix. A producer's name and
version inside a shared fixture means no sibling implementation can reproduce
it, and Golden regression fires on every version bump with no contract change.

They measured it as normative-only for them, since examples/ingest-golden-*
does not exist in commons. Measured here, it is observable today: four fixtures
carry `generated:` on line 8 inside the byte-compared prefix, and
ingest-golden-file is both the shared fixture against the sibling
implementation and the baseline po-claude runs the pilot against.

So we back (d) -- `process:<fixed id>` per v0.2 section 7. Second reason,
visible only from this side: it keeps _is_ingest_owned an equality check.
Under (b) the collision gate must either prefix-match and ignore the version --
parsing a value we had just made structured -- or stop recognizing files an
earlier version wrote, which fails safe but breaks re-running a manifest across
a version bump.

The cost of (d) is recorded rather than minimized: producer identity leaves the
bundle, and a log does not travel through Door C. The decision stays commons'
to ratify.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
 2026-07-26 17:09:20 +02:00
Kjell Tore Guttormsen
f7385fafdb docs(okf-v0.2): the pilot answered before the tag existed, and it changed the tag 
All three pilots accepted. Their measurements land in the plan now rather than
as failed runs later, which is the entire return on asking before building.

Six corrections, each measured by the consumer rather than reasoned by us:

- A-E6 specified `okf_version: "0.2"` quoted. Catalog measured the quoted form
  failing their shape regex with exit 1, and a UTF-8 BOM making the marker
  invisible while still exiting 0. Both now stated as raw-byte expectations.
- A-E2 said the diff lies only *inside* the frontmatter block. No bundle whose
  root index has no frontmatter block can satisfy that together with A-E6.
- B-E3 covers one of catalog's three gates; the other two cannot see our
  fixture. Recorded so the report is not read as broader than it is.
- Our claim that no gate keys on the version value was wrong as formulated:
  check-okf-parity.mjs puts the raw value in its signature. It compares without
  judging -- verdict-invariant, not value-blind.
- C-E1's ground truth is 524 non-reserved documents, not 522; 522 was correct
  at d2c12d2.
- `materialize_bundle` gains keyword-only `profile=DEFAULT` on po-claude's
  request. The signature already has a `*`, so their three-positional call path
  stays source-compatible and additivity becomes a property of the signature
  instead of something a consumer measures.

Also recorded: po-claude closed their index-parser defect themselves (f41264f),
so do not design around it; the assumption under it -- "a generated index has no
frontmatter" -- is not peculiar to them. Dropping the superseded `timestamp`
draws one catalog warning per concept file, decided after the pilot, not before.

The estimate's open caveat is discharged by reading profiles.py:197: emit is one
formatting path, the validator already admits list values, so only the emitter
side is missing and the sizing stands.

CLAUDE.md gains a standing constraint: consumer content stays at form level in
public files. The wiki's bundle is private pending an Anthropic ToS assessment,
their report goes back through coord, and a push is not reversible.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
 2026-07-26 17:03:05 +02:00
Kjell Tore Guttormsen
95b1ef0bcd docs(okf-v0.2): the pilot tag needs far less than the deliverable list implied 
Walking each pilot test back to what it actually exercises produces a result
the deliverable ordering had obscured: none of the three needs D1's
structured reader.

Test A emits v0.2 -- `generated` is an inline flow mapping, a formatted
string, and `sources` needs the emitter to accept a block-list value. Neither
requires reading one back. Test B runs catalog's gate on Test A's output.
Test C validates the wiki's v0.1-shaped documents against a v0.2 variant
profile; their documents carry no v0.2 families, so block-list reading is
never reached.

The structured reader exists to consume third-party v0.2 bundles -- D3, Door
C -- and no pilot repo sends us one. So D1 splits: the emitter half is in the
tag, and the reader half moves behind the pilot where the feedback can inform
it. Two consequences worth stating plainly.

V2 is off the critical path. It chooses the reader's design, and the reader
now lands after the pilot; the tag is not waiting on a decision.

The tag is roughly a session of work, not a phase: an emitter change, a
profile constant following the existing STRICT_V1 pattern, a profile-aware
ownership predicate, `sources` derivation, one generated fixture.

Risk concentrates rather than spreads. `_is_ingest_owned` is the pre-mutation
collision gate, so a defect there is expensive and quiet -- its
characterization test is written before it is touched. The emitter change
runs through DEFAULT's path, so V-A6 is proven by the golden suite rather
than assumed.

The sizing itself carries a caveat in the doc: it treats
FrontmatterSchema.emit as a single formatting function, inferred from its
tests rather than read. That is a premise like any other and gets verified
before the estimate is relied on.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
 2026-07-26 16:56:45 +02:00
Kjell Tore Guttormsen
5de590ebdf docs(okf-v0.2): the pilot tests state their expected results up front 
The pilot invitations asked for feedback without specifying what to run or
what we expect to see. That returns a description, not a measurement: only a
stated expectation lets someone else's run falsify our model instead of
merely confirming it.

Each of the three tests now carries a numbered procedure, numbered expected
outcomes, and an explicit list of what would surprise us. Naming the
surprises is the load-bearing part -- a consumer who sees something odd but
passing otherwise has no reason to mention it.

Test A (producer, portfolio-optimiser-claude) runs one already-working
manifest twice with the same explicit ingested_at, once under DEFAULT as
baseline and once under OKF_V0_2, and diffs. A-E1 is the stop condition: if
the DEFAULT run is not byte-identical to their current pinned output, we have
broken a v0.1 consumer and the pilot ends there. The named surprises include
`at` differing from the ingested_at they passed, which would mean a
wall-clock crept in, and a collision refusal on a file that IS theirs, which
is the inverse of the fail-safe we predicted -- we expect foreign files to be
refused, so a false refusal of their own is the defect.

Test B (gate, catalog) runs their unmodified gate on the v0.2 fixture and on
a 0.1 control. B-E3 -- that no gate other than the version gate behaves
differently between the two -- is what converts our reading of their form
regex into a measurement. A membership list or equality comparison anywhere
in their chain is exactly what this is paid for.

Test C (expressiveness, wiki) is run by us, read-only at a recorded commit,
and the report goes to them. What we ask them to check is the part we cannot
measure from outside: whether a field we called an optional addition is in
fact load-bearing in their pipeline, and whether "no change required" holds
operationally rather than only formally.

Feedback gains a per-expectation verdict line so three independent runs are
comparable and a disagreement is located rather than merely reported. The
request also says outright that a wrong expectation is a better result than a
clean run, since a clean run only confirms what we already believed.

Inputs come from this repo at the pre-release tag; nothing is transported
through the mailbox except the specification.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
 2026-07-26 14:10:40 +02:00
Kjell Tore Guttormsen
a27017e635 docs(okf-v0.2): ship to a pilot set before general availability 
Operator directive 2026-07-26: consumers are better served by getting the
latest version early and reporting back than by us holding it until we judge
it finished.

This closes a real gap. Every test in the plan -- including V-A8 against
upstream's reference implementation -- asks whether output is conformant.
None asks whether it is usable: whether a bundle is awkward to construct,
whether a rejection message is actionable, whether the profile can express
what a consumer's actual data needs. Only real data surfaces that.

Pilot set is three repos, one axis each, chosen for signal:
portfolio-optimiser-claude for the producer path (one real manifest run),
catalog for gate acceptance (their gate on our fixture, measured rather than
inferred from the regex), and claude-code-llm-wiki for expressiveness across
522 real documents.

The load-bearing design detail is that most of the pilot is read-only on our
side: we run the v0.2 reader over real consumer bundles and send the report.
That needs no adoption, no writes into their trees (O2 holds), and no change
to a contract their operator ratified. Only the producer axis asks a
consumer to do anything, and it asks for one run.

Excluded with reasons rather than silently: okr (Node side not yet lifted),
linkedin-studio (v0.2's provenance families would put implicit pressure on
the ingest/published carve-out we agreed not to normalize), commons (they
are deciding V1 -- a participant, not a test site).

Shipping a provisional surface without owing stability rests on three rules,
not on saying "provisional": OKF_LATEST does not point at v0.2 until GA, so
flipping it is the GA event rather than a merge side effect; the docstring
and CHANGELOG name the pilot repos; and breaking changes during the pilot
get no deprecation cycle. Stating that last one up front is what buys the
freedom to act on feedback -- discovering it later is what turns a pilot
into a de-facto release.

Feedback is requested in five named parts, because unstructured reports are
not comparable across three repos. The one that matters most is "what was
awkward but worked": workarounds are the highest-value signal and never
appear as a failure.

GA exit criteria are testable, and carry one honest limit: a three-repo
pilot exercises only what those three use, so `sources` with usage_window,
multi-verifier `verified`, and Attested Computation will likely go
unexercised. Those stay marked provisional at GA instead of being silently
promoted -- claiming otherwise would be the same unearned-claim pattern that
"conform first, claim after" exists to prevent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
 2026-07-26 13:50:28 +02:00
Kjell Tore Guttormsen
cb23de4700 docs(okf-v0.2): supporting the latest OKF version becomes standing policy 
Operator directive 2026-07-26: the library always supports the current
latest version of Google OKF. v0.2 shipped 2026-07-25, so v0.2 support is
committed work rather than something a consumer has to request. This
overrides the previous default answer to open question V3 ("no until a
named consumer asks"), which is kept in the plan marked superseded so the
override reads as deliberate.

Support is additive: a new profile, never a migration of the existing two.
That single design choice is what makes an always-latest policy sustainable,
and it resolves the tension the directive would otherwise create with three
constraints that do not yield to it:

- DEFAULT states commons' ingest-spec section 5 layer, so its `generated`
  shape is commons' call. Under the additive design this stops blocking us,
  which takes commons off the critical path.
- STRICT_V1 mirrors the proving consumer's ratified contract; changing
  another repo's contract from here would violate O2.
- v0.2 defers the attestation receipt and verdict wire formats upstream, so
  the format is supportable and the unspecified runtime is not. It re-enters
  scope when upstream specifies it.

This is also the first time the phase-3 profile abstraction is forced by
something outside this repo rather than by a second consumer, which is the
better test of whether the seam was cut in the right place.

Deliverables D1-D6 replace the earlier decision-round framing: a frontmatter
model that can carry block lists (`sources`, multi-verifier `verified`), an
OKF_V0_2 profile plus an OKF_LATEST alias whose moving-target tradeoff is
documented rather than hidden, Door C conformance against the consumer
tolerance rules, `Attested Computation` round-trip, v0.2 golden fixtures, and
a release-checklist re-check so the standing policy cannot decay silently.

Two new assumptions carry the weight. V-A7 forbids any profile from emitting
`timestamp` together with a malformed `generated`, since that combination
would have neither a valid `generated.at` nor an eligible section 13.1
fallback. V-A8 validates our own v0.2 fixture against upstream's reference
implementation, because every other test in the suite only asks whether we
agree with ourselves.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
 2026-07-26 13:41:30 +02:00
Kjell Tore Guttormsen
ca453b2895 docs(okf-v0.2): the alignment track lands before the contract freezes 
Everything built so far targets OKF v0.1. Upstream published v0.2 on
2026-07-25, so the plan records how the library relates to it and, more
importantly, who owns each decision.

Read from the spec itself rather than secondhand, which corrected two
readings that a summary had gotten wrong:

- STRICT_V1's `timestamp` is NOT a defect. Section 13.1 grants consumers a
  documented fallback to legacy `timestamp` precisely when `generated` is
  absent, and STRICT_V1 emits no `generated`. Nothing is asked of the wiki.
- The one measured shape problem is DEFAULT's `generated: "true"`, because
  v0.2 requires `generated.by` within `generated`. That key was not reserved
  in v0.1, so it was legal when written; v0.2 claimed the name. DEFAULT
  states commons' ingest-spec section 5 layer, so the fix is commons' call
  and is raised there as open question V1 rather than patched locally.

Two findings shrink the work. The canonical form for `generated` and a
single `verified` is an inline flow mapping, which the existing scalar
parser already round-trips as an opaque string, so block-list support is
only needed for `sources` and multi-verifier `verified` -- and only if a
named consumer asks. And the collision degrades safely: `_is_ingest_owned`
returns False for a v0.2 mapping, so a foreign concept is refused rather
than overwritten.

The track sits between Phases 3 and 4 because Phase 4 freezes the
cross-runtime contract. Freezing a v0.1 shape into two runtimes would let
the shared fixture suite certify the drift instead of catching it.

Self-imposed rule, since the spec does not require it: conform first, claim
after. Declaring `okf_version: "0.2"` is a MAY with no conformance
checkpoint, so claiming it early would be permitted -- and would be the same
class of true-sounding misleading claim as reporting a 0.2.0 measurement
under a 0.3.1 heading.

Also moves the guard 0.3.1 measurement procedure out of session state and
into execution-order.md, where it belongs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
 2026-07-26 11:42:37 +02:00
Kjell Tore Guttormsen
c5e6f083e6 docs(phase-4): record linkedin-studio's two named requirements before the freeze 
They named both on a live channel today, having withdrawn the half of their
original question that turned out not to bind them. Recorded here rather than in
a reply so the scope freeze has to answer them explicitly.

`write_index` ownership was genuinely ambiguous in this plan — they read
deliverable 4's "write" and asked which line owns it. It is deliverable 3, and
the plan now says so.

Their vocabulary requirement is already the Python half's shape:
`TypePolicy(allowed=None)` is the default and rewrites nothing, so there is no
DEFAULT_TYPE to snap to. The reference checker's collapse is a known deviation to
resolve with the convention owner under step 1, not a behaviour to lift.
 2026-07-25 20:34:54 +02:00
Kjell Tore Guttormsen
9436427520 feat(phase-3): the index policy becomes configurable, with the reader that judges it 
`IndexPolicy` grew five judging fields and `IndexPolicy.violations`, closing the
gap a test has been pinning since `848e395`. `STRICT_V1.index` is now the proving
consumer's shape rather than DEFAULT's.

The design was settled by a conflict rather than by a preference. The convention
owner reported that an index is an authored count of a directory's children and
that a validator enumerating the directory has implemented the wrong contract.
Checked against the proving consumer before adopting it, the two turn out to be
directly opposed: gate BUNDLE_INDEX_COMPLETE (validate.py:1081-1120, ERROR) builds
its expected set by enumerating the directory and demands an exact bidirectional
match, and bundle.py:498-567 writes every index from a tree walk. Neither is
incoherent inside its own spec, so authored-versus-derived is a policy field in
both directions and a library invariant in neither.

Consequences encoded rather than documented: nothing here enumerates a directory
— the caller supplies the listing, `violations` refuses one when the profile's
index is authored and refuses to run without one when it is derived, so code
written to the wrong contract fails at the call instead of passing every test one
would think to write. Root and nested indexes are asymmetric (confirmed in both
consumers, different key sets). A per-entry description is template-level, so
`render_link` refuses both a missing description and an unwanted one.

DEFAULT keeps upstream's root-only index and judges nothing, for the same measured
reason it carries no required frontmatter key set: upstream binds `index.md` to
the bundle root alone, so a judging default would condemn conforming bundles.

25 new tests, 493 green. C1 re-proven: `git diff --stat examples/` empty.
 2026-07-25 20:30:54 +02:00
Kjell Tore Guttormsen
848e3953fc feat(phase-3): STRICT_V1's type enum and frontmatter schema, with the reader 
Phase 3 step 3, frontmatter half. FrontmatterSchema grows the four judging
fields the proving consumer's contract needs — required, allowed, nullable,
key_pattern — and `violations` is what reads them, so no field lands as an
untested claim.

STRICT_V1 requires the eight keys their operator ratified on 2026-07-25 inside
the eleven-key allowlist that doubles as emission order, both derived from one
tuple so they cannot drift; `summary` is the single nullable key; a value is a
string scalar or a NON-EMPTY LIST OF STRINGS, because their Release pages carry
block lists and a scalars-only schema would reject pages they ship today.

Measured, not assumed: DEFAULT must NOT grow a required set. Door A emits seven
keys, Door B six, and Door C writes an imported concept verbatim with whatever
the sender wrote, so any required set on DEFAULT would declare invalid the
bundles this library itself produces. The plan's other cross-profile direction —
a wiki bundle rejected under DEFAULT — therefore has to come from index and path
shape, not from frontmatter. Recorded here because the plan implied otherwise.

Violations are reported, not raised, like TypeRejection, and sorted by
(key, code): a report that depended on mapping order would not be reproducible.
An unsatisfiable schema — a required or nullable key outside its own allowlist —
fails at construction.

Two boundaries kept explicit. `violations` judges an already-parsed mapping;
parsing strict frontmatter needs a parser this library does not have, which is
the same reason Door C writes verbatim. And STRICT_V1's index policy is still
DEFAULT's, pinned by a test as a known gap: per-directory indexes and an entry
description have no field on IndexPolicy yet, and nothing consumes a profile's
index policy today.

C1 re-proven: 443 existing tests unmodified and green (468 total), and
`git diff --stat examples/` empty.
 2026-07-25 15:29:26 +02:00
Kjell Tore Guttormsen
cf1f7984c4 docs(phase-3): the four open questions are answered; required set is eight 
The proving consumer answered Q1-Q4 on 2026-07-25, each measured or read out
of their code that day. No row in the mapping moved. Three of the four answers
carry a sharper reason than this table drafted, and one reverses a correction
this table made a day earlier.

Q2 is the substantive change. This table had corrected the plan's eight-key
required set down to four, against the consumer's REQUIRED_KEYS constant.
Their operator ratified eight: the constant is their emit-path minimum, not
their contract, and all eight keys are present on 522/522 of their documents.
STRICT_V1 is therefore deliberately stricter than the consumer's own
validator; if they ever ship a class lacking source_sha they bump the profile
rather than bend the bundle, and they say so before shipping it. The eleven-key
allowlist is unchanged and still a different thing from the required set:
version/date are layer-specific and summary is the one nullable key, so all
three stay optional-when-absent.

Q1, Q3 and Q4 keep their Consumer/Guard assignments, now with the reasons that
justify them: gate 5 escalates on run state (disposition == verbatim) rather
than document class, so a profile structurally cannot see it; gate 4's
changelog rule is a vendor slug list that is not scoped to concepts/ at all;
gates 19 and 22 need external state, one of them by construction never
blocking. Assumption C2 is confirmed at 25 gates and the index shape is
confirmed non-negotiable.

Both reviews - the consumer's and the operator's - are now in, so the port may
begin. The Guard column is unaffected by all of it.
 2026-07-25 15:21:59 +02:00
Kjell Tore Guttormsen
6f42c10608 refactor(phase-3): the bundle contract becomes a profile object 
Phase 3 step 1. What Phases 1 and 2 hard-coded about a valid bundle now
lives on one frozen `BundleProfile`, and `DEFAULT` states exactly the
ingest-spec v1 + Phase 2 contract. Nothing observable changes: the 425
existing tests are unmodified and green, and `git diff --stat examples/`
is empty, which is assumption C1's whole proof. Both oracles are real
rather than nominal — the golden suite compares `read_bytes()`, and Door
B pins its frontmatter block as an exact string.

Moved onto the profile, each one previously a constant with a reader:
the index name and its managed-link shape (template plus pattern, kept
honest by a round-trip test), the three filename namespaces
(`ingest-`/`inbox-`/`import-` and `.md`), the reserved `verdict` layer
(duplicated in `manifest` and `inbox` before this), the frontmatter key
order, and the `source_query` whitespace collapse.

Two details are worth naming. The DEFAULT key order spans both doors:
Door A's seven keys and Door B's six are subsequences of one nine-key
order, so a single canonical order reproduces both doors byte-for-byte.
And emission follows commons decision D1 — ordered prefix, then any
remaining keys sorted — which is the mechanism the proving consumer's
hash registry needs; under DEFAULT the tail is always empty.

`TypePolicy` refuses the reserved layer at CONSTRUCTION (assumption C3),
so a profile admitting `verdict` cannot be built, let alone passed to a
door. It reports refusals rather than raising them, because Door A
refuses with `ManifestError` and Door B with `MaterializationError` for
the same type — the wording and the stable code come from the policy,
the exception class stays each door's own.

Deliberately NOT moved, each for a stated reason: the required and
allowlisted key sets the proving consumer needs (no code reads them
until the STRICT_V1 validator exists, and a field nothing reads is a
claim nothing tests), the id grammar (a pattern the slugger derives a
separator class from, not a flat name), `NAME_MAX_BYTES` (a filesystem
fact, not a contract choice), and every disposition/origin/channel
vocabulary (guard territory, always). The profile is also not exported
from the package root yet — the optional `profile` argument on the
flows is step 3's plumbing change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
 2026-07-25 15:09:14 +02:00
Kjell Tore Guttormsen
08ca68fee2 docs(phase-3): fold in the proving consumer's review of the split table 
Their answer crossed with the draft rather than responding to it, so the
two readings of the same validator are independent. They agree on every
substantive point -- four required keys, three reserved basenames,
summary as the only nullable key, the index structure, the closed sets --
and on the column assignments. That is corroboration, not confirmation
bias, and it is worth more than either reading alone.

Four refinements folded in:

- Emission order is an ordered prefix followed by remaining keys SORTED,
  not just an ordered list. This is the commons D1 outcome and it is what
  keeps regeneration byte-identical; normalising key order any other way
  breaks their BUNDLE_HASH_REGISTRY gate, which is why D1 mattered.
- Values may be a non-empty LIST of strings, not only scalars. Same block
  -list shape Door C already refuses to round-trip, which is why imported
  concepts are written verbatim -- a scalars-only parser would destroy
  their Release pages, and a scalars-only profile would reject them.
- Non-required allowlisted keys are validated against closed sets WHEN
  PRESENT, not required to be present.
- Internal-link resolution generalises only because path_policy owns the
  layer layout; noted on the gate row as their caveat, accepted.

Gate 5's row now distinguishes the Guard COLUMN (functionality that is
the guard's and is never reimplemented) from the call SITE (theirs, in
their acquisition stage) -- the one place our wording and theirs differed.
 2026-07-25 12:34:03 +02:00
Kjell Tore Guttormsen
678f9b2b07 docs: two corrections from the convention owner and the spec owner 
Both arrived within minutes of the v0.4.0 announcement and both change
what we would otherwise have built or claimed.

CATALOG -- a profile field we must NOT add. Their steg 5 measured bundle
placement across four consumers: exactly one installs a bundle anywhere,
the other three take the bundle directory as a per-run argument with no
default, no discovery and no $HOME expansion. Placement is unmandated, so
BundleProfile gets no root-location or install-path field, "no
installation root" is a conforming consumer shape, and write policy must
not be coupled to placement. Recorded in the split table, which also now
spells the proving consumer's layer prefixes bundle-relative
(concepts/, releases/) rather than repeating the bundle/ prefix their own
repository layout carries -- that prefix was placement leaking into a
contract.

COMMONS -- our "Fixed" framing outruns their frozen text. Removing every
ingest-stamped file is verbatim what ingest-spec v1 section 5 mandates:
v1 assumes one manifest per bundle and defers the multi-manifest case as
a named extension point. So v0.4.0 implements that extension point rather
than fixing a defect against the spec. The mechanic survives ratification
either way -- stem matching follows from their own {stem}@{h} stamp
definition, since {h} changes on every content edit -- but the prose is
queued for amendment and is not ratified, so the changelog now says so
instead of claiming conformance it does not have.
 2026-07-25 12:32:21 +02:00
Kjell Tore Guttormsen
d8e466791c docs(phase-3): split table over all 25 gates, and two corrected premises 
Phase 3 starts with a written mapping of the proving consumer's gates to
exactly one of Profile / Guard / Consumer, reviewed before any porting.
Drafted here against claude-code-llm-wiki's validate.py and bundle.py at
their 9ee5a8e, which is their single gate entrypoint, so the enumeration
is exhaustive against that commit and no other.

Reading the validator corrected two things this plan asserted before it
had been read, both of which would have produced a wrong STRICT_V1:

- Required frontmatter keys are FOUR (type, title, description,
  timestamp), not the eight the plan listed. Those eight were a
  truncation of FRONTMATTER_KEY_ORDER, which is an eleven-key allowlist
  doubling as canonical emission order. frontmatter_schema therefore
  needs three fields -- required, allowlisted, ordered -- not one
  required/optional pair.
- Reserved files are THREE basenames, not two: README.md is reserved
  like index.md and log.md but in a third way, present and exempt from
  the document gates while still content-scanned. A two-valued
  required/required-absent policy cannot express it.

Two of the 25 gates bundle checks from different columns
(LLM_OUTPUT_LINT, BUNDLE_URL_ALLOWLIST). They are decomposed per check
rather than assigned wholesale to a dominant column, which would have
moved the residue to the wrong side of the boundary.

Also recorded: index policy is the widest gap between the two profiles
(per-bundle vs per-directory, two link formats, append-if-absent vs
exact match, root index with or without frontmatter), and C4 is
confirmed with the operator -- profiles in code, no config-file loading
in v1.

Four open questions are addressed to the proving consumer in the
artifact; no porting until they and the table come back reviewed.
 2026-07-25 12:30:51 +02:00
Kjell Tore Guttormsen
4ea00a902f docs(install): correct how the guard reaches a consumer, measured both ways 
The v0.4.0 release commit claimed the [tool.uv.sources] entry "never
reaches consumers". Measured against the pushed tag in clean
environments, that is wrong in one direction and understated in the
other:

- uv, installing this package from git, DOES pick the guard up from the
  tag automatically. uv reads pyproject.toml when it builds from the
  source tree, so the source applies to the consumer too, not just to
  development here.
- pip does not read it, and fails outright: "No matching distribution
  found for llm-ingestion-guard<0.3,>=0.2". The guard must be installed
  from its own tag FIRST. Verified that the guard-first order then
  installs cleanly and that the shipped adapter runs against the real
  guard in that environment.

What did hold is the part that matters for the pin: the built wheel
carries `Requires-Dist: llm-ingestion-guard<0.3,>=0.2`, so the range is
still the declared dependency and the git URL is still a channel.

README now states the pip order as the default instruction, with the uv
shortcut beside it, because a consumer hitting the resolution error would
otherwise read it as a broken release.
 2026-07-25 12:24:54 +02:00
33 changed files with 5450 additions and 127 deletions
Download patch file Download diff file Expand all files Collapse all files

69
CHANGELOG.md
View file

@ -5,6 +5,63 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [0.5.0a1] — 2026-07-31
**Pre-release. PROVISIONAL surface.** Shipped to a named pilot set —
`portfolio-optimiser-claude`, the plugin marketplace catalog, and
`claude-code-llm-wiki` — so that real data can find what fixtures cannot. The
v0.2 surface may change on their feedback **without a deprecation cycle**.
Saying so up front is what buys the freedom to act on the feedback; discovering
it later is what would make a pilot a de-facto release. Do not pin this tag
outside the pilot set.
Support for a new upstream version is **additive — a new profile, never a
migration**. `OKF_LATEST` still points at `DEFAULT`; flipping it is the GA
event, not a side effect of this tag.
### Added
- **`OKF_V0_2` profile.** Google OKF v0.2 as a profile alongside `DEFAULT` and
`STRICT_V1`. It closes nothing: §14 forbids a conformant consumer to reject on
an unknown `type` value or on unknown additional keys, so `type` is the only
required key. It NAMES `okf_version` but never carries its value — that value
tracks the upstream Google version and belongs to catalog (decision E1), so
the caller supplies it via `materialize_bundle(..., root_frontmatter_values=…)`
and the profile fixes only the key and its position.
- **`root_frontmatter_values`** on `materialize_bundle`, keyword-only: the
mechanism behind "a profile names a key, a caller owns its value".
- **`sources` emitted as an inline flow sequence**, populated from the manifest.
- **The v0.2 golden fixture**, compared byte-for-byte like the others.
### Changed
- **`profile` on `materialize_bundle` is keyword-only**, so existing positional
call sites stay source-compatible.
- **`_is_ingest_owned` is profile-aware.** Ownership is a policy on the profile,
not a literal; recognition stays one-way.
### Fixed
- **`root_frontmatter` permitted a key without requiring it — the two had been
the same profile field.** Upstream says MAY where the field said MUST, so a
root index that legally omitted an optional key was rejected. `root_frontmatter`
now permits and orders; the new `root_frontmatter_required` requires.
`STRICT_V1` keeps requiring its three (unchanged for its consumer), `OKF_V0_2`
requires none, `DEFAULT` is untouched. Measured against the eight root indexes
available locally: 7 of 8 failed before, 0 of 8 after.
### Notes
- **`DEFAULT` and `STRICT_V1` are byte-stable across this release.** The emit
path is byte-identical; the golden suite is the check.
- **Known, deliberately unfixed:** `TypePolicy`'s closed branch does not strip
quotes from a declared type. All three call sites pass `DEFAULT.types`, and
both `DEFAULT` and `OKF_V0_2` set `allowed=None`, so the branch is
unreachable in shipped code — it fires only for a caller constructing
`STRICT_V1` and calling `rejection` directly. Fixing it would repair a write
path that never sees quotes, and would fix the meaning of a quote character
without a value model that can express one, in a line-oriented format.
## [0.4.0] — 2026-07-25
Phase 2. The bundle inbox (Door B) and external-bundle import (Door C) ship,
@ -90,6 +147,17 @@ and with them this library's first — and only ever — runtime dependency.
- Phase 2's binary extraction is **not** in this release: the `[extract]`
extra is declared but empty, and `pdf`/`docx`/`xlsx` therefore fail fast.
That is the one outstanding item from the phase, and it ships separately.
- **The ownership change above is an extension point, not a spec fix.** Filed
under "Fixed" because it stops silent data loss, but the spec owner
(`portfolio-optimiser-commons`) has since pointed out that removing *every*
ingest-stamped file is verbatim what ingest-spec v1 §5 mandates: v1 assumes
one manifest per bundle and defers multiple manifests feeding one bundle as
a named extension point. So this release implements that extension point and
outruns the frozen text rather than conforming to it. The mechanic itself is
not at risk — matching by manifest stem follows from the spec's own
`{stem}@{h}` stamp definition, since `{h}` changes on every content edit —
but the surrounding prose is queued for amendment in commons and is not
ratified. Treat multi-manifest bundles as ahead of the spec until it is.
- Exception `__cause__` preservation is now pinned by a conformance suite, one
test per fail-fast wrap site, alongside the existing `.code` suite.
@ -176,6 +244,7 @@ Phase 1 (Door A) implemented against the normative `ingest-spec.md` owned by
- The Door A public surface: `materialize_bundle` plus the typed error hierarchy
rooted in `IngestError`.
[0.5.0a1]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.4.0...v0.5.0a1
[0.4.0]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.3.2...v0.4.0
[0.3.2]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.3.1...v0.3.2
[0.3.1]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.3.0...v0.3.1

75
CLAUDE.md
View file

@ -49,8 +49,11 @@ are the library baseline. First consumer: `portfolio-optimiser-claude`.
2. **Phase 2 — Doors B/C (Python).** Bundle inbox and external-bundle
import, guard-gated.
3. **Phase 3 — Configurable bundle contract.** Types, layers, frontmatter
sets, and reserved-file policy become config instead of constants;
proving consumer is `claude-code-llm-wiki` (`strict-v1` profile).
sets, index shape, and reserved-file policy become config instead of
constants; proving consumer is `claude-code-llm-wiki` (`strict-v1`
profile). Two consumers hold opposite postures on whether an index is
authored or directory-derived, so neither is a library invariant and
nothing here enumerates a directory unless the profile says derived.
4. **Phase 4 — Node half (`node/`).** Zero-dependency Node/ESM package
(importable *and* CLI-invokable, vendorable per plugin — matching the
marketplace precedent) for the second-brain world: bundle check, index
@ -58,8 +61,68 @@ are the library baseline. First consumer: `portfolio-optimiser-claude`.
(docx/pdf/eml/html → md). Covers okr, linkedin-studio, ms-ai-architect,
and the marketplace catalog.
5. **Phase 5 — MCP as a way to populate a bundle.** Committed by the operator
2026-07-27; the design is open, the scope is not. Plan:
`docs/plan/mcp-bundle-population.md`. The fork that must be settled before
any code — whether we are the MCP **server** (an agent calls our doors as
tools) or an MCP **client** (a manifest source type that pulls from someone
else's server) — decides both the owner and the dependency question, and the
two are not variants of one feature.
The two halves share the OKF contract and fixture suite, **not code**.
### Upstream version policy (standing, non-negotiable)
**The library always supports the current latest version of Google OKF.** Set by
the operator 2026-07-26. Phases 1–3 were built against v0.1; v0.2 shipped
2026-07-25, so v0.2 support is committed work — not contingent on a consumer
asking for it. Plan: `docs/plan/okf-v0.2-alignment.md`.
Support is **additive, expressed as a new profile**, never a migration of
existing ones. This is what makes the policy sustainable instead of a recurring
crisis, and it is bounded by three facts that do not yield to it:
- `DEFAULT` states commons' ingest-spec §5 layer — its `generated` shape is
commons' call, raised there, never patched locally.
- `STRICT_V1` mirrors the proving consumer's ratified contract — changing another
repo's contract from here violates O2.
- `okf_version`'s *value* belongs to catalog (decision E1).
**Rollout is pilot-first.** A new upstream version reaches a small pilot set on a
pre-release tag and is revised on their feedback before general availability —
consumers testing real data find what fixtures cannot. `OKF_LATEST` means the
latest version supported as *stable*, so flipping that alias is the GA event, not
a merge side effect.
Two invariants fall out: no profile hard-codes an upstream version, and no bundle
declares a version its shape has not earned. The first has a mechanism, not just
an intention: **a profile names a key, a caller owns its value.** `okf_version`
is declared through `materialize_bundle(..., root_frontmatter_values=...)`
because its value tracks the upstream Google version and belongs to catalog
(decision E1) — a constant here would claim a decision we do not own, and would
be the one thing to chase on every upstream release. Where upstream itself defers a
contract — v0.2's attestation receipt and verdict wire formats — the format is
supported and the unspecified runtime is not; it re-enters scope when upstream
specifies it. Because "always latest" decays silently, the release checklist
carries an upstream-version re-check.
**Every upstream release runs `docs/upstream-okf-upgrade-runbook.md`.** Pin the
commit, enumerate the whole `okf/` tree, **read the shipped example bundles and not
only `SPEC.md`**, classify the diff, measure our exposure and each consumer's, plan
additively, pilot before GA, then inform every OKF-consuming repo. The runbook is
not optional and not a summary of good intentions: each step names the concrete
failure it prevents, and all of them are failures that happened during v0.1 → v0.2.
**This repo is a black box for its consumers.** The target cost of an upstream
release to a consuming repo is **a re-run, nothing more**: support is additive
(a new profile, never a migration), existing profiles stay byte-stable, new public
parameters are keyword-only with defaults so positional call sites stay
source-compatible, and consumer golden fixtures must not churn. The boundary is
stated every time rather than glossed — the library absorbs *shape* changes, not
upstream changes to content a consumer authored (v0.2's `timestamp` and
`# Citations` supersessions). For that class the deliverable is a measured exposure
report per consumer, sent before they ask.
Phase 4 preconditions (coordination, not unilateral moves):
- Lifts okr's reference implementations (`okf-check.mjs`, `okf-index.mjs`,
innboks libs) in agreement with okr and the marketplace catalog; the
@ -119,6 +182,14 @@ and fixtures, never code.
and `docs/oppstartsprompt.md` are LOCAL-ONLY (gitignored) — never commit
session state or internal briefs. No secrets, sober English prose, no
marketing language.
- **Consumer content stays at form level in public files.** Some consumers we
read are private (`claude-code-llm-wiki` is, pending an Anthropic ToS
assessment). Key names, counts, gate names, profile fields and contract shapes
are publishable; page bodies, full title or path lists from a consumer's
bundle, and Anthropic-derived prose are not. Findings about a private
consumer's data go back to them through coord, never as a file here. This
costs nothing — every question this library asks of a consumer is about shapes
and key sets — and it is not reversible once pushed.
- After `git commit`: push to Forgejo (`git push origin`) immediately.
Never GitHub.

69
README.md
View file

@ -93,9 +93,9 @@ verification criteria:
[plan](docs/plan/phase-1-door-a.md).
2. Bundle inbox and external-bundle import (Python), guard-gated —
[plan](docs/plan/phase-2-doors-b-c.md).
3. Configurable bundle contract (types, layers, frontmatter sets, and
reserved-file policy as configuration), enabling stricter bundle
profiles such as `strict-v1` —
3. Configurable bundle contract (types, layers, frontmatter sets, index
shape, and reserved-file policy as configuration), enabling stricter
bundle profiles such as `strict-v1` —
[plan](docs/plan/phase-3-configurable-contract.md).
4. A `node/` half: a zero-dependency Node/ESM package (importable and
CLI-invokable, vendored per consumer) providing bundle checking, index
@ -104,6 +104,48 @@ verification criteria:
contract and fixture suite, not code —
[plan](docs/plan/phase-4-node-half.md).
## Upstream OKF versions
The library targets the current latest version of Google's OKF. Support is
**additive** — a new upstream version arrives as a new profile, never as a
migration of an existing one — so upgrading the library does not change the
bytes an existing profile emits.
| Profile | Contract | Status |
|---|---|---|
| `DEFAULT` | commons' ingest-spec §5 layer (OKF v0.1 semantics) | stable |
| `STRICT_V1` | a consumer's ratified v0.1 contract | stable |
| `OKF_V0_2` | OKF v0.2 | **provisional**, pre-release only |
| `OKF_LATEST` | alias for the latest version supported as *stable* | currently `DEFAULT` |
`OKF_V0_2` ships first as a pre-release to a named pilot set and may change on
their feedback without a deprecation cycle. Pin the versioned constant rather
than `OKF_LATEST` unless you have explicitly opted into tracking; `OKF_LATEST`
moves at general availability, which is a deliberate release event rather than
a side effect of an upgrade.
Selecting a profile is keyword-only, so existing call sites are unaffected:
```python
materialize_bundle(manifest, bundle_dir, ingested_at, profile=OKF_V0_2)
```
A bundle may declare the version it targets. OKF v0.2 §12 makes this a MAY, and
puts the declaration in the bundle-root `index.md`'s frontmatter block. The
profile names the key; the **caller supplies the value**, because that value
tracks the upstream version and is not this library's to decide:
```python
materialize_bundle(
manifest, bundle_dir, ingested_at,
profile=OKF_V0_2,
root_frontmatter_values={"okf_version": "0.2"},
)
```
Omit the argument and no frontmatter block is written. Offering a key the
profile does not name is refused before anything is written to disk.
## Non-goals
- Verdict/feedback machinery from the method specification (stays in the
@ -115,18 +157,29 @@ verification criteria:
## Requirements
Python 3.10+, and exactly one runtime dependency — the security boundary,
`llm-ingestion-guard>=0.2,<0.3`. Everything else is stdlib. Until that
package is published to an index, install it from its tag:
`llm-ingestion-guard>=0.2,<0.3`. Everything else is stdlib.
That guard is not on a package index yet, so **with pip, install it first** —
otherwise installing this package fails with `No matching distribution found
for llm-ingestion-guard`:
```
pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v0.2.0"
pip install "llm-ingestion-okf @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v0.4.0"
```
With uv, one command is enough — `uv pip install "llm-ingestion-okf @ git+…@v0.4.0"`
resolves the guard from the tag on its own, because uv reads the
`[tool.uv.sources]` entry in this project's `pyproject.toml` when it builds
from the source tree. Both paths were measured on 2026-07-25.
A git URL is a PEP 508 direct reference and pins one exact tag, so it is an
install-time *channel*, not the pin: the range above stays the declared
dependency and resolves normally once the package index exists. The optional
`[extract]` extra (pdf/docx/xlsx parsers) is not populated yet. The planned
Node half targets Node/ESM with zero npm dependencies.
dependency — the built wheel carries `Requires-Dist:
llm-ingestion-guard<0.3,>=0.2` — and resolves normally once the package index
exists. The optional `[extract]` extra (pdf/docx/xlsx parsers) is not
populated yet. The planned Node half targets Node/ESM with zero npm
dependencies.
## License

312
docs/phase-3-split-table.md Normal file
View file

@ -0,0 +1,312 @@
# Phase 3 split table — the proving consumer's 25 gates
Status: **reviewed by the proving consumer and by the operator (2026-07-25);
the port may begin.** Phase 3's plan requires this mapping before any porting.
Nothing in the consumer's repository is modified by this repository.
The proving consumer (`claude-code-llm-wiki`) answered the same questions from
their own code on 2026-07-25, crossing with this draft rather than responding
to it. Two independent readings of the same source agreed on every substantive
point: the required-key set as their code spells it, three reserved basenames,
`summary` as the only nullable key, the index structure, and the closed sets.
Their answer added four refinements, each folded in below and marked. Their
column assignments match this table's, with one wording difference resolved in
gate 5's row. A later round answered the four open questions outright — see
"Answered by the proving consumer" — and moved the required set from what their
code enforces to what their operator ratified.
Read from the consumer's working tree on 2026-07-25:
`tools/wiki_ingest/validate.py` and `tools/wiki_ingest/bundle.py` at commit
`9ee5a8e` (repository HEAD `20bcdd6`). Their `validate.py` is the single
entrypoint for every per-run gate, so this table is exhaustive over the gate
set as it stands at that commit — assumption C2 is satisfied only as long as
that remains true, and a gate added there after this date is not in this table.
## The three columns
| Column | Meaning | Goes where |
|---|---|---|
| Profile | Structural bundle contract: types, frontmatter keys, paths, reserved files, index shape | This phase, `BundleProfile` |
| Guard | Content safety: carriers, secrets, injection, active content | `llm-ingestion-guard` calls — never reimplemented here |
| Consumer | Domain logic: source pipelines, LLM contract, promotion, run accounting | Stays in the consumer's repository |
## Stage A — detection/acquisition
| # | Gate | Default class | Column | Note |
|---|---|---|---|---|
| 1 | `INGEST_LEDGER_DIVERGENCE` | ALERT (halts) | Consumer | Upstream re-poll reconciliation; no bundle structure involved. |
| 2 | `INGEST_CHANGELOG_PARSE` | ERROR | Consumer | Source-format parsing for one specific upstream. |
| 3 | `INGEST_EXPECTED_SET` | ALERT | Consumer | npm/changelog reconciliation against known deltas. |
| 4 | `INGEST_DOCS_CORPUS_SANITY` | ERROR | Consumer | Corpus size/shape thresholds for one named corpus. Its second half — "a concept must not derive from the changelog page" — is a layer-provenance rule expressed as a URL test; see open question Q3. |
| 5 | `INGEST_INPUT_INJECTION_SCAN` | WARN (ERROR when verbatim) | Guard | Carrier scan plus semantic lexicon on fetched input. Guard *column* means the functionality is the guard's and is never reimplemented — not that the call site moves here. The consumer keeps the call, as they state, since it sits in their acquisition stage. The disposition-aware escalation is Consumer policy over a Guard verdict; see Q1. |
## Stage B — LLM-output contract
| # | Gate | Default class | Column | Note |
|---|---|---|---|---|
| 6 | `LLM_STOP_REASON` | ERROR | Consumer | |
| 7 | `LLM_SCHEMA` | ERROR | Consumer | Frozen run-record schema plus summary length caps. |
| 8 | `LLM_OUTPUT_LINT` | ERROR | mixed | Decomposed below. |
| 9 | `LLM_GROUNDING_ANCHOR` | ERROR | Consumer | Anchor resolution against their concepts layer. |
| 10 | `LLM_SOURCE_TRACE` | ERROR | Consumer | Verbatim-quote and dropped-citation detectors. |
| 11 | `LLM_SEAM_LINKS` | ERROR | Consumer | Proposed links restricted to existing concept IDs. |
| 12 | `LLM_BUDGET` | ERROR | Consumer | Per-run call/token/cost caps. |
## Stage C — bundle conformance and publish
| # | Gate | Default class | Column | Note |
|---|---|---|---|---|
| 13 | `BUNDLE_FRONTMATTER_PARSEABLE` | ERROR | Profile | Every non-reserved `.md` opens with a complete fence. |
| 14 | `BUNDLE_TYPE_ENUM` | ERROR | Profile | Closed enum `Concept`/`Guide`/`Reference`/`Release`. |
| 15 | `BUNDLE_RESERVED_FILES` | ERROR | Profile | `log.md` required-absent; `index.md` structure. Three reserved basenames, not two — see "What STRICT_V1 must express". |
| 16 | `BUNDLE_YAML_STRICT` | ERROR | Profile | Key allowlist, required keys, and the StrictYAML scalar subset. |
| 17 | `BUNDLE_LAYER_PATH_TYPE` | ERROR | Profile | Path prefix ⇄ `layer` ⇄ `source_tier` ⇄ `type` agreement. |
| 18 | `BUNDLE_LINKS_INTERNAL` | ERROR | Profile | Every relative cross-link resolves inside the bundle. Consumer's caveat, accepted: link resolution assumes their layer layout, so it generalises only because `path_policy` owns the layout too. |
| 19 | `BUNDLE_LINKS_EXTERNAL` | WARN (never blocks) | Consumer | Liveness polling; needs network state this library does not collect. |
| 20 | `BUNDLE_URL_ALLOWLIST` | ERROR | mixed | Decomposed below. |
| 21 | `BUNDLE_INDEX_COMPLETE` | ERROR | Profile | Per-directory index exactness plus root-index metadata keys. |
| 22 | `BUNDLE_HASH_REGISTRY` | ERROR | Consumer | Their page-hash snapshotter must cover the concepts layer exactly. |
| 23 | `BUNDLE_SANITIZATION` | ERROR | Guard | Carrier-freedom of published text. |
| 24 | `BUNDLE_BLAST_RADIUS` | ERROR | Consumer | Per-version diff caps from their run manifest. |
| 25 | `BUNDLE_SECRET_SCAN` | ERROR | Guard | Credential patterns over bundle text and run artifacts. |
## The two mixed gates, decomposed
The plan asks for exactly one column per gate. Two gates do not honour that as
written, because each bundles checks belonging to different columns. Splitting
them is the honest mapping; the alternative — assigning the whole gate to its
dominant column — would quietly move the residue to the wrong side.
**Gate 8, `LLM_OUTPUT_LINT`** runs `_lint_text` over enrichment summaries and
verbatim release bodies:
| Check | Column |
|---|---|
| Off-allowlist URL | Consumer (their editorial pinning; see gate 20) |
| HTML tag in prose | Guard — active content is squarely the guard's territory at v0.3.0 |
| base64-like blob | Guard |
| Credential pattern | Guard |
| Consumer-directed imperative | Guard — this is the injection posture, expressed as their house rule |
| Injection lexicon and hex blob | Guard |
**Gate 20, `BUNDLE_URL_ALLOWLIST`** bundles two unrelated rules:
| Check | Column |
|---|---|
| URL host/prefix on the pinned allowlist | Consumer |
| Domain is new versus the previous bundle state | Consumer (drift detection over their own retained state) |
Neither half is Profile: a URL allowlist is editorial policy about which
sources a bundle may cite, not a statement about bundle structure. It is listed
as mixed only because the checks have different natures, and both land in the
same column — noted rather than merged, so a later reviewer can see it was
considered.
## What STRICT_V1 must express
The phase-3 plan sketched this profile before the validator was read. Two of
its claims did not survive contact and are corrected here; the plan file
carries the same corrections. One of the two was then corrected a second time
by the consumer's operator — the sequence is recorded rather than smoothed
over, because the end state is deliberately neither what the plan drafted nor
what their code says.
1. **Required keys are eight — settled by the consumer, not read off their
code.** Their `REQUIRED_KEYS` constant is four (`type`, `title`,
`description`, `timestamp`), and this table first corrected the plan down to
it. Asked whether four was the contract or an artefact of unfinished triage
(Q2), they measured the whole bundle and their operator ratified eight:
`type`, `title`, `description`, `timestamp`, `layer`, `source_tier`,
`source_url`, `source_sha`, present on 522/522 documents. The four-key
constant is their *emit-path* minimum, not their contract.
Two consequences are carried deliberately:
- **STRICT_V1 is stricter than the consumer's own validator.** Should they
ever ship a document class that legitimately lacks `source_sha` — a
hand-written Guide, say — this profile rejects what their validator
passes. The agreed resolution is to bump the profile, not to bend the
bundle, and they send the requirement before shipping such a class rather
than after.
- **The allowlist remains a different thing from the required set.**
`FRONTMATTER_KEY_ORDER` is eleven keys — the eight above plus `version`,
`date`, `summary` — and doubles as canonical emission order. `version` and
`date` are layer-specific (present on 351/522) and `summary` on 349/522,
so all three stay optional-when-absent and are validated against closed
sets **only when present**; a profile that required them would reject
pages the consumer ships today. So `frontmatter_schema` needs three
fields, not one: required, allowlisted, and ordered.
2. **Three reserved basenames, not two.** `index.md`, `log.md`, and
`README.md`. README is bundle documentation, reserved like `index.md`: it
carries no frontmatter, is skipped by the per-document gates, is never
listed in a generated index, and is still subject to the link, URL,
sanitization and secret gates. A two-value reserved-file policy
(required-present / required-absent) cannot express it; the third value is
"present-and-exempt-from-document-gates".
Beyond the frontmatter set, `BundleProfile` must carry:
- **Type policy:** closed enum, four values.
- **Nullable keys:** `summary` alone may be `null` — the single deliberate
escape from strings-only frontmatter.
- **Value shapes:** a string scalar, **or a non-empty list of strings**. This
is the same block-list shape Door C already refuses to round-trip through
this library's line-oriented parser, which is why an imported concept is
written verbatim. A profile that assumes scalars-only rejects the consumer's
own `Release` pages, and a parser that assumes scalars-only would silently
destroy them.
- **Key-name grammar:** `^[a-z_]+$`.
- **Scalar encoding:** the StrictYAML subset (JSON-double-quoted scalars), so
a regeneration over the same data is byte-identical.
- **Emission order — carry this over intact:** the ordered prefix above,
followed by any remaining keys **sorted**. This is the door-A model settled
by commons decision D1, and it is what makes regeneration byte-identical.
A profile that normalises key order any other way breaks the consumer's
`BUNDLE_HASH_REGISTRY` gate, which is what made that decision matter.
- **Layer/path policy:** `concepts/` ⇄ layer `concepts`, source_tier
`official-docs`, any type except `Release`; `releases/` ⇄ layer `releases`,
source_tier `changelog-derived`, type `Release` exactly. Their constants
spell these `bundle/concepts` and `bundle/releases` because their validator
walks a repository whose bundle lives in `bundle/`. That prefix is where
*they* keep the bundle, not part of the contract — see "No placement field"
below.
- **Index policy**, which differs from this library's default in four ways:
one index per *directory* rather than one per bundle; entries formatted
`* [Title](link) - description` rather than `- [label](target)`; index
contents must match the directory exactly, both directions; and the root
index carries frontmatter pinning `okf_version`, `bundle_profile`,
`okf_spec_commit`.
## No placement field — a field the profile must NOT grow
The catalog reported on 2026-07-25 (their commit `3ccb5f6`,
`docs/okf-second-brain/spec.md` §1 and §8) that bundle placement is
**unmandated**, measured against four consumers rather than taken on report:
exactly one installs a bundle anywhere; for the other three the bundle
directory is a caller-supplied argument per run, with no default, no discovery
and no `$HOME` expansion. Discovery, where it happens, keys on the §3 root
marker (`index.md` plus `okf_version`), never on a path.
Three consequences for `BundleProfile`, recorded here because this phase's
non-goal is precisely "do not make the profile too narrow for the Node-world
conventions in phase 4":
- No root-location or install-path field. There is no path form to express.
- "No installation root" is a conforming consumer shape, so a profile that
assumes every consumer has one would be too narrow in exactly the warned way.
- Installing and writing are separate axes: a consumer may write into a bundle
it never placed, so write policy must not be tied to a placement concept.
Every path in this document is therefore bundle-relative, and the bundle
directory stays what it is today in both doors — an argument.
The index-policy divergence is the largest single item. Under the default
profile the observable output must stay byte-identical (assumption C1, proven
by the golden suite), so `index_policy` has to express both shapes rather than
generalising one into the other.
## Answered by the proving consumer
All four questions were answered on 2026-07-25, each measured or read out of
their code that day rather than recalled. **No row in the mapping moved**; the
reasons under three of the four are sharper than the ones this table drafted.
- **Q1 — gate tiers stay Consumer/Guard.** This repository's position was
accepted, and the escalation turns out not to be a document-class rule at
all: gate 5 escalates WARN to ERROR when the per-version *record* carries
`disposition == "verbatim"` (`validate.py:528-540`), i.e. when the LLM
rewrite layer was skipped and the input lint is the only filter left. That is
run state, not bundle shape — a profile could not see it even if it wanted
to. A profile describes structure; it does not set security tiers.
- **Q2 — eight, ratified by their operator.** Folded into "What STRICT_V1 must
express" above, together with the consequence both sides accepted.
- **Q3 — Consumer, for a sharper reason than "URL predicate".** The check is
`is_changelog_url()` (`docs_corpus.py:181-184`): it lowercases the last URL
segment and tests membership in `("changelog", "release-notes")` — a
vendor-specific slug list for one upstream. Two details worth carrying: the
gate is **not** scoped to `concepts/` at all (it walks every non-reserved
document across both layers, `validate.py:480-492`, despite a rationale
string that says "concepts layer"), and it spares their own release pages
only because their `source_url` ends in `CHANGELOG.md`, whose last segment is
`changelog.md` rather than `changelog`. That is a coupling, not a design. A
structural profile could express "layer X may not carry source_tier Y"; it
could not express this.
- **Q4 — both stay Consumer.** Gate 19 sits in their `NEVER_BLOCKING_GATES`
(`validate.py:86-88`) by construction, precisely because liveness is a flake
magnet: a profile that owned it would inherit a gate that can never block.
Gate 22 is their provenance model, the one commons decision D1 settled in
their favour, and it is why the ordered-prefix emission matters. The profile
gets no notion of external state from either.
Two confirmations arrived unasked and are recorded because later work leans on
them:
- **Assumption C2 holds.** `validate.py` is still the single entrypoint for
every per-run gate, and the registry at lines 50-78 is the whole set — 5
`INGEST_*`, 7 `LLM_*`, 13 `BUNDLE_*` = 25. A change reaches this repository
as a message before it can be read out of their code.
- **The index shape is not negotiable downward**, which makes "express both
shapes rather than generalise one into the other" the right call: one index
per directory, exactly one `# ` heading, entries `* [Title](link) -
description` and nothing else, directory and index matching in both
directions, and a root index carrying `okf_version` / `bundle_profile` /
`okf_spec_commit` frontmatter in that key order.
## The index policy, and the conflict that shaped it
The convention owner reported on 2026-07-25 that the second-brain spec's §3
mandates an `index.md` at every directory level, that `IndexPolicy` as it then
stood could not express that, and — the load-bearing part — that an index is an
**authored** count of a directory's children and never a filesystem lookup. On
their reading, an index reader or validator that enumerates a directory to
build or check an index has implemented the wrong contract, with a silent
failure mode: code written to the wrong reading passes every test one would
think to write.
That was checked against the proving consumer before it was adopted, and the
two consumers turn out to be **directly opposed on exactly this point**:
- `validate.py:1081-1120`, gate `BUNDLE_INDEX_COMPLETE` (severity ERROR),
builds its expected set by enumerating the directory and fails unless the
index matches it exactly in both directions.
- `bundle.py:498-567` writes every index from what a tree walk finds, and its
own docstring calls `index.md` machine-generated.
So one consumer mandates the derivation the other calls the wrong contract.
Neither is incoherent inside its own spec, and this repository is not the venue
to adjudicate between them. What it settles is narrower and sufficient:
**authored-versus-derived cannot be a library invariant in either direction.**
It is a policy field (`entries_match_directory`), as are per-directory scope,
the heading requirement, whether prose is admitted, and the root-index key set.
Three consequences that outlive this phase:
1. **Nothing in this library enumerates a directory.** The caller supplies the
listing; `IndexPolicy.violations` refuses one when the profile's index is
authored, and refuses to run without one when it is derived. The
convention owner's construction rule is enforced at the call site rather
than documented and hoped for.
2. **Root and nested indexes are asymmetric**, confirmed independently in both
consumers: the root carries frontmatter (`okf_version`/`bundle_profile`/
`okf_spec_commit` for the wiki, `okf_version`/`okf_layout` for the catalog)
and nested indexes carry none. Different key sets, same shape — which is
why it is a profile field and not a constant.
3. **A per-entry description is template-level, not a separate requirement.**
The wiki requires `* [Title](link) - description`; the catalog requires
prose in the index and no per-entry description. A profile that wants none
simply does not name `{description}` in its template.
`DEFAULT` keeps upstream's root-only index and judges nothing, for the same
measured reason it carries no required frontmatter key set: upstream OKF binds
`index.md` to the bundle root alone, so a default demanding one per level would
declare upstream-conforming bundles invalid.
## What this table does not do
It does not port anything. Per the plan, porting starts only after this
mapping is reviewed, and the Guard column must contain no rows implemented in
this repository — the phase-2 grep gate over guard functionality still applies
unchanged.
Both reviews are in as of 2026-07-25 — the proving consumer's (above) and the
operator's — so the port may begin. The Guard-column constraint is unaffected
by either.

76
docs/plan/execution-order.md
View file

@ -8,12 +8,14 @@ own doc:
- `docs/plan/phase-1-door-a.md` — DONE, shipped at v0.3.2.
- `docs/plan/phase-2-doors-b-c.md`
- `docs/plan/phase-3-configurable-contract.md`
- `docs/plan/okf-v0.2-alignment.md` — a policy track, not a phase; slots between
Phases 3 and 4 for the reason given below.
- `docs/plan/phase-4-node-half.md`
Headline order: **Stage 0 (guard gate) → Phase 2 → Phase 3 → Phase 4 code**,
with Phase 2 split guard-independent-first and Phase 4's *coordination* started
in parallel from the beginning. The quality argument for this order is below —
it is not merely the roadmap numbering.
Headline order: **Stage 0 (guard gate) → Phase 2 → Phase 3 → OKF v0.2 alignment
decisions → Phase 4 code**, with Phase 2 split guard-independent-first and
Phase 4's *coordination* started in parallel from the beginning. The quality
argument for this order is below — it is not merely the roadmap numbering.
## Stage 0 — Guard readiness (gate; do first, cheap)
@ -33,6 +35,36 @@ persist-gate design must know now, not at the integration step.
Outcome recorded in STATE before Phase 2 code starts.
### Stage 0 revisited — measuring guard 0.3.1 before widening the pin
The pin held is `>=0.2,<0.3`, which can only ever resolve `v0.2.0`. The guard repo
gates its own 1.0 on our fixture suite being measured against `v0.3.1`, so this
measurement is owed outward and is that repo's only named blocker. Order is
*measure first, widen after*: widening the range before measuring would ship a
window admitting a version the fixtures never ran against.
Procedure — do not vary it, each step exists because of a specific failure mode:
1. Scratch venv **outside** `.venv`: install the package `--no-deps`, then the
guard from the `v0.3.1` tag. Do not touch `pyproject.toml`.
2. **Assert the resolved version** via `importlib.metadata` *before* running
anything, and carry it in the reported result. An unmodified tree resolves
`0.2.0` and runs green — a true claim about 0.2.0 published under a heading
that says 0.3.1. This step is what makes the result honest.
3. Run `tests/test_guard_adapter.py` (19 tests, real scanner) and the full suite.
Watch percent-escapes specifically: the slugger builds filenames from titles,
and the guard has recorded that as its most likely false positive. Report raw
URLs, not counts.
4. **Send the result whichever way it goes**, and state explicitly that 19
fixture tests are not a corpus — their gate is "does 0.3.1 change our
verdicts", not "how often does the false positive fire".
5. Green → one release: `>=0.2,<0.4`, uv tag `v0.3.1`, CLAUDE.md and README
updated together. Red → no bump, and the finding goes back to the guard.
Until this lands, consumers of v0.4.0 resolve guard 0.2.0 (`pyproject.toml:25`).
That is a silent under-defend: no build fails, so nothing surfaces it. The
release in step 5 is what closes it.
## Phase 2 — Doors B/C (split: guard-independent first)
Internal order follows the phase-2 doc's TDD steps, deliberately:
@ -68,6 +100,38 @@ policy differs per consumer and becomes configurable in Phase 3." This is where
the contract becomes explicit and frozen — the precondition for a clean Node
port.
## OKF v0.2 support — between 3 and 4, and it is real code
Everything through Phase 3 targets OKF v0.1. Upstream published v0.2 on
2026-07-25, and the standing operator policy is that the library always supports
the current latest OKF version (see `CLAUDE.md`). So this is a committed
implementation track, not a decision round:
`docs/plan/okf-v0.2-alignment.md`.
**Why it must land before Phase 4 code.** Phase 4 ports a *frozen, explicit*
contract and uses the shared fixture suite as the cross-runtime conformance
oracle. Freezing that contract before v0.2 is supported means freezing a v0.1
shape into two runtimes instead of one, and the fixture suite would then certify
the drift rather than catch it. Worse, the Node half would need the same v0.2
work done twice, in two languages, against a contract that had already been
declared final. The expensive version of this mistake is discovered after the
Node half exists.
**Why it is not a phase of its own.** It adds no new door and no new stage — it
adds a *profile*, which is precisely what Phase 3 built the seam for. Support is
additive: `DEFAULT` (commons') and `STRICT_V1` (the wiki's) keep their contracts
untouched, and v0.2 arrives as a third profile plus an `OKF_LATEST` alias. This
is the first time the Phase 3 abstraction is forced by something outside this
repo rather than by a second consumer, which is the better test of it.
**Order inside the track.** Characterization and audit first — they add tests and
change no behavior. Then the frontmatter model that can carry block lists
(`sources`, multi-verifier `verified`), then the profile, then the golden
fixtures that make the support claim testable. The `okf_version: 0.2`
declaration comes last — **unquoted**, per A-E6 — once the shape has earned it. Commons' answer on
`DEFAULT`'s `generated` runs in parallel and no longer blocks: the additive
design took it off the critical path.
## Phase 4 — Node half (two clocks)
**Coordination clock — start early, parallel to Stages 0–3, but only the
@ -117,6 +181,8 @@ These are the objective checks that a stage is done and the next may begin:
second-brain spec).
4. **Phase 4 code start:** each shape-independent coordination step has a
recorded sign-off; the contract-shape agreement is signed off *after* the
Phase 3 split-table, not before.
Phase 3 split-table, not before; and the OKF v0.2 open questions V1–V5 are
each answered or explicitly deferred *in writing*, so the frozen contract
states its relation to upstream on purpose rather than by omission.
5. **Throughout:** `mypy --strict src/`, `ruff check .`, `ruff format --check .`
clean; boundary grep-gate empty.

107
docs/plan/mcp-bundle-population.md Normal file
View file

@ -0,0 +1,107 @@
# Phase 5 — MCP as a way to populate a bundle
**Status: scoped, not designed.** Committed by the operator 2026-07-27. This
document records the commitment, the fork it turns on, and the constraints any
design has to satisfy. It deliberately does not choose an implementation:
the fork below decides who owns the work, and choosing it wrongly is expensive
in a way writing it later is not.
## The fork — settle this first
"Support MCP to populate a bundle" names two different pieces of work. They
share a protocol and nothing else.
**(A) We are the MCP server.** An agent connects to us and calls tools —
`add_concept`, `list_bundle`, `generate_index` — and the bundle fills up through
our doors. The natural home is **Door B**, which already answers "here is
content, make it OKF": MCP would be a protocol surface over an existing door
rather than a new ingestion path.
**(B) We are an MCP client.** A manifest names an MCP server as a source, the
same way it names `file`/`sql`/`http` today, and Door A pulls concepts from it.
**Default reading, for confirmation rather than assumption:** the phrase "to
populate a bundle" points at (A). Recorded as a reading, not a decision — the
operator has not been asked which one, and the two have different owners.
## Why the fork decides the owner
(B) adds a **source type**, and the source type set is normative in commons'
`ingest-spec.md`. This repo implements that spec and does not edit it. So (B)
cannot start here: it starts as a request to commons, and it lands here only
after they have specified it. (A) adds no source type and no spec surface — the
concepts arrive through Door B exactly as dropped files do — so (A) is ours to
design.
That asymmetry is the whole reason this document exists before any code.
## Constraints any design inherits
None of these are negotiable by this feature, and each one has already refused
something:
1. **Exactly one runtime dependency, ever** (`llm-ingestion-guard`), enforced by
a packaging test. An MCP SDK would be the second. Either the protocol is
hand-rolled over stdlib JSON-RPC on stdio, or it lives behind an optional
extra the way binary parsers do in `[extract]` — never in the core install.
2. **No model calls anywhere in the run path.** MCP is model-facing, and that is
fine in exactly one direction: under (A) the model is the *caller* and we
stay deterministic tools. A design where this library calls a model to decide
what to write is out of scope, and would be a different product.
3. **The persist gate still applies.** Content arriving over MCP is untrusted by
construction. It goes through Door B's injected guard gate like every other
dropped file; MCP is a transport, never a reason to bypass a gate. Security
stays the guard's, as always.
4. **Network access is an explicit per-run opt-in.** A server listening on a
socket, or a client dialing one, is network. The existing fail-fast refusal
is the baseline, not something to reinvent.
5. **Determinism.** `ingested_at` is a required argument with no wall-clock
default. A protocol handler does not get to invent one because a call
arrived "now".
## Open questions
- **Q1 — (A) or (B), or both in order?** Blocks everything else.
- **Q2 — If (A): what is the tool surface?** Door B's public surface is
`process_inbox` over a directory. An MCP tool taking one concept at a time is
a different granularity, and the collision gate and index generation are
currently per-run rather than per-concept. This is the real design work.
**Prior art, measured 2026-07-27 at the pinned upstream commit.** Upstream has
already built this surface in ADK-specific form, and it is worth reading before
we draw our own. `sources/base.py` (38 lines, stdlib only) defines a `Source`
ABC; `tools/source_tools.py` wraps its methods as model-callable tools, which
`agent.py` registers. Three things in it are design signals rather than
incidental:
- **`list_concepts` and `read_concept` are abstract; `sample_rows` is optional
with a `None` default; `find` ships a default implementation.** So the
surface's obligations are *enumerate* and *read*, and sampling is advisory.
A tool surface that made sampling mandatory would be claiming more than the
problem needs.
- **It is per-CONCEPT** (`read_concept(ref)`), which is the granularity this
question is about. That an independent implementation landed on per-concept
is evidence that the per-run shape — Door B's collision gate and index
generation — is the side that has to give, not the tool surface.
- **What not to copy.** `read_concept` returns `dict[str, Any]`, untyped
because a model interprets it; we render bytes and cannot. And `find` is a
linear scan over `list_concepts()`, which assumes total enumeration is cheap
— colliding directly with this library's rule that nothing enumerates unless
a profile says derived.
Read, not vendored. The point is the shape of the question they answered.
- **Q3 — Transport.** stdio is the vendorable, zero-dependency answer and
matches how the marketplace consumes things. HTTP would be a second network
surface with its own gate question.
- **Q4 — Which half?** Phase 4 puts a Node/ESM half next to the Python one. MCP
servers in the plugin world are overwhelmingly Node. If (A) lands in Python
only, the second-brain consumers cannot use it; if it lands in both, the
"share contract and fixtures, not code" rule applies to the tool surface too.
- **Q5 — Does an MCP-populated bundle differ observably** from one Door B wrote
from files? It must not, or the transport has leaked into the artifact.
## Non-goals
- Reimplementing any security check the guard owns.
- Calling a model from this library, under any transport.
- Editing commons' source-type set locally to make (B) possible sooner.

1364
docs/plan/okf-v0.2-alignment.md Normal file
View file

File diff suppressed because it is too large Load diff

52
docs/plan/phase-3-configurable-contract.md
View file

@ -18,21 +18,40 @@ the golden suite is the regression harness for that claim.
profiles are constructed in code by the consumer):
- `type_policy`: closed enum vs. open set; the reserved `verdict` exclusion
stays unconditional (it is a spec invariant, not profile config).
- `frontmatter_schema`: ordered required keys + optional keys, per concept
type where needed (Phase 1's seven-key layer and Phase 2's inbox layer are
expressions of this schema).
- `frontmatter_schema`: three distinct fields, not one — the *required* key
set, the *allowlisted* key set, and the canonical emission *order*.
STRICT_V1 requires eight keys inside an eleven-key allowlist (see
`docs/phase-3-split-table.md`), so a single ordered required/optional pair
cannot express it. Also carries nullable keys and the scalar encoding
subset. Phase 1's seven-key layer and Phase 2's inbox layer are
expressions of this schema.
- `path_policy`: layer/path prefixes, filename namespaces, reserved filename
patterns (`index.md`, `promoted-verdict-*`, …).
- `reserved_files`: per-file required / optional / required-absent policy
(e.g. a profile may demand that `log.md` is absent).
- `index_policy`: managed-link format and per-level index expectations.
- `reserved_files`: per-file policy, THREE values not two — required,
required-absent, and present-but-exempt-from-document-gates (the proving
consumer's `README.md`, which carries no frontmatter and is never indexed
yet is still content-scanned).
- `index_policy`: managed-link format and per-level index expectations. This
is the widest gap between the two profiles: one index per bundle versus
one per directory, `- [label](target)` versus
`* [Title](link) - description`, append-if-absent versus exact match in
both directions, and a root index with or without frontmatter.
2. **Built-in profiles:**
- `DEFAULT` (implicit): exactly the ingest-spec v1 + Phase 2 contract.
- `STRICT_V1`: the `claude-code-llm-wiki` contract — closed type enum
(`Concept`/`Guide`/`Reference`/`Release`), extended required frontmatter
(`Concept`/`Guide`/`Reference`/`Release`), eight required frontmatter keys
(`type`, `title`, `description`, `timestamp`, `layer`, `source_tier`,
`source_url`, `source_sha`), layer path prefixes, `log.md` required-absent,
root-index metadata.
`source_url`, `source_sha`) inside an eleven-key allowlist that doubles as
emission order, layer/path/type/source_tier agreement rules, `log.md`
required-absent, `README.md` reserved-exempt, per-directory indexes, and
root-index metadata (`okf_version`, `bundle_profile`, `okf_spec_commit`).
This line was drafted as eight, corrected to four against the consumer's
`REQUIRED_KEYS` constant on 2026-07-25, and settled at eight by their
operator the same day: the four-key constant is their emit-path minimum,
not their contract, and all eight are present on 522/522 of their
documents. STRICT_V1 is therefore deliberately stricter than the
consumer's own validator; the split table records the agreed consequence.
3. **Plumbing change:** materialization, inbox, import, and index generation take
an optional `profile` argument defaulting to `DEFAULT`. No behavior branches
outside what the profile object expresses.
@ -41,7 +60,10 @@ the golden suite is the regression harness for that claim.
The wiki's existing validator mixes three kinds of gates. First implementation
step is a written mapping of every gate to exactly one column — the mapping is
reviewed with the operator before any porting:
reviewed with the operator before any porting. **Drafted 2026-07-25 in
`docs/phase-3-split-table.md`**, over all 25 gates, and sent to the proving
consumer for review; two gates turned out to bundle checks from different
columns and are decomposed there rather than forced into one.
| Column | Meaning | Goes where |
|---|---|---|
@ -69,9 +91,9 @@ reviewed with the operator before any porting:
| # | Assumption | Test |
|---|---|---|
| C1 | Default-profile refactor is behavior-neutral | Phase 1 golden suite byte-identical; full Phase 1/2 test suite green, unmodified |
| C2 | `strict-v1` requirements are fully enumerable from the wiki's validator | The split table is exhaustive: every wiki gate appears in exactly one column; reviewed with the operator |
| C2 | `strict-v1` requirements are fully enumerable from the wiki's validator | The split table is exhaustive: every wiki gate appears in exactly one column; reviewed with the operator. Drafted 2026-07-25 against `validate.py`/`bundle.py` at their `9ee5a8e`, which is the single gate entrypoint — the enumeration holds only against that commit, so re-check before porting. Two gates required decomposition rather than a single column |
| C3 | Verdict reservation must survive every profile | Named test: no profile construction can permit `type: verdict` |
| C4 | Profiles-in-code (no config files) suffice for the proving consumer | Confirmed with the operator at phase start; config-file loading is an extension point |
| C4 | Profiles-in-code (no config files) suffice for the proving consumer | CONFIRMED with the operator 2026-07-25 at phase start: profiles are constructed in code, config-file loading stays an extension point |
## Non-goals
@ -82,6 +104,12 @@ reviewed with the operator before any porting:
to block it (checked against the catalog's spec during the split-table step).
- Any change to commons' ingest-spec; if profile work reveals spec friction, it
is raised in commons, not patched locally.
- Aligning the contract with upstream OKF v0.2. This phase targets v0.1, and the
profile object is what makes the upstream version a *policy* rather than a
constant: no profile may hard-code an `okf_version` value, and the
frontmatter-set fields must be able to express either the legacy `timestamp`
or v0.2's `generated`. The decisions themselves live in
`docs/plan/okf-v0.2-alignment.md` and are sequenced after this phase.
## Verification

35
docs/plan/phase-4-node-half.md
View file

@ -46,7 +46,40 @@ code.
parity.
3. **linkedin-studio stays plugin-local.** Its `ingest/published/`
provenance-record grammar has a different lifecycle by design and is NOT
normalized into this library. Explicit non-goal, agreed up front.
normalized into this library. Explicit non-goal, agreed up front, and
confirmed by them on 2026-07-25 as their own fixed decision rather than
something conceded to us.
Two requirements they named on 2026-07-25, before the scope freeze, which
it must therefore answer explicitly rather than pass over. Both are about
not breaking what they have, not about building toward them:
- **`write_index` must have a do-nothing mode.** They run a fourth index
mode the spec does not describe: create-if-absent, no managed lines,
the whole file hand-editable. Their scaffold never opens an existing
index, so it cannot normalize one. The requirement is that this
library's writer match that — against an existing index carrying no
managed lines, write nothing, and do not normalize the frame "while it
is open anyway". This is the writer-side sibling of the Phase 3
decision that nothing enumerates a directory unless the profile says
the index is derived. `write_index` is deliverable 3, not deliverable
4's `write`; that was ambiguous in this plan and is now stated.
- **An absent vocabulary means "do not snap", never "snap to default".**
The reference checker collapses any non-matching type to a
`DEFAULT_TYPE`, which would erase their Profile / Operations /
PendingDiff distinction with zero byte change. Vocabulary is
per-bundle, never spec-global. This is already the Python half's shape
— `TypePolicy(allowed=None)` is the default and rewrites nothing — so
the Node half inherits a design rather than adopting a constraint, and
the collapse is a known deviation to resolve with the convention owner
under step 1 rather than lift.
They have offered a minimal tolerance fixture pair (a `brain/` hub whose
`index.md` carries `okf_version` as markdown text, and an
`ingest/published/` record: YAML-free five-line header terminated by
`\n---\n`, no trailing newline). Take the bytes rather than the
description — it is held as a tolerance case, never as a format this
library emits.
4. **Persist gate = guard-as-contract.** There is no Node guard today. The Node
half documents and enforces the guard's *contract* at its persist seam
(fail-closed hook point, no content-safety logic of its own) so a future Node

179
docs/upstream-okf-upgrade-runbook.md Normal file
View file

@ -0,0 +1,179 @@
# Upstream OKF upgrade — runbook
Standing policy (operator, 2026-07-26): **this library always supports the current
latest version of Google OKF.** This document is the procedure that policy runs on.
It exists because a standing policy without a procedure is an intention, and because
the v0.1 → v0.2 round produced five findings that a spec reading alone had settled
wrongly — each step below names the concrete failure it prevents.
Written after the v0.2 round. Every prevented-failure note is something that
actually happened, not a hypothetical.
## The goal that shapes the procedure: this repo is a black box
An upstream release is our problem, not our consumers'. The target cost to a repo
that consumes OKF through this library is **a re-run, and nothing else.**
That is a design commitment with teeth, not an aspiration:
- **Support is additive — a new profile, never a migration.** Existing profiles stay
byte-stable. A consumer who does not opt in sees no change at all.
- **New public parameters are keyword-only with defaults.** A consumer's existing
positional call sites stay source-compatible across an upgrade. This is why
`profile` is specified as `*, profile: BundleProfile = DEFAULT` — **decided, not
yet shipped**: measured 2026-07-26, no door takes the argument at all. It turns
additivity
from something a consumer has to measure into a property of the signature.
- **Consumer golden fixtures must not churn.** If an upgrade rewrites bytes in a
consumer's frozen fixtures, the upgrade is wrong, not the fixture.
**The honest boundary — state it every time, never let the black box be oversold.**
The library absorbs *shape* changes. It cannot absorb upstream changes to content the
consumer authored. v0.2 superseded `timestamp` with `generated.at` and the body
`# Citations` list with `sources`; no library change makes a consumer's existing
`timestamp` field become something else. For that class the deliverable is not
absorption but a **measured exposure report, per consumer, before they ask** — which
is what step 5 produces.
## Trigger
"Always latest" decays silently: nothing fails when upstream ships and we do not
notice. So the re-check is an item on the release checklist — run it at every
release of this library, and record the result **even when unchanged**, because an
unrecorded check is indistinguishable from a skipped one.
Check `GoogleCloudPlatform/knowledge-catalog`, path `okf/`. A version bump appears
as a commit against `okf/SPEC.md` §12 and, in the v0.2 round, as an explicit
migration commit (`okf: migrate format and tooling to Open Knowledge Format v0.2`).
## Step 1 — Pin before reading anything
Record the exact commit SHA and read everything at it.
*Prevents:* reading a moving branch and treating the result as fact. In the v0.2
round both this repo and `portfolio-optimiser-commons` independently read `main`,
which felt like two confirmations and was one unstable reference read twice. Worse,
`SPEC.md` was edited **after** the v0.2 migration commit — so "the v0.2 commit" and
"the current spec text" were different objects, and either alone would have been a
partial answer.
## Step 2 — Enumerate the whole `okf/` tree, not just `SPEC.md`
List every file and directory at the pinned commit before deciding what to read.
*Prevents:* assuming the directory you know about is the one that matters. In the
v0.2 round `okf/` held `SPEC.md`, `README.md`, `pyproject.toml`, `src/`, `tests/`,
`samples/` **and** `bundles/`. The last one held the actual v0.2 example bundles and
was not on anyone's list until the tree was enumerated.
## Step 3 — Read the shipped examples, not only the normative text
This is the step the v0.2 round skipped, and it is the reason this document exists.
**A spec says what is permitted. An example shows what upstream actually emits.**
Those differ, and when we are about to freeze a fixture and hand it to consumers,
the second one is what we are being measured against.
Two traps, both hit in the v0.2 round:
- **A directory called `samples/` was not samples of the format.**
`okf/samples/ga4_merch_store/` is a runner config (`README.md` + `seeds.txt`) that
drives the reference agent to *generate* a bundle. It says nothing about output
shape. The real examples were in `okf/bundles/` — four of them.
- **Fetch tooling that converts to markdown will strip or reformat frontmatter.**
Byte-level questions need byte-level fetches. Check for a BOM, check the trailing
newline, check whether a value is quoted. Those three are exactly the properties
that break downstream gates and exactly the ones a summarizing fetch destroys.
What reading the four v0.2 bundles produced, none of it derivable from `SPEC.md`:
| Finding | Consequence |
|---|---|
| No reference bundle declares `okf_version` at all; root `index.md` has no frontmatter | §12 is a MAY and upstream declines it. Our stricter posture became a *decision* instead of an assumption |
| `generated` is written for **human-authored** content (`by: human:…`) | It can never be an ownership or machine-generated predicate. Governs the collision gate |
| Real frontmatter is multi-line block YAML: block lists of multi-key mappings, nested mappings, flow sequences, booleans, dates | Re-sized the emitter work. It had been scoped against a list of strings |
| §7's canonical tool actor is `<producer>/<version>`, and upstream uses it | Counter-evidence to a recommendation we had already sent to another repo |
## Step 4 — Produce the diff, classified
From the spec's own "Changes from vN" section **plus** the example evidence,
classify every change as:
- **Breaking** — name it, and measure consumer exposure in step 5.
- **Additive** — new optional fields, new conventional headings.
- **Deferred upstream** — a format specified but its runtime protocol left to a
future revision. The format is supportable; an unspecified runtime is not
something to build against. Record it as out of scope *on upstream's deferral*,
not on our preference.
- **Observed but undocumented** — present in an official bundle, not found in the
spec sections read. Record as unresolved. Do not silently promote it to either
category. (v0.2 round: a `not:` family in `metrics/gross-margin.md`.)
## Step 5 — Measure our exposure and each consumer's, in that order
Ours: grep the profiles, the fixtures, and the emitter for every field the diff
touches. Cite file:line. Never reason about a key set — measure it.
Theirs: for each breaking change, state what would have to be true for it to reach
them, then **ask them to measure it against their real data** rather than telling
them whether they are affected. In the v0.2 round both `portfolio-optimiser-claude`
and `linkedin-studio` came back with measured zero exposure and found the framing
useful precisely because it was falsifiable.
## Step 6 — Plan additively, then pilot before general availability
The plan is a new profile, sequenced deliverables, and stated invariants. Rollout is
pilot-first: a pre-release tag to a small pilot set chosen for signal, revised on
their feedback, then GA. Flipping the `OKF_LATEST` alias **is** the GA event, not a
merge side effect.
State each pilot test's expected results **before** the run, numbered, plus what
would surprise us. An expectation we get wrong is a better result than a clean run;
only a stated expectation can be falsified by someone else's run.
**Name fixtures by repo and path, never by path alone.** In the v0.2 round both this
repo and `portfolio-optimiser-claude` had a directory named
`examples/ingest-golden-file/` holding *different* fixtures with zero content
overlap. A pilot baseline was agreed by name, and the name pointed at two files.
## Step 7 — Inform every OKF-consuming repo
All of them, not only the pilots, and not only when something breaks.
Send a **bounded loop of directed messages, not a broadcast** — a broadcast is
inherited by every future repo, which is the wrong lifetime for a
version-specific notice.
Each message carries:
1. The version, the pinned commit, and where to read it.
2. The breaking changes by name, with the shape of the exposure — so they can
measure themselves rather than take our word.
3. **The black-box promise and its boundary**: their cost is a re-run; existing
profiles are byte-stable; new parameters are keyword-only. And the part we
cannot absorb, said plainly.
4. What we want back, if anything, and what happens if they say nothing.
Consumers to cover (2026-07): `portfolio-optimiser-commons` (spec authorship),
`portfolio-optimiser`, `portfolio-optimiser-claude`, `claude-code-llm-wiki`,
`catalog`, `okr`, `linkedin-studio`, `ms-ai-architect`. Re-derive this list each
round rather than trusting it — it is a premise like any other.
## Step 8 — Close the loop in the repo
The plan doc records the pinned commit, the classified diff, the measured exposure,
and every correction a consumer sent back. Corrections stay visible: a premise that
survived verification is only known to have survived if the check is recorded, and a
claim we withdrew is only safely withdrawn if the withdrawal is written where the
claim was.
## Invariants this procedure protects
- No profile hard-codes an upstream version.
- No bundle declares a version its shape has not earned. The spec may permit
declaring conformance without a checkpoint; we decline. *Conform first, claim
after.*
- `DEFAULT` states commons' ingest-spec §5 layer — raised there, never patched here.
- `STRICT_V1` mirrors a consumer's ratified contract — never changed from here.
- The value of `okf_version` belongs to the catalog.
- Security stays the guard's domain in every version.

6
examples/ingest-golden-okf-v0-2/expected-bundle/index.md Normal file
View file

@ -0,0 +1,6 @@
---
okf_version: 0.2
---
Golden OKF v0.2 bundle: regional sales extracted from a CSV source.
- [Regional Sales](ingest-sales.md)

16
examples/ingest-golden-okf-v0-2/expected-bundle/ingest-sales.md Normal file
View file

@ -0,0 +1,16 @@
---
type: dataset
title: Regional Sales
source_system: golden-v0-2-sales
source_query: sales.csv
ingested_at: 2026-07-16T12:00:00Z
ingest_manifest: manifest@a83c86e8f847eca2
generated: { by: process:llm-ingestion-okf, at: 2026-07-16T12:00:00Z }
sources: [{ id: golden-v0-2-sales, resource: fixture }]
---
| region | units | revenue |
| --- | --- | --- |
| nord | 412 | 88120 |
| vest | 297 | 61540 |
| sor | 158 | 33010 |

4
examples/ingest-golden-okf-v0-2/fixture/sales.csv Normal file
View file

@ -0,0 +1,4 @@
region,units,revenue
nord,412,88120
vest,297,61540
sor,158,33010
1 region units revenue
2 nord 412 88120
3 vest 297 61540
4 sor 158 33010

1
examples/ingest-golden-okf-v0-2/ingested-at.txt Normal file
View file

@ -0,0 +1 @@
2026-07-16T12:00:00Z

18
examples/ingest-golden-okf-v0-2/manifest.json Normal file
View file

@ -0,0 +1,18 @@
{
"manifest_version": 1,
"source": {
"type": "file",
"id": "golden-v0-2-sales",
"root": "fixture"
},
"bundle_summary": "Golden OKF v0.2 bundle: regional sales extracted from a CSV source.",
"extractions": [
{
"id": "sales",
"title": "Regional Sales",
"query": "sales.csv",
"okf_type": "dataset",
"max_rows": 10
}
]
}

1
examples/ingest-golden-okf-v0-2/okf-version.txt Normal file
View file

@ -0,0 +1 @@
0.2

15
pyproject.toml
View file

@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "llm-ingestion-okf"
version = "0.4.0"
version = "0.5.0a1"
description = "Shared OKF (Open Knowledge Format) ingestion library: spec-based connectors, bundle inbox, and external-bundle import, with security delegated to llm-ingestion-guard."
readme = "README.md"
license = "MIT"
@ -50,9 +50,14 @@ python_version = "3.10"
module = ["llm_ingestion_guard", "llm_ingestion_guard.*"]
ignore_missing_imports = true
# Development-time install CHANNEL for the guard, which is not on a package
# index yet. It resolves `uv lock`/`uv sync` against the tag; it is uv-specific
# and never reaches consumers — the built wheel carries the range from
# [project.dependencies] above, which is the pin.
# Install CHANNEL for the guard, which is not on a package index yet. It is
# uv-specific, and it reaches further than a dev-only setting: measured
# 2026-07-25, a consumer installing this package from git WITH UV picks the
# guard up from this tag automatically, because uv reads this file when it
# builds from the source tree. pip does not read it — it resolves
# [project.dependencies] alone and fails with "No matching distribution found
# for llm-ingestion-guard" until the guard is installed from its own tag first
# (README). Either way the range above stays the pin: the built wheel carries
# `Requires-Dist: llm-ingestion-guard<0.3,>=0.2`, verified against the wheel.
[tool.uv.sources]
llm-ingestion-guard = { git = "https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git", tag = "v0.2.0" }

2
src/llm_ingestion_okf/__init__.py
View file

@ -73,7 +73,7 @@ from .manifest import (
)
from .materialize import IngestResult, materialize_bundle
__version__ = "0.4.0"
__version__ = "0.5.0a1"
__all__ = [
"BlockedFile",

4
src/llm_ingestion_okf/errors.py
View file

@ -96,6 +96,10 @@ class MaterializationError(IngestError):
- `ingested_at_invalid` — ingested_at is not ISO-8601 UTC with a Z suffix
- `collision_unstamped` — the §3 collision gate: a generated name is
occupied by a file without the ingest stamp
- `source_reference_unquotable` — a manifest source's id or locator
contains a character that would restructure the `sources` flow mapping
(Door A, v0.2 profiles); refused rather than emitted, because the
resulting document parses cleanly into a record no one wrote
- `inbox_slug_empty` — a dropped file's name reduces to an empty slug
under the id grammar (Door B; never an invented fallback name)
- `inbox_slug_too_long` — the generated inbox filename would exceed the

21
src/llm_ingestion_okf/importer.py
View file

@ -34,20 +34,18 @@ from typing import Protocol
from .errors import IngestError, MaterializationError, SourceError
from .extract import decode_text
from .materialize import (
INDEX_NAME,
check_filename_length,
link_in_index,
reduce_to_id_grammar,
validate_ingested_at,
write_bytes,
)
from .profiles import DEFAULT
# An OKF concept is a `.md` document by definition — the guard's path gate
# rejects anything else outright — so nothing else in the source tree is a
# concept, and nothing else is this door's to merge.
_CONCEPT_SUFFIX = ".md"
_FILENAME_PREFIX = "import-"
# concept, and nothing else is this door's to merge. The suffix and the
# `import-` namespace are the profile's (`DEFAULT.paths`).
# The guard's non-blocking floor and its review queue, by VALUE (`Disposition`
# is a `str, Enum`, so the value is the stable thing to compare against across
@ -176,7 +174,7 @@ def import_slug(concept_path: str) -> str:
collapse them onto one filename. A path that reduces to nothing fails fast
rather than being given an invented name.
"""
concept_id = concept_path[: -len(_CONCEPT_SUFFIX)]
concept_id = concept_path[: -len(DEFAULT.paths.concept_suffix)]
slug = reduce_to_id_grammar(concept_id)
if not slug:
raise MaterializationError(
@ -195,7 +193,8 @@ def import_filename(slug: str) -> str:
grammar admits.
"""
return check_filename_length(
f"{_FILENAME_PREFIX}{slug}{_CONCEPT_SUFFIX}", code="import_path_too_long"
f"{DEFAULT.paths.import_prefix}{slug}{DEFAULT.paths.concept_suffix}",
code="import_path_too_long",
)
@ -206,7 +205,7 @@ def _index_label(concept_path: str) -> str:
does not. Fail-fast, never repair — the same rule Door A applies to a
manifest title and Door B to a dropped filename.
"""
label = concept_path[: -len(_CONCEPT_SUFFIX)]
label = concept_path[: -len(DEFAULT.paths.concept_suffix)]
if any(char in label for char in "\n\r[]"):
raise MaterializationError(
f"concept path {concept_path!r} contains '[' or ']', which would break "
@ -229,7 +228,7 @@ def _read_bundle(source: Path) -> tuple[dict[str, str], list[FailedConcept]]:
# glob: glob case-sensitivity follows the FILESYSTEM, so `NOTE.MD`
# would be a concept on APFS and not one on ext4 — the same bundle
# importing differently per platform. The guard folds case here too.
if not path.is_file() or path.suffix.lower() != _CONCEPT_SUFFIX:
if not path.is_file() or path.suffix.lower() != DEFAULT.paths.concept_suffix:
continue
concept_path = path.relative_to(source).as_posix()
try:
@ -409,9 +408,9 @@ def import_bundle(
# §6 index — the last disk mutation, and only when something merged.
if merged:
index_path = bundle / INDEX_NAME
index_path = bundle / DEFAULT.index.name
if not index_path.is_file():
write_bytes(bundle, INDEX_NAME, "")
write_bytes(bundle, DEFAULT.index.name, "")
for entry in merged:
link_in_index(bundle, entry.path.name, _index_label(entry.concept_path))

42
src/llm_ingestion_okf/inbox.py
View file

@ -28,7 +28,6 @@ from pathlib import Path
from .errors import IngestError, MaterializationError, SourceError
from .extract import extract_text
from .materialize import (
INDEX_NAME,
check_filename_length,
link_in_index,
parse_frontmatter,
@ -36,11 +35,7 @@ from .materialize import (
validate_ingested_at,
write_bytes,
)
_RESERVED_OKF_TYPE = "verdict"
_FILENAME_PREFIX = "inbox-"
_FILENAME_SUFFIX = ".md"
from .profiles import DEFAULT
def inbox_slug(source_filename: str) -> str:
@ -73,7 +68,8 @@ def inbox_filename(slug: str) -> str:
not give it.
"""
return check_filename_length(
f"{_FILENAME_PREFIX}{slug}{_FILENAME_SUFFIX}", code="inbox_slug_too_long"
f"{DEFAULT.paths.inbox_prefix}{slug}{DEFAULT.paths.concept_suffix}",
code="inbox_slug_too_long",
)
@ -104,12 +100,11 @@ def render_inbox_concept(
validate_ingested_at(ingested_at)
# The verdict layer is RESERVED: the promotion gate is the only path into
# it, at this door exactly as at Door A's manifest validation.
if okf_type.lower() == _RESERVED_OKF_TYPE:
raise MaterializationError(
f"okf_type must not be {_RESERVED_OKF_TYPE!r} (reserved layer)",
code="okf_type_reserved",
)
# it, at this door exactly as at Door A's manifest validation — the same
# profile decides, each door raises its own typed error.
rejection = DEFAULT.types.rejection(okf_type)
if rejection is not None:
raise MaterializationError(f"okf_type {rejection.reason}", code=rejection.code)
# The title is rendered verbatim into `- [title](target)` and into
# line-oriented frontmatter — met by fail-fast validation, never repair.
if any(char in title for char in "\n\r[]"):
@ -131,8 +126,7 @@ def render_inbox_concept(
"ingested_at": ingested_at,
"generated": "true",
}
rendered = "\n".join(f"{key}: {value}" for key, value in frontmatter.items())
return f"---\n{rendered}\n---\n\n{_normalize_body(text)}"
return f"---\n{DEFAULT.frontmatter.emit(frontmatter)}\n---\n\n{_normalize_body(text)}"
# --- the guard seam -------------------------------------------------------
@ -248,11 +242,9 @@ def process_inbox(
a reserved `okf_type`, and a missing inbox directory.
"""
validate_ingested_at(ingested_at)
if okf_type.lower() == _RESERVED_OKF_TYPE:
raise MaterializationError(
f"okf_type must not be {_RESERVED_OKF_TYPE!r} (reserved layer)",
code="okf_type_reserved",
)
run_rejection = DEFAULT.types.rejection(okf_type)
if run_rejection is not None:
raise MaterializationError(f"okf_type {run_rejection.reason}", code=run_rejection.code)
inbox = Path(inbox_dir)
if not inbox.is_dir():
raise SourceError(f"inbox directory does not exist: {inbox}", code="source_root_missing")
@ -299,7 +291,11 @@ def process_inbox(
# pre-existing curated content by a later file's check.
bundle = Path(bundle_dir)
pre_existing = (
{path.name for path in bundle.glob("*.md") if path.name != INDEX_NAME}
{
path.name
for path in bundle.glob(f"*{DEFAULT.paths.concept_suffix}")
if path.name != DEFAULT.index.name
}
if bundle.is_dir()
else set()
)
@ -369,9 +365,9 @@ def process_inbox(
# §6 index — the last disk mutation, and only when something was written.
if persisted:
index_path = bundle / INDEX_NAME
index_path = bundle / DEFAULT.index.name
if not index_path.is_file():
write_bytes(bundle, INDEX_NAME, "")
write_bytes(bundle, DEFAULT.index.name, "")
for entry in persisted:
link_in_index(
bundle, entry.path.name, unicodedata.normalize("NFC", Path(entry.source_file).stem)

22
src/llm_ingestion_okf/manifest.py
View file

@ -15,11 +15,10 @@ from pathlib import Path
from typing import Any, Union
from .errors import ManifestError
from .profiles import DEFAULT, BundleProfile
_ID_PATTERN = re.compile(r"[a-z0-9][a-z0-9-]*\Z")
_RESERVED_OKF_TYPE = "verdict"
@dataclass(frozen=True)
class FileSource:
@ -60,13 +59,17 @@ class Manifest:
extractions: tuple[Extraction, ...]
def generated_filename(extraction_id: str) -> str:
def generated_filename(extraction_id: str, *, profile: BundleProfile = DEFAULT) -> str:
"""The concept filename for an extraction (spec §5).
The `ingest-` prefix keeps the namespace disjoint from `index.md` and
`promoted-verdict-*` (spec §3) for every id the §4 grammar admits.
The prefix and suffix are the profile's, because the ownership scan globs
on the same two values: a name built from one profile and scanned for under
another is a file the library cannot recognise as its own.
"""
return f"ingest-{extraction_id}.md"
return f"{profile.paths.ingest_prefix}{extraction_id}{profile.paths.concept_suffix}"
def load_manifest(path: Path) -> Manifest:
@ -193,11 +196,12 @@ def _validate_extraction(data: object, index: int) -> Extraction:
okf_type = _require_str(obj["okf_type"], f"{label}.okf_type")
# The verdict layer is RESERVED (spec §3): the promotion gate is the only
# path into it — enforced here, fail-fast, before any source call.
if okf_type.lower() == _RESERVED_OKF_TYPE:
raise ManifestError(
f"{label}.okf_type must not be 'verdict' (reserved layer)", code="okf_type_reserved"
)
# path into it — enforced here, fail-fast, before any source call. The
# profile decides which types a bundle admits; the door raises its own
# typed error, since Door B refuses the same type as a MaterializationError.
rejection = DEFAULT.types.rejection(okf_type)
if rejection is not None:
raise ManifestError(f"{label}.okf_type {rejection.reason}", code=rejection.code)
max_rows = obj["max_rows"]
if not _is_int(max_rows) or max_rows < 1:

229
src/llm_ingestion_okf/materialize.py
View file

@ -13,6 +13,7 @@ import hashlib
import logging
import re
import unicodedata
from collections.abc import Mapping
from dataclasses import dataclass
from pathlib import Path
@ -23,23 +24,18 @@ from .manifest import (
FileSource,
HttpSource,
Manifest,
Source,
SqlSource,
generated_filename,
load_manifest_bytes,
)
from .profiles import DEFAULT, BundleProfile
from .render import render_fenced_block, render_table
_LOGGER = logging.getLogger(__name__)
INDEX_NAME = "index.md"
_INGESTED_AT_RE = re.compile(r"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z$")
# One managed index line: `- [<label>](<target>)`. Anchored full-line —
# removal keys on this exact shape for specific ingest targets, never a bare
# substring (a promoted verdict's line has the same shape but a non-ingest
# target; curated prose mentioning a target inline does not match).
_MANAGED_LINE_RE = re.compile(r"^- \[(?P<label>[^\]]*)\]\((?P<target>[^)]+)\)$")
@dataclass(frozen=True)
class IngestResult:
@ -120,24 +116,6 @@ def check_filename_length(name: str, *, code: str) -> str:
return name
def _collapse_whitespace(value: str) -> str:
# §5 mandates whitespace-run collapse for ONE field only: `source_query`
# (ingest-spec.md:140-141), where a legitimately multi-line SQL SELECT
# must render on one line. Every other value is validated single-line at
# manifest load and emitted verbatim — validation, not repair — so
# operator-supplied bytes (e.g. a title's internal double space) survive.
return " ".join(value.split())
def _render_frontmatter(frontmatter: dict[str, str]) -> str:
# Line-oriented `key: value`, insertion order. Only `source_query` is
# collapsed; all other values pass through verbatim.
return "\n".join(
f"{key}: {_collapse_whitespace(value) if key == 'source_query' else value}"
for key, value in frontmatter.items()
)
def parse_frontmatter(path: Path) -> dict[str, str]:
lines = path.read_text(encoding="utf-8").splitlines()
if not lines or lines[0].strip() != "---":
@ -152,10 +130,18 @@ def parse_frontmatter(path: Path) -> dict[str, str]:
return frontmatter
def _is_ingest_owned(path: Path, manifest_stem: str) -> bool:
# §3/§5 ownership: the ingest stamp is `generated: true` AND an
# `ingest_manifest` reference. Promoted verdict files carry neither key,
# so they can never classify as ingest-owned.
def _is_ingest_owned(path: Path, manifest_stem: str, *, profile: BundleProfile = DEFAULT) -> bool:
# §3/§5 ownership: the profile's ingest stamp AND an `ingest_manifest`
# reference. Promoted verdict files carry neither key, so they can never
# classify as ingest-owned.
#
# The stamp is the PROFILE's because it differs per profile (v0.1 writes
# `generated: true`, v0.2 a `{ by: ..., at: ... }` mapping) — and because the
# emitter and this predicate are coupled through that value. Changing the
# emitted form without the predicate is what makes the library stop
# recognising its own output, firing the collision gate on the files its own
# previous run wrote. `OwnershipPolicy` is where the two meet, so they can
# only be changed together.
#
# §10.2 per-manifest ownership: a file is THIS manifest's to replace only
# when the reference names it by stem. The stamp is `{stem}@{sha256[:16]}`;
@ -165,7 +151,8 @@ def _is_ingest_owned(path: Path, manifest_stem: str) -> bool:
# keeps its own. rsplit strips the trailing `@{sha}`, so a stem that itself
# contains `@` still compares correctly.
frontmatter = parse_frontmatter(path)
if frontmatter.get("generated") != "true":
generated = frontmatter.get("generated")
if generated is None or not profile.ownership.owns(generated):
return False
reference = frontmatter.get("ingest_manifest")
if reference is None:
@ -173,10 +160,70 @@ def _is_ingest_owned(path: Path, manifest_stem: str) -> bool:
return reference.rsplit("@", 1)[0] == manifest_stem
# The characters that terminate or restructure a YAML flow mapping. `:\s`
# catches a colon that would open a nested key; a colon inside `https://host`
# does not, and stays a plain scalar.
_FLOW_UNSAFE_RE = re.compile(r"[,\[\]{}]|:\s")
def _source_locator(source: Source) -> str:
"""Where a manifest source points, per source type.
A filesystem root, the NAME of the environment variable holding the DSN, or
the base URL. `credential_ref` is not a locator and is never returned here:
a credential reference has no reader in a bundle.
"""
if isinstance(source, FileSource):
return source.root
if isinstance(source, SqlSource):
return source.connection_ref
return source.base_url
def _render_sources(source: Source) -> str:
"""§5 `sources` as an inline flow sequence of one flow mapping.
Two keys, not upstream's five: a manifest source has no `author`, no
`last_modified`, and no bundle-internal `resource` in upstream's sense, and
inventing them would be writing fields with no reader.
The flow form rather than upstream's block list, measured and chosen: a
block list read back through this library's line-oriented parser turns each
item line into a KEY nobody wrote — and `_is_ingest_owned` reads through
that same parser. The flow form also satisfies commons' §5 "all values MUST
be single-line", and §11 requires parseable YAML rather than block YAML.
Refusing an unquotable locator is the point of the check rather than a
nicety: `[{ id: x, resource: data, backup }]` is not a parse ERROR, it is a
mapping with a `backup` key nobody wrote. A silently wrong provenance record
is worse than a refused run, and repairing the value by quoting it would
change bytes the operator supplied. Validation, not repair — the same
posture as the filename-length gate.
"""
locator = _source_locator(source)
for label, value in (("id", source.id), ("resource", locator)):
if _FLOW_UNSAFE_RE.search(value):
raise MaterializationError(
f"the source {label} {value!r} contains a character that would "
"restructure the `sources` flow mapping (one of `,[]{}` or a "
"colon followed by whitespace) — refusing to emit a provenance "
"record that parses cleanly into something no one wrote",
code="source_reference_unquotable",
)
return f"[{{ id: {source.id}, resource: {locator} }}]"
def _render_concept_file(
manifest: Manifest, extraction: Extraction, body: str, *, ingested_at: str, stamp: str
manifest: Manifest,
extraction: Extraction,
body: str,
*,
ingested_at: str,
stamp: str,
profile: BundleProfile = DEFAULT,
) -> str:
# §5 frontmatter: exactly these keys, in exactly this order.
# §5 frontmatter: exactly these keys. The order and the `source_query`
# whitespace collapse are the profile's — DEFAULT states the §5 layer.
frontmatter = {
"type": extraction.okf_type,
"title": extraction.title,
@ -184,9 +231,15 @@ def _render_concept_file(
"source_query": extraction.query,
"ingested_at": ingested_at,
"ingest_manifest": stamp,
"generated": "true",
"generated": profile.ownership.stamp(ingested_at),
}
return f"---\n{_render_frontmatter(frontmatter)}\n---\n\n{body}"
# Written only by a profile that NAMES it. `emit` sorts an unnamed key into
# the tail rather than dropping it, so building one mapping for both
# profiles would append `sources` to every v0.1 bundle — additivity is a
# property of what is constructed here, not of the emitter.
if "sources" in profile.frontmatter.order:
frontmatter["sources"] = _render_sources(manifest.source)
return f"---\n{profile.frontmatter.emit(frontmatter)}\n---\n\n{body}"
def write_bytes(bundle_dir: Path, name: str, content: str) -> Path:
@ -199,7 +252,11 @@ def write_bytes(bundle_dir: Path, name: str, content: str) -> Path:
def _update_index_lines(
index_path: Path, removed_targets: set[str], labels_by_target: dict[str, str]
index_path: Path,
removed_targets: set[str],
labels_by_target: dict[str, str],
*,
profile: BundleProfile = DEFAULT,
) -> None:
"""§6 maintenance on an EXISTING index: drop managed lines whose target is
an ingest file removed in this run; refresh in place a managed label that
@ -213,7 +270,7 @@ def _update_index_lines(
for line in lines:
content = line.rstrip("\r\n")
ending = line[len(content) :]
match = _MANAGED_LINE_RE.match(content)
match = profile.index.link_pattern.match(content)
if match is not None:
target = match.group("target")
if target in removed_targets:
@ -221,17 +278,25 @@ def _update_index_lines(
continue
new_label = labels_by_target.get(target)
if new_label is not None and match.group("label") != new_label:
line = f"- [{new_label}]({target})" + ending
line = profile.index.render_link(new_label, target) + ending
changed = True
updated.append(line)
if changed:
index_path.write_bytes("".join(updated).encode("utf-8"))
def link_in_index(bundle_dir: Path, target_name: str, label: str) -> None:
def link_in_index(
bundle_dir: Path, target_name: str, label: str, *, profile: BundleProfile = DEFAULT
) -> None:
# §6: idempotent by target — a link whose target is already present in
# the index is never added twice.
index_path = safe_resolve(bundle_dir, INDEX_NAME)
#
# `profile` is keyword-only with a default because this function is public
# and called from all three doors (A here, B in inbox.py, C in importer.py).
# Doors B and C keep the default, which is the behaviour they already had;
# which profile THEY should own is a separate question, and answering it by
# changing this signature would have decided it silently.
index_path = safe_resolve(bundle_dir, profile.index.name)
body = index_path.read_bytes().decode("utf-8")
if f"]({target_name})" in body:
return
@ -239,7 +304,38 @@ def link_in_index(bundle_dir: Path, target_name: str, label: str) -> None:
# bundle_summary first, but Door B has no summary to invent, so its index
# starts empty and must not open with a blank line.
prefix = body if (body == "" or body.endswith("\n")) else body + "\n"
index_path.write_bytes(f"{prefix}- [{label}]({target_name})\n".encode())
line = profile.index.render_link(label, target_name)
index_path.write_bytes(f"{prefix}{line}\n".encode())
def _render_root_frontmatter(values: Mapping[str, str], *, profile: BundleProfile) -> str:
"""The root index's frontmatter block (§8, §12), or "" when nothing is
declared.
The policy names the keys and fixes their order; the caller supplies the
values. Ordering by the POLICY rather than by the mapping is what keeps two
callers passing the same keys from emitting different bytes — a dict
preserves insertion order, and a golden fixture would then depend on the
order a caller happened to build its argument in.
Values are written verbatim. `okf_version` must reach catalog's shape gate
unquoted, so nothing here may add quoting; the golden fixture asserts that
on raw bytes.
"""
unknown = sorted(set(values) - set(profile.index.root_frontmatter))
if unknown:
raise MaterializationError(
f"root frontmatter key(s) {', '.join(repr(key) for key in unknown)} are not "
f"named by the {profile.index.name} policy, which pins "
f"{profile.index.root_frontmatter or '()'} — writing an unnamed key would "
"put a value in a file no reader of this contract looks at",
code="index_root_frontmatter_unexpected",
)
declared = [key for key in profile.index.root_frontmatter if key in values]
if not declared:
return ""
lines = "".join(f"{key}: {values[key]}\n" for key in declared)
return f"---\n{lines}---\n\n"
def materialize_bundle(
@ -249,6 +345,8 @@ def materialize_bundle(
*,
allow_network: bool = False,
http_get: HttpGet | None = None,
profile: BundleProfile = DEFAULT,
root_frontmatter_values: Mapping[str, str] | None = None,
) -> IngestResult:
"""Materialize a manifest's extractions into an OKF bundle (§5).
@ -260,8 +358,34 @@ def materialize_bundle(
seam (default urllib_get, the only socket path; ignored for `file`/`sql`)
so tests run socket-free (§11). Source calls are logged per §8 (which
source, when, row count) — never cell contents, never secrets.
`profile` selects the bundle contract: `DEFAULT` (commons' ingest-spec §5)
or `OKF_V0_2`. It is keyword-only behind the `*` the signature already
carried, so every three-positional call site stays source-compatible —
support for a new upstream version is additive, never a migration. The
profile governs the emitted frontmatter, the ownership stamp the collision
gate recognises, the concept filenames, and the index; it does NOT reach
manifest type validation, which runs against `DEFAULT` (the two policies
compare equal today). `STRICT_V1` is not supported here: its index policy
sets `per_directory` and `entries_match_directory`, neither of which this
materializer honours.
`root_frontmatter_values` supplies the values for the keys the profile's
index policy names — `okf_version` under `OKF_V0_2` (§8, §12). The split is
deliberate: the profile names the key, the caller owns the value, because
`okf_version`'s value tracks the upstream Google version and belongs to
catalog (E1). Offering a key the policy does not name is refused fail-fast,
before any disk mutation. Omitting the argument emits no block at all — §12
is a MAY, and none of upstream's reference bundles declares it.
The block is written only when the index is CREATED. A re-run into an
existing bundle leaves it untouched, which is what makes the second run
byte-identical to the first (A-E5).
"""
validate_ingested_at(ingested_at)
# Before any source access or disk mutation: a caller error here must not
# leave a partially written bundle behind.
root_frontmatter = _render_root_frontmatter(root_frontmatter_values or {}, profile=profile)
manifest_file = Path(manifest_path)
try:
raw = manifest_file.read_bytes()
@ -323,9 +447,9 @@ def materialize_bundle(
"source call: source=%s ingested_at=%s rows=%d", source.id, ingested_at, row_count
)
content = _render_concept_file(
manifest, extraction, body, ingested_at=ingested_at, stamp=stamp
manifest, extraction, body, ingested_at=ingested_at, stamp=stamp, profile=profile
)
staged.append((generated_filename(extraction.id), content))
staged.append((generated_filename(extraction.id, profile=profile), content))
# Disk phase.
bundle = Path(bundle_dir)
@ -336,8 +460,9 @@ def materialize_bundle(
# ingest stamp are ours to replace.
owned = {
path.name
for path in sorted(bundle.glob("*.md"))
if path.name != INDEX_NAME and _is_ingest_owned(path, manifest_file.stem)
for path in sorted(bundle.glob(f"*{profile.paths.concept_suffix}"))
if path.name != profile.index.name
and _is_ingest_owned(path, manifest_file.stem, profile=profile)
}
# §3 collision gate — BEFORE any mutation: a staged filename occupied by
# a file WITHOUT the stamp is curated content; never overwrite it.
@ -356,16 +481,22 @@ def materialize_bundle(
# §6 index generation — the last disk mutation. A fresh index gets
# bundle_summary as its body; links are appended in extraction order.
index_path = bundle / INDEX_NAME
index_path = bundle / profile.index.name
labels_by_target = {
generated_filename(extraction.id): extraction.title for extraction in manifest.extractions
generated_filename(extraction.id, profile=profile): extraction.title
for extraction in manifest.extractions
}
if not index_path.is_file():
write_bytes(bundle, INDEX_NAME, manifest.bundle_summary + "\n")
write_bytes(bundle, profile.index.name, root_frontmatter + manifest.bundle_summary + "\n")
else:
# Links whose target is an ingest-owned file removed this run MUST be
# removed; all other links — curated and promoted — are preserved.
_update_index_lines(index_path, owned - staged_names, labels_by_target)
_update_index_lines(index_path, owned - staged_names, labels_by_target, profile=profile)
for extraction in manifest.extractions:
link_in_index(bundle, generated_filename(extraction.id), extraction.title)
link_in_index(
bundle,
generated_filename(extraction.id, profile=profile),
extraction.title,
profile=profile,
)
return IngestResult(written=written, stamp=stamp)

764
src/llm_ingestion_okf/profiles.py Normal file
View file

@ -0,0 +1,764 @@
"""The bundle contract as configuration (Phase 3).
What a valid bundle looks like — which concept types exist, which frontmatter
keys are emitted and in which order, which filename namespaces the doors own,
and what an index line looks like — is a profile, not a set of constants
scattered across the doors. `DEFAULT` is exactly the ingest-spec v1 + Phase 2
contract, so nothing observable changes for a caller that never mentions a
profile; the golden fixtures are the byte-level proof.
Two things deliberately do NOT live here. Security is the guard's, always: no
disposition, origin or channel vocabulary belongs on a profile. And the
reserved `verdict` layer is a spec invariant rather than profile config — it is
refused at construction, so a profile admitting it cannot be built, let alone
passed to a door. The `timestamp`/`generated` pair is refused the same way, and
for the same reason: both are things a profile must not be able to express.
Profiles are constructed in code. Config-file loading and inheritance chains
are extension points, not v1 (settled with the operator at phase start).
"""
from __future__ import annotations
import re
from collections.abc import Collection, Mapping, Sequence
from dataclasses import dataclass, field, replace
# The one layer no profile may admit (ingest-spec §3): the promotion gate is
# the only path into it. Compared case-insensitively, as both doors already do.
RESERVED_OKF_TYPE = "verdict"
# The one key PAIR no profile may name (OKF §13.1): `timestamp` is readable as
# a legacy stand-in for `generated.at` only while `generated` is ABSENT, so a
# schema able to name both can describe a document that has neither. Stated as
# key names rather than as a judgement on values because every `generated` a
# schema can express today is a scalar (`_is_legal_value`) and therefore
# malformed as a v0.2 mapping — naming both IS the hazard here. When a value
# model can express a well-formed `generated`, this narrows with it.
# "Naming" spans EVERY field that puts a key in the schema's namespace — `order`,
# `required`, `allowed` and `nullable` alike. A field left out of that union is a
# hole in the gate, not a narrower gate: the namespace is open by default, so a
# key named only by `nullable` is admitted just as surely as an emitted one.
_TIMESTAMP_FALLBACK_PAIR = frozenset({"timestamp", "generated"})
@dataclass(frozen=True)
class TypeRejection:
"""Why a profile refuses an `okf_type`, for the door to frame and raise.
The policy does not raise: Door A refuses with `ManifestError` and Door B
with `MaterializationError`, so the refusal has to be reported rather than
thrown. `reason` completes the sentence "<label>okf_type ..." and `code` is
the stable `IngestError.code` the caller asserts on.
"""
reason: str
code: str
@dataclass(frozen=True)
class TypePolicy:
"""Which `okf_type` values a bundle admits.
`allowed` is `None` for an open set — Phase 1/2 accept any type the
operator names — or a closed enum. Either way the reserved layer is
excluded, and a closed set that names it fails at construction.
"""
allowed: frozenset[str] | None = None
def __post_init__(self) -> None:
if self.allowed is None:
return
named = sorted(value for value in self.allowed if value.lower() == RESERVED_OKF_TYPE)
if named:
raise ValueError(
f"a profile must not admit the reserved {RESERVED_OKF_TYPE!r} layer "
f"(got {', '.join(repr(value) for value in named)}) — the promotion "
"gate is the only path into it (ingest-spec §3)"
)
def rejection(self, okf_type: str) -> TypeRejection | None:
"""The refusal for `okf_type`, or `None` when the profile admits it."""
if okf_type.lower() == RESERVED_OKF_TYPE:
return TypeRejection(
reason=f"must not be {RESERVED_OKF_TYPE!r} (reserved layer)",
code="okf_type_reserved",
)
if self.allowed is not None and okf_type not in self.allowed:
return TypeRejection(
reason=f"must be one of {', '.join(sorted(self.allowed))}, got {okf_type!r}",
code="okf_type_not_allowed",
)
return None
@dataclass(frozen=True)
class SchemaViolation:
"""One way a document's frontmatter departs from the schema.
Reported rather than raised, like `TypeRejection`: a document can depart in
several ways at once, and the caller decides whether that is a refusal, a
report line, or a migration to-do. `code` is the stable machine-readable
part; `reason` completes the sentence "<key> ...".
"""
key: str
reason: str
code: str
def _is_legal_value(value: object) -> bool:
"""A string scalar, or a non-empty list of strings — nothing else.
The list shape is real, not hypothetical: the proving consumer's `Release`
pages carry block lists. A schema that assumed scalars-only would reject
pages they ship today.
"""
if isinstance(value, str):
return True
if isinstance(value, list):
return bool(value) and all(isinstance(item, str) for item in value)
return False
@dataclass(frozen=True)
class FrontmatterSchema:
"""The frontmatter key namespace, what it admits, and how it is emitted.
`order` is the canonical emission order and `collapsed_keys` names the keys
whose whitespace runs collapse to single spaces on the way out. The DEFAULT
order spans both doors' key sets: Door A emits seven of these keys and Door
B six, and each door's subset comes out in exactly the order it wrote by
hand in Phases 1 and 2.
The judging fields — `required`, `allowed`, `nullable`, `key_pattern` — are
read by `violations`. `allowed` is `None` for an open namespace, which is
what DEFAULT keeps: Door A emits seven keys, Door B six, and Door C writes
an imported concept verbatim with whatever the sender wrote, so a closed
namespace or a required set on DEFAULT would declare invalid the very
bundles this library produces.
"""
order: tuple[str, ...]
collapsed_keys: frozenset[str] = field(default_factory=frozenset)
required: frozenset[str] = field(default_factory=frozenset)
allowed: frozenset[str] | None = None
nullable: frozenset[str] = field(default_factory=frozenset)
key_pattern: re.Pattern[str] | None = None
def __post_init__(self) -> None:
named = set(self.order) | set(self.required) | set(self.nullable)
if self.allowed is not None:
named |= set(self.allowed)
if _TIMESTAMP_FALLBACK_PAIR <= named:
raise ValueError(
"a profile must not name both 'timestamp' and 'generated' (OKF "
"§13.1 grants the timestamp fallback only while `generated` is "
"ABSENT, so a schema naming both can describe a document with "
"neither a valid `generated.at` nor an eligible fallback)"
)
if self.allowed is None:
return
for label, keys in (("required", self.required), ("nullable", self.nullable)):
stray = sorted(keys - self.allowed)
if stray:
raise ValueError(
f"{label} keys must be inside the allowlist "
f"(got {', '.join(repr(key) for key in stray)}) — a schema that "
"demands a key it also forbids can never be satisfied"
)
def violations(self, values: Mapping[str, object]) -> tuple[SchemaViolation, ...]:
"""Every way `values` departs from this schema, deterministically ordered.
Sorted by `(key, code)` rather than reported in mapping order: two runs
over the same document must produce the same report, and mapping order
is an accident of how the document was parsed.
Takes an already-parsed mapping. Parsing strict frontmatter is a
separate concern — this library's line-oriented parser cannot represent
the block lists and nulls a strict schema admits, which is why Door C
writes imported concepts verbatim rather than round-tripping them.
"""
found: list[SchemaViolation] = []
for key in self.required - set(values):
found.append(SchemaViolation(key, "is required and absent", "frontmatter_key_missing"))
for key, value in values.items():
if self.key_pattern is not None and not self.key_pattern.fullmatch(key):
found.append(
SchemaViolation(
key,
f"is not a legal key name (must match {self.key_pattern.pattern})",
"frontmatter_key_malformed",
)
)
continue
if self.allowed is not None and key not in self.allowed:
found.append(
SchemaViolation(
key, "is not on the key allowlist", "frontmatter_key_not_allowed"
)
)
continue
if value is None:
if key not in self.nullable:
found.append(SchemaViolation(key, "must not be null", "frontmatter_value_null"))
continue
if not _is_legal_value(value):
found.append(
SchemaViolation(
key,
"must be a string or a non-empty list of strings",
"frontmatter_value_shape",
)
)
return tuple(sorted(found, key=lambda violation: (violation.key, violation.code)))
def emit(self, values: Mapping[str, str]) -> str:
"""Render `values` as line-oriented `key: value`, one line per key.
Commons decision D1: the keys named in `order` come first, in that
order, followed by any remaining keys SORTED. Ordering the tail rather
than trusting insertion order is what makes a regeneration over the
same data byte-identical.
Returns the lines only — the caller owns the `---` fences.
"""
named = [key for key in self.order if key in values]
tail = sorted(key for key in values if key not in self.order)
return "\n".join(f"{key}: {self._render(key, values[key])}" for key in [*named, *tail])
def _render(self, key: str, value: str) -> str:
# §5 mandates whitespace-run collapse for `source_query` only
# (ingest-spec.md:140-141), where a legitimately multi-line SQL SELECT
# must render on one line. Every other value is validated single-line
# at load and emitted verbatim — validation, not repair — so
# operator-supplied bytes (e.g. a title's internal double space)
# survive.
return " ".join(value.split()) if key in self.collapsed_keys else value
# The v0.1 ingest stamp. A literal rather than a configurable value: it is what
# every bundle this library has already written carries, and recognising it is
# what keeps those bundles re-runnable under a later profile.
_V0_1_STAMP = "true"
@dataclass(frozen=True)
class OwnershipPolicy:
"""The `generated` value this profile writes, and the values it owns back.
Ownership is the §3 collision gate's question — may this run replace the
file already sitting at a generated name? — and the answer is a profile's,
because the stamp differs per profile. v0.1 writes the literal `true`; v0.2
writes `generated: { by: <actor>, at: <ingested_at> }` (§5), where the actor
takes §7's `process:<id>` form.
`actor` is `None` for the v0.1 stamp. Where it is set it carries no version,
deliberately: the value sits inside a byte-compared golden, so a producer
version there would fire golden regression on every release without any
contract having changed, and would make a shared cross-implementation
fixture impossible by construction (plan V1(d), operator 2026-07-27).
Recognition is ONE-WAY, and both directions are decisions rather than
accidents. A v0.2 profile owns the v0.1 stamp as well, so a bundle written
under `DEFAULT` re-runs IN PLACE — the black-box promise is that an upstream
release costs a consumer a re-run and nothing more. The reverse is refused:
`DEFAULT` meeting a v0.2 file fails the run rather than replacing a file
whose shape it does not read (V-A3).
The v0.2 test is a PREFIX rather than an equality, because the value carries
`ingested_at` and therefore differs on every run by design. It works because
`parse_frontmatter` returns the whole flow mapping as one opaque string
(V-A2) — no structure this library cannot yet read is parsed here.
"""
actor: str | None = None
def stamp(self, ingested_at: str) -> str:
"""The `generated` value a run at `ingested_at` writes."""
if self.actor is None:
return _V0_1_STAMP
return f"{{ by: {self.actor}, at: {ingested_at} }}"
def owns(self, value: str) -> bool:
"""Whether a `generated` value read back marks this library's output."""
if value == _V0_1_STAMP:
return True
if self.actor is None:
return False
return value.startswith(f"{{ by: {self.actor},")
@dataclass(frozen=True)
class PathPolicy:
"""The filename namespaces the three doors write into.
Each prefix keeps its door's generated names disjoint from `index.md`, from
the other doors, and from `promoted-verdict-*`, for every id the grammar
admits. The id grammar itself stays in `materialize`: it is a pattern the
slugger derives a separator class from, not a name to configure.
"""
concept_suffix: str
ingest_prefix: str
inbox_prefix: str
import_prefix: str
def _split_frontmatter(text: str) -> tuple[dict[str, str], list[str]]:
"""The leading `---` block as ordered keys, and the body lines after it.
Line-oriented, the same shape `materialize.parse_frontmatter` reads —
duplicated rather than imported because `materialize` imports this module,
and because that one takes a path where an index reader has only text.
"""
lines = text.splitlines()
if not lines or lines[0].strip() != "---":
return {}, lines
head: dict[str, str] = {}
for offset, line in enumerate(lines[1:], start=1):
if line.strip() == "---":
return head, lines[offset + 1 :]
key, sep, value = line.partition(":")
if sep:
head[key.strip()] = value.strip()
return head, []
@dataclass(frozen=True)
class IndexViolation:
"""One way an index file departs from the policy.
Reported rather than raised, like `SchemaViolation`. `subject` is whatever
the finding is about — a line, a link target, or a frontmatter key — and
`code` is the stable machine-readable part.
"""
subject: str
reason: str
code: str
@dataclass(frozen=True)
class IndexPolicy:
"""The index file, the shape of its entries, and where indexes must exist.
`link_template` renders a managed line and `link_pattern` recognises one.
Both are carried because §6 does both — append on write, rewrite on
maintenance — and a round-trip test is what keeps the pair honest. The
pattern is anchored to the whole line by construction: removal keys on this
exact shape, never a bare substring, so curated prose that mentions a
target inline survives verbatim.
The judging fields are read by `violations` and `required_indexes`, and
every one of them is off under DEFAULT. They exist because two consumers
disagree about this file in ways no single shape can hold:
- `per_directory` — an index at every level, or only at the bundle root.
Upstream OKF binds `index.md` to the root ALONE, so a default demanding
one per level would declare upstream-conforming bundles invalid. Policy,
never an OKF rule.
- `heading_required` / `allows_prose` — the wiki's index is a generated
heading plus entries and nothing else; the catalog's *requires*
progressive-disclosure prose. Opposite requirements, both expressible.
- `entries_match_directory` — whether the index must match the directory
exactly, in both directions. The wiki enforces this at ERROR
(`BUNDLE_INDEX_COMPLETE`); the catalog holds that an index is an
AUTHORED count of a directory's children and that a validator
enumerating the directory has implemented the wrong contract. Neither
posture is baked in, and this library never enumerates anything: the
caller supplies the listing, and a policy that does not judge it refuses
to be handed one at all — so code written to the wrong contract fails at
the call rather than passing every test one would think to write.
- `root_frontmatter` — the ordered keys the ROOT index may carry, where
nested indexes carry none. Confirmed independently in both consumers, so
the asymmetry is shape rather than one repo's preference. Naming a key
here PERMITS it and fixes its position; it does not demand it.
- `root_frontmatter_required` — the subset that must actually be present.
Separate from the tuple above because the two are different claims, and
collapsing them was P1-F1: `OKF_V0_2` names `okf_version` to fix its
position, but upstream §8/§12 grant it as a MAY, and judging the pair as
one field failed 14 of 17 real bundles — upstream's own four included —
each with exactly this one violation, while our emitter treated the same
key as optional. A profile whose consumer really does demand the keys
says so here, which is what `STRICT_V1` does.
"""
name: str
link_template: str
link_pattern: re.Pattern[str]
per_directory: bool = False
heading_required: bool = False
allows_prose: bool = True
entries_match_directory: bool = False
root_frontmatter: tuple[str, ...] = ()
root_frontmatter_required: frozenset[str] = field(default_factory=frozenset)
def __post_init__(self) -> None:
stray = sorted(self.root_frontmatter_required - set(self.root_frontmatter))
if stray:
raise ValueError(
f"required root keys must be named in the ordered set "
f"(got {', '.join(repr(key) for key in stray)}) — a key demanded "
"but never named could not be judged for position"
)
if self.requires_description and "description" not in self.link_pattern.groupindex:
raise ValueError(
"link_template names {description} but link_pattern has no "
"'description' group — the pair would round-trip in one "
"direction only"
)
@property
def requires_description(self) -> bool:
"""Whether an entry carries a description alongside label and target."""
return "{description}" in self.link_template
def render_link(self, label: str, target: str, description: str | None = None) -> str:
if self.requires_description and description is None:
raise ValueError(
"this index policy's entries carry a description; rendering "
"without one emits a half-written entry that parses as nothing"
)
if not self.requires_description and description is not None:
raise ValueError(
"this index policy's entries carry no description; the value "
"offered would be dropped silently"
)
return self.link_template.format(label=label, target=target, description=description)
def required_indexes(self, directories: Sequence[str]) -> tuple[str, ...]:
"""The index paths this policy requires, given the caller's directories.
Bundle-relative, with `""` for the root. The directories are an
argument precisely so that nothing here reaches the filesystem.
"""
if not self.per_directory:
return (self.name,)
return tuple(sorted(f"{d}/{self.name}" if d else self.name for d in directories))
def violations(
self,
text: str,
*,
is_root: bool = False,
expected_targets: Collection[str] | None = None,
) -> tuple[IndexViolation, ...]:
"""Every way `text` departs from this policy, deterministically ordered.
`expected_targets` is the directory's contents as the CALLER sees them,
required exactly when `entries_match_directory` is set and refused
otherwise. Sorted by `(subject, code)` so two runs over the same file
agree.
"""
if self.entries_match_directory and expected_targets is None:
raise ValueError(
"this index policy judges the index against the directory, so "
"a listing is required — skipping the check silently would "
"pass a gate whose input never arrived"
)
if not self.entries_match_directory and expected_targets is not None:
raise ValueError(
"this index policy's index is authored rather than derived "
"from the directory, so a listing has nothing to judge"
)
found: list[IndexViolation] = []
head, body = _split_frontmatter(text)
found.extend(self._frontmatter_violations(head, is_root=is_root))
headings: list[str] = []
listed: set[str] = set()
for line in body:
stripped = line.strip()
if not stripped:
continue
if line.startswith("# "):
headings.append(line)
continue
match = self.link_pattern.match(line)
if match is not None:
listed.add(match.group("target"))
continue
if not self.allows_prose:
found.append(
IndexViolation(
stripped,
"is neither the heading nor an entry",
"index_prose_not_allowed",
)
)
if self.heading_required:
if not headings:
found.append(
IndexViolation("", "the index carries no `# ` heading", "index_heading_missing")
)
found.extend(
IndexViolation(extra, "is a second `# ` heading", "index_heading_extra")
for extra in headings[1:]
)
if expected_targets is not None:
expected = set(expected_targets)
found.extend(
IndexViolation(
target, "is in the directory but not in the index", "index_entry_missing"
)
for target in expected - listed
)
found.extend(
IndexViolation(
target, "is in the index but not in the directory", "index_entry_unexpected"
)
for target in listed - expected
)
return tuple(sorted(found, key=lambda violation: (violation.subject, violation.code)))
def _frontmatter_violations(
self, head: Mapping[str, str], *, is_root: bool
) -> list[IndexViolation]:
if not self.root_frontmatter:
return []
if not is_root:
# One finding about the block, not one per key: a nested index
# carrying frontmatter is a single structural fact.
first = next(iter(head), None)
if first is None:
return []
return [
IndexViolation(
first,
"is frontmatter on a nested index, which carries none",
"index_root_frontmatter_unexpected",
)
]
found = [
IndexViolation(key, "is pinned on the root index and absent", "index_root_key_missing")
for key in self.root_frontmatter
if key in self.root_frontmatter_required and key not in head
]
present = [key for key in head if key in self.root_frontmatter]
declared = [key for key in self.root_frontmatter if key in head]
if present != declared:
found.append(
IndexViolation(
present[0],
f"breaks the pinned key order {', '.join(self.root_frontmatter)}",
"index_root_key_order",
)
)
return found
@dataclass(frozen=True)
class BundleProfile:
"""One bundle contract: types, frontmatter, filenames, index."""
types: TypePolicy
frontmatter: FrontmatterSchema
paths: PathPolicy
index: IndexPolicy
ownership: OwnershipPolicy = field(default_factory=OwnershipPolicy)
# The ingest-spec v1 + Phase 2 contract, unchanged. Every value here was a
# constant in `manifest`, `materialize`, `inbox` or `importer` before this
# module existed; the golden suite is what proves the move changed no bytes.
DEFAULT = BundleProfile(
types=TypePolicy(allowed=None),
frontmatter=FrontmatterSchema(
# Door A's seven keys and Door B's six, merged into one order that
# contains both as subsequences — neither door's output moves.
order=(
"type",
"title",
"source_system",
"source_query",
"source_file",
"source_sha256",
"ingested_at",
"ingest_manifest",
"generated",
),
collapsed_keys=frozenset({"source_query"}),
),
paths=PathPolicy(
concept_suffix=".md",
ingest_prefix="ingest-",
inbox_prefix="inbox-",
import_prefix="import-",
),
index=IndexPolicy(
name="index.md",
link_template="- [{label}]({target})",
link_pattern=re.compile(r"^- \[(?P<label>[^\]]*)\]\((?P<target>[^)]+)\)$"),
),
)
# `FRONTMATTER_KEY_ORDER` in the proving consumer's `bundle.py`: an eleven-key
# allowlist that doubles as canonical emission order. The first eight are the
# required set their operator ratified on 2026-07-25 (measured present on
# 522/522 documents); the last three are layer-specific and stay optional.
# Deriving both sets from this one tuple is what keeps them from drifting apart
# — their BUNDLE_HASH_REGISTRY gate depends on the order.
_STRICT_V1_KEY_ORDER = (
"type",
"title",
"description",
"timestamp",
"layer",
"source_tier",
"source_url",
"source_sha",
"version",
"date",
"summary",
)
# The `claude-code-llm-wiki` contract, from `docs/phase-3-split-table.md`.
#
# Deliberately stricter than the consumer's own validator: their REQUIRED_KEYS
# constant is the first four keys, which is their emit-path minimum rather than
# their contract. If they ever ship a document class that legitimately lacks
# `source_sha`, the agreed resolution is to bump this profile — they send the
# requirement before shipping the class, not after.
#
# `paths` is DEFAULT's and that is the whole truth for it: the filename
# namespaces are an ingest-spec invariant about what the doors write, not a
# consumer preference.
#
# The index shape is read out of their code rather than recalled:
# `bundle.py:498-524` renders `* [Title](target) - description` with
# subdirectories linking to their own index; `bundle.py:527-567` walks every
# level, writing frontmatter at the bundle root only and a bare `# ` heading
# below it; `validate.py:1081-1120` (gate BUNDLE_INDEX_COMPLETE, ERROR) demands
# the index and the directory match exactly in both directions.
STRICT_V1 = BundleProfile(
types=TypePolicy(allowed=frozenset({"Concept", "Guide", "Reference", "Release"})),
frontmatter=FrontmatterSchema(
order=_STRICT_V1_KEY_ORDER,
required=frozenset(_STRICT_V1_KEY_ORDER[:8]),
allowed=frozenset(_STRICT_V1_KEY_ORDER),
nullable=frozenset({"summary"}),
key_pattern=re.compile(r"^[a-z_]+$"),
),
paths=DEFAULT.paths,
index=IndexPolicy(
name="index.md",
link_template="* [{label}]({target}) - {description}",
link_pattern=re.compile(
r"^\* \[(?P<label>[^\]]*)\]\((?P<target>[^)\s]+)\) - (?P<description>.+)$"
),
per_directory=True,
heading_required=True,
allows_prose=False,
entries_match_directory=True,
root_frontmatter=("okf_version", "bundle_profile", "okf_spec_commit"),
# The proving consumer demands all three, not merely permits them: their
# root index carries exactly these keys in exactly this order on every
# bundle measured (`c5141f8`). Stated explicitly so that separating
# "permitted" from "required" costs them nothing.
root_frontmatter_required=frozenset({"okf_version", "bundle_profile", "okf_spec_commit"}),
),
)
# OKF v0.2, as an ADDITIVE profile: `DEFAULT` states commons' ingest-spec §5
# layer and keeps stating it, so nothing here migrates anything. The key order
# is DEFAULT's followed by the §5 families v0.2 adds, which is also the order
# upstream's own reference bundles emit them in (`generated` before `sources`).
#
# Naming a family is not writing it. `verified`, `status` and `stale_after` are
# expressible so a caller can emit them in canonical order and so the schema can
# judge a document that carries them; Door A writes none of them, because a
# field with no reader is not written. Their structured v0.2 values — block
# lists of mappings — are beyond this library's value model until D1's reader
# lands; that is a reader gap, not an ordering one.
_OKF_V0_2_KEY_ORDER = (
*DEFAULT.frontmatter.order,
"sources",
"verified",
"status",
"stale_after",
)
# PROVISIONAL. Shipped first as a pre-release (`v0.5.0a1`) to a named pilot set
# — `portfolio-optimiser-claude`, the plugin marketplace catalog, and
# `claude-code-llm-wiki` — and this surface may change on their feedback without
# a deprecation cycle. Saying so is what buys the freedom to act on the
# feedback; discovering it later is what would make the pilot a de-facto
# release. The versioned constants are the stable binding.
#
# Two things this profile deliberately does NOT do:
#
# - **It closes nothing.** §14 forbids a conformant consumer to reject on an
# unknown `type` value or on unknown additional keys, so an allowlist or a key
# pattern here would put the profile in violation of the version it is named
# for. `type` is required and is the only one (§4, §11).
# - **It NAMES `okf_version` but never carries its value.** The value tracks the
# upstream Google version and belongs to catalog (decision E1), so a constant
# here would be this repo claiming a decision it does not own — and the one
# that would have to be chased on every upstream release. The caller supplies
# it (`materialize_bundle(..., root_frontmatter_values=...)`); this policy
# fixes only the key and its position.
#
# WHERE it goes was open until 2026-07-31 between upstream's root-index
# frontmatter block and catalog's body-line convention. Catalog verified
# upstream themselves at the pinned commit `3fcbb9f` and reported §8:509-510
# ("Index files contain no frontmatter, with one exception: a bundle-root
# `index.md` MAY carry an `okf_version` key") and §12:773-775 ("in a
# bundle-root `index.md` frontmatter block (the only place frontmatter is
# permitted in an `index.md`)"). Frontmatter it is; their own spec diverges
# from upstream here, and that divergence is theirs to resolve.
#
# Declaring it stays a MAY: none of upstream's four reference bundles carries
# the key at all (catalog grepped `okf/bundles` and `okf/samples`: zero hits),
# so omitting `root_frontmatter_values` emits no block. The JUDGE says the same
# thing since P1-F1 — the key is named here and left out of
# `root_frontmatter_required` — because for one release it did not, and a
# bundle exercising the MAY was reported as violating by the very profile that
# had emitted it correctly.
#
# **Measured limitation (guard 0.2.0, 2026-07-26):** a bundle emitted under this
# profile cannot be read back through a guard-gated import. The guard's T2
# frontmatter grammar admits scalars and flat lists of strings, and refuses every
# route to a mapping — flow on the disallowed-indicator set, block on the
# nested-mapping check, dotted keys on the key pattern. So `generated` as the
# mapping v0.2 specifies has no expressible form through that gate at all. This
# binds what can be IMPORTED (Door C), never what we emit: Door B's
# `screen_output` does not run that parser.
OKF_V0_2 = BundleProfile(
types=TypePolicy(allowed=None),
frontmatter=FrontmatterSchema(
order=_OKF_V0_2_KEY_ORDER,
collapsed_keys=DEFAULT.frontmatter.collapsed_keys,
required=frozenset({"type"}),
),
paths=DEFAULT.paths,
# DEFAULT's index in every respect but one: the root MAY carry `okf_version`
# (§8, §12). Built with `replace` rather than restated so a later change to
# the shared shape cannot drift between the two.
index=replace(DEFAULT.index, root_frontmatter=("okf_version",)),
ownership=OwnershipPolicy(actor="process:llm-ingestion-okf"),
)
# "The latest version supported as STABLE", not the latest present in this
# module. It therefore keeps v0.1 semantics for as long as v0.2 is provisional,
# and flipping it is the GA event — one auditable action rather than a side
# effect of a merge.
#
# The tradeoff is stated rather than hidden: an alias that moves means a consumer
# bound to it inherits upstream's breaking changes on a library upgrade. The
# versioned constants are the stable binding and are what a consumer should pin;
# this is for callers who have explicitly opted into tracking.
OKF_LATEST = DEFAULT

27
tests/test_golden.py
View file

@ -14,9 +14,11 @@ from pathlib import Path
import pytest
from llm_ingestion_okf.materialize import materialize_bundle
from llm_ingestion_okf.profiles import OKF_V0_2
EXAMPLES = Path(__file__).parent.parent / "examples"
INGESTED_AT_NAME = "ingested-at.txt"
OKF_VERSION_NAME = "okf-version.txt"
def fixture_backed_get(fixture_dir: Path) -> object:
@ -42,11 +44,34 @@ def materialize_case(case_dir: Path, out_dir: Path, monkeypatch: pytest.MonkeyPa
allow_network=True,
http_get=fixture_backed_get(case_dir / "fixture"), # type: ignore[arg-type]
)
elif case_dir.name.endswith("okf-v0-2"):
# The declared version is fixture DATA, read from the case, never a
# constant in this file. Its value tracks the upstream Google version
# and belongs to catalog (E1); a literal here would be this repo
# claiming a decision it does not own, and the golden would then have
# to be chased on every upstream release.
materialize_bundle(
case_dir / "manifest.json",
out_dir,
ingested_at,
profile=OKF_V0_2,
root_frontmatter_values={
"okf_version": (case_dir / OKF_VERSION_NAME).read_text(encoding="utf-8").strip()
},
)
else:
materialize_bundle(case_dir / "manifest.json", out_dir, ingested_at)
@pytest.mark.parametrize("case", ["ingest-golden-file", "ingest-golden-sql", "ingest-golden-http"])
@pytest.mark.parametrize(
"case",
[
"ingest-golden-file",
"ingest-golden-sql",
"ingest-golden-http",
"ingest-golden-okf-v0-2",
],
)
def test_golden_case_byte_for_byte(
case: str, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:

333
tests/test_index_policy.py Normal file
View file

@ -0,0 +1,333 @@
"""The index policy as configuration, and the reader that judges it (Phase 3).
Three index shapes have to be expressible without either becoming the default:
- **DEFAULT** — the ingest-spec shape. One index at the bundle root, entries
`- [label](target)`, curated prose alongside them, and no relationship
between the index and the directory it sits in.
- **STRICT_V1** — `claude-code-llm-wiki`, read out of their code rather than
recalled: an index at every level (`bundle.py:527-567`), exactly one `# `
heading, entries `* [Title](link) - description` and nothing else, the root
index carrying `okf_version`/`bundle_profile`/`okf_spec_commit` while nested
indexes carry no frontmatter at all, and gate `BUNDLE_INDEX_COMPLETE`
(`validate.py:1081-1120`, severity ERROR) demanding the index and the
directory match exactly in both directions.
- **The second-brain shape** — not built here, but it must not be excluded.
The catalog reported on 2026-07-25 that their index is an *authored* count
of a directory's children and never a filesystem lookup, that a reader or
validator enumerating the directory has implemented the wrong contract, and
that they require prose in the index but no per-entry description.
The two consumers are directly opposed on exactly one point: the wiki mandates
the directory-derived match the catalog calls the wrong contract. That is what
settles the design. Every one of these differences is a policy field, none is
a library invariant, and `violations` never touches the filesystem — a caller
supplies the listing, and a profile whose index is authored refuses to be
handed one at all.
"""
from __future__ import annotations
import re
import pytest
from llm_ingestion_okf.profiles import DEFAULT, OKF_V0_2, STRICT_V1, IndexPolicy
# --- DEFAULT keeps the Phase 1/2 shape (assumption C1) --------------------
def test_default_index_policy_judges_nothing() -> None:
"""Every new field is off under DEFAULT, so the reader has nothing to say.
Door A emits an index with a curated summary above managed links, Door B
and Door C append to one that may be empty. A judging default would
declare invalid the bundles this library itself produces — the same
reasoning that kept a required key set off DEFAULT's frontmatter schema.
"""
text = "# Sales bundle\n\nCurated prose the operator wrote.\n\n- [Sales](ingest-sales.md)\n"
assert DEFAULT.index.violations(text) == ()
assert DEFAULT.index.violations(text, is_root=True) == ()
def test_default_index_is_root_only_because_upstream_binds_it_there() -> None:
"""Upstream OKF binds `index.md` to the bundle root alone.
A nested directory without its own index is not an error there, which is
why per-directory indexing is policy rather than an OKF rule: a DEFAULT
demanding one per level would declare upstream-conforming bundles invalid.
"""
assert DEFAULT.index.per_directory is False
assert DEFAULT.index.required_indexes(["", "concepts", "concepts/hooks"]) == ("index.md",)
def test_default_index_link_is_unchanged_and_needs_no_description() -> None:
assert DEFAULT.index.requires_description is False
assert DEFAULT.index.render_link("Sales", "ingest-sales.md") == "- [Sales](ingest-sales.md)"
# --- the description on an entry ------------------------------------------
def test_render_link_carries_a_description_when_the_template_names_one() -> None:
assert STRICT_V1.index.requires_description is True
line = STRICT_V1.index.render_link("Hooks", "hooks.md", "How hooks fire.")
assert line == "* [Hooks](hooks.md) - How hooks fire."
def test_render_link_refuses_to_emit_a_half_written_entry() -> None:
"""A template naming `{description}` and a caller omitting it must not
silently render `* [T](t) - `, which parses as nothing and reads as an
entry."""
with pytest.raises(ValueError, match="description"):
STRICT_V1.index.render_link("Hooks", "hooks.md")
def test_a_description_offered_to_a_template_without_one_is_refused() -> None:
"""The mirror case: silently dropping it would lose operator bytes."""
with pytest.raises(ValueError, match="description"):
DEFAULT.index.render_link("Sales", "ingest-sales.md", "dropped on the floor")
def test_strict_v1_link_template_and_pattern_describe_the_same_shape() -> None:
"""Rendering and recognising are separate operations; this is the pair's
round-trip, the same test DEFAULT's shape already carries."""
for label, target, description in (
("Hooks", "hooks.md", "How hooks fire."),
("Concepts", "concepts/index.md", "Machine-generated index of concepts."),
("", "a.md", "-"),
):
line = STRICT_V1.index.render_link(label, target, description)
match = STRICT_V1.index.link_pattern.match(line)
assert match is not None
assert match.group("label") == label
assert match.group("target") == target
assert match.group("description") == description
def test_strict_v1_link_pattern_is_anchored_to_the_whole_line() -> None:
assert STRICT_V1.index.link_pattern.match("see * [Hooks](hooks.md) - x below") is None
# --- per-directory indexes ------------------------------------------------
def test_per_directory_policy_names_one_index_per_level() -> None:
"""The caller supplies the directories; the policy never enumerates them.
This is the catalog's construction rule honoured at the type level: the
library cannot list a directory because it is never given one to list.
"""
assert STRICT_V1.index.per_directory is True
assert STRICT_V1.index.required_indexes(["", "concepts", "concepts/hooks"]) == (
"concepts/hooks/index.md",
"concepts/index.md",
"index.md",
)
# --- the reader: headings -------------------------------------------------
ROOT_HEAD = "---\nokf_version: 0.1\nbundle_profile: strict-v1\nokf_spec_commit: d44368c\n---\n"
def test_strict_index_wants_exactly_one_h1() -> None:
body = "\n* [Hooks](hooks.md) - How hooks fire.\n"
missing = STRICT_V1.index.violations(
ROOT_HEAD + body, is_root=True, expected_targets={"hooks.md"}
)
assert [violation.code for violation in missing] == ["index_heading_missing"]
two = ROOT_HEAD + "\n# Wiki\n\n# Again\n" + body
extra = STRICT_V1.index.violations(two, is_root=True, expected_targets={"hooks.md"})
assert [violation.code for violation in extra] == ["index_heading_extra"]
assert extra[0].subject == "# Again"
def test_a_second_level_heading_is_prose_not_a_heading() -> None:
"""Only `# ` is the heading; `## ` is body text the strict shape forbids."""
text = ROOT_HEAD + "\n# Wiki\n\n## Section\n"
codes = [v.code for v in STRICT_V1.index.violations(text, is_root=True, expected_targets=set())]
assert codes == ["index_prose_not_allowed"]
# --- the reader: prose ----------------------------------------------------
def test_strict_index_admits_nothing_but_the_heading_and_the_entries() -> None:
text = ROOT_HEAD + "\n# Wiki\n\nA sentence the generator would never write.\n"
found = STRICT_V1.index.violations(text, is_root=True, expected_targets=set())
assert [v.code for v in found] == ["index_prose_not_allowed"]
assert found[0].subject == "A sentence the generator would never write."
def test_prose_is_the_catalog_shape_and_stays_expressible() -> None:
"""Their index requires progressive-disclosure prose and no per-entry
description — the opposite of the wiki on both counts, and both have to be
constructible."""
second_brain = IndexPolicy(
name="index.md",
link_template="- [{label}]({target})",
link_pattern=DEFAULT.index.link_pattern,
per_directory=True,
heading_required=True,
allows_prose=True,
root_frontmatter=("okf_version", "okf_layout"),
)
text = "# Notes\n\nWhat lives here and why.\n\n- [A note](a.md)\n"
assert second_brain.violations(text) == ()
assert second_brain.entries_match_directory is False
# --- the reader: directory match, both directions -------------------------
def test_the_match_is_bidirectional_and_names_each_side() -> None:
text = ROOT_HEAD + "\n# Wiki\n\n* [Hooks](hooks.md) - How hooks fire.\n"
found = STRICT_V1.index.violations(
text, is_root=True, expected_targets={"skills.md", "concepts/index.md"}
)
assert [(v.subject, v.code) for v in found] == [
("concepts/index.md", "index_entry_missing"),
("hooks.md", "index_entry_unexpected"),
("skills.md", "index_entry_missing"),
]
def test_a_derived_index_cannot_be_judged_without_the_listing() -> None:
"""Skipping the check silently is the failure mode that matters: a gate
that quietly passes when its input is absent is worse than one that
raises."""
text = ROOT_HEAD + "\n# Wiki\n"
with pytest.raises(ValueError, match="listing"):
STRICT_V1.index.violations(text, is_root=True)
def test_an_authored_index_refuses_a_listing() -> None:
"""The catalog's rule, enforced rather than documented: a profile whose
index is authored cannot be asked to check it against the directory, so
code written to the wrong contract fails at the call instead of passing
every test anyone would think to write."""
with pytest.raises(ValueError, match="authored"):
DEFAULT.index.violations("- [Sales](ingest-sales.md)\n", expected_targets={"a.md"})
# --- the reader: root vs nested frontmatter -------------------------------
def test_the_root_index_pins_its_keys_in_order() -> None:
text = ROOT_HEAD + "\n# Wiki\n"
assert STRICT_V1.index.violations(text, is_root=True, expected_targets=set()) == ()
def test_a_root_index_missing_a_pinned_key_is_reported_per_key() -> None:
text = "---\nokf_version: 0.1\n---\n\n# Wiki\n"
found = STRICT_V1.index.violations(text, is_root=True, expected_targets=set())
assert [(v.subject, v.code) for v in found] == [
("bundle_profile", "index_root_key_missing"),
("okf_spec_commit", "index_root_key_missing"),
]
def test_the_pinned_keys_are_ordered_not_merely_present() -> None:
"""Their `ROOT_INDEX_KEY_ORDER` is what makes a regeneration byte-identical,
so an out-of-order root index is a finding even with every key present."""
text = "---\nbundle_profile: strict-v1\nokf_version: 0.1\nokf_spec_commit: d44368c\n---\n\n# Wiki\n"
found = STRICT_V1.index.violations(text, is_root=True, expected_targets=set())
assert [v.code for v in found] == ["index_root_key_order"]
# --- P1-F1: permitted-and-ordered is not the same claim as required -------
def test_a_permitted_root_key_may_be_omitted_without_a_finding() -> None:
"""P1-F1, measured 2026-07-31: `OKF_V0_2` names `okf_version` because
upstream §8/§12 say a bundle-root index MAY carry it — a permission, not a
demand. Judging the key as required made 14 of the 17 bundles swept fail
with exactly this one violation, upstream's own four reference bundles
among them, while D5's emitter treats the same key as optional (omitting
`root_frontmatter_values` writes no block). We emitted a MAY correctly and
graded it a MUST."""
text = "# Bundle\n\n- [Sales](ingest-sales.md)\n"
assert OKF_V0_2.index.violations(text, is_root=True) == ()
def test_the_permitted_key_is_still_ordered_and_still_root_only() -> None:
"""Separating the two meanings must not cost the checks that were right:
a nested index carrying the key is still a finding, and a root index that
carries it is still judged for order."""
nested = OKF_V0_2.index.violations("---\nokf_version: 0.2\n---\n\n# Concepts\n")
assert [(v.subject, v.code) for v in nested] == [
("okf_version", "index_root_frontmatter_unexpected")
]
root = OKF_V0_2.index.violations("---\nokf_version: 0.2\n---\n\n# Bundle\n", is_root=True)
assert root == ()
def test_a_required_root_key_is_still_reported_when_absent() -> None:
"""`STRICT_V1` mirrors the proving consumer's ratified contract, which
demands all three (they emit all three — measured across their bundle at
`c5141f8`). The fix separates the meanings; it does not relax them."""
found = STRICT_V1.index.violations(
"---\nokf_version: 0.1\n---\n\n# Wiki\n", is_root=True, expected_targets=set()
)
assert [(v.subject, v.code) for v in found] == [
("bundle_profile", "index_root_key_missing"),
("okf_spec_commit", "index_root_key_missing"),
]
def test_a_required_key_outside_the_ordered_set_fails_at_construction() -> None:
"""A key demanded but never named in the ordered tuple could never be
judged for position — the same contradiction `FrontmatterSchema` refuses
when `required` strays outside `allowed`."""
with pytest.raises(ValueError, match="ordered"):
IndexPolicy(
name="index.md",
link_template="- [{label}]({target})",
link_pattern=DEFAULT.index.link_pattern,
root_frontmatter=("okf_version",),
root_frontmatter_required=frozenset({"bundle_profile"}),
)
def test_a_nested_index_carries_no_frontmatter_at_all() -> None:
"""Confirmed independently in two consumers: the wiki writes frontmatter
only at the bundle root (`bundle.py:544-557`), and the catalog reports
their `okf_version`/`okf_layout` markers as root-exclusive. The asymmetry
is shape, not one repo's preference."""
found = STRICT_V1.index.violations(ROOT_HEAD + "\n# Concepts\n", expected_targets=set())
assert [(v.subject, v.code) for v in found] == [
("okf_version", "index_root_frontmatter_unexpected")
]
def test_a_nested_index_without_frontmatter_is_clean() -> None:
assert STRICT_V1.index.violations("# Concepts\n", expected_targets=set()) == ()
# --- determinism ----------------------------------------------------------
def test_violations_are_sorted_so_two_runs_agree() -> None:
text = "---\nokf_spec_commit: d\nokf_version: 0.1\n---\n\nz prose\na prose\n"
found = STRICT_V1.index.violations(text, is_root=True, expected_targets={"m.md"})
assert list(found) == sorted(found, key=lambda v: (v.subject, v.code))
assert found == STRICT_V1.index.violations(text, is_root=True, expected_targets={"m.md"})
def test_index_policy_stays_frozen() -> None:
with pytest.raises(Exception):
STRICT_V1.index.per_directory = False # type: ignore[misc]
def test_the_pattern_group_names_the_template_needs_are_present() -> None:
"""A template naming `{description}` whose pattern cannot recover one is a
pair that round-trips in only one direction."""
with pytest.raises(ValueError, match="description"):
IndexPolicy(
name="index.md",
link_template="* [{label}]({target}) - {description}",
link_pattern=re.compile(r"^\* \[(?P<label>[^\]]*)\]\((?P<target>[^)]+)\)$"),
)

343
tests/test_okf_v0_2_characterization.py Normal file
View file

@ -0,0 +1,343 @@
"""The safety net under OKF v0.2 support (plan steps 1-3).
Every test here characterizes behavior this library ALREADY has, before any
v0.2 code exists. Nothing is changed by them; that is the point. They are
written first because the one part of the v0.2 work we called risky —
`_is_ingest_owned`, the pre-mutation collision gate — is only safe to touch
with its current guarantees pinned.
Three groups, one per plan step:
1. **Characterization.** An inline flow mapping survives the scalar
`parse_frontmatter` verbatim (V-A2), a block list does not, and a v0.2
`generated` mapping can never make a file ingest-owned (V-A3). The last one
converts today's fail-safe from an accident into a documented guarantee:
a v0.2 file at a target name is refused, never overwritten.
2. **Door C consumer tolerance** (D3). OKF §14 forbids a consumer to reject on
five things; Door C is a consumer in that sense. One case per MUST NOT, plus
the §5.2 bare-`verified` mapping.
3. **No profile hard-codes an upstream version** (V4/V-A5). The `okf_version`
VALUE belongs to catalog (decision E1); a profile names the key and expresses
any value, which is what keeps "always the latest OKF version" from turning
into a constant this repo has to chase.
"""
from __future__ import annotations
import dataclasses
import json
import re
from collections.abc import Iterator
from pathlib import Path
from typing import Any
import pytest
from llm_ingestion_okf.errors import MaterializationError
from llm_ingestion_okf.materialize import (
_is_ingest_owned,
materialize_bundle,
parse_frontmatter,
)
from llm_ingestion_okf.profiles import DEFAULT, STRICT_V1, BundleProfile, FrontmatterSchema
from test_import_flow import StubImportGate, place, run
INGESTED_AT = "2026-07-25T12:00:00Z"
# The v0.2 `generated` shape, as an inline flow mapping — the form A-E3 pins
# for the coming profile, and the form the §5 single-line MUST admits.
V0_2_GENERATED = "{ by: llm-ingestion-okf/0.4.0, at: 2026-07-25T12:00:00Z }"
# --- step 1: what the scalar parser already does --------------------------
def test_an_inline_flow_mapping_survives_the_scalar_parser_verbatim(tmp_path: Path) -> None:
"""V-A2. `parse_frontmatter` splits on the FIRST colon, so a flow mapping —
colons and all — comes back as one opaque string. Byte-exact both ways:
what the emitter wrote is what the parser returns.
"""
values = {"type": "dataset", "title": "Orders", "generated": V0_2_GENERATED}
path = tmp_path / "concept.md"
path.write_bytes(f"---\n{DEFAULT.frontmatter.emit(values)}\n---\n\nBody.\n".encode())
assert path.read_bytes().decode("utf-8").splitlines()[3] == f"generated: {V0_2_GENERATED}"
assert parse_frontmatter(path) == values
def test_a_block_list_pollutes_the_scalar_parsers_key_space(tmp_path: Path) -> None:
"""The measured reason `sources` is emitted as an inline flow sequence.
Upstream's canonical `sources` is a block list of multi-key mappings. Read
through this line-oriented parser, each item line becomes a KEY: the list
disappears and `- id` / `resource` appear as frontmatter keys that no
document declared. `_is_ingest_owned` reads through this same parser, so
emitting the block form would have forced the gate and the parser to be
hardened in one step.
"""
path = tmp_path / "concept.md"
path.write_bytes(
b"---\ntype: dataset\nsources:\n - id: margin-standard\n"
b" resource: policies/margin-standard.md\n---\n\nBody.\n"
)
parsed = parse_frontmatter(path)
assert parsed["sources"] == ""
assert parsed["- id"] == "margin-standard"
assert parsed["resource"] == "policies/margin-standard.md"
def _stamped_concept(generated: str) -> bytes:
return (
"---\ntype: dataset\ntitle: Orders\n"
"ingest_manifest: manifest@0123456789abcdef\n"
f"generated: {generated}\n---\n\nBody.\n"
).encode()
def test_a_v0_1_generated_true_is_ingest_owned(tmp_path: Path) -> None:
"""The control for the test below: the stamp as v0.1 writes it DOES own."""
path = tmp_path / "ingest-orders.md"
path.write_bytes(_stamped_concept("true"))
assert _is_ingest_owned(path, "manifest") is True
def test_a_v0_2_generated_mapping_is_never_ingest_owned(tmp_path: Path) -> None:
"""V-A3. `generated` as a v0.2 mapping is not the string `true`, so the
predicate returns False and the file is not ours to replace.
The manifest reference is readable and matches — the refusal comes from
`generated` alone. This is also why `generated` can never become the
ownership predicate on its own (U3): upstream writes it for HAND-AUTHORED
files, so presence proves nothing about who generated what.
"""
path = tmp_path / "ingest-orders.md"
path.write_bytes(_stamped_concept(V0_2_GENERATED))
assert parse_frontmatter(path)["ingest_manifest"] == "manifest@0123456789abcdef"
assert _is_ingest_owned(path, "manifest") is False
def test_a_v0_2_file_at_a_target_name_is_refused_not_overwritten(tmp_path: Path) -> None:
"""The fail-safe, end to end: not-owned means the §3 collision gate refuses
the run before any mutation. A v0.2 shape arriving in a bundle Door A also
writes into can cost a run; it can never cost data.
"""
src = tmp_path / "src"
src.mkdir()
manifest_path = src / "manifest.json"
manifest_path.write_text(
json.dumps(
{
"manifest_version": 1,
"source": {"type": "file", "id": "catalogue-1", "root": "data"},
"bundle_summary": "A test bundle.",
"extractions": [
{
"id": "orders",
"title": "Orders",
"query": "orders.csv",
"okf_type": "dataset",
"max_rows": 100,
}
],
}
),
encoding="utf-8",
)
(src / "data").mkdir()
(src / "data" / "orders.csv").write_text("a,b\n1,x\n", encoding="utf-8", newline="")
bundle = tmp_path / "bundle"
materialize_bundle(manifest_path, bundle, INGESTED_AT)
concept = bundle / "ingest-orders.md"
v0_2_bytes = concept.read_bytes().replace(
b"generated: true", f"generated: {V0_2_GENERATED}".encode()
)
concept.write_bytes(v0_2_bytes)
with pytest.raises(MaterializationError) as excinfo:
materialize_bundle(manifest_path, bundle, INGESTED_AT)
assert excinfo.value.code == "collision_unstamped"
assert concept.read_bytes() == v0_2_bytes
# --- step 2: Door C against the §14 consumer tolerance rules --------------
TOLERANCE_CASES = [
pytest.param(
"---\ntype: Concept\n---\n\nOnly a type.\n",
id="missing-optional-frontmatter",
),
pytest.param(
"---\ntype: Attested Computation\ntitle: Gross margin\n---\n\nA v0.2 type.\n",
id="unknown-type-value",
),
pytest.param(
"---\ntype: dataset\nstale_after: 2027-01-01\nnot_a_key_we_know: x\n---\n\nBody.\n",
id="unknown-additional-keys",
),
pytest.param(
"---\ntype: dataset\n---\n\nSee [the other one](does-not-exist.md).\n",
id="broken-cross-link",
),
pytest.param(
"---\ntype: dataset\nverified: { by: human:jsmith@acme, at: 2026-07-01T09:00:00Z }\n"
"---\n\nBody.\n",
id="bare-verified-mapping",
),
]
@pytest.mark.parametrize("document", TOLERANCE_CASES)
def test_door_c_merges_what_a_consumer_must_not_reject(tmp_path: Path, document: str) -> None:
"""D3/§14. Each case is one thing a conformant consumer MUST NOT reject.
Door C's tolerance is structural rather than lenient: it writes the guard's
bytes verbatim and never parses the sender's frontmatter, so there is no
place for a shape judgement to be made. That is what these cases pin — a
future reader (D1b) that starts judging frontmatter at this door would
break them, which is exactly when we want to hear about it.
The bare-`verified` case is the §5.2 MUST in its Door C form: the mapping is
persisted unmodified. Coercing it to a one-element list is the READER's
obligation and lands with D1b; nothing here reads it.
"""
place(tmp_path / "source", "notes/tolerated.md", document)
result, bundle = run(tmp_path, StubImportGate(), ingested_at=INGESTED_AT)
assert result.failed == ()
assert [entry.concept_path for entry in result.merged] == ["notes/tolerated.md"]
assert (bundle / "import-notes-tolerated.md").read_bytes() == document.encode("utf-8")
def test_door_c_does_not_require_the_source_bundle_to_carry_an_index(tmp_path: Path) -> None:
"""The fifth MUST NOT: a missing `index.md` is not grounds for rejection.
Door C generates the target index itself, so the sender's bundle need not
carry one — asserted explicitly here because every other test in the suite
omits the source index incidentally rather than as a stated rule.
"""
document = "---\ntype: dataset\n---\n\nNo index anywhere in the source.\n"
place(tmp_path / "source", "tables/users.md", document)
assert not (tmp_path / "source" / "index.md").exists()
result, bundle = run(tmp_path, StubImportGate(), ingested_at=INGESTED_AT)
assert result.failed == ()
assert [entry.concept_path for entry in result.merged] == ["tables/users.md"]
assert (bundle / "index.md").read_text(encoding="utf-8") == (
"- [tables/users](import-tables-users.md)\n"
)
# --- step 3: no profile hard-codes an upstream version --------------------
# A version LITERAL — `0.1`, `0.2`, `2026.7.3`. Compiled patterns are walked
# past deliberately: a regex is a shape, and a form gate for version values is
# the opposite of hard-coding one.
_VERSION_LITERAL_RE = re.compile(r"^\d+(\.\d+)+$")
def _strings_in(value: Any) -> Iterator[str]:
"""Every string reachable from a profile, through dataclasses and
collections alike. Recursive so that a version added to a nested policy —
the only place it could plausibly be added — is still found."""
if isinstance(value, str):
yield value
elif dataclasses.is_dataclass(value) and not isinstance(value, type):
for field in dataclasses.fields(value):
yield from _strings_in(getattr(value, field.name))
elif isinstance(value, (tuple, list, set, frozenset)):
for item in value:
yield from _strings_in(item)
@pytest.mark.parametrize("profile", [DEFAULT, STRICT_V1], ids=["DEFAULT", "STRICT_V1"])
def test_no_profile_hard_codes_an_upstream_version(profile: BundleProfile) -> None:
"""V4/V-A5. A profile names `okf_version` as a key and never carries its
value. The value tracks the upstream Google version and belongs to catalog
(decision E1), so a constant here would be this repo claiming a decision it
does not own — and the one that would have to be chased on every upstream
release.
"""
hard_coded = sorted({s for s in _strings_in(profile) if _VERSION_LITERAL_RE.match(s)})
assert hard_coded == []
def test_the_key_name_is_what_a_profile_pins(tmp_path: Path) -> None:
"""The other half of V4: naming the key is allowed and is what STRICT_V1
does. Without this, the test above would also pass on a profile that had
stopped requiring the declaration at all."""
assert "okf_version" in STRICT_V1.index.root_frontmatter
@pytest.mark.parametrize(
"schema_kwargs",
[
pytest.param({"order": ("timestamp", "generated")}, id="both-emitted"),
pytest.param(
{"order": ("timestamp",), "allowed": frozenset({"timestamp", "generated"})},
id="one-emitted-one-admitted",
),
pytest.param(
{"order": (), "required": frozenset({"timestamp", "generated"})},
id="both-required",
),
pytest.param(
{"order": ("timestamp", "title"), "nullable": frozenset({"generated"})},
id="one-emitted-one-nullable",
),
],
)
def test_a_schema_naming_both_timestamp_and_generated_cannot_be_built(
schema_kwargs: dict[str, Any],
) -> None:
"""V-A7. OKF §13.1 grants the `timestamp` fallback only while `generated` is
ABSENT, so a schema able to name both can describe a document with neither a
valid `generated.at` nor an eligible fallback. Refused at construction, the
same shape as the reserved `verdict` layer: a profile that could reach the
combination cannot be built, let alone handed to a door.
Every judging field names keys, `nullable` included. An open namespace
already admits `generated`, so a schema that also gives it a null rule has
named it exactly as surely as one that emits it — the last case is the
construction the proving consumer found the day the gate was written.
"""
with pytest.raises(ValueError, match="timestamp"):
FrontmatterSchema(**schema_kwargs)
def test_the_two_shipped_profiles_sit_on_opposite_sides_of_the_fallback() -> None:
"""Why the gate above costs nothing: neither shipped profile is near the
combination, and they are not near it in opposite directions. `DEFAULT`
emits `generated` and no `timestamp`; `STRICT_V1` emits `timestamp` and no
`generated`, which is precisely what puts the wiki on the §13.1 legacy path
rather than in a defect.
"""
assert "generated" in DEFAULT.frontmatter.order
assert "timestamp" not in DEFAULT.frontmatter.order
assert "timestamp" in STRICT_V1.frontmatter.order
assert "generated" not in STRICT_V1.frontmatter.order
assert STRICT_V1.frontmatter.allowed is not None
assert "generated" not in STRICT_V1.frontmatter.allowed
@pytest.mark.parametrize("declared", ["0.1", "0.2", "1.0", "2026.7.3"])
def test_the_root_frontmatter_policy_expresses_any_version_value(declared: str) -> None:
"""V4. The policy judges presence and ORDER, never the value — so the day
upstream ships v0.3, no index policy in this library has to change.
"""
text = (
f"---\nokf_version: {declared}\nbundle_profile: strict-v1\n"
"okf_spec_commit: 0123456789abcdef\n---\n\n# Bundle\n\n* [A](a.md) - a\n"
)
violations = STRICT_V1.index.violations(text, is_root=True, expected_targets={"a.md"})
assert violations == ()

339
tests/test_okf_v0_2_profile.py Normal file
View file

@ -0,0 +1,339 @@
"""D2 — the `OKF_V0_2` profile, profile-aware ownership, `sources` derivation.
Three things, in the order the risk runs:
1. **The profile constant.** Names the §5 families, requires only `type` (§4:
"the only always-required key"), and closes nothing — §14 forbids a consumer
to reject on unknown types or unknown keys, so an allowlist here would put
the profile in violation of the spec it is named after.
2. **Ownership.** `_is_ingest_owned` is the pre-mutation collision gate, so it
is the one piece of this work whose defects are expensive and quiet. The v0.2
stamp is a flow mapping carrying `ingested_at`, so equality is impossible by
construction and the predicate is a prefix test. The operator decided the
direction: `OKF_V0_2` accepts BOTH stamps, so a bundle written under `DEFAULT`
re-runs IN PLACE; `DEFAULT` does not accept the v0.2 form, so the recognition
is one-way and the fail-safe from V-A3 is preserved.
3. **`sources`.** Derived from the manifest's source as an inline flow sequence
(requirement 1's measured form), with the locator taken verbatim per source
type: `root`, `connection_ref`, `base_url`. `credential_ref` is never a
locator and never reaches frontmatter.
`generated.by` is `process:llm-ingestion-okf` — option (d) from the plan's V1,
chosen by the operator 2026-07-27 over the `<producer>/<version>` form A-E3 was
written against. The value carries no version, which is what keeps a byte-
compared fixture stable across releases. **A-E3 is now stale in the pilot
specification and owes po-claude a correction.**
"""
from __future__ import annotations
import re
from pathlib import Path
import pytest
from llm_ingestion_okf.errors import MaterializationError
from llm_ingestion_okf.manifest import (
Extraction,
FileSource,
HttpSource,
Manifest,
Source,
SqlSource,
)
from llm_ingestion_okf.materialize import _is_ingest_owned, _render_concept_file
from llm_ingestion_okf.profiles import DEFAULT, OKF_LATEST, OKF_V0_2, STRICT_V1, BundleProfile
INGESTED_AT = "2026-07-25T12:00:00Z"
STAMP = "manifest@0123456789abcdef"
V0_2_STAMP = f"{{ by: process:llm-ingestion-okf, at: {INGESTED_AT} }}"
def _manifest(source: Source) -> Manifest:
return Manifest(
manifest_version=1,
source=source,
bundle_summary="A test bundle.",
extractions=(
Extraction(
id="orders", title="Orders", query="orders.csv", okf_type="dataset", max_rows=10
),
),
)
def _concept(source: Source, *, profile: BundleProfile = OKF_V0_2) -> str:
manifest = _manifest(source)
return _render_concept_file(
manifest,
manifest.extractions[0],
"Body.\n",
ingested_at=INGESTED_AT,
stamp=STAMP,
profile=profile,
)
def _stamped_concept(generated: str) -> bytes:
return (
"---\ntype: dataset\ntitle: Orders\n"
f"ingest_manifest: {STAMP}\n"
f"generated: {generated}\n---\n\nBody.\n"
).encode()
# --- the profile constant -------------------------------------------------
def test_the_v0_2_profile_names_the_five_families_it_must_express() -> None:
"""C-E3's list, as emission order rather than as prose. Naming a key is what
lets a caller emit it in canonical order; judging a structured value is D1's
reader and deliberately absent here.
"""
order = OKF_V0_2.frontmatter.order
for key in ("generated", "sources", "verified", "status", "stale_after"):
assert key in order
assert "timestamp" not in order # §13.1: the fallback is for v0.1 documents
assert order.index("generated") < order.index("sources")
def test_the_v0_2_profile_requires_only_type() -> None:
"""§4/§11: `type` is the only always-required key, and a concept carrying
just `type` is fully conformant."""
assert OKF_V0_2.frontmatter.required == frozenset({"type"})
def test_the_v0_2_profile_closes_neither_the_type_set_nor_the_key_namespace() -> None:
"""§14's MUST NOTs, expressed as configuration: a consumer must not reject
on an unknown `type` value or on unknown additional keys. A profile that
closed either would be non-conformant against the version it is named for.
"""
assert OKF_V0_2.types.allowed is None
assert OKF_V0_2.frontmatter.allowed is None
assert OKF_V0_2.frontmatter.key_pattern is None
def test_the_v0_2_profile_keeps_the_source_query_collapse_and_the_default_paths() -> None:
"""The parts that are commons' §5 rather than upstream's: the doors write
the same filenames whatever the OKF version, and a multi-line SELECT still
renders on one line."""
assert OKF_V0_2.frontmatter.collapsed_keys == frozenset({"source_query"})
assert OKF_V0_2.paths == DEFAULT.paths
def test_okf_latest_still_means_v0_1_until_ga() -> None:
"""`OKF_LATEST` is "the latest version supported as STABLE", not the latest
present in the tree. Flipping it is the GA event — a single auditable
action rather than a side effect of this merge — so during the pilot a
consumer bound to the alias keeps v0.1 semantics.
"""
assert OKF_LATEST is DEFAULT
# --- ownership ------------------------------------------------------------
def test_the_default_profile_still_stamps_the_v0_1_literal() -> None:
assert DEFAULT.ownership.stamp(INGESTED_AT) == "true"
def test_the_v0_2_stamp_is_a_flow_mapping_whose_at_is_the_argument_exactly() -> None:
"""A-E3, corrected: the actor is the §7 `process:<id>` form and carries no
version. `at` binds to the already-validated `ingested_at` argument, so no
wall-clock is introduced and two runs with the same argument agree.
"""
assert OKF_V0_2.ownership.stamp(INGESTED_AT) == V0_2_STAMP
assert "llm-ingestion-okf/" not in OKF_V0_2.ownership.stamp(INGESTED_AT)
def test_the_v0_2_predicate_accepts_the_stamp_its_own_profile_writes(tmp_path: Path) -> None:
"""A-E5's second run: a v0.2 bundle re-runs into the directory it wrote."""
path = tmp_path / "ingest-orders.md"
path.write_bytes(_stamped_concept(V0_2_STAMP))
assert _is_ingest_owned(path, "manifest", profile=OKF_V0_2) is True
def test_the_v0_2_predicate_also_accepts_the_v0_1_stamp(tmp_path: Path) -> None:
"""The operator's decision (2026-07-27): a `DEFAULT`-written bundle is
re-runnable IN PLACE under `OKF_V0_2`. Without this the profile switch would
fire `collision_unstamped` on the library's own previous output, and the
black-box promise — an upstream release costs a consumer a re-run, nothing
more — would not hold across the profile change.
"""
path = tmp_path / "ingest-orders.md"
path.write_bytes(_stamped_concept("true"))
assert _is_ingest_owned(path, "manifest", profile=OKF_V0_2) is True
def test_the_default_predicate_still_refuses_the_v0_2_stamp(tmp_path: Path) -> None:
"""Recognition is one-way, which is what preserves V-A3. `DEFAULT` meeting a
v0.2 file refuses the run rather than replacing the file."""
path = tmp_path / "ingest-orders.md"
path.write_bytes(_stamped_concept(V0_2_STAMP))
assert _is_ingest_owned(path, "manifest") is False
assert _is_ingest_owned(path, "manifest", profile=DEFAULT) is False
def test_a_foreign_v0_2_actor_is_never_ingest_owned(tmp_path: Path) -> None:
"""U3 as a test: `generated` is attribution, and upstream writes it for
HAND-AUTHORED files. Presence proves nothing — the actor has to be ours, or
the collision gate would claim a human's file that merely carries the key.
"""
path = tmp_path / "ingest-orders.md"
path.write_bytes(_stamped_concept(f"{{ by: human:jsmith@acme, at: {INGESTED_AT} }}"))
assert _is_ingest_owned(path, "manifest", profile=OKF_V0_2) is False
def test_the_v0_2_stamp_still_has_to_name_this_manifest(tmp_path: Path) -> None:
"""§10.2 per-manifest ownership is unchanged by the profile: a sibling
manifest sharing the bundle keeps its own files."""
path = tmp_path / "ingest-orders.md"
path.write_bytes(_stamped_concept(V0_2_STAMP))
assert _is_ingest_owned(path, "other-manifest", profile=OKF_V0_2) is False
# --- `sources` derivation -------------------------------------------------
SOURCE_CASES = [
pytest.param(FileSource(id="golden-catalogue", root="fixture"), "fixture", id="file"),
pytest.param(
SqlSource(id="golden-db", connection_ref="OKF_GOLDEN_SQL_DB"),
"OKF_GOLDEN_SQL_DB",
id="sql",
),
pytest.param(
HttpSource(id="golden-api", base_url="https://golden.example.test"),
"https://golden.example.test",
id="http",
),
]
@pytest.mark.parametrize("source,resource", SOURCE_CASES)
def test_sources_is_an_inline_flow_sequence_carrying_the_locator(
source: Source, resource: str
) -> None:
"""A-E4. One entry, `id` from the manifest source and `resource` its locator
verbatim — the filesystem root, the env-var NAME holding the DSN, or the base
URL. Upstream's `resource` is a bundle-internal path; ours is not, and
inventing the `author`/`last_modified` upstream also carries would be writing
fields with no reader.
"""
line = f"sources: [{{ id: {source.id}, resource: {resource} }}]"
assert line in _concept(source).splitlines()
def test_sources_follows_generated_in_the_emitted_block() -> None:
"""The order the schema declares is the order that reaches disk."""
lines = _concept(FileSource(id="golden-catalogue", root="fixture")).splitlines()
assert [line.split(":")[0] for line in lines[1:9]] == [
"type",
"title",
"source_system",
"source_query",
"ingested_at",
"ingest_manifest",
"generated",
"sources",
]
def test_the_default_profile_emits_neither_sources_nor_a_v0_2_generated() -> None:
"""V-A6 at the unit level: the seven-key v0.1 block, byte for byte. The
golden suite proves the same thing end to end; this one localizes a failure
to the renderer rather than to a fixture diff.
"""
rendered = _concept(FileSource(id="golden-catalogue", root="fixture"), profile=DEFAULT)
assert rendered == (
"---\n"
"type: dataset\n"
"title: Orders\n"
"source_system: golden-catalogue\n"
"source_query: orders.csv\n"
f"ingested_at: {INGESTED_AT}\n"
f"ingest_manifest: {STAMP}\n"
"generated: true\n"
"---\n"
"\n"
"Body.\n"
)
def test_the_http_credential_reference_never_reaches_the_frontmatter() -> None:
"""A credential reference is not a locator. The convention forbids the
credential itself in frontmatter; the env-var NAME holding it has no reader
in a bundle either, so it is not written.
"""
source = HttpSource(
id="golden-api",
base_url="https://golden.example.test",
credential_ref="OKF_GOLDEN_API_TOKEN",
)
assert "OKF_GOLDEN_API_TOKEN" not in _concept(source)
@pytest.mark.parametrize(
"root",
["data, backup", "data{1}", "data[1]", "key: value"],
ids=["comma", "braces", "brackets", "colon-space"],
)
def test_a_locator_that_would_restructure_the_flow_mapping_is_refused(root: str) -> None:
"""Fail-fast rather than emit a silently WRONG provenance record.
These characters terminate or restructure a YAML flow mapping, and the
failure is quiet rather than loud. Measured with PyYAML 2026-07-27 rather
than argued: `[{ id: a, resource: data, backup }]` raises nothing and yields
`[{'id': 'a', 'resource': 'data', 'backup': None}]` — a clean parse into a
record no one wrote. Refusing is the same posture as the filename-length
gate: validation, not repair.
The same run confirms what IS emitted parses as intended — `generated` to a
two-key mapping, `sources` to a one-element list of one — including a base
URL, whose colons stay inside a plain scalar in flow context.
"""
with pytest.raises(MaterializationError) as excinfo:
_concept(FileSource(id="golden-catalogue", root=root))
assert excinfo.value.code == "source_reference_unquotable"
def test_the_default_profile_is_unaffected_by_an_unquotable_locator() -> None:
"""The refusal is a property of the v0.2 emission, not a new gate on v0.1:
`DEFAULT` never writes the locator, so it has nothing to refuse. A shared
refusal would have broken additivity — a manifest that ran yesterday must
still run."""
rendered = _concept(FileSource(id="golden-catalogue", root="data, backup"), profile=DEFAULT)
assert "data, backup" not in rendered
# --- V-A5 extended to the new profile -------------------------------------
_VERSION_LITERAL_RE = re.compile(r"^\d+(\.\d+)+$")
@pytest.mark.parametrize(
"profile", [DEFAULT, STRICT_V1, OKF_V0_2], ids=["DEFAULT", "STRICT_V1", "OKF_V0_2"]
)
def test_no_profile_hard_codes_an_upstream_version(profile: BundleProfile) -> None:
"""V-A5, restated over the profile the name of which is the one place a
version literal would look natural. `OKF_V0_2` is a NAME; the value belongs
to catalog (E1) and is declared at D5, in the fixture, once.
Kept alongside the characterization suite's copy rather than replacing it:
that one pins the two shipped profiles at the point they were measured.
"""
from test_okf_v0_2_characterization import _strings_in
assert sorted({s for s in _strings_in(profile) if _VERSION_LITERAL_RE.match(s)}) == []

15
tests/test_packaging.py
View file

@ -32,3 +32,18 @@ def test_the_only_runtime_dependency_is_the_security_boundary() -> None:
tomllib = pytest.importorskip("tomllib") # stdlib from 3.11; the pin holds on 3.10 too
pyproject = tomllib.loads((PROJECT_ROOT / "pyproject.toml").read_text(encoding="utf-8"))
assert pyproject["project"]["dependencies"] == ["llm-ingestion-guard>=0.2,<0.3"]
def test_the_declared_version_agrees_with_the_packaged_one() -> None:
"""The two places a version is written must not drift apart.
The install channel is a direct git reference, so a consumer pins a TAG
while pip records `project.version`. Nothing in the run path reads
`__version__` — which is exactly why a stale one survives a green suite,
and why a consumer installing at a pre-release tag can end up with a
package that reports the previous release. This is the only machine check
on that; the tag name itself remains a human step.
"""
tomllib = pytest.importorskip("tomllib")
pyproject = tomllib.loads((PROJECT_ROOT / "pyproject.toml").read_text(encoding="utf-8"))
assert llm_ingestion_okf.__version__ == pyproject["project"]["version"]

224
tests/test_profile.py Normal file
View file

@ -0,0 +1,224 @@
"""The bundle contract as configuration (Phase 3, step 1).
What Phases 1 and 2 hard-coded about a valid bundle — the index name and its
managed-link shape, the three filename namespaces, the reserved layer, and the
frontmatter key order — becomes a profile object here. `DEFAULT` must express
exactly the existing contract: the Phase 1/2 suite and the golden fixtures are
the byte-level proof (assumption C1), and these tests pin the profile's own
behavior.
"""
from __future__ import annotations
import dataclasses
import pytest
from llm_ingestion_okf.profiles import (
DEFAULT,
BundleProfile,
FrontmatterSchema,
TypePolicy,
)
def test_the_profile_and_every_policy_on_it_are_frozen() -> None:
"""A profile is a value, not a mutable registry.
A consumer holds DEFAULT while a run is in flight; a profile that could be
mutated mid-run would make the contract a moving target and the golden
determinism claim unprovable.
"""
for value in (DEFAULT, DEFAULT.types, DEFAULT.frontmatter, DEFAULT.paths, DEFAULT.index):
with pytest.raises(dataclasses.FrozenInstanceError):
value.name = "mutated" # type: ignore[misc]
def test_default_profile_pins_the_phase_1_2_names() -> None:
"""The names the two doors already write, now stated in one place."""
assert DEFAULT.index.name == "index.md"
assert DEFAULT.paths.concept_suffix == ".md"
assert DEFAULT.paths.ingest_prefix == "ingest-"
assert DEFAULT.paths.inbox_prefix == "inbox-"
assert DEFAULT.paths.import_prefix == "import-"
# --- the reserved layer (assumption C3) -----------------------------------
@pytest.mark.parametrize("reserved", ["verdict", "Verdict", "VERDICT"])
def test_no_type_policy_can_admit_the_reserved_verdict_layer(reserved: str) -> None:
"""C3: the verdict reservation is a spec invariant, not profile config.
Refused at CONSTRUCTION, not at use: a profile that admits `verdict` must
not exist at all, or the promotion gate stops being the only path into that
layer the moment someone passes the profile to a door.
"""
with pytest.raises(ValueError, match="verdict"):
TypePolicy(allowed=frozenset({"Concept", reserved}))
def test_a_closed_type_policy_without_the_reserved_layer_constructs() -> None:
policy = TypePolicy(allowed=frozenset({"Concept", "Guide", "Reference", "Release"}))
assert policy.rejection("Concept") is None
def test_default_type_policy_is_open_apart_from_the_reserved_layer() -> None:
"""Phase 1/2 accept any okf_type but `verdict`; DEFAULT must not narrow that."""
assert DEFAULT.types.allowed is None
for admitted in ("Concept", "note", "whatever-the-operator-wants"):
assert DEFAULT.types.rejection(admitted) is None
for reserved in ("verdict", "Verdict", "VERDICT"):
assert DEFAULT.types.rejection(reserved) is not None
def test_the_reserved_rejection_reproduces_both_doors_message_and_code() -> None:
"""The doors keep their own typed errors; the policy supplies the wording.
Door A raises ManifestError and Door B MaterializationError for the same
refusal, so the policy cannot raise — it hands back the reason and the
stable code, and each door frames it. These are the exact strings the
Phase 1/2 tests already assert on.
"""
rejection = DEFAULT.types.rejection("verdict")
assert rejection is not None
assert rejection.code == "okf_type_reserved"
assert f"okf_type {rejection.reason}" == "okf_type must not be 'verdict' (reserved layer)"
def test_a_closed_type_policy_rejects_an_off_enum_type_distinctly() -> None:
"""An off-enum refusal is not the reserved-layer refusal wearing its code."""
policy = TypePolicy(allowed=frozenset({"Concept", "Release"}))
rejection = policy.rejection("Guide")
assert rejection is not None
assert rejection.code == "okf_type_not_allowed"
assert "Concept, Release" in rejection.reason
# --- index policy ---------------------------------------------------------
def test_index_link_template_and_pattern_describe_the_same_shape() -> None:
"""Two fields that must agree, proven to agree rather than trusted to.
Rendering and parsing managed index links are separate operations (§6
appends, and maintenance rewrites), so the profile carries both a template
and a pattern. A round-trip is what keeps them from drifting apart.
"""
for label, target in [("Title", "ingest-x.md"), ("", "inbox-y.md")]:
line = DEFAULT.index.render_link(label, target)
match = DEFAULT.index.link_pattern.match(line)
assert match is not None, f"pattern does not match its own template output: {line!r}"
assert match.group("label") == label
assert match.group("target") == target
def test_default_index_link_is_the_phase_1_shape() -> None:
assert DEFAULT.index.render_link("Sales", "ingest-sales.md") == "- [Sales](ingest-sales.md)"
def test_index_link_pattern_is_anchored_to_the_whole_line() -> None:
"""§6 removal keys on the exact shape, never a bare substring — a curated
line that merely mentions a link must survive verbatim."""
assert DEFAULT.index.link_pattern.match("see - [Sales](ingest-sales.md) below") is None
# --- frontmatter emission -------------------------------------------------
def test_emission_is_the_ordered_prefix_then_the_sorted_tail() -> None:
"""Commons decision D1, carried over intact.
Keys named in the profile's order come first in that order; anything else
follows sorted. That is what makes a regeneration over the same data
byte-identical, which is what the proving consumer's hash registry needs.
"""
schema = FrontmatterSchema(order=("type", "title"))
emitted = schema.emit({"zeta": "3", "title": "T", "alpha": "1", "type": "Concept"})
assert emitted == "type: Concept\ntitle: T\nalpha: 1\nzeta: 3"
def test_emission_skips_ordered_keys_that_are_absent() -> None:
schema = FrontmatterSchema(order=("type", "title", "generated"))
assert schema.emit({"generated": "true", "type": "Concept"}) == "type: Concept\ngenerated: true"
def test_default_emission_reproduces_door_a_key_order() -> None:
"""Byte-identity with `_render_concept_file`'s §5 frontmatter."""
emitted = DEFAULT.frontmatter.emit(
{
"type": "Concept",
"title": "Sales",
"source_system": "crm",
"source_query": "sales.csv",
"ingested_at": "2026-07-03T12:00:00Z",
"ingest_manifest": "m@0123456789abcdef",
"generated": "true",
}
)
assert emitted.splitlines() == [
"type: Concept",
"title: Sales",
"source_system: crm",
"source_query: sales.csv",
"ingested_at: 2026-07-03T12:00:00Z",
"ingest_manifest: m@0123456789abcdef",
"generated: true",
]
def test_default_emission_reproduces_door_b_key_order() -> None:
"""Byte-identity with `render_inbox_concept`'s provenance layer.
Door B's six keys are a different subset of the same schema, and the
canonical order must reproduce BOTH doors — a single order that reordered
either one would change bytes already frozen in the fixtures.
"""
emitted = DEFAULT.frontmatter.emit(
{
"type": "Concept",
"title": "Notes",
"source_file": "notes.md",
"source_sha256": "abc",
"ingested_at": "2026-07-03T12:00:00Z",
"generated": "true",
}
)
assert emitted.splitlines() == [
"type: Concept",
"title: Notes",
"source_file: notes.md",
"source_sha256: abc",
"ingested_at: 2026-07-03T12:00:00Z",
"generated: true",
]
def test_source_query_is_the_only_collapsed_key() -> None:
"""§5 collapses whitespace runs for `source_query` alone (a multi-line
SELECT must render on one line); every other value is validated single-line
at load and emitted verbatim, so an operator's internal double space
survives."""
assert DEFAULT.frontmatter.collapsed_keys == frozenset({"source_query"})
emitted = DEFAULT.frontmatter.emit(
{"title": "two spaces", "source_query": "SELECT a\n FROM t"}
)
assert emitted == "title: two spaces\nsource_query: SELECT a FROM t"
def test_a_profile_is_assembled_from_its_policies() -> None:
"""The profile is its policies and nothing else — no behavior branches
outside what the object expresses.
`ownership` joined the original four at D2, and it is the same rule rather
than an exception to it: the ingest stamp differs per profile, so the
alternative was a version branch inside the collision gate. A policy on the
object is what keeps the emitter and the predicate changeable only together.
"""
assert {field.name for field in dataclasses.fields(BundleProfile)} == {
"types",
"frontmatter",
"paths",
"index",
"ownership",
}

227
tests/test_profile_threading.py Normal file
View file

@ -0,0 +1,227 @@
"""Requirement 6 — `profile` threaded through `materialize_bundle`'s disk phase.
`OKF_V0_2` landed in D2, but nothing outside the library could ask for it: the
content phase already accepted `profile`, while `materialize_bundle` neither
took one nor passed one down. The parameter is keyword-only behind the `*` the
signature already carried, so every three-positional call site stays
source-compatible — which is what the consumer who asked for it needs.
Two of these tests would be untestable by construction if written only against
the shipped profiles: `OKF_V0_2.paths is DEFAULT.paths` and
`.index is DEFAULT.index`, so no assertion can tell `profile.index.name` apart
from `DEFAULT.index.name` when they are the same object. That identity is what
makes the change byte-neutral, and it is also what would let six of the nine
threading sites stay hard-coded with every shipped-profile test still green.
`_SYNTHETIC` exists to close exactly that gap: it renames the index and the
concept files, so a site still reading `DEFAULT` writes to the wrong path and
fails. It is a test instrument, never a supported profile.
What this does NOT claim: the materializer honours `profile.types` (validation
still runs against `DEFAULT` in `manifest.py`; both policies compare equal
today, so nothing is hidden — but the parameter does not reach it), nor
`IndexPolicy`'s judging fields (`per_directory`, `entries_match_directory`,
`root_frontmatter`). `STRICT_V1` sets all three, so passing it here produces a
bundle that does not meet its own profile. Supported here: `DEFAULT` and
`OKF_V0_2`.
"""
from __future__ import annotations
import json
from dataclasses import replace
from pathlib import Path
from typing import Any
import pytest
from llm_ingestion_okf.materialize import materialize_bundle
from llm_ingestion_okf.profiles import DEFAULT, OKF_V0_2, BundleProfile
INGESTED_AT = "2026-07-16T12:00:00Z"
# Renames both halves the disk phase reads: a threading site left on `DEFAULT`
# writes `ingest-orders.md` / `index.md` instead, and the assertions below say
# so by name rather than by a byte diff that cannot point at the cause.
_SYNTHETIC = replace(
OKF_V0_2,
paths=replace(DEFAULT.paths, ingest_prefix="x-", concept_suffix=".mdx"),
index=replace(DEFAULT.index, name="contents.md"),
)
def _manifest_data() -> dict[str, Any]:
return {
"manifest_version": 1,
"source": {"type": "file", "id": "catalogue-1", "root": "data"},
"bundle_summary": "A test bundle.",
"extractions": [
{
"id": "orders",
"title": "Orders",
"query": "orders.csv",
"okf_type": "dataset",
"max_rows": 100,
}
],
}
@pytest.fixture
def file_setup(tmp_path: Path) -> tuple[Path, Path]:
"""A manifest + CSV catalogue in tmp/src, and an empty bundle target path."""
src = tmp_path / "src"
src.mkdir(parents=True)
manifest_path = src / "manifest.json"
manifest_path.write_text(json.dumps(_manifest_data()), encoding="utf-8")
(src / "data").mkdir()
(src / "data" / "orders.csv").write_text("a,b\n1,x\n2,y\n", encoding="utf-8", newline="")
return manifest_path, tmp_path / "bundle"
# --- the profile reaches the content phase ---
def test_v0_2_profile_reaches_the_emitted_frontmatter(file_setup: tuple[Path, Path]) -> None:
"""`_render_concept_file` took `profile` since D2; the call site did not pass
one, so `OKF_V0_2` was unreachable through the only public door.
"""
manifest_path, bundle = file_setup
materialize_bundle(manifest_path, bundle, INGESTED_AT, profile=OKF_V0_2)
text = (bundle / "ingest-orders.md").read_text(encoding="utf-8")
assert f"generated: {{ by: process:llm-ingestion-okf, at: {INGESTED_AT} }}" in text
assert "sources: [{ id: catalogue-1, resource: data }]" in text
assert "generated: true" not in text
def test_default_run_is_unchanged_by_the_new_parameter(file_setup: tuple[Path, Path]) -> None:
"""Byte-neutrality at the signature: passing `DEFAULT` explicitly and passing
nothing must produce the same bundle, or the default is not the default.
"""
manifest_path, bundle = file_setup
materialize_bundle(manifest_path, bundle, INGESTED_AT)
implicit = (bundle / "ingest-orders.md").read_bytes()
index_implicit = (bundle / "index.md").read_bytes()
explicit_dir = bundle.parent / "bundle-explicit"
materialize_bundle(manifest_path, explicit_dir, INGESTED_AT, profile=DEFAULT)
assert (explicit_dir / "ingest-orders.md").read_bytes() == implicit
assert (explicit_dir / "index.md").read_bytes() == index_implicit
assert b"generated: true" in implicit
def test_profile_is_keyword_only(file_setup: tuple[Path, Path]) -> None:
"""The parameter sits behind the `*` the signature already had. A positional
fourth argument must not be accepted, or a later reordering would silently
rebind existing call sites.
"""
manifest_path, bundle = file_setup
with pytest.raises(TypeError):
materialize_bundle(manifest_path, bundle, INGESTED_AT, DEFAULT) # type: ignore[misc]
# --- A-E5: two runs into the SAME directory ---
def test_second_v0_2_run_into_the_same_directory_succeeds(
file_setup: tuple[Path, Path],
) -> None:
"""A-E5, end to end. The emitter and the ownership predicate are coupled
through the stamp value; `OKF_V0_2` changes that value. If the predicate is
not given the same profile the emitter used, the library stops recognising
its own output and the §3 collision gate fires `collision_unstamped` on the
files the previous run wrote. One run compared byte for byte cannot see it.
"""
manifest_path, bundle = file_setup
materialize_bundle(manifest_path, bundle, INGESTED_AT, profile=OKF_V0_2)
first = (bundle / "ingest-orders.md").read_bytes()
first_index = (bundle / "index.md").read_bytes()
materialize_bundle(manifest_path, bundle, INGESTED_AT, profile=OKF_V0_2)
assert (bundle / "ingest-orders.md").read_bytes() == first
assert (bundle / "index.md").read_bytes() == first_index
# --- the paths/index half of the profile reaches disk ---
def test_profile_paths_and_index_name_reach_disk(file_setup: tuple[Path, Path]) -> None:
"""Six of the nine threading sites read `paths`/`index`, which are the same
objects on every shipped profile. `_SYNTHETIC` renames both so the sites are
provable rather than merely regression-covered.
"""
manifest_path, bundle = file_setup
result = materialize_bundle(manifest_path, bundle, INGESTED_AT, profile=_SYNTHETIC)
assert (bundle / "x-orders.mdx").is_file()
assert not (bundle / "ingest-orders.md").exists()
assert (bundle / "contents.md").is_file()
assert not (bundle / "index.md").exists()
assert [path.name for path in result.written] == ["x-orders.mdx"]
assert "- [Orders](x-orders.mdx)" in (bundle / "contents.md").read_text(encoding="utf-8")
def test_second_synthetic_run_replaces_rather_than_colliding(
file_setup: tuple[Path, Path],
) -> None:
"""The §3 ownership scan globs `*{concept_suffix}` and excludes `index.name`.
Left on `DEFAULT`, the glob finds nothing under a renamed suffix, `owned` is
empty, and the gate refuses to overwrite the file this same code just wrote.
The index must also not gain a second link for the same target.
"""
manifest_path, bundle = file_setup
materialize_bundle(manifest_path, bundle, INGESTED_AT, profile=_SYNTHETIC)
first = (bundle / "x-orders.mdx").read_bytes()
materialize_bundle(manifest_path, bundle, INGESTED_AT, profile=_SYNTHETIC)
assert (bundle / "x-orders.mdx").read_bytes() == first
index_text = (bundle / "contents.md").read_text(encoding="utf-8")
assert index_text.count("- [Orders](x-orders.mdx)") == 1
def test_removed_extraction_is_unlinked_under_a_renamed_index(
file_setup: tuple[Path, Path], tmp_path: Path
) -> None:
"""`_update_index_lines` runs only on an EXISTING index, so it is reached
solely by a second run — and only its `link_pattern`/`render_link` are
profile-owned. Dropping an extraction is what forces the removal branch.
"""
manifest_path, bundle = file_setup
materialize_bundle(manifest_path, bundle, INGESTED_AT, profile=_SYNTHETIC)
data = _manifest_data()
data["extractions"][0]["id"] = "invoices"
data["extractions"][0]["title"] = "Invoices"
manifest_path.write_text(json.dumps(data), encoding="utf-8")
(tmp_path / "src" / "data" / "invoices.csv").write_text(
"a,b\n1,x\n", encoding="utf-8", newline=""
)
materialize_bundle(manifest_path, bundle, INGESTED_AT, profile=_SYNTHETIC)
index_text = (bundle / "contents.md").read_text(encoding="utf-8")
assert "- [Invoices](x-invoices.mdx)" in index_text
assert "x-orders.mdx" not in index_text
assert not (bundle / "x-orders.mdx").exists()
def test_synthetic_profile_is_not_a_shipped_profile() -> None:
"""Guards the instrument: if a future profile ever shares these overrides,
the two tests above stop proving anything and would go quietly green.
"""
assert isinstance(_SYNTHETIC, BundleProfile)
assert _SYNTHETIC.paths is not DEFAULT.paths
assert _SYNTHETIC.index is not DEFAULT.index
assert OKF_V0_2.paths is DEFAULT.paths
# `OKF_V0_2.index` stopped being DEFAULT's object at D5, which named
# `okf_version` in its `root_frontmatter`. That is NOT the distinctness this
# guard is about: every field that carries a NAME — the index filename and
# the concept filenames — is still shared, so a threading bug that passed
# the wrong profile would still emit identical bytes under either. Object
# identity was only ever a proxy for that; assert the thing itself, or this
# guard would go green on a change that leaves the instrument blind.
assert OKF_V0_2.index.name == DEFAULT.index.name
assert OKF_V0_2.index.link_template == DEFAULT.index.link_template
assert _SYNTHETIC.index.name != DEFAULT.index.name
assert _SYNTHETIC.paths.concept_suffix != DEFAULT.paths.concept_suffix

215
tests/test_root_frontmatter_emission.py Normal file
View file

@ -0,0 +1,215 @@
"""D5's emission half: the root `index.md` frontmatter block.
Upstream §8 binds `index.md` to "no frontmatter, with one exception: a
bundle-root `index.md` MAY carry an `okf_version` key (§12)", and §12 puts the
declaration "in a bundle-root `index.md` frontmatter block (the only place
frontmatter is permitted in an `index.md`)". Catalog verified both quotations
against the pinned commit `3fcbb9f` on 2026-07-31 and reported the same
reading, so placement is settled: frontmatter, not a body line.
The value never comes from a profile. V4/V-A5 gives `okf_version`'s *value* to
catalog (decision E1) and leaves this library the narrower obligation — name
the key, express any value. So the profile pins `root_frontmatter` and the
caller supplies the mapping.
Two assertions here read RAW BYTES rather than a parsed value, deliberately.
Catalog measured that a quoted value fails their shape regex with `exit 1`, and
that a UTF-8 BOM makes the marker invisible to them while still exiting `0`. A
parsed assertion masks exactly those two defects: `yaml.safe_load` returns the
string `"0.2"` whether or not it was quoted, and strips a BOM before the
caller ever sees it.
"""
from __future__ import annotations
import json
from dataclasses import replace
from pathlib import Path
import pytest
from llm_ingestion_okf.errors import MaterializationError
from llm_ingestion_okf.materialize import materialize_bundle
from llm_ingestion_okf.profiles import DEFAULT, OKF_V0_2
INGESTED_AT = "2026-07-16T12:00:00Z"
def build_case(tmp_path: Path) -> Path:
"""A minimal file-source manifest; the frontmatter block is what is under
test, so the extraction is kept as small as it can be."""
case = tmp_path / "case"
fixture = case / "fixture"
fixture.mkdir(parents=True)
(fixture / "rows.csv").write_text("id,label\n1,alpha\n", encoding="utf-8")
manifest = {
"manifest_version": 1,
"source": {"type": "file", "id": "root-fm", "root": "fixture"},
"bundle_summary": "Root frontmatter case.",
"extractions": [
{
"id": "rows",
"title": "Rows",
"query": "rows.csv",
"okf_type": "dataset",
"max_rows": 10,
}
],
}
(case / "manifest.json").write_text(json.dumps(manifest), encoding="utf-8")
return case
def test_root_index_carries_the_declared_key(tmp_path: Path) -> None:
"""The block opens the file and the body follows it."""
case = build_case(tmp_path)
bundle = tmp_path / "bundle"
materialize_bundle(
case / "manifest.json",
bundle,
INGESTED_AT,
profile=OKF_V0_2,
root_frontmatter_values={"okf_version": "0.2"},
)
assert (bundle / "index.md").read_bytes() == (
b"---\nokf_version: 0.2\n---\n\nRoot frontmatter case.\n- [Rows](ingest-rows.md)\n"
)
def test_the_value_is_unquoted_in_the_raw_bytes(tmp_path: Path) -> None:
"""Catalog's shape regex `/^\\d+(\\.\\d+)*$/` sees the quotes as part of the
value and exits 1. Asserted on bytes: a parsed check passes either way."""
case = build_case(tmp_path)
bundle = tmp_path / "bundle"
materialize_bundle(
case / "manifest.json",
bundle,
INGESTED_AT,
profile=OKF_V0_2,
root_frontmatter_values={"okf_version": "0.2"},
)
raw = (bundle / "index.md").read_bytes()
assert b"okf_version: 0.2\n" in raw
assert b'okf_version: "0.2"' not in raw
assert b"okf_version: '0.2'" not in raw
def test_the_file_carries_no_utf8_bom(tmp_path: Path) -> None:
"""A BOM leaves the marker invisible to catalog's gate while still exiting
0 — the failure that reports success, so it gets its own byte assertion."""
case = build_case(tmp_path)
bundle = tmp_path / "bundle"
materialize_bundle(
case / "manifest.json",
bundle,
INGESTED_AT,
profile=OKF_V0_2,
root_frontmatter_values={"okf_version": "0.2"},
)
raw = (bundle / "index.md").read_bytes()
assert not raw.startswith(b"\xef\xbb\xbf")
assert raw.startswith(b"---\n")
def test_a_key_the_policy_does_not_name_is_refused(tmp_path: Path) -> None:
"""`DEFAULT` names no root-frontmatter key, so offering one is a caller
error rather than something to write. Fail-fast: refused BEFORE any disk
mutation, or a rejected run would still leave a half-written bundle."""
case = build_case(tmp_path)
bundle = tmp_path / "bundle"
with pytest.raises(MaterializationError) as excinfo:
materialize_bundle(
case / "manifest.json",
bundle,
INGESTED_AT,
profile=DEFAULT,
root_frontmatter_values={"okf_version": "0.2"},
)
assert excinfo.value.code == "index_root_frontmatter_unexpected"
assert not bundle.exists()
def test_omitting_the_values_emits_no_frontmatter(tmp_path: Path) -> None:
"""§12 is a MAY, and none of upstream's four reference bundles declares the
key at all (catalog grepped `okf/bundles` and `okf/samples`: zero hits). A
profile that names the key must therefore still emit a bundle without it."""
case = build_case(tmp_path)
bundle = tmp_path / "bundle"
materialize_bundle(case / "manifest.json", bundle, INGESTED_AT, profile=OKF_V0_2)
raw = (bundle / "index.md").read_bytes()
assert not raw.startswith(b"---")
assert b"okf_version" not in raw
def test_key_order_follows_the_policy_not_the_mapping(tmp_path: Path) -> None:
"""The policy pins the order; a caller's dict ordering must not reach the
file, or two callers passing the same keys would emit different bytes."""
case = build_case(tmp_path)
bundle = tmp_path / "bundle"
profile = replace(
OKF_V0_2,
index=replace(OKF_V0_2.index, root_frontmatter=("okf_version", "bundle_profile")),
)
materialize_bundle(
case / "manifest.json",
bundle,
INGESTED_AT,
profile=profile,
root_frontmatter_values={"bundle_profile": "okf-v0-2", "okf_version": "0.2"},
)
raw = (bundle / "index.md").read_bytes()
assert raw.startswith(b"---\nokf_version: 0.2\nbundle_profile: okf-v0-2\n---\n\n")
def test_the_frozen_golden_declares_the_version_gate_safely() -> None:
"""D5's raw-byte guard, asserted on the COMMITTED fixture rather than on a
fresh run.
`test_golden.py` already compares a run against these bytes, but that only
proves the code agrees with itself: regenerate the fixture from a quoted or
BOM-carrying emitter and both sides move together, silently. This asserts
the shape of what we froze and hand to catalog's gate — the one thing a
self-comparison cannot catch.
"""
raw = (
Path(__file__).parent.parent
/ "examples"
/ "ingest-golden-okf-v0-2"
/ "expected-bundle"
/ "index.md"
).read_bytes()
assert raw.startswith(b"---\nokf_version: 0.2\n---\n\n")
assert not raw.startswith(b"\xef\xbb\xbf") # a BOM exits 0 while hiding the marker
assert b'"' not in raw.split(b"---\n", 2)[1] # a quoted value exits 1
assert b"\r\n" not in raw # LF-only, like every other emitted file
def test_reingest_in_place_is_byte_identical(tmp_path: Path) -> None:
"""A-E5. The second run goes into the FIRST run's directory — the shape the
pilot runs — so the frontmatter block must survive §6 index maintenance
rather than being appended a second time or dropped."""
case = build_case(tmp_path)
bundle = tmp_path / "bundle"
call = dict(
profile=OKF_V0_2,
root_frontmatter_values={"okf_version": "0.2"},
)
materialize_bundle(case / "manifest.json", bundle, INGESTED_AT, **call) # type: ignore[arg-type]
first = (bundle / "index.md").read_bytes()
materialize_bundle(case / "manifest.json", bundle, INGESTED_AT, **call) # type: ignore[arg-type]
assert (bundle / "index.md").read_bytes() == first

371
tests/test_strict_v1.py Normal file
View file

@ -0,0 +1,371 @@
"""The proving consumer's contract as a profile (Phase 3, step 3).
`STRICT_V1` is the `claude-code-llm-wiki` bundle contract expressed in the same
object `DEFAULT` uses. Everything pinned here was measured in their repository
on 2026-07-25 and settled in `docs/phase-3-split-table.md`; the eight-key
required set is their operator's ratification, deliberately stricter than the
four-key constant their own validator enforces.
These tests are also the reader that earns the new schema fields. A profile
field no code reads is a claim no test covers, so `required`, `allowed`,
`nullable` and `key_pattern` land together with `FrontmatterSchema.violations`,
which is what reads them.
What is NOT here, and why: `violations` takes an already-parsed mapping. This
library's line-oriented parser cannot represent the block lists and nulls the
consumer's frontmatter admits — that is precisely why Door C writes imported
concepts verbatim — so parsing strict-v1 frontmatter is a separate concern from
judging it, and lands separately.
"""
from __future__ import annotations
import pytest
from llm_ingestion_okf.profiles import (
DEFAULT,
STRICT_V1,
FrontmatterSchema,
)
# The eight keys the consumer's operator ratified on 2026-07-25, measured
# present on 522/522 of their documents.
RATIFIED_REQUIRED = frozenset(
{
"type",
"title",
"description",
"timestamp",
"layer",
"source_tier",
"source_url",
"source_sha",
}
)
# `FRONTMATTER_KEY_ORDER` in their `bundle.py`: the eight above plus three
# layer-specific keys, doubling as canonical emission order.
RATIFIED_ORDER = (
"type",
"title",
"description",
"timestamp",
"layer",
"source_tier",
"source_url",
"source_sha",
"version",
"date",
"summary",
)
def wiki_document() -> dict[str, object]:
"""A conforming `concepts/` page, every allowlisted key present."""
return {
"type": "Concept",
"title": "Hooks",
"description": "Event-driven automation in Claude Code.",
"timestamp": "2026-07-25T12:00:00Z",
"layer": "concepts",
"source_tier": "official-docs",
"source_url": "https://docs.claude.com/en/docs/claude-code/hooks",
"source_sha": "0" * 64,
"version": "2.1.183",
"date": "2026-07-25",
"summary": "What hooks are and when they fire.",
}
def codes(schema: FrontmatterSchema, values: dict[str, object]) -> list[str]:
return [violation.code for violation in schema.violations(values)]
# --- the profile itself ---------------------------------------------------
def test_strict_v1_closes_the_type_enum_to_the_four_document_classes() -> None:
"""Their four classes exactly — an open set would admit anything."""
assert STRICT_V1.types.allowed == frozenset({"Concept", "Guide", "Reference", "Release"})
def test_strict_v1_requires_the_eight_ratified_keys() -> None:
"""Eight, not the four their `REQUIRED_KEYS` constant spells.
Their constant is the emit-path minimum; the contract their operator
ratified is all eight. This is the one place the profile is deliberately
stricter than the consumer's own validator, and the agreed resolution if a
document class ever legitimately lacks `source_sha` is to bump the profile,
not to bend the bundle.
"""
assert STRICT_V1.frontmatter.required == RATIFIED_REQUIRED
def test_strict_v1_allowlist_is_eleven_keys_and_doubles_as_emission_order() -> None:
"""The allowlist and the order are one list, not two that can drift apart.
Their `BUNDLE_HASH_REGISTRY` gate depends on regeneration being
byte-identical, which is what the fixed order buys.
"""
assert STRICT_V1.frontmatter.order == RATIFIED_ORDER
assert STRICT_V1.frontmatter.allowed == frozenset(RATIFIED_ORDER)
def test_the_three_layer_specific_keys_are_optional_not_required() -> None:
"""`version`/`date` sit on 351/522 documents and `summary` on 349/522.
Requiring them would reject pages the consumer ships today.
"""
allowed = STRICT_V1.frontmatter.allowed
assert allowed is not None
for key in ("version", "date", "summary"):
assert key in allowed
assert key not in STRICT_V1.frontmatter.required
def test_summary_is_the_only_nullable_key() -> None:
"""The single deliberate escape from strings-only frontmatter."""
assert STRICT_V1.frontmatter.nullable == frozenset({"summary"})
# --- the reader: a conforming document -----------------------------------
def test_a_conforming_wiki_document_has_no_violations() -> None:
assert STRICT_V1.frontmatter.violations(wiki_document()) == ()
def test_the_optional_keys_may_simply_be_absent() -> None:
"""A `concepts/` page carries neither `version` nor `date`."""
document = wiki_document()
for key in ("version", "date", "summary"):
del document[key]
assert STRICT_V1.frontmatter.violations(document) == ()
# --- the reader: cross-profile rejection ---------------------------------
def test_a_door_a_ingest_document_is_rejected_under_strict_v1() -> None:
"""The plan's cross-profile test, frontmatter half.
A bundle this library generates under `DEFAULT` carries the ingest-spec
layer, which shares only `type` and `title` with the consumer's contract:
six required keys are missing and five of its own keys are off-allowlist.
"""
door_a = {
"type": "dataset",
"title": "Orders",
"source_system": "golden-catalogue",
"source_query": "orders.csv",
"ingested_at": "2026-07-16T12:00:00Z",
"ingest_manifest": "manifest@37674ac20059e788",
"generated": "true",
}
violations = STRICT_V1.frontmatter.violations(door_a)
assert violations != ()
missing = {v.key for v in violations if v.code == "frontmatter_key_missing"}
assert missing == RATIFIED_REQUIRED - {"type", "title"}
off_allowlist = {v.key for v in violations if v.code == "frontmatter_key_not_allowed"}
assert off_allowlist == {
"source_system",
"source_query",
"ingested_at",
"ingest_manifest",
"generated",
}
def test_default_judges_nothing_about_keys_so_both_doors_stay_valid() -> None:
"""`DEFAULT` must NOT grow a required set — measured, not assumed.
Door A emits seven keys, Door B six, and Door C writes an imported concept
verbatim with whatever frontmatter the sender wrote. A required set on
`DEFAULT` would therefore declare invalid the very bundles this library
produces. `DEFAULT` keeps an open namespace; the rejection of a wiki-shaped
bundle under `DEFAULT` comes from the index and path shape instead.
"""
assert DEFAULT.frontmatter.required == frozenset()
assert DEFAULT.frontmatter.allowed is None
assert DEFAULT.frontmatter.violations(wiki_document()) == ()
assert DEFAULT.frontmatter.violations({"type": "dataset", "title": "Orders"}) == ()
# --- the reader: each check ----------------------------------------------
def test_a_missing_required_key_is_named_individually() -> None:
document = wiki_document()
del document["source_sha"]
violations = STRICT_V1.frontmatter.violations(document)
assert [(v.key, v.code) for v in violations] == [("source_sha", "frontmatter_key_missing")]
def test_a_key_outside_the_allowlist_is_refused() -> None:
"""A closed allowlist is what makes the contract enumerable at all."""
document = wiki_document()
document["author"] = "someone"
assert codes(STRICT_V1.frontmatter, document) == ["frontmatter_key_not_allowed"]
def test_a_key_name_outside_the_grammar_is_refused() -> None:
"""`^[a-z_]+$`: a key that reaches the allowlist check malformed would
otherwise be reported only as 'not allowed', which hides the real defect."""
schema = FrontmatterSchema(order=("type",), key_pattern=STRICT_V1.frontmatter.key_pattern)
violations = schema.violations({"Source-URL": "x"})
assert [(v.key, v.code) for v in violations] == [("Source-URL", "frontmatter_key_malformed")]
def test_summary_may_be_null_and_no_other_key_may() -> None:
document = wiki_document()
document["summary"] = None
assert STRICT_V1.frontmatter.violations(document) == ()
document["description"] = None
assert [(v.key, v.code) for v in STRICT_V1.frontmatter.violations(document)] == [
("description", "frontmatter_value_null")
]
def test_a_non_empty_list_of_strings_is_a_legal_value() -> None:
"""The block-list shape their `Release` pages carry.
A profile that assumed scalars-only would reject the consumer's own pages —
and a parser that assumed scalars-only would silently destroy them, which
is why Door C writes imported concepts verbatim.
"""
document = wiki_document()
document["source_url"] = ["https://a.example/one", "https://a.example/two"]
assert STRICT_V1.frontmatter.violations(document) == ()
@pytest.mark.parametrize(
"value",
[[], ["ok", 3], {"nested": "map"}, 7, True],
ids=["empty-list", "list-with-non-string", "mapping", "int", "bool"],
)
def test_a_value_outside_the_scalar_or_string_list_shape_is_refused(value: object) -> None:
document = wiki_document()
document["title"] = value
assert [(v.key, v.code) for v in STRICT_V1.frontmatter.violations(document)] == [
("title", "frontmatter_value_shape")
]
# --- determinism and construction-time invariants -------------------------
def test_violations_come_back_in_a_deterministic_order() -> None:
"""Two runs over the same document must report identically ordered.
Determinism is bit-exact in this library; a violation list that depended on
mapping insertion order would make a report irreproducible.
"""
document = wiki_document()
del document["source_sha"]
del document["layer"]
document["zzz"] = "off-allowlist"
document["aaa"] = "off-allowlist"
reported = [(v.key, v.code) for v in STRICT_V1.frontmatter.violations(document)]
assert reported == sorted(reported)
reversed_document = dict(reversed(list(document.items())))
assert [
(v.key, v.code) for v in STRICT_V1.frontmatter.violations(reversed_document)
] == reported
def test_a_required_key_outside_the_allowlist_cannot_be_constructed() -> None:
"""An unsatisfiable schema is a defect in the profile, not in a document."""
with pytest.raises(ValueError, match="required"):
FrontmatterSchema(
order=("type", "title"),
required=frozenset({"type", "absent_from_allowlist"}),
allowed=frozenset({"type", "title"}),
)
def test_a_nullable_key_outside_the_allowlist_cannot_be_constructed() -> None:
with pytest.raises(ValueError, match="nullable"):
FrontmatterSchema(
order=("type",),
allowed=frozenset({"type"}),
nullable=frozenset({"summary"}),
)
def test_strict_v1_path_namespaces_are_the_spec_namespaces() -> None:
"""The door filename prefixes are an ingest-spec invariant, not a preference.
They describe what Doors A/B/C write, so they are the same under any
profile; the consumer's own files are written by their pipeline, not by a
door, and are unaffected either way.
"""
assert STRICT_V1.paths == DEFAULT.paths
def test_strict_v1_index_policy_is_the_consumers_shape() -> None:
"""The gap this test used to pin is closed; it now states the shape.
Every value is read out of their code rather than recalled — the entry
template and the root/nested frontmatter split from `bundle.py:498-567`,
the bidirectional match from gate `BUNDLE_INDEX_COMPLETE`
(`validate.py:1081-1120`, severity ERROR). The reader that judges all of
it lives on `IndexPolicy.violations`, so no field here is written without
something that reads it.
"""
index = STRICT_V1.index
assert index != DEFAULT.index
assert index.name == DEFAULT.index.name
assert index.render_link("Hooks", "hooks.md", "How hooks fire.") == (
"* [Hooks](hooks.md) - How hooks fire."
)
assert index.per_directory is True
assert index.heading_required is True
assert index.allows_prose is False
assert index.entries_match_directory is True
assert index.root_frontmatter == ("okf_version", "bundle_profile", "okf_spec_commit")
def test_a_conforming_wiki_root_index_passes_and_a_default_one_does_not() -> None:
"""The plan's second cross-profile test: a bundle valid under one profile
is refused by the other. It could not come from frontmatter — DEFAULT has
no required key set, by measurement — so it comes from index shape.
"""
conforming = (
"---\nokf_version: 0.1\nbundle_profile: strict-v1\n"
"okf_spec_commit: d44368c\n---\n\n# Wiki\n\n"
"* [Concepts](concepts/index.md) - Machine-generated index of concepts.\n"
)
assert (
STRICT_V1.index.violations(conforming, is_root=True, expected_targets={"concepts/index.md"})
== ()
)
default_shaped = "# Wiki\n\nCurated prose.\n\n- [Concepts](concepts/index.md)\n"
codes = {
violation.code
for violation in STRICT_V1.index.violations(
default_shaped, is_root=True, expected_targets={"concepts/index.md"}
)
}
assert codes == {
"index_root_key_missing",
"index_prose_not_allowed",
"index_entry_missing",
}
# ...and the reverse direction: DEFAULT judges neither of them, because a
# judging default would condemn the bundles this library itself writes.
assert DEFAULT.index.violations(conforming, is_root=True) == ()
assert DEFAULT.index.violations(default_shaped, is_root=True) == ()
def test_strict_v1_emission_is_the_ordered_prefix_then_the_sorted_tail() -> None:
"""Commons decision D1, unchanged by the stricter namespace."""
emitted = STRICT_V1.frontmatter.emit(
{"title": "Hooks", "type": "Concept", "description": "d", "timestamp": "t"}
)
assert emitted == "type: Concept\ntitle: Hooks\ndescription: d\ntimestamp: t"