llm-ingestion-okf/docs/phase-3-split-table.md
Kjell Tore Guttormsen 678f9b2b07 docs: two corrections from the convention owner and the spec owner
Both arrived within minutes of the v0.4.0 announcement and both change
what we would otherwise have built or claimed.

CATALOG -- a profile field we must NOT add. Their steg 5 measured bundle
placement across four consumers: exactly one installs a bundle anywhere,
the other three take the bundle directory as a per-run argument with no
default, no discovery and no $HOME expansion. Placement is unmandated, so
BundleProfile gets no root-location or install-path field, "no
installation root" is a conforming consumer shape, and write policy must
not be coupled to placement. Recorded in the split table, which also now
spells the proving consumer's layer prefixes bundle-relative
(concepts/, releases/) rather than repeating the bundle/ prefix their own
repository layout carries -- that prefix was placement leaking into a
contract.

COMMONS -- our "Fixed" framing outruns their frozen text. Removing every
ingest-stamped file is verbatim what ingest-spec v1 section 5 mandates:
v1 assumes one manifest per bundle and defers the multi-manifest case as
a named extension point. So v0.4.0 implements that extension point rather
than fixing a defect against the spec. The mechanic survives ratification
either way -- stem matching follows from their own {stem}@{h} stamp
definition, since {h} changes on every content edit -- but the prose is
queued for amendment and is not ratified, so the changelog now says so
instead of claiming conformance it does not have.
2026-07-25 12:32:21 +02:00

11 KiB

Phase 3 split table — the proving consumer's 25 gates

Status: draft, awaiting review. Phase 3's plan requires this mapping before any porting, reviewed with the operator and with the proving consumer (claude-code-llm-wiki). Nothing here is implemented yet, and nothing in the consumer's repository is modified by this repository.

Read from the consumer's working tree on 2026-07-25: tools/wiki_ingest/validate.py and tools/wiki_ingest/bundle.py at commit 9ee5a8e (repository HEAD 20bcdd6). Their validate.py is the single entrypoint for every per-run gate, so this table is exhaustive over the gate set as it stands at that commit — assumption C2 is satisfied only as long as that remains true, and a gate added there after this date is not in this table.

The three columns

Column Meaning Goes where
Profile Structural bundle contract: types, frontmatter keys, paths, reserved files, index shape This phase, BundleProfile
Guard Content safety: carriers, secrets, injection, active content llm-ingestion-guard calls — never reimplemented here
Consumer Domain logic: source pipelines, LLM contract, promotion, run accounting Stays in the consumer's repository

Stage A — detection/acquisition

# Gate Default class Column Note
1 INGEST_LEDGER_DIVERGENCE ALERT (halts) Consumer Upstream re-poll reconciliation; no bundle structure involved.
2 INGEST_CHANGELOG_PARSE ERROR Consumer Source-format parsing for one specific upstream.
3 INGEST_EXPECTED_SET ALERT Consumer npm/changelog reconciliation against known deltas.
4 INGEST_DOCS_CORPUS_SANITY ERROR Consumer Corpus size/shape thresholds for one named corpus. Its second half — "a concept must not derive from the changelog page" — is a layer-provenance rule expressed as a URL test; see open question Q3.
5 INGEST_INPUT_INJECTION_SCAN WARN (ERROR when verbatim) Guard Carrier scan plus semantic lexicon on fetched input. The disposition-aware escalation is Consumer policy over a Guard verdict; see open question Q1.

Stage B — LLM-output contract

# Gate Default class Column Note
6 LLM_STOP_REASON ERROR Consumer
7 LLM_SCHEMA ERROR Consumer Frozen run-record schema plus summary length caps.
8 LLM_OUTPUT_LINT ERROR mixed Decomposed below.
9 LLM_GROUNDING_ANCHOR ERROR Consumer Anchor resolution against their concepts layer.
10 LLM_SOURCE_TRACE ERROR Consumer Verbatim-quote and dropped-citation detectors.
11 LLM_SEAM_LINKS ERROR Consumer Proposed links restricted to existing concept IDs.
12 LLM_BUDGET ERROR Consumer Per-run call/token/cost caps.

Stage C — bundle conformance and publish

# Gate Default class Column Note
13 BUNDLE_FRONTMATTER_PARSEABLE ERROR Profile Every non-reserved .md opens with a complete fence.
14 BUNDLE_TYPE_ENUM ERROR Profile Closed enum Concept/Guide/Reference/Release.
15 BUNDLE_RESERVED_FILES ERROR Profile log.md required-absent; index.md structure. Three reserved basenames, not two — see "What STRICT_V1 must express".
16 BUNDLE_YAML_STRICT ERROR Profile Key allowlist, required keys, and the StrictYAML scalar subset.
17 BUNDLE_LAYER_PATH_TYPE ERROR Profile Path prefix ⇄ layersource_tiertype agreement.
18 BUNDLE_LINKS_INTERNAL ERROR Profile Every relative cross-link resolves inside the bundle.
19 BUNDLE_LINKS_EXTERNAL WARN (never blocks) Consumer Liveness polling; needs network state this library does not collect.
20 BUNDLE_URL_ALLOWLIST ERROR mixed Decomposed below.
21 BUNDLE_INDEX_COMPLETE ERROR Profile Per-directory index exactness plus root-index metadata keys.
22 BUNDLE_HASH_REGISTRY ERROR Consumer Their page-hash snapshotter must cover the concepts layer exactly.
23 BUNDLE_SANITIZATION ERROR Guard Carrier-freedom of published text.
24 BUNDLE_BLAST_RADIUS ERROR Consumer Per-version diff caps from their run manifest.
25 BUNDLE_SECRET_SCAN ERROR Guard Credential patterns over bundle text and run artifacts.

The two mixed gates, decomposed

The plan asks for exactly one column per gate. Two gates do not honour that as written, because each bundles checks belonging to different columns. Splitting them is the honest mapping; the alternative — assigning the whole gate to its dominant column — would quietly move the residue to the wrong side.

Gate 8, LLM_OUTPUT_LINT runs _lint_text over enrichment summaries and verbatim release bodies:

Check Column
Off-allowlist URL Consumer (their editorial pinning; see gate 20)
HTML tag in prose Guard — active content is squarely the guard's territory at v0.3.0
base64-like blob Guard
Credential pattern Guard
Consumer-directed imperative Guard — this is the injection posture, expressed as their house rule
Injection lexicon and hex blob Guard

Gate 20, BUNDLE_URL_ALLOWLIST bundles two unrelated rules:

Check Column
URL host/prefix on the pinned allowlist Consumer
Domain is new versus the previous bundle state Consumer (drift detection over their own retained state)

Neither half is Profile: a URL allowlist is editorial policy about which sources a bundle may cite, not a statement about bundle structure. It is listed as mixed only because the checks have different natures, and both land in the same column — noted rather than merged, so a later reviewer can see it was considered.

What STRICT_V1 must express

The phase-3 plan sketched this profile before the validator was read. Two of its claims do not survive contact and are corrected here; the plan file is corrected in the same commit.

  1. Required keys are four, not eight. REQUIRED_KEYS = (type, title, description, timestamp). The eight-key list in the plan was a truncation of FRONTMATTER_KEY_ORDER, which is a different thing: an eleven-key allowlist that doubles as canonical emission order — type, title, description, timestamp, layer, source_tier, source_url, source_sha, version, date, summary. So frontmatter_schema needs three fields, not one: required, allowlisted, and ordered.
  2. Three reserved basenames, not two. index.md, log.md, and README.md. README is bundle documentation, reserved like index.md: it carries no frontmatter, is skipped by the per-document gates, is never listed in a generated index, and is still subject to the link, URL, sanitization and secret gates. A two-value reserved-file policy (required-present / required-absent) cannot express it; the third value is "present-and-exempt-from-document-gates".

Beyond the frontmatter set, BundleProfile must carry:

  • Type policy: closed enum, four values.
  • Nullable keys: summary alone may be null — the single deliberate escape from strings-only frontmatter.
  • Scalar encoding: the StrictYAML subset (JSON-double-quoted scalars), so a regeneration over the same data is byte-identical.
  • Layer/path policy: concepts/ ⇄ layer concepts, source_tier official-docs, any type except Release; releases/ ⇄ layer releases, source_tier changelog-derived, type Release exactly. Their constants spell these bundle/concepts and bundle/releases because their validator walks a repository whose bundle lives in bundle/. That prefix is where they keep the bundle, not part of the contract — see "No placement field" below.
  • Index policy, which differs from this library's default in four ways: one index per directory rather than one per bundle; entries formatted * [Title](link) - description rather than - [label](target); index contents must match the directory exactly, both directions; and the root index carries frontmatter pinning okf_version, bundle_profile, okf_spec_commit.

No placement field — a field the profile must NOT grow

The catalog reported on 2026-07-25 (their commit 3ccb5f6, docs/okf-second-brain/spec.md §1 and §8) that bundle placement is unmandated, measured against four consumers rather than taken on report: exactly one installs a bundle anywhere; for the other three the bundle directory is a caller-supplied argument per run, with no default, no discovery and no $HOME expansion. Discovery, where it happens, keys on the §3 root marker (index.md plus okf_version), never on a path.

Three consequences for BundleProfile, recorded here because this phase's non-goal is precisely "do not make the profile too narrow for the Node-world conventions in phase 4":

  • No root-location or install-path field. There is no path form to express.
  • "No installation root" is a conforming consumer shape, so a profile that assumes every consumer has one would be too narrow in exactly the warned way.
  • Installing and writing are separate axes: a consumer may write into a bundle it never placed, so write policy must not be tied to a placement concept.

Every path in this document is therefore bundle-relative, and the bundle directory stays what it is today in both doors — an argument.

The index-policy divergence is the largest single item. Under the default profile the observable output must stay byte-identical (assumption C1, proven by the golden suite), so index_policy has to express both shapes rather than generalising one into the other.

Open questions for the proving consumer

  • Q1. Gate 5 escalates the injection scan from WARN to ERROR on a verbatim disposition, because no LLM rewrite layer sits in front of the published text. That is a policy tier chosen per document, over a guard verdict. Should the profile be able to express "this document class gets a stricter gate tier", or does that stay entirely in the consumer's hands? This repository's position is the latter — the profile describes structure, not security tiers — but it is the consumer's call, since it decides where their gate lives after the port.
  • Q2. Is the four-key required set final, or is it four only because seed triage has not finished? layer, source_tier, source_url and source_sha are allowlisted and, in practice, present on every generated document — if they are meant to be required, STRICT_V1 should require them.
  • Q3. Gate 4's second half asserts that no concepts/ page may derive from the changelog page. That reads as a layer-provenance rule, which is Profile territory, but it is expressed as a URL predicate over one specific upstream, which is Consumer territory. Mapped to Consumer here. Correct?
  • Q4. Gates 19 and 22 are mapped to Consumer because they need state this library never collects (link liveness, page hashes). If either is expected to move, the profile would have to grow a notion of external state, which is a larger change than phase 3 scopes.

What this table does not do

It does not port anything. Per the plan, porting starts only after this mapping is reviewed, and the Guard column must contain no rows implemented in this repository — the phase-2 grep gate over guard functionality still applies unchanged.