docs(plan): add detailed phase 1-4 implementation plans

Phase 1 (Door A): ingest-spec v1 implementation + §11 golden fixtures,
pinned to the current spec revision, with TDD order and byte-exact
verification criteria. Phase 2 (Doors B/C): guard-gated inbox and
external bundle import. Phase 3: configurable bundle contract via a
profile object, default profile locked to the golden suite. Phase 4:
zero-dep Node half, coordination-first with owner sign-offs.

Each plan carries explicit key assumptions with tests and a
verification section.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QeqhJpYQyghASjiJo5EhGg
This commit is contained in:
Kjell Tore Guttormsen 2026-07-16 10:58:08 +02:00
commit b0ad2aedfb
4 changed files with 450 additions and 0 deletions

124
docs/plan/phase-1-door-a.md Normal file
View file

@ -0,0 +1,124 @@
# Phase 1 plan — Door A: ingest-spec implementation + §11 golden fixtures
Status: approved roadmap phase (see `CLAUDE.md`); this plan details it before any code.
Normative basis: `ingest-spec.md` v1 in `portfolio-optimiser-commons` (RFC-2119).
This plan was written against spec revision `sha256:959484e1952d28fa…` — if the spec has
changed when implementation starts, re-read it first and update this plan.
## Goal
A stdlib-only Python implementation of the full ingest-spec contract:
manifest → connector (`file`/`sql`/`http`) → deterministic materialization of
`ingest-{id}.md` files → index generation — plus the §11 golden fixture suite
(byte-exact) that no repo ships today. Zero model calls; zero runtime dependencies.
## Deliverables
1. **Manifest layer** — parse + fail-fast schema validation BEFORE any source call
(spec §4): `manifest_version == 1`, required top-level fields, polymorphic
`source.type`, extraction list validation (id grammar `[a-z0-9][a-z0-9-]*`,
unique ids, single-line titles, positive `max_rows`), and the verdict
reservation (spec §3): `okf_type` must not be `verdict` (case-insensitive) and
generated names must not fall in `promoted-verdict-*` or be `index.md`.
Hand-rolled validation (no pydantic — zero-dep rule); typed error hierarchy
rooted in `IngestError`.
2. **Connectors** (spec §4):
- `file`: CSV under `root`, boundary-checked fail-closed path resolution,
`utf-8-sig` decoding, streaming row cap, ragged-row rejection.
- `sql`: sqlite via `connection_ref` env-var resolution, single read-only
SELECT, bool-value rejection, sqlite errors wrapped in typed errors,
streaming row cap.
- `http`: GET against `base_url` + path, `credential_ref` env-var resolution,
hard network opt-in gate (spec §8): refuse fail-fast unless the per-run flag
is set; the flag is a run argument, never a manifest field. Transport behind
a small seam so tests inject a mock (spec §11: no live sources in tests).
3. **Materialization** (spec §5): explicit required `ingested_at`
(ISO-8601 UTC `Z`, regex-validated), frontmatter with exactly the seven keys in
spec order, body rendering per source type (cell escaping `\``\\`, `|`
`\|`, newline → space; integers plain decimal; floats shortest round-trip;
SQL NULL → empty string; any other type is an error), `ingest_manifest` stamp
`{stem}@{sha256(manifest bytes)[:16]}`, LF-only output with exactly one
trailing newline. Replacement semantics: stage everything in memory, run the
collision gate (a generated name colliding with an unstamped existing file is
an error) BEFORE any mutation, then remove stamped files, write the new set,
update the index.
4. **Index generation** (spec §6): create with `bundle_summary` when missing;
otherwise preserve every unmanaged line byte for byte; managed links
`- [{title}](ingest-{id}.md)` appended in extraction order, idempotent by
target; on re-materialization remove only links whose target was an
ingest-owned file removed this run.
5. **Public API** — one entry point, roughly
`materialize_bundle(manifest_path, bundle_dir, ingested_at, *, allow_network=False) -> IngestResult`,
plus the typed error hierarchy. Exact naming settled during TDD; the shape
(three explicit inputs + opt-in flag) is fixed by spec §5/§8.
6. **Golden fixtures** (spec §11): `examples/ingest-golden-file/` and
`examples/ingest-golden-sql/` (conformance MUSTs), plus
`examples/ingest-golden-http/` against mock payloads (documented as the
optional extension exercised). Each case: `manifest.json`, `fixture/`,
`ingested-at.txt`, `expected-bundle/`. Runnable offline without credentials.
7. **Load-bearing test table** (spec §11): one named test per seam — provenance
stamping, navigability, verdict reservation, re-ingest layer safety over a
promoted verdict, golden byte regression, network gate. (The spec-integrity
seam belongs to commons, which owns the spec.)
## Implementation baseline
The stricter behaviors named in `CLAUDE.md` are the floor, not options: streaming
row caps, `utf-8-sig`, in-memory staging with pre-mutation collision gate,
validated `ingested_at`, typed `IngestError`.
## TDD order
Each step starts with a failing test:
1. Manifest validation (schema, id grammar, verdict reservation, fail-fast order —
validation error must occur with zero source calls).
2. Body renderers (escaping, number forms, NULL, type rejection) as pure functions.
3. `file` connector (fixture catalogue; boundary check; BOM; caps).
4. `sql` connector (fixture sqlite file; env-var resolution; read-only; caps).
5. Materialization + collision gate + replacement semantics.
6. Index generation (fresh, preserve, idempotent, removal rules).
7. `http` connector + network gate (mock transport).
8. Golden fixtures authored, then the byte-for-byte conformance test.
9. Load-bearing table completed; each seam test verified red when its seam is
detached (mutation check by hand, noted in the test docstring).
## Key assumptions (each with its test)
| # | Assumption | Test |
|---|---|---|
| A1 | Spec v1 unchanged since this plan | Compare `sha256(ingest-spec.md)` against the hash above at implementation start |
| A2 | stdlib suffices (`csv`, `sqlite3`, `json`, `hashlib`, `urllib`, `re`) | Import smoke test + CI assert that `pyproject.toml` declares no runtime dependencies |
| A3 | Byte determinism holds across runs/platforms | Run materialization twice → `diff -r` empty; golden suite in CI |
| A4 | Float shortest round-trip (`repr`) is stable in Python ≥ 3.10 | Guaranteed since Python 3.1; covered by golden bytes anyway |
## Non-goals
- Consumer adoption (`portfolio-optimiser-claude`, then `portfolio-optimiser`)
is separate coordinated work in those repos — this phase only makes the library
adoption-ready. Spec changes, if any prove necessary, go via commons.
- Doors B/C, configurable contracts, Node — later phases.
- Per-row file splitting, multi-manifest bundles, incremental re-ingest,
approval registries — spec-declared extension points, not v1.
## Verification
All must pass before the phase is declared done:
1. `pytest` green, including the golden byte-for-byte comparison for all three
fixture cases.
2. `mypy --strict src/` clean; `ruff check .` and `ruff format --check .` clean.
3. Determinism: run the materializer twice on the same golden inputs into two
fresh directories; `diff -r` between them and against `expected-bundle/` is
empty.
4. Idempotence: re-run over the just-materialized bundle; `diff -r` still empty.
5. Layer safety: fixture bundle containing a `promoted-verdict-*` file and its
index link → after re-ingest both survive byte-identical (named test).
6. Verdict reservation: manifest with `okf_type: Verdict` rejected at validation,
before any source call (named test asserts no source I/O happened).
7. Network gate: `http` manifest without the opt-in flag fails fast with a typed
error; no socket use (mock transport asserts zero calls).
8. No wall-clock: `grep -rn "datetime.now\|time.time\|utcnow" src/` returns
nothing.
9. Zero deps: `pyproject.toml` has an empty runtime dependency list.
10. Offline: full test suite passes with network access disabled.

View file

@ -0,0 +1,123 @@
# Phase 2 plan — Door B (bundle inbox) + Door C (external bundle import)
Status: approved roadmap phase (see `CLAUDE.md`); details settled here before code.
Depends on: Phase 1 (materialization + index primitives are reused, never duplicated).
This phase adds the library's first — and only permitted — runtime dependency:
`llm-ingestion-guard>=0.2,<0.3`.
## Goal
Two guard-gated persist paths on top of the Phase 1 plumbing:
- **Door B — bundle inbox:** convert operator-dropped files into OKF concept
files. All file-type→text extraction lives here (the guard is text-only).
- **Door C — external bundle import:** assess a third-party OKF bundle per
concept via the guard's `okf.import_bundle`; merge, materialize, and index
only the non-rejected concepts.
The boundary rule is absolute: the guard answers "is this content safe to
persist?"; this library only connects, extracts, materializes, and indexes.
No scanning, sanitizing, or quarantine logic is implemented here.
## Door B — deliverables
1. **Extraction registry** — file extension → extractor:
- Core (stdlib only): `md`/`txt` passthrough, `csv` → markdown table (reuses
the Phase 1 renderer), `json` → verbatim inside a fenced block, `html`
text via `html.parser`.
- Optional (`[extract]` extra only): `pdf`, `docx`, `xlsx`. Without the extra
installed those types are rejected fail-fast with a typed error naming the
extra — never a silent skip, never a bundled parser in core.
2. **Inbox flow** — an explicit operator command (mirrors the spec §9 HITL rule:
never automatic, no watcher, no scheduler):
`process_inbox(inbox_dir, bundle_dir, ingested_at, ...) -> InboxResult`.
Per file: read bytes → extract text → guard gate → materialize concept →
index link. `ingested_at` explicit as in Phase 1; no wall-clock anywhere.
3. **Guard gate (persist gate #1):** extracted text passes through the guard
(`prepare_input`/`screen_output` bookends with an upload-grade policy) before
any write. Disposition handling: blocking dispositions → the file is NOT
persisted and is reported in `InboxResult`; quarantine-grade dispositions →
reported for operator review, not persisted (a quarantine directory is an
extension point, not v1); pass/warn → persisted with the report attached to
the result.
4. **Inbox provenance layer** — machine-generated files carry the honesty
marker, analogous to spec §7: `type`, `title`, `source_file` (inbox-relative
name), `source_sha256` (of the original bytes), `ingested_at`, `generated: true`.
Filenames namespaced `inbox-{slug}.md` with the slug reduced to the Phase 1
id grammar — disjoint from `index.md`, `promoted-verdict-*`, and `ingest-*`.
The verdict reservation and the pre-mutation collision gate apply unchanged.
## Door C — deliverables
1. **Import flow** — explicit operator command:
`import_bundle(source_dir, bundle_dir, ingested_at, *, origin, channel) -> ImportResult`.
Reads the external bundle as `{relative path -> text}`, hands it to the
guard's `okf.import_bundle(bundle, origin=…, channel=…, allow_reserved=False)`,
and merges ONLY concepts whose per-concept result carries no error.
2. **Merge + index:** accepted concepts go through the Phase 1 staging/collision
gate and index primitives (idempotent linking, curated content preserved).
Rejected concepts are reported per concept with the guard's reason — never
partially written.
3. **Import report:** the guard's log body (`BundleResult.log()`) is returned to
the caller in `ImportResult`; persisting it into the bundle is NOT done in v1
(reserved-file policy differs per consumer and becomes configurable in
Phase 3).
## Design decisions fixed by this plan
- Trust follows `origin`, never `channel` (guard doctrine) — callers must supply
both; no defaults.
- Extraction failure (corrupt file, missing extra) is a typed error on that file;
the run continues with the remaining files and the result reports per-file
outcomes. A guard FAIL-grade disposition on one file likewise never aborts the
whole inbox run.
- Determinism: same inbox bytes + same `ingested_at` + same guard version ⇒
byte-identical persisted output. Guard verdicts are part of that input surface.
## TDD order
1. Extraction registry: core types (fixtures per type), fail-fast on unknown
extensions, fail-fast on optional types without `[extract]`.
2. Inbox provenance rendering + filename slugging (pure functions).
3. Door B flow against a stub guard (in tests only — a test double standing in
for the pinned guard API, so persist/reject branches are exercised
deterministically; the real guard is exercised in integration tests).
4. Door B integration with the real guard: benign fixture persists; a fixture
the guard fails-secure on is not persisted.
5. Door C flow: mixed fixture bundle (accepted + rejected concepts) → only
accepted concepts merged; collision and curated-preservation semantics reused
from Phase 1 tests.
## Key assumptions (each with its test)
| # | Assumption | Test |
|---|---|---|
| B1 | Guard 0.2 API matches the pinned surface (`prepare_input`, `screen_output`, `okf.import_bundle`, disposition enum) | Import-and-signature smoke test that fails on upgrade drift |
| B2 | Guard is installable where this library's CI runs | Resolve the distribution channel with the operator before implementation starts (risk: not yet decided) |
| B3 | `html.parser`-based extraction is adequate for v1 | Golden fixtures for representative HTML; anything richer is explicitly out of scope |
| B4 | Guard verdicts are deterministic for fixed input + version | Same fixture run twice → identical `InboxResult` |
## Non-goals
- Any security decision logic (scan, sanitize, lexicons, fencing) — guard only.
- Quarantine storage/workflow and re-scan over time — extension points; the
guard's companion-expectations list is revisited in Phase 3.
- Model-assisted enrichment of inbox content — never in the run path.
- Watchers, schedulers, implicit triggers.
## Verification
1. `pytest` green with the guard installed; `mypy --strict src/`; `ruff check .`
and `ruff format --check .` clean.
2. `pdf` file without `[extract]` → typed error naming the extra; with
`pip install .[extract]` the same file extracts (integration test, may be
skipped in environments without the extra — but must run in CI).
3. Persist-gate proof: a fixture the guard fails-secure on results in ZERO new
files in the bundle and an explicit rejection entry in the result (named
test asserts the bundle directory is byte-identical before/after).
4. Door C partial-merge proof: mixed fixture → accepted concepts present,
rejected absent, index links only for accepted, curated lines untouched.
5. Phase 1 golden suite still passes byte-for-byte (no regression from reuse).
6. Grep-gate: `grep -rn "sanitize\|quarantine\|lexicon" src/` shows no local
security reimplementation (guard imports only).
7. `pyproject.toml` runtime dependencies == exactly `llm-ingestion-guard>=0.2,<0.3`.

View file

@ -0,0 +1,100 @@
# Phase 3 plan — Configurable bundle contract
Status: approved roadmap phase (see `CLAUDE.md`); details settled here before code.
Depends on: Phases 12 (their behavior becomes the *default profile* and must not change).
Proving consumer: `claude-code-llm-wiki` with a `strict-v1` profile.
## Goal
Everything the first two phases hard-code about what a valid bundle looks like —
concept types, layers/path prefixes, frontmatter sets and ordering, reserved-file
policy — becomes configuration on a profile object instead of constants. The
observable behavior under the default profile is byte-identical to Phase 1/2;
the golden suite is the regression harness for that claim.
## Design
1. **`BundleProfile`** — a frozen dataclass (stdlib, no config-file DSL in v1;
profiles are constructed in code by the consumer):
- `type_policy`: closed enum vs. open set; the reserved `verdict` exclusion
stays unconditional (it is a spec invariant, not profile config).
- `frontmatter_schema`: ordered required keys + optional keys, per concept
type where needed (Phase 1's seven-key layer and Phase 2's inbox layer are
expressions of this schema).
- `path_policy`: layer/path prefixes, filename namespaces, reserved filename
patterns (`index.md`, `promoted-verdict-*`, …).
- `reserved_files`: per-file required / optional / required-absent policy
(e.g. a profile may demand that `log.md` is absent).
- `index_policy`: managed-link format and per-level index expectations.
2. **Built-in profiles:**
- `DEFAULT` (implicit): exactly the ingest-spec v1 + Phase 2 contract.
- `STRICT_V1`: the `claude-code-llm-wiki` contract — closed type enum
(`Concept`/`Guide`/`Reference`/`Release`), extended required frontmatter
(`type`, `title`, `description`, `timestamp`, `layer`, `source_tier`,
`source_url`, `source_sha`), layer path prefixes, `log.md` required-absent,
root-index metadata.
3. **Plumbing change:** materialization, inbox, import, and index generation take
an optional `profile` argument defaulting to `DEFAULT`. No behavior branches
outside what the profile object expresses.
## The split table (settled before code)
The wiki's existing validator mixes three kinds of gates. First implementation
step is a written mapping of every gate to exactly one column — the mapping is
reviewed with the operator before any porting:
| Column | Meaning | Goes where |
|---|---|---|
| Profile | Structural bundle contract (types, keys, paths, reserved files) | This phase, `BundleProfile` |
| Guard | Content safety (sanitize, quarantine, fencing, secrets, budgets) | Guard calls — never reimplemented here |
| Consumer | Domain logic (source pipelines, promotion, staging) | Stays in the consumer repo |
## TDD order
1. `BundleProfile` type + `DEFAULT` profile; refactor Phase 1/2 constants onto
it with the full existing test suite as the safety net.
2. Golden suite re-run — byte-for-byte identical under `DEFAULT` (the phase's
load-bearing regression test).
3. `STRICT_V1` profile + fixtures: a wiki-shaped bundle that validates under
`STRICT_V1` and is rejected under `DEFAULT` (extra frontmatter keys are fine
under preserve-unknown; the *rejection* comes from the missing ingest-spec
layer), and an ingest-spec bundle rejected under `STRICT_V1` (missing
required keys, `log.md` policy).
4. Reserved-file policy tests (required-absent enforcement).
5. Split-table review artifact committed under `docs/` and cross-checked against
the implemented profile fields.
## Key assumptions (each with its test)
| # | Assumption | Test |
|---|---|---|
| C1 | Default-profile refactor is behavior-neutral | Phase 1 golden suite byte-identical; full Phase 1/2 test suite green, unmodified |
| C2 | `strict-v1` requirements are fully enumerable from the wiki's validator | The split table is exhaustive: every wiki gate appears in exactly one column; reviewed with the operator |
| C3 | Verdict reservation must survive every profile | Named test: no profile construction can permit `type: verdict` |
| C4 | Profiles-in-code (no config files) suffice for the proving consumer | Confirmed with the operator at phase start; config-file loading is an extension point |
## Non-goals
- Porting the wiki's security stack — those gates map to the Guard column.
- Config-file/DSL profile loading, profile inheritance chains — extension points.
- Adopting the second-brain (Node-world) conventions — that contract enters in
Phase 4; this phase only ensures the profile object is expressive enough not
to block it (checked against the catalog's spec during the split-table step).
- Any change to commons' ingest-spec; if profile work reveals spec friction, it
is raised in commons, not patched locally.
## Verification
1. `pytest` green; `mypy --strict src/`; `ruff check .` and
`ruff format --check .` clean.
2. Golden suite byte-for-byte under `DEFAULT` — zero fixture changes in the diff
(`git diff --stat examples/` empty for this phase).
3. `STRICT_V1` fixture bundle validates; cross-profile rejection tests pass in
both directions (step 3 above).
4. C3 named test: constructing a profile that admits `verdict` raises at
construction time.
5. The split table exists under `docs/`, every wiki gate row has exactly one
column, and the Guard column contains no rows implemented in this repo
(grep-gate from Phase 2 still clean).
6. Public API check: Phase 1/2 call sites compile unchanged (profile argument is
optional) — verified by leaving existing tests untouched.

View file

@ -0,0 +1,103 @@
# Phase 4 plan — the Node half (`node/`)
Status: approved roadmap phase (see `CLAUDE.md`); details settled here before code.
Depends on: Phase 3 (the profile object must be able to express the second-brain
contract; that expressiveness is checked during Phase 3's split-table step).
Hard precondition: the coordination steps below — this phase does not start with
code, it starts with agreements.
## Goal
A zero-dependency Node/ESM package under `node/` for the second-brain world:
bundle check, index generation, inbox split/frontmatter/write, and document
conversion (docx/pdf/eml/html → md). Importable as a library AND invokable as a
CLI (`node <path> <args>`, exit code + stdout — the established plugin
invocation pattern), consumed by vendoring per plugin, not npm publishing.
The Python and Node halves share the OKF contract and the fixture suite, never
code.
## Coordination first (ordered, each with an owner sign-off)
1. **okr agreement.** The reference implementations (`okf-check.mjs`,
`okf-index.mjs`, the inbox libraries and vocabulary/link modules) are lifted
only in agreement with okr; their test suite is ported alongside the code as
the seed corpus. Known deviations between okr's current behavior and the
catalog spec are resolved WITH the convention owner before the lift — the
library implements the agreed contract, not one repo's drift.
2. **Catalog stays convention owner.** The second-brain spec remains in the
marketplace catalog; this repo implements it. After the lift, the catalog
re-pins its shared check gate to this repo's copy — acceptance is behavior
parity on the shared fixture suite, and the re-pin follows the marketplace's
release-tag pinning discipline.
3. **linkedin-studio stays plugin-local.** Its `ingest/published/`
provenance-record grammar has a different lifecycle by design and is NOT
normalized into this library. Explicit non-goal, agreed up front.
4. **Persist gate = guard-as-contract.** There is no Node guard today. The Node
half documents and enforces the guard's *contract* at its persist seam
(fail-closed hook point, no content-safety logic of its own) so a future Node
guard from the security repo can drop in. No security reimplementation in
either runtime.
5. **ms-ai-architect as greenfield consumer.** Its designed-but-unbuilt needs
(writer, index generator, checker, retrieval support) are the acceptance
sketch for the package's API surface.
## Deliverables
1. **`node/` package skeleton:** ESM, `package.json` with NO `dependencies`,
`node:` builtins only, `node:test` for tests, no build step (vendorable as
plain files).
2. **Bundle check** (`okf check`): the catalog spec's rules — `type:` required,
per-level `index.md` without frontmatter, root `okf_version`, canonical
`resource` field, preserve-unknown / tolerate-broken posture.
3. **Index generation** (`okf index`): per-level index maintenance matching the
reference behavior.
4. **Inbox primitives:** split, frontmatter emit/parse, relations, write — the
lifted library modules under this repo's test suite.
5. **Document conversion** (`okf convert`): docx/pdf/eml/html → md. Conversion
parsers cannot be zero-dep; this mirrors the Python `[extract]` pattern —
the core package stays dependency-free, the conversion module lazy-loads
optional parsers and rejects those types fail-fast when they are absent.
Which parsers, and whether they are vendored or peer-installed, is settled in
coordination step 1 (okr had concrete candidates).
6. **CLI:** one entry (`node/bin/okf.mjs check|index|inbox|convert …`),
deterministic stdout, meaningful exit codes, no interactive prompts.
7. **Shared fixture suite:** cross-runtime fixtures for the second-brain
contract, consumed by BOTH the Node tests and (via the Phase 3 profile) the
Python tests, so the halves cannot silently diverge.
## Key assumptions (each with its test)
| # | Assumption | Test |
|---|---|---|
| D1 | okr's reference code + tests transfer cleanly to this repo's layout | Ported suite green before any behavior change; parity run against okr's copy on the shared fixtures |
| D2 | The catalog spec is the single contract both halves implement | Phase 3 profile expresses it; cross-runtime fixture suite passes in both runtimes |
| D3 | Zero-dep core is compatible with doc conversion | Lazy-load pattern proven by a test run WITHOUT optional parsers installed: core commands work, `convert` fails fast with a clear message |
| D4 | Vendoring workflow works for consumers | Dry-run: copy `node/` into a scratch plugin layout, run the CLI from there, exit codes intact |
## Non-goals
- npm publishing (vendoring only), bundling/transpiling, TypeScript build chain.
- Normalizing linkedin-studio's published-record grammar.
- A Node guard, or ANY content-safety logic — guard-as-contract until the
security repo ships a Node implementation.
- Python↔Node code sharing of any kind (contract and fixtures only).
## Verification
1. `node --test node/` green; ported reference tests included.
2. Zero-dep proof: `package.json` has no `dependencies`/`optionalDependencies`;
grep for non-`node:` imports in core modules returns nothing (conversion
module exempted, lazy-load only).
3. Parity: this repo's check gate and the currently vendored reference copy
produce identical verdicts on the full shared fixture suite (scripted
comparison, part of CI until the catalog re-pins).
4. Cross-runtime agreement: Python (Phase 3 profile) and Node checkers give the
same accept/reject verdict per shared fixture (scripted matrix run).
5. CLI contract: each subcommand exercised via `node node/bin/okf.mjs …` in
tests; exit codes asserted (0 clean, non-zero on violation), stdout
deterministic (two runs diff-empty).
6. D3 test: suite run in an environment without optional parsers — core green,
`convert` on a docx fails fast naming the missing parser.
7. Coordination artifacts: each of the five steps above has a recorded sign-off
before its dependent code lands (tracked in the session state, summarized in
the commit history).