docs(plan): add detailed phase 1-4 implementation plans
Phase 1 (Door A): ingest-spec v1 implementation + §11 golden fixtures, pinned to the current spec revision, with TDD order and byte-exact verification criteria. Phase 2 (Doors B/C): guard-gated inbox and external bundle import. Phase 3: configurable bundle contract via a profile object, default profile locked to the golden suite. Phase 4: zero-dep Node half, coordination-first with owner sign-offs. Each plan carries explicit key assumptions with tests and a verification section. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QeqhJpYQyghASjiJo5EhGg
This commit is contained in:
parent
e57afecf72
commit
b0ad2aedfb
4 changed files with 450 additions and 0 deletions
124
docs/plan/phase-1-door-a.md
Normal file
124
docs/plan/phase-1-door-a.md
Normal file
|
|
@ -0,0 +1,124 @@
|
||||||
|
# Phase 1 plan — Door A: ingest-spec implementation + §11 golden fixtures
|
||||||
|
|
||||||
|
Status: approved roadmap phase (see `CLAUDE.md`); this plan details it before any code.
|
||||||
|
Normative basis: `ingest-spec.md` v1 in `portfolio-optimiser-commons` (RFC-2119).
|
||||||
|
This plan was written against spec revision `sha256:959484e1952d28fa…` — if the spec has
|
||||||
|
changed when implementation starts, re-read it first and update this plan.
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
A stdlib-only Python implementation of the full ingest-spec contract:
|
||||||
|
manifest → connector (`file`/`sql`/`http`) → deterministic materialization of
|
||||||
|
`ingest-{id}.md` files → index generation — plus the §11 golden fixture suite
|
||||||
|
(byte-exact) that no repo ships today. Zero model calls; zero runtime dependencies.
|
||||||
|
|
||||||
|
## Deliverables
|
||||||
|
|
||||||
|
1. **Manifest layer** — parse + fail-fast schema validation BEFORE any source call
|
||||||
|
(spec §4): `manifest_version == 1`, required top-level fields, polymorphic
|
||||||
|
`source.type`, extraction list validation (id grammar `[a-z0-9][a-z0-9-]*`,
|
||||||
|
unique ids, single-line titles, positive `max_rows`), and the verdict
|
||||||
|
reservation (spec §3): `okf_type` must not be `verdict` (case-insensitive) and
|
||||||
|
generated names must not fall in `promoted-verdict-*` or be `index.md`.
|
||||||
|
Hand-rolled validation (no pydantic — zero-dep rule); typed error hierarchy
|
||||||
|
rooted in `IngestError`.
|
||||||
|
2. **Connectors** (spec §4):
|
||||||
|
- `file`: CSV under `root`, boundary-checked fail-closed path resolution,
|
||||||
|
`utf-8-sig` decoding, streaming row cap, ragged-row rejection.
|
||||||
|
- `sql`: sqlite via `connection_ref` env-var resolution, single read-only
|
||||||
|
SELECT, bool-value rejection, sqlite errors wrapped in typed errors,
|
||||||
|
streaming row cap.
|
||||||
|
- `http`: GET against `base_url` + path, `credential_ref` env-var resolution,
|
||||||
|
hard network opt-in gate (spec §8): refuse fail-fast unless the per-run flag
|
||||||
|
is set; the flag is a run argument, never a manifest field. Transport behind
|
||||||
|
a small seam so tests inject a mock (spec §11: no live sources in tests).
|
||||||
|
3. **Materialization** (spec §5): explicit required `ingested_at`
|
||||||
|
(ISO-8601 UTC `Z`, regex-validated), frontmatter with exactly the seven keys in
|
||||||
|
spec order, body rendering per source type (cell escaping `\` → `\\`, `|` →
|
||||||
|
`\|`, newline → space; integers plain decimal; floats shortest round-trip;
|
||||||
|
SQL NULL → empty string; any other type is an error), `ingest_manifest` stamp
|
||||||
|
`{stem}@{sha256(manifest bytes)[:16]}`, LF-only output with exactly one
|
||||||
|
trailing newline. Replacement semantics: stage everything in memory, run the
|
||||||
|
collision gate (a generated name colliding with an unstamped existing file is
|
||||||
|
an error) BEFORE any mutation, then remove stamped files, write the new set,
|
||||||
|
update the index.
|
||||||
|
4. **Index generation** (spec §6): create with `bundle_summary` when missing;
|
||||||
|
otherwise preserve every unmanaged line byte for byte; managed links
|
||||||
|
`- [{title}](ingest-{id}.md)` appended in extraction order, idempotent by
|
||||||
|
target; on re-materialization remove only links whose target was an
|
||||||
|
ingest-owned file removed this run.
|
||||||
|
5. **Public API** — one entry point, roughly
|
||||||
|
`materialize_bundle(manifest_path, bundle_dir, ingested_at, *, allow_network=False) -> IngestResult`,
|
||||||
|
plus the typed error hierarchy. Exact naming settled during TDD; the shape
|
||||||
|
(three explicit inputs + opt-in flag) is fixed by spec §5/§8.
|
||||||
|
6. **Golden fixtures** (spec §11): `examples/ingest-golden-file/` and
|
||||||
|
`examples/ingest-golden-sql/` (conformance MUSTs), plus
|
||||||
|
`examples/ingest-golden-http/` against mock payloads (documented as the
|
||||||
|
optional extension exercised). Each case: `manifest.json`, `fixture/`,
|
||||||
|
`ingested-at.txt`, `expected-bundle/`. Runnable offline without credentials.
|
||||||
|
7. **Load-bearing test table** (spec §11): one named test per seam — provenance
|
||||||
|
stamping, navigability, verdict reservation, re-ingest layer safety over a
|
||||||
|
promoted verdict, golden byte regression, network gate. (The spec-integrity
|
||||||
|
seam belongs to commons, which owns the spec.)
|
||||||
|
|
||||||
|
## Implementation baseline
|
||||||
|
|
||||||
|
The stricter behaviors named in `CLAUDE.md` are the floor, not options: streaming
|
||||||
|
row caps, `utf-8-sig`, in-memory staging with pre-mutation collision gate,
|
||||||
|
validated `ingested_at`, typed `IngestError`.
|
||||||
|
|
||||||
|
## TDD order
|
||||||
|
|
||||||
|
Each step starts with a failing test:
|
||||||
|
|
||||||
|
1. Manifest validation (schema, id grammar, verdict reservation, fail-fast order —
|
||||||
|
validation error must occur with zero source calls).
|
||||||
|
2. Body renderers (escaping, number forms, NULL, type rejection) as pure functions.
|
||||||
|
3. `file` connector (fixture catalogue; boundary check; BOM; caps).
|
||||||
|
4. `sql` connector (fixture sqlite file; env-var resolution; read-only; caps).
|
||||||
|
5. Materialization + collision gate + replacement semantics.
|
||||||
|
6. Index generation (fresh, preserve, idempotent, removal rules).
|
||||||
|
7. `http` connector + network gate (mock transport).
|
||||||
|
8. Golden fixtures authored, then the byte-for-byte conformance test.
|
||||||
|
9. Load-bearing table completed; each seam test verified red when its seam is
|
||||||
|
detached (mutation check by hand, noted in the test docstring).
|
||||||
|
|
||||||
|
## Key assumptions (each with its test)
|
||||||
|
|
||||||
|
| # | Assumption | Test |
|
||||||
|
|---|---|---|
|
||||||
|
| A1 | Spec v1 unchanged since this plan | Compare `sha256(ingest-spec.md)` against the hash above at implementation start |
|
||||||
|
| A2 | stdlib suffices (`csv`, `sqlite3`, `json`, `hashlib`, `urllib`, `re`) | Import smoke test + CI assert that `pyproject.toml` declares no runtime dependencies |
|
||||||
|
| A3 | Byte determinism holds across runs/platforms | Run materialization twice → `diff -r` empty; golden suite in CI |
|
||||||
|
| A4 | Float shortest round-trip (`repr`) is stable in Python ≥ 3.10 | Guaranteed since Python 3.1; covered by golden bytes anyway |
|
||||||
|
|
||||||
|
## Non-goals
|
||||||
|
|
||||||
|
- Consumer adoption (`portfolio-optimiser-claude`, then `portfolio-optimiser`)
|
||||||
|
is separate coordinated work in those repos — this phase only makes the library
|
||||||
|
adoption-ready. Spec changes, if any prove necessary, go via commons.
|
||||||
|
- Doors B/C, configurable contracts, Node — later phases.
|
||||||
|
- Per-row file splitting, multi-manifest bundles, incremental re-ingest,
|
||||||
|
approval registries — spec-declared extension points, not v1.
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
All must pass before the phase is declared done:
|
||||||
|
|
||||||
|
1. `pytest` green, including the golden byte-for-byte comparison for all three
|
||||||
|
fixture cases.
|
||||||
|
2. `mypy --strict src/` clean; `ruff check .` and `ruff format --check .` clean.
|
||||||
|
3. Determinism: run the materializer twice on the same golden inputs into two
|
||||||
|
fresh directories; `diff -r` between them and against `expected-bundle/` is
|
||||||
|
empty.
|
||||||
|
4. Idempotence: re-run over the just-materialized bundle; `diff -r` still empty.
|
||||||
|
5. Layer safety: fixture bundle containing a `promoted-verdict-*` file and its
|
||||||
|
index link → after re-ingest both survive byte-identical (named test).
|
||||||
|
6. Verdict reservation: manifest with `okf_type: Verdict` rejected at validation,
|
||||||
|
before any source call (named test asserts no source I/O happened).
|
||||||
|
7. Network gate: `http` manifest without the opt-in flag fails fast with a typed
|
||||||
|
error; no socket use (mock transport asserts zero calls).
|
||||||
|
8. No wall-clock: `grep -rn "datetime.now\|time.time\|utcnow" src/` returns
|
||||||
|
nothing.
|
||||||
|
9. Zero deps: `pyproject.toml` has an empty runtime dependency list.
|
||||||
|
10. Offline: full test suite passes with network access disabled.
|
||||||
123
docs/plan/phase-2-doors-b-c.md
Normal file
123
docs/plan/phase-2-doors-b-c.md
Normal file
|
|
@ -0,0 +1,123 @@
|
||||||
|
# Phase 2 plan — Door B (bundle inbox) + Door C (external bundle import)
|
||||||
|
|
||||||
|
Status: approved roadmap phase (see `CLAUDE.md`); details settled here before code.
|
||||||
|
Depends on: Phase 1 (materialization + index primitives are reused, never duplicated).
|
||||||
|
This phase adds the library's first — and only permitted — runtime dependency:
|
||||||
|
`llm-ingestion-guard>=0.2,<0.3`.
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Two guard-gated persist paths on top of the Phase 1 plumbing:
|
||||||
|
|
||||||
|
- **Door B — bundle inbox:** convert operator-dropped files into OKF concept
|
||||||
|
files. All file-type→text extraction lives here (the guard is text-only).
|
||||||
|
- **Door C — external bundle import:** assess a third-party OKF bundle per
|
||||||
|
concept via the guard's `okf.import_bundle`; merge, materialize, and index
|
||||||
|
only the non-rejected concepts.
|
||||||
|
|
||||||
|
The boundary rule is absolute: the guard answers "is this content safe to
|
||||||
|
persist?"; this library only connects, extracts, materializes, and indexes.
|
||||||
|
No scanning, sanitizing, or quarantine logic is implemented here.
|
||||||
|
|
||||||
|
## Door B — deliverables
|
||||||
|
|
||||||
|
1. **Extraction registry** — file extension → extractor:
|
||||||
|
- Core (stdlib only): `md`/`txt` passthrough, `csv` → markdown table (reuses
|
||||||
|
the Phase 1 renderer), `json` → verbatim inside a fenced block, `html` →
|
||||||
|
text via `html.parser`.
|
||||||
|
- Optional (`[extract]` extra only): `pdf`, `docx`, `xlsx`. Without the extra
|
||||||
|
installed those types are rejected fail-fast with a typed error naming the
|
||||||
|
extra — never a silent skip, never a bundled parser in core.
|
||||||
|
2. **Inbox flow** — an explicit operator command (mirrors the spec §9 HITL rule:
|
||||||
|
never automatic, no watcher, no scheduler):
|
||||||
|
`process_inbox(inbox_dir, bundle_dir, ingested_at, ...) -> InboxResult`.
|
||||||
|
Per file: read bytes → extract text → guard gate → materialize concept →
|
||||||
|
index link. `ingested_at` explicit as in Phase 1; no wall-clock anywhere.
|
||||||
|
3. **Guard gate (persist gate #1):** extracted text passes through the guard
|
||||||
|
(`prepare_input`/`screen_output` bookends with an upload-grade policy) before
|
||||||
|
any write. Disposition handling: blocking dispositions → the file is NOT
|
||||||
|
persisted and is reported in `InboxResult`; quarantine-grade dispositions →
|
||||||
|
reported for operator review, not persisted (a quarantine directory is an
|
||||||
|
extension point, not v1); pass/warn → persisted with the report attached to
|
||||||
|
the result.
|
||||||
|
4. **Inbox provenance layer** — machine-generated files carry the honesty
|
||||||
|
marker, analogous to spec §7: `type`, `title`, `source_file` (inbox-relative
|
||||||
|
name), `source_sha256` (of the original bytes), `ingested_at`, `generated: true`.
|
||||||
|
Filenames namespaced `inbox-{slug}.md` with the slug reduced to the Phase 1
|
||||||
|
id grammar — disjoint from `index.md`, `promoted-verdict-*`, and `ingest-*`.
|
||||||
|
The verdict reservation and the pre-mutation collision gate apply unchanged.
|
||||||
|
|
||||||
|
## Door C — deliverables
|
||||||
|
|
||||||
|
1. **Import flow** — explicit operator command:
|
||||||
|
`import_bundle(source_dir, bundle_dir, ingested_at, *, origin, channel) -> ImportResult`.
|
||||||
|
Reads the external bundle as `{relative path -> text}`, hands it to the
|
||||||
|
guard's `okf.import_bundle(bundle, origin=…, channel=…, allow_reserved=False)`,
|
||||||
|
and merges ONLY concepts whose per-concept result carries no error.
|
||||||
|
2. **Merge + index:** accepted concepts go through the Phase 1 staging/collision
|
||||||
|
gate and index primitives (idempotent linking, curated content preserved).
|
||||||
|
Rejected concepts are reported per concept with the guard's reason — never
|
||||||
|
partially written.
|
||||||
|
3. **Import report:** the guard's log body (`BundleResult.log()`) is returned to
|
||||||
|
the caller in `ImportResult`; persisting it into the bundle is NOT done in v1
|
||||||
|
(reserved-file policy differs per consumer and becomes configurable in
|
||||||
|
Phase 3).
|
||||||
|
|
||||||
|
## Design decisions fixed by this plan
|
||||||
|
|
||||||
|
- Trust follows `origin`, never `channel` (guard doctrine) — callers must supply
|
||||||
|
both; no defaults.
|
||||||
|
- Extraction failure (corrupt file, missing extra) is a typed error on that file;
|
||||||
|
the run continues with the remaining files and the result reports per-file
|
||||||
|
outcomes. A guard FAIL-grade disposition on one file likewise never aborts the
|
||||||
|
whole inbox run.
|
||||||
|
- Determinism: same inbox bytes + same `ingested_at` + same guard version ⇒
|
||||||
|
byte-identical persisted output. Guard verdicts are part of that input surface.
|
||||||
|
|
||||||
|
## TDD order
|
||||||
|
|
||||||
|
1. Extraction registry: core types (fixtures per type), fail-fast on unknown
|
||||||
|
extensions, fail-fast on optional types without `[extract]`.
|
||||||
|
2. Inbox provenance rendering + filename slugging (pure functions).
|
||||||
|
3. Door B flow against a stub guard (in tests only — a test double standing in
|
||||||
|
for the pinned guard API, so persist/reject branches are exercised
|
||||||
|
deterministically; the real guard is exercised in integration tests).
|
||||||
|
4. Door B integration with the real guard: benign fixture persists; a fixture
|
||||||
|
the guard fails-secure on is not persisted.
|
||||||
|
5. Door C flow: mixed fixture bundle (accepted + rejected concepts) → only
|
||||||
|
accepted concepts merged; collision and curated-preservation semantics reused
|
||||||
|
from Phase 1 tests.
|
||||||
|
|
||||||
|
## Key assumptions (each with its test)
|
||||||
|
|
||||||
|
| # | Assumption | Test |
|
||||||
|
|---|---|---|
|
||||||
|
| B1 | Guard 0.2 API matches the pinned surface (`prepare_input`, `screen_output`, `okf.import_bundle`, disposition enum) | Import-and-signature smoke test that fails on upgrade drift |
|
||||||
|
| B2 | Guard is installable where this library's CI runs | Resolve the distribution channel with the operator before implementation starts (risk: not yet decided) |
|
||||||
|
| B3 | `html.parser`-based extraction is adequate for v1 | Golden fixtures for representative HTML; anything richer is explicitly out of scope |
|
||||||
|
| B4 | Guard verdicts are deterministic for fixed input + version | Same fixture run twice → identical `InboxResult` |
|
||||||
|
|
||||||
|
## Non-goals
|
||||||
|
|
||||||
|
- Any security decision logic (scan, sanitize, lexicons, fencing) — guard only.
|
||||||
|
- Quarantine storage/workflow and re-scan over time — extension points; the
|
||||||
|
guard's companion-expectations list is revisited in Phase 3.
|
||||||
|
- Model-assisted enrichment of inbox content — never in the run path.
|
||||||
|
- Watchers, schedulers, implicit triggers.
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
1. `pytest` green with the guard installed; `mypy --strict src/`; `ruff check .`
|
||||||
|
and `ruff format --check .` clean.
|
||||||
|
2. `pdf` file without `[extract]` → typed error naming the extra; with
|
||||||
|
`pip install .[extract]` the same file extracts (integration test, may be
|
||||||
|
skipped in environments without the extra — but must run in CI).
|
||||||
|
3. Persist-gate proof: a fixture the guard fails-secure on results in ZERO new
|
||||||
|
files in the bundle and an explicit rejection entry in the result (named
|
||||||
|
test asserts the bundle directory is byte-identical before/after).
|
||||||
|
4. Door C partial-merge proof: mixed fixture → accepted concepts present,
|
||||||
|
rejected absent, index links only for accepted, curated lines untouched.
|
||||||
|
5. Phase 1 golden suite still passes byte-for-byte (no regression from reuse).
|
||||||
|
6. Grep-gate: `grep -rn "sanitize\|quarantine\|lexicon" src/` shows no local
|
||||||
|
security reimplementation (guard imports only).
|
||||||
|
7. `pyproject.toml` runtime dependencies == exactly `llm-ingestion-guard>=0.2,<0.3`.
|
||||||
100
docs/plan/phase-3-configurable-contract.md
Normal file
100
docs/plan/phase-3-configurable-contract.md
Normal file
|
|
@ -0,0 +1,100 @@
|
||||||
|
# Phase 3 plan — Configurable bundle contract
|
||||||
|
|
||||||
|
Status: approved roadmap phase (see `CLAUDE.md`); details settled here before code.
|
||||||
|
Depends on: Phases 1–2 (their behavior becomes the *default profile* and must not change).
|
||||||
|
Proving consumer: `claude-code-llm-wiki` with a `strict-v1` profile.
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Everything the first two phases hard-code about what a valid bundle looks like —
|
||||||
|
concept types, layers/path prefixes, frontmatter sets and ordering, reserved-file
|
||||||
|
policy — becomes configuration on a profile object instead of constants. The
|
||||||
|
observable behavior under the default profile is byte-identical to Phase 1/2;
|
||||||
|
the golden suite is the regression harness for that claim.
|
||||||
|
|
||||||
|
## Design
|
||||||
|
|
||||||
|
1. **`BundleProfile`** — a frozen dataclass (stdlib, no config-file DSL in v1;
|
||||||
|
profiles are constructed in code by the consumer):
|
||||||
|
- `type_policy`: closed enum vs. open set; the reserved `verdict` exclusion
|
||||||
|
stays unconditional (it is a spec invariant, not profile config).
|
||||||
|
- `frontmatter_schema`: ordered required keys + optional keys, per concept
|
||||||
|
type where needed (Phase 1's seven-key layer and Phase 2's inbox layer are
|
||||||
|
expressions of this schema).
|
||||||
|
- `path_policy`: layer/path prefixes, filename namespaces, reserved filename
|
||||||
|
patterns (`index.md`, `promoted-verdict-*`, …).
|
||||||
|
- `reserved_files`: per-file required / optional / required-absent policy
|
||||||
|
(e.g. a profile may demand that `log.md` is absent).
|
||||||
|
- `index_policy`: managed-link format and per-level index expectations.
|
||||||
|
2. **Built-in profiles:**
|
||||||
|
- `DEFAULT` (implicit): exactly the ingest-spec v1 + Phase 2 contract.
|
||||||
|
- `STRICT_V1`: the `claude-code-llm-wiki` contract — closed type enum
|
||||||
|
(`Concept`/`Guide`/`Reference`/`Release`), extended required frontmatter
|
||||||
|
(`type`, `title`, `description`, `timestamp`, `layer`, `source_tier`,
|
||||||
|
`source_url`, `source_sha`), layer path prefixes, `log.md` required-absent,
|
||||||
|
root-index metadata.
|
||||||
|
3. **Plumbing change:** materialization, inbox, import, and index generation take
|
||||||
|
an optional `profile` argument defaulting to `DEFAULT`. No behavior branches
|
||||||
|
outside what the profile object expresses.
|
||||||
|
|
||||||
|
## The split table (settled before code)
|
||||||
|
|
||||||
|
The wiki's existing validator mixes three kinds of gates. First implementation
|
||||||
|
step is a written mapping of every gate to exactly one column — the mapping is
|
||||||
|
reviewed with the operator before any porting:
|
||||||
|
|
||||||
|
| Column | Meaning | Goes where |
|
||||||
|
|---|---|---|
|
||||||
|
| Profile | Structural bundle contract (types, keys, paths, reserved files) | This phase, `BundleProfile` |
|
||||||
|
| Guard | Content safety (sanitize, quarantine, fencing, secrets, budgets) | Guard calls — never reimplemented here |
|
||||||
|
| Consumer | Domain logic (source pipelines, promotion, staging) | Stays in the consumer repo |
|
||||||
|
|
||||||
|
## TDD order
|
||||||
|
|
||||||
|
1. `BundleProfile` type + `DEFAULT` profile; refactor Phase 1/2 constants onto
|
||||||
|
it with the full existing test suite as the safety net.
|
||||||
|
2. Golden suite re-run — byte-for-byte identical under `DEFAULT` (the phase's
|
||||||
|
load-bearing regression test).
|
||||||
|
3. `STRICT_V1` profile + fixtures: a wiki-shaped bundle that validates under
|
||||||
|
`STRICT_V1` and is rejected under `DEFAULT` (extra frontmatter keys are fine
|
||||||
|
under preserve-unknown; the *rejection* comes from the missing ingest-spec
|
||||||
|
layer), and an ingest-spec bundle rejected under `STRICT_V1` (missing
|
||||||
|
required keys, `log.md` policy).
|
||||||
|
4. Reserved-file policy tests (required-absent enforcement).
|
||||||
|
5. Split-table review artifact committed under `docs/` and cross-checked against
|
||||||
|
the implemented profile fields.
|
||||||
|
|
||||||
|
## Key assumptions (each with its test)
|
||||||
|
|
||||||
|
| # | Assumption | Test |
|
||||||
|
|---|---|---|
|
||||||
|
| C1 | Default-profile refactor is behavior-neutral | Phase 1 golden suite byte-identical; full Phase 1/2 test suite green, unmodified |
|
||||||
|
| C2 | `strict-v1` requirements are fully enumerable from the wiki's validator | The split table is exhaustive: every wiki gate appears in exactly one column; reviewed with the operator |
|
||||||
|
| C3 | Verdict reservation must survive every profile | Named test: no profile construction can permit `type: verdict` |
|
||||||
|
| C4 | Profiles-in-code (no config files) suffice for the proving consumer | Confirmed with the operator at phase start; config-file loading is an extension point |
|
||||||
|
|
||||||
|
## Non-goals
|
||||||
|
|
||||||
|
- Porting the wiki's security stack — those gates map to the Guard column.
|
||||||
|
- Config-file/DSL profile loading, profile inheritance chains — extension points.
|
||||||
|
- Adopting the second-brain (Node-world) conventions — that contract enters in
|
||||||
|
Phase 4; this phase only ensures the profile object is expressive enough not
|
||||||
|
to block it (checked against the catalog's spec during the split-table step).
|
||||||
|
- Any change to commons' ingest-spec; if profile work reveals spec friction, it
|
||||||
|
is raised in commons, not patched locally.
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
1. `pytest` green; `mypy --strict src/`; `ruff check .` and
|
||||||
|
`ruff format --check .` clean.
|
||||||
|
2. Golden suite byte-for-byte under `DEFAULT` — zero fixture changes in the diff
|
||||||
|
(`git diff --stat examples/` empty for this phase).
|
||||||
|
3. `STRICT_V1` fixture bundle validates; cross-profile rejection tests pass in
|
||||||
|
both directions (step 3 above).
|
||||||
|
4. C3 named test: constructing a profile that admits `verdict` raises at
|
||||||
|
construction time.
|
||||||
|
5. The split table exists under `docs/`, every wiki gate row has exactly one
|
||||||
|
column, and the Guard column contains no rows implemented in this repo
|
||||||
|
(grep-gate from Phase 2 still clean).
|
||||||
|
6. Public API check: Phase 1/2 call sites compile unchanged (profile argument is
|
||||||
|
optional) — verified by leaving existing tests untouched.
|
||||||
103
docs/plan/phase-4-node-half.md
Normal file
103
docs/plan/phase-4-node-half.md
Normal file
|
|
@ -0,0 +1,103 @@
|
||||||
|
# Phase 4 plan — the Node half (`node/`)
|
||||||
|
|
||||||
|
Status: approved roadmap phase (see `CLAUDE.md`); details settled here before code.
|
||||||
|
Depends on: Phase 3 (the profile object must be able to express the second-brain
|
||||||
|
contract; that expressiveness is checked during Phase 3's split-table step).
|
||||||
|
Hard precondition: the coordination steps below — this phase does not start with
|
||||||
|
code, it starts with agreements.
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
A zero-dependency Node/ESM package under `node/` for the second-brain world:
|
||||||
|
bundle check, index generation, inbox split/frontmatter/write, and document
|
||||||
|
conversion (docx/pdf/eml/html → md). Importable as a library AND invokable as a
|
||||||
|
CLI (`node <path> <args>`, exit code + stdout — the established plugin
|
||||||
|
invocation pattern), consumed by vendoring per plugin, not npm publishing.
|
||||||
|
The Python and Node halves share the OKF contract and the fixture suite, never
|
||||||
|
code.
|
||||||
|
|
||||||
|
## Coordination first (ordered, each with an owner sign-off)
|
||||||
|
|
||||||
|
1. **okr agreement.** The reference implementations (`okf-check.mjs`,
|
||||||
|
`okf-index.mjs`, the inbox libraries and vocabulary/link modules) are lifted
|
||||||
|
only in agreement with okr; their test suite is ported alongside the code as
|
||||||
|
the seed corpus. Known deviations between okr's current behavior and the
|
||||||
|
catalog spec are resolved WITH the convention owner before the lift — the
|
||||||
|
library implements the agreed contract, not one repo's drift.
|
||||||
|
2. **Catalog stays convention owner.** The second-brain spec remains in the
|
||||||
|
marketplace catalog; this repo implements it. After the lift, the catalog
|
||||||
|
re-pins its shared check gate to this repo's copy — acceptance is behavior
|
||||||
|
parity on the shared fixture suite, and the re-pin follows the marketplace's
|
||||||
|
release-tag pinning discipline.
|
||||||
|
3. **linkedin-studio stays plugin-local.** Its `ingest/published/`
|
||||||
|
provenance-record grammar has a different lifecycle by design and is NOT
|
||||||
|
normalized into this library. Explicit non-goal, agreed up front.
|
||||||
|
4. **Persist gate = guard-as-contract.** There is no Node guard today. The Node
|
||||||
|
half documents and enforces the guard's *contract* at its persist seam
|
||||||
|
(fail-closed hook point, no content-safety logic of its own) so a future Node
|
||||||
|
guard from the security repo can drop in. No security reimplementation in
|
||||||
|
either runtime.
|
||||||
|
5. **ms-ai-architect as greenfield consumer.** Its designed-but-unbuilt needs
|
||||||
|
(writer, index generator, checker, retrieval support) are the acceptance
|
||||||
|
sketch for the package's API surface.
|
||||||
|
|
||||||
|
## Deliverables
|
||||||
|
|
||||||
|
1. **`node/` package skeleton:** ESM, `package.json` with NO `dependencies`,
|
||||||
|
`node:` builtins only, `node:test` for tests, no build step (vendorable as
|
||||||
|
plain files).
|
||||||
|
2. **Bundle check** (`okf check`): the catalog spec's rules — `type:` required,
|
||||||
|
per-level `index.md` without frontmatter, root `okf_version`, canonical
|
||||||
|
`resource` field, preserve-unknown / tolerate-broken posture.
|
||||||
|
3. **Index generation** (`okf index`): per-level index maintenance matching the
|
||||||
|
reference behavior.
|
||||||
|
4. **Inbox primitives:** split, frontmatter emit/parse, relations, write — the
|
||||||
|
lifted library modules under this repo's test suite.
|
||||||
|
5. **Document conversion** (`okf convert`): docx/pdf/eml/html → md. Conversion
|
||||||
|
parsers cannot be zero-dep; this mirrors the Python `[extract]` pattern —
|
||||||
|
the core package stays dependency-free, the conversion module lazy-loads
|
||||||
|
optional parsers and rejects those types fail-fast when they are absent.
|
||||||
|
Which parsers, and whether they are vendored or peer-installed, is settled in
|
||||||
|
coordination step 1 (okr had concrete candidates).
|
||||||
|
6. **CLI:** one entry (`node/bin/okf.mjs check|index|inbox|convert …`),
|
||||||
|
deterministic stdout, meaningful exit codes, no interactive prompts.
|
||||||
|
7. **Shared fixture suite:** cross-runtime fixtures for the second-brain
|
||||||
|
contract, consumed by BOTH the Node tests and (via the Phase 3 profile) the
|
||||||
|
Python tests, so the halves cannot silently diverge.
|
||||||
|
|
||||||
|
## Key assumptions (each with its test)
|
||||||
|
|
||||||
|
| # | Assumption | Test |
|
||||||
|
|---|---|---|
|
||||||
|
| D1 | okr's reference code + tests transfer cleanly to this repo's layout | Ported suite green before any behavior change; parity run against okr's copy on the shared fixtures |
|
||||||
|
| D2 | The catalog spec is the single contract both halves implement | Phase 3 profile expresses it; cross-runtime fixture suite passes in both runtimes |
|
||||||
|
| D3 | Zero-dep core is compatible with doc conversion | Lazy-load pattern proven by a test run WITHOUT optional parsers installed: core commands work, `convert` fails fast with a clear message |
|
||||||
|
| D4 | Vendoring workflow works for consumers | Dry-run: copy `node/` into a scratch plugin layout, run the CLI from there, exit codes intact |
|
||||||
|
|
||||||
|
## Non-goals
|
||||||
|
|
||||||
|
- npm publishing (vendoring only), bundling/transpiling, TypeScript build chain.
|
||||||
|
- Normalizing linkedin-studio's published-record grammar.
|
||||||
|
- A Node guard, or ANY content-safety logic — guard-as-contract until the
|
||||||
|
security repo ships a Node implementation.
|
||||||
|
- Python↔Node code sharing of any kind (contract and fixtures only).
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
1. `node --test node/` green; ported reference tests included.
|
||||||
|
2. Zero-dep proof: `package.json` has no `dependencies`/`optionalDependencies`;
|
||||||
|
grep for non-`node:` imports in core modules returns nothing (conversion
|
||||||
|
module exempted, lazy-load only).
|
||||||
|
3. Parity: this repo's check gate and the currently vendored reference copy
|
||||||
|
produce identical verdicts on the full shared fixture suite (scripted
|
||||||
|
comparison, part of CI until the catalog re-pins).
|
||||||
|
4. Cross-runtime agreement: Python (Phase 3 profile) and Node checkers give the
|
||||||
|
same accept/reject verdict per shared fixture (scripted matrix run).
|
||||||
|
5. CLI contract: each subcommand exercised via `node node/bin/okf.mjs …` in
|
||||||
|
tests; exit codes asserted (0 clean, non-zero on violation), stdout
|
||||||
|
deterministic (two runs diff-empty).
|
||||||
|
6. D3 test: suite run in an environment without optional parsers — core green,
|
||||||
|
`convert` on a docx fails fast naming the missing parser.
|
||||||
|
7. Coordination artifacts: each of the five steps above has a recorded sign-off
|
||||||
|
before its dependent code lands (tracked in the session state, summarized in
|
||||||
|
the commit history).
|
||||||
Loading…
Add table
Add a link
Reference in a new issue