diff --git a/docs/second-brain/brief-sb-s1.md b/docs/second-brain/brief-sb-s1.md new file mode 100644 index 0000000..3a00b19 --- /dev/null +++ b/docs/second-brain/brief-sb-s1.md @@ -0,0 +1,113 @@ +# Task Brief — SB-S1 (Ingest + gold signal) + +> **Status:** DRAFT for adversarial brief-review (light-Voyage step 2). Not yet operator-approved. +> **Arc:** `architecture.md` (approved 2026-06-23). **Predecessor:** SB-S0 landed (`8c92719`) — `scripts/brain/` (types/id/profile/scaffold/cli, 34 tests, gate-wired). **This brief:** the second slice in detail; S2–S4 stay one-line in the arc. +> **Inputs:** `architecture.md` §"evolution loop" steps 1–2, §"invariants" (provenance-weighted learning); SB-S0 code under `scripts/brain/`; the silo inventory (`docs/second-brain/research/silo-inventory.md`). +> **Review delta (2026-06-23):** folded the brief-review (PROCEED_WITH_RISKS). Resolver standardised on `dataRoot` (M-resolver); D3 voice-samples↔ingest relationship pinned (M-D3); SC5 hardened to a gate-checkable lint (M-SC5); test-runner floor edits named as deliverables (m-floor); input path + D4 net-new note + Q3 sharpened (minors). + +--- + +## 1. Objective + +Capture the **gold signal** — the user's actual *published* posts — into `ingest/published/` tagged `provenance=published`, and **wire the voice/profile-learning surface to learn from that published signal only**, never from AI-drafted content. This is the one invariant the whole second-brain stands or falls on (the model-collapse guard): a content engine that learns its own voice from what it itself drafted collapses toward its own priors. SB-S1 lays the gold signal **before** the consolidation loop (SB-S2) that will consume it. + +## 2. Background (why this slice, why now) + +- **The invariant already has a single-agent precedent.** `voice-scrubber` enforces *"gold standard = approved Norwegian editions, never the English post corpus."* SB-S1 generalises that one agent's rule into a system-wide, data-backed invariant: learning consumes `provenance=published` only. +- **The signal is genuinely new, not a re-count of existing silos.** A published post lands today in three non-referencing places — state `## Recent Posts` (hook + topic only), the dead `analytics/content-history.md`, and `analytics/posts/*.json` (metrics, from CSV). **None holds the full post text.** Voice learning needs the verbatim text; `ingest/published/` holding it is additive, not redundant. (Cross-silo *threading* of a shared id is SB-S3, explicitly out of scope here.) +- **The provenance shape is already in place.** SB-S0 pinned `Provenance = human | published | ai-draft` and put a `provenance` field on every `ProfileFact`. SB-S1 is the first slice to actually *produce* `published` records and *gate* a learning consumer on them — it spends the spine S0 built. +- **Manual-first is the contract.** Auto-ingest (EU/EEA DMA portability) is a later additive tributary (SB-S4); SB-S1's ingest is the manual path: the user supplies the published post text, the CLI lands it as a gold record. + +## 3. Confirmed decisions (inherited from the approved arc — not for re-litigation) + +1. **Boundary:** engine → plugin · user data → data dir · cockpit → Maskinrommet. +2. **Plain-text/JSON + git, no vector/graph DB.** Claude is the retrieval engine. +3. **Provenance-weighted learning** (`published` only, never `ai-draft`) is an invariant, not a toggle. +4. **Tributaries keep their own schema/lifecycle** — SB-S1 does NOT reshape voice-samples, specifics, trends, or analytics stores. +5. **No new seam function** — ingest paths resolve through the brain package's own inlined `dataRoot(sub)` resolver (`scripts/brain/src/dataRoot.ts`, the repo idiom — NOT the hooks-side `getDataRoot()` seam, a different function in a different package). So the twin-sync surface (`data-root.mjs ⇄ storage.ts`) stays untouched. + +Review SHOULD challenge: the S1/S2 boundary (§5/§7), the store format (§5.1), feasibility, hidden coupling, testability, scope leak, missing risks. Review SHOULD NOT re-open the five decisions above unless internally contradictory or technically impossible. + +## 4. Scope — where SB-S1 sits in the arc + +| Slice | One-line scope | This brief | +|-------|----------------|------------| +| SB-S0 — Foundation | scaffold + profile fold + id/provenance spine | ✅ landed | +| **SB-S1 — Ingest + gold signal** | manual import → `ingest/published/` `provenance=published`; wire the voice/profile-learning surface to published-only | **← this** | +| SB-S2 — Evolution loop | sleep-time consolidation: journal+published+tributary deltas → profile **diff** (evidence/provenance/timestamps); threshold promotion; contradiction reconciliation; freshness nudge. **Owns session-start wiring of init.** | later | +| SB-S3 — Cross-silo graph + ops centre | thread the id through tributaries (post↔specific↔trend↔analytics); flesh out `operations.md`; retire dead `content-history.md` | later | +| SB-S4 *(optional)* | EU/EEA DMA portability as auto-tributary into `ingest/inbox/` | later | + +## 5. SB-S1 in detail (the build target) + +### 5.1 Deliverables + +1. **A published-record data layer (TS, `scripts/brain/`).** A `PublishedRecord` shape and pure parse/serialize functions for a published post file. Reuse the SB-S0 idioms: `mintEntityId` for the canonical id, the constrained-header line-grammar idea (NO YAML dep), `normalizeProvenance` for the provenance token. Fields (proposed, open to the plan): `id` (sha256[:12] of normalized post text) · `provenance` (always `published`) · `published_date` (YYYY-MM-DD) · `captured_at` (YYYY-MM-DD) · `source` (`manual` | future connector) · the **verbatim post body**. Pure functions, deterministic, unit-tested. + +2. **A `brain ingest` CLI** extending `scripts/brain/src/cli.ts` (same dispatch idiom). Proposed subcommands (final shape open to the plan): + - `brain ingest ` — read a file containing one published post, mint id, write `ingest/published/.md` with `provenance=published`. **Idempotent:** same text → same id → no duplicate (compare-then-skip, matching `initBrain`). + - `brain ingest --scan-inbox` — process every file in `ingest/inbox/` into `ingest/published/`, then mark/clear the inbox source (drop-zone → processed, per the architecture's `inbox`→`published` model). + - `brain published list` — inspect the gold corpus (id · `provenance` · date · first line). Surfacing `provenance` per record makes the invariant *observable* — the operator can eyeball that nothing `ai-draft` leaked into `published/`. + +3. **Wire the voice/profile-learning surface to published-only (the invariant made enforceable).** Edit `agents/voice-trainer.md`'s "Gather" step. The pinned relationship to the existing corpus (resolves brief-review M-D3): + - **ADD** `ingest/published/` (`provenance=published`) as the **primary gold source**, ranked above the legacy `voice-samples/` corpus. + - **KEEP** `voice-samples/` as a human-curated tributary — **not reshaped, not redirected, not abandoned** (§7 honoured). By its existing contract its files are the user's own real writing (human-supplied via `/setup`/`/onboarding`); they are not `ai-draft`. voice-trainer keeps reading it but treats `ingest/published/` as the higher-trust signal. + - **FORBID** learning voice/profile patterns from any content marked `provenance=ai-draft` — the explicit negative rule that generalises `voice-scrubber`'s *"gold standard = approved editions, never the draft corpus"* precedent to the system invariant. + - **FENCE the auto-append trap:** the `voice-samples/` template hints at a (not-yet-built) Stop-hook that auto-appends posts after content sessions. S1 states explicitly that any such future auto-capture MUST tag provenance and admit `published` only — never blind-append generated drafts into a learning corpus. (Forward-looking guard, no S1 code; names the exact model-collapse trap so a later slice cannot reintroduce it.) + - **voice-trainer stays Sonnet** (no model/frontmatter change → no structure-lint model-consistency break). + +4. **Manual-import contract doc** — `docs/second-brain/ingest-manual-import.md`: the published-record file format, the CLI usage, and the published-only learning rule. (Design doc → `docs/`, NOT `references/`, so the reference-doc count guard does not fire.) **Net-new (verified):** SB-S0 shipped the `ingest/{inbox,published}` *dirs* only; no contract doc exists yet — the architecture's "SB-S0 … manual-import contract" wording referred to the dir shape, not a doc. + +5. **Tests (TDD, failing-first)** — published-record parse/serialize round-trip (incl. a body with `]`/`|`/quotes/newlines), id determinism + dedup, CLI idempotency on re-ingest and `--scan-inbox`, source-absent / empty-inbox no-op. **Plus the SC5 published-only structure-lint** (a new gate check; see SC5). Gate stays green. + +6. **Gate floor edits (named — Edit, not Write, on existing files → pathguard-safe):** bump `BRAIN_TESTS_FLOOR` in `scripts/test-runner.sh` to the new brain-suite count, and bump the assertion-count floor (`ASSERT_BASELINE_FLOOR` / the `TOTAL_CHECKS` anti-erosion floor) by the number of new `pass()` lines added (the BRAIN suite line is unchanged-count; the new SC5 structure-lint check adds one). Exact line numbers confirmed in the plan against the live `scripts/test-runner.sh`. + +7. **Docs** — flip the `architecture.md` SB-S1 row to "landed"; this brief + the SB-S1 plan persisted under `docs/second-brain/`. + +### 5.2 The S1/S2 boundary (the decision the review must pin) + +- **SB-S1 ships:** the gold-signal **data layer** (records + CLI), and wires the **existing on-demand learning surface** (`voice-trainer`, invoked when the user runs a voice command) to the published-only rule. This is a data module + an agent-doc edit — no scheduler, no automatic profile mutation. +- **SB-S2 owns:** the **automated** sleep-time consolidation loop (session-start cadence) that reads `published` + journal + tributary deltas and proposes `profile.md` **diffs** with evidence/provenance/timestamps + threshold promotion. **No profile.md is auto-mutated in S1.** + +This boundary keeps S1 a clean, testable data+invariant slice and leaves all *automatic profile evolution* to S2. It is the #1 thing for the operator to confirm. + +## 6. Success criteria (SB-S1 — testable) + +- **SC1 (ingest, runtime):** with `LINKEDIN_STUDIO_DATA` set to a temp dir, `brain ingest ` creates `ingest/published/.md` with `provenance=published` and the verbatim body, at the runtime data-path via the brain package's `dataRoot`. **`ingest` creates `ingest/published/` on demand** (mkdir-recursive) — it does NOT hard-depend on a prior `brain init`, so ingest-before-init works. Verified by a test asserting file existence + content with no prior init. +- **SC2 (record round-trip):** a `PublishedRecord` serializes and parses back to an identical record, including a body containing `]`, `|`, quotes, and embedded newlines. Verified by a parse/serialize identity unit test. +- **SC3 (id determinism + dedup):** the same post text mints the same id (and re-ingesting it is a no-op — no duplicate file); two different posts mint different ids. Verified by id + idempotency unit tests. +- **SC4 (scan-inbox):** files dropped in `ingest/inbox/` are processed into `ingest/published/` by `--scan-inbox`; an empty inbox is a clean no-op; re-running does not re-create already-published records. Verified by a temp-dir test. +- **SC5 (published-only invariant, gate-checkable):** a **new structure-lint check** in `scripts/test-runner.sh` (matching the Section 8–17 honesty-lint idiom: positive + negative assertion + a non-vacuity self-test) asserts that `agents/voice-trainer.md` (a) names `ingest/published/` in its gather/source instructions AND (b) carries the explicit negative string forbidding learning from `provenance=ai-draft`. The self-test proves the check is non-vacuous (it would FAIL on a fixture missing either string). The honest runtime limit (an agent instruction can still be ignored at runtime) is stated in the doc — but the *presence* of the wiring is gate-enforced, not checklist-trusted. (Resolves brief-review M-SC5: no "OR checklist" escape hatch.) +- **SC6 (no regression / no new seam):** `scripts/test-runner.sh` green; floors hold (trends ≥24, specifics ≥28, contract ≥33, **brain ≥ its new floor**); hook tests pass; SB-S1 adds **no new seam function** so `data-root.test.mjs` twin-sync is untouched. + +## 7. Non-goals / scope fence (SB-S1 does NOT) + +- **No consolidation loop / no automatic profile mutation** (SB-S2). `brain ingest` writes only under `ingest/`, never to `brain/profile.md`. +- **No session-start wiring** of init *or* ingest (SB-S2 owns scaffold wiring; ingest stays a manual CLI in S1). +- **No cross-silo id threading** — ingest mints its own record id but does NOT weave it into trends/specifics/analytics/state (SB-S3). +- **No reconciliation of the triple-recorded post** and **no retirement of `content-history.md`** (SB-S3). +- **No connector / no LinkedIn API** (SB-S4). `source` allows a future connector value but no connector code ships. +- **No tributary-schema edits** — voice-samples, specifics, trends, analytics stores keep their current shape. (`voice-trainer.md` is edited; the voice-samples *store format* is not.) +- **No new seam function**, no state-file / two-roots reconciliation, no GUI/cockpit. + +## 8. Constraints / risks + +- **TDD iron law:** no production code without a failing test first. +- **Code placement:** all new code is TS under `scripts/brain/` — **no new `.mjs` under `hooks/scripts/`**, so the `pre-write-pathguard.mjs` Category-6 guard does not fire (it matches `…/hooks/scripts/*.mjs` on Write only; SB-S1 writes none). `agents/voice-trainer.md` is an **existing** file → Edit, not Write → unaffected regardless. +- **Public repo (`open/` Forgejo):** confirm before push; push only inside the window; STATE.md stays gitignored. +- **Structure lint:** SB-S1 adds no `agents/`, `commands/`, `references/`, or `skills/` file (ingest is a CLI subcommand in an existing package; voice-trainer is edited; the contract doc lives in `docs/`). So no count guard fires absent a version bump. **Version bump is an open question** (§9): the agent behavioural change may warrant 0.5.0 → 0.5.1. +- **Node v25 test quirk:** `node --test ` fails — use `node --import tsx --test tests/*.test.ts`; run TS tools from their own dir; `npm install` in `scripts/brain` before the gate or the suite warn-skips. +- **Risk — store-format ambiguity:** the published-record file format is the one data-shaping decision in S1 (mirrors SB-S0's fold risk). Mitigate by pinning the grammar + a round-trip test (SC2) before the CLI wires it. +- **Risk — invariant enforceability:** "wire voice-trainer to published-only" is an agent-doc instruction, not executable code; an instruction can be ignored at runtime. Mitigate by making SC5 a *gate-checkable* lint where feasible (grep for the rule's presence), and by stating the limit honestly in the doc. +- **Risk — S1/S2 scope creep:** the temptation is to start consuming `published/` into `profile.md` here. Fence held by §7 + SC (no test asserts profile mutation in S1). + +## 9. Open questions for the operator (settle at "go") + +1. **S1/S2 boundary (§5.2):** confirm S1 = data layer + voice-trainer invariant wiring, S2 = the automatic profile-diff loop. (Recommended.) +2. **Profile vs voice in S1:** confirm that *profile* learning is invariant-only in S1 (provenance already on facts; no profile mutation) and that the only *active* consumer wired is `voice-trainer`. (Recommended.) +3. **Store format (~decided, confirm):** published records as **file-per-post constrained-header markdown** (the `brain/profile.md` line-grammar idiom, no YAML) — matching the architecture's `inbox`→`published` per-file drop-zone. This deliberately does NOT reuse the specifics-bank single-JSON-index idiom (that store is a topic-tagged bank, not a per-file drop-zone). Confirm. +4. **Version bump:** take 0.5.0 → 0.5.1, or no bump? SB-S0 set a no-bump precedent for count-neutral slices, but SB-S1 differs: it changes a *shipped agent's learning behaviour* (`voice-trainer`), a user-visible capability change — a stronger case for 0.5.1 than SB-S0 had. (Lean 0.5.1.) + +## 10. Definition of done (SB-S1) + +SC1–SC6 pass; deliverables 1–7 exist; non-goals untouched; the published-only invariant is wired into `voice-trainer` + documented + **gate-enforced (SC5 structure-lint)**; the BRAIN + assertion floors raised; committed (push per window, confirm first); `architecture.md` SB-S1 row → "landed"; version/CLAUDE.md/README touched only if a bump is taken. diff --git a/docs/second-brain/plan-sb-s1.md b/docs/second-brain/plan-sb-s1.md new file mode 100644 index 0000000..1617370 --- /dev/null +++ b/docs/second-brain/plan-sb-s1.md @@ -0,0 +1,160 @@ +# Implementation Plan — SB-S1 (Ingest + gold signal) + +> **Status:** review-hardened (`plan-critic` REVISE 63→folded; `scope-guardian` GAP→folded). Ready for operator approval / "go". +> **Brief:** `brief-sb-s1.md` (review-hardened). **Arc:** `architecture.md`. **Predecessor:** SB-S0 (`scripts/brain/`). +> **Review delta (2026-06-23):** B1 separator pinned to one exact string + edge battery; B2 id → verbatim-body hash (no `normalizeContent`) + collision-safe write; floors pinned EMPIRICALLY (not hand-arithmetic); SC5 grep → exact `grep -F` literals + `AI-generated` negative probe; CLI = net-new flag infra + `init` regression; per-step On-failure/Checkpoint + 2 circuit-breakers; D3 auto-append FENCE restored; voice-trainer anchors fixed (:136/:144); malformed-file try/catch + scanInbox `*.md`-only filter. +> **Scope:** SB-S1 only — the published-record data layer + `brain ingest` CLI, and wiring `voice-trainer` to the published-only invariant (gate-enforced). **No consolidation loop, no automatic profile mutation, no session-start wiring, no cross-silo threading, no connector** (brief §7). + +--- + +## 1. What SB-S1 delivers (recap) + +A new TS module `scripts/brain/src/ingest.ts` (matching the SB-S0/specifics-bank idiom) defining a `PublishedRecord` shape with a pinned file-per-post **constrained-header grammar** (no YAML) that round-trips, a content-hash id over the **verbatim body** (`sha256(body)[:12]` — byte-identity dedup, no normalization), and idempotent, collision-safe IO (`writePublished`, `scanInbox`, `listPublished`) under the brain package's `dataRoot`. The `brain` CLI grows `ingest` + `published list` subcommands. `agents/voice-trainer.md` is wired to add `ingest/published/` as the primary gold source and forbid learning from `provenance=ai-draft`, and that wiring is **gate-enforced** by a new structure-lint (Section 16c). All unit-tested, TDD (failing test first). Gate stays green; BRAIN + assertion floors rise. + +## 2. Package additions (`scripts/brain/`) — no new package, extend the existing one + +``` +scripts/brain/ + src/ + ingest.ts # NEW — PublishedRecord type + parse/serialize (pure) + writePublished/scanInbox/listPublished (IO) + id.ts # EDIT — add normalizeContent() + mintContentId() (the specifics-bank content-hash idiom) + cli.ts # EDIT — add `ingest` + `published` dispatch (parseFlags idiom from specifics-bank/src/cli.ts) + (types.ts, profile.ts, scaffold.ts, dataRoot.ts unchanged) + tests/ + ingest.test.ts # NEW — record round-trip + id determinism/dedup (pure) + publish.test.ts # NEW — temp-dir IO: write/idempotency/scan-inbox/create-on-demand + fixtures/ + sample-inbox-post.md # NEW — a synthetic published post for scan-inbox + round-trip tests +``` + +No `package.json`/`tsconfig.json` change (devDeps already tsx+typescript; no new dep — the grammar is regex, no YAML). The test command (`node --import tsx --test tests/*.test.ts`) already globs `tests/*.test.ts`, so the two new suites are picked up automatically. + +## 3. Module designs + +### 3.1 `id.ts` addition — `mintContentId` on the VERBATIM body (no normalization — fixes plan-critic B2) +**Verify-first (step 1):** mirror the `specificId` hash idiom at `scripts/specifics-bank/src/bank.ts:57` (the function is named `specificId`, NOT `mintContentId` — that name is net-new here; `normalizeContent` lives at `bank.ts:52`). **Decision: do NOT reuse `normalizeContent` for a post body.** +```ts +// mintContentId: sha256 of the VERBATIM body, first 12 hex. The id IS byte-identity. +export function mintContentId(text: string): string { + return createHash("sha256").update(text).digest("hex").slice(0, 12); +} +``` +- **Why verbatim, not `normalizeContent`:** `normalizeContent` (bank.ts:52) does `trim().toLowerCase().replace(/\s+/g," ")` — correct for *short* specifics dedupe, but for a full post body it collapses ALL whitespace + case, so two **structurally different** published posts (same words, different line breaks/paragraphing/case) would mint the **same id** and the second would be silently skipped on write (data loss of the exact gold signal SB-S1 exists to capture, plan-critic B2). Hashing the verbatim body makes the id true byte-identity: only an identical re-ingest collides; any real difference mints a distinct id. +- `mintContentId` is content-keyed — distinct from `mintEntityId({kind,key})` which slugifies a *label* (a body must never be slugified). Pure, deterministic. SC3. +- **Defence-in-depth (write-side, §3.3):** even with byte-identity hashing, `writePublished` compares bodies on any id-collision and never silently drops a differing body — so no hash strategy can cause data loss. + +### 3.2 `ingest.ts` — `PublishedRecord` + constrained-header grammar (no YAML) +```ts +export interface PublishedRecord { + id: string; // mintContentId(body) — 12 hex, the dedupe key + filename stem + provenance: 'published'; // S1 only ever writes `published`; the type pins it + published_date: string; // YYYY-MM-DD (operator-supplied or defaults to captured_at) + captured_at: string; // YYYY-MM-DD (ingest run date; supplied by caller — pure fn) + source: string; // 'manual' (default) | future connector token + body: string; // the VERBATIM post text (may contain ] | quotes newlines ---) +} +``` +**File grammar** (`ingest/published/.md`) — the EXACT byte layout, used identically by serialize and parse (fixes plan-critic B1): +``` +id: <12hex> +provenance: published +published_date: YYYY-MM-DD +captured_at: YYYY-MM-DD +source: manual +--- + +``` +- **Pinned separator (one string, both directions):** the five header lines, then a line that is exactly `---`, then the verbatim body. Concretely: `serialize` = `headerLines.join("\n") + "\n---\n" + body`. There is **NO** blank-line padding around the `---` and **NO** `# Published Record` title (both removed — they were the B1 contradiction). The header block is fixed at exactly 5 lines. +- **`parsePublishedRecord(text)`** → split on the FIRST occurrence of `\n---\n`. Left = header (parse the 5 scalars by the in-tree scalar-regex idiom: `^id:\s*([0-9a-f]{12})$/m`, `^published_date:\s*(\d{4}-\d{2}-\d{2})$/m`, etc.). Right = `body`, **verbatim, byte-exact** (a body that itself contains a `\n---\n` is preserved — we split on the FIRST sentinel only, via `indexOf`, never a global split). **`provenance`** is parsed and asserted to equal `published` (a `published/` record with any other provenance is a corruption signal → throw/skip, never silently accepted — fixes plan-critic minor 11; the type pins `'published'`). +- **`parse(serialize(rec)) === rec`** over all six fields incl. the body (SC2). **Edge cases the round-trip test MUST cover** (plan-critic B1): empty body (`""`), body = `"\n"`, body with trailing newlines, body starting with `---`, body containing `\n---\n` mid-text, and a body whose first line is header-shaped (e.g. `id: 0123456789ab`). Because the header is a fixed 5-line block terminated by the first `\n---\n`, a header-shaped body line cannot leak into the header (it sits after the sentinel). The only constraint on the body: none needed — it is captured byte-exact after the first sentinel. + +### 3.3 `ingest.ts` — IO (impure; caller supplies the run date, like `scaffold.ts`) +- **`writePublished(rec): {written: boolean; path: string; collision?: boolean}`** — `dataRoot('ingest/published')`, mkdir-recursive (**create-on-demand — no hard dependency on `brain init`**, SC1), path = `.md`. **Collision-safe compare-then-skip (fixes plan-critic B2 defence-in-depth):** if `.md` exists, READ it, parse its body: (a) body byte-identical to `rec.body` → true duplicate → `{written:false}` (idempotent, no clobber — the `initBrain` discipline); (b) body DIFFERS (astronomically rare 48-bit collision, or a truncation artifact) → write to a disambiguated path `-2.md` (next free suffix) and return `{written:true, collision:true}` — **never silently drop a differing body.** +- **`ingestText({body, source?, published_date?, captured_at})`** — `mintContentId(body)`, build the record (`provenance:'published'`; `published_date` defaults to `captured_at` when absent), `writePublished`. Returns the record + written flag. +- **`scanInbox({captured_at})`** — read **top-level `*.md` files only**, skipping dotfiles (`.DS_Store` etc. — macOS), no recursion (fixes plan-critic minor 9), in `dataRoot('ingest/inbox')` (empty/absent dir → clean no-op, SC4); for each, `ingestText({body: fileContent, source:'manual', captured_at})`. **Non-destructive in S1:** inbox files are LEFT in place (idempotent via the published dedup — a re-scan re-skips); auto-move/delete of processed inbox files is deferred (noted §8). Returns `{processed[], skipped[]}`. +- **`listPublished()`** — read `ingest/published/*.md`; **parse each inside a try/catch — a malformed/hand-dropped file is skipped+counted, never crashes the command** (`parsePublishedRecord` can throw via the provenance assertion / `normalizeProvenance`, id.ts:45; fixes plan-critic minor 8). Return `{records: [{id, provenance, published_date, firstLine}], skipped: number}` sorted by date. Surfaces `provenance` so the operator can eyeball that nothing `ai-draft` leaked (brief §5.1 D2). + +### 3.4 `cli.ts` — add NET-NEW flag-routing infra (not just "extend dispatch" — fixes plan-critic major 6) +The brain `cli.ts` currently has **no** `parseFlags`, **no** `--json`, **no** `today()` — it dispatches purely on positional `argv[2]` (just `init`). So this step ADDS that infrastructure (copy `parseFlags` from `specifics-bank/src/cli.ts:41`; a local `today()` — `ingest.ts` itself stays pure with a caller-supplied date, so the CLI's `today()` is the only clock and does not shadow `scaffold.ts:28`'s private one). The **existing `init` positional branch is preserved verbatim**, coexisting with the new flag-driven branches: +- **`brain ingest --file [--source ] [--date ]`** — read the file, `ingestText`, print `Wrote ingest/published/.md` / `Duplicate — already published: ` / `Collision → wrote -2.md`. +- **`brain ingest --scan-inbox [--source ]`** — `scanInbox`, print the processed/skipped split. +- **`brain published list [--json]`** — `listPublished`; human form prints `id · provenance · date · first-line`; `--json` emits `{records:[{id,provenance,published_date,firstLine}], skipped}` (the `listPublished` return verbatim — pinned shape, fixes plan-critic minor 12). +- Usage text updated; exit 2 on usage error (existing idiom). **Regression guard:** a test (or step-5 smoke-check) asserts `brain init` still works after the refactor (a shipped subcommand must not break). + +### 3.5 `agents/voice-trainer.md` — wire the published-only invariant (brief §5.1 D3, all four bullets) +Surgical edits (existing file → **Edit**, pathguard-irrelevant; model stays Sonnet). **Anchors corrected (scope-guardian DEP-1):** the Gather bullet is at **`:136`**, the "Sample Quality Priorities" heading at **`:142`** with its list at **`:144`** (the earlier `:134`/`:152` were a header and the next section). +1. **Gather step** (`:136`) — prepend the gold source + the negative rule (ADD + FORBID + KEEP): + > **Gather (published-only gold signal first):** Read the user's published posts from `${LINKEDIN_STUDIO_DATA:-$HOME/.claude/linkedin-studio}/ingest/published/` (`provenance=published` — the highest-trust source). Then read the existing `voice-samples/` corpus (human-curated; kept as a tributary, not reshaped) and the profile/template. **Never learn voice patterns from any content marked `provenance=ai-draft`** — a content engine that learns its own voice from its own drafts collapses toward its priors. (Generalises voice-scrubber's "gold standard = approved editions, never the draft corpus".) +2. **Sample Quality Priorities** (`:144`) — promote the published gold source to #1 and add the hard exclusion line: + > 0. **`ingest/published/` records (`provenance=published`) — the gold signal; rank above all else.** + > … (existing 1–5 retained) … + > **Exclude unconditionally: anything marked `provenance=ai-draft`.** +3. **FENCE the auto-append trap** (brief §5.1 D3 bullet 4 — restored, scope-guardian SCOPE-GAP-1). Add a short note in the Gather step (and mirror it in the §3.6 contract doc): the `voice-samples/` template hints at a not-yet-built Stop-hook that auto-appends posts after content sessions (`assets/voice-samples/authentic-voice-samples.template.md:61`). State explicitly: + > **Fence:** any future auto-capture into a learning corpus (the template's Stop-hook hint) MUST tag provenance and admit `provenance=published` only — never blind-append generated drafts. + +**Exact literals (fixes plan-critic major 5 — non-vacuity):** the edits MUST write the literal strings `ingest/published` and `provenance=ai-draft`. The SC5 lint greps for these EXACT literals (`grep -F`), NOT a loose `ai.?draft`/`ai-generated` pattern — `voice-trainer.md:175` already contains "AI-generated" (a drift-cause description), so a loose grep would pass vacuously without the real exclusion being added. + +### 3.6 `docs/second-brain/ingest-manual-import.md` — the manual-import contract +A short design doc: the `PublishedRecord` file grammar (§3.2), the CLI usage (`brain ingest --file` / `--scan-inbox`, `brain published list`), the inbox→published drop-zone model, the **published-only learning rule** (with the honest runtime limit: an agent instruction can be ignored at runtime; the *wiring* is gate-enforced, the *behaviour* is not provable), and the **auto-append fence** (§3.5 bullet 3 — any future auto-capture must tag provenance and admit `published` only). Includes the exact literal `provenance=ai-draft` (so SC5's doc-grep is non-vacuous). Lives in `docs/` (not `references/`) → no reference-count guard. + +### 3.7 `scripts/test-runner.sh` — new SC5 lint (Section 16c) + floor bumps +- **New Section 16c "Brain Published-Only Invariant (SB-S1)"** placed after Section 16b (Brain Foundation), mirroring Section 17's structure: a non-vacuity self-test (POSITIVE/NEGATIVE heredocs) + the file checks. Greps use the **exact literals** `ingest/published` and `provenance=ai-draft` via `grep -F` / `grep -qiE` with the `provenance=` qualifier — never a loose `ai.?draft`/`ai-generated` pattern (plan-critic major 5). Three `pass()` lines (none in a loop): + 1. **self-test** — POSITIVE probe contains BOTH literals → must pass; NEGATIVE probes: one missing `ingest/published`, one missing `provenance=ai-draft`, **and one containing `AI-generated` but NOT `provenance=ai-draft`** (proves the check doesn't pass on voice-trainer's pre-existing drift prose) → each must fail. Non-vacuous like Sections 8/13/17. + 2. **`voice-trainer.md`** contains `ingest/published` AND `provenance=ai-draft`. + 3. **`docs/second-brain/ingest-manual-import.md`** contains `provenance=ai-draft` (the rule is documented). +- **`BRAIN_TESTS_FLOOR=34` (`:714`) → the actual post-implementation count.** Pinned at step 8 by reading the brain suite's real `tests N` AFTER the new suites land (TDD determines it; do NOT guess). **Step 8 must first assert the brain suite actually RAN (`BR_TESTS` non-empty — i.e. tsx installed) before pinning**, else a warn-skip would let the gate go green while SB-S1's own tests never ran (plan-critic major 4). +- **`ASSERT_BASELINE_FLOOR=75` (`:798`) → the actual printed `TOTAL_CHECKS`.** The floor counts **runtime** `PASS+FAIL` increments, not static `pass()` call-sites (the file has 72 static sites but a 75 floor because some checks loop). The three new Section 16c checks are non-looping, so the delta is +3 → 78 — **but pin it empirically: run the gate at step 8, read the printed `TOTAL_CHECKS`, set the floor to that exact number** rather than hand-arithmetic (plan-critic major 3). Update the inline history comment at `:796`. + +## 4. Files created / edited + +**Created:** `scripts/brain/src/ingest.ts` · `scripts/brain/tests/{ingest,publish}.test.ts` · `scripts/brain/fixtures/sample-inbox-post.md` · `docs/second-brain/ingest-manual-import.md`. + +**Edited:** `scripts/brain/src/id.ts` (+normalizeContent/+mintContentId) · `scripts/brain/src/cli.ts` (+ingest/+published) · `agents/voice-trainer.md` (gather + priorities) · `scripts/test-runner.sh` (Section 16c + two floor bumps) · `docs/second-brain/architecture.md` (flip SB-S1 row → landed). + +**Conditional (only if Q4 = bump to 0.5.1):** `CLAUDE.md` header + README badge + CHANGELOG + the marketplace catalog `ref` (via `catalog/scripts/release-plugin.mjs` — the polyrepo release path; tag + catalog bump atomic). Decided at "go". + +## 5. TDD test plan (failing-first) → success criteria + +| Test (`tests/*.test.ts`) | Asserts | SC | +|---|---|---| +| `ingest.test.ts` | `parsePublishedRecord(serializePublishedRecord(rec)) === rec` over the **B1 edge-case battery**: empty body, body=`"\n"`, trailing newlines, body starting with `---`, body containing `\n---\n` mid-text, header-shaped first body line (`id: 0123456789ab`), and a body with `]`/`|`/quotes; `mintContentId` deterministic (same body→same id; whitespace/case difference→DIFFERENT id, proving no normalize-collision); a `published/` record with non-`published` provenance throws (corruption assertion) | SC2, SC3 | +| `publish.test.ts` | `LINKEDIN_STUDIO_DATA`=tmp, NO prior init → `writePublished` creates `ingest/published/.md` with `provenance=published` + verbatim body (create-on-demand); re-ingest identical body = `{written:false}`, no dup; **id-collision with a DIFFERING body → disambiguated `-2.md`, no data loss**; `scanInbox` processes top-level `*.md` only (skips a `.DS_Store` fixture), → published; empty/absent inbox = no-op; re-scan re-skips; `listPublished` skips a malformed dropped file without throwing; **`brain init` still works after the CLI refactor** (regression) | SC1, SC3, SC4 | +| (gate) `scripts/test-runner.sh` Section 16c | self-test non-vacuous (incl. the `AI-generated`-but-not-`ai-draft` negative probe); `voice-trainer.md` + contract doc carry the exact literals | SC5 | +| (gate) `scripts/test-runner.sh` | green; trends ≥24, specifics ≥28, contract ≥33, **brain ≥ new floor (brain suite confirmed RAN, not skipped)**; hooks pass; assertion floor = printed `TOTAL_CHECKS` | SC6 | +| (n/a) | no new seam fn; `data-root.test.mjs` twin-sync untouched | SC6 | + +Each test written **before** its module (iron law): red → implement → green. + +## 6. Step sequence + +**Discipline (plan-critic major 7):** each TDD step is a **Checkpoint** — commit after its cycle goes green (small, revertible commits). Each step carries an **On-failure** clause; two hard **circuit-breakers**: if step 3 (round-trip) cannot be made green after pinning the separator, HALT (the grammar is wrong — do not proceed); if step 8 (full gate) is not green, HALT (do not commit a red gate). + +1. **Verify-first:** read `scripts/specifics-bank/src/bank.ts:52,57` → confirm the `specificId` hash idiom (mirror it as `mintContentId` on the VERBATIM body; do NOT reuse `normalizeContent`). Confirm `scripts/brain/node_modules/.bin/tsx` exists (else `npm install` in `scripts/brain` — the gate warn-skips a depless suite; "green" must not mean green-because-skipped). *On-failure: if tsx won't install, STOP and report — the slice cannot be gate-verified.* +2. TDD `id.ts` `mintContentId` (`ingest.test.ts` id cases: determinism + whitespace/case → different id). → SC3. *Checkpoint.* +3. TDD `ingest.ts` grammar parse/serialize over the B1 edge-case battery (`ingest.test.ts` round-trip). → SC2. *On-failure: CIRCUIT-BREAKER — if identity can't hold, the separator is wrong; halt and re-pin §3.2 before any further step.* *Checkpoint.* +4. Write `fixtures/sample-inbox-post.md` (+ a `.DS_Store` + a malformed file in test temp dirs); TDD `ingest.ts` IO — `writePublished` (incl. collision→disambiguate), `ingestText`, `scanInbox` (filter), `listPublished` (try/catch) (`publish.test.ts`). → SC1, SC4. *Checkpoint.* +5. Add the CLI flag-routing infra + `ingest`/`published` branches (preserve `init`); regression-test `brain init`; smoke-run against a temp `LINKEDIN_STUDIO_DATA` (ingest a file, scan an inbox, list). *On-failure: if `brain init` regresses, revert the CLI refactor and isolate.* *Checkpoint.* +6. Edit `agents/voice-trainer.md` (`:136` gather + `:144` priorities + the `provenance=ai-draft` exclusion + the auto-append fence — exact literals). *Checkpoint.* +7. Write `docs/second-brain/ingest-manual-import.md` (incl. the `provenance=ai-draft` literal). *Checkpoint.* +8. Add `scripts/test-runner.sh` Section 16c (self-test + voice-trainer + doc checks, exact-literal greps); run the gate, **confirm the brain suite RAN (`BR_TESTS` non-empty)**, then pin `BRAIN_TESTS_FLOOR` to the real `tests N` and `ASSERT_BASELINE_FLOOR` to the printed `TOTAL_CHECKS`; re-run the FULL gate green. → SC5, SC6. *On-failure: CIRCUIT-BREAKER — a red gate is never committed.* *Checkpoint.* +9. Flip `architecture.md` SB-S1 row → landed. **Exact edit (scope-guardian/plan-critic minor 10):** the row at `architecture.md:78` begins `| **SB-S1 — Ingest + gold signal** |` — change its trailing status cell to the "landed" marker matching SB-S0's row style (verify the exact current cell text at edit time). Persist brief + plan. **Commit; confirm before push (PUBLIC `open/` Forgejo), only inside the window; STATE.md stays gitignored.** Conditional version bump per Q4. + +## 7. Scope fence (echo — SB-S1 does NOT) + +No consolidation loop · **no write to `brain/profile.md`** (ingest writes only under `ingest/`) · no session-start wiring · no id threading into tributaries · no triple-post reconciliation · no `content-history.md` retirement · no tributary-schema edits (voice-samples store untouched; only `voice-trainer.md` instructions edited) · no LinkedIn API/connector · no new seam function · no state-file/two-roots reconciliation · no GUI. + +## 8. Known limits / deferred (honest) + +- **Invariant is wired, not behaviourally proven.** SC5 gate-enforces that `voice-trainer` *carries* the published-only wiring + exclusion; it cannot prove the agent *obeys* it at runtime (an instruction can be ignored). Stated in the contract doc. The strongest feasible enforcement short of running the agent. +- **scan-inbox is non-destructive in S1** — processed inbox files are left in place (idempotent via published dedup). Auto-move/delete of processed inbox files (the full drop-zone lifecycle) is deferred; the operator clears inbox manually for now. +- **`published_date` defaults to `captured_at`** when the operator doesn't pass `--date` (and for scan-inbox, which has no per-file date). A real published date is operator-supplied; SB-S2's loop can refine from analytics. +- **`voice-samples/` stays un-provenanced.** S1 keeps it as a human-curated tributary and does not retro-tag it (that would be a tributary-schema change, §7). The published-only invariant bites on the NEW gold source + the ai-draft exclusion; voice-samples is trusted as human by its existing contract. +- **No cross-silo id.** A published post ingested here mints a content-hash id but is NOT yet linked to its analytics/state/specifics records (SB-S3). + +## 9. Risks + +- **Grammar identity edge (B1):** a body containing `\n---\n` could confuse the header/body split — mitigated by splitting on the **first** sentinel only (`indexOf`, never global) + the fixed 5-line header; the B1 edge-case battery (§5) is the proof. Circuit-breaker at step 3. +- **Silent data loss (B2) — CLOSED:** id is now `sha256(verbatim body)[:12]` (no `normalizeContent`), so structurally-different posts never collide; and `writePublished` compares bodies on any id-collision and disambiguates rather than skipping — so no hash strategy can drop a differing gold record. Tested in `publish.test.ts`. +- **Gate "green" honest only with deps installed** — `npm install scripts/brain` (step 1) AND step 8 asserts `BR_TESTS` non-empty before pinning the floor; otherwise a warn-skip lets the gate go green while SB-S1's own tests never ran. +- **Floor-bump arithmetic (corrected):** `ASSERT_BASELINE_FLOOR` counts **runtime** `PASS+FAIL`, not static `pass()` sites — so do NOT hand-compute 75+3; read the printed `TOTAL_CHECKS` at step 8 and set the floor to it. `BRAIN_TESTS_FLOOR` = the actual brain `tests N` after the new suites land. Both pinned empirically at step 8, never guessed. +- **Scope creep into S2:** the temptation is to consume `published/` into `profile.md`. Fenced by §7 + the test plan (no test asserts profile mutation; `writePublished` targets only `ingest/`).