# Task Brief — SB-S1 (Ingest + gold signal)

> **Status:** DRAFT for adversarial brief-review (light-Voyage step 2). Not yet operator-approved.
> **Arc:** `architecture.md` (approved 2026-06-23). **Predecessor:** SB-S0 landed (`8c92719`) — `scripts/brain/` (types/id/profile/scaffold/cli, 34 tests, gate-wired). **This brief:** the second slice in detail; S2–S4 stay one-line in the arc.
> **Inputs:** `architecture.md` §"evolution loop" steps 1–2, §"invariants" (provenance-weighted learning); SB-S0 code under `scripts/brain/`; the silo inventory (`docs/second-brain/research/silo-inventory.md`).
> **Review delta (2026-06-23):** folded the brief-review (PROCEED_WITH_RISKS). Resolver standardised on `dataRoot` (M-resolver); D3 voice-samples↔ingest relationship pinned (M-D3); SC5 hardened to a gate-checkable lint (M-SC5); test-runner floor edits named as deliverables (m-floor); input path + D4 net-new note + Q3 sharpened (minors).

---

## 1. Objective

Capture the **gold signal** — the user's actual *published* posts — into `ingest/published/` tagged `provenance=published`, and **wire the voice/profile-learning surface to learn from that published signal only**, never from AI-drafted content. This is the one invariant the whole second-brain stands or falls on (the model-collapse guard): a content engine that learns its own voice from what it itself drafted collapses toward its own priors. SB-S1 lays the gold signal **before** the consolidation loop (SB-S2) that will consume it.

## 2. Background (why this slice, why now)

- **The invariant already has a single-agent precedent.** `voice-scrubber` enforces *"gold standard = approved Norwegian editions, never the English post corpus."* SB-S1 generalises that one agent's rule into a system-wide, data-backed invariant: learning consumes `provenance=published` only.
- **The signal is genuinely new, not a re-count of existing silos.** A published post lands today in three non-referencing places — state `## Recent Posts` (hook + topic only), the dead `analytics/content-history.md`, and `analytics/posts/*.json` (metrics, from CSV). **None holds the full post text.** Voice learning needs the verbatim text; `ingest/published/` holding it is additive, not redundant. (Cross-silo *threading* of a shared id is SB-S3, explicitly out of scope here.)
- **The provenance shape is already in place.** SB-S0 pinned `Provenance = human | published | ai-draft` and put a `provenance` field on every `ProfileFact`. SB-S1 is the first slice to actually *produce* `published` records and *gate* a learning consumer on them — it spends the spine S0 built.
- **Manual-first is the contract.** Auto-ingest (EU/EEA DMA portability) is a later additive tributary (SB-S4); SB-S1's ingest is the manual path: the user supplies the published post text, the CLI lands it as a gold record.

## 3. Confirmed decisions (inherited from the approved arc — not for re-litigation)

1. **Boundary:** engine → plugin · user data → data dir · cockpit → Maskinrommet.
2. **Plain-text/JSON + git, no vector/graph DB.** Claude is the retrieval engine.
3. **Provenance-weighted learning** (`published` only, never `ai-draft`) is an invariant, not a toggle.
4. **Tributaries keep their own schema/lifecycle** — SB-S1 does NOT reshape voice-samples, specifics, trends, or analytics stores.
5. **No new seam function** — ingest paths resolve through the brain package's own inlined `dataRoot(sub)` resolver (`scripts/brain/src/dataRoot.ts`, the repo idiom — NOT the hooks-side `getDataRoot()` seam, a different function in a different package). So the twin-sync surface (`data-root.mjs ⇄ storage.ts`) stays untouched.

Review SHOULD challenge: the S1/S2 boundary (§5/§7), the store format (§5.1), feasibility, hidden coupling, testability, scope leak, missing risks. Review SHOULD NOT re-open the five decisions above unless internally contradictory or technically impossible.

## 4. Scope — where SB-S1 sits in the arc

| Slice | One-line scope | This brief |
|-------|----------------|------------|
| SB-S0 — Foundation | scaffold + profile fold + id/provenance spine | ✅ landed |
| **SB-S1 — Ingest + gold signal** | manual import → `ingest/published/` `provenance=published`; wire the voice/profile-learning surface to published-only | **← this** |
| SB-S2 — Evolution loop | sleep-time consolidation: journal+published+tributary deltas → profile **diff** (evidence/provenance/timestamps); threshold promotion; contradiction reconciliation; freshness nudge. **Owns session-start wiring of init.** | later |
| SB-S3 — Cross-silo graph + ops centre | thread the id through tributaries (post↔specific↔trend↔analytics); flesh out `operations.md`; retire dead `content-history.md` | later |
| SB-S4 *(optional)* | EU/EEA DMA portability as auto-tributary into `ingest/inbox/` | later |

## 5. SB-S1 in detail (the build target)

### 5.1 Deliverables

1. **A published-record data layer (TS, `scripts/brain/`).** A `PublishedRecord` shape and pure parse/serialize functions for a published post file. Reuse the SB-S0 idioms: `mintEntityId` for the canonical id, the constrained-header line-grammar idea (NO YAML dep), `normalizeProvenance` for the provenance token. Fields (proposed, open to the plan): `id` (sha256[:12] of normalized post text) · `provenance` (always `published`) · `published_date` (YYYY-MM-DD) · `captured_at` (YYYY-MM-DD) · `source` (`manual` | future connector) · the **verbatim post body**. Pure functions, deterministic, unit-tested.

2. **A `brain ingest` CLI** extending `scripts/brain/src/cli.ts` (same dispatch idiom). Proposed subcommands (final shape open to the plan):
   - `brain ingest <file>` — read a file containing one published post, mint id, write `ingest/published/<id>.md` with `provenance=published`. **Idempotent:** same text → same id → no duplicate (compare-then-skip, matching `initBrain`).
   - `brain ingest --scan-inbox` — process every file in `ingest/inbox/` into `ingest/published/`, then mark/clear the inbox source (drop-zone → processed, per the architecture's `inbox`→`published` model).
   - `brain published list` — inspect the gold corpus (id · `provenance` · date · first line). Surfacing `provenance` per record makes the invariant *observable* — the operator can eyeball that nothing `ai-draft` leaked into `published/`.

3. **Wire the voice/profile-learning surface to published-only (the invariant made enforceable).** Edit `agents/voice-trainer.md`'s "Gather" step. The pinned relationship to the existing corpus (resolves brief-review M-D3):
   - **ADD** `ingest/published/` (`provenance=published`) as the **primary gold source**, ranked above the legacy `voice-samples/` corpus.
   - **KEEP** `voice-samples/` as a human-curated tributary — **not reshaped, not redirected, not abandoned** (§7 honoured). By its existing contract its files are the user's own real writing (human-supplied via `/setup`/`/onboarding`); they are not `ai-draft`. voice-trainer keeps reading it but treats `ingest/published/` as the higher-trust signal.
   - **FORBID** learning voice/profile patterns from any content marked `provenance=ai-draft` — the explicit negative rule that generalises `voice-scrubber`'s *"gold standard = approved editions, never the draft corpus"* precedent to the system invariant.
   - **FENCE the auto-append trap:** the `voice-samples/` template hints at a (not-yet-built) Stop-hook that auto-appends posts after content sessions. S1 states explicitly that any such future auto-capture MUST tag provenance and admit `published` only — never blind-append generated drafts into a learning corpus. (Forward-looking guard, no S1 code; names the exact model-collapse trap so a later slice cannot reintroduce it.)
   - **voice-trainer stays Sonnet** (no model/frontmatter change → no structure-lint model-consistency break).

4. **Manual-import contract doc** — `docs/second-brain/ingest-manual-import.md`: the published-record file format, the CLI usage, and the published-only learning rule. (Design doc → `docs/`, NOT `references/`, so the reference-doc count guard does not fire.) **Net-new (verified):** SB-S0 shipped the `ingest/{inbox,published}` *dirs* only; no contract doc exists yet — the architecture's "SB-S0 … manual-import contract" wording referred to the dir shape, not a doc.

5. **Tests (TDD, failing-first)** — published-record parse/serialize round-trip (incl. a body with `]`/`|`/quotes/newlines), id determinism + dedup, CLI idempotency on re-ingest and `--scan-inbox`, source-absent / empty-inbox no-op. **Plus the SC5 published-only structure-lint** (a new gate check; see SC5). Gate stays green.

6. **Gate floor edits (named — Edit, not Write, on existing files → pathguard-safe):** bump `BRAIN_TESTS_FLOOR` in `scripts/test-runner.sh` to the new brain-suite count, and bump the assertion-count floor (`ASSERT_BASELINE_FLOOR` / the `TOTAL_CHECKS` anti-erosion floor) by the number of new `pass()` lines added (the BRAIN suite line is unchanged-count; the new SC5 structure-lint check adds one). Exact line numbers confirmed in the plan against the live `scripts/test-runner.sh`.

7. **Docs** — flip the `architecture.md` SB-S1 row to "landed"; this brief + the SB-S1 plan persisted under `docs/second-brain/`.

### 5.2 The S1/S2 boundary (the decision the review must pin)

- **SB-S1 ships:** the gold-signal **data layer** (records + CLI), and wires the **existing on-demand learning surface** (`voice-trainer`, invoked when the user runs a voice command) to the published-only rule. This is a data module + an agent-doc edit — no scheduler, no automatic profile mutation.
- **SB-S2 owns:** the **automated** sleep-time consolidation loop (session-start cadence) that reads `published` + journal + tributary deltas and proposes `profile.md` **diffs** with evidence/provenance/timestamps + threshold promotion. **No profile.md is auto-mutated in S1.**

This boundary keeps S1 a clean, testable data+invariant slice and leaves all *automatic profile evolution* to S2. It is the #1 thing for the operator to confirm.

## 6. Success criteria (SB-S1 — testable)

- **SC1 (ingest, runtime):** with `LINKEDIN_STUDIO_DATA` set to a temp dir, `brain ingest <file>` creates `ingest/published/<id>.md` with `provenance=published` and the verbatim body, at the runtime data-path via the brain package's `dataRoot`. **`ingest` creates `ingest/published/` on demand** (mkdir-recursive) — it does NOT hard-depend on a prior `brain init`, so ingest-before-init works. Verified by a test asserting file existence + content with no prior init.
- **SC2 (record round-trip):** a `PublishedRecord` serializes and parses back to an identical record, including a body containing `]`, `|`, quotes, and embedded newlines. Verified by a parse/serialize identity unit test.
- **SC3 (id determinism + dedup):** the same post text mints the same id (and re-ingesting it is a no-op — no duplicate file); two different posts mint different ids. Verified by id + idempotency unit tests.
- **SC4 (scan-inbox):** files dropped in `ingest/inbox/` are processed into `ingest/published/` by `--scan-inbox`; an empty inbox is a clean no-op; re-running does not re-create already-published records. Verified by a temp-dir test.
- **SC5 (published-only invariant, gate-checkable):** a **new structure-lint check** in `scripts/test-runner.sh` (matching the Section 8–17 honesty-lint idiom: positive + negative assertion + a non-vacuity self-test) asserts that `agents/voice-trainer.md` (a) names `ingest/published/` in its gather/source instructions AND (b) carries the explicit negative string forbidding learning from `provenance=ai-draft`. The self-test proves the check is non-vacuous (it would FAIL on a fixture missing either string). The honest runtime limit (an agent instruction can still be ignored at runtime) is stated in the doc — but the *presence* of the wiring is gate-enforced, not checklist-trusted. (Resolves brief-review M-SC5: no "OR checklist" escape hatch.)
- **SC6 (no regression / no new seam):** `scripts/test-runner.sh` green; floors hold (trends ≥24, specifics ≥28, contract ≥33, **brain ≥ its new floor**); hook tests pass; SB-S1 adds **no new seam function** so `data-root.test.mjs` twin-sync is untouched.

## 7. Non-goals / scope fence (SB-S1 does NOT)

- **No consolidation loop / no automatic profile mutation** (SB-S2). `brain ingest` writes only under `ingest/`, never to `brain/profile.md`.
- **No session-start wiring** of init *or* ingest (SB-S2 owns scaffold wiring; ingest stays a manual CLI in S1).
- **No cross-silo id threading** — ingest mints its own record id but does NOT weave it into trends/specifics/analytics/state (SB-S3).
- **No reconciliation of the triple-recorded post** and **no retirement of `content-history.md`** (SB-S3).
- **No connector / no LinkedIn API** (SB-S4). `source` allows a future connector value but no connector code ships.
- **No tributary-schema edits** — voice-samples, specifics, trends, analytics stores keep their current shape. (`voice-trainer.md` is edited; the voice-samples *store format* is not.)
- **No new seam function**, no state-file / two-roots reconciliation, no GUI/cockpit.

## 8. Constraints / risks

- **TDD iron law:** no production code without a failing test first.
- **Code placement:** all new code is TS under `scripts/brain/` — **no new `.mjs` under `hooks/scripts/`**, so the `pre-write-pathguard.mjs` Category-6 guard does not fire (it matches `…/hooks/scripts/*.mjs` on Write only; SB-S1 writes none). `agents/voice-trainer.md` is an **existing** file → Edit, not Write → unaffected regardless.
- **Public repo (`open/` Forgejo):** confirm before push; push only inside the window; STATE.md stays gitignored.
- **Structure lint:** SB-S1 adds no `agents/`, `commands/`, `references/`, or `skills/` file (ingest is a CLI subcommand in an existing package; voice-trainer is edited; the contract doc lives in `docs/`). So no count guard fires absent a version bump. **Version bump is an open question** (§9): the agent behavioural change may warrant 0.5.0 → 0.5.1.
- **Node v25 test quirk:** `node --test <dir>` fails — use `node --import tsx --test tests/*.test.ts`; run TS tools from their own dir; `npm install` in `scripts/brain` before the gate or the suite warn-skips.
- **Risk — store-format ambiguity:** the published-record file format is the one data-shaping decision in S1 (mirrors SB-S0's fold risk). Mitigate by pinning the grammar + a round-trip test (SC2) before the CLI wires it.
- **Risk — invariant enforceability:** "wire voice-trainer to published-only" is an agent-doc instruction, not executable code; an instruction can be ignored at runtime. Mitigate by making SC5 a *gate-checkable* lint where feasible (grep for the rule's presence), and by stating the limit honestly in the doc.
- **Risk — S1/S2 scope creep:** the temptation is to start consuming `published/` into `profile.md` here. Fence held by §7 + SC (no test asserts profile mutation in S1).

## 9. Open questions for the operator (settle at "go")

1. **S1/S2 boundary (§5.2):** confirm S1 = data layer + voice-trainer invariant wiring, S2 = the automatic profile-diff loop. (Recommended.)
2. **Profile vs voice in S1:** confirm that *profile* learning is invariant-only in S1 (provenance already on facts; no profile mutation) and that the only *active* consumer wired is `voice-trainer`. (Recommended.)
3. **Store format (~decided, confirm):** published records as **file-per-post constrained-header markdown** (the `brain/profile.md` line-grammar idiom, no YAML) — matching the architecture's `inbox`→`published` per-file drop-zone. This deliberately does NOT reuse the specifics-bank single-JSON-index idiom (that store is a topic-tagged bank, not a per-file drop-zone). Confirm.
4. **Version bump:** take 0.5.0 → 0.5.1, or no bump? SB-S0 set a no-bump precedent for count-neutral slices, but SB-S1 differs: it changes a *shipped agent's learning behaviour* (`voice-trainer`), a user-visible capability change — a stronger case for 0.5.1 than SB-S0 had. (Lean 0.5.1.)

## 10. Definition of done (SB-S1)

SC1–SC6 pass; deliverables 1–7 exist; non-goals untouched; the published-only invariant is wired into `voice-trainer` + documented + **gate-enforced (SC5 structure-lint)**; the BRAIN + assertion floors raised; committed (push per window, confirm first); `architecture.md` SB-S1 row → "landed"; version/CLAUDE.md/README touched only if a bump is taken.