diff --git a/docs/second-brain/brief-sb-s2.md b/docs/second-brain/brief-sb-s2.md new file mode 100644 index 0000000..32ec337 --- /dev/null +++ b/docs/second-brain/brief-sb-s2.md @@ -0,0 +1,129 @@ +# Task Brief — SB-S2 (Evolution loop) + +> **Status:** DRAFT for adversarial brief-review (light-Voyage step 2). Not yet operator-approved. +> **Arc:** `architecture.md` (approved 2026-06-23). **Predecessors:** SB-S0 (scaffold + profile spine) + SB-S1 (ingest + published gold signal), both landed + released (v0.5.1). +> **Operator scope decisions (2026-06-23, settled before this brief):** (1) **journal-capture DEFERRED** — S2 consumes `published` + tributary deltas only; (2) **deterministic CLI engine + the invoking session does the fuzzy extraction** (count-neutral, NO new agent); (3) **motor-only** — S2 evolves `brain/profile.md` but wiring a *reader* is S3 (accepted, flagged). +> **Inputs:** `architecture.md` §"evolution loop" + §"invariants"; SB-S0/S1 code (`scripts/brain/`); the session-start surface (`hooks/scripts/session-start.mjs`). +> **Review delta (2026-06-23):** folded the brief-review (REVISE). B1+B2 (the state-file write idiom is false AND the brain CLI can't reach the HOME-scoped state file) → resolved by storing consolidation state in a **brain data-root sidecar** (`brain/consolidation-state.json`), reachable by both `--apply` (dataRoot) and the session-start nudge (getDataRoot) — no state-file involvement. M3 `--propose` emits TWO artifacts (md + json). M4 **supersede dropped from S2** (keep-both only; supersede→S3). M5 SC1 split per-rule. Minors: SC3 split, candidate-file schema + gate-validation, freshness nudge = new-published-count only (no profile.md parse at session-start), consolidation matches `kind:'observed'` only (folded `profile-field` static seeds immutable in S2). + +--- + +## 1. Objective + +Build the **compounding mechanism**: an operator-invoked, sleep-time **consolidation pass** that reads the new published gold signal + tributary deltas, proposes a **diff** to the two-layer `brain/profile.md` (never a silent overwrite) — with evidence-threshold promotion, contradiction reconciliation (bias: keep-both-timestamped), provenance-gated learning (`published`/`human` only, never `ai-draft`), and temporal decay flagging (in the diff, not at session-start) — plus a **"consolidation-due" nudge** on the session-start surface (new-published count + days-since-last-run; per-fact stale nudging deferred), and the **session-start wiring of the scaffold init**. This is the loop that turns the gold signal (SB-S1) into an ever-improving, drift-resistant profile. + +## 2. Background (why this shape, given the surfaces) + +- **The loop cannot run AI at session-start.** `session-start.mjs` is zero-dep, must NOT spawn `tsx`, 10s timeout (the B-S3 rule). So the consolidation *pass* (which needs fuzzy fact-extraction) is **operator-invoked** ("sleep-time" = when the operator runs it), and session-start only emits a deterministic **nudge** ("consolidation due / N new published posts"). This is faithful to the architecture's "sleep-time consolidation, not inline cost." +- **journal/ has no producer.** SB-S0 created the empty dir; nothing writes episodes. Per the operator decision, S2 consumes `published` (SB-S1, structured, provenance-tagged) + tributary deltas (specifics/trends/analytics — all have real timestamps); **journal-capture is its own later sub-slice.** +- **The fuzzy extraction lives in the command/session layer, not a new agent** (the specifics-bank precedent: the deterministic store + CLI ship, the elicitation interview lives upstream). So S2 ships a deterministic **engine + CLI**; the invoking session reads the gathered deltas and produces candidate facts. Count-neutral (no `agents/` file → no reload, no count-lint bump). +- **profile.md has no reader yet.** S2 evolves it into a (currently) unconsumed artifact; wiring content consumers is S3. Accepted: the value is that the profile compounds NOW so S3's reader inherits rich data. Flagged honestly (§9). +- **The diff is operator-gated** — matches the longform `SendUserFile` + `[OPERATØR]` idiom (editorial-review): proposed, presentable, never silently applied; confirm/edit/roll-back via git. Anti-sycophancy: the profile is evidence to *test*, presented neutrally. + +## 3. Confirmed decisions (inherited arc + operator scope — not for re-litigation) + +1. **Boundary:** engine → plugin · user data → data dir · cockpit → Maskinrommet. +2. **Invariants (architecture §"Invariants"):** provenance-weighted learning (`published`/`human` only, never `ai-draft`); episodic/semantic split; evidence-threshold promotion; contradiction → keep-both-timestamped; temporal-validity + freshness nudge; anti-sycophancy default; zero required curation. +3. **Operator scope (§Status):** journal deferred · deterministic CLI + session extraction (no new agent) · motor-only (no reader until S3). +4. **No new seam function** — brain code uses the package's own `dataRoot`; the session-start edit uses the existing `getDataRoot` (`data-root.mjs`). +5. **No AI at session-start** (zero-dep nudge only); the consolidation pass is operator-invoked; **no auto-apply** of any diff. + +Review SHOULD challenge: feasibility, the engine's classification rules (§5.1), the session-start edit's zero-dep safety, the diff/apply data shapes, hidden coupling, testability, scope leak, missing risks. Review SHOULD NOT re-open the §3 decisions unless internally contradictory or technically impossible. + +## 4. Scope — where SB-S2 sits in the arc + +| Slice | One-line scope | This brief | +|-------|----------------|------------| +| SB-S0 — Foundation | scaffold + profile fold + id/provenance spine | ✅ landed | +| SB-S1 — Ingest + gold signal | manual import → `ingest/published/`; voice-trainer published-only | ✅ landed (v0.5.1) | +| **SB-S2 — Evolution loop** | consolidation engine → profile **diff** (threshold/contradiction/decay, provenance-gated) + session-start freshness/consolidation-due nudge + scaffold-ensure wiring | **← this** | +| SB-S3 — Cross-silo graph + ops centre | thread the id through tributaries; wire profile.md READERS; flesh out `operations.md`; retire dead `content-history.md` | later | +| SB-S4 *(optional)* | EU/EEA DMA portability auto-tributary | later | + +## 5. SB-S2 in detail (the build target) + +### 5.1 Deliverables + +1. **Deterministic consolidation engine** (`scripts/brain/src/consolidate.ts`) — PURE functions, no IO, heavily tested. Core: `proposeDiff({ current: ProfileDoc, candidates: Candidate[], today, opts }) → ProfileDiff`. A `Candidate` = `{ key, value, provenance, source, observed_date }`. **Matching:** every candidate is keyed `mintEntityId({kind:'observed', key})` and matched ONLY against existing `observed`-kind facts. The SB-S0 **folded `profile-field` static seeds are immutable in S2** — a candidate that semantically overlaps a folded field becomes its own `observed` fact (the keep-both philosophy; reconciling observed-vs-folded is operator-gated / S3). This avoids the `profile-field`↔`observed` id mismatch (fixes brief-review NICE). Classification rules: + - **Reject** any `provenance: 'ai-draft'` candidate outright (model-collapse guard) — it never enters or bumps a fact. + - **Add** (no matching `observed` fact, provenance ∈ {published, human}) → a new **dynamic** fact, `evidence_count: 1`, dates = `observed_date`/`today`. + - **Evidence-bump** (matching fact, SAME value) → `evidence_count++`, `last_seen = today` (no other field touched). + - **Promote** (a dynamic fact whose `evidence_count` reaches threshold `N`, default 3) → move dynamic→**static**. + - **Conflict** (matching key, DIFFERENT value) → **keep-both-timestamped**: add the new value as its own fact, keep the old fact untouched (no evidence-bump of the old), and record the pair in `conflicts[]` — the architecture's explicit bias against silent overwrite. **No supersede in S2** (flipping `status:superseded` is deferred to S3; S2 never demotes/supersedes a fact). + - **Decay-flag** — dynamic facts whose `last_seen` is older than `DECAY_DAYS` (default 90) are listed in `staleFlags[]` (information only; S2 never auto-demotes). + `ProfileDiff` = `{ additions[], evidenceBumps[], promotions[], conflicts[], staleFlags[] }` (typed; NO `supersessions[]` — that arm is S3). The engine NEVER mutates `profile.md` — it returns proposals over a deep-frozen-safe input. `applyDiff(current, diff) → ProfileDoc` is a separate pure function (used only by the gated apply path); its output round-trips through `parseProfile`/`serializeProfile`. + +2. **`brain consolidate` CLI** (extend `scripts/brain/src/cli.ts`) — the operator-invoked loop: + - `brain consolidate --gather [--json]` — read-only: gather the deltas since the last run (new `published` records + each tributary's newest-timestamp summary + the current profile) and print them, for the **invoking session** to read and turn into candidate facts. (The fuzzy extraction lives here, in the session — no new agent.) + - `brain consolidate --propose --candidates ` — **validate** the candidate file against the `Candidate` shape (malformed/missing field → non-zero exit, no profile touch — the only enforceable session↔engine contract), run `proposeDiff` over the current profile + the candidates, and write **TWO artifacts** to `brain/`: `pending-diff.md` (operator-readable, for the `SendUserFile` gate) AND `pending-diff.json` (the typed `ProfileDiff`, machine-applyable). Print both paths. Does NOT touch `profile.md`. + - `brain consolidate --apply --diff brain/pending-diff.json --confirm` — apply the confirmed JSON diff to `profile.md` (the only path that writes the profile), then update the consolidation sidecar (D3). **Operator-gated** (requires `--confirm`); never auto-runs. + +3. **Last-consolidation state — a brain data-root sidecar** `brain/consolidation-state.json` (`{ "last_run": "YYYY-MM-DD" }`), written by `--apply` and read by the session-start nudge. **This resolves brief-review B1+B2:** both the brain CLI (`dataRoot`) and the session-start hook (`getDataRoot`) resolve the data root, so the sidecar is reachable by both — whereas the plugin state file lives in HOME (`~/.claude/linkedin-studio.local.md`), a SIBLING of the data root the brain CLI cannot reach through its own resolver, and `state-updater.mjs` has no `last_import_date` writer to mimic (that scalar is written by the import *command* via LLM, not a function). Keeping consolidation state in the brain's own data-root territory is also conceptually right (brain state, not plugin state) and zero-dep JSON for the hook to read. + +4. **Session-start wiring** (Edit `hooks/scripts/session-start.mjs` — existing file, zero-dep, pathguard-safe; all reads via the existing `getDataRoot`, the trend-nudge twin idiom): + - **Scaffold-ensure (zero-dep):** mkdir the brain/ + ingest/ dirs if absent (idempotent, no tsx); if `brain/profile.md` is absent, emit a nudge to run `brain init` (seeding needs the tsx fold — kept manual, not inlined). + - **Brain consolidation-due nudge (new-published-count only):** read `brain/consolidation-state.json` `last_run` (zero-dep JSON parse) + count `ingest/published/*.md` records (a `readdir` count — **no `profile.md` parse**, to respect the 10s budget + avoid a fragile zero-dep fact parser). Trend-nudge discipline: `daysSince(last_run)` + threshold + null-safe (a never-consolidated brain with no published records never nags). Emit e.g. "N published posts captured, last consolidation M days ago → run `brain consolidate`." + - **Out of scope (honest):** **stale-FACT** nudging (parsing `profile.md` last_seen at session-start) is NOT shipped in S2 — it needs a zero-dep fact parser + cost, and bites harder once a reader exists. S2's brain-wide nudge = the new-published/consolidation-due signal only; per-fact decay is computed inside `proposeDiff` (`staleFlags[]`), surfaced in the diff, not at session-start. + +5. **Operator-gated diff presentation** — the `--propose` artifact is a readable markdown diff (additions/bumps/promotions/conflicts/stale, sorted, neutral/anti-sycophantic framing); the invoking session `SendUserFile`s it and presents the `[OPERATØR]` gate before any `--apply` (the editorial-review idiom). + +6. **Docs** — `docs/second-brain/consolidation-loop.md` (the loop contract: CLI usage, the engine rules, the operator-gate, the honest limits); this brief + the SB-S2 plan; flip the `architecture.md` SB-S2 row to landed. + +7. **Tests + gate** — engine unit tests (every classification rule + idempotency + provenance-gating + threshold + keep-both + decay + `applyDiff` round-trip), CLI tests (gather/propose/apply, no-auto-apply, last-run recording), and a **session-start nudge test** in the hooks test suite (consolidation-due nudge fires/null-safe). Bump `BRAIN_TESTS_FLOOR` + the assertion floor (per the SB-S1-established empirical method) + the hook-test count if applicable. + +### 5.2 What "owns the session-start wiring of the scaffold init" means here + +S2 makes the brain **present + nudged** at session-start without running tsx: it zero-dep-ensures the dirs and nudges `brain init` (profile seeding stays the manual tsx CLI) + nudges `brain consolidate` when due. It does NOT run the consolidation pass automatically (that needs AI → operator-invoked). + +## 6. Success criteria (SB-S2 — testable) + +Engine rules each get their OWN criterion (fixes brief-review M5; N=3, DECAY_DAYS=90 pinned as the test constants — confirm §9 Q1 at "go" first): +- **SC1a (add):** a candidate with no matching `observed` fact + provenance ∈ {published,human} populates `additions[]` as a dynamic fact, `evidence_count:1`. +- **SC1b (reject ai-draft):** an `ai-draft` candidate yields an EMPTY diff — no add, no bump, no promote (the model-collapse guard, in code). +- **SC1c (evidence-bump):** a candidate matching an existing fact with the SAME value populates `evidenceBumps[]` (`evidence_count++`, `last_seen=today`), no other field changed. +- **SC1d (promote):** a dynamic fact reaching `evidence_count ≥ 3` populates `promotions[]` (dynamic→static). +- **SC1e (conflict→keep-both):** a candidate matching a key with a DIFFERENT value populates `conflicts[]`, ADDS the new value as its own fact, and leaves the old fact untouched (no evidence-bump, no supersede) — both retained. +- **SC1f (decay-flag):** a dynamic fact with `last_seen` older than 90 days is listed in `staleFlags[]`; no auto-demotion. +- **SC1g (folded immutable):** a candidate overlapping a folded `profile-field` static seed creates a separate `observed` fact and never mutates the folded one. +- **SC2 (immutability):** `proposeDiff` does not mutate its `current`/`candidates` inputs (asserted by structural equality of the inputs after the call). +- **SC3 (apply round-trip):** `applyDiff(current, proposeDiff(...))` yields a `ProfileDoc` that round-trips exactly through `parseProfile`/`serializeProfile`. +- **SC4 (idempotency):** re-running propose→apply with the same candidates over an already-applied profile yields no spurious additions (evidence-bumps/`last_seen` only; no duplicate facts). +- **SC5 (CLI gating + validation):** `--gather` and `--propose` never write `profile.md` (asserted byte-unchanged after each); `--propose` writes `pending-diff.{md,json}` to `brain/` and REJECTS a malformed candidate file (non-zero exit, no profile touch); only `--apply --confirm` writes `profile.md` and updates `brain/consolidation-state.json`. Verified by temp-dir CLI tests. +- **SC6 (session-start nudge, zero-dep):** the consolidation-due nudge fires when published records exist + `last_run` is stale/absent, and is silent when the brain is absent / nothing new; the edit adds no `tsx`/heavy dep and parses no `profile.md`. Verified by a hook-suite test (HOME + `LINKEDIN_STUDIO_DATA` set to separate temp dirs — the `session-start-trends-staleness.test.mjs` harness pattern — asserting the nudge string appears / is absent). +- **SC7 (no regression / no new seam / count-neutral):** `scripts/test-runner.sh` green; floors hold (trends ≥24, specifics ≥28, contract ≥33, brain ≥ new floor); hook tests pass; **no new agent/command/ref/skill** (counts unchanged); no new seam function (twin-sync untouched); `compile-hooks.py --check` clean if `hooks.json` is regenerated. + +## 7. Non-goals / scope fence (SB-S2 does NOT) + +- **No journal-capture** (deferred sub-slice) — S2 reads `published` + tributary deltas only; nothing writes `brain/journal/`. +- **No new agent** — the fuzzy extraction is done by the invoking session (no `agents/*.md`, no reload, no agent-count bump). +- **No profile.md READER** — no content agent/command is wired to consume `brain/profile.md` (S3). +- **No cross-silo id threading** into tributaries (S3); no `content-history.md` retirement / triple-post reconciliation (S3). +- **No AI at session-start** — only a deterministic nudge; the consolidation pass is operator-invoked. +- **No auto-apply** — no diff is ever applied without `--apply --confirm`; `profile.md` is mutated by exactly one gated path. +- **No new hook `.mjs`** — only the existing `session-start.mjs` is edited (pathguard: Write of a new `hooks/scripts/*.mjs` is blocked). +- **No new seam function**, no connector (S4), no GUI. + +## 8. Constraints / risks + +- **TDD iron law:** no production code without a failing test first. +- **Code placement / pathguard:** engine + CLI are TS under `scripts/brain/` (safe). The session-start change is an **Edit** of the existing `hooks/scripts/session-start.mjs` (Edit is unaffected by the Category-6 Write guard) — **no new `.mjs` under `hooks/scripts/`**. +- **Zero-dep hook discipline:** the session-start edit must NOT import tsx/the brain TS, must stay within the 10s timeout, and must use direct raw reads (the trend-nudge twin idiom). The brain consolidation is NOT run there. +- **Hook compile:** if the SessionStart entry changes shape, regenerate `hooks.json` via `compile-hooks.py` (drift-guarded). Editing only the script body needs no recompile. +- **Consolidation state is a brain sidecar, NOT the plugin state file** (fixes brief-review B1+B2): `brain/consolidation-state.json` under the data root, written by `--apply` (dataRoot) + read by the session-start hook (getDataRoot, zero-dep JSON). Do NOT route it through `~/.claude/linkedin-studio.local.md` — the brain CLI cannot reach HOME, and `state-updater.mjs` has no `last_import_date`-style writer to mimic (that scalar is an LLM/command write). Reading it at session-start is a `JSON.parse(readFileSync(...))`, the trend-`trends.json`-read twin. +- **Public repo (`open/`):** confirm before push; window; STATE.md gitignored. +- **Count-neutral:** no `agents/commands/references/skills` file added → no count guard fires (absent a version bump). **Version bump** likely 0.5.1 → 0.5.2 (new capability + session-start behavior) — confirm at "go". +- **Risk — engine rule ambiguity:** the conflict-vs-supersede call is the fuzzy heart. Mitigation: bias to keep-both-timestamped (deterministic + safe), reserve supersede for an explicit confirm signal; pin each rule with a unit test before wiring the CLI. +- **Risk — candidate extraction quality:** the *value* of the loop depends on the session extracting good candidates from the gathered deltas; the engine only guarantees the mechanics. Honest limit (the architecture's "voice fidelity is hard"): S2 ships the machinery, not a guarantee of insight. +- **Risk — unconsumed output:** profile.md has no reader until S3; S2's value is deferred (compounding for later). Accepted by the operator; stated in `consolidation-loop.md`. +- **Risk — session-start cost:** the nudge must stay cheap; counting `published/` files + one scalar read is O(files) — bound it (e.g. count via readdir, no parse) to respect the 10s budget. + +## 9. Open questions for the operator (settle at "go") + +1. **Promotion threshold N + decay window:** default `N = 3` independent observations to promote dynamic→static, `DECAY_DAYS = 90` for stale-flagging. Confirm or adjust. (Recommend N=3 / 90d — pinned as the test constants.) +2. **Version bump:** 0.5.1 → 0.5.2 for the new consolidation capability + session-start nudge? (Lean 0.5.2, consistent with SB-S1.) + +*Resolved by the brief-review (no longer open):* conflict = **keep-both only, no supersede in S2** (supersede→S3); diff artifacts = `brain/pending-diff.{md,json}` (git-trackable per "roll-back via git"). + +## 10. Definition of done (SB-S2) + +SC1a–g + SC2–SC7 pass; deliverables 1–7 exist; non-goals untouched; the engine is provenance-gated + keep-both-only (no supersede) in CODE (not just docs); consolidation state is the brain sidecar (not the plugin state file); the consolidation pass is operator-invoked + the diff operator-gated (no auto-apply); session-start nudge zero-dep (no profile.md parse) + tested; committed (push per window, confirm first); `architecture.md` SB-S2 row → landed; version/CLAUDE.md/README touched only if a bump is taken. diff --git a/docs/second-brain/plan-sb-s2.md b/docs/second-brain/plan-sb-s2.md new file mode 100644 index 0000000..41259d1 --- /dev/null +++ b/docs/second-brain/plan-sb-s2.md @@ -0,0 +1,160 @@ +# Implementation Plan — SB-S2 (Evolution loop) + +> **Status:** review-hardened (`plan-critic` REPLAN 58→folded; `scope-guardian` ALIGNED). Ready for operator approval / "go". +> **Brief:** `brief-sb-s2.md` (review-hardened). **Arc:** `architecture.md`. **Predecessors:** SB-S0/S1 (`scripts/brain/`, v0.5.1). +> **Review delta (2026-06-23):** B1 conflict id → distinct (primary `mintEntityId(key)` + alt `mintContentId('observed-alt:'+key+'::'+value+'::'+date)`, no duplicate ids); B2 `--gather` reads published files directly for bodies (not `listPublished`), tributary reads CUT; B3 hook tests run SEPARATELY (`node --test hooks/scripts/__tests__/`) as an explicit step — NOT via test-runner.sh (which runs no hook test) → SC7 reworded, assertion floor +0; M4 new hook-test `.mjs` authored via Bash-heredoc (pathguard-safe under either guard); M5 scaffold-ensure runs UNCONDITIONALLY (fresh-install path); M6 `\\n` escaped-newline idiom; M8 per-step commit msg + revert target; minors: single-line value validation, static-decay-exempt, tsx fail-loud. +> **Scope:** the deterministic consolidation engine + `brain consolidate` CLI (operator-invoked, operator-gated) + the session-start consolidation-due nudge + scaffold-ensure. **No journal-capture, no new agent, no profile.md reader, no supersede, no cross-silo threading, no auto-apply, no AI at session-start** (brief §7). + +--- + +## 1. What SB-S2 delivers (recap) + +A new PURE TS engine `scripts/brain/src/consolidate.ts` (`proposeDiff` + `applyDiff` over the SB-S0 two-layer `ProfileDoc`), a `brain consolidate` CLI (`--gather` / `--propose` / `--apply --confirm`) that closes the loop operator-gated, a brain data-root sidecar `brain/consolidation-state.json` for the last-run timestamp, and a zero-dep **consolidation-due nudge** + **scaffold-ensure** edit to `hooks/scripts/session-start.mjs`. Count-neutral (no agent/command/ref/skill). All TDD; gate stays green; floors rise. + +## 2. Package additions (`scripts/brain/`) — extend the existing package + +``` +scripts/brain/ + src/ + consolidate.ts # NEW — Candidate/ProfileDiff types + proposeDiff + applyDiff (pure) + consolidation-state IO + cli.ts # EDIT — add `consolidate` dispatch (--gather/--propose/--apply) + (types.ts, id.ts, profile.ts, ingest.ts, scaffold.ts, dataRoot.ts unchanged) + tests/ + consolidate.test.ts # NEW — engine: per-rule (SC1a–g) + immutability + round-trip + idempotency + consolidate-cli.test.ts # NEW — CLI gating + candidate-validation + sidecar (SC5) +hooks/scripts/ + session-start.mjs # EDIT (existing → pathguard-safe) — scaffold-ensure + consolidation-due nudge + __tests__/ + session-start-brain-consolidation.test.mjs # NEW — SC6 (mirror of session-start-trends-staleness.test.mjs) +``` + +No new dep (engine is pure TS; the hook stays zero-dep). The brain `test` script already globs `tests/*.test.ts`. + +## 3. Module designs + +### 3.1 `consolidate.ts` — types +```ts +import { mintEntityId, mintContentId, slugify } from "./id.js"; +import { SCHEMA_VERSION } from "./types.js"; +import type { ProfileDoc, ProfileFact, Provenance } from "./types.js"; + +export interface Candidate { + key: string; // the fact's stable label/topic/concept + value: string; // single-line (no embedded newline/CR — profile grammar) + provenance: Provenance; // published | human | ai-draft + source: string; // e.g. "published:" | "manual" + observed_date: string; // YYYY-MM-DD +} +export interface ProfileDiff { + additions: ProfileFact[]; // new dynamic facts (primary + conflict-alt) + evidenceBumps: { id: string; newCount: number; last_seen: string }[]; + promotions: { id: string }[]; // dynamic→static (post-bump count ≥ N) + conflicts: { primaryId: string; primaryValue: string; altId: string }[]; // both retained, distinct ids + staleFlags: { id: string; last_seen: string; daysStale: number }[]; +} +export interface ConsolidateOpts { promoteThreshold?: number; decayDays?: number } // defaults 3 / 90 +const OBSERVED_KIND = "observed"; +``` + +### 3.2 `consolidate.ts` — `proposeDiff` (pure; the classification heart) — id model fixes plan-critic B1 +**Id model (the B1 fix — no duplicate ids):** +- A concept's **primary fact** id = `mintEntityId({ kind: OBSERVED_KIND, key })` (key-only → one primary per concept). +- A conflict **alt fact** id = `mintContentId(\`observed-alt:${key}::${value}::${observed_date}\`)` (SB-S1's verbatim sha256[:12] — byte-distinct, so primary and alt never collide and two alts for the same concept differ by value/date). Matching always starts from the *candidate's* `key`, so we never need to recover a key from an existing fact. + +```ts +proposeDiff({ current, candidates, today, opts }): ProfileDiff +``` +- `N = opts.promoteThreshold ?? 3`, `DECAY = opts.decayDays ?? 90`. +- Build a `byId` lookup over `current.dynamic ∪ current.static`. (Folded `profile-field` static seeds use a DIFFERENT kind → their ids never collide with `observed` ids → immutable, SC1g.) +- For each candidate: + - `provenance === 'ai-draft'` → skip entirely (SC1b). + - `primaryId = mintEntityId({kind:OBSERVED_KIND, key})`; `prev = byId.get(primaryId)`. + - **no `prev`** → `additions` gets a new dynamic `ProfileFact` (`id:primaryId`, `evidence_count:1`, `first_seen:observed_date`, `last_seen:today`, `status:'active'`, `provenance`) (SC1a). + - **`prev`, `prev.value === candidate.value`** → `evidenceBumps` (`newCount = prev.evidence_count+1`, `last_seen:today`); if `prev` is in `dynamic` AND `newCount ≥ N` → also `promotions[{id:primaryId}]` (SC1c/SC1d). + - **`prev`, `prev.value !== candidate.value`** → CONFLICT (SC1e): build an alt fact `{id: mintContentId(\`observed-alt:${key}::${candidate.value}::${observed_date}\`), value:candidate.value, dynamic, evidence_count:1, …}`, push it to `additions`, record `conflicts[{primaryId, primaryValue: prev.value, altId}]`, and leave `prev` UNTOUCHED (no bump). **No supersede** (S3). (If the alt id already exists from a prior run → treat as the same-value bump path on the alt id, so re-running is idempotent.) +- After the candidate pass: `staleFlags` = every **`current.dynamic`** fact whose `daysSince(last_seen) > DECAY` (SC1f). **Static facts are decay-exempt by design** (promoted = stable). Informational only — no mutation. +- **Purity:** never mutate `current`/`candidates` (build new objects); SC2 asserts inputs structurally unchanged. + +### 3.3 `consolidate.ts` — `applyDiff` (pure) + sidecar IO +- `applyDiff(current, diff): ProfileDoc` — returns a NEW doc: append `additions` (which already includes both new primary facts AND conflict alt facts) to `dynamic`; apply `evidenceBumps` (update count+last_seen on the fact with the matching id); move `promotions` facts from `dynamic`→`static`. `conflicts[]` is informational (the alt fact is already in `additions`); `staleFlags` cause NO mutation. **Id-uniqueness invariant:** because primary ids (`mintEntityId(key)`) and alt ids (`mintContentId(...)`) are byte-distinct, no two facts share an id — so the bump/promote target is unambiguous and the doc stays well-formed. Output round-trips through `parseProfile`/`serializeProfile` (SC3). +- **Sidecar IO** (the only impure bit, kept here for cohesion): `readConsolidationState(): {last_run: string|null}` + `writeConsolidationState(date)` over `dataRoot('brain/consolidation-state.json')` (`{ "last_run": "YYYY-MM-DD" }`). Tiny JSON; absent → `{last_run:null}`. + +### 3.4 `cli.ts` — `consolidate` dispatch (extend the SB-S1 router) +Add `command === "consolidate"` with a `--gather` / `--propose` / `--apply` mode (the parseFlags idiom already present): +- **`--gather [--json]`** — read-only. Reads the published gold corpus **directly** (`readdir(dataRoot('ingest/published'))` → `parsePublishedRecord` each — `listPublished()` returns no body/`captured_at`, so it can't feed extraction; fixes plan-critic B2), filters to records with `published_date > readConsolidationState().last_run` (null → all), and prints a digest of `{id, published_date, body}` per new record + the current `brain/profile.md` facts (parsed), for the **invoking session** to turn into a `Candidate[]` JSON. **Tributary newest-timestamp reads are CUT** (informational-only, underspecified — the published corpus is the S2 signal; plan-critic M12). Writes nothing. +- **`--propose --candidates `** — read + **validate** each candidate against the `Candidate` shape: every item has key/value/provenance∈{human,published,ai-draft}/source/observed_date, AND **`key`/`value` contain no newline or carriage-return** (the profile grammar is single-line — a `\n` would corrupt serialization; plan-critic M9). Any violation → non-zero exit, no write (SC5). `proposeDiff` over the current profile. Write `brain/pending-diff.json` (the typed `ProfileDiff`) AND `brain/pending-diff.md` (operator-readable: sections Additions / Evidence-bumps / Promotions / ⚠ Conflicts / Stale, neutral framing). Print both paths. Does NOT touch `profile.md`. +- **`--apply --diff --confirm`** — require `--confirm` (else refuse). Read the JSON diff, `parseProfile(brain/profile.md)`, `applyDiff`, write `brain/profile.md` (`serializeProfile`), then `writeConsolidationState(today())`. The ONLY path that writes the profile. +- Usage text + exit 2 on misuse. `init`/`ingest`/`published` branches unchanged (regression-guarded). + +### 3.5 `hooks/scripts/session-start.mjs` — scaffold-ensure + consolidation-due nudge (Edit, zero-dep) +- Add `readdirSync` to the existing `node:fs` import (line 5). +- **Helpers (twins of `trendsNewestCapture`, lines 38–52):** + - `brainLastRun(path)` — `JSON.parse(readFileSync)` of `consolidation-state.json` → `last_run` or null (try/catch → null). + - `countPublished(dir)` — `existsSync(dir) ? readdirSync(dir).filter(f=>f.endsWith('.md')).length : 0` (no parse — cost-bounded). +- **Scaffold-ensure — UNCONDITIONAL** (fixes plan-critic M5: the `reminders`/context block runs only inside `if (existsSync(STATE_FILE))`, so a fresh install (else branch ~line 432) would never scaffold/nudge). Place it next to the M0 migration block (~line 73), BEFORE the state-file branch, so it always runs: `for (const d of ['brain/journal','ingest/inbox','ingest/published']) mkdirSync(join(getDataRoot(''),d),{recursive:true})` (idempotent, zero-dep; confirm `getDataRoot('')` returns the bare data root — step 1). The optional "run `brain init` to seed your profile" nudge is appended to **`context` directly** (not `reminders`) right after scaffold-ensure when `!existsSync(join(getDataRoot('brain'),'profile.md'))`, so it survives the no-state-file path. +- **Consolidation-due nudge** (inside the reminders block, beside the trends nudge ~line 342 — an established user with published content has a state file, so the reminders path is correct here). **Use the `\\n` escaped-newline idiom** (the file un-escapes once via `context.replace(/\\n/g,'\n')` at line 479 — a raw `\n` would mis-render; plan-critic M6): + ```js + const published = countPublished(join(getDataRoot('ingest'), 'published')); + const lastRun = brainLastRun(join(getDataRoot('brain'), 'consolidation-state.json')); + const daysSinceRun = daysSince(lastRun); + if (published > 0 && (lastRun === null || (daysSinceRun !== null && daysSinceRun >= 7))) { + const since = lastRun === null ? 'never run' : `${daysSinceRun}d ago`; + reminders += `- ${published} published post(s) captured, last brain consolidation ${since}. Run \\\`brain consolidate\\\` to evolve your profile.\\n`; + } + ``` + Null-safe + fires only when published records exist (a never-used brain never nags — the trend-nudge discipline). **No `profile.md` parse.** + +## 4. Files created / edited + +**Created:** `scripts/brain/src/consolidate.ts` · `scripts/brain/tests/{consolidate,consolidate-cli}.test.ts` · `hooks/scripts/__tests__/session-start-brain-consolidation.test.mjs` · `docs/second-brain/consolidation-loop.md`. + +**Edited:** `scripts/brain/src/cli.ts` (+consolidate) · `hooks/scripts/session-start.mjs` (scaffold-ensure + nudge + `readdirSync` import) · `scripts/test-runner.sh` (BRAIN floor + assertion floor) · `docs/second-brain/architecture.md` (flip SB-S2 row). + +**Conditional (Q2 = bump to 0.5.2):** `plugin.json` + README badge + CLAUDE.md header + CHANGELOG; polyrepo release via `release-plugin.mjs` (tag + catalog), pushed per window + confirm. + +## 5. TDD test plan (failing-first) → success criteria + +| Test | Asserts | SC | +|---|---|---| +| `consolidate.test.ts` | add / reject-ai-draft / evidence-bump / promote-at-3 / **conflict-keep-both with DISTINCT primary+alt ids, old fact untouched** / decay-flag (dynamic only) / **static-decay-exempt** / folded-immutable; input immutability; `applyDiff∘proposeDiff` round-trips via parse/serialize; idempotent (re-run → bumps not dup facts, alt-id stable) | SC1a–g, SC2, SC3, SC4 | +| `consolidate-cli.test.ts` | temp-dir: `--gather` reads published bodies (filtered by `published_date>last_run`) + leaves `profile.md` byte-unchanged; `--propose` leaves profile.md unchanged, writes `pending-diff.{md,json}`, REJECTS a malformed candidate file AND a candidate whose value contains `\n` (non-zero, no write); `--apply --confirm` writes profile.md + `consolidation-state.json`; `--apply` without `--confirm` refuses; `init`/`ingest`/`published` still work | SC5 | +| `session-start-brain-consolidation.test.mjs` (authored via **Bash-heredoc** — pathguard-safe) | subprocess + isolated HOME/`LINKEDIN_STUDIO_DATA` (trends-staleness harness): nudge fires (rendered on its own line) when published records exist + last_run stale/absent; silent when brain absent / no published; scaffold-ensure runs on the fresh-install (no-state-file) path; hook still emits valid JSON | SC6 | +| (separate) `node --test hooks/scripts/__tests__/` | the new SC6 test + all existing hook tests pass — **run as an explicit step, NOT via test-runner.sh** (which runs no hook test; plan-critic B3) | SC6 | +| (gate) `scripts/test-runner.sh` | green; floors hold; brain ≥ new floor; counts unchanged; `compile-hooks.py --check` clean if hooks.json regenerated. (No new test-runner section → assertion floor +0.) | SC7 | + +Each test written **before** its module (iron law). **Circuit-breakers:** halt if the engine round-trip (SC3) can't be made green, or if the full gate (step 8) is red. + +## 6. Step sequence (each step: TDD red→green; Checkpoint = commit with the named message; On-failure → revert the named target + halt + report) + +1. **Verify-first:** re-confirm `proposeDiff` inputs against `types.ts` (`ProfileFact`/`ProfileDoc`/`Provenance`) + `mintEntityId`/`mintContentId`/`slugify` signatures + `daysSince`/`getDataRoot('')` behavior in session-start.mjs (does the empty subdir return the bare data root?). **tsx fail-loud:** assert `scripts/brain/node_modules/.bin/tsx` present (the gate brain section *warn-skips* if absent — for this slice that would let the gate go green with SB-S2's tests never run; `npm install` and FAIL if still absent; plan-critic M11). *On-failure: stop, report — the slice cannot be verified.* +2. TDD `consolidate.ts` types + `proposeDiff` per-rule (`consolidate.test.ts`, SC1a–g + SC2; incl. the distinct primary/alt id assertion). *Checkpoint: `feat(linkedin-studio): SB-S2 consolidation engine — proposeDiff classification [skip-docs]`. On-failure: revert `scripts/brain/src/consolidate.ts` + its test.* +3. TDD `applyDiff` + round-trip + idempotency (SC3/SC4). *Circuit-breaker: if the parse/serialize round-trip can't be made green, the id/shape is wrong — halt.* *Checkpoint: fold into step-2 commit or `feat(...): SB-S2 applyDiff + round-trip [skip-docs]`.* +4. TDD sidecar IO (`readConsolidationState`/`writeConsolidationState`) — fold into the consolidate-cli temp-dir tests. *Checkpoint with step 5.* +5. Extend `cli.ts` `consolidate` (gather/propose/apply); TDD `consolidate-cli.test.ts` (SC5) incl. candidate validation (shape + no-newline), no-confirm refusal, `--gather` body-read, `init`/`ingest`/`published` regression. *Checkpoint: `feat(linkedin-studio): SB-S2 brain consolidate CLI [skip-docs]`. On-failure: revert `cli.ts` + the cli test.* +6. Edit `session-start.mjs` (import `readdirSync`; UNCONDITIONAL scaffold-ensure near the migration block; consolidation-due nudge in the reminders block, `\\n` idiom); author `session-start-brain-consolidation.test.mjs` **via Bash-heredoc** (pathguard-safe); run `node --test hooks/scripts/__tests__/` (the WHOLE hook suite, since the gate doesn't — SC6 + no hook regression). If `hooks.json` shape unchanged (only script body edited) no recompile; else `compile-hooks.py` + verify `--check`. *Checkpoint: `feat(linkedin-studio): SB-S2 session-start scaffold-ensure + consolidation nudge [skip-docs]`. On-failure: `git checkout hooks/scripts/session-start.mjs` (do NOT leave the hook emitting invalid JSON) + halt.* +7. Write `docs/second-brain/consolidation-loop.md` (CLI usage, engine rules, the candidate-file schema as the session↔engine contract, the operator-gate, honest limits incl. no-reader-until-S3 + the loop's value-depends-on-extraction caveat). *Checkpoint: `docs(...)`.* +8. Bump `scripts/test-runner.sh` `BRAIN_TESTS_FLOOR` to the real `tests N` (assert the brain suite RAN — `BR_TESTS` non-empty, not warn-skipped). **`ASSERT_BASELINE_FLOOR` unchanged (+0): SB-S2 adds NO new test-runner section** (the hook test runs separately, the new TS tests run inside the existing brain `npm test` which contributes one unchanged `pass()` line) — confirm by reading the printed `TOTAL_CHECKS` is unchanged. Run the FULL gate green AND the hook suite green. *Circuit-breaker: no red gate committed.* *Checkpoint: `chore(...): SB-S2 gate brain floor`.* +9. Flip `architecture.md` SB-S2 row → landed (exact edit: the `| **SB-S2 — Evolution loop** |` row → add the `✅ *landed*` marker, matching SB-S0/S1 row style); persist brief + plan. **Commit; confirm before push (PUBLIC `open/`), window; STATE.md gitignored.** Conditional 0.5.2 release per Q2 (tag + catalog via `release-plugin.mjs`). + +## 7. Scope fence (echo — SB-S2 does NOT) + +No journal-capture · no new agent (session extracts) · no `profile.md` reader (S3) · no supersede (keep-both only; S3) · no cross-silo id threading (S3) · no `content-history.md` retirement · no AI at session-start (deterministic nudge only) · no auto-apply (only `--apply --confirm` writes profile.md) · no new hook `.mjs` (edit existing session-start.mjs) · no new seam function · no connector (S4) · no GUI. + +## 8. Known limits / deferred (honest) + +- **The loop's VALUE depends on the session's candidate extraction** — the engine guarantees only the mechanics (threshold/conflict/decay/provenance-gate). Garbage candidates → a garbage diff (but the operator gate + the candidate-schema validation catch shape errors, not quality). Documented in `consolidation-loop.md`. +- **profile.md has no reader until S3** — S2 evolves an artifact nothing consumes yet; the value is deferred compounding. Stated honestly. +- **Evidence inflates only on genuinely-new candidates** — `--gather` is delta-gated (published since `last_run`), so a re-run after `--apply` surfaces nothing; the engine itself would bump again given the same candidate, but the loop never re-feeds one. Documented. +- **No supersede / no stale-fact demotion in S2** — conflicts keep both; stale facts are flagged, never auto-removed (operator/ S3). +- **Session-start nudge is new-published-count only** — no per-fact staleness at session-start (no zero-dep profile parser). + +## 9. Risks + +- **Duplicate-id corruption (was plan-critic B1) — CLOSED:** primary ids (`mintEntityId(key)`) and conflict-alt ids (`mintContentId('observed-alt:'+key+'::'+value+'::'+date)`) are byte-distinct, so no two facts share an id; the bump/promote target is unambiguous and `parseProfile`/`serializeProfile` stays well-formed. Pinned by SC1e (distinct ids) + SC3 (round-trip) + SC4 (idempotency). +- **`--gather` data source (was plan-critic B2) — CLOSED:** gather reads the published files directly (`parsePublishedRecord` → body), not `listPublished()` (which lacks body/`captured_at`); filtered by `published_date > last_run`. +- **Hook tests not gate-run (was plan-critic B3):** `scripts/test-runner.sh` runs NO hook test, so SC6 is verified by an explicit `node --test hooks/scripts/__tests__/` step (the whole suite), NOT the structure gate. SC7 reworded; no false "hook tests pass at the gate" claim. **Flagged (not done):** wiring the hook suite into test-runner.sh is a separate hardening opportunity. +- **Engine rule ambiguity (conflict vs bump):** deterministic value-equality decides bump-vs-conflict; keep-both is the safe default; each rule pinned by its own SC1x test before the CLI wires it. +- **Sidecar reachability (was the brief-review blocker) — CLOSED:** both `--apply` (dataRoot) and session-start (getDataRoot) resolve the SAME data root → the sidecar closes the loop; SC6 sets HOME + data root to separate temp dirs (the trends-test pattern) to prove it. +- **Zero-dep hook discipline:** no tsx, no profile parse, bounded readdir, `\\n` idiom; scaffold-ensure runs unconditionally (fresh-install path); SC6 asserts valid JSON output + the nudge rendering. +- **Gate floors:** BRAIN floor = actual post-impl count (assert suite RAN, not warn-skipped — step 1 fail-loud on missing tsx); `ASSERT_BASELINE_FLOOR` +0 (no new test-runner section). Confirm `TOTAL_CHECKS` unchanged at step 8. +- **Pathguard:** session-start.mjs is an existing file → Edit (safe). The new hook TEST `.mjs` under `hooks/scripts/__tests__/` is authored via **Bash-heredoc** — safe whether or not the active Category-6 guard matches `__tests__/` (the verified llm-security `.mjs` guard matches `hooks/scripts/.*\.mjs`; heredoc sidesteps it regardless).