linkedin-studio/docs/second-brain/plan-sb-s2.md

Implementation Plan — SB-S2 (Evolution loop)

Status: review-hardened (plan-critic REPLAN 58→folded; scope-guardian ALIGNED). Ready for operator approval / "go". Brief: brief-sb-s2.md (review-hardened). Arc: architecture.md. Predecessors: SB-S0/S1 (scripts/brain/, v0.5.1). Review delta (2026-06-23): B1 conflict id → distinct (primary mintEntityId(key) + alt mintContentId('observed-alt:'+key+'::'+value+'::'+date), no duplicate ids); B2 --gather reads published files directly for bodies (not listPublished), tributary reads CUT; B3 hook tests run SEPARATELY (node --test hooks/scripts/__tests__/) as an explicit step — NOT via test-runner.sh (which runs no hook test) → SC7 reworded, assertion floor +0; M4 new hook-test .mjs authored via Bash-heredoc (pathguard-safe under either guard); M5 scaffold-ensure runs UNCONDITIONALLY (fresh-install path); M6 \\n escaped-newline idiom; M8 per-step commit msg + revert target; minors: single-line value validation, static-decay-exempt, tsx fail-loud. Scope: the deterministic consolidation engine + brain consolidate CLI (operator-invoked, operator-gated) + the session-start consolidation-due nudge + scaffold-ensure. No journal-capture, no new agent, no profile.md reader, no supersede, no cross-silo threading, no auto-apply, no AI at session-start (brief §7).

---

1. What SB-S2 delivers (recap)

A new PURE TS engine scripts/brain/src/consolidate.ts (proposeDiff + applyDiff over the SB-S0 two-layer ProfileDoc), a brain consolidate CLI (--gather / --propose / --apply --confirm) that closes the loop operator-gated, a brain data-root sidecar brain/consolidation-state.json for the last-run timestamp, and a zero-dep consolidation-due nudge + scaffold-ensure edit to hooks/scripts/session-start.mjs. Count-neutral (no agent/command/ref/skill). All TDD; gate stays green; floors rise.

2. Package additions (scripts/brain/) — extend the existing package

scripts/brain/
  src/
    consolidate.ts     # NEW — Candidate/ProfileDiff types + proposeDiff + applyDiff (pure) + consolidation-state IO
    cli.ts             # EDIT — add `consolidate` dispatch (--gather/--propose/--apply)
    (types.ts, id.ts, profile.ts, ingest.ts, scaffold.ts, dataRoot.ts unchanged)
  tests/
    consolidate.test.ts   # NEW — engine: per-rule (SC1a–g) + immutability + round-trip + idempotency
    consolidate-cli.test.ts  # NEW — CLI gating + candidate-validation + sidecar (SC5)
hooks/scripts/
  session-start.mjs    # EDIT (existing → pathguard-safe) — scaffold-ensure + consolidation-due nudge
  __tests__/
    session-start-brain-consolidation.test.mjs  # NEW — SC6 (mirror of session-start-trends-staleness.test.mjs)

No new dep (engine is pure TS; the hook stays zero-dep). The brain test script already globs tests/*.test.ts.

3. Module designs

3.1 consolidate.ts — types

import { mintEntityId, mintContentId, slugify } from "./id.js";
import { SCHEMA_VERSION } from "./types.js";
import type { ProfileDoc, ProfileFact, Provenance } from "./types.js";

export interface Candidate {
  key: string;            // the fact's stable label/topic/concept
  value: string;          // single-line (no embedded newline/CR — profile grammar)
  provenance: Provenance; // published | human | ai-draft
  source: string;         // e.g. "published:<id>" | "manual"
  observed_date: string;  // YYYY-MM-DD
}
export interface ProfileDiff {
  additions: ProfileFact[];                                   // new dynamic facts (primary + conflict-alt)
  evidenceBumps: { id: string; newCount: number; last_seen: string }[];
  promotions: { id: string }[];                               // dynamic→static (post-bump count ≥ N)
  conflicts: { primaryId: string; primaryValue: string; altId: string }[]; // both retained, distinct ids
  staleFlags: { id: string; last_seen: string; daysStale: number }[];
}
export interface ConsolidateOpts { promoteThreshold?: number; decayDays?: number } // defaults 3 / 90
const OBSERVED_KIND = "observed";

3.2 consolidate.ts — proposeDiff (pure; the classification heart) — id model fixes plan-critic B1

Id model (the B1 fix — no duplicate ids):

• A concept's primary fact id = mintEntityId({ kind: OBSERVED_KIND, key }) (key-only → one primary per concept).
• A conflict alt fact id = mintContentId(\observed-alt:${key}::${value}::${observed_date}`)(SB-S1's verbatim sha256[:12] — byte-distinct, so primary and alt never collide and two alts for the same concept differ by value/date). Matching always starts from the *candidate's*key`, so we never need to recover a key from an existing fact.

proposeDiff({ current, candidates, today, opts }): ProfileDiff

• N = opts.promoteThreshold ?? 3, DECAY = opts.decayDays ?? 90.
• Build a byId lookup over current.dynamic ∪ current.static. (Folded profile-field static seeds use a DIFFERENT kind → their ids never collide with observed ids → immutable, SC1g.)
• For each candidate:
  • provenance === 'ai-draft' → skip entirely (SC1b).
  • primaryId = mintEntityId({kind:OBSERVED_KIND, key}); prev = byId.get(primaryId).
  • no prev → additions gets a new dynamic ProfileFact (id:primaryId, evidence_count:1, first_seen:observed_date, last_seen:today, status:'active', provenance) (SC1a).
  • prev, prev.value === candidate.value → evidenceBumps (newCount = prev.evidence_count+1, last_seen:today); if prev is in dynamic AND newCount ≥ N → also promotions[{id:primaryId}] (SC1c/SC1d).
  • prev, prev.value !== candidate.value → CONFLICT (SC1e): build an alt fact {id: mintContentId(\observed-alt:${key}::${candidate.value}::${observed_date}`), value:candidate.value, dynamic, evidence_count:1, …}, push it to additions, record conflicts[{primaryId, primaryValue: prev.value, altId}], and leave prev` UNTOUCHED (no bump). No supersede (S3). (If the alt id already exists from a prior run → treat as the same-value bump path on the alt id, so re-running is idempotent.)
• After the candidate pass: staleFlags = every current.dynamic fact whose daysSince(last_seen) > DECAY (SC1f). Static facts are decay-exempt by design (promoted = stable). Informational only — no mutation.
• Purity: never mutate current/candidates (build new objects); SC2 asserts inputs structurally unchanged.

3.3 consolidate.ts — applyDiff (pure) + sidecar IO

• applyDiff(current, diff): ProfileDoc — returns a NEW doc: append additions (which already includes both new primary facts AND conflict alt facts) to dynamic; apply evidenceBumps (update count+last_seen on the fact with the matching id); move promotions facts from dynamic→static. conflicts[] is informational (the alt fact is already in additions); staleFlags cause NO mutation. Id-uniqueness invariant: because primary ids (mintEntityId(key)) and alt ids (mintContentId(...)) are byte-distinct, no two facts share an id — so the bump/promote target is unambiguous and the doc stays well-formed. Output round-trips through parseProfile/serializeProfile (SC3).
• Sidecar IO (the only impure bit, kept here for cohesion): readConsolidationState(): {last_run: string|null} + writeConsolidationState(date) over dataRoot('brain/consolidation-state.json') ({ "last_run": "YYYY-MM-DD" }). Tiny JSON; absent → {last_run:null}.

3.4 cli.ts — consolidate dispatch (extend the SB-S1 router)

Add command === "consolidate" with a --gather / --propose / --apply mode (the parseFlags idiom already present):

• --gather [--json] — read-only. Reads the published gold corpus directly (readdir(dataRoot('ingest/published')) → parsePublishedRecord each — listPublished() returns no body/captured_at, so it can't feed extraction; fixes plan-critic B2), filters to records with published_date > readConsolidationState().last_run (null → all), and prints a digest of {id, published_date, body} per new record + the current brain/profile.md facts (parsed), for the invoking session to turn into a Candidate[] JSON. Tributary newest-timestamp reads are CUT (informational-only, underspecified — the published corpus is the S2 signal; plan-critic M12). Writes nothing.
• --propose --candidates <file.json> — read + validate each candidate against the Candidate shape: every item has key/value/provenance∈{human,published,ai-draft}/source/observed_date, AND key/value contain no newline or carriage-return (the profile grammar is single-line — a \n would corrupt serialization; plan-critic M9). Any violation → non-zero exit, no write (SC5). proposeDiff over the current profile. Write brain/pending-diff.json (the typed ProfileDiff) AND brain/pending-diff.md (operator-readable: sections Additions / Evidence-bumps / Promotions / ⚠ Conflicts / Stale, neutral framing). Print both paths. Does NOT touch profile.md.
• --apply --diff <path> --confirm — require --confirm (else refuse). Read the JSON diff, parseProfile(brain/profile.md), applyDiff, write brain/profile.md (serializeProfile), then writeConsolidationState(today()). The ONLY path that writes the profile.
• Usage text + exit 2 on misuse. init/ingest/published branches unchanged (regression-guarded).

3.5 hooks/scripts/session-start.mjs — scaffold-ensure + consolidation-due nudge (Edit, zero-dep)

• Add readdirSync to the existing node:fs import (line 5).
• Helpers (twins of trendsNewestCapture, lines 38–52):
  • brainLastRun(path) — JSON.parse(readFileSync) of consolidation-state.json → last_run or null (try/catch → null).
  • countPublished(dir) — existsSync(dir) ? readdirSync(dir).filter(f=>f.endsWith('.md')).length : 0 (no parse — cost-bounded).
• Scaffold-ensure — UNCONDITIONAL (fixes plan-critic M5: the reminders/context block runs only inside if (existsSync(STATE_FILE)), so a fresh install (else branch ~line 432) would never scaffold/nudge). Place it next to the M0 migration block (~line 73), BEFORE the state-file branch, so it always runs: for (const d of ['brain/journal','ingest/inbox','ingest/published']) mkdirSync(join(getDataRoot(''),d),{recursive:true}) (idempotent, zero-dep; confirm getDataRoot('') returns the bare data root — step 1). The optional "run brain init to seed your profile" nudge is appended to context directly (not reminders) right after scaffold-ensure when !existsSync(join(getDataRoot('brain'),'profile.md')), so it survives the no-state-file path.
• Consolidation-due nudge (inside the reminders block, beside the trends nudge ~line 342 — an established user with published content has a state file, so the reminders path is correct here). Use the \\n escaped-newline idiom (the file un-escapes once via context.replace(/\\n/g,'\n') at line 479 — a raw \n would mis-render; plan-critic M6):

  const published = countPublished(join(getDataRoot('ingest'), 'published'));
  const lastRun = brainLastRun(join(getDataRoot('brain'), 'consolidation-state.json'));
  const daysSinceRun = daysSince(lastRun);
  if (published > 0 && (lastRun === null || (daysSinceRun !== null && daysSinceRun >= 7))) {
    const since = lastRun === null ? 'never run' : `${daysSinceRun}d ago`;
    reminders += `- ${published} published post(s) captured, last brain consolidation ${since}. Run \\\`brain consolidate\\\` to evolve your profile.\\n`;
  }
  

  Null-safe + fires only when published records exist (a never-used brain never nags — the trend-nudge discipline). No profile.md parse.

4. Files created / edited

Created: scripts/brain/src/consolidate.ts · scripts/brain/tests/{consolidate,consolidate-cli}.test.ts · hooks/scripts/__tests__/session-start-brain-consolidation.test.mjs · docs/second-brain/consolidation-loop.md.

Edited: scripts/brain/src/cli.ts (+consolidate) · hooks/scripts/session-start.mjs (scaffold-ensure + nudge + readdirSync import) · scripts/test-runner.sh (BRAIN floor + assertion floor) · docs/second-brain/architecture.md (flip SB-S2 row).

Conditional (Q2 = bump to 0.5.2): plugin.json + README badge + CLAUDE.md header + CHANGELOG; polyrepo release via release-plugin.mjs (tag + catalog), pushed per window + confirm.

5. TDD test plan (failing-first) → success criteria

Test	Asserts	SC
consolidate.test.ts	add / reject-ai-draft / evidence-bump / promote-at-3 / conflict-keep-both with DISTINCT primary+alt ids, old fact untouched / decay-flag (dynamic only) / static-decay-exempt / folded-immutable; input immutability; applyDiff∘proposeDiff round-trips via parse/serialize; idempotent (re-run → bumps not dup facts, alt-id stable)	SC1a–g, SC2, SC3, SC4
consolidate-cli.test.ts	temp-dir: --gather reads published bodies (filtered by published_date>last_run) + leaves profile.md byte-unchanged; --propose leaves profile.md unchanged, writes pending-diff.{md,json}, REJECTS a malformed candidate file AND a candidate whose value contains \n (non-zero, no write); --apply --confirm writes profile.md + consolidation-state.json; --apply without --confirm refuses; init/ingest/published still work	SC5
session-start-brain-consolidation.test.mjs (authored via Bash-heredoc — pathguard-safe)	subprocess + isolated HOME/LINKEDIN_STUDIO_DATA (trends-staleness harness): nudge fires (rendered on its own line) when published records exist + last_run stale/absent; silent when brain absent / no published; scaffold-ensure runs on the fresh-install (no-state-file) path; hook still emits valid JSON	SC6
(separate) node --test hooks/scripts/__tests__/	the new SC6 test + all existing hook tests pass — run as an explicit step, NOT via test-runner.sh (which runs no hook test; plan-critic B3)	SC6
(gate) scripts/test-runner.sh	green; floors hold; brain ≥ new floor; counts unchanged; compile-hooks.py --check clean if hooks.json regenerated. (No new test-runner section → assertion floor +0.)	SC7

Each test written before its module (iron law). Circuit-breakers: halt if the engine round-trip (SC3) can't be made green, or if the full gate (step 8) is red.

6. Step sequence (each step: TDD red→green; Checkpoint = commit with the named message; On-failure → revert the named target + halt + report)

1. Verify-first: re-confirm proposeDiff inputs against types.ts (ProfileFact/ProfileDoc/Provenance) + mintEntityId/mintContentId/slugify signatures + daysSince/getDataRoot('') behavior in session-start.mjs (does the empty subdir return the bare data root?). tsx fail-loud: assert scripts/brain/node_modules/.bin/tsx present (the gate brain section warn-skips if absent — for this slice that would let the gate go green with SB-S2's tests never run; npm install and FAIL if still absent; plan-critic M11). On-failure: stop, report — the slice cannot be verified.
2. TDD consolidate.ts types + proposeDiff per-rule (consolidate.test.ts, SC1a–g + SC2; incl. the distinct primary/alt id assertion). Checkpoint: feat(linkedin-studio): SB-S2 consolidation engine — proposeDiff classification [skip-docs]. On-failure: revert scripts/brain/src/consolidate.ts + its test.
3. TDD applyDiff + round-trip + idempotency (SC3/SC4). Circuit-breaker: if the parse/serialize round-trip can't be made green, the id/shape is wrong — halt. Checkpoint: fold into step-2 commit or feat(...): SB-S2 applyDiff + round-trip [skip-docs].
4. TDD sidecar IO (readConsolidationState/writeConsolidationState) — fold into the consolidate-cli temp-dir tests. Checkpoint with step 5.
5. Extend cli.ts consolidate (gather/propose/apply); TDD consolidate-cli.test.ts (SC5) incl. candidate validation (shape + no-newline), no-confirm refusal, --gather body-read, init/ingest/published regression. Checkpoint: feat(linkedin-studio): SB-S2 brain consolidate CLI [skip-docs]. On-failure: revert cli.ts + the cli test.
6. Edit session-start.mjs (import readdirSync; UNCONDITIONAL scaffold-ensure near the migration block; consolidation-due nudge in the reminders block, \\n idiom); author session-start-brain-consolidation.test.mjs via Bash-heredoc (pathguard-safe); run node --test hooks/scripts/__tests__/ (the WHOLE hook suite, since the gate doesn't — SC6 + no hook regression). If hooks.json shape unchanged (only script body edited) no recompile; else compile-hooks.py + verify --check. Checkpoint: feat(linkedin-studio): SB-S2 session-start scaffold-ensure + consolidation nudge [skip-docs]. On-failure: git checkout hooks/scripts/session-start.mjs (do NOT leave the hook emitting invalid JSON) + halt.
7. Write docs/second-brain/consolidation-loop.md (CLI usage, engine rules, the candidate-file schema as the session↔engine contract, the operator-gate, honest limits incl. no-reader-until-S3 + the loop's value-depends-on-extraction caveat). Checkpoint: docs(...).
8. Bump scripts/test-runner.sh BRAIN_TESTS_FLOOR to the real tests N (assert the brain suite RAN — BR_TESTS non-empty, not warn-skipped). ASSERT_BASELINE_FLOOR unchanged (+0): SB-S2 adds NO new test-runner section (the hook test runs separately, the new TS tests run inside the existing brain npm test which contributes one unchanged pass() line) — confirm by reading the printed TOTAL_CHECKS is unchanged. Run the FULL gate green AND the hook suite green. Circuit-breaker: no red gate committed. Checkpoint: chore(...): SB-S2 gate brain floor.
9. Flip architecture.md SB-S2 row → landed (exact edit: the | **SB-S2 — Evolution loop** | row → add the ✅ *landed* marker, matching SB-S0/S1 row style); persist brief + plan. Commit; confirm before push (PUBLIC open/), window; STATE.md gitignored. Conditional 0.5.2 release per Q2 (tag + catalog via release-plugin.mjs).

7. Scope fence (echo — SB-S2 does NOT)

No journal-capture · no new agent (session extracts) · no profile.md reader (S3) · no supersede (keep-both only; S3) · no cross-silo id threading (S3) · no content-history.md retirement · no AI at session-start (deterministic nudge only) · no auto-apply (only --apply --confirm writes profile.md) · no new hook .mjs (edit existing session-start.mjs) · no new seam function · no connector (S4) · no GUI.

8. Known limits / deferred (honest)

• The loop's VALUE depends on the session's candidate extraction — the engine guarantees only the mechanics (threshold/conflict/decay/provenance-gate). Garbage candidates → a garbage diff (but the operator gate + the candidate-schema validation catch shape errors, not quality). Documented in consolidation-loop.md.
• profile.md has no reader until S3 — S2 evolves an artifact nothing consumes yet; the value is deferred compounding. Stated honestly.
• Evidence inflates only on genuinely-new candidates — --gather is delta-gated (published since last_run), so a re-run after --apply surfaces nothing; the engine itself would bump again given the same candidate, but the loop never re-feeds one. Documented.
• No supersede / no stale-fact demotion in S2 — conflicts keep both; stale facts are flagged, never auto-removed (operator/ S3).
• Session-start nudge is new-published-count only — no per-fact staleness at session-start (no zero-dep profile parser).

9. Risks

• Duplicate-id corruption (was plan-critic B1) — CLOSED: primary ids (mintEntityId(key)) and conflict-alt ids (mintContentId('observed-alt:'+key+'::'+value+'::'+date)) are byte-distinct, so no two facts share an id; the bump/promote target is unambiguous and parseProfile/serializeProfile stays well-formed. Pinned by SC1e (distinct ids) + SC3 (round-trip) + SC4 (idempotency).
• --gather data source (was plan-critic B2) — CLOSED: gather reads the published files directly (parsePublishedRecord → body), not listPublished() (which lacks body/captured_at); filtered by published_date > last_run.
• Hook tests not gate-run (was plan-critic B3): scripts/test-runner.sh runs NO hook test, so SC6 is verified by an explicit node --test hooks/scripts/__tests__/ step (the whole suite), NOT the structure gate. SC7 reworded; no false "hook tests pass at the gate" claim. Flagged (not done): wiring the hook suite into test-runner.sh is a separate hardening opportunity.
• Engine rule ambiguity (conflict vs bump): deterministic value-equality decides bump-vs-conflict; keep-both is the safe default; each rule pinned by its own SC1x test before the CLI wires it.
• Sidecar reachability (was the brief-review blocker) — CLOSED: both --apply (dataRoot) and session-start (getDataRoot) resolve the SAME data root → the sidecar closes the loop; SC6 sets HOME + data root to separate temp dirs (the trends-test pattern) to prove it.
• Zero-dep hook discipline: no tsx, no profile parse, bounded readdir, \\n idiom; scaffold-ensure runs unconditionally (fresh-install path); SC6 asserts valid JSON output + the nudge rendering.
• Gate floors: BRAIN floor = actual post-impl count (assert suite RAN, not warn-skipped — step 1 fail-loud on missing tsx); ASSERT_BASELINE_FLOOR +0 (no new test-runner section). Confirm TOTAL_CHECKS unchanged at step 8.
• Pathguard: session-start.mjs is an existing file → Edit (safe). The new hook TEST .mjs under hooks/scripts/__tests__/ is authored via Bash-heredoc — safe whether or not the active Category-6 guard matches __tests__/ (the verified llm-security .mjs guard matches hooks/scripts/.*\.mjs; heredoc sidesteps it regardless).