linkedin-studio/docs/second-brain/plan-sb-s2.md
Kjell Tore Guttormsen 204665e90b docs(linkedin-studio): second-brain SB-S2 brief + plan (review-hardened)
Slice SB-S2 (Evolution loop): operator-invoked consolidation engine → profile
diff (threshold-promotion N=3, conflict keep-both with distinct ids, decay-flag
90d, provenance-gated) + brain/consolidation-state.json sidecar + zero-dep
session-start consolidation-due nudge + scaffold-ensure. Light-Voyage-hardened:
brief-reviewer (REVISE→folded: state-file→sidecar fix), plan-critic (REPLAN
58→folded: distinct-id model, gather reads bodies, hook-tests-run-separately),
scope-guardian (ALIGNED). Operator scope: journal deferred · deterministic CLI
no-agent · motor-only no-reader. Defaults N=3/90d, version → 0.5.2.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RigJBiRFNtFZKCz21qNbQ4
2026-06-23 16:57:17 +02:00

160 lines
22 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Implementation Plan — SB-S2 (Evolution loop)
> **Status:** review-hardened (`plan-critic` REPLAN 58→folded; `scope-guardian` ALIGNED). Ready for operator approval / "go".
> **Brief:** `brief-sb-s2.md` (review-hardened). **Arc:** `architecture.md`. **Predecessors:** SB-S0/S1 (`scripts/brain/`, v0.5.1).
> **Review delta (2026-06-23):** B1 conflict id → distinct (primary `mintEntityId(key)` + alt `mintContentId('observed-alt:'+key+'::'+value+'::'+date)`, no duplicate ids); B2 `--gather` reads published files directly for bodies (not `listPublished`), tributary reads CUT; B3 hook tests run SEPARATELY (`node --test hooks/scripts/__tests__/`) as an explicit step — NOT via test-runner.sh (which runs no hook test) → SC7 reworded, assertion floor +0; M4 new hook-test `.mjs` authored via Bash-heredoc (pathguard-safe under either guard); M5 scaffold-ensure runs UNCONDITIONALLY (fresh-install path); M6 `\\n` escaped-newline idiom; M8 per-step commit msg + revert target; minors: single-line value validation, static-decay-exempt, tsx fail-loud.
> **Scope:** the deterministic consolidation engine + `brain consolidate` CLI (operator-invoked, operator-gated) + the session-start consolidation-due nudge + scaffold-ensure. **No journal-capture, no new agent, no profile.md reader, no supersede, no cross-silo threading, no auto-apply, no AI at session-start** (brief §7).
---
## 1. What SB-S2 delivers (recap)
A new PURE TS engine `scripts/brain/src/consolidate.ts` (`proposeDiff` + `applyDiff` over the SB-S0 two-layer `ProfileDoc`), a `brain consolidate` CLI (`--gather` / `--propose` / `--apply --confirm`) that closes the loop operator-gated, a brain data-root sidecar `brain/consolidation-state.json` for the last-run timestamp, and a zero-dep **consolidation-due nudge** + **scaffold-ensure** edit to `hooks/scripts/session-start.mjs`. Count-neutral (no agent/command/ref/skill). All TDD; gate stays green; floors rise.
## 2. Package additions (`scripts/brain/`) — extend the existing package
```
scripts/brain/
src/
consolidate.ts # NEW — Candidate/ProfileDiff types + proposeDiff + applyDiff (pure) + consolidation-state IO
cli.ts # EDIT — add `consolidate` dispatch (--gather/--propose/--apply)
(types.ts, id.ts, profile.ts, ingest.ts, scaffold.ts, dataRoot.ts unchanged)
tests/
consolidate.test.ts # NEW — engine: per-rule (SC1ag) + immutability + round-trip + idempotency
consolidate-cli.test.ts # NEW — CLI gating + candidate-validation + sidecar (SC5)
hooks/scripts/
session-start.mjs # EDIT (existing → pathguard-safe) — scaffold-ensure + consolidation-due nudge
__tests__/
session-start-brain-consolidation.test.mjs # NEW — SC6 (mirror of session-start-trends-staleness.test.mjs)
```
No new dep (engine is pure TS; the hook stays zero-dep). The brain `test` script already globs `tests/*.test.ts`.
## 3. Module designs
### 3.1 `consolidate.ts` — types
```ts
import { mintEntityId, mintContentId, slugify } from "./id.js";
import { SCHEMA_VERSION } from "./types.js";
import type { ProfileDoc, ProfileFact, Provenance } from "./types.js";
export interface Candidate {
key: string; // the fact's stable label/topic/concept
value: string; // single-line (no embedded newline/CR — profile grammar)
provenance: Provenance; // published | human | ai-draft
source: string; // e.g. "published:<id>" | "manual"
observed_date: string; // YYYY-MM-DD
}
export interface ProfileDiff {
additions: ProfileFact[]; // new dynamic facts (primary + conflict-alt)
evidenceBumps: { id: string; newCount: number; last_seen: string }[];
promotions: { id: string }[]; // dynamic→static (post-bump count ≥ N)
conflicts: { primaryId: string; primaryValue: string; altId: string }[]; // both retained, distinct ids
staleFlags: { id: string; last_seen: string; daysStale: number }[];
}
export interface ConsolidateOpts { promoteThreshold?: number; decayDays?: number } // defaults 3 / 90
const OBSERVED_KIND = "observed";
```
### 3.2 `consolidate.ts` — `proposeDiff` (pure; the classification heart) — id model fixes plan-critic B1
**Id model (the B1 fix — no duplicate ids):**
- A concept's **primary fact** id = `mintEntityId({ kind: OBSERVED_KIND, key })` (key-only → one primary per concept).
- A conflict **alt fact** id = `mintContentId(\`observed-alt:${key}::${value}::${observed_date}\`)` (SB-S1's verbatim sha256[:12] — byte-distinct, so primary and alt never collide and two alts for the same concept differ by value/date). Matching always starts from the *candidate's* `key`, so we never need to recover a key from an existing fact.
```ts
proposeDiff({ current, candidates, today, opts }): ProfileDiff
```
- `N = opts.promoteThreshold ?? 3`, `DECAY = opts.decayDays ?? 90`.
- Build a `byId` lookup over `current.dynamic current.static`. (Folded `profile-field` static seeds use a DIFFERENT kind → their ids never collide with `observed` ids → immutable, SC1g.)
- For each candidate:
- `provenance === 'ai-draft'` → skip entirely (SC1b).
- `primaryId = mintEntityId({kind:OBSERVED_KIND, key})`; `prev = byId.get(primaryId)`.
- **no `prev`** → `additions` gets a new dynamic `ProfileFact` (`id:primaryId`, `evidence_count:1`, `first_seen:observed_date`, `last_seen:today`, `status:'active'`, `provenance`) (SC1a).
- **`prev`, `prev.value === candidate.value`** → `evidenceBumps` (`newCount = prev.evidence_count+1`, `last_seen:today`); if `prev` is in `dynamic` AND `newCount ≥ N` → also `promotions[{id:primaryId}]` (SC1c/SC1d).
- **`prev`, `prev.value !== candidate.value`** → CONFLICT (SC1e): build an alt fact `{id: mintContentId(\`observed-alt:${key}::${candidate.value}::${observed_date}\`), value:candidate.value, dynamic, evidence_count:1, …}`, push it to `additions`, record `conflicts[{primaryId, primaryValue: prev.value, altId}]`, and leave `prev` UNTOUCHED (no bump). **No supersede** (S3). (If the alt id already exists from a prior run → treat as the same-value bump path on the alt id, so re-running is idempotent.)
- After the candidate pass: `staleFlags` = every **`current.dynamic`** fact whose `daysSince(last_seen) > DECAY` (SC1f). **Static facts are decay-exempt by design** (promoted = stable). Informational only — no mutation.
- **Purity:** never mutate `current`/`candidates` (build new objects); SC2 asserts inputs structurally unchanged.
### 3.3 `consolidate.ts` — `applyDiff` (pure) + sidecar IO
- `applyDiff(current, diff): ProfileDoc` — returns a NEW doc: append `additions` (which already includes both new primary facts AND conflict alt facts) to `dynamic`; apply `evidenceBumps` (update count+last_seen on the fact with the matching id); move `promotions` facts from `dynamic``static`. `conflicts[]` is informational (the alt fact is already in `additions`); `staleFlags` cause NO mutation. **Id-uniqueness invariant:** because primary ids (`mintEntityId(key)`) and alt ids (`mintContentId(...)`) are byte-distinct, no two facts share an id — so the bump/promote target is unambiguous and the doc stays well-formed. Output round-trips through `parseProfile`/`serializeProfile` (SC3).
- **Sidecar IO** (the only impure bit, kept here for cohesion): `readConsolidationState(): {last_run: string|null}` + `writeConsolidationState(date)` over `dataRoot('brain/consolidation-state.json')` (`{ "last_run": "YYYY-MM-DD" }`). Tiny JSON; absent → `{last_run:null}`.
### 3.4 `cli.ts` — `consolidate` dispatch (extend the SB-S1 router)
Add `command === "consolidate"` with a `--gather` / `--propose` / `--apply` mode (the parseFlags idiom already present):
- **`--gather [--json]`** — read-only. Reads the published gold corpus **directly** (`readdir(dataRoot('ingest/published'))``parsePublishedRecord` each — `listPublished()` returns no body/`captured_at`, so it can't feed extraction; fixes plan-critic B2), filters to records with `published_date > readConsolidationState().last_run` (null → all), and prints a digest of `{id, published_date, body}` per new record + the current `brain/profile.md` facts (parsed), for the **invoking session** to turn into a `Candidate[]` JSON. **Tributary newest-timestamp reads are CUT** (informational-only, underspecified — the published corpus is the S2 signal; plan-critic M12). Writes nothing.
- **`--propose --candidates <file.json>`** — read + **validate** each candidate against the `Candidate` shape: every item has key/value/provenance∈{human,published,ai-draft}/source/observed_date, AND **`key`/`value` contain no newline or carriage-return** (the profile grammar is single-line — a `\n` would corrupt serialization; plan-critic M9). Any violation → non-zero exit, no write (SC5). `proposeDiff` over the current profile. Write `brain/pending-diff.json` (the typed `ProfileDiff`) AND `brain/pending-diff.md` (operator-readable: sections Additions / Evidence-bumps / Promotions / ⚠ Conflicts / Stale, neutral framing). Print both paths. Does NOT touch `profile.md`.
- **`--apply --diff <path> --confirm`** — require `--confirm` (else refuse). Read the JSON diff, `parseProfile(brain/profile.md)`, `applyDiff`, write `brain/profile.md` (`serializeProfile`), then `writeConsolidationState(today())`. The ONLY path that writes the profile.
- Usage text + exit 2 on misuse. `init`/`ingest`/`published` branches unchanged (regression-guarded).
### 3.5 `hooks/scripts/session-start.mjs` — scaffold-ensure + consolidation-due nudge (Edit, zero-dep)
- Add `readdirSync` to the existing `node:fs` import (line 5).
- **Helpers (twins of `trendsNewestCapture`, lines 3852):**
- `brainLastRun(path)``JSON.parse(readFileSync)` of `consolidation-state.json``last_run` or null (try/catch → null).
- `countPublished(dir)``existsSync(dir) ? readdirSync(dir).filter(f=>f.endsWith('.md')).length : 0` (no parse — cost-bounded).
- **Scaffold-ensure — UNCONDITIONAL** (fixes plan-critic M5: the `reminders`/context block runs only inside `if (existsSync(STATE_FILE))`, so a fresh install (else branch ~line 432) would never scaffold/nudge). Place it next to the M0 migration block (~line 73), BEFORE the state-file branch, so it always runs: `for (const d of ['brain/journal','ingest/inbox','ingest/published']) mkdirSync(join(getDataRoot(''),d),{recursive:true})` (idempotent, zero-dep; confirm `getDataRoot('')` returns the bare data root — step 1). The optional "run `brain init` to seed your profile" nudge is appended to **`context` directly** (not `reminders`) right after scaffold-ensure when `!existsSync(join(getDataRoot('brain'),'profile.md'))`, so it survives the no-state-file path.
- **Consolidation-due nudge** (inside the reminders block, beside the trends nudge ~line 342 — an established user with published content has a state file, so the reminders path is correct here). **Use the `\\n` escaped-newline idiom** (the file un-escapes once via `context.replace(/\\n/g,'\n')` at line 479 — a raw `\n` would mis-render; plan-critic M6):
```js
const published = countPublished(join(getDataRoot('ingest'), 'published'));
const lastRun = brainLastRun(join(getDataRoot('brain'), 'consolidation-state.json'));
const daysSinceRun = daysSince(lastRun);
if (published > 0 && (lastRun === null || (daysSinceRun !== null && daysSinceRun >= 7))) {
const since = lastRun === null ? 'never run' : `${daysSinceRun}d ago`;
reminders += `- ${published} published post(s) captured, last brain consolidation ${since}. Run \\\`brain consolidate\\\` to evolve your profile.\\n`;
}
```
Null-safe + fires only when published records exist (a never-used brain never nags — the trend-nudge discipline). **No `profile.md` parse.**
## 4. Files created / edited
**Created:** `scripts/brain/src/consolidate.ts` · `scripts/brain/tests/{consolidate,consolidate-cli}.test.ts` · `hooks/scripts/__tests__/session-start-brain-consolidation.test.mjs` · `docs/second-brain/consolidation-loop.md`.
**Edited:** `scripts/brain/src/cli.ts` (+consolidate) · `hooks/scripts/session-start.mjs` (scaffold-ensure + nudge + `readdirSync` import) · `scripts/test-runner.sh` (BRAIN floor + assertion floor) · `docs/second-brain/architecture.md` (flip SB-S2 row).
**Conditional (Q2 = bump to 0.5.2):** `plugin.json` + README badge + CLAUDE.md header + CHANGELOG; polyrepo release via `release-plugin.mjs` (tag + catalog), pushed per window + confirm.
## 5. TDD test plan (failing-first) → success criteria
| Test | Asserts | SC |
|---|---|---|
| `consolidate.test.ts` | add / reject-ai-draft / evidence-bump / promote-at-3 / **conflict-keep-both with DISTINCT primary+alt ids, old fact untouched** / decay-flag (dynamic only) / **static-decay-exempt** / folded-immutable; input immutability; `applyDiff∘proposeDiff` round-trips via parse/serialize; idempotent (re-run → bumps not dup facts, alt-id stable) | SC1ag, SC2, SC3, SC4 |
| `consolidate-cli.test.ts` | temp-dir: `--gather` reads published bodies (filtered by `published_date>last_run`) + leaves `profile.md` byte-unchanged; `--propose` leaves profile.md unchanged, writes `pending-diff.{md,json}`, REJECTS a malformed candidate file AND a candidate whose value contains `\n` (non-zero, no write); `--apply --confirm` writes profile.md + `consolidation-state.json`; `--apply` without `--confirm` refuses; `init`/`ingest`/`published` still work | SC5 |
| `session-start-brain-consolidation.test.mjs` (authored via **Bash-heredoc** — pathguard-safe) | subprocess + isolated HOME/`LINKEDIN_STUDIO_DATA` (trends-staleness harness): nudge fires (rendered on its own line) when published records exist + last_run stale/absent; silent when brain absent / no published; scaffold-ensure runs on the fresh-install (no-state-file) path; hook still emits valid JSON | SC6 |
| (separate) `node --test hooks/scripts/__tests__/` | the new SC6 test + all existing hook tests pass — **run as an explicit step, NOT via test-runner.sh** (which runs no hook test; plan-critic B3) | SC6 |
| (gate) `scripts/test-runner.sh` | green; floors hold; brain ≥ new floor; counts unchanged; `compile-hooks.py --check` clean if hooks.json regenerated. (No new test-runner section → assertion floor +0.) | SC7 |
Each test written **before** its module (iron law). **Circuit-breakers:** halt if the engine round-trip (SC3) can't be made green, or if the full gate (step 8) is red.
## 6. Step sequence (each step: TDD red→green; Checkpoint = commit with the named message; On-failure → revert the named target + halt + report)
1. **Verify-first:** re-confirm `proposeDiff` inputs against `types.ts` (`ProfileFact`/`ProfileDoc`/`Provenance`) + `mintEntityId`/`mintContentId`/`slugify` signatures + `daysSince`/`getDataRoot('')` behavior in session-start.mjs (does the empty subdir return the bare data root?). **tsx fail-loud:** assert `scripts/brain/node_modules/.bin/tsx` present (the gate brain section *warn-skips* if absent — for this slice that would let the gate go green with SB-S2's tests never run; `npm install` and FAIL if still absent; plan-critic M11). *On-failure: stop, report — the slice cannot be verified.*
2. TDD `consolidate.ts` types + `proposeDiff` per-rule (`consolidate.test.ts`, SC1ag + SC2; incl. the distinct primary/alt id assertion). *Checkpoint: `feat(linkedin-studio): SB-S2 consolidation engine — proposeDiff classification [skip-docs]`. On-failure: revert `scripts/brain/src/consolidate.ts` + its test.*
3. TDD `applyDiff` + round-trip + idempotency (SC3/SC4). *Circuit-breaker: if the parse/serialize round-trip can't be made green, the id/shape is wrong — halt.* *Checkpoint: fold into step-2 commit or `feat(...): SB-S2 applyDiff + round-trip [skip-docs]`.*
4. TDD sidecar IO (`readConsolidationState`/`writeConsolidationState`) — fold into the consolidate-cli temp-dir tests. *Checkpoint with step 5.*
5. Extend `cli.ts` `consolidate` (gather/propose/apply); TDD `consolidate-cli.test.ts` (SC5) incl. candidate validation (shape + no-newline), no-confirm refusal, `--gather` body-read, `init`/`ingest`/`published` regression. *Checkpoint: `feat(linkedin-studio): SB-S2 brain consolidate CLI [skip-docs]`. On-failure: revert `cli.ts` + the cli test.*
6. Edit `session-start.mjs` (import `readdirSync`; UNCONDITIONAL scaffold-ensure near the migration block; consolidation-due nudge in the reminders block, `\\n` idiom); author `session-start-brain-consolidation.test.mjs` **via Bash-heredoc** (pathguard-safe); run `node --test hooks/scripts/__tests__/` (the WHOLE hook suite, since the gate doesn't — SC6 + no hook regression). If `hooks.json` shape unchanged (only script body edited) no recompile; else `compile-hooks.py` + verify `--check`. *Checkpoint: `feat(linkedin-studio): SB-S2 session-start scaffold-ensure + consolidation nudge [skip-docs]`. On-failure: `git checkout hooks/scripts/session-start.mjs` (do NOT leave the hook emitting invalid JSON) + halt.*
7. Write `docs/second-brain/consolidation-loop.md` (CLI usage, engine rules, the candidate-file schema as the session↔engine contract, the operator-gate, honest limits incl. no-reader-until-S3 + the loop's value-depends-on-extraction caveat). *Checkpoint: `docs(...)`.*
8. Bump `scripts/test-runner.sh` `BRAIN_TESTS_FLOOR` to the real `tests N` (assert the brain suite RAN — `BR_TESTS` non-empty, not warn-skipped). **`ASSERT_BASELINE_FLOOR` unchanged (+0): SB-S2 adds NO new test-runner section** (the hook test runs separately, the new TS tests run inside the existing brain `npm test` which contributes one unchanged `pass()` line) — confirm by reading the printed `TOTAL_CHECKS` is unchanged. Run the FULL gate green AND the hook suite green. *Circuit-breaker: no red gate committed.* *Checkpoint: `chore(...): SB-S2 gate brain floor`.*
9. Flip `architecture.md` SB-S2 row → landed (exact edit: the `| **SB-S2 — Evolution loop** |` row → add the `*landed*` marker, matching SB-S0/S1 row style); persist brief + plan. **Commit; confirm before push (PUBLIC `open/`), window; STATE.md gitignored.** Conditional 0.5.2 release per Q2 (tag + catalog via `release-plugin.mjs`).
## 7. Scope fence (echo — SB-S2 does NOT)
No journal-capture · no new agent (session extracts) · no `profile.md` reader (S3) · no supersede (keep-both only; S3) · no cross-silo id threading (S3) · no `content-history.md` retirement · no AI at session-start (deterministic nudge only) · no auto-apply (only `--apply --confirm` writes profile.md) · no new hook `.mjs` (edit existing session-start.mjs) · no new seam function · no connector (S4) · no GUI.
## 8. Known limits / deferred (honest)
- **The loop's VALUE depends on the session's candidate extraction** — the engine guarantees only the mechanics (threshold/conflict/decay/provenance-gate). Garbage candidates → a garbage diff (but the operator gate + the candidate-schema validation catch shape errors, not quality). Documented in `consolidation-loop.md`.
- **profile.md has no reader until S3** — S2 evolves an artifact nothing consumes yet; the value is deferred compounding. Stated honestly.
- **Evidence inflates only on genuinely-new candidates** — `--gather` is delta-gated (published since `last_run`), so a re-run after `--apply` surfaces nothing; the engine itself would bump again given the same candidate, but the loop never re-feeds one. Documented.
- **No supersede / no stale-fact demotion in S2** — conflicts keep both; stale facts are flagged, never auto-removed (operator/ S3).
- **Session-start nudge is new-published-count only** — no per-fact staleness at session-start (no zero-dep profile parser).
## 9. Risks
- **Duplicate-id corruption (was plan-critic B1) — CLOSED:** primary ids (`mintEntityId(key)`) and conflict-alt ids (`mintContentId('observed-alt:'+key+'::'+value+'::'+date)`) are byte-distinct, so no two facts share an id; the bump/promote target is unambiguous and `parseProfile`/`serializeProfile` stays well-formed. Pinned by SC1e (distinct ids) + SC3 (round-trip) + SC4 (idempotency).
- **`--gather` data source (was plan-critic B2) — CLOSED:** gather reads the published files directly (`parsePublishedRecord` → body), not `listPublished()` (which lacks body/`captured_at`); filtered by `published_date > last_run`.
- **Hook tests not gate-run (was plan-critic B3):** `scripts/test-runner.sh` runs NO hook test, so SC6 is verified by an explicit `node --test hooks/scripts/__tests__/` step (the whole suite), NOT the structure gate. SC7 reworded; no false "hook tests pass at the gate" claim. **Flagged (not done):** wiring the hook suite into test-runner.sh is a separate hardening opportunity.
- **Engine rule ambiguity (conflict vs bump):** deterministic value-equality decides bump-vs-conflict; keep-both is the safe default; each rule pinned by its own SC1x test before the CLI wires it.
- **Sidecar reachability (was the brief-review blocker) — CLOSED:** both `--apply` (dataRoot) and session-start (getDataRoot) resolve the SAME data root → the sidecar closes the loop; SC6 sets HOME + data root to separate temp dirs (the trends-test pattern) to prove it.
- **Zero-dep hook discipline:** no tsx, no profile parse, bounded readdir, `\\n` idiom; scaffold-ensure runs unconditionally (fresh-install path); SC6 asserts valid JSON output + the nudge rendering.
- **Gate floors:** BRAIN floor = actual post-impl count (assert suite RAN, not warn-skipped — step 1 fail-loud on missing tsx); `ASSERT_BASELINE_FLOOR` +0 (no new test-runner section). Confirm `TOTAL_CHECKS` unchanged at step 8.
- **Pathguard:** session-start.mjs is an existing file → Edit (safe). The new hook TEST `.mjs` under `hooks/scripts/__tests__/` is authored via **Bash-heredoc** — safe whether or not the active Category-6 guard matches `__tests__/` (the verified llm-security `.mjs` guard matches `hooks/scripts/.*\.mjs`; heredoc sidesteps it regardless).