Kjell Tore Guttormsen
68f6283d8a
The LAST second-brain slice; the S0–S3e arc is now complete.
(b) Retire the dead, zero-reader content-history.md across its 8 plumbing
surfaces: the flaky Stop-hook writer prose, the template (git-rm'd), the
migrate-data.mjs B1 MOVE entry + its test assertions, .gitignore, the gate
SC2_CLASSES guard, the data-path ref-doc, and a session-start comment. SC1
grep = 0; migrate suite 5/5 (R8 idempotency demonstrated).
(c) `brain reconcile` — read-side triple-post reconciliation joining silo 1
(## Recent Posts, auto-tracked creation) to the silo 2↔3 graph, surfacing the
coverage gap: posts created via the plugin but never `brain ingest`-ed. New
pure core scripts/brain/src/reconcile.ts (parseRecentPosts tracks the WRITER
format state-updater.mjs:116, NOT the date-only pruner; reconcileRecentPosts
matches hook→record.body→graph for the in-graph/in-brain-only/orphaned tiers;
loadRecentPosts reads STATE_FILE via the canonical getStateFile() chain — a new
cross-seam read, the state file lives outside the brain dataRoot). Wired as the
`brain reconcile` CLI subcommand (inline parsePublishedRecord loader, not the
body-less listPublished). Read-only: never writes the state silo.
Tests (verified live): gate 99/0/0 (ASSERT floor 82→84; new Section 16f
self-test + CLI grep) · brain 127/127 (floor 114→127, +13 reconcile) · hook
suite 136/136. SC4/SC6 end-to-end run is a real behavioural pass (STATE_FILE
seam read + fallback). Honest limit: read-side cannot reconstruct un-captured
specifics/trends — auto-capture is the flagged follow-up.
Brief/plan: docs/second-brain/{brief,plan}-sb-s3e.md (fbad29d). Go-before-code
gate cleared (operator: retire · read-side · build-now).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RigJBiRFNtFZKCz21qNbQ4
102 lines
14 KiB
Markdown
102 lines
14 KiB
Markdown
# Second Brain — Architecture Design
|
||
|
||
> **Status:** architecture **approved by operator 2026-06-23**. **SB-S0 (Foundation) + SB-S1 (Ingest + gold signal) + SB-S2 (Evolution loop) landed 2026-06-23** (`scripts/brain/`, 82 tests, gate-wired; ingest CLI + published-only invariant + operator-gated consolidation loop + session-start nudge); S3–S4 remain design-phase.
|
||
> **Boundary (confirmed 2026-06-23):** the **engine** (store schema · evolution loop · ingest seam) → **the plugin** (domain-general, shareable); the **user's data** (posts · articles · newsletters · plans · ideas) → the **per-user data dir** (`${LINKEDIN_STUDIO_DATA:-$HOME/.claude/linkedin-studio}/`, survives reinstall); the **personal cockpit** (the operator's day-to-day operations centre) → **Maskinrommet** (a thin layer that reads/writes *through* the plugin's store, never a fork of the engine).
|
||
> **Research inputs (three parallel threads, 2026-06-23):** `research/connector-egress.md` · `research/secondbrain-sota.md` · `research/silo-inventory.md`.
|
||
|
||
## The problem
|
||
|
||
A "second brain" stores everything about one creator — posts, articles, newsletters, data, plans, ideas — and compounds it into an ever-improving, user-aligned profile. It is **memory AND an operations centre**.
|
||
|
||
The three research threads reframed the task in a decisive way:
|
||
|
||
1. **There is nothing to migrate.** M0 already routed all **12 existing per-user silos** through one tested seam (`hooks/scripts/data-root.mjs` `getDataRoot()` + its TS twin `scripts/analytics/src/utils/storage.ts`), default `${LINKEDIN_STUDIO_DATA:-$HOME/.claude/linkedin-studio}/`. They all survive reinstall today. The task is **unify**, not move.
|
||
2. **Not a vector/graph DB.** For *our* case the portable, auditable, drift-resistant 2026 standard is **plain-text + JSON + git** — because **Claude *is* the semantic-retrieval engine** (it reads files in-context). A heavyweight store adds infra, breaks portability, and invites lock-in for no gain.
|
||
3. **Auto-ingest exists — for an EU/EEA member.** The DMA Member Data Portability API is self-serve in EU/EEA + Switzerland and gives automated pull of **content** (post text + articles), but **not** received-analytics. Analytics stays the existing manual CSV path. Scraping is a ToS breach with active enforcement → never baked in. So ingest is **manual-first as the contract**; any connector is an additive tributary.
|
||
|
||
The real gap (from the silo inventory): the 12 silos are **per-user but siloed and heterogeneous** — three storage idioms, two roots, no cross-references, provenance reinvented under different field names in each. A published post lands in three non-referencing places with no shared id. The question *"which raw material actually performs?"* (specific → post → measured analytics) is **unanswerable today** — and it is exactly what a second brain should answer.
|
||
|
||
## The shape: a thin Markdown hub over typed tributaries
|
||
|
||
```
|
||
${LINKEDIN_STUDIO_DATA:-$HOME/.claude/linkedin-studio}/
|
||
brain/
|
||
index.md # MOC / hub — one-screen pointer to every tributary + freshness flags
|
||
# (the "memory AND ops centre" front page)
|
||
profile.md # SEMANTIC, two-layer: §static (stable) + §dynamic (evolving).
|
||
# Each fact: value · first_seen · last_seen · evidence_count ·
|
||
# provenance(human|published|ai-draft) · status(active|superseded)
|
||
operations.md # PLANS / IDEAS + the "who I am now" anchor (frozen-past-self guard)
|
||
journal/ # EPISODIC, append-only: YYYY-MM-session.md — raw, never edited;
|
||
# the source the consolidation loop distils FROM
|
||
ingest/
|
||
inbox/ # NEW drop-zone: manual import OR (EU/EEA) connector output
|
||
published/ # processed, provenance=published — the GOLD signal
|
||
voice-samples/ # tributary (style) — stays its own store
|
||
specifics-bank/ # tributary (raw material) — stays its own store
|
||
trends/ # tributary (external signal) — stays its own store
|
||
analytics/ # tributary (performance) — stays its own store
|
||
```
|
||
|
||
### Tributaries vs hub — the explicit call
|
||
|
||
- **Stay separate (tributaries), feed the hub via distilled pointer-summaries:** voice-samples, specifics-bank, trends, analytics, ingest. Each has a *different dedupe key + lifecycle* (trends: title+url, fast decay · specifics-bank: human-only, near-permanent · analytics: numeric, immutable). Forcing one schema would destroy those distinctions — the silo inventory warns of this explicitly. The hub holds a distilled summary of each + a pointer.
|
||
- **Fold INTO the hub:** the flat `profile/user-profile.md` → two-layer `brain/profile.md`. Scattered `audience-insights/` / `examples/` become dynamic-layer sections or tributary-summaries referenced from `index.md`. The **operations/plans centre is genuinely new**.
|
||
|
||
### Alternatives considered (and rejected)
|
||
|
||
- **Vector/graph DB (Pinecone/Neo4j/Zep-style):** rejected — adds infra, breaks plain-text portability + git audit/rollback, invites lock-in, and duplicates retrieval that Claude already does in-context. The four SOTA ideas that port cleanly to files (episodic/semantic split, provenance-weighting, evidence-threshold promotion, temporal-validity frontmatter) give most of the value; the vector/graph machinery does not port and is not needed.
|
||
- **One unified schema across all silos:** rejected — destroys each silo's distinct dedupe key + lifecycle. Keep tributaries; add a thin hub + a cross-silo id.
|
||
|
||
## Invariants (the spine)
|
||
|
||
- **Provenance-weighted learning (the critical guard):** profile/voice learn from `provenance=published` (human-edited) **only**, never `ai-draft`. This is the model-collapse guard for our exact risk — a content engine that learns voice from what it itself drafted collapses toward its own priors. Generalises `voice-scrubber`'s existing rule ("gold standard = approved editions, not the English post corpus") to a system-wide invariant.
|
||
- **One canonical entity id + one provenance shape**, threaded through the silos → makes the post ↔ specific ↔ trend ↔ analytics graph assemblable. Fixes the inventory's hardest problem.
|
||
- **Episodic / semantic physically separated** (raw events in `journal/`, distilled facts in `profile.md`).
|
||
- **Evidence-threshold promotion (anti-overfit):** the dynamic layer decays; a fact enters the **static** layer only on explicit confirmation OR ≥N independent observations. Single weird sessions never reshape identity.
|
||
- **Temporal-validity + freshness nudge:** every fact carries `last_seen`; retrieval weights recent over old; the session-start nudge flags facts not refreshed in T days (extend the existing trend-staleness nudge brain-wide).
|
||
- **Anti-sycophancy as a built-in default, not a toggle:** consolidation + content agents treat the profile as evidence to *test*, not flatter. (Personalization context measurably *increases* sycophancy, so counter-pressure it every time the profile is used.)
|
||
- **Frozen-past-self guard:** `operations.md` holds a periodic user-authored "where I'm headed now" anchor that deprecates older inferences.
|
||
- **Zero required curation (graveyard guard):** the loop compounds value with ~zero mandatory upkeep; curation is available but never required. git = free rollback.
|
||
|
||
## The evolution loop (the compounding mechanism)
|
||
|
||
Run on a cadence (reuse the existing session-start staleness-nudge surface) as a **sleep-time consolidation pass**, not inline cost:
|
||
|
||
1. **Capture (episodic, auto):** real-signal sessions append to `brain/journal/`. The ingest seam (`ingest/inbox/`) takes the user's actual published posts — manual import first; published output tagged `provenance=published`.
|
||
2. **Consolidate (reflection):** an Opus agent reads new journal + published + tributary deltas, proposes profile updates as a **diff**, never a silent overwrite. Each candidate carries `evidence_count`, `provenance`, timestamps.
|
||
3. **Promote with thresholds:** static-layer entry needs confirmation or ≥N observations; one-offs stay in the decaying dynamic layer.
|
||
4. **Reconcile contradictions:** classify temporal-update (supersede, keep old `status: superseded` + date) · contradictory (keep both, timestamped) · condition-dependent (scope) · distractor (drop). Bias toward "keep both + surface the conflict" over silent overwrite.
|
||
5. **Decay + freshness:** weight recent over old; flag stale facts at session-start.
|
||
6. **Curate (optional, never required):** the diff is presentable; confirm/edit/delete/roll-back via git.
|
||
|
||
## Build sequence (full ambition, incremental — one slice per session)
|
||
|
||
| Slice | Content | Why this order |
|
||
|-------|---------|----------------|
|
||
| **SB-S0 — Foundation** ✅ *landed 2026-06-23* | `brain/` scaffold + two-layer `profile.md` (fold in `user-profile.md`) + `index.md` MOC + `operations.md` + `journal/`; entity-id + provenance shape as a small typed, tested module; `ingest/` dirs + manual-import contract. **No loop yet.** Shipped as `scripts/brain/` (TS, 34 tests, gate-wired BRAIN floor); fold = P1 labeled-scalars + P2 expertise (checkbox-prefs deferred, §8 of `plan-sb-s0.md`). | Smallest thing that stands up and is testable; locks the id/provenance spine everything hangs on |
|
||
| **SB-S1 — Ingest + gold signal** ✅ *landed 2026-06-23* | Manual import → `ingest/published/` with `provenance=published` (CLI `brain ingest`); voice-trainer wired to learn from published-only, never `ai-draft`, gate-enforced. Shipped as `scripts/brain/src/ingest.ts` + `docs/second-brain/ingest-manual-import.md` (v0.5.1). **No profile.md mutation** (SB-S2). | The gold signal before the loop that consumes it |
|
||
| **SB-S2 — Evolution loop** ✅ *landed 2026-06-23* | Operator-invoked, operator-gated consolidation: `brain consolidate` (`--gather`/`--propose`/`--apply --confirm`) → profile **diff** w/ evidence_count/provenance/timestamps; threshold-promotion (N=3); contradiction → keep-both with distinct ids (no supersede); decay-flag (90d); `consolidation-state.json` sidecar; zero-dep session-start consolidation-due nudge + scaffold-ensure. Shipped as `scripts/brain/src/consolidate.ts` + `docs/second-brain/consolidation-loop.md` (v0.5.2). **Operator decisions:** journal deferred · no new agent (session extracts) · motor-only (no reader until S3). | The compounding mechanism |
|
||
| **SB-S3 — Cross-silo graph + ops centre** | Thread the id through tributaries (post↔specific↔trend↔analytics assemblable); flesh out `operations.md`; retire the dead `content-history.md` + triple-post reconciliation. **S3a ✅ first reader · S3b ✅ supersede · S3c ✅ cross-silo id-threading** (hub-side: the published record carries the `specifics`/`trends` ids it was built from + a pure analytics resolver — `scripts/brain/src/assemble.ts`, `brain assemble`; tributaries untouched). **S3d ✅ ops centre** (`operations.md` is a read tributary; `strategy-advisor` honours the dated "who I am now" anchor that *deprecates older inferences* — advisory/reader-side). **S3e ✅ hygiene + reconciliation** (dead `content-history.md` retired across its 8 plumbing surfaces; `brain reconcile` reconciles silo 1 `## Recent Posts` ↔ the silo 2↔3 graph read-side, surfacing the coverage gap — created posts never `brain ingest`-ed; auto-capture of specifics/trends = a flagged follow-up). **The second-brain arc is complete** (S4 EØS DMA-connector optional). | Finally answers "which raw material performs?" |
|
||
| *(later / optional)* **SB-S4** | EU/EEA DMA portability API as an auto-tributary into `ingest/inbox/` | Additive; never a dependency |
|
||
|
||
## What's genuinely hard (honest flags)
|
||
|
||
1. **Voice fidelity to a private individual is limited even at SOTA** — the real win is grounded *content* (specifics-bank) over mimicked *style*. Manage expectations.
|
||
2. **Detecting that a high-relevance fact went stale is UNSOLVED field-wide** (STALE benchmark: all tested models/frameworks fail). We timestamp + nudge; we cannot auto-detect "changed jobs." Keep a human in that loop.
|
||
3. **Contradiction classification** (real change vs context-scoped vs noise) is emerging + error-prone → bias to keep-both-timestamped.
|
||
4. **The connector** — no clean LinkedIn self-serve content API outside EU/EEA. Build the manual ingest seam as the contract; any connector is a tributary, never a dependency. Newsletter-edition coverage by the portability `ARTICLES` domain is **not fully verified** — check against a real export.
|
||
5. **Sycophancy is structural** — app-level mitigations reduce, not eliminate.
|
||
6. **Avoiding the graveyard is a product problem** — the loop must compound value with ~zero required curation or it dies in 6 months like 82% of second brains.
|
||
|
||
## Verification (SB-S0, when we build it)
|
||
|
||
- Set `LINKEDIN_STUDIO_DATA` to a temp dir, run init → assert `brain/{index,profile,operations}.md` + `journal/` + `ingest/{inbox,published}` exist via the `getDataRoot` seam.
|
||
- `profile.md` parses two-layer (§static / §dynamic); `user-profile.md` fields fold in without loss (diff check).
|
||
- entity-id module: deterministic id mint + provenance shape, unit-tested.
|
||
- **No regression:** gate `scripts/test-runner.sh` 89/0/0 green; trends 24/24; specifics 28/28; contract 33/33.
|
||
- **Key assumption to test early:** "Claude-as-retrieval-engine over plain files is sufficient (no vector DB)" — testable with a retrieval scenario once the brain holds content. Marked as assumption until proven.
|
||
|
||
## Bottom line
|
||
|
||
A thin two-layer Markdown hub (`brain/`: semantic `profile.md` + episodic `journal/` + ops `operations.md` + `index.md` MOC) over the existing typed tributaries, fed by a provenance-tagged ingest seam, maintained by a sleep-time consolidation loop with evidence-threshold promotion, temporal-validity reconciliation, and a built-in anti-sycophancy / anti-collapse stance. Every silo stays a tributary except the flat `user-profile.md`, which folds in.
|