Kjell Tore Guttormsen
d2ec7482be
Second sub-slice of SB-S3 (after S3a's reader landed). Adds an operator-gated,
explicitly-signalled supersede op to the consolidation engine: a temporal-update
candidate retires a stale fact (status: superseded, re-minted to an archival id,
replaced in place, retained as audit) and installs the new winner under the
canonical key-id. Bounded to scripts/brain/ — pure, deterministic, fully
unit-testable.
Operator-confirmed scope: temporal-update→supersede ONLY · winner→dynamic
(re-earn) · minimal `supersedes?: string` · retain superseded as audit.
Light-Voyage-hardened:
- brief-reviewer APPROVE-WITH-FIXES — 8 FIX folded (the load-bearing one: id
lifecycle → winner-takes-key-id, retired fact re-minted, so mintEntityId(key)
always points at the live fact).
- plan-critic REVISE → 2 BLOCK + 6 FIX folded: applyDiff must REPLACE-IN-PLACE
(not copy → would dup the canonical key-id); proposeDiff must value-guard the
supersede fork (a re-sent stale signal would self-supersede every run); plus
--gather active-only filter + decay excludes superseded (else retired facts
leak back as live context / spam stale flags).
- scope-guardian ALIGNED (0 creep / 0 gap).
Also reconciles brief-sb-s3a.md's stale DRAFT status (S3a landed 4fa411f).
No code yet — awaiting go-before-code.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RigJBiRFNtFZKCz21qNbQ4
111 lines
19 KiB
Markdown
111 lines
19 KiB
Markdown
# Brief — SB-S3b: the supersede arm
|
||
|
||
> **Slice:** SB-S3b (second sub-slice of SB-S3). **Status:** DRAFT — awaiting operator "go" before any code. Light-Voyage: brief-review APPROVE-WITH-FIXES (8 FIX folded; 3 RISK carried to the plan).
|
||
> **Predecessors:** SB-S0 (id/profile substrate — reserved the `superseded` status) · SB-S1 (published gold) · SB-S2 (consolidation motor — keep-both, *no supersede*) · **SB-S3a (first reader `strategy-advisor` — LANDED `4fa411f`, gate 95/0/0, ASSERT floor 80).**
|
||
> **Grounded in:** a read-only code map of `scripts/brain/` (file:line cited throughout) + a brief-review pass against the same code.
|
||
|
||
## 1. Operator decision (2026-06-23)
|
||
|
||
SB-S3 was decomposed into four sub-slices; after S3a (reader) landed, the recommended order is **S3b (supersede) → S3c (id-threading) → S3d (hygiene+ops, last)**. S3b is next because it has the **lowest blast radius** (bounded to the `scripts/brain/` TS package — pure functions, fully unit-testable, unlike S3a's behavioural SCs), it **completes the consolidation motor's one explicit TODO** (`consolidate.ts:11`: *"no supersede in S2 — that's S3"*), and it **strengthens the read path S3a just shipped** (a profile with supersede gives the new reader cleaner, non-contradictory facts).
|
||
|
||
## 2. Why supersede, and the central design decision
|
||
|
||
S2's engine handles a value-conflict by **keep-both**: the old fact is left untouched and a new alt fact is added with a distinct content-id (`consolidate.ts:120-129`). That is correct for a *genuine contradiction* (two views coexist), but wrong for a **temporal update** — when a new fact *replaces* an outdated one (the user changed roles, a number moved, a stance evolved). Today there is **no way to retire the stale fact**; it persists forever, and the S3a reader has to weigh a fact the user has already moved past.
|
||
|
||
The `superseded` status was reserved at SB-S0 (`types.ts:17`, `:23`; grammar `profile.ts:58`; round-trip proven `profile.test.ts:35`) precisely for this — **but nothing ever sets it.** S3b is the code that does.
|
||
|
||
**Central design decision (the "fuzzy heart", `brief-sb-s2.md:115`) — supersede is driven by an EXPLICIT signal, never engine auto-classification.** The S2 architecture splits cleanly: the **session** does the fuzzy extraction, the **engine** is deterministic mechanics, the **operator** gates the diff before apply. Classifying *temporal-update vs contradictory* is a fuzzy judgement — so it stays where the fuzziness already lives (the session's candidate extraction), not baked into a pure function. A candidate carries an explicit "this supersedes the prior fact for key X" intent; `proposeDiff` **deterministically** routes it; the operator **sees and confirms** the supersession in the rendered diff before `--apply --confirm` writes it. This honours the S2 promise (`brief-sb-s2.md`: *"reserve supersede for an explicit confirm signal"*) and keeps the engine deterministic and testable.
|
||
|
||
**Consequence (the engine has no classification safety net).** Because the engine never validates that a supersede *should* be a supersede, the **operator gate is the only place a mis-classification is caught** — and supersede is the first op that *retires* state (a mis-call silently retires a fact that should have been kept-both, and `status: superseded` makes it look intentional). Two obligations follow: SC6's rendering must make every supersession **visually unmistakable** (old value → new value, both shown, flagged as a retirement), and the doc reconciliation (§3.7) must add an honest-limit line mirroring `consolidation-loop.md:64` — *the engine guarantees supersede mechanics; whether a contradiction is really a temporal update is the session's + operator's judgement.*
|
||
|
||
## 3. Scope — what is IN (S3b)
|
||
|
||
1. **Candidate supersede signal.** Extend the transient `Candidate` (`consolidate.ts:33-39`) with an explicit, optional supersede intent — minimal shape: `supersedes?: string` (the **key** of the active prior fact this candidate replaces). Set only by the session when it judges a temporal update; absent for ordinary candidates (fully backward-compatible). `validateCandidates` (`cli.ts:147-157`) gains an optional single-line check for the new field.
|
||
|
||
2. **A sixth diff array.** `ProfileDiff` (`consolidate.ts:41-47`) gains `supersedes: { oldId, oldValue, newId, newValue }[]` (parallel to the existing `conflicts[]` — the five-array shape means a new op is a new array, not a union edit).
|
||
|
||
3. **`proposeDiff` routing (deterministic), with a precise matching rule.** "Matching active prior fact" = the fact returned by `byId.get(mintEntityId({kind:'observed', key}))` — i.e. the live fact at the canonical key-id (see §3.5: retired facts no longer hold the key-id, so this is unambiguously the *active* one). When a candidate carries `supersedes: <key>` AND that lookup finds a fact, route to `supersedes[]`. When the signal is present but **no fact is found at the key-id**, degrade gracefully to a plain **add** (defined, tested behaviour — never a crash). Candidates without the signal are untouched (keep-both/bump/promote/decay exactly as S2).
|
||
|
||
4. **`applyDiff` mutation (the first `superseded` writer), state-checked.** Add a branch (`consolidate.ts:144-162`) that, for each `supersedes[]` entry: (a) re-mints the retired fact to a distinct archival id and sets its `status: "superseded"`; (b) installs the new winner under the canonical key-id (see §3.5). The branch is **state-checked, not blind**: a `supersedes[]` entry whose `oldId` is absent from `current` or already `superseded` is **skipped** (this is what makes re-apply idempotent and guards stale diffs — §5). Superseded facts are **retained** in the doc (audit), per `architecture.md:69` (*"keep old `status: superseded` + date"*) — never deleted. Stays a **pure** function returning a new doc.
|
||
|
||
5. **Id discipline — winner takes the key-id; retired fact is re-minted off it.** Because `mintEntityId` is value-independent (`id.ts:33`), only one fact may hold the canonical key-id, and it must be the **live** one. So on supersede: the **new winner takes `mintEntityId({observed, key})`** (the canonical key-id), and the **retired fact is re-minted to a distinct archival id** (`mintContentId` over an archival seed, e.g. `superseded:${key}::${oldValue}::${oldFirstSeen}`, so it is stable and collision-free). This preserves the **no-duplicate-id** invariant AND keeps `mintEntityId(key)` always pointing at the live fact — so every downstream match (future same-key candidates, the S3a reader) resolves to the winner exactly as before supersede existed. *(This inverts the naïve "keep old id, mint winner fresh", which would leave the key-id on a retired fact and resurrect it on the next same-key candidate — the brief-review's [FIX 1].)*
|
||
|
||
6. **CLI rendering, non-vacuously checked.** `renderDiffMd` (`cli.ts:132-145`) gains a `## Supersessions (old → new)` section so the operator sees every retirement before confirming. **No new subcommand** — supersede rides the existing `--propose` / `--apply --confirm` flow; the sole `profile.md` writer stays `cli.ts:215`. Because the operator confirms via the **JSON** diff while reading the **MD**, a **brain-suite test** (`consolidate-cli.test.ts`, repo non-vacuity idiom) asserts **every `supersedes[]` JSON entry produces a corresponding rendered MD line** and a 0-entry diff renders no section — not merely that a section exists (RISK C). *(This is a brain-suite test, not a `test-runner.sh` Section — so it is counted by `BRAIN_TESTS_FLOOR`, not by a new unconditional structure-lint check; see §3.7.)*
|
||
|
||
7. **Gate + doc reconciliation.** **Verified baseline (S3a landed):** `BRAIN_TESTS_FLOOR` = 82 (`test-runner.sh:716`), `ASSERT_BASELINE_FLOOR` = 80 (`:913`). Bump `BRAIN_TESTS_FLOOR` by the new brain-test count; **`ASSERT_BASELINE_FLOOR` stays 80** — no new *unconditional `test-runner.sh` structure-lint* check is added (the render-fidelity check lives in the brain suite, §3.6). `BRAIN_TESTS_FLOOR` is the only floor that moves. Reconcile the now-stale "no supersede" claims (`consolidation-loop.md:34`, `:71`; `consolidate.ts:11` docstring) to the true state, and add the honest-limit line from §2.
|
||
|
||
8. **`--gather` active-only filter (plan-critic FIX 8b — completeness).** `runConsolidate --gather` (`cli.ts:174`) emits `profileFacts` to the extraction session with **no status filter** today. After supersede lands, that would re-present `status:superseded` archival facts as live context — re-introducing the exact retired values this slice exists to retire (§2). The gather output is filtered to `status === "active"` (one line). Without this the slice regresses its own goal.
|
||
|
||
9. **Decay excludes superseded (plan-critic FIX 8c — completeness).** A retained *dynamic* superseded fact is still in `current.dynamic`, so the decay scan (`consolidate.ts:132-134`) would throw `staleFlags` for it forever. The decay filter excludes `status:superseded` (static archival facts are already decay-exempt — dynamic-only scan). Supersede and decay stay orthogonal.
|
||
|
||
## 4. Non-goals — what is OUT (deferred)
|
||
|
||
- **Engine auto-classification of temporal-update vs contradictory.** The signal is explicit (§2). The engine never guesses.
|
||
- **The other two contradiction-typology branches** — *condition-dependent (scope)* and *distractor (drop)* (`architecture.md:69`, `research/secondbrain-sota.md:37`). S3b ships **temporal-update → supersede ONLY**; contradictory stays keep-both (S2). The other two are later, purely-additive arrays (deferring them leaves no broken half-state — confirmed by brief-review).
|
||
- **Auto-demotion of stale facts.** Decay (`staleFlags`, `consolidate.ts:132-134`) stays **advisory** — never mutates status or removes a fact. Supersede and decay stay orthogonal (no shared field). A stale fact is *not* auto-superseded.
|
||
- **Cross-silo id-threading (S3c)** · **content-history retirement / operations.md / triple-post reconciliation (S3d)** · **any new reader** (S3a's reader unchanged; `content-optimizer` is later).
|
||
- **Chained / multi-fact supersede as a FEATURE.** One active prior → one new winner per candidate. Chains (A→B→C in one pass) and one-supersedes-many are out. **But the intra-batch SEMANTICS are defined, not undefined** (RISK A): the supersede fork carries a `touched`-style guard (analogous to `consolidate.ts:97`) so that within one candidate batch, two candidates targeting the same key, or a candidate superseding a fact another candidate just created, resolve deterministically (first-writer-wins + the later one degrades to add/bump) rather than corrupting. Deferring the *feature* is fine; the *batch rule* is in scope.
|
||
- **Grammar changes.** None needed — `superseded` is already accepted end-to-end.
|
||
|
||
## 5. Boundaries / invariants (must hold)
|
||
|
||
- **Operator-gated, single disk-writer.** No new write path to disk; `--apply --confirm` (`cli.ts:215`) stays the sole `profile.md` writer; `--apply` refuses without `--confirm`.
|
||
- **Pure, idempotent engine.** `proposeDiff`/`applyDiff` stay pure (no I/O). Idempotency holds at BOTH layers: `proposeDiff` produces no new `supersedes[]` entry once a supersession has landed (winner matches by value → bump; old fact already superseded), and `applyDiff`'s supersede branch is **state-checked** so re-applying the same diff does not double-install the winner or double-flip status.
|
||
- **Stale-diff safety (named assumption + guard).** A pending diff is meant to be applied against the profile it was proposed from; a profile mutated between propose and apply is operator error. S3b does not silently corrupt in that case: `applyDiff` **skips** any `supersedes[]` entry whose `oldId` is absent or already `superseded` in `current` (the §3.4 state-check), rather than retiring the wrong fact.
|
||
- **Audit-preserving.** Superseded facts are retained with `status: superseded`, never deleted — history is inspectable.
|
||
- **No-duplicate-id.** Every fact keeps a distinct id; the canonical key-id is held only by the live winner; the retired fact moves to a distinct archival id.
|
||
- **Backward-compatible.** Candidates without the signal behave exactly as S2; a profile with zero supersessions serialises byte-identically to today (`serializeProfile` unchanged; grammar already round-trips `superseded`).
|
||
- **Decay stays advisory** and orthogonal to supersede.
|
||
- **No grammar / no cross-silo threading / no new reader** in S3b.
|
||
- **TDD iron law** — each engine rule is pinned by a failing unit test before the CLI is wired (mirrors `brief-sb-s2.md:115`).
|
||
|
||
## 6. Success criteria (testable — deterministic, unit-level)
|
||
|
||
- **SC1 — routing:** a candidate with `supersedes: <key>` + a fact at `mintEntityId({observed, key})` → `proposeDiff` emits exactly one `supersedes[]` entry (oldId/oldValue + newId/newValue) and **no** `conflicts[]`/keep-both entry for it. *(unit)*
|
||
- **SC2 — mutation + concrete ids:** `applyDiff` produces a doc where (a) the **winner** carries `id == mintEntityId({observed, key})`, `status: "active"`, `evidence_count: 1`, the new value; (b) the **retired** fact carries `status: "superseded"` and an id `!= mintEntityId({observed, key})` (the archival id); (c) the retired fact is **still present** (count of facts for the key = 2, one active + one superseded). *(unit)*
|
||
- **SC3a — propose-level idempotency:** after a supersession has landed, a second `gather→propose` cycle with the *same* now-current value produces **no** new `supersedes[]` entry (the winner matches by value → bump). *(unit)*
|
||
- **SC3b — apply-level idempotency:** `applyDiff` of a supersede diff onto a doc where that supersession is already applied does **not** double-install the winner, double-flip status, or create a duplicate id (the §3.4 state-check skips it). *(unit)*
|
||
- **SC4 — graceful no-target:** a `supersedes: <key>` signal with **no fact at that key-id** (active or otherwise — superseded facts no longer hold the key-id) degrades to a plain add; no throw. *(unit)*
|
||
- **SC5 — round-trip through the real path:** `parseProfile(serializeProfile(applyDiff(cur, supersedeDiff)))` deep-equals `applyDiff(cur, supersedeDiff)` — a superseded fact *produced by the engine* round-trips, not just the hand-built `profile.test.ts:35` fixture. *(unit)*
|
||
- **SC6 — CLI gate + render fidelity:** `--propose` renders a `## Supersessions` section in `pending-diff.md` with one line per `supersedes[]` entry (old → new, both values shown), and a 0-entry diff renders no section (decoy); `--apply` writes the superseded status ONLY with `--confirm` (refuses otherwise). *(cli — brain suite)*
|
||
- **SC7 — no regression:** contradictory candidates (no signal) still keep-both; bump/promote/decay/published-only all unchanged (regression pins). *(unit)*
|
||
- **SC8 — intra-batch determinism:** a batch with two candidates superseding the same key (or one superseding a fact another candidate creates in the same batch) resolves deterministically per the §4 `touched`-guard rule — no duplicate id, no double-supersede. *(unit)*
|
||
- **SC9 — gate green:** brain suite ≥ bumped `BRAIN_TESTS_FLOOR`; `scripts/test-runner.sh` green; **`ASSERT_BASELINE_FLOOR` unchanged at 80** (the render-fidelity check is a brain-suite test, so `BRAIN_TESTS_FLOOR` is the only floor that moves). *(plan-critic FIX 7)*
|
||
- **SC10 — gather active-only:** `--gather` output excludes `status:superseded` facts (no retired value re-presented as live context). *(cli)*
|
||
- **SC11 — decay excludes superseded:** a retained dynamic superseded fact throws no `staleFlags`. *(unit)*
|
||
- **SC12 — supersede-wins ordering:** a diff that bumps/promotes AND supersedes the same id resolves to supersede-wins (the superseded fact is not bumped/promoted). *(unit)*
|
||
|
||
## 7. Verification
|
||
|
||
- **Deterministic (the whole slice):** SC1–SC9 are unit/CLI tests in `scripts/brain/tests/` + the structure-lint gate. **Unlike S3a, S3b has no behavioural-only SC** — supersede is pure engine mechanics, fully exercisable in the harness. A key reason it's a good second slice.
|
||
- **Run:** `(cd scripts/brain && npm test)` then `bash scripts/test-runner.sh`.
|
||
|
||
## 8. Open questions — RESOLVED (operator 2026-06-23)
|
||
|
||
The id-lifecycle is resolved — winner-takes-key-id (§3.5), per brief-review [FIX 1]. The four forks are now operator-confirmed:
|
||
|
||
1. **Q-SCOPE → temporal-update → supersede ONLY** ✅ (condition-dependent + distractor deferred). Clean cut, no broken half-state.
|
||
2. **Q-LAYER → winner enters DYNAMIC and re-earns promotion** ✅ (N=3). A superseded *static* fact stays static-but-superseded (decay-exempt — `consolidate.ts:132` scans dynamic only — acceptable as retained audit, not live). No static-inherit.
|
||
3. **Q-SIGNAL → minimal `supersedes?: string` (target key)** ✅ (widen later only if condition-dependent/distractor are built).
|
||
4. **Q-RETAIN → retain superseded facts as audit trail** ✅ (no pruning in S3b; any pruning is an S3d hygiene concern).
|
||
|
||
## 9. Brief-review (light-Voyage) — folded
|
||
|
||
`voyage:brief-reviewer` verdict: **APPROVE-WITH-FIXES** (PROCEED_WITH_RISKS). All 8 [FIX] folded above:
|
||
- **[FIX 1]** post-supersede id lifecycle → §3.5 inverted to winner-takes-key-id + retired-fact-re-minted (the one latent-corruption fix). ✅
|
||
- **[FIX 2]** SC3 idempotency layer → split into SC3a (propose) + SC3b (apply, state-checked). ✅
|
||
- **[FIX 3]** SC2 concrete id/status/count assertions (non-vacuous). ✅
|
||
- **[FIX 4]** stale-diff trust boundary → §5 named assumption + §3.4 skip-guard. ✅
|
||
- **[FIX 5]** floor baseline → §3.7 states the **verified** post-S3a baseline (82 / 80; S3a landed `4fa411f`). ✅
|
||
- **[FIX 6]** SC5 round-trips through `applyDiff` output, not the hand fixture. ✅
|
||
- **[FIX 7]** Q-LAYER decided default → §3 IN-scope; only static-inherit stays open (§8.2). ✅
|
||
- **[FIX 8]** SC4 names the scanned fact-set (the key-id lookup). ✅
|
||
|
||
The 3 [RISK] are carried as explicit plan obligations:
|
||
- **[RISK A]** intra-batch supersede semantics → defined in §4 + pinned by SC8 (`touched`-guard).
|
||
- **[RISK B]** operator gate is the only classification net → §2 consequence + §3.6 unmistakable render + §3.7 honest-limit doc line.
|
||
- **[RISK C]** `renderDiffMd` JSON↔MD divergence → §3.6 render-fidelity check + SC6 lint.
|
||
|
||
Brief-review confirmed (no fix): byte-identical backward-compat, five→six-array idiom, no grammar change, the "temporal-update only" scope cut, and the optional-field backward-compat of `Candidate.supersedes?`.
|
||
|
||
**Plan-critic pass (folded back into this brief).** Reviewing the plan, `voyage:plan-critic` surfaced two latent-corruption BLOCKs (applyDiff must replace-in-place not copy; proposeDiff must value-guard the supersede fork) — both resolved in the plan — and two completeness gaps that belong in the brief's scope: the **`--gather` active-only filter (§3.8)** and **decay excludes superseded (§3.9)**, with SC10/SC11/SC12 added to §6. `voyage:scope-guardian`: **ALIGNED** (0 creep / 0 gap). The plan (`plan-sb-s3b.md`) carries the full BLOCK/FIX ledger.
|