diff --git a/docs/research-engine/brief-re-r2a.md b/docs/research-engine/brief-re-r2a.md new file mode 100644 index 0000000..ce6e185 --- /dev/null +++ b/docs/research-engine/brief-re-r2a.md @@ -0,0 +1,208 @@ +# Brief — RE-R2a: capture bridge (item→store) + publishedAt persistence + +> **Slice:** RE-R2a (research-engine rung-2, slice 2 — the data layer of R2). The first of the two +> halves R2 ("the visible topic-stream") was split into at the 2026-06-24 go-gate: **R2a** = the pure +> `scripts/trends/` data layer (this brief); **R2b** = the dated morning-brief artifact + session-start +> surfacing (separate brief, after R2a lands). The split was chosen because (b)+(c) are code-independent +> of (a), R2a mirrors R1's pure-TDD shape exactly (lowest risk), and it closes the capture loop the store +> was built for. +> **Predecessor:** RE-R1 (`brief-re-r1.md`) delivered the validated ingress envelope (`item.ts`: +> `normalizeItem`/`normalizeItems`) and the triage scorer (`score.ts`) behind a CLI seam, and **explicitly +> deferred the item→store bridge** ("injecting the store's `capturedAt` and persisting `publishedAt` — is +> R2 orchestration", `brief-re-r1.md` §3 B1). R2a builds exactly that deferred bridge. +> **Substrate:** `docs/research-engine-concepts.local.md` §1 (hull 3 "store-schema mangler felt" / the +> item→store gap), §3 B1 (one schema downstream never branches on). **TDD-order:** RED tests land before code. + +## 1. Operator decision context (2026-06-24) + +The research engine is **Tier-1** (operator, 2026-06-23): *"hele min arbeidsflyt hviler på at jeg får en +jevn strøm av gode forslag til tema å skrive om."* At the R2 go-gate (2026-06-24) the operator chose to +build R2 as **two sequenced slices, foundation-first**: R2a (this — the bridge + schema, pure trends/) before +R2b (the visible brief + surfacing). R2a delivers no *visible* change; it closes the capture loop so the +store accumulates publish-dated history that R2b's brief and R3's freshness window will rank on. + +## 2. The gap — grounded in code + +- **`normalize` (R1) and the store are disconnected.** `item.ts` validates a raw source item into the + canonical `TrendItem` envelope `{source,title,url,publishedAt?,topics[],summary?}` (`item.ts:21-38`), but + **nothing folds that envelope into the store.** The only ingress to the store is `cli.ts add` (`cli.ts:111-135`), + which builds a `TrendInput` from raw flags — it never calls `normalizeItem`, so a capture path that polls → + normalizes → stores **does not exist**. The bridge is the missing link R1 named and deferred. +- **`publishedAt` is carried but dropped.** The envelope carries the source's own publish date + (`item.ts:28-33`, "NOT persisted in R1"), but `TrendRecord` (`types.ts:26-41`) has no `publishedAt` field and + `addTrend` (`store.ts:111-130`) never persists it. So every capture **loses** the publish date — the field + B4 freshness (R3) and R2b's "fresh today" sort will both need. R2a persists it now, before history accumulates + undated, so the loop is publish-dated from this slice forward. +- **`SCHEMA_VERSION` is pinned at 1 with no migration path exercised.** `loadStore` (`store.ts:74-81`) reads + `parsed.schemaVersion ?? SCHEMA_VERSION` but the store has never had a second version, so the forward-migration + path is untested. Adding `publishedAt` is the first additive-optional schema bump (v1→v2) and the right moment + to prove a lossless migrate-on-load. + +## 3. Scope — what is IN (RE-R2a) + +### B-bridge — `itemToInput` (pure mapping, `scripts/trends/src/item.ts`, EDIT) +A pure `itemToInput(item: TrendItem, capturedAt: string): TrendInput` that maps the validated envelope to a +store input by **injecting `capturedAt`** (the store's "when WE saw it", supplied by the caller — never derived +inside the pure function) and **carrying `publishedAt`/`topics`/`summary`/`title`/`url`/`source` through verbatim**. +It does **not** re-validate (the item is already validated by `normalizeItem`) and does **not** derive an `id` +(the store owns id via `addTrend`→`trendId`). Lives in `item.ts` (it is the item's mapping to the store; `item.ts` +already imports from `store.ts`) — **not** a new single-function `bridge.ts` (single-use abstraction). + +### Schema migration — `publishedAt` persisted (`types.ts` + `store.ts`, EDIT) +- `types.ts`: `SCHEMA_VERSION 1 → 2`; add `publishedAt?: string` to `TrendRecord` (after `capturedAt`), with a + comment distinguishing it from `capturedAt` (source publish-date; forward-compat for B4 freshness). +- `store.ts`: `TrendInput` gains `publishedAt?: string`; `addTrend` persists it when present (same + conditional-spread idiom as `summary`, `store.ts:126`). **On re-capture/merge it is NOT overwritten** — + it joins the store's documented "FIRST sighting's source + capturedAt are kept (provenance of first sight)" + rule (`store.ts:108-110`); no back-fill (a deferred enhancement, §4). +- `loadStore`: **forward migrate-on-load** — stamp the in-memory store to `Math.max(onDisk, SCHEMA_VERSION)`, + where `onDisk = typeof parsed.schemaVersion === "number" ? parsed.schemaVersion : SCHEMA_VERSION` (a string / + `NaN` / absent version falls back to current, never crashes). v1→v2 is purely additive-optional (old records are + already valid v2 records that simply lack the optional field), so the migration is the stamp alone — **no record + rewrite; lossless + idempotent for any well-formed v1/v2 store**. A subsequent `saveStore` persists the v2 stamp. + **The existing non-array `trends` coercion (`store.ts:79`, `Array.isArray(parsed.trends) ? … : []`) is UNCHANGED + and out of R2a's migration scope** — losslessness is claimed only for well-formed stores, not for a corrupt + `trends` field. (No downgrade branch: a hypothetical future on-disk > current can't happen yet — Karpathy #2, no + handling for impossible scenarios; noted as a deferred risk.) + +### CLI `capture` subcommand (`scripts/trends/src/cli.ts`, EDIT) +`echo '' | node --import tsx src/cli.ts capture [--store ] [--json]` — reads the raw +JSON payload **from stdin** (same stdin contract as `normalize`/`score`, so `--json` stays an *output* toggle), +runs it through `normalizeItem`/`normalizeItems`, maps each valid item via `itemToInput(item, today())`, folds +each into the store via `addTrend`, and `saveStore`s once. Emits a summary — `{added, duplicates, merged, errors}` ++ store path/count — human-readable by default, JSON with `--json`. Content-invalid items are reported in the +summary (`errors[]`), **never via the exit code**; exit **2** only on a malformed invocation (unparseable/empty +stdin), matching `normalize`/`score`. Note the contract difference from `add`: `capture` items must carry `source` +(the normalizer requires it — no "manual" default), because a capture is from a real source. + +### Wiring + gate (see Open Question #1 — included by default, trimmable to minimal) +- `agents/trend-spotter.md` (EDIT): re-point **Step 4.5** (`trend-spotter.md:282-301`) from N× flag-based `add` to + a single batch `capture` — the agent builds a raw-item JSON batch (it already builds JSON for `score`) and pipes + it to `capture` in one call. Strictly better than `add` (it normalizes + carries `publishedAt`; one call, not N). + The replacement prose carries the literal `src/cli.ts capture`. Kept domain-general (no vendor/sector tokens). +- `scripts/test-runner.sh` (EDIT): bump `TRENDS_TESTS_FLOOR` (62 → **live recount**, stays inside the deps guard). + Add **Section 16h** ("Trends Capture Wiring"), mirroring Section 16g's three-check shape, placed **after Section + 17 / before Section 18** (anti-erosion last): (1) a non-vacuity self-test; (2) `grep` that `cli.ts` has a `capture` + handler (deps-absent proof the path exists); (3) `grep -qF "src/cli.ts capture" agents/trend-spotter.md`. These are + **unconditional** → bump `ASSERT_BASELINE_FLOOR` 87 → **live recount** (expected ~90). + +## 4. Non-goals — what is OUT (deferred) + +- **The dated morning-brief artifact (B3)** — R2b. *R2a closes the capture loop; R2b makes the stream visible.* +- **Session-start surfacing** of the brief (hull 4) — R2b. R2a does **not** touch `hooks/**`. +- **`publishedAt` back-fill on re-capture** (fill an absent existing `publishedAt` from a later sighting) — deferred; + first-sight provenance is kept, matching the existing merge rule. Revisit if undated-first-sight proves common. +- **`add --published-at` flag** — deferred (was a proposed "minor"; folded out at light-Voyage). The flag-based + `add` is the *manual* single-trend path; it stays publish-date-free for now. The store layer still GAINS + `publishedAt` (on `TrendInput` + `addTrend`) — that is what `capture` needs — but exposing it on the `add` CLI + (and the `isValidIso`-export-vs-inline decision it would force) is out of R2a. Trivial to add any later slice. +- **`relevance` / `saturation` / `status` / lifecycle fields** (hull 5) — R3. R2a adds **only** `publishedAt`. +- **Freshness window / dedup-vs-seen-log / autonomous trigger (B4)** — R3. +- **Research-deepening (A1–A4), adapter sub-agents, MCP fetch fan-out** — R2b/R3. +- **A store-reading brief / ranking on accumulated relevance** — R2b/R3 (needs persisted scores). + +## 5. Boundaries / invariants (must hold) + +- **TDD iron law:** the failing migration / `itemToInput` / `capture` tests land BEFORE the implementation. + RED proofs recorded (logic-RED, not import-RED). **NB:** only the **v1→v2** load + round-trip are genuinely RED + (old `loadStore` returns 1); v2-idempotence + missing/non-numeric-`schemaVersion` pass against old code, so they + are **GREEN-only regression guards**, not RED cases (see plan Step 1). +- **Lossless migration (well-formed stores):** every existing **well-formed** v1 store loads as v2 with **records + intact** (no `publishedAt` invented, no topic/summary/capturedAt/url/title change); idempotent (v2 → v2). A + corrupt `trends` field is coerced by the existing (unchanged) `Array.isArray` guard — explicitly **out of R2a's + losslessness claim**, not a regression introduced here. +- **First-sight provenance preserved:** re-capturing an existing trend never overwrites its `publishedAt`, + `capturedAt`, or `source` — only topics union (unchanged from R1). +- **No scoring change:** `score.ts` and `references/trend-scoring-modes.md` (the SSOT) are untouched — R2a is the + data layer, not the scorer. +- **Domain-general:** de-niche guard (Section 17) stays green; no vendor/sector tokens enter the edited + `trend-spotter.md` prose. +- **House conventions:** ESM + `node:test` + `tsx`; data-seam stays inline; **no new `.mjs` under `hooks/scripts/`** + (R2a touches no hook); `.gitignore` already covers `scripts/trends/{node_modules,build}`. +- **No new `references/*.md`**, **no new agent/command** (counts stay 19/29/27); no new `.ts` source file + (bridge lives in `item.ts`). Brief+plan live under `docs/` (uncounted), TRACKED like `docs/second-brain/*`. +- **Counts recounted live at land**, never pinned/guessed. + +## 6. Success criteria (testable) + +- **SC1 (bridge)** — `itemToInput(item, "2026-06-24")` returns a `TrendInput` with `capturedAt` injected (`=== "2026-06-24"`), + `publishedAt`/`topics`/`summary`/`title`/`url`/`source` carried verbatim, and **no `id`**. Absent `publishedAt` + on the item → absent on the input (key omitted, not `undefined`-valued). **Field-confusion guard:** for an item + whose `publishedAt` differs from the injected `capturedAt`, the result's `capturedAt !== result.publishedAt` + (proves the bridge never confuses the two dates — the whole point of the slice). +- **SC2 (persist)** — `addTrend` with a `publishedAt` persists it on the new record; a record without it omits the + key. On a re-capture (same title+url) the existing `publishedAt` is **unchanged** (first-sight kept), and only + topics union — `merged` reflects topic change alone. **No back-fill:** a re-capture carrying a `publishedAt` onto + a record that **lacked** one does **not** add it (Open Q#2) — the absent→present case is the one where "first-sight + kept" is counterintuitive, so it is pinned by a test. +- **SC3 (migrate)** — `loadStore` on a v1 store fixture (`{schemaVersion:1, trends:[…without publishedAt]}`) returns + `schemaVersion === 2` with every record intact and **no `publishedAt` invented**. Also pinned: a store with + **missing** `schemaVersion` → stamped 2 (records intact); an **empty/absent** store → `{schemaVersion:2, trends:[]}`; + a **non-numeric** `schemaVersion` (`"weird"`/`NaN`) → falls back to 2 (records intact); idempotent on a v2 store; + a round-trip `loadStore`→`saveStore` writes `schemaVersion: 2`. +- **SC4 (capture CLI)** — `echo '' | … capture` normalizes + folds: a well-formed item is added (or reported + duplicate/merged), a content-invalid item appears in the summary `errors[]`, the store file is written once, and + the summary counts are correct — **`added + merged + duplicates + errors.length === payload.length`** (the tally + is derived from `AddResult {added, merged}`, which has no `duplicates` field: `added`=`res.added`, + `merged`=`!res.added && res.merged`, `duplicates`=`!res.added && !res.merged`). The captured record's `capturedAt` + is a today-shaped ISO date **distinct from the item's `publishedAt`** (field-confusion guard at the ingress; the + exact-value assertion lives in `item.test.ts` with an injected date, since `capture` reads the wall clock). Exit + **2** on empty/unparseable stdin; **0** on a well-formed call even with content-invalid items. `--json` emits the + summary object. +- **SC5 (gate)** — `(cd scripts/trends && npm test)` green at the bumped `TRENDS_TESTS_FLOOR`; new Section 16h green; + `ASSERT_BASELINE_FLOOR` bumped to the live recount; overall gate `FAIL=0`. +- **SC6 (wiring + de-niche)** — `trend-spotter.md` references `src/cli.ts capture` (Section 16h grep green); de-niche + guard (Section 17) green; structure counts unchanged (19/29/27). *(If Open Q#1 → minimal: SC6 drops the wiring/16h + clauses; de-niche + counts still asserted.)* + +## 7. Verification + +**Deterministic (gate):** `bash scripts/test-runner.sh` → `FAIL=0`; trends suite ≥ new floor; new Section 16h +self-test + greps pass; Section 17 de-niche green; ref/agent/command counts unchanged. +**Behavioural (manual):** `echo '<2-item batch incl. one publishedAt + one invalid>' | node --import tsx src/cli.ts +capture --store /tmp/r2a-trends.json --json`; confirm the valid item lands with `publishedAt` persisted, the invalid +one is in `errors[]`, and `node … list --store /tmp/r2a-trends.json --json` shows the persisted `publishedAt`. +Then load an authored v1 fixture and confirm `status --json` reports it migrated (count intact). + +## 8. Open questions for the go-gate + +1. **Wire `trend-spotter.md` + add Section 16h, or keep R2a minimal? — RESOLVED at go-gate (operator, 2026-06-24): WIRE.** + Build R2a with the agent re-point (Step 4.5 `add`→`capture`) + Section 16h grep/self-test + ASSERT-floor bump. The + minimal alternative (bridge code + tests only, no agent edit) was declined — wiring mirrors R1's "the lift is real + and grep-able" discipline and makes the bridge actually used (one normalizing `capture` vs N× `add`). +2. **`publishedAt` merge policy.** Proposed **keep first-sight** (no back-fill), matching the existing source/capturedAt + provenance rule (now pinned by the absent→present SC2 test). Confirm, or request back-fill-if-absent (adds a branch + + a `merged` semantic question). +3. **`add --published-at` flag — RESOLVED at light-Voyage: deferred** (see §4). The manual `add` stays publish-date-free; + `capture` is the path that carries `publishedAt`. Noted here only so the resolution is traceable; no go-gate action. + +## 9. Light-Voyage review — folded + +Three Opus reviewers ran on the drafts, each verifying claims against live code. **scope-guardian: ALIGNED** +(every SC1–SC6 traces to a step; zero creep; all §4 non-goals held; 2 minors). **brief-reviewer: +PROCEED_WITH_RISKS** (2 majors, 4 minors). **plan-critic: REVISE** (1 blocker, 5 majors, 5 minors). All findings +folded; see `plan-re-r2a.md` §Plan-critic — folded for the per-step resolution: + +- **[BLOCKER, folded]** "v2 load idempotent" cannot be a RED case — old `loadStore` (`store.ts:78`) returns + `parsed.schemaVersion ?? SCHEMA_VERSION`, so a v2 fixture already loads as 2 against unchanged code. → §6 SC3 + + §5 now split the migration cases: **only v1→2 + round-trip are RED**; v2-idempotence + missing/non-numeric are + **GREEN-only regression guards**. +- **[MAJOR, folded]** "lossless / byte-for-byte intact" over-claimed vs `loadStore`'s non-array `trends` coercion + (`store.ts:79`). → §5 + §3 scope losslessness to **well-formed** v1/v2 stores; the coercion is unchanged + out of + scope (not a regression). +- **[MAJOR, folded]** `capturedAt` injection unverified at the new `capture` ingress (the field-confusion the slice + exists to prevent). → SC1 + SC4 now pin `capturedAt !== publishedAt` (item.test exact-value; cli.test shape + + distinct). +- **[MAJOR, folded]** `capture` tally `{added, duplicates, merged}` mis-maps onto `AddResult {added, merged}` (no + `duplicates` field). → SC4 pins the exact derivation + `sum === payload.length`. +- **[MAJOR, folded]** `add --published-at` validation referenced the non-exported `isValidIso` (`item.ts:51`). → + **`add --published-at` deferred entirely** (§4), removing the export-vs-inline decision and keeping `item.ts`'s + edit to `itemToInput` only. +- **[MAJOR, folded]** README "Capture … `add`" framing would contradict the agent's `add`→`capture` re-point. → + plan Step 5 now corrects the README framing (`add` = manual single-trend; `capture` = normalizing batch), not just + appends. +- **[MAJOR, folded]** empty "folded" placeholders shipped in the doc bodies. → this section + plan §Plan-critic now + filled. +- **[MINOR, folded in plan]** missing-/non-numeric-`schemaVersion` test, absent→present back-fill test, stub-replacement + note, `import type { TrendInput }` line, `grep -qF` for the 16h literal, `TRENDS_TESTS_FLOOR` breakdown-comment update, + README in the code commit, headless per-step clauses N/A (in-session execution). diff --git a/docs/research-engine/plan-re-r2a.md b/docs/research-engine/plan-re-r2a.md new file mode 100644 index 0000000..4143288 --- /dev/null +++ b/docs/research-engine/plan-re-r2a.md @@ -0,0 +1,192 @@ +# Plan — RE-R2a: capture bridge (item→store) + publishedAt persistence + +> **Brief:** `docs/research-engine/brief-re-r2a.md`. **Slice:** RE-R2a (research-engine rung-2, slice 2 — R2 data layer). +> **TDD-order:** RED (migration + bridge + capture tests as logic-RED) → GREEN (`types.ts`/`store.ts` migration + persist) → +> GREEN (`item.ts` bridge) → GREEN (`cli.ts capture` + tests) → wire `trend-spotter.md` → gate floors → behavioural → land. +> **Counts recounted live at land, never pinned/guessed.** +> **Light-Voyage hardened:** scope-guardian / brief-reviewer / plan-critic findings folded (see §Plan-critic — folded). + +## Goal + +Build the item→store bridge RE-R1 deferred: a pure `itemToInput` mapping the validated `TrendItem` envelope to a +store input (injecting `capturedAt`, carrying `publishedAt`), a `publishedAt` schema bump (v1→v2) with a lossless +migrate-on-load, and a `capture` CLI that closes the poll→normalize→store loop. No scoring change; no hook touch; the +dated brief + surfacing are R2b. + +## Files touched (exhaustive — for scope-guardian) + +| File | Change | SC | +|---|---|---| +| `scripts/trends/src/types.ts` | **EDIT** — `SCHEMA_VERSION 1→2`; `publishedAt?: string` on `TrendRecord` (after `capturedAt`, with the capturedAt-distinction comment) | SC3 | +| `scripts/trends/src/store.ts` | **EDIT** — `TrendInput` +`publishedAt?`; `addTrend` persists it (conditional-spread, first-sight kept on merge); `loadStore` forward migrate-on-load (`max(onDisk, SCHEMA_VERSION)`) | SC2, SC3 | +| `scripts/trends/src/item.ts` | **EDIT** — pure `itemToInput(item, capturedAt): TrendInput` (injects capturedAt; no id; no re-validate) | SC1 | +| `scripts/trends/src/cli.ts` | **EDIT** — `capture` subcommand only (stdin→normalize→bridge→addTrend→saveStore→summary; exit 2 bad invocation; `--json`). *(`add --published-at` deferred at light-Voyage — see brief §4.)* | SC4 | +| `scripts/trends/tests/store.test.ts` | **EDIT** — migration cases (v1→v2 lossless, idempotent, round-trip) + `publishedAt` persist/first-sight-merge cases | SC2, SC3 | +| `scripts/trends/tests/item.test.ts` | **EDIT** — `itemToInput` mapping cases (capturedAt inject, carry-through, no id, absent publishedAt omitted) | SC1 | +| `scripts/trends/tests/cli.test.ts` | **EDIT** — `capture` happy path (stdin→store), duplicate/merge, content-invalid in errors[], exit-2 bad invocation, `--json` summary | SC4 | +| `scripts/trends/README.md` | **EDIT** — add `publishedAt?` to the record-shape block + a `capture` example (honest schema/CLI doc) | — | +| `agents/trend-spotter.md` | **EDIT (Open Q#1, default)** — Step 4.5 `add`→`capture`; carries literal `src/cli.ts capture`; domain-general | SC6 | +| `scripts/test-runner.sh` | **EDIT** — `TRENDS_TESTS_FLOOR` 62→recount (stays inside deps guard); NEW unconditional **Section 16h** (before Section 18); `ASSERT_BASELINE_FLOOR` 87→recount; anti-erosion header enumeration | SC5, SC6 | +| `docs/research-engine/{brief,plan}-re-r2a.md` | **NEW** — slice docs (TRACKED, like `docs/second-brain/*`) | — | +| `STATE.md` | **EDIT at land** — Telling-block reconcile (trends floor, ASSERT floor, gate total). *Land bookkeeping, not slice scope; LOCAL-ONLY.* | — | + +**Not touched (scope fence):** `scripts/trends/src/score.ts` (no scoring change) · `references/trend-scoring-modes.md` + `references/*` (SSOT unchanged, no new ref) · `hooks/**` (no surfacing — R2b) · `config/trends-sources.template.md` · no new `.ts` source file (bridge in `item.ts`) · `agents/*` count (19) · `commands/*` (29) · `.gitignore` (trends lines present). + +## Step 1 — (RED) failing tests for migration + bridge + capture + +Extend the three existing test files against the not-yet-changed code. Logic-RED (not import-RED), stub-by-assertion-type. +**Critical RED-vs-GREEN-guard split** (plan-critic blocker): against old code (`SCHEMA_VERSION=1`, `loadStore` returns +`parsed.schemaVersion ?? SCHEMA_VERSION`), only some assertions actually fail: +- **Genuinely RED** (old code fails): v1-fixture load → `schemaVersion===2` (old returns 1); v1 round-trip + `loadStore`→`saveStore` writes `schemaVersion:2` (old writes 1); `addTrend({…, publishedAt})` persists it (old + drops it); `itemToInput` carries/injects correctly (add a thin **throwing** stub for the not-yet-existing export so + the case fails on assertion, not on `undefined is not a function`). +- **GREEN-only regression guards** (pass against old code — NOT labelled RED): a v2 fixture load → `schemaVersion===2` + (old already returns 2 via `?? `); **missing** `schemaVersion` → 2 (old `??` already yields current); **non-numeric** + `schemaVersion` → 2; empty/absent store → `{schemaVersion:2,trends:[]}`. These are written in Step 1 but documented + as regression guards, so the RED proof is not falsely claimed for them. + +`store.test.ts`: the RED migration cases above + the regression-guard cases; `addTrend` without `publishedAt` omits the +key; re-capture (same title+url) leaves existing `publishedAt` unchanged and only unions topics; **absent→present** +re-capture (first sighting lacked `publishedAt`, re-capture carries one) does **NOT** add it (no back-fill, Open Q#2), +`merged` reflects topic change alone. +`item.test.ts`: `itemToInput(item,"2026-06-24")` → `capturedAt` injected (`==="2026-06-24"`), all fields carried verbatim, +**no `id`**, item without `publishedAt` → input without the key; **field-confusion guard:** an item whose `publishedAt` +differs from the injected date → `result.capturedAt !== result.publishedAt`. +`cli.test.ts`: `capture` happy (one valid item piped → store gains it, `added:1`); a batch with one invalid item → +`errors[]` carries it, valid one added, exit 0; the summary tally **sums to the input size**; the captured record's +`capturedAt` matches `/^\d{4}-\d{2}-\d{2}$/` and `!==` the item's `publishedAt`; empty/unparseable stdin → exit 2; +`--json` emits the summary object. + +**RED proof (record in commit):** `(cd scripts/trends && npm test)` → the **genuinely-RED** cases fail on assertion +(logic-RED), not module-not-found; the regression-guard cases may pass pre-change (documented, not claimed RED). + +## Step 2 — (GREEN) schema migration: `types.ts` + `store.ts` + +`types.ts`: `SCHEMA_VERSION = 2`; add `publishedAt?: string` to `TrendRecord` after `capturedAt` with the +distinction comment (source publish-date; forward-compat for B4; distinct from capturedAt). +`store.ts`: `TrendInput` gains `publishedAt?`; `addTrend` adds `...(input.publishedAt !== undefined ? { publishedAt: input.publishedAt } : {})` to the new-record literal (after `capturedAt`); the merge branch is **unchanged** (topics union only — first-sight `publishedAt` kept, no back-fill). `loadStore` returns `schemaVersion: Math.max(onDisk, SCHEMA_VERSION)` where `onDisk = typeof parsed.schemaVersion === "number" ? parsed.schemaVersion : SCHEMA_VERSION` (forward-only stamp handling string/`NaN`/absent → current; never crashes); **the existing `Array.isArray(parsed.trends) ? … : []` coercion (`store.ts:79`) is preserved verbatim — a corrupt `trends` field stays out of the losslessness claim**. Make Step 1's RED migration + persist cases (and the regression guards) green. + +## Step 3 — (GREEN) bridge: `itemToInput` in `item.ts` + +Add `import type { TrendInput } from "./store.js";` (item.ts already imports the `normalizeField` *value* from there; +this adds the *type* — dependency direction `item.ts → store.ts` stays acyclic). Then **replace the Step-1 throwing +stub** with `export function itemToInput(item: TrendItem, capturedAt: string): TrendInput` returning +`{ source, title, url, topics: [...item.topics], capturedAt, ...(item.publishedAt !== undefined ? { publishedAt: item.publishedAt } : {}), ...(item.summary !== undefined ? { summary: item.summary } : {}) }`. No `id`; no re-validation (the envelope is already validated). Confirm no throwing stub survives into GREEN. Make Step 1's bridge cases green. + +## Step 4 — (GREEN) CLI `capture` + `cli.test.ts` + +Add `capture` to `cli.ts`'s `main` dispatch: read stdin via the existing `readStdinJson()` (its empty/unparseable path +already does `usage()`→exit 2); `Array.isArray(payload) ? normalizeItems(payload) : normalizeItem(payload)`; for each +valid item, `itemToInput(item, today())` → `addTrend(store, res.store…)`; `saveStore` once. **Tally derived from +`AddResult {added, merged}` (no `duplicates` field, `store.ts:35-41`):** `added += res.added ? 1 : 0`; +`merged += (!res.added && res.merged) ? 1 : 0`; `duplicates += (!res.added && !res.merged) ? 1 : 0`. Human summary by +default; `--json` prints `{added, duplicates, merged, errors}`. **`add --published-at` is NOT added (deferred).** Write +the `cli.test.ts` cases (subprocess: `node --import tsx src/cli.ts capture` with piped stdin + a `--store` temp path), +including an explicit `added + merged + duplicates + errors.length === payload.length` assertion and a +`capturedAt` shape (`/^\d{4}-\d{2}-\d{2}$/`) + `!== publishedAt` check. **`capturedAt` *exact-value* assertions live in +`item.test.ts` (injected fixed date), never in `cli.test.ts` (which reads the wall clock — would be flaky).** + +## Step 5 — wire `trend-spotter.md` (Open Q#1, default) + README + +Replace Step 4.5's N× `add` block (`trend-spotter.md:282-301`) with: build a raw-item JSON batch (the same items +already scored), pipe it to **`scripts/trends/src/cli.ts capture`** in one call — it normalizes + folds + persists +`publishedAt`. Replacement prose **must contain the literal `src/cli.ts capture`** (Section 16h `grep -qF`). Keep the +"skip silently if no deps" escape hatch + domain-general phrasing (no vendor/sector tokens — Section 17). Update +`scripts/trends/README.md`: add `publishedAt?` to the record-shape block, add a `capture` CLI example, **and correct +the `add` framing** — `README.md:47-64` currently calls `add` "the capture path"; after the re-point `add` is the +**manual single-trend** path and `capture` is the **normalizing batch** path. Fix the framing, don't just append (else +the README contradicts the new agent wiring). + +## Step 6 — gate: floors + new unconditional Section 16h + +In `scripts/test-runner.sh`: +- Bump `TRENDS_TESTS_FLOOR` 62 → **live recount** after Steps 1–4. Stays **inside** the `if [ -x …/tsx ]` deps guard + (conditional — do not hoist). **Also update the inline breakdown comment** at `test-runner.sh:695` + (`# store 24 + RE-R1: item 18 + score 16 + cli 4`) — extend it with the R2a additions (e.g. + `+ RE-R2a: store +N + item +M + cli +K`), so the comment doesn't drift from the number (the exact erosion the + anti-erosion section guards against). +- Add **Section 16h** ("Trends Capture Wiring"), mirroring Section 16g's three-check shape, **after Section 17 / + before Section 18**. Three **unconditional**, deps-absent-safe checks (pure `grep`/self-test, no `tsx`): + (1) a non-vacuity self-test (accept a probe carrying `src/cli.ts capture`, reject one without it); + (2) `grep -qF 'command === "capture"' scripts/trends/src/cli.ts` — deps-absent proof the handler exists (`grep -qF` + exact, mirroring 16g lines 1058/1064 — not `grep -E`); + (3) `grep -qF "src/cli.ts capture" agents/trend-spotter.md`. +- Bump `ASSERT_BASELINE_FLOOR` 87 → **live recount** (= 87 + the new unconditional emitters in 16h; expected ~90, + **recounted at land, not pinned**). Update the Section-18 anti-erosion header enumeration comment. + +## Step 7 — behavioural verification + +`(cd scripts/trends && npm install)` if needed, then: +`echo '[{"source":"tavily","title":"X","url":"https://e/x","topics":["a"],"publishedAt":"2026-06-20"},{"title":"bad"}]' | node --import tsx src/cli.ts capture --store /tmp/r2a-trends.json --json` → +confirm `added:1`, the invalid entry in `errors[]`, and `node … list --store /tmp/r2a-trends.json --json` shows the +persisted `publishedAt`. Author a v1 fixture (`{"schemaVersion":1,"trends":[{…no publishedAt}]}`), run `status --json` +→ confirm count intact (migration lossless). Run full `bash scripts/test-runner.sh` → `FAIL=0`. + +## Step 8 — land + +Recount all touched floors live; reconcile STATE.md "Telling" block (trends N/N, ASSERT floor, gate total). Commit +order (house style): **(1)** docs commit `docs/research-engine/{brief,plan}-re-r2a.md` (no suffix, tracked); **(2)** +code commit `scripts/trends/*` + `agents/trend-spotter.md` + `scripts/trends/README.md` + `scripts/test-runner.sh` +with `[skip-docs]`. Push freely (window restriction lifted 2026-06-24; gitleaks runs at commit; `origin` = PUBLIC +`open/` — STATE/`*.local.*` never pushed). No version bump (additive; `v0.5.2` dev). + +## Verification (testable) + +| SC | Check | Command | Expected | +|---|---|---|---| +| — | RED proof | `(cd scripts/trends && npm test)` after Step 1 | new cases fail on assertion (logic-RED), not module-not-found | +| SC1 | bridge | `npm test` (item.test.ts) | capturedAt injected (`==="2026-06-24"`); fields carried; no `id`; absent publishedAt omitted; `capturedAt !== publishedAt` (field-confusion guard) | +| SC2 | persist | `npm test` (store.test.ts) | publishedAt persisted when present / omitted when absent; first-sight kept on merge; absent→present re-capture does NOT back-fill | +| SC3 | migrate | `npm test` (store.test.ts) | v1→v2 lossless (records intact, no publishedAt invented); missing/non-numeric schemaVersion → 2; empty store → v2/[]; idempotent; round-trip writes v2 | +| SC4 | capture | `npm test` (cli.test.ts) + manual | normalize+fold; errors[] for invalid; tally sums to input size; capturedAt today-shaped & ≠ publishedAt; exit 2 bad stdin / 0 well-formed; `--json` summary | +| SC5 | gate | `bash scripts/test-runner.sh` | `FAIL=0`; trends ≥ new floor; Section 16h green; ASSERT floor = live recount | +| SC6 | wiring + de-niche | Section 16h greps + Section 17 | `src/cli.ts capture` in trend-spotter.md; no vendor/sector tokens; counts 19/29/27 | + +## Risks + +- **R1 — migration eats or rewrites records.** *Mitigated:* migrate-on-load is a version *stamp* only (`max(onDisk, + current)`); records pass through untouched; SC3 pins lossless + idempotent + round-trip on a real v1 fixture. +- **R2 — `publishedAt` first-sight vs back-fill ambiguity.** *Mitigated:* brief Open Q#2 settles it at first-sight + (no back-fill); SC2 pins "existing publishedAt unchanged on re-capture"; back-fill explicitly deferred. +- **R3 — `capture` exit-code semantics drift** (content-invalid item leaking into a non-zero exit). *Mitigated:* SC4 + pins exit 2 = malformed invocation only; content-invalid → `errors[]` at exit 0; mirrors `normalize`/`score`. +- **R4 — editing `trend-spotter.md` trips the de-niche guard.** *Mitigated:* Section 17 runs in the gate; replacement + prose is pillar/source-driven and vendor/sector-free; only the store-fold mechanism changes (`add`→`capture`). +- **R5 — new gate checks must survive a deps-absent fresh clone.** *Mitigated:* Section 16h is pure `grep`/self-test on + tracked source (no `tsx`) → unconditional; `TRENDS_TESTS_FLOOR` stays inside the deps guard. +- **R6 — future schema downgrade (on-disk > current) silently drops unknown fields on save.** *Mitigated/deferred:* + cannot happen pre-v3; `max()` already refuses to downgrade the stamp; field-preservation-on-save is an R-future + concern, noted not handled (no impossible-scenario code). + +## Plan-critic — folded + +plan-critic returned **REVISE** (1 blocker, 5 majors, 5 minors); brief-reviewer **PROCEED_WITH_RISKS**; +scope-guardian **ALIGNED**. Resolution, each verified against live code: + +- **[BLOCKER] "v2 load idempotent" is not RED** (`store.ts:78` `?? ` already returns 2 for a v2 fixture under old + `SCHEMA_VERSION=1`). ✅ Step 1 now splits **genuinely-RED** (v1→2 load + round-trip + addTrend-persist + itemToInput) + from **GREEN-only regression guards** (v2-idempotence, missing/non-numeric/empty); the RED proof is claimed only for + the former. +- **[MAJOR] capture tally mis-maps onto `AddResult`** (no `duplicates` field, `store.ts:35-41`). ✅ Step 4 pins the + exact derivation (`added`/`merged`/`duplicates` from `res.added`+`res.merged`) + a `sum === payload.length` test. +- **[MAJOR] `TRENDS_TESTS_FLOOR` breakdown comment (`:695`) left stale.** ✅ Step 6 now extends the inline breakdown + comment alongside the number. +- **[MAJOR] `import isValidIso` impossible** (`item.ts:51` private). ✅ `add --published-at` **deferred entirely** + (brief §4) — no export-vs-inline decision, `item.ts` edit stays `itemToInput`-only. +- **[MAJOR] README `add`-as-capture framing would contradict the re-point.** ✅ Step 5 corrects the framing + (`add` = manual single-trend; `capture` = normalizing batch), not just appends. +- **[MAJOR] empty "folded" placeholders.** ✅ this section + brief §9 filled. +- **[MINOR] stub cleanup unstated** ✅ Step 3 states the throwing stub is **replaced**, none survives into GREEN. + **[MINOR] `today()` capturedAt untestable in cli.test** ✅ Step 4 routes exact-value assertions to `item.test.ts`, + cli.test asserts shape + `≠ publishedAt`. **[MINOR] 16h grep flag** ✅ Step 6 specifies `grep -qF`. + **[MINOR] non-numeric schemaVersion untested** ✅ added to Step 1 regression guards + SC3. **[MINOR] README under + `[skip-docs]`** ✅ kept in the code commit (it documents the shipped code, like R1's `trend-spotter.md`); noted. +- **[brief-reviewer MAJOR] capturedAt injection unverified at the new ingress** ✅ SC1 + SC4 field-confusion guards. + **[brief-reviewer MAJOR] "lossless" over-claim** ✅ §5/§3 scope it to well-formed stores; malformed-`trends` + coercion unchanged + out of scope. **[brief-reviewer MINOR] absent→present back-fill** ✅ pinned by an SC2 test. +- **[plan-critic headless-readiness 60]** N/A — R2a is executed **in-session, operator-driven** (driftsmodell), not + as a headless autonomous run, so per-step revert/halt clauses aren't needed (R1's plan had none either). + +**scope-guardian — ALIGNED:** every SC1–SC6 traces to a step; zero creep (README is in-change documentation); every +§4 non-goal held (no hook touch, no brief artifact, no relevance/saturation/status, no back-fill, no scoring change).