linkedin-studio/docs/research-engine/plan-re-r2a.md

# Plan — RE-R2a: capture bridge (item→store) + publishedAt persistence

> **Brief:** `docs/research-engine/brief-re-r2a.md`. **Slice:** RE-R2a (research-engine rung-2, slice 2 — R2 data layer).
> **TDD-order:** RED (migration + bridge + capture tests as logic-RED) → GREEN (`types.ts`/`store.ts` migration + persist) →
> GREEN (`item.ts` bridge) → GREEN (`cli.ts capture` + tests) → wire `trend-spotter.md` → gate floors → behavioural → land.
> **Counts recounted live at land, never pinned/guessed.**
> **Light-Voyage hardened:** scope-guardian / brief-reviewer / plan-critic findings folded (see §Plan-critic — folded).

## Goal

Build the item→store bridge RE-R1 deferred: a pure `itemToInput` mapping the validated `TrendItem` envelope to a
store input (injecting `capturedAt`, carrying `publishedAt`), a `publishedAt` schema bump (v1→v2) with a lossless
migrate-on-load, and a `capture` CLI that closes the poll→normalize→store loop. No scoring change; no hook touch; the
dated brief + surfacing are R2b.

## Files touched (exhaustive — for scope-guardian)

| File | Change | SC |
|---|---|---|
| `scripts/trends/src/types.ts` | **EDIT** — `SCHEMA_VERSION 1→2`; `publishedAt?: string` on `TrendRecord` (after `capturedAt`, with the capturedAt-distinction comment) | SC3 |
| `scripts/trends/src/store.ts` | **EDIT** — `TrendInput` +`publishedAt?`; `addTrend` persists it (conditional-spread, first-sight kept on merge); `loadStore` forward migrate-on-load (`max(onDisk, SCHEMA_VERSION)`) | SC2, SC3 |
| `scripts/trends/src/item.ts` | **EDIT** — pure `itemToInput(item, capturedAt): TrendInput` (injects capturedAt; no id; no re-validate) | SC1 |
| `scripts/trends/src/cli.ts` | **EDIT** — `capture` subcommand only (stdin→normalize→bridge→addTrend→saveStore→summary; exit 2 bad invocation; `--json`). *(`add --published-at` deferred at light-Voyage — see brief §4.)* | SC4 |
| `scripts/trends/tests/store.test.ts` | **EDIT** — migration cases (v1→v2 lossless, idempotent, round-trip) + `publishedAt` persist/first-sight-merge cases | SC2, SC3 |
| `scripts/trends/tests/item.test.ts` | **EDIT** — `itemToInput` mapping cases (capturedAt inject, carry-through, no id, absent publishedAt omitted) | SC1 |
| `scripts/trends/tests/cli.test.ts` | **EDIT** — `capture` happy path (stdin→store), duplicate/merge, content-invalid in errors[], exit-2 bad invocation, `--json` summary | SC4 |
| `scripts/trends/README.md` | **EDIT** — add `publishedAt?` to the record-shape block + a `capture` example (honest schema/CLI doc) | — |
| `agents/trend-spotter.md` | **EDIT (Open Q#1, default)** — Step 4.5 `add`→`capture`; carries literal `src/cli.ts capture`; domain-general | SC6 |
| `scripts/test-runner.sh` | **EDIT** — `TRENDS_TESTS_FLOOR` 62→recount (stays inside deps guard); NEW unconditional **Section 16h** (before Section 18); `ASSERT_BASELINE_FLOOR` 87→recount; anti-erosion header enumeration | SC5, SC6 |
| `docs/research-engine/{brief,plan}-re-r2a.md` | **NEW** — slice docs (TRACKED, like `docs/second-brain/*`) | — |
| `STATE.md` | **EDIT at land** — Telling-block reconcile (trends floor, ASSERT floor, gate total). *Land bookkeeping, not slice scope; LOCAL-ONLY.* | — |

**Not touched (scope fence):** `scripts/trends/src/score.ts` (no scoring change) · `references/trend-scoring-modes.md` + `references/*` (SSOT unchanged, no new ref) · `hooks/**` (no surfacing — R2b) · `config/trends-sources.template.md` · no new `.ts` source file (bridge in `item.ts`) · `agents/*` count (19) · `commands/*` (29) · `.gitignore` (trends lines present).

## Step 1 — (RED) failing tests for migration + bridge + capture

Extend the three existing test files against the not-yet-changed code. Logic-RED (not import-RED), stub-by-assertion-type.
**Critical RED-vs-GREEN-guard split** (plan-critic blocker): against old code (`SCHEMA_VERSION=1`, `loadStore` returns
`parsed.schemaVersion ?? SCHEMA_VERSION`), only some assertions actually fail:
- **Genuinely RED** (old code fails): v1-fixture load → `schemaVersion===2` (old returns 1); v1 round-trip
  `loadStore`→`saveStore` writes `schemaVersion:2` (old writes 1); `addTrend({…, publishedAt})` persists it (old
  drops it); `itemToInput` carries/injects correctly (add a thin **throwing** stub for the not-yet-existing export so
  the case fails on assertion, not on `undefined is not a function`).
- **GREEN-only regression guards** (pass against old code — NOT labelled RED): a v2 fixture load → `schemaVersion===2`
  (old already returns 2 via `?? `); **missing** `schemaVersion` → 2 (old `??` already yields current); **non-numeric**
  `schemaVersion` → 2; empty/absent store → `{schemaVersion:2,trends:[]}`. These are written in Step 1 but documented
  as regression guards, so the RED proof is not falsely claimed for them.

`store.test.ts`: the RED migration cases above + the regression-guard cases; `addTrend` without `publishedAt` omits the
key; re-capture (same title+url) leaves existing `publishedAt` unchanged and only unions topics; **absent→present**
re-capture (first sighting lacked `publishedAt`, re-capture carries one) does **NOT** add it (no back-fill, Open Q#2),
`merged` reflects topic change alone.
`item.test.ts`: `itemToInput(item,"2026-06-24")` → `capturedAt` injected (`==="2026-06-24"`), all fields carried verbatim,
**no `id`**, item without `publishedAt` → input without the key; **field-confusion guard:** an item whose `publishedAt`
differs from the injected date → `result.capturedAt !== result.publishedAt`.
`cli.test.ts`: `capture` happy (one valid item piped → store gains it, `added:1`); a batch with one invalid item →
`errors[]` carries it, valid one added, exit 0; the summary tally **sums to the input size**; the captured record's
`capturedAt` matches `/^\d{4}-\d{2}-\d{2}$/` and `!==` the item's `publishedAt`; empty/unparseable stdin → exit 2;
`--json` emits the summary object.

**RED proof (record in commit):** `(cd scripts/trends && npm test)` → the **genuinely-RED** cases fail on assertion
(logic-RED), not module-not-found; the regression-guard cases may pass pre-change (documented, not claimed RED).

## Step 2 — (GREEN) schema migration: `types.ts` + `store.ts`

`types.ts`: `SCHEMA_VERSION = 2`; add `publishedAt?: string` to `TrendRecord` after `capturedAt` with the
distinction comment (source publish-date; forward-compat for B4; distinct from capturedAt).
`store.ts`: `TrendInput` gains `publishedAt?`; `addTrend` adds `...(input.publishedAt !== undefined ? { publishedAt: input.publishedAt } : {})` to the new-record literal (after `capturedAt`); the merge branch is **unchanged** (topics union only — first-sight `publishedAt` kept, no back-fill). `loadStore` returns `schemaVersion: Math.max(onDisk, SCHEMA_VERSION)` where `onDisk = typeof parsed.schemaVersion === "number" ? parsed.schemaVersion : SCHEMA_VERSION` (forward-only stamp handling string/`NaN`/absent → current; never crashes); **the existing `Array.isArray(parsed.trends) ? … : []` coercion (`store.ts:79`) is preserved verbatim — a corrupt `trends` field stays out of the losslessness claim**. Make Step 1's RED migration + persist cases (and the regression guards) green.

## Step 3 — (GREEN) bridge: `itemToInput` in `item.ts`

Add `import type { TrendInput } from "./store.js";` (item.ts already imports the `normalizeField` *value* from there;
this adds the *type* — dependency direction `item.ts → store.ts` stays acyclic). Then **replace the Step-1 throwing
stub** with `export function itemToInput(item: TrendItem, capturedAt: string): TrendInput` returning
`{ source, title, url, topics: [...item.topics], capturedAt, ...(item.publishedAt !== undefined ? { publishedAt: item.publishedAt } : {}), ...(item.summary !== undefined ? { summary: item.summary } : {}) }`. No `id`; no re-validation (the envelope is already validated). Confirm no throwing stub survives into GREEN. Make Step 1's bridge cases green.

## Step 4 — (GREEN) CLI `capture` + `cli.test.ts`

Add `capture` to `cli.ts`'s `main` dispatch: read stdin via the existing `readStdinJson()` (its empty/unparseable path
already does `usage()`→exit 2); `Array.isArray(payload) ? normalizeItems(payload) : normalizeItem(payload)`; for each
valid item, `itemToInput(item, today())` → `addTrend(store, res.store…)`; `saveStore` once. **Tally derived from
`AddResult {added, merged}` (no `duplicates` field, `store.ts:35-41`):** `added += res.added ? 1 : 0`;
`merged += (!res.added && res.merged) ? 1 : 0`; `duplicates += (!res.added && !res.merged) ? 1 : 0`. Human summary by
default; `--json` prints `{added, duplicates, merged, errors}`. **`add --published-at` is NOT added (deferred).** Write
the `cli.test.ts` cases (subprocess: `node --import tsx src/cli.ts capture` with piped stdin + a `--store` temp path),
including an explicit `added + merged + duplicates + errors.length === payload.length` assertion and a
`capturedAt` shape (`/^\d{4}-\d{2}-\d{2}$/`) + `!== publishedAt` check. **`capturedAt` *exact-value* assertions live in
`item.test.ts` (injected fixed date), never in `cli.test.ts` (which reads the wall clock — would be flaky).**

## Step 5 — wire `trend-spotter.md` (Open Q#1, default) + README

Replace Step 4.5's N× `add` block (`trend-spotter.md:282-301`) with: build a raw-item JSON batch (the same items
already scored), pipe it to **`scripts/trends/src/cli.ts capture`** in one call — it normalizes + folds + persists
`publishedAt`. Replacement prose **must contain the literal `src/cli.ts capture`** (Section 16h `grep -qF`). Keep the
"skip silently if no deps" escape hatch + domain-general phrasing (no vendor/sector tokens — Section 17). Update
`scripts/trends/README.md`: add `publishedAt?` to the record-shape block, add a `capture` CLI example, **and correct
the `add` framing** — `README.md:47-64` currently calls `add` "the capture path"; after the re-point `add` is the
**manual single-trend** path and `capture` is the **normalizing batch** path. Fix the framing, don't just append (else
the README contradicts the new agent wiring).

## Step 6 — gate: floors + new unconditional Section 16h

In `scripts/test-runner.sh`:
- Bump `TRENDS_TESTS_FLOOR` 62 → **live recount** after Steps 1–4. Stays **inside** the `if [ -x …/tsx ]` deps guard
  (conditional — do not hoist). **Also update the inline breakdown comment** at `test-runner.sh:695`
  (`# store 24 + RE-R1: item 18 + score 16 + cli 4`) — extend it with the R2a additions (e.g.
  `+ RE-R2a: store +N + item +M + cli +K`), so the comment doesn't drift from the number (the exact erosion the
  anti-erosion section guards against).
- Add **Section 16h** ("Trends Capture Wiring"), mirroring Section 16g's three-check shape, **after Section 17 /
  before Section 18**. Three **unconditional**, deps-absent-safe checks (pure `grep`/self-test, no `tsx`):
  (1) a non-vacuity self-test (accept a probe carrying `src/cli.ts capture`, reject one without it);
  (2) `grep -qF 'command === "capture"' scripts/trends/src/cli.ts` — deps-absent proof the handler exists (`grep -qF`
  exact, mirroring 16g lines 1058/1064 — not `grep -E`);
  (3) `grep -qF "src/cli.ts capture" agents/trend-spotter.md`.
- Bump `ASSERT_BASELINE_FLOOR` 87 → **live recount** (= 87 + the new unconditional emitters in 16h; expected ~90,
  **recounted at land, not pinned**). Update the Section-18 anti-erosion header enumeration comment.

## Step 7 — behavioural verification

`(cd scripts/trends && npm install)` if needed, then:
`echo '[{"source":"tavily","title":"X","url":"https://e/x","topics":["a"],"publishedAt":"2026-06-20"},{"title":"bad"}]' | node --import tsx src/cli.ts capture --store /tmp/r2a-trends.json --json` →
confirm `added:1`, the invalid entry in `errors[]`, and `node … list --store /tmp/r2a-trends.json --json` shows the
persisted `publishedAt`. Author a v1 fixture (`{"schemaVersion":1,"trends":[{…no publishedAt}]}`), run `status --json`
→ confirm count intact (migration lossless). Run full `bash scripts/test-runner.sh` → `FAIL=0`.

## Step 8 — land

Recount all touched floors live; reconcile STATE.md "Telling" block (trends N/N, ASSERT floor, gate total). Commit
order (house style): **(1)** docs commit `docs/research-engine/{brief,plan}-re-r2a.md` (no suffix, tracked); **(2)**
code commit `scripts/trends/*` + `agents/trend-spotter.md` + `scripts/trends/README.md` + `scripts/test-runner.sh`
with `[skip-docs]`. Push freely (window restriction lifted 2026-06-24; gitleaks runs at commit; `origin` = PUBLIC
`open/` — STATE/`*.local.*` never pushed). No version bump (additive; `v0.5.2` dev).

## Verification (testable)

| SC | Check | Command | Expected |
|---|---|---|---|
| — | RED proof | `(cd scripts/trends && npm test)` after Step 1 | new cases fail on assertion (logic-RED), not module-not-found |
| SC1 | bridge | `npm test` (item.test.ts) | capturedAt injected (`==="2026-06-24"`); fields carried; no `id`; absent publishedAt omitted; `capturedAt !== publishedAt` (field-confusion guard) |
| SC2 | persist | `npm test` (store.test.ts) | publishedAt persisted when present / omitted when absent; first-sight kept on merge; absent→present re-capture does NOT back-fill |
| SC3 | migrate | `npm test` (store.test.ts) | v1→v2 lossless (records intact, no publishedAt invented); missing/non-numeric schemaVersion → 2; empty store → v2/[]; idempotent; round-trip writes v2 |
| SC4 | capture | `npm test` (cli.test.ts) + manual | normalize+fold; errors[] for invalid; tally sums to input size; capturedAt today-shaped & ≠ publishedAt; exit 2 bad stdin / 0 well-formed; `--json` summary |
| SC5 | gate | `bash scripts/test-runner.sh` | `FAIL=0`; trends ≥ new floor; Section 16h green; ASSERT floor = live recount |
| SC6 | wiring + de-niche | Section 16h greps + Section 17 | `src/cli.ts capture` in trend-spotter.md; no vendor/sector tokens; counts 19/29/27 |

## Risks

- **R1 — migration eats or rewrites records.** *Mitigated:* migrate-on-load is a version *stamp* only (`max(onDisk,
  current)`); records pass through untouched; SC3 pins lossless + idempotent + round-trip on a real v1 fixture.
- **R2 — `publishedAt` first-sight vs back-fill ambiguity.** *Mitigated:* brief Open Q#2 settles it at first-sight
  (no back-fill); SC2 pins "existing publishedAt unchanged on re-capture"; back-fill explicitly deferred.
- **R3 — `capture` exit-code semantics drift** (content-invalid item leaking into a non-zero exit). *Mitigated:* SC4
  pins exit 2 = malformed invocation only; content-invalid → `errors[]` at exit 0; mirrors `normalize`/`score`.
- **R4 — editing `trend-spotter.md` trips the de-niche guard.** *Mitigated:* Section 17 runs in the gate; replacement
  prose is pillar/source-driven and vendor/sector-free; only the store-fold mechanism changes (`add`→`capture`).
- **R5 — new gate checks must survive a deps-absent fresh clone.** *Mitigated:* Section 16h is pure `grep`/self-test on
  tracked source (no `tsx`) → unconditional; `TRENDS_TESTS_FLOOR` stays inside the deps guard.
- **R6 — future schema downgrade (on-disk > current) silently drops unknown fields on save.** *Mitigated/deferred:*
  cannot happen pre-v3; `max()` already refuses to downgrade the stamp; field-preservation-on-save is an R-future
  concern, noted not handled (no impossible-scenario code).

## Plan-critic — folded

plan-critic returned **REVISE** (1 blocker, 5 majors, 5 minors); brief-reviewer **PROCEED_WITH_RISKS**;
scope-guardian **ALIGNED**. Resolution, each verified against live code:

- **[BLOCKER] "v2 load idempotent" is not RED** (`store.ts:78` `?? ` already returns 2 for a v2 fixture under old
  `SCHEMA_VERSION=1`). ✅ Step 1 now splits **genuinely-RED** (v1→2 load + round-trip + addTrend-persist + itemToInput)
  from **GREEN-only regression guards** (v2-idempotence, missing/non-numeric/empty); the RED proof is claimed only for
  the former.
- **[MAJOR] capture tally mis-maps onto `AddResult`** (no `duplicates` field, `store.ts:35-41`). ✅ Step 4 pins the
  exact derivation (`added`/`merged`/`duplicates` from `res.added`+`res.merged`) + a `sum === payload.length` test.
- **[MAJOR] `TRENDS_TESTS_FLOOR` breakdown comment (`:695`) left stale.** ✅ Step 6 now extends the inline breakdown
  comment alongside the number.
- **[MAJOR] `import isValidIso` impossible** (`item.ts:51` private). ✅ `add --published-at` **deferred entirely**
  (brief §4) — no export-vs-inline decision, `item.ts` edit stays `itemToInput`-only.
- **[MAJOR] README `add`-as-capture framing would contradict the re-point.** ✅ Step 5 corrects the framing
  (`add` = manual single-trend; `capture` = normalizing batch), not just appends.
- **[MAJOR] empty "folded" placeholders.** ✅ this section + brief §9 filled.
- **[MINOR] stub cleanup unstated** ✅ Step 3 states the throwing stub is **replaced**, none survives into GREEN.
  **[MINOR] `today()` capturedAt untestable in cli.test** ✅ Step 4 routes exact-value assertions to `item.test.ts`,
  cli.test asserts shape + `≠ publishedAt`. **[MINOR] 16h grep flag** ✅ Step 6 specifies `grep -qF`.
  **[MINOR] non-numeric schemaVersion untested** ✅ added to Step 1 regression guards + SC3. **[MINOR] README under
  `[skip-docs]`** ✅ kept in the code commit (it documents the shipped code, like R1's `trend-spotter.md`); noted.
- **[brief-reviewer MAJOR] capturedAt injection unverified at the new ingress** ✅ SC1 + SC4 field-confusion guards.
  **[brief-reviewer MAJOR] "lossless" over-claim** ✅ §5/§3 scope it to well-formed stores; malformed-`trends`
  coercion unchanged + out of scope. **[brief-reviewer MINOR] absent→present back-fill** ✅ pinned by an SC2 test.
- **[plan-critic headless-readiness 60]** N/A — R2a is executed **in-session, operator-driven** (driftsmodell), not
  as a headless autonomous run, so per-step revert/halt clauses aren't needed (R1's plan had none either).

**scope-guardian — ALIGNED:** every SC1–SC6 traces to a step; zero creep (README is in-change documentation); every
§4 non-goal held (no hook touch, no brief artifact, no relevance/saturation/status, no back-fill, no scoring change).