# Brief — RE-R2a: capture bridge (item→store) + publishedAt persistence

> **Slice:** RE-R2a (research-engine rung-2, slice 2 — the data layer of R2). The first of the two
> halves R2 ("the visible topic-stream") was split into at the 2026-06-24 go-gate: **R2a** = the pure
> `scripts/trends/` data layer (this brief); **R2b** = the dated morning-brief artifact + session-start
> surfacing (separate brief, after R2a lands). The split was chosen because (b)+(c) are code-independent
> of (a), R2a mirrors R1's pure-TDD shape exactly (lowest risk), and it closes the capture loop the store
> was built for.
> **Predecessor:** RE-R1 (`brief-re-r1.md`) delivered the validated ingress envelope (`item.ts`:
> `normalizeItem`/`normalizeItems`) and the triage scorer (`score.ts`) behind a CLI seam, and **explicitly
> deferred the item→store bridge** ("injecting the store's `capturedAt` and persisting `publishedAt` — is
> R2 orchestration", `brief-re-r1.md` §3 B1). R2a builds exactly that deferred bridge.
> **Substrate:** `docs/research-engine-concepts.local.md` §1 (hull 3 "store-schema mangler felt" / the
> item→store gap), §3 B1 (one schema downstream never branches on). **TDD-order:** RED tests land before code.

## 1. Operator decision context (2026-06-24)

The research engine is **Tier-1** (operator, 2026-06-23): *"hele min arbeidsflyt hviler på at jeg får en
jevn strøm av gode forslag til tema å skrive om."* At the R2 go-gate (2026-06-24) the operator chose to
build R2 as **two sequenced slices, foundation-first**: R2a (this — the bridge + schema, pure trends/) before
R2b (the visible brief + surfacing). R2a delivers no *visible* change; it closes the capture loop so the
store accumulates publish-dated history that R2b's brief and R3's freshness window will rank on.

## 2. The gap — grounded in code

- **`normalize` (R1) and the store are disconnected.** `item.ts` validates a raw source item into the
  canonical `TrendItem` envelope `{source,title,url,publishedAt?,topics[],summary?}` (`item.ts:21-38`), but
  **nothing folds that envelope into the store.** The only ingress to the store is `cli.ts add` (`cli.ts:111-135`),
  which builds a `TrendInput` from raw flags — it never calls `normalizeItem`, so a capture path that polls →
  normalizes → stores **does not exist**. The bridge is the missing link R1 named and deferred.
- **`publishedAt` is carried but dropped.** The envelope carries the source's own publish date
  (`item.ts:28-33`, "NOT persisted in R1"), but `TrendRecord` (`types.ts:26-41`) has no `publishedAt` field and
  `addTrend` (`store.ts:111-130`) never persists it. So every capture **loses** the publish date — the field
  B4 freshness (R3) and R2b's "fresh today" sort will both need. R2a persists it now, before history accumulates
  undated, so the loop is publish-dated from this slice forward.
- **`SCHEMA_VERSION` is pinned at 1 with no migration path exercised.** `loadStore` (`store.ts:74-81`) reads
  `parsed.schemaVersion ?? SCHEMA_VERSION` but the store has never had a second version, so the forward-migration
  path is untested. Adding `publishedAt` is the first additive-optional schema bump (v1→v2) and the right moment
  to prove a lossless migrate-on-load.

## 3. Scope — what is IN (RE-R2a)

### B-bridge — `itemToInput` (pure mapping, `scripts/trends/src/item.ts`, EDIT)
A pure `itemToInput(item: TrendItem, capturedAt: string): TrendInput` that maps the validated envelope to a
store input by **injecting `capturedAt`** (the store's "when WE saw it", supplied by the caller — never derived
inside the pure function) and **carrying `publishedAt`/`topics`/`summary`/`title`/`url`/`source` through verbatim**.
It does **not** re-validate (the item is already validated by `normalizeItem`) and does **not** derive an `id`
(the store owns id via `addTrend`→`trendId`). Lives in `item.ts` (it is the item's mapping to the store; `item.ts`
already imports from `store.ts`) — **not** a new single-function `bridge.ts` (single-use abstraction).

### Schema migration — `publishedAt` persisted (`types.ts` + `store.ts`, EDIT)
- `types.ts`: `SCHEMA_VERSION 1 → 2`; add `publishedAt?: string` to `TrendRecord` (after `capturedAt`), with a
  comment distinguishing it from `capturedAt` (source publish-date; forward-compat for B4 freshness).
- `store.ts`: `TrendInput` gains `publishedAt?: string`; `addTrend` persists it when present (same
  conditional-spread idiom as `summary`, `store.ts:126`). **On re-capture/merge it is NOT overwritten** —
  it joins the store's documented "FIRST sighting's source + capturedAt are kept (provenance of first sight)"
  rule (`store.ts:108-110`); no back-fill (a deferred enhancement, §4).
- `loadStore`: **forward migrate-on-load** — stamp the in-memory store to `Math.max(onDisk, SCHEMA_VERSION)`,
  where `onDisk = typeof parsed.schemaVersion === "number" ? parsed.schemaVersion : SCHEMA_VERSION` (a string /
  `NaN` / absent version falls back to current, never crashes). v1→v2 is purely additive-optional (old records are
  already valid v2 records that simply lack the optional field), so the migration is the stamp alone — **no record
  rewrite; lossless + idempotent for any well-formed v1/v2 store**. A subsequent `saveStore` persists the v2 stamp.
  **The existing non-array `trends` coercion (`store.ts:79`, `Array.isArray(parsed.trends) ? … : []`) is UNCHANGED
  and out of R2a's migration scope** — losslessness is claimed only for well-formed stores, not for a corrupt
  `trends` field. (No downgrade branch: a hypothetical future on-disk > current can't happen yet — Karpathy #2, no
  handling for impossible scenarios; noted as a deferred risk.)

### CLI `capture` subcommand (`scripts/trends/src/cli.ts`, EDIT)
`echo '<raw item | batch>' | node --import tsx src/cli.ts capture [--store <path>] [--json]` — reads the raw
JSON payload **from stdin** (same stdin contract as `normalize`/`score`, so `--json` stays an *output* toggle),
runs it through `normalizeItem`/`normalizeItems`, maps each valid item via `itemToInput(item, today())`, folds
each into the store via `addTrend`, and `saveStore`s once. Emits a summary — `{added, duplicates, merged, errors}`
+ store path/count — human-readable by default, JSON with `--json`. Content-invalid items are reported in the
summary (`errors[]`), **never via the exit code**; exit **2** only on a malformed invocation (unparseable/empty
stdin), matching `normalize`/`score`. Note the contract difference from `add`: `capture` items must carry `source`
(the normalizer requires it — no "manual" default), because a capture is from a real source.

### Wiring + gate (see Open Question #1 — included by default, trimmable to minimal)
- `agents/trend-spotter.md` (EDIT): re-point **Step 4.5** (`trend-spotter.md:282-301`) from N× flag-based `add` to
  a single batch `capture` — the agent builds a raw-item JSON batch (it already builds JSON for `score`) and pipes
  it to `capture` in one call. Strictly better than `add` (it normalizes + carries `publishedAt`; one call, not N).
  The replacement prose carries the literal `src/cli.ts capture`. Kept domain-general (no vendor/sector tokens).
- `scripts/test-runner.sh` (EDIT): bump `TRENDS_TESTS_FLOOR` (62 → **live recount**, stays inside the deps guard).
  Add **Section 16h** ("Trends Capture Wiring"), mirroring Section 16g's three-check shape, placed **after Section
  17 / before Section 18** (anti-erosion last): (1) a non-vacuity self-test; (2) `grep` that `cli.ts` has a `capture`
  handler (deps-absent proof the path exists); (3) `grep -qF "src/cli.ts capture" agents/trend-spotter.md`. These are
  **unconditional** → bump `ASSERT_BASELINE_FLOOR` 87 → **live recount** (expected ~90).

## 4. Non-goals — what is OUT (deferred)

- **The dated morning-brief artifact (B3)** — R2b. *R2a closes the capture loop; R2b makes the stream visible.*
- **Session-start surfacing** of the brief (hull 4) — R2b. R2a does **not** touch `hooks/**`.
- **`publishedAt` back-fill on re-capture** (fill an absent existing `publishedAt` from a later sighting) — deferred;
  first-sight provenance is kept, matching the existing merge rule. Revisit if undated-first-sight proves common.
- **`add --published-at` flag** — deferred (was a proposed "minor"; folded out at light-Voyage). The flag-based
  `add` is the *manual* single-trend path; it stays publish-date-free for now. The store layer still GAINS
  `publishedAt` (on `TrendInput` + `addTrend`) — that is what `capture` needs — but exposing it on the `add` CLI
  (and the `isValidIso`-export-vs-inline decision it would force) is out of R2a. Trivial to add any later slice.
- **`relevance` / `saturation` / `status` / lifecycle fields** (hull 5) — R3. R2a adds **only** `publishedAt`.
- **Freshness window / dedup-vs-seen-log / autonomous trigger (B4)** — R3.
- **Research-deepening (A1–A4), adapter sub-agents, MCP fetch fan-out** — R2b/R3.
- **A store-reading brief / ranking on accumulated relevance** — R2b/R3 (needs persisted scores).

## 5. Boundaries / invariants (must hold)

- **TDD iron law:** the failing migration / `itemToInput` / `capture` tests land BEFORE the implementation.
  RED proofs recorded (logic-RED, not import-RED). **NB:** only the **v1→v2** load + round-trip are genuinely RED
  (old `loadStore` returns 1); v2-idempotence + missing/non-numeric-`schemaVersion` pass against old code, so they
  are **GREEN-only regression guards**, not RED cases (see plan Step 1).
- **Lossless migration (well-formed stores):** every existing **well-formed** v1 store loads as v2 with **records
  intact** (no `publishedAt` invented, no topic/summary/capturedAt/url/title change); idempotent (v2 → v2). A
  corrupt `trends` field is coerced by the existing (unchanged) `Array.isArray` guard — explicitly **out of R2a's
  losslessness claim**, not a regression introduced here.
- **First-sight provenance preserved:** re-capturing an existing trend never overwrites its `publishedAt`,
  `capturedAt`, or `source` — only topics union (unchanged from R1).
- **No scoring change:** `score.ts` and `references/trend-scoring-modes.md` (the SSOT) are untouched — R2a is the
  data layer, not the scorer.
- **Domain-general:** de-niche guard (Section 17) stays green; no vendor/sector tokens enter the edited
  `trend-spotter.md` prose.
- **House conventions:** ESM + `node:test` + `tsx`; data-seam stays inline; **no new `.mjs` under `hooks/scripts/`**
  (R2a touches no hook); `.gitignore` already covers `scripts/trends/{node_modules,build}`.
- **No new `references/*.md`**, **no new agent/command** (counts stay 19/29/27); no new `.ts` source file
  (bridge lives in `item.ts`). Brief+plan live under `docs/` (uncounted), TRACKED like `docs/second-brain/*`.
- **Counts recounted live at land**, never pinned/guessed.

## 6. Success criteria (testable)

- **SC1 (bridge)** — `itemToInput(item, "2026-06-24")` returns a `TrendInput` with `capturedAt` injected (`=== "2026-06-24"`),
  `publishedAt`/`topics`/`summary`/`title`/`url`/`source` carried verbatim, and **no `id`**. Absent `publishedAt`
  on the item → absent on the input (key omitted, not `undefined`-valued). **Field-confusion guard:** for an item
  whose `publishedAt` differs from the injected `capturedAt`, the result's `capturedAt !== result.publishedAt`
  (proves the bridge never confuses the two dates — the whole point of the slice).
- **SC2 (persist)** — `addTrend` with a `publishedAt` persists it on the new record; a record without it omits the
  key. On a re-capture (same title+url) the existing `publishedAt` is **unchanged** (first-sight kept), and only
  topics union — `merged` reflects topic change alone. **No back-fill:** a re-capture carrying a `publishedAt` onto
  a record that **lacked** one does **not** add it (Open Q#2) — the absent→present case is the one where "first-sight
  kept" is counterintuitive, so it is pinned by a test.
- **SC3 (migrate)** — `loadStore` on a v1 store fixture (`{schemaVersion:1, trends:[…without publishedAt]}`) returns
  `schemaVersion === 2` with every record intact and **no `publishedAt` invented**. Also pinned: a store with
  **missing** `schemaVersion` → stamped 2 (records intact); an **empty/absent** store → `{schemaVersion:2, trends:[]}`;
  a **non-numeric** `schemaVersion` (`"weird"`/`NaN`) → falls back to 2 (records intact); idempotent on a v2 store;
  a round-trip `loadStore`→`saveStore` writes `schemaVersion: 2`.
- **SC4 (capture CLI)** — `echo '<batch>' | … capture` normalizes + folds: a well-formed item is added (or reported
  duplicate/merged), a content-invalid item appears in the summary `errors[]`, the store file is written once, and
  the summary counts are correct — **`added + merged + duplicates + errors.length === payload.length`** (the tally
  is derived from `AddResult {added, merged}`, which has no `duplicates` field: `added`=`res.added`,
  `merged`=`!res.added && res.merged`, `duplicates`=`!res.added && !res.merged`). The captured record's `capturedAt`
  is a today-shaped ISO date **distinct from the item's `publishedAt`** (field-confusion guard at the ingress; the
  exact-value assertion lives in `item.test.ts` with an injected date, since `capture` reads the wall clock). Exit
  **2** on empty/unparseable stdin; **0** on a well-formed call even with content-invalid items. `--json` emits the
  summary object.
- **SC5 (gate)** — `(cd scripts/trends && npm test)` green at the bumped `TRENDS_TESTS_FLOOR`; new Section 16h green;
  `ASSERT_BASELINE_FLOOR` bumped to the live recount; overall gate `FAIL=0`.
- **SC6 (wiring + de-niche)** — `trend-spotter.md` references `src/cli.ts capture` (Section 16h grep green); de-niche
  guard (Section 17) green; structure counts unchanged (19/29/27). *(If Open Q#1 → minimal: SC6 drops the wiring/16h
  clauses; de-niche + counts still asserted.)*

## 7. Verification

**Deterministic (gate):** `bash scripts/test-runner.sh` → `FAIL=0`; trends suite ≥ new floor; new Section 16h
self-test + greps pass; Section 17 de-niche green; ref/agent/command counts unchanged.
**Behavioural (manual):** `echo '<2-item batch incl. one publishedAt + one invalid>' | node --import tsx src/cli.ts
capture --store /tmp/r2a-trends.json --json`; confirm the valid item lands with `publishedAt` persisted, the invalid
one is in `errors[]`, and `node … list --store /tmp/r2a-trends.json --json` shows the persisted `publishedAt`.
Then load an authored v1 fixture and confirm `status --json` reports it migrated (count intact).

## 8. Open questions for the go-gate

1. **Wire `trend-spotter.md` + add Section 16h, or keep R2a minimal? — RESOLVED at go-gate (operator, 2026-06-24): WIRE.**
   Build R2a with the agent re-point (Step 4.5 `add`→`capture`) + Section 16h grep/self-test + ASSERT-floor bump. The
   minimal alternative (bridge code + tests only, no agent edit) was declined — wiring mirrors R1's "the lift is real
   and grep-able" discipline and makes the bridge actually used (one normalizing `capture` vs N× `add`).
2. **`publishedAt` merge policy.** Proposed **keep first-sight** (no back-fill), matching the existing source/capturedAt
   provenance rule (now pinned by the absent→present SC2 test). Confirm, or request back-fill-if-absent (adds a branch
   + a `merged` semantic question).
3. **`add --published-at` flag — RESOLVED at light-Voyage: deferred** (see §4). The manual `add` stays publish-date-free;
   `capture` is the path that carries `publishedAt`. Noted here only so the resolution is traceable; no go-gate action.

## 9. Light-Voyage review — folded

Three Opus reviewers ran on the drafts, each verifying claims against live code. **scope-guardian: ALIGNED**
(every SC1–SC6 traces to a step; zero creep; all §4 non-goals held; 2 minors). **brief-reviewer:
PROCEED_WITH_RISKS** (2 majors, 4 minors). **plan-critic: REVISE** (1 blocker, 5 majors, 5 minors). All findings
folded; see `plan-re-r2a.md` §Plan-critic — folded for the per-step resolution:

- **[BLOCKER, folded]** "v2 load idempotent" cannot be a RED case — old `loadStore` (`store.ts:78`) returns
  `parsed.schemaVersion ?? SCHEMA_VERSION`, so a v2 fixture already loads as 2 against unchanged code. → §6 SC3 +
  §5 now split the migration cases: **only v1→2 + round-trip are RED**; v2-idempotence + missing/non-numeric are
  **GREEN-only regression guards**.
- **[MAJOR, folded]** "lossless / byte-for-byte intact" over-claimed vs `loadStore`'s non-array `trends` coercion
  (`store.ts:79`). → §5 + §3 scope losslessness to **well-formed** v1/v2 stores; the coercion is unchanged + out of
  scope (not a regression).
- **[MAJOR, folded]** `capturedAt` injection unverified at the new `capture` ingress (the field-confusion the slice
  exists to prevent). → SC1 + SC4 now pin `capturedAt !== publishedAt` (item.test exact-value; cli.test shape +
  distinct).
- **[MAJOR, folded]** `capture` tally `{added, duplicates, merged}` mis-maps onto `AddResult {added, merged}` (no
  `duplicates` field). → SC4 pins the exact derivation + `sum === payload.length`.
- **[MAJOR, folded]** `add --published-at` validation referenced the non-exported `isValidIso` (`item.ts:51`). →
  **`add --published-at` deferred entirely** (§4), removing the export-vs-inline decision and keeping `item.ts`'s
  edit to `itemToInput` only.
- **[MAJOR, folded]** README "Capture … `add`" framing would contradict the agent's `add`→`capture` re-point. →
  plan Step 5 now corrects the README framing (`add` = manual single-trend; `capture` = normalizing batch), not just
  appends.
- **[MAJOR, folded]** empty "folded" placeholders shipped in the doc bodies. → this section + plan §Plan-critic now
  filled.
- **[MINOR, folded in plan]** missing-/non-numeric-`schemaVersion` test, absent→present back-fill test, stub-replacement
  note, `import type { TrendInput }` line, `grep -qF` for the 16h literal, `TRENDS_TESTS_FLOOR` breakdown-comment update,
  README in the code commit, headless per-step clauses N/A (in-session execution).