Kjell Tore Guttormsen
b4e500fad4
R2 ("the visible topic-stream") split into two sequenced slices at the
2026-06-24 go-gate, foundation-first: R2a (this — the pure scripts/trends/
data layer) before R2b (dated morning-brief + session-start surfacing).
R2a builds the item→store bridge RE-R1 deferred: a pure itemToInput mapping
the validated TrendItem envelope to a store input (injecting capturedAt,
carrying publishedAt), a publishedAt schema bump (SCHEMA_VERSION 1→2) with a
lossless forward migrate-on-load, and a `capture` CLI closing the
poll→normalize→store loop. No scoring change, no hook touch.
Go-gate: WIRE (operator) — re-point trend-spotter Step 4.5 add→capture +
Section 16h grep/self-test. Q2 publishedAt-merge = first-sight, no back-fill.
Light-Voyage hardened: scope-guardian ALIGNED, brief-reviewer
PROCEED_WITH_RISKS, plan-critic REVISE — all folded (BLOCKER: v2-idempotence
is GREEN-only not RED; capture tally mapped exactly onto AddResult; "lossless"
scoped to well-formed stores; add --published-at deferred; README framing fix).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VmHCQjJHUyWwxGAVVjNLgp
208 lines
18 KiB
Markdown
208 lines
18 KiB
Markdown
# Brief — RE-R2a: capture bridge (item→store) + publishedAt persistence
|
||
|
||
> **Slice:** RE-R2a (research-engine rung-2, slice 2 — the data layer of R2). The first of the two
|
||
> halves R2 ("the visible topic-stream") was split into at the 2026-06-24 go-gate: **R2a** = the pure
|
||
> `scripts/trends/` data layer (this brief); **R2b** = the dated morning-brief artifact + session-start
|
||
> surfacing (separate brief, after R2a lands). The split was chosen because (b)+(c) are code-independent
|
||
> of (a), R2a mirrors R1's pure-TDD shape exactly (lowest risk), and it closes the capture loop the store
|
||
> was built for.
|
||
> **Predecessor:** RE-R1 (`brief-re-r1.md`) delivered the validated ingress envelope (`item.ts`:
|
||
> `normalizeItem`/`normalizeItems`) and the triage scorer (`score.ts`) behind a CLI seam, and **explicitly
|
||
> deferred the item→store bridge** ("injecting the store's `capturedAt` and persisting `publishedAt` — is
|
||
> R2 orchestration", `brief-re-r1.md` §3 B1). R2a builds exactly that deferred bridge.
|
||
> **Substrate:** `docs/research-engine-concepts.local.md` §1 (hull 3 "store-schema mangler felt" / the
|
||
> item→store gap), §3 B1 (one schema downstream never branches on). **TDD-order:** RED tests land before code.
|
||
|
||
## 1. Operator decision context (2026-06-24)
|
||
|
||
The research engine is **Tier-1** (operator, 2026-06-23): *"hele min arbeidsflyt hviler på at jeg får en
|
||
jevn strøm av gode forslag til tema å skrive om."* At the R2 go-gate (2026-06-24) the operator chose to
|
||
build R2 as **two sequenced slices, foundation-first**: R2a (this — the bridge + schema, pure trends/) before
|
||
R2b (the visible brief + surfacing). R2a delivers no *visible* change; it closes the capture loop so the
|
||
store accumulates publish-dated history that R2b's brief and R3's freshness window will rank on.
|
||
|
||
## 2. The gap — grounded in code
|
||
|
||
- **`normalize` (R1) and the store are disconnected.** `item.ts` validates a raw source item into the
|
||
canonical `TrendItem` envelope `{source,title,url,publishedAt?,topics[],summary?}` (`item.ts:21-38`), but
|
||
**nothing folds that envelope into the store.** The only ingress to the store is `cli.ts add` (`cli.ts:111-135`),
|
||
which builds a `TrendInput` from raw flags — it never calls `normalizeItem`, so a capture path that polls →
|
||
normalizes → stores **does not exist**. The bridge is the missing link R1 named and deferred.
|
||
- **`publishedAt` is carried but dropped.** The envelope carries the source's own publish date
|
||
(`item.ts:28-33`, "NOT persisted in R1"), but `TrendRecord` (`types.ts:26-41`) has no `publishedAt` field and
|
||
`addTrend` (`store.ts:111-130`) never persists it. So every capture **loses** the publish date — the field
|
||
B4 freshness (R3) and R2b's "fresh today" sort will both need. R2a persists it now, before history accumulates
|
||
undated, so the loop is publish-dated from this slice forward.
|
||
- **`SCHEMA_VERSION` is pinned at 1 with no migration path exercised.** `loadStore` (`store.ts:74-81`) reads
|
||
`parsed.schemaVersion ?? SCHEMA_VERSION` but the store has never had a second version, so the forward-migration
|
||
path is untested. Adding `publishedAt` is the first additive-optional schema bump (v1→v2) and the right moment
|
||
to prove a lossless migrate-on-load.
|
||
|
||
## 3. Scope — what is IN (RE-R2a)
|
||
|
||
### B-bridge — `itemToInput` (pure mapping, `scripts/trends/src/item.ts`, EDIT)
|
||
A pure `itemToInput(item: TrendItem, capturedAt: string): TrendInput` that maps the validated envelope to a
|
||
store input by **injecting `capturedAt`** (the store's "when WE saw it", supplied by the caller — never derived
|
||
inside the pure function) and **carrying `publishedAt`/`topics`/`summary`/`title`/`url`/`source` through verbatim**.
|
||
It does **not** re-validate (the item is already validated by `normalizeItem`) and does **not** derive an `id`
|
||
(the store owns id via `addTrend`→`trendId`). Lives in `item.ts` (it is the item's mapping to the store; `item.ts`
|
||
already imports from `store.ts`) — **not** a new single-function `bridge.ts` (single-use abstraction).
|
||
|
||
### Schema migration — `publishedAt` persisted (`types.ts` + `store.ts`, EDIT)
|
||
- `types.ts`: `SCHEMA_VERSION 1 → 2`; add `publishedAt?: string` to `TrendRecord` (after `capturedAt`), with a
|
||
comment distinguishing it from `capturedAt` (source publish-date; forward-compat for B4 freshness).
|
||
- `store.ts`: `TrendInput` gains `publishedAt?: string`; `addTrend` persists it when present (same
|
||
conditional-spread idiom as `summary`, `store.ts:126`). **On re-capture/merge it is NOT overwritten** —
|
||
it joins the store's documented "FIRST sighting's source + capturedAt are kept (provenance of first sight)"
|
||
rule (`store.ts:108-110`); no back-fill (a deferred enhancement, §4).
|
||
- `loadStore`: **forward migrate-on-load** — stamp the in-memory store to `Math.max(onDisk, SCHEMA_VERSION)`,
|
||
where `onDisk = typeof parsed.schemaVersion === "number" ? parsed.schemaVersion : SCHEMA_VERSION` (a string /
|
||
`NaN` / absent version falls back to current, never crashes). v1→v2 is purely additive-optional (old records are
|
||
already valid v2 records that simply lack the optional field), so the migration is the stamp alone — **no record
|
||
rewrite; lossless + idempotent for any well-formed v1/v2 store**. A subsequent `saveStore` persists the v2 stamp.
|
||
**The existing non-array `trends` coercion (`store.ts:79`, `Array.isArray(parsed.trends) ? … : []`) is UNCHANGED
|
||
and out of R2a's migration scope** — losslessness is claimed only for well-formed stores, not for a corrupt
|
||
`trends` field. (No downgrade branch: a hypothetical future on-disk > current can't happen yet — Karpathy #2, no
|
||
handling for impossible scenarios; noted as a deferred risk.)
|
||
|
||
### CLI `capture` subcommand (`scripts/trends/src/cli.ts`, EDIT)
|
||
`echo '<raw item | batch>' | node --import tsx src/cli.ts capture [--store <path>] [--json]` — reads the raw
|
||
JSON payload **from stdin** (same stdin contract as `normalize`/`score`, so `--json` stays an *output* toggle),
|
||
runs it through `normalizeItem`/`normalizeItems`, maps each valid item via `itemToInput(item, today())`, folds
|
||
each into the store via `addTrend`, and `saveStore`s once. Emits a summary — `{added, duplicates, merged, errors}`
|
||
+ store path/count — human-readable by default, JSON with `--json`. Content-invalid items are reported in the
|
||
summary (`errors[]`), **never via the exit code**; exit **2** only on a malformed invocation (unparseable/empty
|
||
stdin), matching `normalize`/`score`. Note the contract difference from `add`: `capture` items must carry `source`
|
||
(the normalizer requires it — no "manual" default), because a capture is from a real source.
|
||
|
||
### Wiring + gate (see Open Question #1 — included by default, trimmable to minimal)
|
||
- `agents/trend-spotter.md` (EDIT): re-point **Step 4.5** (`trend-spotter.md:282-301`) from N× flag-based `add` to
|
||
a single batch `capture` — the agent builds a raw-item JSON batch (it already builds JSON for `score`) and pipes
|
||
it to `capture` in one call. Strictly better than `add` (it normalizes + carries `publishedAt`; one call, not N).
|
||
The replacement prose carries the literal `src/cli.ts capture`. Kept domain-general (no vendor/sector tokens).
|
||
- `scripts/test-runner.sh` (EDIT): bump `TRENDS_TESTS_FLOOR` (62 → **live recount**, stays inside the deps guard).
|
||
Add **Section 16h** ("Trends Capture Wiring"), mirroring Section 16g's three-check shape, placed **after Section
|
||
17 / before Section 18** (anti-erosion last): (1) a non-vacuity self-test; (2) `grep` that `cli.ts` has a `capture`
|
||
handler (deps-absent proof the path exists); (3) `grep -qF "src/cli.ts capture" agents/trend-spotter.md`. These are
|
||
**unconditional** → bump `ASSERT_BASELINE_FLOOR` 87 → **live recount** (expected ~90).
|
||
|
||
## 4. Non-goals — what is OUT (deferred)
|
||
|
||
- **The dated morning-brief artifact (B3)** — R2b. *R2a closes the capture loop; R2b makes the stream visible.*
|
||
- **Session-start surfacing** of the brief (hull 4) — R2b. R2a does **not** touch `hooks/**`.
|
||
- **`publishedAt` back-fill on re-capture** (fill an absent existing `publishedAt` from a later sighting) — deferred;
|
||
first-sight provenance is kept, matching the existing merge rule. Revisit if undated-first-sight proves common.
|
||
- **`add --published-at` flag** — deferred (was a proposed "minor"; folded out at light-Voyage). The flag-based
|
||
`add` is the *manual* single-trend path; it stays publish-date-free for now. The store layer still GAINS
|
||
`publishedAt` (on `TrendInput` + `addTrend`) — that is what `capture` needs — but exposing it on the `add` CLI
|
||
(and the `isValidIso`-export-vs-inline decision it would force) is out of R2a. Trivial to add any later slice.
|
||
- **`relevance` / `saturation` / `status` / lifecycle fields** (hull 5) — R3. R2a adds **only** `publishedAt`.
|
||
- **Freshness window / dedup-vs-seen-log / autonomous trigger (B4)** — R3.
|
||
- **Research-deepening (A1–A4), adapter sub-agents, MCP fetch fan-out** — R2b/R3.
|
||
- **A store-reading brief / ranking on accumulated relevance** — R2b/R3 (needs persisted scores).
|
||
|
||
## 5. Boundaries / invariants (must hold)
|
||
|
||
- **TDD iron law:** the failing migration / `itemToInput` / `capture` tests land BEFORE the implementation.
|
||
RED proofs recorded (logic-RED, not import-RED). **NB:** only the **v1→v2** load + round-trip are genuinely RED
|
||
(old `loadStore` returns 1); v2-idempotence + missing/non-numeric-`schemaVersion` pass against old code, so they
|
||
are **GREEN-only regression guards**, not RED cases (see plan Step 1).
|
||
- **Lossless migration (well-formed stores):** every existing **well-formed** v1 store loads as v2 with **records
|
||
intact** (no `publishedAt` invented, no topic/summary/capturedAt/url/title change); idempotent (v2 → v2). A
|
||
corrupt `trends` field is coerced by the existing (unchanged) `Array.isArray` guard — explicitly **out of R2a's
|
||
losslessness claim**, not a regression introduced here.
|
||
- **First-sight provenance preserved:** re-capturing an existing trend never overwrites its `publishedAt`,
|
||
`capturedAt`, or `source` — only topics union (unchanged from R1).
|
||
- **No scoring change:** `score.ts` and `references/trend-scoring-modes.md` (the SSOT) are untouched — R2a is the
|
||
data layer, not the scorer.
|
||
- **Domain-general:** de-niche guard (Section 17) stays green; no vendor/sector tokens enter the edited
|
||
`trend-spotter.md` prose.
|
||
- **House conventions:** ESM + `node:test` + `tsx`; data-seam stays inline; **no new `.mjs` under `hooks/scripts/`**
|
||
(R2a touches no hook); `.gitignore` already covers `scripts/trends/{node_modules,build}`.
|
||
- **No new `references/*.md`**, **no new agent/command** (counts stay 19/29/27); no new `.ts` source file
|
||
(bridge lives in `item.ts`). Brief+plan live under `docs/` (uncounted), TRACKED like `docs/second-brain/*`.
|
||
- **Counts recounted live at land**, never pinned/guessed.
|
||
|
||
## 6. Success criteria (testable)
|
||
|
||
- **SC1 (bridge)** — `itemToInput(item, "2026-06-24")` returns a `TrendInput` with `capturedAt` injected (`=== "2026-06-24"`),
|
||
`publishedAt`/`topics`/`summary`/`title`/`url`/`source` carried verbatim, and **no `id`**. Absent `publishedAt`
|
||
on the item → absent on the input (key omitted, not `undefined`-valued). **Field-confusion guard:** for an item
|
||
whose `publishedAt` differs from the injected `capturedAt`, the result's `capturedAt !== result.publishedAt`
|
||
(proves the bridge never confuses the two dates — the whole point of the slice).
|
||
- **SC2 (persist)** — `addTrend` with a `publishedAt` persists it on the new record; a record without it omits the
|
||
key. On a re-capture (same title+url) the existing `publishedAt` is **unchanged** (first-sight kept), and only
|
||
topics union — `merged` reflects topic change alone. **No back-fill:** a re-capture carrying a `publishedAt` onto
|
||
a record that **lacked** one does **not** add it (Open Q#2) — the absent→present case is the one where "first-sight
|
||
kept" is counterintuitive, so it is pinned by a test.
|
||
- **SC3 (migrate)** — `loadStore` on a v1 store fixture (`{schemaVersion:1, trends:[…without publishedAt]}`) returns
|
||
`schemaVersion === 2` with every record intact and **no `publishedAt` invented**. Also pinned: a store with
|
||
**missing** `schemaVersion` → stamped 2 (records intact); an **empty/absent** store → `{schemaVersion:2, trends:[]}`;
|
||
a **non-numeric** `schemaVersion` (`"weird"`/`NaN`) → falls back to 2 (records intact); idempotent on a v2 store;
|
||
a round-trip `loadStore`→`saveStore` writes `schemaVersion: 2`.
|
||
- **SC4 (capture CLI)** — `echo '<batch>' | … capture` normalizes + folds: a well-formed item is added (or reported
|
||
duplicate/merged), a content-invalid item appears in the summary `errors[]`, the store file is written once, and
|
||
the summary counts are correct — **`added + merged + duplicates + errors.length === payload.length`** (the tally
|
||
is derived from `AddResult {added, merged}`, which has no `duplicates` field: `added`=`res.added`,
|
||
`merged`=`!res.added && res.merged`, `duplicates`=`!res.added && !res.merged`). The captured record's `capturedAt`
|
||
is a today-shaped ISO date **distinct from the item's `publishedAt`** (field-confusion guard at the ingress; the
|
||
exact-value assertion lives in `item.test.ts` with an injected date, since `capture` reads the wall clock). Exit
|
||
**2** on empty/unparseable stdin; **0** on a well-formed call even with content-invalid items. `--json` emits the
|
||
summary object.
|
||
- **SC5 (gate)** — `(cd scripts/trends && npm test)` green at the bumped `TRENDS_TESTS_FLOOR`; new Section 16h green;
|
||
`ASSERT_BASELINE_FLOOR` bumped to the live recount; overall gate `FAIL=0`.
|
||
- **SC6 (wiring + de-niche)** — `trend-spotter.md` references `src/cli.ts capture` (Section 16h grep green); de-niche
|
||
guard (Section 17) green; structure counts unchanged (19/29/27). *(If Open Q#1 → minimal: SC6 drops the wiring/16h
|
||
clauses; de-niche + counts still asserted.)*
|
||
|
||
## 7. Verification
|
||
|
||
**Deterministic (gate):** `bash scripts/test-runner.sh` → `FAIL=0`; trends suite ≥ new floor; new Section 16h
|
||
self-test + greps pass; Section 17 de-niche green; ref/agent/command counts unchanged.
|
||
**Behavioural (manual):** `echo '<2-item batch incl. one publishedAt + one invalid>' | node --import tsx src/cli.ts
|
||
capture --store /tmp/r2a-trends.json --json`; confirm the valid item lands with `publishedAt` persisted, the invalid
|
||
one is in `errors[]`, and `node … list --store /tmp/r2a-trends.json --json` shows the persisted `publishedAt`.
|
||
Then load an authored v1 fixture and confirm `status --json` reports it migrated (count intact).
|
||
|
||
## 8. Open questions for the go-gate
|
||
|
||
1. **Wire `trend-spotter.md` + add Section 16h, or keep R2a minimal? — RESOLVED at go-gate (operator, 2026-06-24): WIRE.**
|
||
Build R2a with the agent re-point (Step 4.5 `add`→`capture`) + Section 16h grep/self-test + ASSERT-floor bump. The
|
||
minimal alternative (bridge code + tests only, no agent edit) was declined — wiring mirrors R1's "the lift is real
|
||
and grep-able" discipline and makes the bridge actually used (one normalizing `capture` vs N× `add`).
|
||
2. **`publishedAt` merge policy.** Proposed **keep first-sight** (no back-fill), matching the existing source/capturedAt
|
||
provenance rule (now pinned by the absent→present SC2 test). Confirm, or request back-fill-if-absent (adds a branch
|
||
+ a `merged` semantic question).
|
||
3. **`add --published-at` flag — RESOLVED at light-Voyage: deferred** (see §4). The manual `add` stays publish-date-free;
|
||
`capture` is the path that carries `publishedAt`. Noted here only so the resolution is traceable; no go-gate action.
|
||
|
||
## 9. Light-Voyage review — folded
|
||
|
||
Three Opus reviewers ran on the drafts, each verifying claims against live code. **scope-guardian: ALIGNED**
|
||
(every SC1–SC6 traces to a step; zero creep; all §4 non-goals held; 2 minors). **brief-reviewer:
|
||
PROCEED_WITH_RISKS** (2 majors, 4 minors). **plan-critic: REVISE** (1 blocker, 5 majors, 5 minors). All findings
|
||
folded; see `plan-re-r2a.md` §Plan-critic — folded for the per-step resolution:
|
||
|
||
- **[BLOCKER, folded]** "v2 load idempotent" cannot be a RED case — old `loadStore` (`store.ts:78`) returns
|
||
`parsed.schemaVersion ?? SCHEMA_VERSION`, so a v2 fixture already loads as 2 against unchanged code. → §6 SC3 +
|
||
§5 now split the migration cases: **only v1→2 + round-trip are RED**; v2-idempotence + missing/non-numeric are
|
||
**GREEN-only regression guards**.
|
||
- **[MAJOR, folded]** "lossless / byte-for-byte intact" over-claimed vs `loadStore`'s non-array `trends` coercion
|
||
(`store.ts:79`). → §5 + §3 scope losslessness to **well-formed** v1/v2 stores; the coercion is unchanged + out of
|
||
scope (not a regression).
|
||
- **[MAJOR, folded]** `capturedAt` injection unverified at the new `capture` ingress (the field-confusion the slice
|
||
exists to prevent). → SC1 + SC4 now pin `capturedAt !== publishedAt` (item.test exact-value; cli.test shape +
|
||
distinct).
|
||
- **[MAJOR, folded]** `capture` tally `{added, duplicates, merged}` mis-maps onto `AddResult {added, merged}` (no
|
||
`duplicates` field). → SC4 pins the exact derivation + `sum === payload.length`.
|
||
- **[MAJOR, folded]** `add --published-at` validation referenced the non-exported `isValidIso` (`item.ts:51`). →
|
||
**`add --published-at` deferred entirely** (§4), removing the export-vs-inline decision and keeping `item.ts`'s
|
||
edit to `itemToInput` only.
|
||
- **[MAJOR, folded]** README "Capture … `add`" framing would contradict the agent's `add`→`capture` re-point. →
|
||
plan Step 5 now corrects the README framing (`add` = manual single-trend; `capture` = normalizing batch), not just
|
||
appends.
|
||
- **[MAJOR, folded]** empty "folded" placeholders shipped in the doc bodies. → this section + plan §Plan-critic now
|
||
filled.
|
||
- **[MINOR, folded in plan]** missing-/non-numeric-`schemaVersion` test, absent→present back-fill test, stub-replacement
|
||
note, `import type { TrendInput }` line, `grep -qF` for the 16h literal, `TRENDS_TESTS_FLOOR` breakdown-comment update,
|
||
README in the code commit, headless per-step clauses N/A (in-session execution).
|