linkedin-studio/docs/research-engine/plan-re-r1.md

# Plan — RE-R1: item-schema (B1) + triage-scorer (B2) as tested code

> **Brief:** `docs/research-engine/brief-re-r1.md`. **Slice:** RE-R1 (research-engine rung-2, slice 1).
> **TDD-order:** RED (item + score tests as logic-RED) → GREEN (item.ts, score.ts) → GREEN (CLI + cli tests) → wire trend-spotter prose → gate floors → behavioural → land. **Counts recounted live at land, never pinned/guessed.**
> **Light-Voyage hardened:** scope-guardian ALIGNED; brief-reviewer + plan-critic findings folded (see §Plan-critic — folded).

## Goal

Move the research engine's deterministic core — the canonical item envelope (B1) and the composite/band/threshold arithmetic (B2) — out of `agents/trend-spotter.md` prose into pure, tested TypeScript under `scripts/trends/`, behind a CLI seam the agent and a future orchestrator call. No store-schema change; the five judgment scores stay with the model; wiring `normalizeItem` into the store is R2.

## Files touched (exhaustive — for scope-guardian)

| File | Change | SC |
|---|---|---|
| `scripts/trends/src/item.ts` | **NEW** — `TrendItem` ingress envelope + `normalizeItem` / `normalizeItems` (pure, validating; no `id`) | SC1 |
| `scripts/trends/src/score.ts` | **NEW** — `ScoreMode`, per-mode weight consts (mirror SSOT), `composite`, `band`, `triage` | SC2, SC3, SC4 |
| `scripts/trends/src/cli.ts` | **EDIT** — add `normalize` + `score` subcommands (stdin JSON in, JSON out, exit 2 on bad invocation) | SC5 |
| `scripts/trends/tests/item.test.ts` | **NEW** — normalize: required-field errors, whitespace/topic dedupe, publishedAt validate, batch partition | SC1 |
| `scripts/trends/tests/score.test.ts` | **NEW** — golden composite (both modes: all-tens=10.0 + `{10,8,6,4,2}`=7.0 + weights-sum-1.0 + pinned weights), range guard, band boundaries + pinned action strings, triage gate/rank | SC2, SC3, SC4 |
| `scripts/trends/tests/cli.test.ts` | **NEW** — subprocess: `normalize`/`score` happy path (stdin→JSON) + exit-2 bad invocation | SC5 |
| `agents/trend-spotter.md` | **EDIT** — replace L124-137 "compute composite/bands yourself" with a pointer naming `src/cli.ts score` as the deterministic-step owner; domain-general | SC7 |
| `scripts/test-runner.sh` | **EDIT** — `TRENDS_TESTS_FLOOR` 24→recount (stays inside deps guard); NEW unconditional Section **16g** (before Section 18); `ASSERT_BASELINE_FLOOR` 84→recount; header enumeration | SC6, SC7 |
| `docs/research-engine/{brief,plan}-re-r1.md` | **NEW** — slice docs (TRACKED, like `docs/second-brain/*`) | — |
| `STATE.md` | **EDIT at land** — Telling-block reconcile (floors, gate total). *Land bookkeeping, not slice scope; LOCAL-ONLY.* | — |

**Not touched (scope fence):** `scripts/trends/src/{store.ts,types.ts}` (no schema change, `SCHEMA_VERSION` stays 1) · `references/trend-scoring-modes.md` (SSOT unchanged) · `references/*` (no new ref doc) · `config/trends-sources.template.md` (source list — not wired in R1) · `agents/*` count (19) · `commands/*` (29) · `hooks/**` · `.gitignore` (trends lines already present).

## Step 1 — (RED) failing tests for B1 + B2

Write `tests/item.test.ts` and `tests/score.test.ts` against not-yet-existing modules. Make them **logic-RED**, not import-RED, with a stub strategy split by assertion type:
- **arithmetic / "returns X" tests** → stub returns a *wrong constant* (so the equality assertion fails on value, not on a throw);
- **"should throw" validation tests** → stub returns a valid-looking value (so the `assert.throws` fails because nothing threw).

`item.test.ts`: well-formed raw → canonical item (fields verbatim; topics normalized+deduped); missing/empty `source|title|url` → `{ok:false}` naming the field; whitespace collapse via the same normalization as `store.normalizeField`; `publishedAt` present-and-ISO → kept, absent → undefined, present-and-invalid → `{ok:false}`; `normalizeItems` partitions a batch into `{items, errors}`; the canonical item carries **no `id`**.

`score.test.ts`: `composite` for both modes on (a) all-tens → exactly **10.0**, (b) the asymmetric vector `{10,8,6,4,2}` in dimension order → **7.0**; a dimension = 0 or 11 → throws; a **pinned-weights** assertion (each mode's five constants equal the SSOT values, Σ=1.0); `band` at 8.0 / 6.0 / 4.0 / 2.0 / below → correct priority + **the exact SSOT action string** (kortform + long-form); `triage` with threshold 4.0 → kept (≥4.0, sorted composite-desc, annotated composite+band) and dropped (<4.0).

**RED proof (record in commit):** `(cd scripts/trends && npm test)` → new cases fail with assertion errors (not module-not-found).

## Step 2 — (GREEN) implement `item.ts`

Implement the `TrendItem` ingress envelope `{source,title,url,publishedAt?,topics[],summary?}` + `normalizeItem`/`normalizeItems` to pass Step 1's item cases. Reuse `normalizeField` (import from `./store.js`) for whitespace; topic normalize + dedupe. `publishedAt` validated with a strict ISO-date check; **carried, not persisted** (comment: the *source's* publish date, forward-compat for B4 freshness — distinct from the store's `capturedAt`). **Do not derive or carry `id`** — the store owns it via `addTrend`→`trendId`; the envelope has no id field. **Do not wire to the store** — the item→`TrendInput` bridge (`capturedAt` injection) is R2.

## Step 3 — (GREEN) implement `score.ts`

Encode the two weight-sets as `const` records mirroring `trend-scoring-modes.md` (header comment: the SSOT is the human source, "ordering is the signal, not a measured coefficient", + the SSOT path). Implement `composite` (validate each dimension ∈[1,10], weighted sum), `band` (the five-range map → `{priority, kortformAction, longformAction}` using the **exact SSOT action strings**), and `triage` (`kept`/`dropped` + composite-desc sort + per-item composite/band annotation). Make Step 1's score cases green (incl. the pinned-weights + pinned-action-string assertions).

## Step 4 — (GREEN) CLI subcommands + `cli.test.ts`

Add `normalize` and `score` to `cli.ts`'s `main` dispatch. Both **read the JSON payload from stdin** (not a flag — the existing `--json` is an *output* toggle and must keep that meaning) and **print JSON to stdout**. `normalize` → canonical items / `{ok:false}` error entries. `score` → `triage` with `--mode`/`--threshold`, prints `{kept, dropped}`. Exit **2** on a malformed invocation (unparseable stdin, missing required flag) via the existing `usage()` path; **0** otherwise. Write `tests/cli.test.ts` (subprocess: spawn `node --import tsx src/cli.ts <sub>` with a piped stdin payload) covering a happy path + an exit-2 bad-invocation for each new subcommand.

## Step 5 — wire `trend-spotter.md` (prose pointer)

Replace the L124-137 "score 5 dims, take the weighted composite, apply the bands yourself" instruction with: the agent supplies the five 1–10 judgment scores, then pipes them to **`scripts/trends/src/cli.ts score`** (the deterministic owner of composite + bands + threshold). The replacement prose **must contain the literal `src/cli.ts score`** (the exact string Step 6's `grep -qF` matches). Keep it domain-general — no vendor/sector tokens (Section 17). The agent still owns mode selection and the qualitative scoring.

## Step 6 — gate: floors + new unconditional section

In `scripts/test-runner.sh`:
- Bump `TRENDS_TESTS_FLOOR` 24 → **live recount** after Steps 1–4 (24 + new item/score/cli cases). It **stays inside** the `if [ -x scripts/trends/node_modules/.bin/tsx ]` deps guard (conditional — do not hoist it out; that would break fresh-clone safety).
- Add **Section 16g** (label it 16g; place it **after Section 17 / before Section 18**, since Section 18 anti-erosion must run last). Three **unconditional**, deps-absent-safe checks (pure `grep`, no `tsx`): (1) `score.ts` encodes both `kortform` and `long-form` weight-sets; (2) `grep -qF "src/cli.ts score" agents/trend-spotter.md`; (3) a non-vacuity self-test for those greps (house pattern, per Sections 16c–17).
- Bump `ASSERT_BASELINE_FLOOR` 84 → **live recount** (= 84 + the count of new unconditional `pass`/`fail` emitters in 16g; expected ~87 with the self-test, **recounted at land, not pinned**). Update the section-header enumeration comment.

## Step 7 — behavioural verification

`(cd scripts/trends && npm install)` if needed, then run (verified invocation form, not `npm run start`):
`echo '<3-item sample>' | node --import tsx src/cli.ts normalize` and
`echo '<scored sample>' | node --import tsx src/cli.ts score --mode kortform --threshold 4.0`;
confirm by hand that one ≥4.0 item is kept (correct band/action) and one <4.0 is dropped. Run full `bash scripts/test-runner.sh` → `FAIL=0`.

## Step 8 — land

Recount all touched floors live; reconcile STATE.md "Telling" block (trends N/N, ASSERT floor, gate total). Commit order (house style: `feat … [skip-docs]` for code; plain for docs): **(1)** docs commit `docs/research-engine/{brief,plan}-re-r1.md` (no suffix, tracked); **(2)** code commit `scripts/trends/*` + `agents/trend-spotter.md` + `scripts/test-runner.sh` with `[skip-docs]`. Check the push window (`date '+%u %H:%M'`); `origin` is the PUBLIC `open/` remote → **confirm with operator before push**. No version bump (additive; `v0.5.2` dev).

## Verification (testable)

| SC | Check | Command | Expected |
|---|---|---|---|
| — | RED proof | `(cd scripts/trends && npm test)` after Step 1 | new cases fail on assertion (logic-RED), not module-not-found |
| SC1 | normalize | `npm test` (item.test.ts) | required-field `{ok:false}` + dedupe + publishedAt + batch-partition cases green; no `id` on item |
| SC2 | composite | `npm test` (score.test.ts) | all-tens=10.0 both modes; `{10,8,6,4,2}`=7.0 both modes; pinned weights; range guard throws |
| SC3 | bands | `npm test` (score.test.ts) | 8.0/6.0/4.0/2.0 boundaries → correct priority + exact SSOT action string |
| SC4 | triage | `npm test` (score.test.ts) | kept ≥ threshold ranked desc + annotated; dropped below |
| SC5 | CLI | `npm test` (cli.test.ts) + manual stdin run | JSON out; exit 2 malformed invocation / 0 well-formed |
| SC6 | gate | `bash scripts/test-runner.sh` | `FAIL=0`; trends ≥ new floor; ASSERT floor = live recount |
| SC7 | wiring + de-niche | Section 16g greps + Section 17 | `src/cli.ts score` present in trend-spotter.md; no vendor/sector tokens; counts 19/29/27 |

## Risks

- **R1 — SSOT/code drift.** Weights, band thresholds, AND the ten action strings now live in both `trend-scoring-modes.md` and `score.ts`. *Mitigated:* `score.test.ts` pins all three (weights + Σ=1.0 + thresholds + exact action strings) against the SSOT values, with an SSOT-path comment naming the markdown as the human source. (A markdown-table-parsing cross-check is deferred — out of scope.)
- **R2 — `publishedAt` carried but not persisted; `capturedAt` not on the item.** Could read as a dangling field. *Mitigated:* explicit comment (source publish-date, forward-compat for B4) + the brief's non-goal; the store bridge (capturedAt injection) is explicitly R2; the scorer does not depend on either.
- **R3 — editing `trend-spotter.md` could trip the de-niche guard.** *Mitigated:* Section 17 runs in the gate; replacement prose is pillar-driven and vendor/sector-free.
- **R4 — new gate checks must survive a deps-absent fresh clone.** *Mitigated:* the three new checks are pure `grep`/self-test on tracked source files (no `tsx`), so they are unconditional and safe; the arithmetic proof stays inside the deps-gated suite; `TRENDS_TESTS_FLOOR` stays inside the deps guard.
- **R5 — default threshold (4.0) may not match operator intent.** *Mitigated:* single config param; brief open question #1 settles it at the go-gate.
- **R6 — CLI `--json` semantic collision.** *Mitigated:* new subcommands take payload via **stdin**, leaving `--json` as the existing output toggle; cli.test.ts encodes the stdin contract.

## Plan-critic — folded

plan-critic returned **REVISE** (1 blocker, 6 majors, 4 minors); brief-reviewer **PROCEED_WITH_RISKS**; scope-guardian **ALIGNED**. Resolution, each verified against live code:

- **[BLOCKER] `TrendInput` shape mismatch** (`store.ts:26-33` requires `capturedAt`, no `publishedAt`). ✅ Step 2 no longer claims a direct map; the item→store bridge is deferred to R2; envelope carries no `id`.
- **[MAJOR] gate-section placement / Section-18-last** ✅ Step 6 pins **16g, before Section 18**.
- **[MAJOR] `ASSERT_BASELINE_FLOOR` hard-pinned 86** ✅ Step 6 now **live recount** (~+3 with the house self-test), not pinned.
- **[MAJOR] `TRENDS_TESTS_FLOOR` could be hoisted out of the deps guard** ✅ Step 6 states it stays conditional.
- **[MAJOR] CLI `--json` input/output overload** (`cli.ts:79` output toggle) ✅ Step 4 reads payload from **stdin**.
- **[MAJOR] band action-string drift unguarded** ✅ Step 1/3 + R1 pin the thresholds + action strings.
- **[MAJOR] Step 5/6 grep literal unpinned** ✅ pinned to `src/cli.ts score` in both steps.
- **[MINOR] RED-stub strategy** ✅ Step 1 splits stub by assertion type. **[MINOR] `npm run start` unverified** ✅ Step 7 uses `node --import tsx src/cli.ts`. **[MINOR] commit grouping** ✅ Step 8 fixes order. **[MINOR] empty folded sections** ✅ filled.
- **[scope-guardian MINOR] STATE.md + `config/trends-sources.template.md`** ✅ STATE.md added as a land-bookkeeping row; template added to the scope fence.

**scope-guardian — ALIGNED:** every SC1–SC7 traces to a step; zero scope creep; every NON-goal respected.