voyage/docs/storm-measurement.md

# STORM adoption gate — pre-registered measurement protocol

**Status:** thresholds registered, **no measurement run yet.** This document is
committed *before* the first measurement by design: a threshold chosen after
seeing the numbers is not a threshold. **Harness:** `scripts/storm-measure.mjs`
(pinned by `tests/scripts/storm-measure.test.mjs`). **Decides:** whether the
bounded Phase 5 conversation loop (`commands/trekresearch.md` §Loop bound) is
worth turning on by default.

---

## 1. What this gate measures — and what it does not

The gate measures **source and coverage breadth**:

| Metric | Definition | Arm |
|---|---|---|
| `unique_sources` | distinct external sources cited by the run | between-arm median gain |
| dimensions over baseline | `(dimensions − dimensions_baseline) / dimensions_baseline` | within-run median across the treatment arm |

It does **not** measure outline quality, answer correctness, synthesis
usefulness, or operator satisfaction. A breadth win is a *necessary* condition
for adoption, never a sufficient one. If the loop widens coverage by 40% and
the resulting briefs read worse, the correct action is still decline — the
number does not overrule a reading of the artifacts.

The second metric is a within-run delta by construction (`dimensions_baseline`
is the interview dimension count, `dimensions` the post-Phase-4.5 list), so it
needs no control arm; the control arm's value is 0 because the loop is inert
below `effort: high`.

## 2. Pre-registered thresholds

Either metric clearing the bar is enough. The brief pre-registers "median
forbedring ≥ 30 % på (a) eller (b) → adopt. < 15 % → decline" — a breadth win
on one axis counts, because either axis widening is the effect the loop claims.
Adopt is evaluated first, so a run that clears the adopt bar on one metric is
an adopt even when the other metric sits under the decline bar.

| Median gain | Verdict | Action |
|---|---|---|
| ≥ 30% on **either** metric | **adopt** | Flip the `VOYAGE_STORM_ENABLED` default (see §5) |
| < 15% on **either** metric (and no adopt) | **decline** | Leave the mechanism default-off. This is a **no-op**: nothing is rolled back |
| both metrics in 15% – 30% | **inconclusive** | Keep default-off, gather more runs, re-measure |
| either arm empty | **insufficient-data** | Not a decline — measure more |

`ADOPT_THRESHOLD = 0.30` and `DECLINE_THRESHOLD = 0.15` are exported constants
in `scripts/storm-measure.mjs` and pinned by the test suite. Changing them is a
deliberate, reviewable act, not a tuning knob to be nudged toward a result.

## 3. Excluded runs (the honest denominator)

Runs with `empty_turns > 0` are **excluded from the gain and reported as a
count**. An empty turn is one that spent budget and returned no findings, or
findings without citations. Including those runs decides adoption on a broken
denominator — the loop looks cheap because its failures are averaged into its
successes. The harness prints the excluded count on every invocation; if that
count is a large fraction of the treatment arm, the finding is about the loop's
reliability, and it should be read before the gain figure is read at all.

Rows predating the Step 9 measurement fields carry no `effort` and are dropped
as `legacy` with a count. A stats file where *no* row carries `effort` is a hard
error, not an empty treatment group: a schema gap must never present itself as
"no gain".

## 4. The measurement runs (operator-run, outside this plan)

The measurement itself is **not** part of the implementation plan that built
this harness. It is an operator-run gate between that plan and any adopt commit.

Protocol:

- **n ≥ 5 runs per arm.** Fewer, and the median is an anecdote.
- **The same question set in both arms.** Two briefs, run as `--project` runs.
- The arms differ in exactly one thing: whether the loop is enabled.
- Both arms append to the same `trekresearch-stats.jsonl`; `effort` is the
  grouping key that separates them.

```bash
# Control arm (loop inert) — n >= 5
claude -p "/trekresearch --project .claude/projects/<brief-standard-effort>"

# Treatment arm (loop live, effort: high in the brief's phase_signals) — n >= 5
VOYAGE_STORM_ENABLED=1 \
  claude -p "/trekresearch --project .claude/projects/<brief-high-effort>"

# The gate
node scripts/storm-measure.mjs --stats "${CLAUDE_PLUGIN_DATA}/trekresearch-stats.jsonl"

# Machine-readable, for a decision record
node scripts/storm-measure.mjs --json --stats "${CLAUDE_PLUGIN_DATA}/trekresearch-stats.jsonl"

# SC activation check — did the latest high-effort run discover any dimension
# beyond its interview baseline? Exit 0 = yes.
node scripts/storm-measure.mjs --activation-check --stats "${CLAUDE_PLUGIN_DATA}/trekresearch-stats.jsonl"
```

## 5. What "adopt" concretely means

Adoption is **one constant**: `isStormEnabled()` in
`lib/util/research-loop-cap.mjs` currently requires `VOYAGE_STORM_ENABLED === '1'`.
Adopt = make the loop's budget non-zero without that opt-in, in a commit that
cites the measurement output.

This asymmetry is deliberate and was designed in before any code was written:

- **decline costs nothing** — the mechanism ships default-off, so declining is
  doing nothing. No revert, no removal from a command file two other steps
  already rewrote.
- **adopt costs one constant** — plus the enforcement hook
  (`hooks/scripts/pre-agent-cap.mjs`) already in place to bound what gets turned
  on, and the operator-visible cap-exhaustion message already required by
  Phase 5's exit conditions.

## 6. Reading the result honestly

- The gate measures breadth. Say "breadth" in the decision record, not "quality".
- Report the excluded count alongside the gain, always. A 35% gain computed
  after excluding 6 of 10 treatment runs is a finding about instability.
- `insufficient-data` is not a decline. Do not resolve it by lowering n.
- A verdict computed from a stats file mixing several question sets measures the
  question sets, not the loop.