voyage/docs/storm-measurement.md

# STORM adoption gate — pre-registered measurement protocol

**Status:** thresholds registered, **no measurement run yet.** This document is
committed *before* the first measurement by design: a threshold chosen after
seeing the numbers is not a threshold. **Harness:** `scripts/storm-measure.mjs`
(pinned by `tests/scripts/storm-measure.test.mjs`). **Decides:** whether the
bounded Phase 5 conversation loop (`commands/trekresearch.md` §Loop bound) is
worth turning on by default.

---

## 1. What this gate measures — and what it does not

The gate measures **source and coverage breadth**:

| Metric | Definition | Arm |
|---|---|---|
| `unique_sources` | distinct external sources cited by the run | between-arm median gain |
| dimensions over baseline | `(dimensions − dimensions_baseline) / dimensions_baseline` | within-run median across the treatment arm |

It does **not** measure outline quality, answer correctness, synthesis
usefulness, or operator satisfaction. A breadth win is a *necessary* condition
for adoption, never a sufficient one. If the loop widens coverage by 40% and
the resulting briefs read worse, the correct action is still decline — the
number does not overrule a reading of the artifacts.

The second metric is a within-run delta by construction (`dimensions_baseline`
is the interview dimension count, `dimensions` the post-Phase-4.5 list), so it
needs no control arm; the control arm's value is 0 because the loop is inert
below `effort: high`.

## 2. Pre-registered thresholds

Either metric clearing the bar is enough. The brief pre-registers "median
forbedring ≥ 30 % på (a) eller (b) → adopt. < 15 % → decline" — a breadth win
on one axis counts, because either axis widening is the effect the loop claims.
Adopt is evaluated first, so a run that clears the adopt bar on one metric is
an adopt even when the other metric sits under the decline bar.

| Median gain | Verdict | Action |
|---|---|---|
| ≥ 30% on **either** metric | **adopt** | Flip the `VOYAGE_STORM_ENABLED` default (see §5) |
| < 15% on **either** metric (and no adopt) | **decline** | Leave the mechanism default-off. This is a **no-op**: nothing is rolled back |
| both metrics in 15% – 30% | **inconclusive** | Keep default-off, gather more runs, re-measure |
| either arm empty | **insufficient-data** | Not a decline — measure more |

`ADOPT_THRESHOLD = 0.30` and `DECLINE_THRESHOLD = 0.15` are exported constants
in `scripts/storm-measure.mjs` and pinned by the test suite. Changing them is a
deliberate, reviewable act, not a tuning knob to be nudged toward a result.

## 3. Excluded runs (the honest denominator)

Runs with `empty_turns > 0` are **excluded from the gain and reported as a
count**. An empty turn is one that spent budget and returned no findings, or
findings without citations. A run whose `empty_turns` cannot be **read** as a
number is excluded on the same footing: a field that says `"many"` is not
evidence of zero empty turns, and treating it as one would let the least
trustworthy run back into the denominator in the direction that flatters
adoption. Including those runs decides adoption on a broken
denominator — the loop looks cheap because its failures are averaged into its
successes. The harness prints the excluded count on every invocation; if that
count is a large fraction of the treatment arm, the finding is about the loop's
reliability, and it should be read before the gain figure is read at all.

Rows predating the Step 9 measurement fields carry no `effort` and are dropped
as `legacy` with a count. A stats file where *no* row carries `effort` is a hard
error, not an empty treatment group: a schema gap must never present itself as
"no gain".

## 4. The measurement runs (operator-run, outside this plan)

The measurement itself is **not** part of the implementation plan that built
this harness. It is an operator-run gate between that plan and any adopt commit.

Protocol:

- **n ≥ 5 runs per arm.** Fewer, and the median is an anecdote.
- **The same question set in both arms.** Two briefs, run as `--project` runs.
- The arms differ in exactly one thing: whether the loop is enabled.
- Both arms append to the same `trekresearch-stats.jsonl`; `effort` is the
  grouping key that separates them.

Each run is a Claude Code session the operator opens (Voyage starts none of
them; changed 2026-09-27 before any run — the thresholds above are untouched):

```text
# Control arm (loop inert) — n >= 5, in a session opened with VOYAGE_STORM_ENABLED unset
/trekresearch --project .claude/projects/<brief-standard-effort>

# Treatment arm (loop live, effort: high in the brief's phase_signals) — n >= 5,
# in a session opened with VOYAGE_STORM_ENABLED=1 in its environment
/trekresearch --project .claude/projects/<brief-high-effort>
```

```bash
# The gate
node scripts/storm-measure.mjs --stats "${CLAUDE_PLUGIN_DATA}/trekresearch-stats.jsonl"

# Machine-readable, for a decision record
node scripts/storm-measure.mjs --json --stats "${CLAUDE_PLUGIN_DATA}/trekresearch-stats.jsonl"

# SC activation check — BOTH halves: did the latest high-effort run discover at
# least one dimension, AND is its final list a true SUPERSET of the interview
# ones? The second half is read from the run's own `dimensions_baseline_preserved`
# attestation, because the dimension NAMES that would show it directly are prose
# the exporter allowlist denies. A run that does not attest it FAILS — dropping
# two interview dimensions and appending three discovered ones is a +1 count
# delta and not a superset. Exit 0 = both halves hold.
node scripts/storm-measure.mjs --activation-check --stats "${CLAUDE_PLUGIN_DATA}/trekresearch-stats.jsonl"
```

## 5. What "adopt" concretely means

Adoption is **one constant**: `isStormEnabled()` in
`lib/util/research-loop-cap.mjs` currently requires `VOYAGE_STORM_ENABLED === '1'`.
Adopt = make the loop's budget non-zero without that opt-in, in a commit that
cites the measurement output.

This asymmetry is deliberate and was designed in before any code was written:

- **decline costs nothing** — the mechanism ships default-off, so declining is
  doing nothing. No revert, no removal from a command file two other steps
  already rewrote.
- **adopt costs one constant** — plus the enforcement hook
  (`hooks/scripts/pre-agent-cap.mjs`) already in place to bound what gets turned
  on, and the operator-visible cap-exhaustion message already required by
  Phase 5's exit conditions.

## 6. Reading the result honestly

- The gate measures breadth. Say "breadth" in the decision record, not "quality".
- Report the excluded count alongside the gain, always. A 35% gain computed
  after excluding 6 of 10 treatment runs is a finding about instability.
- `insufficient-data` is not a decline. Do not resolve it by lowering n.
- A verdict computed from a stats file mixing several question sets measures the
  question sets, not the loop.