voyage/docs/storm-measurement.md

STORM adoption gate — pre-registered measurement protocol

Status: thresholds registered, no measurement run yet. This document is committed before the first measurement by design: a threshold chosen after seeing the numbers is not a threshold. Harness: scripts/storm-measure.mjs (pinned by tests/scripts/storm-measure.test.mjs). Decides: whether the bounded Phase 5 conversation loop (commands/trekresearch.md §Loop bound) is worth turning on by default.

---

1. What this gate measures — and what it does not

The gate measures source and coverage breadth:

Metric	Definition	Arm
unique_sources	distinct external sources cited by the run	between-arm median gain
dimensions over baseline	(dimensions − dimensions_baseline) / dimensions_baseline	within-run median across the treatment arm

It does not measure outline quality, answer correctness, synthesis usefulness, or operator satisfaction. A breadth win is a necessary condition for adoption, never a sufficient one. If the loop widens coverage by 40% and the resulting briefs read worse, the correct action is still decline — the number does not overrule a reading of the artifacts.

The second metric is a within-run delta by construction (dimensions_baseline is the interview dimension count, dimensions the post-Phase-4.5 list), so it needs no control arm; the control arm's value is 0 because the loop is inert below effort: high.

2. Pre-registered thresholds

Either metric clearing the bar is enough. The brief pre-registers "median forbedring ≥ 30 % på (a) eller (b) → adopt. < 15 % → decline" — a breadth win on one axis counts, because either axis widening is the effect the loop claims. Adopt is evaluated first, so a run that clears the adopt bar on one metric is an adopt even when the other metric sits under the decline bar.

Median gain	Verdict	Action
≥ 30% on either metric	adopt	Flip the VOYAGE_STORM_ENABLED default (see §5)
< 15% on either metric (and no adopt)	decline	Leave the mechanism default-off. This is a no-op: nothing is rolled back
both metrics in 15% – 30%	inconclusive	Keep default-off, gather more runs, re-measure
either arm empty	insufficient-data	Not a decline — measure more

ADOPT_THRESHOLD = 0.30 and DECLINE_THRESHOLD = 0.15 are exported constants in scripts/storm-measure.mjs and pinned by the test suite. Changing them is a deliberate, reviewable act, not a tuning knob to be nudged toward a result.

3. Excluded runs (the honest denominator)

Runs with empty_turns > 0 are excluded from the gain and reported as a count. An empty turn is one that spent budget and returned no findings, or findings without citations. Including those runs decides adoption on a broken denominator — the loop looks cheap because its failures are averaged into its successes. The harness prints the excluded count on every invocation; if that count is a large fraction of the treatment arm, the finding is about the loop's reliability, and it should be read before the gain figure is read at all.

Rows predating the Step 9 measurement fields carry no effort and are dropped as legacy with a count. A stats file where no row carries effort is a hard error, not an empty treatment group: a schema gap must never present itself as "no gain".

4. The measurement runs (operator-run, outside this plan)

The measurement itself is not part of the implementation plan that built this harness. It is an operator-run gate between that plan and any adopt commit.

Protocol:

• n ≥ 5 runs per arm. Fewer, and the median is an anecdote.
• The same question set in both arms. Two briefs, run as --project runs.
• The arms differ in exactly one thing: whether the loop is enabled.
• Both arms append to the same trekresearch-stats.jsonl; effort is the grouping key that separates them.

# Control arm (loop inert) — n >= 5
claude -p "/trekresearch --project .claude/projects/<brief-standard-effort>"

# Treatment arm (loop live, effort: high in the brief's phase_signals) — n >= 5
VOYAGE_STORM_ENABLED=1 \
  claude -p "/trekresearch --project .claude/projects/<brief-high-effort>"

# The gate
node scripts/storm-measure.mjs --stats "${CLAUDE_PLUGIN_DATA}/trekresearch-stats.jsonl"

# Machine-readable, for a decision record
node scripts/storm-measure.mjs --json --stats "${CLAUDE_PLUGIN_DATA}/trekresearch-stats.jsonl"

# SC activation check — did the latest high-effort run discover any dimension
# beyond its interview baseline? Exit 0 = yes.
node scripts/storm-measure.mjs --activation-check --stats "${CLAUDE_PLUGIN_DATA}/trekresearch-stats.jsonl"

5. What "adopt" concretely means

Adoption is one constant: isStormEnabled() in lib/util/research-loop-cap.mjs currently requires VOYAGE_STORM_ENABLED === '1'. Adopt = make the loop's budget non-zero without that opt-in, in a commit that cites the measurement output.

This asymmetry is deliberate and was designed in before any code was written:

• decline costs nothing — the mechanism ships default-off, so declining is doing nothing. No revert, no removal from a command file two other steps already rewrote.
• adopt costs one constant — plus the enforcement hook (hooks/scripts/pre-agent-cap.mjs) already in place to bound what gets turned on, and the operator-visible cap-exhaustion message already required by Phase 5's exit conditions.

6. Reading the result honestly

• The gate measures breadth. Say "breadth" in the decision record, not "quality".
• Report the excluded count alongside the gain, always. A 35% gain computed after excluding 6 of 10 treatment runs is a finding about instability.
• insufficient-data is not a decline. Do not resolve it by lowering n.
• A verdict computed from a stats file mixing several question sets measures the question sets, not the loop.