voyage/docs/storm-measurement.md

STORM adoption gate — pre-registered measurement protocol

Status: thresholds registered, no measurement run yet. This document is committed before the first measurement by design: a threshold chosen after seeing the numbers is not a threshold. Harness: scripts/storm-measure.mjs (pinned by tests/scripts/storm-measure.test.mjs). Decides: whether the bounded Phase 5 conversation loop (commands/trekresearch.md §Loop bound) is worth turning on by default.

---

1. What this gate measures — and what it does not

The gate measures source and coverage breadth:

Metric	Definition	Arm
unique_sources	distinct external sources cited by the run	between-arm median gain
dimensions over baseline	(dimensions − dimensions_baseline) / dimensions_baseline	within-run median across the treatment arm

It does not measure outline quality, answer correctness, synthesis usefulness, or operator satisfaction. A breadth win is a necessary condition for adoption, never a sufficient one. If the loop widens coverage by 40% and the resulting briefs read worse, the correct action is still decline — the number does not overrule a reading of the artifacts.

The second metric is a within-run delta by construction (dimensions_baseline is the interview dimension count, dimensions the post-Phase-4.5 list), so it needs no control arm; the control arm's value is 0 because the loop is inert below effort: high.

2. Pre-registered thresholds

Either metric clearing the bar is enough. The brief pre-registers "median forbedring ≥ 30 % på (a) eller (b) → adopt. < 15 % → decline" — a breadth win on one axis counts, because either axis widening is the effect the loop claims. Adopt is evaluated first, so a run that clears the adopt bar on one metric is an adopt even when the other metric sits under the decline bar.

Median gain	Verdict	Action
≥ 30% on either metric	adopt	Flip the VOYAGE_STORM_ENABLED default (see §5)
< 15% on either metric (and no adopt)	decline	Leave the mechanism default-off. This is a no-op: nothing is rolled back
both metrics in 15% – 30%	inconclusive	Keep default-off, gather more runs, re-measure
either arm empty	insufficient-data	Not a decline — measure more

ADOPT_THRESHOLD = 0.30 and DECLINE_THRESHOLD = 0.15 are exported constants in scripts/storm-measure.mjs and pinned by the test suite. Changing them is a deliberate, reviewable act, not a tuning knob to be nudged toward a result.

3. Excluded runs (the honest denominator)

Runs with empty_turns > 0 are excluded from the gain and reported as a count. An empty turn is one that spent budget and returned no findings, or findings without citations. A run whose empty_turns cannot be read as a number is excluded on the same footing: a field that says "many" is not evidence of zero empty turns, and treating it as one would let the least trustworthy run back into the denominator in the direction that flatters adoption. Including those runs decides adoption on a broken denominator — the loop looks cheap because its failures are averaged into its successes. The harness prints the excluded count on every invocation; if that count is a large fraction of the treatment arm, the finding is about the loop's reliability, and it should be read before the gain figure is read at all.

Rows predating the Step 9 measurement fields carry no effort and are dropped as legacy with a count. A stats file where no row carries effort is a hard error, not an empty treatment group: a schema gap must never present itself as "no gain".

4. The measurement runs (operator-run, outside this plan)

The measurement itself is not part of the implementation plan that built this harness. It is an operator-run gate between that plan and any adopt commit.

Protocol:

• n ≥ 5 runs per arm. Fewer, and the median is an anecdote.
• The same question set in both arms. Two briefs, run as --project runs.
• The arms differ in exactly one thing: whether the loop is enabled.
• Both arms append to the same trekresearch-stats.jsonl; effort is the grouping key that separates them.

# Control arm (loop inert) — n >= 5
claude -p "/trekresearch --project .claude/projects/<brief-standard-effort>"

# Treatment arm (loop live, effort: high in the brief's phase_signals) — n >= 5
VOYAGE_STORM_ENABLED=1 \
  claude -p "/trekresearch --project .claude/projects/<brief-high-effort>"

# The gate
node scripts/storm-measure.mjs --stats "${CLAUDE_PLUGIN_DATA}/trekresearch-stats.jsonl"

# Machine-readable, for a decision record
node scripts/storm-measure.mjs --json --stats "${CLAUDE_PLUGIN_DATA}/trekresearch-stats.jsonl"

# SC activation check — BOTH halves: did the latest high-effort run discover at
# least one dimension, AND is its final list a true SUPERSET of the interview
# ones? The second half is read from the run's own `dimensions_baseline_preserved`
# attestation, because the dimension NAMES that would show it directly are prose
# the exporter allowlist denies. A run that does not attest it FAILS — dropping
# two interview dimensions and appending three discovered ones is a +1 count
# delta and not a superset. Exit 0 = both halves hold.
node scripts/storm-measure.mjs --activation-check --stats "${CLAUDE_PLUGIN_DATA}/trekresearch-stats.jsonl"

5. What "adopt" concretely means

Adoption is one constant: isStormEnabled() in lib/util/research-loop-cap.mjs currently requires VOYAGE_STORM_ENABLED === '1'. Adopt = make the loop's budget non-zero without that opt-in, in a commit that cites the measurement output.

This asymmetry is deliberate and was designed in before any code was written:

• decline costs nothing — the mechanism ships default-off, so declining is doing nothing. No revert, no removal from a command file two other steps already rewrote.
• adopt costs one constant — plus the enforcement hook (hooks/scripts/pre-agent-cap.mjs) already in place to bound what gets turned on, and the operator-visible cap-exhaustion message already required by Phase 5's exit conditions.

6. Reading the result honestly

• The gate measures breadth. Say "breadth" in the decision record, not "quality".
• Report the excluded count alongside the gain, always. A 35% gain computed after excluding 6 of 10 treatment runs is a finding about instability.
• insufficient-data is not a decline. Do not resolve it by lowering n.
• A verdict computed from a stats file mixing several question sets measures the question sets, not the loop.