STORM adoption gate — pre-registered measurement protocol

Status: thresholds registered, no measurement run yet. This document is committed before the first measurement by design: a threshold chosen after seeing the numbers is not a threshold. Harness: scripts/storm-measure.mjs (pinned by tests/scripts/storm-measure.test.mjs). Decides: whether the bounded Phase 5 conversation loop (commands/trekresearch.md §Loop bound) is worth turning on by default.

1. What this gate measures — and what it does not

The gate measures source and coverage breadth:

Metric	Definition	Arm
`unique_sources`	distinct external sources cited by the run	between-arm median gain
dimensions over baseline	`(dimensions − dimensions_baseline) / dimensions_baseline`	within-run median across the treatment arm

It does not measure outline quality, answer correctness, synthesis usefulness, or operator satisfaction. A breadth win is a necessary condition for adoption, never a sufficient one. If the loop widens coverage by 40% and the resulting briefs read worse, the correct action is still decline — the number does not overrule a reading of the artifacts.

The second metric is a within-run delta by construction (dimensions_baseline is the interview dimension count, dimensions the post-Phase-4.5 list), so it needs no control arm; the control arm's value is 0 because the loop is inert below effort: high.

2. Pre-registered thresholds

Both metrics must clear the bar. A strong result on one axis does not carry a weak result on the other — the loop's claim is breadth on both.

Median gain (BOTH metrics)	Verdict	Action
≥ 30%	adopt	Flip the `VOYAGE_STORM_ENABLED` default (see §5)
< 15%	decline	Leave the mechanism default-off. This is a no-op: nothing is rolled back
15% – 30%	inconclusive	Keep default-off, gather more runs, re-measure
either arm empty	insufficient-data	Not a decline — measure more

ADOPT_THRESHOLD = 0.30 and DECLINE_THRESHOLD = 0.15 are exported constants in scripts/storm-measure.mjs and pinned by the test suite. Changing them is a deliberate, reviewable act, not a tuning knob to be nudged toward a result.

3. Excluded runs (the honest denominator)

Runs with empty_turns > 0 are excluded from the gain and reported as a count. An empty turn is one that spent budget and returned no findings, or findings without citations. Including those runs decides adoption on a broken denominator — the loop looks cheap because its failures are averaged into its successes. The harness prints the excluded count on every invocation; if that count is a large fraction of the treatment arm, the finding is about the loop's reliability, and it should be read before the gain figure is read at all.

Rows predating the Step 9 measurement fields carry no effort and are dropped as legacy with a count. A stats file where no row carries effort is a hard error, not an empty treatment group: a schema gap must never present itself as "no gain".

4. The measurement runs (operator-run, outside this plan)

The measurement itself is not part of the implementation plan that built this harness. It is an operator-run gate between that plan and any adopt commit.

Protocol:

n ≥ 5 runs per arm. Fewer, and the median is an anecdote.
The same question set in both arms. Two briefs, run as --project runs.
The arms differ in exactly one thing: whether the loop is enabled.
Both arms append to the same trekresearch-stats.jsonl; effort is the grouping key that separates them.

# Control arm (loop inert) — n >= 5
claude -p "/trekresearch --project .claude/projects/<brief-standard-effort>"

# Treatment arm (loop live, effort: high in the brief's phase_signals) — n >= 5
VOYAGE_STORM_ENABLED=1 \
  claude -p "/trekresearch --project .claude/projects/<brief-high-effort>"

# The gate
node scripts/storm-measure.mjs --stats "${CLAUDE_PLUGIN_DATA}/trekresearch-stats.jsonl"

# Machine-readable, for a decision record
node scripts/storm-measure.mjs --json --stats "${CLAUDE_PLUGIN_DATA}/trekresearch-stats.jsonl"

# SC activation check — did the latest high-effort run discover any dimension
# beyond its interview baseline? Exit 0 = yes.
node scripts/storm-measure.mjs --activation-check --stats "${CLAUDE_PLUGIN_DATA}/trekresearch-stats.jsonl"

5. What "adopt" concretely means

Adoption is one constant: isStormEnabled() in lib/util/research-loop-cap.mjs currently requires VOYAGE_STORM_ENABLED === '1'. Adopt = make the loop's budget non-zero without that opt-in, in a commit that cites the measurement output.

This asymmetry is deliberate and was designed in before any code was written:

decline costs nothing — the mechanism ships default-off, so declining is doing nothing. No revert, no removal from a command file two other steps already rewrote.
adopt costs one constant — plus the enforcement hook (hooks/scripts/pre-agent-cap.mjs) already in place to bound what gets turned on, and the operator-visible cap-exhaustion message already required by Phase 5's exit conditions.

6. Reading the result honestly

The gate measures breadth. Say "breadth" in the decision record, not "quality".
Report the excluded count alongside the gain, always. A 35% gain computed after excluding 6 of 10 treatment runs is a finding about instability.
insufficient-data is not a decline. Do not resolve it by lowering n.
A verdict computed from a stats file mixing several question sets measures the question sets, not the loop.

5.6 KiB Raw Blame History Unescape Escape