Kjell Tore Guttormsen
dc245af408
The SC asks two things of an effort: high run: that it discovered at least one dimension AND that the dimension list in the output brief is a TRUE SUPERSET of the interview-derived ones. activationCheck computed dimensions - dimensions_baseline >= 1 and returned ok on that alone. A count delta says nothing about membership: a run that dropped two interview dimensions and appended three discovered ones is +1 and passed the check while violating the second half outright. Supersetness was asserted only by Phase 4.5's prose contract that discovery appends - nothing read it. The stats record cannot carry the dimension names that would show it directly: names are free prose, and lib/exporters/field-allowlist.mjs denies prose by omission (question, project_dir, brief_path are already excluded for that reason). So the run attests membership with dimensions_baseline_preserved, a boolean set in a new Phase 4.5 step 5, and the gate refuses to call activation OK without it. An ABSENT attestation fails rather than falling back to the old count-only rule - otherwise legacy rows would keep passing on the defect. Producer side wired end to end: the record format, the measurement-fields prose (five fields -> six), the exporter allowlist, the jsonl-schemas fixture row, and the --activation-check comment in docs/storm-measurement.md. Review finding d2786604458207a5a73478cdcb6a54bbdb92141d (MINOR). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LuGhWAbWyRFBFeemfhxoVv
131 lines
6.6 KiB
Markdown
131 lines
6.6 KiB
Markdown
# STORM adoption gate — pre-registered measurement protocol
|
||
|
||
**Status:** thresholds registered, **no measurement run yet.** This document is
|
||
committed *before* the first measurement by design: a threshold chosen after
|
||
seeing the numbers is not a threshold. **Harness:** `scripts/storm-measure.mjs`
|
||
(pinned by `tests/scripts/storm-measure.test.mjs`). **Decides:** whether the
|
||
bounded Phase 5 conversation loop (`commands/trekresearch.md` §Loop bound) is
|
||
worth turning on by default.
|
||
|
||
---
|
||
|
||
## 1. What this gate measures — and what it does not
|
||
|
||
The gate measures **source and coverage breadth**:
|
||
|
||
| Metric | Definition | Arm |
|
||
|---|---|---|
|
||
| `unique_sources` | distinct external sources cited by the run | between-arm median gain |
|
||
| dimensions over baseline | `(dimensions − dimensions_baseline) / dimensions_baseline` | within-run median across the treatment arm |
|
||
|
||
It does **not** measure outline quality, answer correctness, synthesis
|
||
usefulness, or operator satisfaction. A breadth win is a *necessary* condition
|
||
for adoption, never a sufficient one. If the loop widens coverage by 40% and
|
||
the resulting briefs read worse, the correct action is still decline — the
|
||
number does not overrule a reading of the artifacts.
|
||
|
||
The second metric is a within-run delta by construction (`dimensions_baseline`
|
||
is the interview dimension count, `dimensions` the post-Phase-4.5 list), so it
|
||
needs no control arm; the control arm's value is 0 because the loop is inert
|
||
below `effort: high`.
|
||
|
||
## 2. Pre-registered thresholds
|
||
|
||
Either metric clearing the bar is enough. The brief pre-registers "median
|
||
forbedring ≥ 30 % på (a) eller (b) → adopt. < 15 % → decline" — a breadth win
|
||
on one axis counts, because either axis widening is the effect the loop claims.
|
||
Adopt is evaluated first, so a run that clears the adopt bar on one metric is
|
||
an adopt even when the other metric sits under the decline bar.
|
||
|
||
| Median gain | Verdict | Action |
|
||
|---|---|---|
|
||
| ≥ 30% on **either** metric | **adopt** | Flip the `VOYAGE_STORM_ENABLED` default (see §5) |
|
||
| < 15% on **either** metric (and no adopt) | **decline** | Leave the mechanism default-off. This is a **no-op**: nothing is rolled back |
|
||
| both metrics in 15% – 30% | **inconclusive** | Keep default-off, gather more runs, re-measure |
|
||
| either arm empty | **insufficient-data** | Not a decline — measure more |
|
||
|
||
`ADOPT_THRESHOLD = 0.30` and `DECLINE_THRESHOLD = 0.15` are exported constants
|
||
in `scripts/storm-measure.mjs` and pinned by the test suite. Changing them is a
|
||
deliberate, reviewable act, not a tuning knob to be nudged toward a result.
|
||
|
||
## 3. Excluded runs (the honest denominator)
|
||
|
||
Runs with `empty_turns > 0` are **excluded from the gain and reported as a
|
||
count**. An empty turn is one that spent budget and returned no findings, or
|
||
findings without citations. A run whose `empty_turns` cannot be **read** as a
|
||
number is excluded on the same footing: a field that says `"many"` is not
|
||
evidence of zero empty turns, and treating it as one would let the least
|
||
trustworthy run back into the denominator in the direction that flatters
|
||
adoption. Including those runs decides adoption on a broken
|
||
denominator — the loop looks cheap because its failures are averaged into its
|
||
successes. The harness prints the excluded count on every invocation; if that
|
||
count is a large fraction of the treatment arm, the finding is about the loop's
|
||
reliability, and it should be read before the gain figure is read at all.
|
||
|
||
Rows predating the Step 9 measurement fields carry no `effort` and are dropped
|
||
as `legacy` with a count. A stats file where *no* row carries `effort` is a hard
|
||
error, not an empty treatment group: a schema gap must never present itself as
|
||
"no gain".
|
||
|
||
## 4. The measurement runs (operator-run, outside this plan)
|
||
|
||
The measurement itself is **not** part of the implementation plan that built
|
||
this harness. It is an operator-run gate between that plan and any adopt commit.
|
||
|
||
Protocol:
|
||
|
||
- **n ≥ 5 runs per arm.** Fewer, and the median is an anecdote.
|
||
- **The same question set in both arms.** Two briefs, run as `--project` runs.
|
||
- The arms differ in exactly one thing: whether the loop is enabled.
|
||
- Both arms append to the same `trekresearch-stats.jsonl`; `effort` is the
|
||
grouping key that separates them.
|
||
|
||
```bash
|
||
# Control arm (loop inert) — n >= 5
|
||
claude -p "/trekresearch --project .claude/projects/<brief-standard-effort>"
|
||
|
||
# Treatment arm (loop live, effort: high in the brief's phase_signals) — n >= 5
|
||
VOYAGE_STORM_ENABLED=1 \
|
||
claude -p "/trekresearch --project .claude/projects/<brief-high-effort>"
|
||
|
||
# The gate
|
||
node scripts/storm-measure.mjs --stats "${CLAUDE_PLUGIN_DATA}/trekresearch-stats.jsonl"
|
||
|
||
# Machine-readable, for a decision record
|
||
node scripts/storm-measure.mjs --json --stats "${CLAUDE_PLUGIN_DATA}/trekresearch-stats.jsonl"
|
||
|
||
# SC activation check — BOTH halves: did the latest high-effort run discover at
|
||
# least one dimension, AND is its final list a true SUPERSET of the interview
|
||
# ones? The second half is read from the run's own `dimensions_baseline_preserved`
|
||
# attestation, because the dimension NAMES that would show it directly are prose
|
||
# the exporter allowlist denies. A run that does not attest it FAILS — dropping
|
||
# two interview dimensions and appending three discovered ones is a +1 count
|
||
# delta and not a superset. Exit 0 = both halves hold.
|
||
node scripts/storm-measure.mjs --activation-check --stats "${CLAUDE_PLUGIN_DATA}/trekresearch-stats.jsonl"
|
||
```
|
||
|
||
## 5. What "adopt" concretely means
|
||
|
||
Adoption is **one constant**: `isStormEnabled()` in
|
||
`lib/util/research-loop-cap.mjs` currently requires `VOYAGE_STORM_ENABLED === '1'`.
|
||
Adopt = make the loop's budget non-zero without that opt-in, in a commit that
|
||
cites the measurement output.
|
||
|
||
This asymmetry is deliberate and was designed in before any code was written:
|
||
|
||
- **decline costs nothing** — the mechanism ships default-off, so declining is
|
||
doing nothing. No revert, no removal from a command file two other steps
|
||
already rewrote.
|
||
- **adopt costs one constant** — plus the enforcement hook
|
||
(`hooks/scripts/pre-agent-cap.mjs`) already in place to bound what gets turned
|
||
on, and the operator-visible cap-exhaustion message already required by
|
||
Phase 5's exit conditions.
|
||
|
||
## 6. Reading the result honestly
|
||
|
||
- The gate measures breadth. Say "breadth" in the decision record, not "quality".
|
||
- Report the excluded count alongside the gain, always. A 35% gain computed
|
||
after excluding 6 of 10 treatment runs is a finding about instability.
|
||
- `insufficient-data` is not a decline. Do not resolve it by lowering n.
|
||
- A verdict computed from a stats file mixing several question sets measures the
|
||
question sets, not the loop.
|