Kjell Tore Guttormsen
7d5e4fcbcc
Two defects in the adoption gate, both in the direction that flatters
adoption.
1. A non-numeric empty_turns counted as an eligible run.
Number('many') is NaN, and the test was `Number.isFinite(empty) &&
empty > 0`, so NaN fell through to the eligible branch. Measured
before: 0 -> eligible, 2 -> excluded, undefined -> eligible, null ->
eligible, 'many' -> ELIGIBLE, NaN -> ELIGIBLE. The exclusion is one of
the two properties docs/storm-measurement.md names as carrying this
gate's honesty, and the run whose bookkeeping broke is the run whose
numbers deserve the least trust. Now excluded. Absent and null stay
eligible via `?? 0` - a field never written is a genuine zero on a run
where the loop never armed.
2. The printed threshold line said "adopt >= 30.0% on BOTH - decline <
15.0% on BOTH" while decideVerdict evaluates OR on both sides. S82
restored the pre-registered OR rule in the logic (c37bf50d) and left
this line describing the stricter AND gate, one line above the verdict
that OR produced. The summary is the only form of the rule most readers
ever see, so it now states EITHER on both sides and that adopt wins
ties. Found while fixing (1); not a review finding.
A test pins the printed line against the string "on BOTH" so the two
cannot drift apart again silently.
Review finding 24a76c21ffc694cd782cd449212c9502d31aeda6 (MINOR).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LuGhWAbWyRFBFeemfhxoVv
126 lines
6.2 KiB
Markdown
126 lines
6.2 KiB
Markdown
# STORM adoption gate — pre-registered measurement protocol
|
||
|
||
**Status:** thresholds registered, **no measurement run yet.** This document is
|
||
committed *before* the first measurement by design: a threshold chosen after
|
||
seeing the numbers is not a threshold. **Harness:** `scripts/storm-measure.mjs`
|
||
(pinned by `tests/scripts/storm-measure.test.mjs`). **Decides:** whether the
|
||
bounded Phase 5 conversation loop (`commands/trekresearch.md` §Loop bound) is
|
||
worth turning on by default.
|
||
|
||
---
|
||
|
||
## 1. What this gate measures — and what it does not
|
||
|
||
The gate measures **source and coverage breadth**:
|
||
|
||
| Metric | Definition | Arm |
|
||
|---|---|---|
|
||
| `unique_sources` | distinct external sources cited by the run | between-arm median gain |
|
||
| dimensions over baseline | `(dimensions − dimensions_baseline) / dimensions_baseline` | within-run median across the treatment arm |
|
||
|
||
It does **not** measure outline quality, answer correctness, synthesis
|
||
usefulness, or operator satisfaction. A breadth win is a *necessary* condition
|
||
for adoption, never a sufficient one. If the loop widens coverage by 40% and
|
||
the resulting briefs read worse, the correct action is still decline — the
|
||
number does not overrule a reading of the artifacts.
|
||
|
||
The second metric is a within-run delta by construction (`dimensions_baseline`
|
||
is the interview dimension count, `dimensions` the post-Phase-4.5 list), so it
|
||
needs no control arm; the control arm's value is 0 because the loop is inert
|
||
below `effort: high`.
|
||
|
||
## 2. Pre-registered thresholds
|
||
|
||
Either metric clearing the bar is enough. The brief pre-registers "median
|
||
forbedring ≥ 30 % på (a) eller (b) → adopt. < 15 % → decline" — a breadth win
|
||
on one axis counts, because either axis widening is the effect the loop claims.
|
||
Adopt is evaluated first, so a run that clears the adopt bar on one metric is
|
||
an adopt even when the other metric sits under the decline bar.
|
||
|
||
| Median gain | Verdict | Action |
|
||
|---|---|---|
|
||
| ≥ 30% on **either** metric | **adopt** | Flip the `VOYAGE_STORM_ENABLED` default (see §5) |
|
||
| < 15% on **either** metric (and no adopt) | **decline** | Leave the mechanism default-off. This is a **no-op**: nothing is rolled back |
|
||
| both metrics in 15% – 30% | **inconclusive** | Keep default-off, gather more runs, re-measure |
|
||
| either arm empty | **insufficient-data** | Not a decline — measure more |
|
||
|
||
`ADOPT_THRESHOLD = 0.30` and `DECLINE_THRESHOLD = 0.15` are exported constants
|
||
in `scripts/storm-measure.mjs` and pinned by the test suite. Changing them is a
|
||
deliberate, reviewable act, not a tuning knob to be nudged toward a result.
|
||
|
||
## 3. Excluded runs (the honest denominator)
|
||
|
||
Runs with `empty_turns > 0` are **excluded from the gain and reported as a
|
||
count**. An empty turn is one that spent budget and returned no findings, or
|
||
findings without citations. A run whose `empty_turns` cannot be **read** as a
|
||
number is excluded on the same footing: a field that says `"many"` is not
|
||
evidence of zero empty turns, and treating it as one would let the least
|
||
trustworthy run back into the denominator in the direction that flatters
|
||
adoption. Including those runs decides adoption on a broken
|
||
denominator — the loop looks cheap because its failures are averaged into its
|
||
successes. The harness prints the excluded count on every invocation; if that
|
||
count is a large fraction of the treatment arm, the finding is about the loop's
|
||
reliability, and it should be read before the gain figure is read at all.
|
||
|
||
Rows predating the Step 9 measurement fields carry no `effort` and are dropped
|
||
as `legacy` with a count. A stats file where *no* row carries `effort` is a hard
|
||
error, not an empty treatment group: a schema gap must never present itself as
|
||
"no gain".
|
||
|
||
## 4. The measurement runs (operator-run, outside this plan)
|
||
|
||
The measurement itself is **not** part of the implementation plan that built
|
||
this harness. It is an operator-run gate between that plan and any adopt commit.
|
||
|
||
Protocol:
|
||
|
||
- **n ≥ 5 runs per arm.** Fewer, and the median is an anecdote.
|
||
- **The same question set in both arms.** Two briefs, run as `--project` runs.
|
||
- The arms differ in exactly one thing: whether the loop is enabled.
|
||
- Both arms append to the same `trekresearch-stats.jsonl`; `effort` is the
|
||
grouping key that separates them.
|
||
|
||
```bash
|
||
# Control arm (loop inert) — n >= 5
|
||
claude -p "/trekresearch --project .claude/projects/<brief-standard-effort>"
|
||
|
||
# Treatment arm (loop live, effort: high in the brief's phase_signals) — n >= 5
|
||
VOYAGE_STORM_ENABLED=1 \
|
||
claude -p "/trekresearch --project .claude/projects/<brief-high-effort>"
|
||
|
||
# The gate
|
||
node scripts/storm-measure.mjs --stats "${CLAUDE_PLUGIN_DATA}/trekresearch-stats.jsonl"
|
||
|
||
# Machine-readable, for a decision record
|
||
node scripts/storm-measure.mjs --json --stats "${CLAUDE_PLUGIN_DATA}/trekresearch-stats.jsonl"
|
||
|
||
# SC activation check — did the latest high-effort run discover any dimension
|
||
# beyond its interview baseline? Exit 0 = yes.
|
||
node scripts/storm-measure.mjs --activation-check --stats "${CLAUDE_PLUGIN_DATA}/trekresearch-stats.jsonl"
|
||
```
|
||
|
||
## 5. What "adopt" concretely means
|
||
|
||
Adoption is **one constant**: `isStormEnabled()` in
|
||
`lib/util/research-loop-cap.mjs` currently requires `VOYAGE_STORM_ENABLED === '1'`.
|
||
Adopt = make the loop's budget non-zero without that opt-in, in a commit that
|
||
cites the measurement output.
|
||
|
||
This asymmetry is deliberate and was designed in before any code was written:
|
||
|
||
- **decline costs nothing** — the mechanism ships default-off, so declining is
|
||
doing nothing. No revert, no removal from a command file two other steps
|
||
already rewrote.
|
||
- **adopt costs one constant** — plus the enforcement hook
|
||
(`hooks/scripts/pre-agent-cap.mjs`) already in place to bound what gets turned
|
||
on, and the operator-visible cap-exhaustion message already required by
|
||
Phase 5's exit conditions.
|
||
|
||
## 6. Reading the result honestly
|
||
|
||
- The gate measures breadth. Say "breadth" in the decision record, not "quality".
|
||
- Report the excluded count alongside the gain, always. A 35% gain computed
|
||
after excluding 6 of 10 treatment runs is a finding about instability.
|
||
- `insufficient-data` is not a decline. Do not resolve it by lowering n.
|
||
- A verdict computed from a stats file mixing several question sets measures the
|
||
question sets, not the loop.
|