open/voyage
1
0
Fork 0
Code Issues Releases 1 Activity

fix(storm-measure): exclude an unreadable empty_turns, and stop printing BOTH for an OR rule

Browse source 
Two defects in the adoption gate, both in the direction that flatters
adoption.

1. A non-numeric empty_turns counted as an eligible run.
   Number('many') is NaN, and the test was `Number.isFinite(empty) &&
   empty > 0`, so NaN fell through to the eligible branch. Measured
   before: 0 -> eligible, 2 -> excluded, undefined -> eligible, null ->
   eligible, 'many' -> ELIGIBLE, NaN -> ELIGIBLE. The exclusion is one of
   the two properties docs/storm-measurement.md names as carrying this
   gate's honesty, and the run whose bookkeeping broke is the run whose
   numbers deserve the least trust. Now excluded. Absent and null stay
   eligible via `?? 0` - a field never written is a genuine zero on a run
   where the loop never armed.

2. The printed threshold line said "adopt >= 30.0% on BOTH - decline <
   15.0% on BOTH" while decideVerdict evaluates OR on both sides. S82
   restored the pre-registered OR rule in the logic (c37bf50d) and left
   this line describing the stricter AND gate, one line above the verdict
   that OR produced. The summary is the only form of the rule most readers
   ever see, so it now states EITHER on both sides and that adopt wins
   ties. Found while fixing (1); not a review finding.

A test pins the printed line against the string "on BOTH" so the two
cannot drift apart again silently.

Review finding 24a76c21ffc694cd782cd449212c9502d31aeda6 (MINOR).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LuGhWAbWyRFBFeemfhxoVv
This commit is contained in:
Kjell Tore Guttormsen 2026-08-12 22:59:08 +02:00
commit 7d5e4fcbcc
3 changed files with 61 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

6
docs/storm-measurement.md
View file

@ -52,7 +52,11 @@ deliberate, reviewable act, not a tuning knob to be nudged toward a result.
Runs with `empty_turns > 0` are **excluded from the gain and reported as a
count**. An empty turn is one that spent budget and returned no findings, or
findings without citations. Including those runs decides adoption on a broken
findings without citations. A run whose `empty_turns` cannot be **read** as a
number is excluded on the same footing: a field that says `"many"` is not
evidence of zero empty turns, and treating it as one would let the least
trustworthy run back into the denominator in the direction that flatters
adoption. Including those runs decides adoption on a broken
denominator — the loop looks cheap because its failures are averaged into its
successes. The harness prints the excluded count on every invocation; if that
count is a large fraction of the treatment arm, the finding is about the loop's