open/voyage
1
0
Fork 0
Code Issues Releases 5 Activity

docs(proevesett): stability with its denominator; v2's 8 of 8 was one sample

Browse source 
New § Stability: why case 4 was unstable (it could halt on the intent gate,
5 of 6 here before the fix, 6 of 12 and 7 of 10 in the PM re-measurement),
the three changes, the case-4 known-positive (3 of 3 passed with the gate's
code off until names-code; then 0 of 3), and the says-fail cause left
unmeasured (the failing trace was deleted; 0 of 13 kept replies used
another spelling).

After the fix (a384ce6): three full runs (one --runs 3, two --runs 1),
every case 5 of 5, each full run 8 of 8; case 4 alone 10 of 10, 15 of 15
in total. The status line no longer lets 8 of 8 read as a stable state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-23 10:59:59 +02:00
commit e7687650c4
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
1 changed files with 66 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

68
docs/proevesett.md
View file

@ -5,6 +5,11 @@
expectations were committed *before* its first run; the run did not change them.
Red is data here, not a defect in the suite. What v2 changed and why: § v2 below.
**v2's first 8 of 8 was one sample, not a stable state.** Measured the same day,
before the stability fix: case 4 passed 5 of 6 here, 6 of 12 and 7 of 10 in the PM
re-measurement. After the fix: every case **5 of 5** over three full runs, case 4
also **10 of 10** on its own. Numbers per case, with the denominator: § Stability.
The unit suite (`npm test`) proves Voyage's modules in isolation. This suite proves
something the unit suite cannot: that the **commands, as a model reads and follows
them in a real headless session**, keep the promises they make. Each case is a real
@ -101,7 +106,7 @@ All new patterns were run against the ten traces of the PM re-measurement. There
were no false positives, and the known-positive and known-negative still split.
**v2 first run: 8 of 8** (graders 36 of 36, 65 s, $1.97; the tool's list-price
estimate).
estimate). That was one run. It did not hold: see § Stability.
**Known-positive for the new case, demonstrated** in a scratch copy (never this
tree):
@ -112,6 +117,64 @@ tree):
measures whether the gate works, not whether that exact line is there. The line is
pinned by `tests/validators/intent-approval.test.mjs`, which executes it.
## Stability (2026-09-23): case 4 measures its own gate
**What was unstable.** The PM re-measurement of v2 got 6 of 8 and 7 of 8 in two full
runs. Case 4 `plan-halts-without-phase-signals` passed **6 of 12** on the v2 commit
and **7 of 10** on the commit before it. The failure was the same every time. The
case-4 brief had no intent marker, so the child sometimes ran the intent gate first
and halted on `BRIEF_INTENT_NOT_APPROVED`. That halt was correct, but the brief-validator
never ran, so `validator-ran` and `halt-names-signals` failed. The case measured the
wrong gate. Re-measured here before the fix: **5 of 6**, and the one failure had the
same trace.
| Change | Why |
|--------|-----|
| The case-4 scaffold writes a current `intent_approved_hash` into the brief | The intent gate now passes, so only the sequencing gate can stop the case. The marker is written literally, not by `intent-approval.mjs --stamp`: `--stamp` emits a `brief-approved` record into the live stats that the yardstick reads |
| New case-4 grader `no-intent-halt` (trace, `not_contains`, any `BRIEF_INTENT_*` code) | If the marker ever goes stale, the case fails and names the reason. A halt on the intent gate no longer counts as a pass |
| New case-4 grader `names-code` (the validator's `BRIEF_V51_MISSING_SIGNALS` code in the trace) | The known-positive below: without it, the case passed with the gate's code switched off |
| `says-fail` / `says-pass` take `FAILED` / `PASSED` as the verdict word | v2's word boundary rejected them. v1's raw substring accepted them. A correct reply must not fail the case |
**Known-positive for case 4, demonstrated** in a scratch copy (never this tree). The
fault planted is the one the case exists to catch: the `BRIEF_V51_MISSING_SIGNALS`
check in `brief-validator.mjs` was made `if (false)`.
- Without `names-code`, the case still passed **3 of 3**. The child read the rule in
the command prose, halted, named `phase_signals`, and even pointed at the planted
`if (false)`. A correct halt does not show that the gate's code fired.
- With `names-code`, it fell **0 of 3**. `names-code` failed each time, and the other
five graders passed.
- Before committing, the pattern was checked on this session's recorded traces. It
matched 10 of 10 green runs and 0 of 3 fault runs.
**`says-fail` (case 6), cause not measured.** The PM saw 1 failure in 7 runs on
`says-fail`. That run's trace was deleted, so the reply cannot be read. The
hypothesis was that the reply said `FAILED`. It is not confirmed: 0 of 13 kept
replies (8 here, 5 from the PM) used any spelling but `FAIL`. What is certain is that the
v2 pattern rejects `FAILED`, and a reply saying `FAILED` is correct. So the pattern is
widened, and the failure stays unexplained. If it comes back, it is not this.
**Measured stability after the fix** (`a384ce6`, Claude Code 2.1.280). Full run =
all eight cases, the command under § How to run it. There were three: one with
`--runs 3` and two with `--runs 1`. That gives 5 children per case.
| Case | Full runs | Alone | Total |
|------|-----------|-------|-------|
| `plan-halts-without-intent-approval` | 5 of 5 | – | 5 of 5 |
| `plan-halts-without-phase-signals` | 5 of 5 | 10 of 10 | **15 of 15** |
| `plan-project-not-initialized` | 5 of 5 | – | 5 of 5 |
| `plan-rejects-unknown-export` | 5 of 5 | – | 5 of 5 |
| `plan-requires-brief` | 5 of 5 | – | 5 of 5 |
| `review-requires-project` | 5 of 5 | – | 5 of 5 |
| `review-validate-flags-bad-finding-id` | 5 of 5 | 8 of 8 (v2 grader, before widening) | 13 of 13 |
| `review-validate-passes-clean-review` | 5 of 5 | – | 5 of 5 |
In 4 of the 15 case-4 runs, the child ran the intent gate as its own call before the
brief-validator. In 1 more, it ran both gates in one call. All 5 passed the intent
gate, went on to the sequencing gate and passed the case. Before the fix, a run that
ran the intent gate alone first halted there. Every full run gave **8 of 8**.
A pass rate of 15 of 15 does not prove the rate is 1: it bounds it. Five runs per
case is a small denominator. The claim is "no failure in N", not "cannot fail".
## Measured facts about the harness this suite relies on
These were measured by probe cases in a scratch copy, not taken from the docs.
@ -177,4 +240,5 @@ Because `runs: 1` is a sample, an unstable case such as `review-requires-project
will sometimes stop a change that is innocent. That is the intended price. Moving the guard
into code removes the instability; editing the grader does not. v2 did the move: the
parser now writes the line, and the grader reads the parser's code. One green run does
not show the instability is gone. That takes more runs.
not show the instability is gone. That takes more runs. § Stability has them:
`review-requires-project` 5 of 5, and case 4 15 of 15 after its own fix.