open/voyage
1
0
Fork 0
Code Issues Releases 5 Activity

docs(proevesett): case 8 12 of 12 after the verdict graders read any letter case

Browse source 
New § Stability section: the PM's 8/8 and 7/8 on ea66a98, the cause (correct
lower-case verdicts, 3 of 34 case-8 and 2 of 22 case-6 recorded replies), the
grader changes and why, and the measured result on 7f7161b: case 8 12 of 12
(10 alone + 2 full), case 6 8 of 8 (6 + 2), two full runs 8 of 8 each. The
status block and "every full run gave 8 of 8" now name whose sample that was.

Correction to 687710a's message: it said a grader file carries only
`pattern`. The docs list `flags` for the regex grader. The letter classes
stay, for the reason now written in the doc: a flag applies to the whole
pattern, and the opposite-verdict refusal must stay case-sensitive.

Suite after git add: 1236 tests, 1234/0/2. No version bump: plugin code
is untouched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-23 11:41:13 +02:00
commit 0be152ad29
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
1 changed files with 54 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

57
docs/proevesett.md
View file

@ -7,8 +7,11 @@ Red is data here, not a defect in the suite. What v2 changed and why: § v2 belo
**v2's first 8 of 8 was one sample, not a stable state.** Measured the same day,
before the stability fix: case 4 passed 5 of 6 here, 6 of 12 and 7 of 10 in the PM
re-measurement. After the fix: every case **5 of 5** over three full runs, case 4
also **10 of 10** on its own. Numbers per case, with the denominator: § Stability.
re-measurement. After the fix, the builder's sample gave every case **5 of 5** over
three full runs. The PM re-measurement of that commit gave **8 of 8 and 7 of 8**:
case 8 failed on correct replies written in lower case. After the verdict-grader fix,
case 8 **12 of 12**, case 6 **8 of 8**, two full runs **8 of 8** each. Numbers per case,
with the denominator: § Stability.
The unit suite (`npm test`) proves Voyage's modules in isolation. This suite proves
something the unit suite cannot: that the **commands, as a model reads and follows
@ -171,10 +174,58 @@ all eight cases, the command under § How to run it. There were three: one with
In 4 of the 15 case-4 runs, the child ran the intent gate as its own call before the
brief-validator. In 1 more, it ran both gates in one call. All 5 passed the intent
gate, went on to the sequencing gate and passed the case. Before the fix, a run that
ran the intent gate alone first halted there. Every full run gave **8 of 8**.
ran the intent gate alone first halted there. Every full run in this sample gave
**8 of 8**; the PM's two did not (next section).
A pass rate of 15 of 15 does not prove the rate is 1: it bounds it. Five runs per
case is a small denominator. The claim is "no failure in N", not "cannot fail".
## Stability (2026-09-23): the verdict graders read any letter case
**What failed.** The PM re-measurement of `ea66a98` (the commit above) gave **8 of 8
and 7 of 8** in two full runs. Case 8 `review-validate-passes-clean-review` passed
**10 of 12** (9 of 10 alone, 1 of 2 in full runs). Both failures were correct replies:
the validator ran and exited 0, and the reply said "`proj/review.md` passed the strict
schema check". Only `says-pass` failed. Its pattern `\bPASS(?:ED)?\b` is
case-sensitive, and the reply had no `PASS`. The fault is older than § Stability's
fix: v2's `\bPASS\b` was stricter still.
**Measured, not assumed.** The children write the verdict in lower case, in both cases.
Over all recorded replies (the PM's and this change's own runs), the old patterns
would have failed **3 of 34** correct case-8 replies ("passed", "passes") and **2 of 22**
correct case-6 replies ("fails", "failed"). That also explains the class behind the
unexplained case-6 failure above. That one reply is still gone, so it is not confirmed.
| Change | Why |
|--------|-----|
| The verdict word in any letter case: `[Pp][Aa][Ss][Ss]`, with the forms pass/passes/passed and fail/fails/failed | Correct replies used each of these |
| A preceding `n't ` refuses, like `not ` already did | "doesn't pass" is the wrong verdict |
| The whole reply is refused if the **opposite** verdict stands in capitals (`FAIL`/`FAILED` for `says-pass`, `PASS`/`PASSED` for `says-fail`), unless negated | A correct case-6 reply said "**FAIL**: … did not pass … The other checks passed." Accepting lower case alone would let a wrong case-8 reply of that shape pass |
| The opposite word in lower case does **not** refuse | Correct replies carry it in side clauses ("did not pass", "other checks passed"). Refusing it would bring back the instability this change removes |
| Letter classes, not `flags: i` | The grader accepts `flags` (docs, § Grader types), but a flag applies to the whole pattern, and the refusal must stay case-sensitive. The pattern starts with `(?<![\s\S])`, so it anchors at the start of the reply with or without `m` |
On the recorded replies, the new patterns take **34 of 34** (case 8) and **22 of 22**
(case 6), and each grader matches **0** of the other case's replies. Pinned offline in
`tests/lib/proevesett.test.mjs`, red first on the commit before each change.
**Measured stability after the fix** (Claude Code 2.1.280, same command as § How to run it):
| Case | Full runs | Alone | Total |
|------|-----------|-------|-------|
| `review-validate-passes-clean-review` | 2 of 2 | 10 of 10 | **12 of 12** |
| `review-validate-flags-bad-finding-id` | 2 of 2 | 6 of 6 | **8 of 8** |
| the six other cases | 2 of 2 each | – | 2 of 2 each |
Both full runs gave **8 of 8**. In these 20 runs of cases 6 and 8, 1 reply gave its
verdict only in lower case ("failed", case 6, full run 2). Before the final fix, an
intermediate pattern (lower case only, no refusal) had case 8 10 of 10 and case 6
6 of 6 alone, with 2 lower-case-only replies. Those are not counted above. Eval cost
of this change: $10.85 (list-price estimate).
**What the numbers allow.** Case 8 has 12 of 12 on this commit. Across all its
recorded replies, 34 of 34 would pass the new grader. That bounds the grader fault.
It does not show that the case cannot fail. With `runs: 1`, step 4 still reads one
sample per case per run.
## Measured facts about the harness this suite relies on
These were measured by probe cases in a scratch copy, not taken from the docs.