Measured on 687710a: a correct case-6 reply said "**FAIL**: ... did not pass
... The other checks passed." With lower case accepted, says-pass matches that
reply, so a wrong case-8 reply of the same shape would pass the case. The test
pins that a reply whose capitalised verdict is the opposite one is refused,
while the opposite word in lower case (a side clause) or negated is not.
Red on 687710a: 11 of 12 in tests/lib/proevesett.test.mjs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The PM re-measurement of ea66a98 had case 8 pass 10 of 12. Both failures were
correct replies that gave the verdict only in lower case ("passed"), and
\bPASS(?:ED)?\b is case-sensitive. The test pins lower-case pass/passes/passed
and fail/fails/failed as verdicts, keeps "not passed", "doesn't pass", "failed"
refused by says-pass, and checks that each case's recorded reply does not
satisfy the other case's grader.
Red on ea66a98: 10 of 11 in tests/lib/proevesett.test.mjs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Known-positive for case 4, in a scratch copy (never this tree):
brief-validator.mjs with the BRIEF_V51_MISSING_SIGNALS check made
'if (false)'. The case still passed 3 of 3: the child read the rule in
the command prose, halted, named phase_signals, and even found the
planted 'if (false)'. So the halt alone does not show the validator
fired, and no grader checked what expected_outcome already promised
('brief-validator ... reports BRIEF_V51_MISSING_SIGNALS').
New test: case 4 needs a names-code grader reading the validator's JSON
code from the trace (same form as the intent case). Red: 1 of 10.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The scaffold writes a current intent_approved_hash (sha256 over ## Intent +
## Goal) into the case-4 brief. Chose a literal marker over running
intent-approval.mjs --stamp in the scaffold because --stamp emits a
brief-approved record into the live stats the yardstick reads: every eval
run would count as an approval.
New grader no-intent-halt (regex, trace, not_contains
code\W{1,12}BRIEF_INTENT_): a halt at the intent gate fails the case
instead of passing it for the wrong gate.
says-fail / says-pass take FAILED / PASSED as the verdict word. v2's word
boundary dropped them. The one says-fail failure (PM, 1 of 7) is not
explained: its trace was deleted, and 0 of 13 kept replies used another
spelling than FAIL.
Case 4 alone, 10 runs: 10 of 10. In 3 of 10 the child ran the intent gate
first, passed it and went on to the brief-validator. That order halted the
case before. Before the fix, same day: 5 of 6.
Green: tests/lib/proevesett.test.mjs 9 of 9.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Case 4 (plan-halts-without-phase-signals) measured on dc9b480, 6 runs:
5 of 6. The failing run called only intent-approval.mjs, halted on
BRIEF_INTENT_NOT_APPROVED, and so failed validator-ran and
halt-names-signals: the fixture has no intent marker, so the case can
measure the wrong gate. PM: 6 of 12 on dc9b480, 7 of 10 on 66e1fa1.
New tests: the case-4 fixture must pass intent-approval --check and fail
only BRIEF_V51_MISSING_SIGNALS; a not_contains grader must fail the case
on any BRIEF_INTENT_* code in the trace. says-fail / says-pass must take
FAILED / PASSED as the verdict word (v2's word boundary dropped them).
Red: 3 of 9 in tests/lib/proevesett.test.mjs fail, each for its own reason.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Offline structure tests for evals/ (the suite itself needs headless runs):
- no case reaches the steg 1 intent gate (PM: 0 traces touched it). The
new case's scaffold is in: its brief passes brief-validator --soft
(2.1 WITH phase_signals) and --check gives BRIEF_INTENT_NOT_APPROVED —
that test is green, so the fixture reaches the gate.
- review-requires-project: trekreview.md composes 'Error: --project <dir>
is required.' in prose (8/10 in the PM run, backticks broke the regex),
and its arg-parser line passes "$@", which the Bash tool never has.
- no-error-code misses REVIEW_WRONG_TYPE; PASS/FAIL graders are raw
substrings; no-write cannot see a write through Bash.
7 tests, 6 red.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
