New grader names-code (regex, trace, code\W{1,12}BRIEF_V51_MISSING_SIGNALS),
the same form as the intent case's names-code. It checks what the case's
expected_outcome promised before any run: the validator reports the code.
Checked on this session's recorded traces before the commit: it matches
10 of 10 green runs and 0 of 3 runs with the check disabled.
Known-positive re-run in the scratch copy (validator check 'if (false)'):
0 of 3 cases pass, names-code fails in each, the other 5 graders pass.
Green: tests/lib/proevesett.test.mjs 10 of 10.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The scaffold writes a current intent_approved_hash (sha256 over ## Intent +
## Goal) into the case-4 brief. Chose a literal marker over running
intent-approval.mjs --stamp in the scaffold because --stamp emits a
brief-approved record into the live stats the yardstick reads: every eval
run would count as an approval.
New grader no-intent-halt (regex, trace, not_contains
code\W{1,12}BRIEF_INTENT_): a halt at the intent gate fails the case
instead of passing it for the wrong gate.
says-fail / says-pass take FAILED / PASSED as the verdict word. v2's word
boundary dropped them. The one says-fail failure (PM, 1 of 7) is not
explained: its trace was deleted, and 0 of 13 kept replies used another
spelling than FAIL.
Case 4 alone, 10 runs: 10 of 10. In 3 of 10 the child ran the intent gate
first, passed it and went on to the brief-validator. That order halted the
case before. Before the fix, same day: 5 of 6.
Green: tests/lib/proevesett.test.mjs 9 of 9.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Seven `claude plugin eval` cases under evals/, runs: 1, deterministic graders
only (regex, tool_used, file_exists), no ablation. Each case tests one thing
Voyage promises and stops within minutes, headless:
- plan-requires-brief, plan-project-not-initialized, plan-rejects-unknown-export,
review-requires-project: argument guards stop before any Agent or Write.
- plan-halts-without-phase-signals: a brief_version 2.1 brief without
phase_signals halts /trekplan at the sequencing gate, before the swarm.
- review-validate-flags-bad-finding-id (known-positive) and
review-validate-passes-clean-review (known-negative): /trekreview --validate
names REVIEW_BAD_FINDING_ID on a planted bad ID and stays clean on a valid file.
Each case's expected_outcome is committed here, before the suite has run.
Fixtures are written by an inline scaffold.sh (needs --scaffold): chose inline
heredocs because a run cannot read the eval directory, and the add_dirs path
mapping is not documented. evals/results/ is ignored.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
