open/voyage
1
0
Fork 0
Code Issues Releases 5 Activity

test(proevesett): red — no case for the intent gate, case 5 composed in prose, weak graders

Browse source 
Offline structure tests for evals/ (the suite itself needs headless runs):
- no case reaches the steg 1 intent gate (PM: 0 traces touched it). The
  new case's scaffold is in: its brief passes brief-validator --soft
  (2.1 WITH phase_signals) and --check gives BRIEF_INTENT_NOT_APPROVED —
  that test is green, so the fixture reaches the gate.
- review-requires-project: trekreview.md composes 'Error: --project <dir>
  is required.' in prose (8/10 in the PM run, backticks broke the regex),
  and its arg-parser line passes "$@", which the Bash tool never has.
- no-error-code misses REVIEW_WRONG_TYPE; PASS/FAIL graders are raw
  substrings; no-write cannot see a write through Bash.
7 tests, 6 red.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-23 09:26:31 +02:00
commit 348aa95639
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
3 changed files with 231 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

4
evals/plan-halts-without-intent-approval/case.yaml Normal file
View file

@ -0,0 +1,4 @@
schema_version: "1.1"
name: plan-halts-without-intent-approval
context:
scaffold_script: scaffold.sh