test(proevesett): red — no case for the intent gate, case 5 composed in prose, weak graders

Offline structure tests for evals/ (the suite itself needs headless runs):
- no case reaches the steg 1 intent gate (PM: 0 traces touched it). The
  new case's scaffold is in: its brief passes brief-validator --soft
  (2.1 WITH phase_signals) and --check gives BRIEF_INTENT_NOT_APPROVED —
  that test is green, so the fixture reaches the gate.
- review-requires-project: trekreview.md composes 'Error: --project <dir>
  is required.' in prose (8/10 in the PM run, backticks broke the regex),
  and its arg-parser line passes "$@", which the Bash tool never has.
- no-error-code misses REVIEW_WRONG_TYPE; PASS/FAIL graders are raw
  substrings; no-write cannot see a write through Bash.
7 tests, 6 red.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-23 09:26:31 +02:00
commit 348aa95639
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
3 changed files with 231 additions and 0 deletions

View file

@ -0,0 +1,4 @@
schema_version: "1.1"
name: plan-halts-without-intent-approval
context:
scaffold_script: scaffold.sh

View file

@ -0,0 +1,45 @@
#!/bin/bash
# Seeds the empty run workspace with a project folder whose brief passes the
# sequencing gate (brief_version 2.1 WITH phase_signals) but carries no
# intent_approved_hash, so /trekplan must stop at the intent-approval gate.
# Self-contained on purpose: the run cannot read the eval directory.
set -eu
mkdir -p proj
cat > proj/brief.md <<'BRIEF'
---
type: trekbrief
brief_version: "2.1"
created: 2026-09-23
task: "Eval fixture: add a greeting helper"
slug: eval-greeting-intent
project_dir: proj/
research_topics: 0
research_status: skipped
auto_research: false
interview_turns: 1
source: fixture
phase_signals:
- phase: research
effort: low
- phase: plan
effort: standard
- phase: execute
effort: standard
- phase: review
effort: standard
---
# Task: add a greeting helper
## Intent
Eval fixture for the Voyage proevesett. Not a real task.
## Goal
A function `greet(name)` returns `Hello, <name>!`.
## Success Criteria
- `greet("Ada")` returns `Hello, Ada!`.
BRIEF