test(proevesett): red — no case for the intent gate, case 5 composed in prose, weak graders
Offline structure tests for evals/ (the suite itself needs headless runs): - no case reaches the steg 1 intent gate (PM: 0 traces touched it). The new case's scaffold is in: its brief passes brief-validator --soft (2.1 WITH phase_signals) and --check gives BRIEF_INTENT_NOT_APPROVED — that test is green, so the fixture reaches the gate. - review-requires-project: trekreview.md composes 'Error: --project <dir> is required.' in prose (8/10 in the PM run, backticks broke the regex), and its arg-parser line passes "$@", which the Bash tool never has. - no-error-code misses REVIEW_WRONG_TYPE; PASS/FAIL graders are raw substrings; no-write cannot see a write through Bash. 7 tests, 6 red. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
4592b40873
commit
348aa95639
3 changed files with 231 additions and 0 deletions
45
evals/plan-halts-without-intent-approval/scaffold.sh
Executable file
45
evals/plan-halts-without-intent-approval/scaffold.sh
Executable file
|
|
@ -0,0 +1,45 @@
|
|||
#!/bin/bash
|
||||
# Seeds the empty run workspace with a project folder whose brief passes the
|
||||
# sequencing gate (brief_version 2.1 WITH phase_signals) but carries no
|
||||
# intent_approved_hash, so /trekplan must stop at the intent-approval gate.
|
||||
# Self-contained on purpose: the run cannot read the eval directory.
|
||||
set -eu
|
||||
mkdir -p proj
|
||||
cat > proj/brief.md <<'BRIEF'
|
||||
---
|
||||
type: trekbrief
|
||||
brief_version: "2.1"
|
||||
created: 2026-09-23
|
||||
task: "Eval fixture: add a greeting helper"
|
||||
slug: eval-greeting-intent
|
||||
project_dir: proj/
|
||||
research_topics: 0
|
||||
research_status: skipped
|
||||
auto_research: false
|
||||
interview_turns: 1
|
||||
source: fixture
|
||||
phase_signals:
|
||||
- phase: research
|
||||
effort: low
|
||||
- phase: plan
|
||||
effort: standard
|
||||
- phase: execute
|
||||
effort: standard
|
||||
- phase: review
|
||||
effort: standard
|
||||
---
|
||||
|
||||
# Task: add a greeting helper
|
||||
|
||||
## Intent
|
||||
|
||||
Eval fixture for the Voyage proevesett. Not a real task.
|
||||
|
||||
## Goal
|
||||
|
||||
A function `greet(name)` returns `Hello, <name>!`.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- `greet("Ada")` returns `Hello, Ada!`.
|
||||
BRIEF
|
||||
Loading…
Add table
Add a link
Reference in a new issue