test(evals): Voyage's fixed test set as a plugin eval suite, expectations written before the first run
Seven `claude plugin eval` cases under evals/, runs: 1, deterministic graders only (regex, tool_used, file_exists), no ablation. Each case tests one thing Voyage promises and stops within minutes, headless: - plan-requires-brief, plan-project-not-initialized, plan-rejects-unknown-export, review-requires-project: argument guards stop before any Agent or Write. - plan-halts-without-phase-signals: a brief_version 2.1 brief without phase_signals halts /trekplan at the sequencing gate, before the swarm. - review-validate-flags-bad-finding-id (known-positive) and review-validate-passes-clean-review (known-negative): /trekreview --validate names REVIEW_BAD_FINDING_ID on a planted bad ID and stays clean on a valid file. Each case's expected_outcome is committed here, before the suite has run. Fixtures are written by an inline scaffold.sh (needs --scaffold): chose inline heredocs because a run cannot read the eval directory, and the add_dirs path mapping is not documented. evals/results/ is ignored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
1779411b49
commit
d311f3b49a
40 changed files with 422 additions and 0 deletions
6
evals/plan-project-not-initialized/graders/no-agent.md
Normal file
6
evals/plan-project-not-initialized/graders/no-agent.md
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
type: tool_used
|
||||
tool: Agent
|
||||
min: 0
|
||||
max: 0
|
||||
---
|
||||
|
|
@ -0,0 +1,5 @@
|
|||
---
|
||||
type: file_exists
|
||||
path: "**/plan.md"
|
||||
exists: false
|
||||
---
|
||||
|
|
@ -0,0 +1,4 @@
|
|||
---
|
||||
type: regex
|
||||
pattern: "project directory not initialized"
|
||||
---
|
||||
11
evals/plan-project-not-initialized/prompt.md
Normal file
11
evals/plan-project-not-initialized/prompt.md
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
---
|
||||
description: "/trekplan --project on a missing folder stops before any exploration."
|
||||
expected_outcome: "'Error: project directory not initialized. Run /trekbrief to create it.'; no Agent call, no plan.md."
|
||||
tags: [proevesett]
|
||||
runs: 1
|
||||
max_turns: 8
|
||||
timeout_seconds: 300
|
||||
allowed_tools: [Bash, Read, Glob, Grep, Write, Agent]
|
||||
---
|
||||
|
||||
/voyage:trekplan --project missing-project
|
||||
Loading…
Add table
Add a link
Reference in a new issue