test(evals): Voyage's fixed test set as a plugin eval suite, expectations written before the first run

Seven `claude plugin eval` cases under evals/, runs: 1, deterministic graders
only (regex, tool_used, file_exists), no ablation. Each case tests one thing
Voyage promises and stops within minutes, headless:

- plan-requires-brief, plan-project-not-initialized, plan-rejects-unknown-export,
  review-requires-project: argument guards stop before any Agent or Write.
- plan-halts-without-phase-signals: a brief_version 2.1 brief without
  phase_signals halts /trekplan at the sequencing gate, before the swarm.
- review-validate-flags-bad-finding-id (known-positive) and
  review-validate-passes-clean-review (known-negative): /trekreview --validate
  names REVIEW_BAD_FINDING_ID on a planted bad ID and stays clean on a valid file.

Each case's expected_outcome is committed here, before the suite has run.
Fixtures are written by an inline scaffold.sh (needs --scaffold): chose inline
heredocs because a run cannot read the eval directory, and the add_dirs path
mapping is not documented. evals/results/ is ignored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-23 07:52:34 +02:00
commit d311f3b49a
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
40 changed files with 422 additions and 0 deletions

View file

@ -0,0 +1,4 @@
schema_version: "1.1"
name: plan-halts-without-phase-signals
context:
scaffold_script: scaffold.sh

View file

@ -0,0 +1,4 @@
---
type: regex
pattern: "phase_signals"
---

View file

@ -0,0 +1,6 @@
---
type: tool_used
tool: Agent
min: 0
max: 0
---

View file

@ -0,0 +1,5 @@
---
type: file_exists
path: "**/plan.md"
exists: false
---

View file

@ -0,0 +1,5 @@
---
type: tool_used
tool: Bash
input_match: "brief-validator"
---

View file

@ -0,0 +1,11 @@
---
description: "/trekplan on a brief_version 2.1 brief without phase_signals halts at the sequencing gate."
expected_outcome: "brief-validator runs, reports BRIEF_V51_MISSING_SIGNALS, and /trekplan halts with the phase_signals hint; no Agent call (no exploration swarm), no plan.md."
tags: [proevesett]
runs: 1
max_turns: 15
timeout_seconds: 300
allowed_tools: [Bash, Read, Glob, Grep, Write, Agent]
---
/voyage:trekplan --project proj

View file

@ -0,0 +1,35 @@
#!/bin/bash
# Seeds the empty run workspace with a project folder whose brief declares
# brief_version 2.1 but carries neither phase_signals nor phase_signals_partial.
# Self-contained on purpose: the run cannot read the eval directory.
set -eu
mkdir -p proj
cat > proj/brief.md <<'BRIEF'
---
type: trekbrief
brief_version: "2.1"
created: 2026-09-23
task: "Eval fixture: add a greeting helper"
slug: eval-greeting
project_dir: proj/
research_topics: 0
research_status: skipped
auto_research: false
interview_turns: 1
source: fixture
---
# Task: add a greeting helper
## Intent
Eval fixture for the Voyage proevesett. Not a real task.
## Goal
A function `greet(name)` returns `Hello, <name>!`.
## Success Criteria
- `greet("Ada")` returns `Hello, Ada!`.
BRIEF