test(evals): Voyage's fixed test set as a plugin eval suite, expectations written before the first run

Seven `claude plugin eval` cases under evals/, runs: 1, deterministic graders
only (regex, tool_used, file_exists), no ablation. Each case tests one thing
Voyage promises and stops within minutes, headless:

- plan-requires-brief, plan-project-not-initialized, plan-rejects-unknown-export,
  review-requires-project: argument guards stop before any Agent or Write.
- plan-halts-without-phase-signals: a brief_version 2.1 brief without
  phase_signals halts /trekplan at the sequencing gate, before the swarm.
- review-validate-flags-bad-finding-id (known-positive) and
  review-validate-passes-clean-review (known-negative): /trekreview --validate
  names REVIEW_BAD_FINDING_ID on a planted bad ID and stays clean on a valid file.

Each case's expected_outcome is committed here, before the suite has run.
Fixtures are written by an inline scaffold.sh (needs --scaffold): chose inline
heredocs because a run cannot read the eval directory, and the add_dirs path
mapping is not documented. evals/results/ is ignored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

.gitignore

@@ -9,6 +9,10 @@ test-results/
 playwright-report/
 blob-report/
 
+# `claude plugin eval` run output (aggregate-result.json, report.html). Results are
+# data about one run on one machine; the suite (evals/<case>/) is what is tracked.
+/evals/results/
+
 # Editor files
 *.swp
 *.swo

evals/plan-halts-without-phase-signals/case.yaml

@@ -0,0 +1,4 @@
+schema_version: "1.1"
+name: plan-halts-without-phase-signals
+context:
+  scaffold_script: scaffold.sh

evals/plan-halts-without-phase-signals/graders/halt-names-signals.md

@@ -0,0 +1,4 @@
+---
+type: regex
+pattern: "phase_signals"
+---

evals/plan-halts-without-phase-signals/graders/no-agent.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Agent
+min: 0
+max: 0
+---

evals/plan-halts-without-phase-signals/graders/no-plan-file.md

@@ -0,0 +1,5 @@
+---
+type: file_exists
+path: "**/plan.md"
+exists: false
+---

evals/plan-halts-without-phase-signals/graders/validator-ran.md

@@ -0,0 +1,5 @@
+---
+type: tool_used
+tool: Bash
+input_match: "brief-validator"
+---

evals/plan-halts-without-phase-signals/prompt.md

@@ -0,0 +1,11 @@
+---
+description: "/trekplan on a brief_version 2.1 brief without phase_signals halts at the sequencing gate."
+expected_outcome: "brief-validator runs, reports BRIEF_V51_MISSING_SIGNALS, and /trekplan halts with the phase_signals hint; no Agent call (no exploration swarm), no plan.md."
+tags: [proevesett]
+runs: 1
+max_turns: 15
+timeout_seconds: 300
+allowed_tools: [Bash, Read, Glob, Grep, Write, Agent]
+---
+
+/voyage:trekplan --project proj

evals/plan-halts-without-phase-signals/scaffold.sh

@@ -0,0 +1,35 @@
+#!/bin/bash
+# Seeds the empty run workspace with a project folder whose brief declares
+# brief_version 2.1 but carries neither phase_signals nor phase_signals_partial.
+# Self-contained on purpose: the run cannot read the eval directory.
+set -eu
+mkdir -p proj
+cat > proj/brief.md <<'BRIEF'
+---
+type: trekbrief
+brief_version: "2.1"
+created: 2026-09-23
+task: "Eval fixture: add a greeting helper"
+slug: eval-greeting
+project_dir: proj/
+research_topics: 0
+research_status: skipped
+auto_research: false
+interview_turns: 1
+source: fixture
+---
+
+# Task: add a greeting helper
+
+## Intent
+
+Eval fixture for the Voyage proevesett. Not a real task.
+
+## Goal
+
+A function `greet(name)` returns `Hello, <name>!`.
+
+## Success Criteria
+
+- `greet("Ada")` returns `Hello, Ada!`.
+BRIEF

evals/plan-project-not-initialized/graders/no-agent.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Agent
+min: 0
+max: 0
+---

evals/plan-project-not-initialized/graders/no-plan-file.md

@@ -0,0 +1,5 @@
+---
+type: file_exists
+path: "**/plan.md"
+exists: false
+---

evals/plan-project-not-initialized/graders/not-initialized.md

@@ -0,0 +1,4 @@
+---
+type: regex
+pattern: "project directory not initialized"
+---

evals/plan-project-not-initialized/prompt.md

@@ -0,0 +1,11 @@
+---
+description: "/trekplan --project on a missing folder stops before any exploration."
+expected_outcome: "'Error: project directory not initialized. Run /trekbrief to create it.'; no Agent call, no plan.md."
+tags: [proevesett]
+runs: 1
+max_turns: 8
+timeout_seconds: 300
+allowed_tools: [Bash, Read, Glob, Grep, Write, Agent]
+---
+
+/voyage:trekplan --project missing-project

evals/plan-rejects-unknown-export/graders/format-rejected.md

@@ -0,0 +1,4 @@
+---
+type: regex
+pattern: "format 'pr' is not supported"
+---

evals/plan-rejects-unknown-export/graders/no-agent.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Agent
+min: 0
+max: 0
+---

evals/plan-rejects-unknown-export/graders/no-write.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Write
+min: 0
+max: 0
+---

evals/plan-rejects-unknown-export/prompt.md

@@ -0,0 +1,11 @@
+---
+description: "/trekplan --export with a removed format names the format and stops."
+expected_outcome: "'Error: export format 'pr' is not supported.'; no Agent call, no file written."
+tags: [proevesett]
+runs: 1
+max_turns: 8
+timeout_seconds: 300
+allowed_tools: [Bash, Read, Glob, Grep, Write, Agent]
+---
+
+/voyage:trekplan --export pr plan.md

evals/plan-requires-brief/graders/no-agent.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Agent
+min: 0
+max: 0
+---

evals/plan-requires-brief/graders/no-write.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Write
+min: 0
+max: 0
+---

evals/plan-requires-brief/graders/usage-stop.md

@@ -0,0 +1,4 @@
+---
+type: regex
+pattern: "A brief is required"
+---

evals/plan-requires-brief/prompt.md

@@ -0,0 +1,11 @@
+---
+description: "/trekplan with no arguments prints usage and stops."
+expected_outcome: "Usage block ending in 'A brief is required. Produce one with /trekbrief first.'; no Agent call, no file written."
+tags: [proevesett]
+runs: 1
+max_turns: 8
+timeout_seconds: 300
+allowed_tools: [Bash, Read, Glob, Grep, Write, Agent]
+---
+
+/voyage:trekplan

evals/review-requires-project/graders/no-agent.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Agent
+min: 0
+max: 0
+---

evals/review-requires-project/graders/no-write.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Write
+min: 0
+max: 0
+---

evals/review-requires-project/graders/project-required.md

@@ -0,0 +1,4 @@
+---
+type: regex
+pattern: "--project <dir> is required"
+---

evals/review-requires-project/prompt.md

@@ -0,0 +1,11 @@
+---
+description: "/trekreview with no arguments demands --project and stops."
+expected_outcome: "'Error: --project <dir> is required.' plus usage; no Agent call, no file written."
+tags: [proevesett]
+runs: 1
+max_turns: 8
+timeout_seconds: 300
+allowed_tools: [Bash, Read, Glob, Grep, Write, Agent]
+---
+
+/voyage:trekreview

evals/review-validate-flags-bad-finding-id/case.yaml

@@ -0,0 +1,4 @@
+schema_version: "1.1"
+name: review-validate-flags-bad-finding-id
+context:
+  scaffold_script: scaffold.sh

evals/review-validate-flags-bad-finding-id/graders/names-rule.md

@@ -0,0 +1,5 @@
+---
+type: regex
+pattern: "REVIEW_BAD_FINDING_ID"
+target: trace
+---

evals/review-validate-flags-bad-finding-id/graders/no-agent.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Agent
+min: 0
+max: 0
+---

evals/review-validate-flags-bad-finding-id/graders/no-write.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Write
+min: 0
+max: 0
+---

evals/review-validate-flags-bad-finding-id/graders/says-fail.md

@@ -0,0 +1,4 @@
+---
+type: regex
+pattern: "FAIL"
+---

evals/review-validate-flags-bad-finding-id/graders/validator-ran.md

@@ -0,0 +1,5 @@
+---
+type: tool_used
+tool: Bash
+input_match: "review-validator"
+---

evals/review-validate-flags-bad-finding-id/prompt.md

@@ -0,0 +1,11 @@
+---
+description: "KNOWN-POSITIVE. /trekreview --validate on a review.md whose finding ID is not 40-char hex fails and names the rule."
+expected_outcome: "review-validator runs and reports REVIEW_BAD_FINDING_ID; the one-line summary says FAIL; no Agent call, no file written."
+tags: [proevesett]
+runs: 1
+max_turns: 10
+timeout_seconds: 300
+allowed_tools: [Bash, Read, Glob, Grep, Write, Agent]
+---
+
+/voyage:trekreview --project proj --validate

evals/review-validate-flags-bad-finding-id/scaffold.sh

@@ -0,0 +1,79 @@
+#!/bin/bash
+# Seeds the empty run workspace with a project folder whose review.md carries a finding ID that is not 40-char hex.
+# Self-contained on purpose: the run cannot read the eval directory, and the
+# fixtures are written inline so no path outside the workspace is involved.
+set -eu
+mkdir -p proj
+cat > proj/brief.md <<'BRIEF'
+---
+type: trekbrief
+brief_version: "2.1"
+created: 2026-09-23
+task: "Eval fixture: add a greeting helper"
+slug: eval-greeting
+project_dir: proj/
+research_topics: 0
+research_status: skipped
+auto_research: false
+interview_turns: 1
+source: fixture
+phase_signals:
+  - phase: plan
+    effort: low
+  - phase: review
+    effort: low
+---
+
+# Task: add a greeting helper
+
+## Intent
+
+Eval fixture for the Voyage proevesett. Not a real task.
+
+## Goal
+
+A function `greet(name)` returns `Hello, <name>!`.
+
+## Success Criteria
+
+- `greet("Ada")` returns `Hello, Ada!`.
+BRIEF
+cat > proj/review.md <<'REVIEW'
+---
+type: trekreview
+review_version: "1.0"
+created: 2026-09-23
+task: "Eval fixture: add a greeting helper"
+slug: eval-greeting
+project_dir: proj/
+brief_path: proj/brief.md
+scope_sha_start: 0123456789abcdef0123456789abcdef01234567
+scope_sha_end: fedcba9876543210fedcba9876543210fedcba98
+reviewed_files_count: 1
+verdict: ALLOW
+findings:
+  - F-001
+---
+
+# Review: add a greeting helper
+
+## Executive Summary
+
+Eval fixture. One SUGGESTION, verdict ALLOW.
+
+## Coverage
+
+One file reviewed: `src/greet.js`.
+
+## Findings (SUGGESTION)
+
+- `src/greet.js:1` - name the parameter more descriptively.
+
+## Remediation Summary
+
+Nothing blocks. The suggestion is optional.
+
+```json
+{"findings":[{"id":"d2d0e27875ae9ef0d818cb08bb6f14e6d33c4232","severity":"SUGGESTION","file":"src/greet.js","line":1,"rule_key":"naming","title":"Parameter name","detail":"Eval fixture.","recommended_action":"Rename the parameter."}]}
+```
+REVIEW

evals/review-validate-passes-clean-review/case.yaml

@@ -0,0 +1,4 @@
+schema_version: "1.1"
+name: review-validate-passes-clean-review
+context:
+  scaffold_script: scaffold.sh

evals/review-validate-passes-clean-review/graders/no-agent.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Agent
+min: 0
+max: 0
+---

evals/review-validate-passes-clean-review/graders/no-error-code.md

@@ -0,0 +1,6 @@
+---
+type: regex
+pattern: "REVIEW_(MISSING_FIELD|BAD_FINDING_ID|BAD_FINDINGS_TYPE|MISSING_SECTION|NOT_FOUND|READ_ERROR)"
+target: trace
+match: not_contains
+---

evals/review-validate-passes-clean-review/graders/no-write.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Write
+min: 0
+max: 0
+---

evals/review-validate-passes-clean-review/graders/says-pass.md

@@ -0,0 +1,4 @@
+---
+type: regex
+pattern: "PASS"
+---

evals/review-validate-passes-clean-review/graders/validator-ran.md

@@ -0,0 +1,5 @@
+---
+type: tool_used
+tool: Bash
+input_match: "review-validator"
+---

evals/review-validate-passes-clean-review/prompt.md

@@ -0,0 +1,11 @@
+---
+description: "KNOWN-NEGATIVE. /trekreview --validate on a valid review.md passes and raises no error code."
+expected_outcome: "review-validator runs; the one-line summary says PASS; no REVIEW_* error code appears anywhere in the trace; no Agent call, no file written."
+tags: [proevesett]
+runs: 1
+max_turns: 10
+timeout_seconds: 300
+allowed_tools: [Bash, Read, Glob, Grep, Write, Agent]
+---
+
+/voyage:trekreview --project proj --validate

evals/review-validate-passes-clean-review/scaffold.sh

@@ -0,0 +1,79 @@
+#!/bin/bash
+# Seeds the empty run workspace with a project folder whose review.md is valid.
+# Self-contained on purpose: the run cannot read the eval directory, and the
+# fixtures are written inline so no path outside the workspace is involved.
+set -eu
+mkdir -p proj
+cat > proj/brief.md <<'BRIEF'
+---
+type: trekbrief
+brief_version: "2.1"
+created: 2026-09-23
+task: "Eval fixture: add a greeting helper"
+slug: eval-greeting
+project_dir: proj/
+research_topics: 0
+research_status: skipped
+auto_research: false
+interview_turns: 1
+source: fixture
+phase_signals:
+  - phase: plan
+    effort: low
+  - phase: review
+    effort: low
+---
+
+# Task: add a greeting helper
+
+## Intent
+
+Eval fixture for the Voyage proevesett. Not a real task.
+
+## Goal
+
+A function `greet(name)` returns `Hello, <name>!`.
+
+## Success Criteria
+
+- `greet("Ada")` returns `Hello, Ada!`.
+BRIEF
+cat > proj/review.md <<'REVIEW'
+---
+type: trekreview
+review_version: "1.0"
+created: 2026-09-23
+task: "Eval fixture: add a greeting helper"
+slug: eval-greeting
+project_dir: proj/
+brief_path: proj/brief.md
+scope_sha_start: 0123456789abcdef0123456789abcdef01234567
+scope_sha_end: fedcba9876543210fedcba9876543210fedcba98
+reviewed_files_count: 1
+verdict: ALLOW
+findings:
+  - d2d0e27875ae9ef0d818cb08bb6f14e6d33c4232
+---
+
+# Review: add a greeting helper
+
+## Executive Summary
+
+Eval fixture. One SUGGESTION, verdict ALLOW.
+
+## Coverage
+
+One file reviewed: `src/greet.js`.
+
+## Findings (SUGGESTION)
+
+- `src/greet.js:1` - name the parameter more descriptively.
+
+## Remediation Summary
+
+Nothing blocks. The suggestion is optional.
+
+```json
+{"findings":[{"id":"d2d0e27875ae9ef0d818cb08bb6f14e6d33c4232","severity":"SUGGESTION","file":"src/greet.js","line":1,"rule_key":"naming","title":"Parameter name","detail":"Eval fixture.","recommended_action":"Rename the parameter."}]}
+```
+REVIEW