test(evals): Voyage's fixed test set as a plugin eval suite, expectations written before the first run
Seven `claude plugin eval` cases under evals/, runs: 1, deterministic graders only (regex, tool_used, file_exists), no ablation. Each case tests one thing Voyage promises and stops within minutes, headless: - plan-requires-brief, plan-project-not-initialized, plan-rejects-unknown-export, review-requires-project: argument guards stop before any Agent or Write. - plan-halts-without-phase-signals: a brief_version 2.1 brief without phase_signals halts /trekplan at the sequencing gate, before the swarm. - review-validate-flags-bad-finding-id (known-positive) and review-validate-passes-clean-review (known-negative): /trekreview --validate names REVIEW_BAD_FINDING_ID on a planted bad ID and stays clean on a valid file. Each case's expected_outcome is committed here, before the suite has run. Fixtures are written by an inline scaffold.sh (needs --scaffold): chose inline heredocs because a run cannot read the eval directory, and the add_dirs path mapping is not documented. evals/results/ is ignored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
1779411b49
commit
d311f3b49a
40 changed files with 422 additions and 0 deletions
4
evals/review-validate-flags-bad-finding-id/case.yaml
Normal file
4
evals/review-validate-flags-bad-finding-id/case.yaml
Normal file
|
|
@ -0,0 +1,4 @@
|
|||
schema_version: "1.1"
|
||||
name: review-validate-flags-bad-finding-id
|
||||
context:
|
||||
scaffold_script: scaffold.sh
|
||||
|
|
@ -0,0 +1,5 @@
|
|||
---
|
||||
type: regex
|
||||
pattern: "REVIEW_BAD_FINDING_ID"
|
||||
target: trace
|
||||
---
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
type: tool_used
|
||||
tool: Agent
|
||||
min: 0
|
||||
max: 0
|
||||
---
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
type: tool_used
|
||||
tool: Write
|
||||
min: 0
|
||||
max: 0
|
||||
---
|
||||
|
|
@ -0,0 +1,4 @@
|
|||
---
|
||||
type: regex
|
||||
pattern: "FAIL"
|
||||
---
|
||||
|
|
@ -0,0 +1,5 @@
|
|||
---
|
||||
type: tool_used
|
||||
tool: Bash
|
||||
input_match: "review-validator"
|
||||
---
|
||||
11
evals/review-validate-flags-bad-finding-id/prompt.md
Normal file
11
evals/review-validate-flags-bad-finding-id/prompt.md
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
---
|
||||
description: "KNOWN-POSITIVE. /trekreview --validate on a review.md whose finding ID is not 40-char hex fails and names the rule."
|
||||
expected_outcome: "review-validator runs and reports REVIEW_BAD_FINDING_ID; the one-line summary says FAIL; no Agent call, no file written."
|
||||
tags: [proevesett]
|
||||
runs: 1
|
||||
max_turns: 10
|
||||
timeout_seconds: 300
|
||||
allowed_tools: [Bash, Read, Glob, Grep, Write, Agent]
|
||||
---
|
||||
|
||||
/voyage:trekreview --project proj --validate
|
||||
79
evals/review-validate-flags-bad-finding-id/scaffold.sh
Executable file
79
evals/review-validate-flags-bad-finding-id/scaffold.sh
Executable file
|
|
@ -0,0 +1,79 @@
|
|||
#!/bin/bash
|
||||
# Seeds the empty run workspace with a project folder whose review.md carries a finding ID that is not 40-char hex.
|
||||
# Self-contained on purpose: the run cannot read the eval directory, and the
|
||||
# fixtures are written inline so no path outside the workspace is involved.
|
||||
set -eu
|
||||
mkdir -p proj
|
||||
cat > proj/brief.md <<'BRIEF'
|
||||
---
|
||||
type: trekbrief
|
||||
brief_version: "2.1"
|
||||
created: 2026-09-23
|
||||
task: "Eval fixture: add a greeting helper"
|
||||
slug: eval-greeting
|
||||
project_dir: proj/
|
||||
research_topics: 0
|
||||
research_status: skipped
|
||||
auto_research: false
|
||||
interview_turns: 1
|
||||
source: fixture
|
||||
phase_signals:
|
||||
- phase: plan
|
||||
effort: low
|
||||
- phase: review
|
||||
effort: low
|
||||
---
|
||||
|
||||
# Task: add a greeting helper
|
||||
|
||||
## Intent
|
||||
|
||||
Eval fixture for the Voyage proevesett. Not a real task.
|
||||
|
||||
## Goal
|
||||
|
||||
A function `greet(name)` returns `Hello, <name>!`.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- `greet("Ada")` returns `Hello, Ada!`.
|
||||
BRIEF
|
||||
cat > proj/review.md <<'REVIEW'
|
||||
---
|
||||
type: trekreview
|
||||
review_version: "1.0"
|
||||
created: 2026-09-23
|
||||
task: "Eval fixture: add a greeting helper"
|
||||
slug: eval-greeting
|
||||
project_dir: proj/
|
||||
brief_path: proj/brief.md
|
||||
scope_sha_start: 0123456789abcdef0123456789abcdef01234567
|
||||
scope_sha_end: fedcba9876543210fedcba9876543210fedcba98
|
||||
reviewed_files_count: 1
|
||||
verdict: ALLOW
|
||||
findings:
|
||||
- F-001
|
||||
---
|
||||
|
||||
# Review: add a greeting helper
|
||||
|
||||
## Executive Summary
|
||||
|
||||
Eval fixture. One SUGGESTION, verdict ALLOW.
|
||||
|
||||
## Coverage
|
||||
|
||||
One file reviewed: `src/greet.js`.
|
||||
|
||||
## Findings (SUGGESTION)
|
||||
|
||||
- `src/greet.js:1` - name the parameter more descriptively.
|
||||
|
||||
## Remediation Summary
|
||||
|
||||
Nothing blocks. The suggestion is optional.
|
||||
|
||||
```json
|
||||
{"findings":[{"id":"d2d0e27875ae9ef0d818cb08bb6f14e6d33c4232","severity":"SUGGESTION","file":"src/greet.js","line":1,"rule_key":"naming","title":"Parameter name","detail":"Eval fixture.","recommended_action":"Rename the parameter."}]}
|
||||
```
|
||||
REVIEW
|
||||
Loading…
Add table
Add a link
Reference in a new issue