Kjell Tore Guttormsen
d311f3b49a
Seven `claude plugin eval` cases under evals/, runs: 1, deterministic graders only (regex, tool_used, file_exists), no ablation. Each case tests one thing Voyage promises and stops within minutes, headless: - plan-requires-brief, plan-project-not-initialized, plan-rejects-unknown-export, review-requires-project: argument guards stop before any Agent or Write. - plan-halts-without-phase-signals: a brief_version 2.1 brief without phase_signals halts /trekplan at the sequencing gate, before the swarm. - review-validate-flags-bad-finding-id (known-positive) and review-validate-passes-clean-review (known-negative): /trekreview --validate names REVIEW_BAD_FINDING_ID on a planted bad ID and stays clean on a valid file. Each case's expected_outcome is committed here, before the suite has run. Fixtures are written by an inline scaffold.sh (needs --scaffold): chose inline heredocs because a run cannot read the eval directory, and the add_dirs path mapping is not documented. evals/results/ is ignored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
35 lines
766 B
Bash
Executable file
35 lines
766 B
Bash
Executable file
#!/bin/bash
|
|
# Seeds the empty run workspace with a project folder whose brief declares
|
|
# brief_version 2.1 but carries neither phase_signals nor phase_signals_partial.
|
|
# Self-contained on purpose: the run cannot read the eval directory.
|
|
set -eu
|
|
mkdir -p proj
|
|
cat > proj/brief.md <<'BRIEF'
|
|
---
|
|
type: trekbrief
|
|
brief_version: "2.1"
|
|
created: 2026-09-23
|
|
task: "Eval fixture: add a greeting helper"
|
|
slug: eval-greeting
|
|
project_dir: proj/
|
|
research_topics: 0
|
|
research_status: skipped
|
|
auto_research: false
|
|
interview_turns: 1
|
|
source: fixture
|
|
---
|
|
|
|
# Task: add a greeting helper
|
|
|
|
## Intent
|
|
|
|
Eval fixture for the Voyage proevesett. Not a real task.
|
|
|
|
## Goal
|
|
|
|
A function `greet(name)` returns `Hello, <name>!`.
|
|
|
|
## Success Criteria
|
|
|
|
- `greet("Ada")` returns `Hello, Ada!`.
|
|
BRIEF
|