voyage/docs/proevesett.md

Prøvesett — Voyage's fixed test set as a plugin eval suite

Status: v1, seven cases, first run 6 of 7 (2026-09-23, Claude Code 2.1.280). The cases and their expectations were committed before the first run; the run did not change them. Red is data here, not a defect in the suite.

The unit suite (npm test) proves Voyage's modules in isolation. This suite proves something the unit suite cannot: that the commands, as a model reads and follows them in a real headless session, keep the promises they make. Each case is a real claude -p child with only Voyage loaded.

What "pass" and "fail" mean (tool definition, verbatim)

Source: Claude Code docs, Test plugins with evals (code.claude.com/docs/en/plugin-evals), § How a case is scored and § Grader types.

A run's score is the fraction of its graders that passed, weighted if you set weights, and the case's score is the mean across its runs. A case passes when its score meets the --threshold, 1.0 by default.

The four graders this suite uses, and when each passes:

Grader	Passes when (verbatim)
regex	"The JavaScript regex pattern is found in the target. Set match: not_contains to require absence"
tool_used	"The number of calls to tool whose JSON-encoded input matches the optional input_match regex is between min, default 1, and max, default unlimited. To assert a tool was never called, set both min: 0 and max: 0"
file_exists	"A file Claude created matches the path glob, or none does with exists: false. Only files created during the run count"

regex looks at last_message ("Claude's final response text. This is the default") unless the case sets target: trace ("The session as JSON, one message per line").

With runs: 1 and the default threshold, a case passes only when every grader passes, and fails when any single one does not. n of N below counts cases, not graders.

The cases

Every case has runs: 1, deterministic graders only (no llm, no baseline — they cost judge calls), and runs without the no-plugin baseline arm. Each tests one thing Voyage promises and ends in seconds. Nothing starts the /trekplan exploration swarm or the /trekreview reviewer agents; the no-agent grader in every case asserts it.

Case	Voyage promise under test	Expected (committed before the run)	Run 1
plan-requires-brief	/trekplan without a brief prints usage and stops	"A brief is required"; no Agent, no Write	pass
plan-project-not-initialized	/trekplan --project on a missing folder stops before exploring	"project directory not initialized"; no Agent, no plan.md	pass
plan-rejects-unknown-export	a removed export format is named and refused	"format 'pr' is not supported"; no Agent, no Write	pass
plan-halts-without-phase-signals	a brief_version 2.1 brief without phase_signals halts at the sequencing gate	brief-validator runs, reply names phase_signals; no Agent, no plan.md	pass
review-requires-project	/trekreview without --project refuses	"--project is required"; no Agent, no Write	fail (2 of 3 graders)
review-validate-flags-bad-finding-id	known-positive: --validate fails a review whose finding ID is not 40-char hex, and names the rule	review-validator runs; REVIEW_BAD_FINDING_ID in the trace; reply says FAIL; no Agent, no Write	pass
review-validate-passes-clean-review	known-negative: --validate passes a valid review and raises nothing	review-validator runs; reply says PASS; no REVIEW_* error code in the trace; no Agent, no Write	pass

The one red case

review-requires-project failed on its text grader only: the child replied in one turn without reproducing the exact error line Error: --project <dir> is required. The Agent and Write guards held. A diagnostic re-run of the same case alone passed, with the exact line. So the case is unstable, not broken. It is still red on purpose. This guard lives only in the command's prose, and a model does not always repeat prose word for word. Loosening the pattern after seeing the result would fit the grader to the run. If this should hold every time, the fix belongs in Voyage: let code emit the message, as the validators already emit their rule codes (REVIEW_BAD_FINDING_ID is printed by review-validator.mjs, not composed by the model — that is why names-rule greps the trace, not the reply).

Known-positive, demonstrated

Proof that the known-positive case can fail: in a scratch copy of Voyage (never this tree), the finding-ID check in lib/validators/review-validator.mjs was disabled. The validator then accepted the planted F-001 ID, and review-validate-flags-bad-finding-id dropped from 1.0 to 0.6. names-rule and says-fail failed; validator-ran, no-agent and no-write still passed. The case therefore falls when this part of Voyage breaks.

First run, recorded

Result	6 of 7 cases passed (graders 25 of 26; overall score 0.952 = mean of the case scores)
Children	7 claude -p runs (7 cases × 1 run × 1 arm)
Wall time	54 s (the tool reports durationSeconds: 53)
Cost	$1.67, the tool's list-price estimate; on a subscription it counts against usage limits instead
Output	evals/results/<timestamp>/aggregate-result.json + report.html (gitignored)

Measured facts about the harness this suite relies on

These were measured by probe cases in a scratch copy, not taken from the docs.

• Commands are invocable in the child as /voyage:<command>. The command body is expanded into the prompt; no Skill tool call is involved.
• ${CLAUDE_PLUGIN_ROOT} is substituted in the expanded command text, so the node ${CLAUDE_PLUGIN_ROOT}/lib/... lines in the commands reach the real validators. In the child's Bash environment the variable is empty. A case prompt must never rely on it directly.
• The plugin directory is readable from the child's OS sandbox, so Voyage's validators run for real under --allow-tools Bash.
• Fixtures come from an inline scaffold.sh that writes them with heredocs. A run starts in an empty workspace and "can't read the eval directory". The docs do not say how an add_dirs path maps into the run, so the suite does not depend on it.

What the suite does NOT cover

• The agent swarms. It never tests the /trekplan exploration, synthesis or review phases, the /trekreview reviewers and coordinator, or /trekresearch. Every case asserts zero Agent calls. Whether the swarm plans or reviews well is outside this suite. The offline gold-scored eval (tests/lib/gold-eval.test.mjs, see docs/eval-corpus/README.md) covers the coordinator contract on recorded payloads.
• /trekbrief, because its interview needs AskUserQuestion, which a headless child does not have. /trekexecute, /trekcontinue and /trekendsession have no case yet.
• Plugin contribution. The suite runs without the no-plugin baseline (--ablation none), so it does not measure how much Voyage adds over plain Claude. It measures only whether Voyage keeps its own promises.
• Variance. With runs: 1, a single run can pass or fail by chance; the red case above shows exactly that. One result is a sample, not a rate.
• Semantic quality of any output. Every grader is a pattern, a tool count or a file glob.

How to run it

From the plugin root:

claude plugin eval . --trust-plugin --ablation none --runs 1 --scaffold --no-publish --allow-tools Bash Write

• --scaffold runs each case's scaffold.sh as you, outside the sandbox. The three scaffolds in this suite only write fixture files into the run's workspace.
• --allow-tools Bash Write grants the tools the commands need (Bash runs under the OS sandbox). Write is granted so that the no-write and no-plan-file graders measure a refusal, not a missing permission. Agent is listed in every case's allowed_tools for the same reason.
• The exit code is 1 whenever any case scores below 1.0.

How veikart step 4 ("Trygg å endre") uses it

The suite is the fixed yardstick that a rule change is measured against:

1. Run the suite on the base commit and record n of N (aggregates.casesPassed / aggregates.casesTotal in aggregate-result.json).
2. Run it again with the change applied.
3. If n is smaller, the change stops. A lower n is not argued away. It is cleared only by showing that the same case is also red on the base commit, measured the same day. Otherwise the change broke a promise.
4. A new promise gets a new case, and its expectation is committed before its first run, the same as here. N grows; it never shrinks to make n look better.

Because runs: 1 is a sample, an unstable case such as review-requires-project will sometimes stop a change that is innocent. That is the intended price. Moving the guard into code removes the instability; editing the grader does not.