voyage/docs/proevesett.md

# Prøvesett — Voyage's fixed test set as a plugin eval suite

**Status:** v2, eight cases. v1 (seven cases) first run **6 of 7**; v2 first run
**8 of 8** (both 2026-09-23, Claude Code 2.1.280). Each version's cases and
expectations were committed *before* its first run; the run did not change them.
Red is data here, not a defect in the suite. What v2 changed and why: § v2 below.

The unit suite (`npm test`) proves Voyage's modules in isolation. This suite proves
something the unit suite cannot: that the **commands, as a model reads and follows
them in a real headless session**, keep the promises they make. Each case is a real
`claude -p` child with only Voyage loaded.

## What "pass" and "fail" mean (tool definition, verbatim)

Source: Claude Code docs, *Test plugins with evals* (`code.claude.com/docs/en/plugin-evals`),
§ How a case is scored and § Grader types.

> A run's score is the fraction of its graders that passed, weighted if you set weights,
> and the case's score is the mean across its runs. A case passes when its score meets the
> `--threshold`, `1.0` by default.

The four graders this suite uses, and when each passes:

| Grader | Passes when (verbatim) |
|--------|------------------------|
| `regex` | "The JavaScript regex `pattern` is found in the target. Set `match: not_contains` to require absence" |
| `tool_used` | "The number of calls to `tool` whose JSON-encoded input matches the optional `input_match` regex is between `min`, default 1, and `max`, default unlimited. To assert a tool was never called, set both `min: 0` and `max: 0`" |
| `file_exists` | "A file Claude created matches the `path` glob, or none does with `exists: false`. Only files created during the run count" |

`regex` looks at `last_message` ("Claude's final response text. This is the default")
unless the case sets `target: trace` ("The session as JSON, one message per line").

With `runs: 1` and the default threshold, a case **passes** only when **every** grader
passes, and **fails** when any single one does not. `n of N` below counts cases, not graders.

## The cases

Every case has `runs: 1`, deterministic graders only (no `llm`, no `baseline` — they cost
judge calls), and runs without the no-plugin baseline arm. Each tests **one** thing Voyage
promises and ends in seconds. Nothing starts the `/trekplan` exploration swarm or the
`/trekreview` reviewer agents; the `no-agent` grader in every case asserts it.

| Case | Voyage promise under test | Expected (committed before the run) | Run 1 |
|------|---------------------------|--------------------------------------|-------|
| `plan-requires-brief` | `/trekplan` without a brief prints usage and stops | "A brief is required"; no Agent, no Write | pass |
| `plan-project-not-initialized` | `/trekplan --project` on a missing folder stops before exploring | "project directory not initialized"; no Agent, no `plan.md` | pass |
| `plan-rejects-unknown-export` | a removed export format is named and refused | "format 'pr' is not supported"; no Agent, no Write | pass |
| `plan-halts-without-phase-signals` | a `brief_version` 2.1 brief without `phase_signals` halts at the sequencing gate | brief-validator runs, reply names `phase_signals`; no Agent, no `plan.md` | pass |
| `review-requires-project` | `/trekreview` without `--project` refuses | "--project <dir> is required"; no Agent, no Write | **fail** (2 of 3 graders) |
| `review-validate-flags-bad-finding-id` | **known-positive:** `--validate` fails a review whose finding ID is not 40-char hex, and names the rule | review-validator runs; `REVIEW_BAD_FINDING_ID` in the trace; reply says FAIL; no Agent, no Write | pass |
| `review-validate-passes-clean-review` | **known-negative:** `--validate` passes a valid review and raises nothing | review-validator runs; reply says PASS; no `REVIEW_*` error code in the trace; no Agent, no Write | pass |

The v1 table above is kept as it was run. v2 rows and grader changes: § v2.

### The one red case (v1; fixed in v2, see § v2)

`review-requires-project` failed on its text grader only: the child replied in one turn
without reproducing the exact error line `Error: --project <dir> is required.` The
Agent and Write guards held. A diagnostic re-run of the same case alone passed, with the
exact line. So the case is **unstable, not broken**. It is still red on purpose. This
guard lives only in the command's prose, and a model does not always repeat prose word
for word. Loosening the pattern after seeing the result would fit the grader to the run.
If this should hold every time, the fix belongs in Voyage: let code emit the message, as
the validators already emit their rule codes (`REVIEW_BAD_FINDING_ID` is printed by
`review-validator.mjs`, not composed by the model — that is why `names-rule` greps the
trace, not the reply).

### Known-positive, demonstrated

Proof that the known-positive case can fail: in a scratch copy of Voyage (never this
tree), the finding-ID check in `lib/validators/review-validator.mjs` was disabled. The
validator then accepted the planted `F-001` ID, and
`review-validate-flags-bad-finding-id` dropped from **1.0 to 0.6**. `names-rule` and
`says-fail` failed; `validator-ran`, `no-agent` and `no-write` still passed. The case
therefore falls when this part of Voyage breaks.

## First run, recorded

| | |
|---|---|
| Result | **6 of 7** cases passed (graders 25 of 26; overall score 0.952 = mean of the case scores) |
| Children | 7 `claude -p` runs (7 cases × 1 run × 1 arm) |
| Wall time | 54 s (the tool reports `durationSeconds: 53`) |
| Cost | $1.67, the tool's list-price estimate; on a subscription it counts against usage limits instead |
| Output | `evals/results/<timestamp>/aggregate-result.json` + `report.html` (gitignored) |

## v2 (2026-09-23): the intent gate gets a case, case 5's error moves into code

Expectations committed in `3892a83` before the v2 run. Offline structure tests in
`tests/lib/proevesett.test.mjs` pin what each grader can see.

| Change | Why |
|--------|-----|
| **New case `plan-halts-without-intent-approval`**: a `brief_version` 2.1 brief *with* `phase_signals` and no `intent_approved_hash`. Graders: `gate-ran` (`intent-approval.mjs --check` in a Bash call), `names-code` (the gate's own JSON `code` for `BRIEF_INTENT_NOT_APPROVED` in the trace), `no-agent`, `no-plan-file` | Veikart steg 1's gate had no case. No v1 trace reached it: case 4 halts at the sequencing gate first |
| **`review-requires-project`**: the arg parser got a CLI that checks `--project` itself and prints `Error: --project <dir> is required.` + usage (exit 1, code `ARG_REQUIRED_MISSING`). `trekreview.md` runs it with `$ARGUMENTS` and relays its stderr. The reply-regex grader `project-required` is replaced by `parser-ran` + `names-missing-project` (the parser's code in the trace, the same form as `names-rule`) | The fix this doc recommended for the red case. Also: the prose passed `"$@"`, which the Bash tool never has. The parser never ran (0 Bash calls in the case-5 traces), so the model composed the line from prose. That failed 2 of 10 on backticks |
| `no-error-code` → `code\W{1,12}REVIEW_[A-Z_]+`, bound to the validator's JSON output | The listed codes missed `REVIEW_WRONG_TYPE` and `REVIEW_VERSION_FORMAT` |
| `says-pass` / `says-fail` reject a preceding `NOT` and a longer word | Raw substrings: "did not PASS" passed `says-pass` |
| `no-bash-write` beside every `no-write`: redirect, `tee`, `touch`, `cp`, `mv`, `rm`, `sed -i` in the Bash command | `no-write` sees only the Write tool |

All new patterns were run against the ten traces of the PM re-measurement. There
were no false positives, and the known-positive and known-negative still split.

**v2 first run: 8 of 8** (graders 36 of 36, 65 s, $1.97; the tool's list-price
estimate).

**Known-positive for the new case, demonstrated** in a scratch copy (never this
tree):
- With `--check` made to accept every brief (`intent-approval.mjs`), the case dropped
  to **0.75**. `names-code` failed; the other graders passed.
- Removing only the gate's command line from `trekplan.md` did **not** fell the case
  (1.00). The child rebuilt the `--check` call from the prose around it. So the case
  measures whether the gate works, not whether that exact line is there. The line is
  pinned by `tests/validators/intent-approval.test.mjs`, which executes it.

## Measured facts about the harness this suite relies on

These were measured by probe cases in a scratch copy, not taken from the docs.

- **Commands are invocable** in the child as `/voyage:<command>`. The command body is
  expanded into the prompt; no `Skill` tool call is involved.
- **`${CLAUDE_PLUGIN_ROOT}` is substituted in the expanded command text**, so the
  `node ${CLAUDE_PLUGIN_ROOT}/lib/...` lines in the commands reach the real validators.
  In the child's Bash **environment** the variable is **empty**. A case prompt must
  never rely on it directly.
- **The plugin directory is readable** from the child's OS sandbox, so Voyage's
  validators run for real under `--allow-tools Bash`.
- **Fixtures come from an inline `scaffold.sh`** that writes them with heredocs. A run
  starts in an empty workspace and "can't read the eval directory". The docs do not say
  how an `add_dirs` path maps into the run, so the suite does not depend on it.

## What the suite does NOT cover

- **The agent swarms.** It never tests the `/trekplan` exploration, synthesis or review
  phases, the `/trekreview` reviewers and coordinator, or `/trekresearch`. Every case
  asserts zero Agent calls. Whether the swarm plans or reviews *well* is outside this
  suite. The offline gold-scored eval (`tests/lib/gold-eval.test.mjs`, see
  `docs/eval-corpus/README.md`) covers the coordinator contract on recorded payloads.
- **`/trekbrief`**, because its interview needs `AskUserQuestion`, which a headless child
  does not have. **`/trekexecute`, `/trekcontinue` and `/trekendsession`** have no case yet.
- **Plugin contribution.** The suite runs without the no-plugin baseline (`--ablation none`),
  so it does not measure how much Voyage adds over plain Claude. It measures only whether
  Voyage keeps its own promises.
- **Variance.** With `runs: 1`, a single run can pass or fail by chance; the red case
  above shows exactly that. One result is a sample, not a rate.
- **Semantic quality** of any output. Every grader is a pattern, a tool count or a file glob.

## How to run it

From the plugin root:

```bash
claude plugin eval . --trust-plugin --ablation none --runs 1 --scaffold --no-publish --allow-tools Bash Write
```

- `--scaffold` runs each case's `scaffold.sh` as you, outside the sandbox. The four
  scaffolds in this suite only write fixture files into the run's workspace.
- `--allow-tools Bash Write` grants the tools the commands need (Bash runs under the OS
  sandbox). Write is granted so that the `no-write` and `no-plan-file` graders measure a
  refusal, not a missing permission. `Agent` is listed in every case's `allowed_tools`
  for the same reason.
- The exit code is 1 whenever any case scores below 1.0.

## How veikart step 4 ("Trygg å endre") uses it

The suite is the fixed yardstick that a rule change is measured against:

1. Run the suite on the base commit and record `n of N` (`aggregates.casesPassed` /
   `aggregates.casesTotal` in `aggregate-result.json`).
2. Run it again with the change applied.
3. **If `n` is smaller, the change stops.** A lower `n` is not argued away. It is cleared
   only by showing that the same case is also red on the base commit, measured the same
   day. Otherwise the change broke a promise.
4. A new promise gets a new case, and its expectation is committed before its first run,
   the same as here. `N` grows; it never shrinks to make `n` look better.

Because `runs: 1` is a sample, an unstable case such as `review-requires-project` in v1
will sometimes stop a change that is innocent. That is the intended price. Moving the guard
into code removes the instability; editing the grader does not. v2 did the move: the
parser now writes the line, and the grader reads the parser's code. One green run does
not show the instability is gone. That takes more runs.