docs(proevesett): the eval suite's first run is 6 of 7, and the known-positive falls on a planted fault
docs/proevesett.md records what the suite covers and what it does not, how to
run it, the verbatim pass/fail definition with its source, and the first run:
6 of 7 cases (graders 25 of 26), 7 claude -p children, 54 s.
The red case, review-requires-project, lost only its text grader: the child
paraphrased a guard that lives in prose. A diagnostic re-run passed, so it is
unstable, not broken, and it stays red - loosening the pattern after the run
would fit the grader to the result.
Known-positive shown in a scratch copy: with the finding-ID check disabled in
review-validator.mjs, review-validate-flags-bad-finding-id drops from 1.0 to 0.6.
Also recorded, measured by probe: ${CLAUDE_PLUGIN_ROOT} is substituted in the
expanded command text but empty in the child's Bash env, and the plugin
directory is readable from the child's sandbox.
Veikart step 4 uses the suite as its yardstick: a rule change that lowers n stops.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
d311f3b49a
commit
0eb18f39e6
SSH key fingerprint:
SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
1 changed files with 147 additions and 0 deletions
147
docs/proevesett.md
Normal file
147
docs/proevesett.md
Normal file
|
|
@ -0,0 +1,147 @@
|
|||
# Prøvesett — Voyage's fixed test set as a plugin eval suite
|
||||
|
||||
**Status:** v1, seven cases, first run **6 of 7** (2026-09-23, Claude Code 2.1.280).
|
||||
The cases and their expectations were committed *before* the first run; the
|
||||
run did not change them. Red is data here, not a defect in the suite.
|
||||
|
||||
The unit suite (`npm test`) proves Voyage's modules in isolation. This suite proves
|
||||
something the unit suite cannot: that the **commands, as a model reads and follows
|
||||
them in a real headless session**, keep the promises they make. Each case is a real
|
||||
`claude -p` child with only Voyage loaded.
|
||||
|
||||
## What "pass" and "fail" mean (tool definition, verbatim)
|
||||
|
||||
Source: Claude Code docs, *Test plugins with evals* (`code.claude.com/docs/en/plugin-evals`),
|
||||
§ How a case is scored and § Grader types.
|
||||
|
||||
> A run's score is the fraction of its graders that passed, weighted if you set weights,
|
||||
> and the case's score is the mean across its runs. A case passes when its score meets the
|
||||
> `--threshold`, `1.0` by default.
|
||||
|
||||
The four graders this suite uses, and when each passes:
|
||||
|
||||
| Grader | Passes when (verbatim) |
|
||||
|--------|------------------------|
|
||||
| `regex` | "The JavaScript regex `pattern` is found in the target. Set `match: not_contains` to require absence" |
|
||||
| `tool_used` | "The number of calls to `tool` whose JSON-encoded input matches the optional `input_match` regex is between `min`, default 1, and `max`, default unlimited. To assert a tool was never called, set both `min: 0` and `max: 0`" |
|
||||
| `file_exists` | "A file Claude created matches the `path` glob, or none does with `exists: false`. Only files created during the run count" |
|
||||
|
||||
`regex` looks at `last_message` ("Claude's final response text. This is the default")
|
||||
unless the case sets `target: trace` ("The session as JSON, one message per line").
|
||||
|
||||
With `runs: 1` and the default threshold, a case **passes** only when **every** grader
|
||||
passes, and **fails** when any single one does not. `n of N` below counts cases, not graders.
|
||||
|
||||
## The cases
|
||||
|
||||
Every case has `runs: 1`, deterministic graders only (no `llm`, no `baseline` — they cost
|
||||
judge calls), and runs without the no-plugin baseline arm. Each tests **one** thing Voyage
|
||||
promises and ends in seconds. Nothing starts the `/trekplan` exploration swarm or the
|
||||
`/trekreview` reviewer agents; the `no-agent` grader in every case asserts it.
|
||||
|
||||
| Case | Voyage promise under test | Expected (committed before the run) | Run 1 |
|
||||
|------|---------------------------|--------------------------------------|-------|
|
||||
| `plan-requires-brief` | `/trekplan` without a brief prints usage and stops | "A brief is required"; no Agent, no Write | pass |
|
||||
| `plan-project-not-initialized` | `/trekplan --project` on a missing folder stops before exploring | "project directory not initialized"; no Agent, no `plan.md` | pass |
|
||||
| `plan-rejects-unknown-export` | a removed export format is named and refused | "format 'pr' is not supported"; no Agent, no Write | pass |
|
||||
| `plan-halts-without-phase-signals` | a `brief_version` 2.1 brief without `phase_signals` halts at the sequencing gate | brief-validator runs, reply names `phase_signals`; no Agent, no `plan.md` | pass |
|
||||
| `review-requires-project` | `/trekreview` without `--project` refuses | "--project <dir> is required"; no Agent, no Write | **fail** (2 of 3 graders) |
|
||||
| `review-validate-flags-bad-finding-id` | **known-positive:** `--validate` fails a review whose finding ID is not 40-char hex, and names the rule | review-validator runs; `REVIEW_BAD_FINDING_ID` in the trace; reply says FAIL; no Agent, no Write | pass |
|
||||
| `review-validate-passes-clean-review` | **known-negative:** `--validate` passes a valid review and raises nothing | review-validator runs; reply says PASS; no `REVIEW_*` error code in the trace; no Agent, no Write | pass |
|
||||
|
||||
### The one red case
|
||||
|
||||
`review-requires-project` failed on its text grader only: the child replied in one turn
|
||||
without reproducing the exact error line `Error: --project <dir> is required.` The
|
||||
Agent and Write guards held. A diagnostic re-run of the same case alone passed, with the
|
||||
exact line. So the case is **unstable, not broken**. It is still red on purpose. This
|
||||
guard lives only in the command's prose, and a model does not always repeat prose word
|
||||
for word. Loosening the pattern after seeing the result would fit the grader to the run.
|
||||
If this should hold every time, the fix belongs in Voyage: let code emit the message, as
|
||||
the validators already emit their rule codes (`REVIEW_BAD_FINDING_ID` is printed by
|
||||
`review-validator.mjs`, not composed by the model — that is why `names-rule` greps the
|
||||
trace, not the reply).
|
||||
|
||||
### Known-positive, demonstrated
|
||||
|
||||
Proof that the known-positive case can fail: in a scratch copy of Voyage (never this
|
||||
tree), the finding-ID check in `lib/validators/review-validator.mjs` was disabled. The
|
||||
validator then accepted the planted `F-001` ID, and
|
||||
`review-validate-flags-bad-finding-id` dropped from **1.0 to 0.6**. `names-rule` and
|
||||
`says-fail` failed; `validator-ran`, `no-agent` and `no-write` still passed. The case
|
||||
therefore falls when this part of Voyage breaks.
|
||||
|
||||
## First run, recorded
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Result | **6 of 7** cases passed (graders 25 of 26; overall score 0.952 = mean of the case scores) |
|
||||
| Children | 7 `claude -p` runs (7 cases × 1 run × 1 arm) |
|
||||
| Wall time | 54 s (the tool reports `durationSeconds: 53`) |
|
||||
| Cost | $1.67, the tool's list-price estimate; on a subscription it counts against usage limits instead |
|
||||
| Output | `evals/results/<timestamp>/aggregate-result.json` + `report.html` (gitignored) |
|
||||
|
||||
## Measured facts about the harness this suite relies on
|
||||
|
||||
These were measured by probe cases in a scratch copy, not taken from the docs.
|
||||
|
||||
- **Commands are invocable** in the child as `/voyage:<command>`. The command body is
|
||||
expanded into the prompt; no `Skill` tool call is involved.
|
||||
- **`${CLAUDE_PLUGIN_ROOT}` is substituted in the expanded command text**, so the
|
||||
`node ${CLAUDE_PLUGIN_ROOT}/lib/...` lines in the commands reach the real validators.
|
||||
In the child's Bash **environment** the variable is **empty**. A case prompt must
|
||||
never rely on it directly.
|
||||
- **The plugin directory is readable** from the child's OS sandbox, so Voyage's
|
||||
validators run for real under `--allow-tools Bash`.
|
||||
- **Fixtures come from an inline `scaffold.sh`** that writes them with heredocs. A run
|
||||
starts in an empty workspace and "can't read the eval directory". The docs do not say
|
||||
how an `add_dirs` path maps into the run, so the suite does not depend on it.
|
||||
|
||||
## What the suite does NOT cover
|
||||
|
||||
- **The agent swarms.** It never tests the `/trekplan` exploration, synthesis or review
|
||||
phases, the `/trekreview` reviewers and coordinator, or `/trekresearch`. Every case
|
||||
asserts zero Agent calls. Whether the swarm plans or reviews *well* is outside this
|
||||
suite. The offline gold-scored eval (`tests/lib/gold-eval.test.mjs`, see
|
||||
`docs/eval-corpus/README.md`) covers the coordinator contract on recorded payloads.
|
||||
- **`/trekbrief`**, because its interview needs `AskUserQuestion`, which a headless child
|
||||
does not have. **`/trekexecute`, `/trekcontinue` and `/trekendsession`** have no case yet.
|
||||
- **Plugin contribution.** The suite runs without the no-plugin baseline (`--ablation none`),
|
||||
so it does not measure how much Voyage adds over plain Claude. It measures only whether
|
||||
Voyage keeps its own promises.
|
||||
- **Variance.** With `runs: 1`, a single run can pass or fail by chance; the red case
|
||||
above shows exactly that. One result is a sample, not a rate.
|
||||
- **Semantic quality** of any output. Every grader is a pattern, a tool count or a file glob.
|
||||
|
||||
## How to run it
|
||||
|
||||
From the plugin root:
|
||||
|
||||
```bash
|
||||
claude plugin eval . --trust-plugin --ablation none --runs 1 --scaffold --no-publish --allow-tools Bash Write
|
||||
```
|
||||
|
||||
- `--scaffold` runs each case's `scaffold.sh` as you, outside the sandbox. The three
|
||||
scaffolds in this suite only write fixture files into the run's workspace.
|
||||
- `--allow-tools Bash Write` grants the tools the commands need (Bash runs under the OS
|
||||
sandbox). Write is granted so that the `no-write` and `no-plan-file` graders measure a
|
||||
refusal, not a missing permission. `Agent` is listed in every case's `allowed_tools`
|
||||
for the same reason.
|
||||
- The exit code is 1 whenever any case scores below 1.0.
|
||||
|
||||
## How veikart step 4 ("Trygg å endre") uses it
|
||||
|
||||
The suite is the fixed yardstick that a rule change is measured against:
|
||||
|
||||
1. Run the suite on the base commit and record `n of N` (`aggregates.casesPassed` /
|
||||
`aggregates.casesTotal` in `aggregate-result.json`).
|
||||
2. Run it again with the change applied.
|
||||
3. **If `n` is smaller, the change stops.** A lower `n` is not argued away. It is cleared
|
||||
only by showing that the same case is also red on the base commit, measured the same
|
||||
day. Otherwise the change broke a promise.
|
||||
4. A new promise gets a new case, and its expectation is committed before its first run,
|
||||
the same as here. `N` grows; it never shrinks to make `n` look better.
|
||||
|
||||
Because `runs: 1` is a sample, an unstable case such as `review-requires-project` will
|
||||
sometimes stop a change that is innocent. That is the intended price. Moving the guard
|
||||
into code removes the instability; editing the grader does not.
|
||||
Loading…
Add table
Add a link
Reference in a new issue