merge: Voyage's fixed test set as a plugin-eval suite (0eb18f3) onto step 1's intent gate

.gitignore

@@ -9,6 +9,10 @@ test-results/
 playwright-report/
 blob-report/
 
+# `claude plugin eval` run output (aggregate-result.json, report.html). Results are
+# data about one run on one machine; the suite (evals/<case>/) is what is tracked.
+/evals/results/
+
 # Editor files
 *.swp
 *.swo

docs/proevesett.md

@@ -0,0 +1,147 @@
+# Prøvesett — Voyage's fixed test set as a plugin eval suite
+
+**Status:** v1, seven cases, first run **6 of 7** (2026-09-23, Claude Code 2.1.280).
+The cases and their expectations were committed *before* the first run; the
+run did not change them. Red is data here, not a defect in the suite.
+
+The unit suite (`npm test`) proves Voyage's modules in isolation. This suite proves
+something the unit suite cannot: that the **commands, as a model reads and follows
+them in a real headless session**, keep the promises they make. Each case is a real
+`claude -p` child with only Voyage loaded.
+
+## What "pass" and "fail" mean (tool definition, verbatim)
+
+Source: Claude Code docs, *Test plugins with evals* (`code.claude.com/docs/en/plugin-evals`),
+§ How a case is scored and § Grader types.
+
+> A run's score is the fraction of its graders that passed, weighted if you set weights,
+> and the case's score is the mean across its runs. A case passes when its score meets the
+> `--threshold`, `1.0` by default.
+
+The four graders this suite uses, and when each passes:
+
+| Grader | Passes when (verbatim) |
+|--------|------------------------|
+| `regex` | "The JavaScript regex `pattern` is found in the target. Set `match: not_contains` to require absence" |
+| `tool_used` | "The number of calls to `tool` whose JSON-encoded input matches the optional `input_match` regex is between `min`, default 1, and `max`, default unlimited. To assert a tool was never called, set both `min: 0` and `max: 0`" |
+| `file_exists` | "A file Claude created matches the `path` glob, or none does with `exists: false`. Only files created during the run count" |
+
+`regex` looks at `last_message` ("Claude's final response text. This is the default")
+unless the case sets `target: trace` ("The session as JSON, one message per line").
+
+With `runs: 1` and the default threshold, a case **passes** only when **every** grader
+passes, and **fails** when any single one does not. `n of N` below counts cases, not graders.
+
+## The cases
+
+Every case has `runs: 1`, deterministic graders only (no `llm`, no `baseline` — they cost
+judge calls), and runs without the no-plugin baseline arm. Each tests **one** thing Voyage
+promises and ends in seconds. Nothing starts the `/trekplan` exploration swarm or the
+`/trekreview` reviewer agents; the `no-agent` grader in every case asserts it.
+
+| Case | Voyage promise under test | Expected (committed before the run) | Run 1 |
+|------|---------------------------|--------------------------------------|-------|
+| `plan-requires-brief` | `/trekplan` without a brief prints usage and stops | "A brief is required"; no Agent, no Write | pass |
+| `plan-project-not-initialized` | `/trekplan --project` on a missing folder stops before exploring | "project directory not initialized"; no Agent, no `plan.md` | pass |
+| `plan-rejects-unknown-export` | a removed export format is named and refused | "format 'pr' is not supported"; no Agent, no Write | pass |
+| `plan-halts-without-phase-signals` | a `brief_version` 2.1 brief without `phase_signals` halts at the sequencing gate | brief-validator runs, reply names `phase_signals`; no Agent, no `plan.md` | pass |
+| `review-requires-project` | `/trekreview` without `--project` refuses | "--project <dir> is required"; no Agent, no Write | **fail** (2 of 3 graders) |
+| `review-validate-flags-bad-finding-id` | **known-positive:** `--validate` fails a review whose finding ID is not 40-char hex, and names the rule | review-validator runs; `REVIEW_BAD_FINDING_ID` in the trace; reply says FAIL; no Agent, no Write | pass |
+| `review-validate-passes-clean-review` | **known-negative:** `--validate` passes a valid review and raises nothing | review-validator runs; reply says PASS; no `REVIEW_*` error code in the trace; no Agent, no Write | pass |
+
+### The one red case
+
+`review-requires-project` failed on its text grader only: the child replied in one turn
+without reproducing the exact error line `Error: --project <dir> is required.` The
+Agent and Write guards held. A diagnostic re-run of the same case alone passed, with the
+exact line. So the case is **unstable, not broken**. It is still red on purpose. This
+guard lives only in the command's prose, and a model does not always repeat prose word
+for word. Loosening the pattern after seeing the result would fit the grader to the run.
+If this should hold every time, the fix belongs in Voyage: let code emit the message, as
+the validators already emit their rule codes (`REVIEW_BAD_FINDING_ID` is printed by
+`review-validator.mjs`, not composed by the model — that is why `names-rule` greps the
+trace, not the reply).
+
+### Known-positive, demonstrated
+
+Proof that the known-positive case can fail: in a scratch copy of Voyage (never this
+tree), the finding-ID check in `lib/validators/review-validator.mjs` was disabled. The
+validator then accepted the planted `F-001` ID, and
+`review-validate-flags-bad-finding-id` dropped from **1.0 to 0.6**. `names-rule` and
+`says-fail` failed; `validator-ran`, `no-agent` and `no-write` still passed. The case
+therefore falls when this part of Voyage breaks.
+
+## First run, recorded
+
+| | |
+|---|---|
+| Result | **6 of 7** cases passed (graders 25 of 26; overall score 0.952 = mean of the case scores) |
+| Children | 7 `claude -p` runs (7 cases × 1 run × 1 arm) |
+| Wall time | 54 s (the tool reports `durationSeconds: 53`) |
+| Cost | $1.67, the tool's list-price estimate; on a subscription it counts against usage limits instead |
+| Output | `evals/results/<timestamp>/aggregate-result.json` + `report.html` (gitignored) |
+
+## Measured facts about the harness this suite relies on
+
+These were measured by probe cases in a scratch copy, not taken from the docs.
+
+- **Commands are invocable** in the child as `/voyage:<command>`. The command body is
+  expanded into the prompt; no `Skill` tool call is involved.
+- **`${CLAUDE_PLUGIN_ROOT}` is substituted in the expanded command text**, so the
+  `node ${CLAUDE_PLUGIN_ROOT}/lib/...` lines in the commands reach the real validators.
+  In the child's Bash **environment** the variable is **empty**. A case prompt must
+  never rely on it directly.
+- **The plugin directory is readable** from the child's OS sandbox, so Voyage's
+  validators run for real under `--allow-tools Bash`.
+- **Fixtures come from an inline `scaffold.sh`** that writes them with heredocs. A run
+  starts in an empty workspace and "can't read the eval directory". The docs do not say
+  how an `add_dirs` path maps into the run, so the suite does not depend on it.
+
+## What the suite does NOT cover
+
+- **The agent swarms.** It never tests the `/trekplan` exploration, synthesis or review
+  phases, the `/trekreview` reviewers and coordinator, or `/trekresearch`. Every case
+  asserts zero Agent calls. Whether the swarm plans or reviews *well* is outside this
+  suite. The offline gold-scored eval (`tests/lib/gold-eval.test.mjs`, see
+  `docs/eval-corpus/README.md`) covers the coordinator contract on recorded payloads.
+- **`/trekbrief`**, because its interview needs `AskUserQuestion`, which a headless child
+  does not have. **`/trekexecute`, `/trekcontinue` and `/trekendsession`** have no case yet.
+- **Plugin contribution.** The suite runs without the no-plugin baseline (`--ablation none`),
+  so it does not measure how much Voyage adds over plain Claude. It measures only whether
+  Voyage keeps its own promises.
+- **Variance.** With `runs: 1`, a single run can pass or fail by chance; the red case
+  above shows exactly that. One result is a sample, not a rate.
+- **Semantic quality** of any output. Every grader is a pattern, a tool count or a file glob.
+
+## How to run it
+
+From the plugin root:
+
+```bash
+claude plugin eval . --trust-plugin --ablation none --runs 1 --scaffold --no-publish --allow-tools Bash Write
+```
+
+- `--scaffold` runs each case's `scaffold.sh` as you, outside the sandbox. The three
+  scaffolds in this suite only write fixture files into the run's workspace.
+- `--allow-tools Bash Write` grants the tools the commands need (Bash runs under the OS
+  sandbox). Write is granted so that the `no-write` and `no-plan-file` graders measure a
+  refusal, not a missing permission. `Agent` is listed in every case's `allowed_tools`
+  for the same reason.
+- The exit code is 1 whenever any case scores below 1.0.
+
+## How veikart step 4 ("Trygg å endre") uses it
+
+The suite is the fixed yardstick that a rule change is measured against:
+
+1. Run the suite on the base commit and record `n of N` (`aggregates.casesPassed` /
+   `aggregates.casesTotal` in `aggregate-result.json`).
+2. Run it again with the change applied.
+3. **If `n` is smaller, the change stops.** A lower `n` is not argued away. It is cleared
+   only by showing that the same case is also red on the base commit, measured the same
+   day. Otherwise the change broke a promise.
+4. A new promise gets a new case, and its expectation is committed before its first run,
+   the same as here. `N` grows; it never shrinks to make `n` look better.
+
+Because `runs: 1` is a sample, an unstable case such as `review-requires-project` will
+sometimes stop a change that is innocent. That is the intended price. Moving the guard
+into code removes the instability; editing the grader does not.

evals/plan-halts-without-phase-signals/case.yaml

@@ -0,0 +1,4 @@
+schema_version: "1.1"
+name: plan-halts-without-phase-signals
+context:
+  scaffold_script: scaffold.sh

evals/plan-halts-without-phase-signals/graders/halt-names-signals.md

@@ -0,0 +1,4 @@
+---
+type: regex
+pattern: "phase_signals"
+---

evals/plan-halts-without-phase-signals/graders/no-agent.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Agent
+min: 0
+max: 0
+---

evals/plan-halts-without-phase-signals/graders/no-plan-file.md

@@ -0,0 +1,5 @@
+---
+type: file_exists
+path: "**/plan.md"
+exists: false
+---

evals/plan-halts-without-phase-signals/graders/validator-ran.md

@@ -0,0 +1,5 @@
+---
+type: tool_used
+tool: Bash
+input_match: "brief-validator"
+---

evals/plan-halts-without-phase-signals/prompt.md

@@ -0,0 +1,11 @@
+---
+description: "/trekplan on a brief_version 2.1 brief without phase_signals halts at the sequencing gate."
+expected_outcome: "brief-validator runs, reports BRIEF_V51_MISSING_SIGNALS, and /trekplan halts with the phase_signals hint; no Agent call (no exploration swarm), no plan.md."
+tags: [proevesett]
+runs: 1
+max_turns: 15
+timeout_seconds: 300
+allowed_tools: [Bash, Read, Glob, Grep, Write, Agent]
+---
+
+/voyage:trekplan --project proj

evals/plan-halts-without-phase-signals/scaffold.sh

@@ -0,0 +1,35 @@
+#!/bin/bash
+# Seeds the empty run workspace with a project folder whose brief declares
+# brief_version 2.1 but carries neither phase_signals nor phase_signals_partial.
+# Self-contained on purpose: the run cannot read the eval directory.
+set -eu
+mkdir -p proj
+cat > proj/brief.md <<'BRIEF'
+---
+type: trekbrief
+brief_version: "2.1"
+created: 2026-09-23
+task: "Eval fixture: add a greeting helper"
+slug: eval-greeting
+project_dir: proj/
+research_topics: 0
+research_status: skipped
+auto_research: false
+interview_turns: 1
+source: fixture
+---
+
+# Task: add a greeting helper
+
+## Intent
+
+Eval fixture for the Voyage proevesett. Not a real task.
+
+## Goal
+
+A function `greet(name)` returns `Hello, <name>!`.
+
+## Success Criteria
+
+- `greet("Ada")` returns `Hello, Ada!`.
+BRIEF

evals/plan-project-not-initialized/graders/no-agent.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Agent
+min: 0
+max: 0
+---

evals/plan-project-not-initialized/graders/no-plan-file.md

@@ -0,0 +1,5 @@
+---
+type: file_exists
+path: "**/plan.md"
+exists: false
+---

evals/plan-project-not-initialized/graders/not-initialized.md

@@ -0,0 +1,4 @@
+---
+type: regex
+pattern: "project directory not initialized"
+---

evals/plan-project-not-initialized/prompt.md

@@ -0,0 +1,11 @@
+---
+description: "/trekplan --project on a missing folder stops before any exploration."
+expected_outcome: "'Error: project directory not initialized. Run /trekbrief to create it.'; no Agent call, no plan.md."
+tags: [proevesett]
+runs: 1
+max_turns: 8
+timeout_seconds: 300
+allowed_tools: [Bash, Read, Glob, Grep, Write, Agent]
+---
+
+/voyage:trekplan --project missing-project

evals/plan-rejects-unknown-export/graders/format-rejected.md

@@ -0,0 +1,4 @@
+---
+type: regex
+pattern: "format 'pr' is not supported"
+---

evals/plan-rejects-unknown-export/graders/no-agent.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Agent
+min: 0
+max: 0
+---

evals/plan-rejects-unknown-export/graders/no-write.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Write
+min: 0
+max: 0
+---

evals/plan-rejects-unknown-export/prompt.md

@@ -0,0 +1,11 @@
+---
+description: "/trekplan --export with a removed format names the format and stops."
+expected_outcome: "'Error: export format 'pr' is not supported.'; no Agent call, no file written."
+tags: [proevesett]
+runs: 1
+max_turns: 8
+timeout_seconds: 300
+allowed_tools: [Bash, Read, Glob, Grep, Write, Agent]
+---
+
+/voyage:trekplan --export pr plan.md

evals/plan-requires-brief/graders/no-agent.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Agent
+min: 0
+max: 0
+---

evals/plan-requires-brief/graders/no-write.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Write
+min: 0
+max: 0
+---

evals/plan-requires-brief/graders/usage-stop.md

@@ -0,0 +1,4 @@
+---
+type: regex
+pattern: "A brief is required"
+---

evals/plan-requires-brief/prompt.md

@@ -0,0 +1,11 @@
+---
+description: "/trekplan with no arguments prints usage and stops."
+expected_outcome: "Usage block ending in 'A brief is required. Produce one with /trekbrief first.'; no Agent call, no file written."
+tags: [proevesett]
+runs: 1
+max_turns: 8
+timeout_seconds: 300
+allowed_tools: [Bash, Read, Glob, Grep, Write, Agent]
+---
+
+/voyage:trekplan

evals/review-requires-project/graders/no-agent.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Agent
+min: 0
+max: 0
+---

evals/review-requires-project/graders/no-write.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Write
+min: 0
+max: 0
+---

evals/review-requires-project/graders/project-required.md

@@ -0,0 +1,4 @@
+---
+type: regex
+pattern: "--project <dir> is required"
+---

evals/review-requires-project/prompt.md

@@ -0,0 +1,11 @@
+---
+description: "/trekreview with no arguments demands --project and stops."
+expected_outcome: "'Error: --project <dir> is required.' plus usage; no Agent call, no file written."
+tags: [proevesett]
+runs: 1
+max_turns: 8
+timeout_seconds: 300
+allowed_tools: [Bash, Read, Glob, Grep, Write, Agent]
+---
+
+/voyage:trekreview

evals/review-validate-flags-bad-finding-id/case.yaml

@@ -0,0 +1,4 @@
+schema_version: "1.1"
+name: review-validate-flags-bad-finding-id
+context:
+  scaffold_script: scaffold.sh

evals/review-validate-flags-bad-finding-id/graders/names-rule.md

@@ -0,0 +1,5 @@
+---
+type: regex
+pattern: "REVIEW_BAD_FINDING_ID"
+target: trace
+---

evals/review-validate-flags-bad-finding-id/graders/no-agent.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Agent
+min: 0
+max: 0
+---

evals/review-validate-flags-bad-finding-id/graders/no-write.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Write
+min: 0
+max: 0
+---

evals/review-validate-flags-bad-finding-id/graders/says-fail.md

@@ -0,0 +1,4 @@
+---
+type: regex
+pattern: "FAIL"
+---

evals/review-validate-flags-bad-finding-id/graders/validator-ran.md

@@ -0,0 +1,5 @@
+---
+type: tool_used
+tool: Bash
+input_match: "review-validator"
+---

evals/review-validate-flags-bad-finding-id/prompt.md

@@ -0,0 +1,11 @@
+---
+description: "KNOWN-POSITIVE. /trekreview --validate on a review.md whose finding ID is not 40-char hex fails and names the rule."
+expected_outcome: "review-validator runs and reports REVIEW_BAD_FINDING_ID; the one-line summary says FAIL; no Agent call, no file written."
+tags: [proevesett]
+runs: 1
+max_turns: 10
+timeout_seconds: 300
+allowed_tools: [Bash, Read, Glob, Grep, Write, Agent]
+---
+
+/voyage:trekreview --project proj --validate

evals/review-validate-flags-bad-finding-id/scaffold.sh

@@ -0,0 +1,79 @@
+#!/bin/bash
+# Seeds the empty run workspace with a project folder whose review.md carries a finding ID that is not 40-char hex.
+# Self-contained on purpose: the run cannot read the eval directory, and the
+# fixtures are written inline so no path outside the workspace is involved.
+set -eu
+mkdir -p proj
+cat > proj/brief.md <<'BRIEF'
+---
+type: trekbrief
+brief_version: "2.1"
+created: 2026-09-23
+task: "Eval fixture: add a greeting helper"
+slug: eval-greeting
+project_dir: proj/
+research_topics: 0
+research_status: skipped
+auto_research: false
+interview_turns: 1
+source: fixture
+phase_signals:
+  - phase: plan
+    effort: low
+  - phase: review
+    effort: low
+---
+
+# Task: add a greeting helper
+
+## Intent
+
+Eval fixture for the Voyage proevesett. Not a real task.
+
+## Goal
+
+A function `greet(name)` returns `Hello, <name>!`.
+
+## Success Criteria
+
+- `greet("Ada")` returns `Hello, Ada!`.
+BRIEF
+cat > proj/review.md <<'REVIEW'
+---
+type: trekreview
+review_version: "1.0"
+created: 2026-09-23
+task: "Eval fixture: add a greeting helper"
+slug: eval-greeting
+project_dir: proj/
+brief_path: proj/brief.md
+scope_sha_start: 0123456789abcdef0123456789abcdef01234567
+scope_sha_end: fedcba9876543210fedcba9876543210fedcba98
+reviewed_files_count: 1
+verdict: ALLOW
+findings:
+  - F-001
+---
+
+# Review: add a greeting helper
+
+## Executive Summary
+
+Eval fixture. One SUGGESTION, verdict ALLOW.
+
+## Coverage
+
+One file reviewed: `src/greet.js`.
+
+## Findings (SUGGESTION)
+
+- `src/greet.js:1` - name the parameter more descriptively.
+
+## Remediation Summary
+
+Nothing blocks. The suggestion is optional.
+
+```json
+{"findings":[{"id":"d2d0e27875ae9ef0d818cb08bb6f14e6d33c4232","severity":"SUGGESTION","file":"src/greet.js","line":1,"rule_key":"naming","title":"Parameter name","detail":"Eval fixture.","recommended_action":"Rename the parameter."}]}
+```
+REVIEW

evals/review-validate-passes-clean-review/case.yaml

@@ -0,0 +1,4 @@
+schema_version: "1.1"
+name: review-validate-passes-clean-review
+context:
+  scaffold_script: scaffold.sh

evals/review-validate-passes-clean-review/graders/no-agent.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Agent
+min: 0
+max: 0
+---

evals/review-validate-passes-clean-review/graders/no-error-code.md

@@ -0,0 +1,6 @@
+---
+type: regex
+pattern: "REVIEW_(MISSING_FIELD|BAD_FINDING_ID|BAD_FINDINGS_TYPE|MISSING_SECTION|NOT_FOUND|READ_ERROR)"
+target: trace
+match: not_contains
+---

evals/review-validate-passes-clean-review/graders/no-write.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Write
+min: 0
+max: 0
+---

evals/review-validate-passes-clean-review/graders/says-pass.md

@@ -0,0 +1,4 @@
+---
+type: regex
+pattern: "PASS"
+---

evals/review-validate-passes-clean-review/graders/validator-ran.md

@@ -0,0 +1,5 @@
+---
+type: tool_used
+tool: Bash
+input_match: "review-validator"
+---

evals/review-validate-passes-clean-review/prompt.md

@@ -0,0 +1,11 @@
+---
+description: "KNOWN-NEGATIVE. /trekreview --validate on a valid review.md passes and raises no error code."
+expected_outcome: "review-validator runs; the one-line summary says PASS; no REVIEW_* error code appears anywhere in the trace; no Agent call, no file written."
+tags: [proevesett]
+runs: 1
+max_turns: 10
+timeout_seconds: 300
+allowed_tools: [Bash, Read, Glob, Grep, Write, Agent]
+---
+
+/voyage:trekreview --project proj --validate

evals/review-validate-passes-clean-review/scaffold.sh

@@ -0,0 +1,79 @@
+#!/bin/bash
+# Seeds the empty run workspace with a project folder whose review.md is valid.
+# Self-contained on purpose: the run cannot read the eval directory, and the
+# fixtures are written inline so no path outside the workspace is involved.
+set -eu
+mkdir -p proj
+cat > proj/brief.md <<'BRIEF'
+---
+type: trekbrief
+brief_version: "2.1"
+created: 2026-09-23
+task: "Eval fixture: add a greeting helper"
+slug: eval-greeting
+project_dir: proj/
+research_topics: 0
+research_status: skipped
+auto_research: false
+interview_turns: 1
+source: fixture
+phase_signals:
+  - phase: plan
+    effort: low
+  - phase: review
+    effort: low
+---
+
+# Task: add a greeting helper
+
+## Intent
+
+Eval fixture for the Voyage proevesett. Not a real task.
+
+## Goal
+
+A function `greet(name)` returns `Hello, <name>!`.
+
+## Success Criteria
+
+- `greet("Ada")` returns `Hello, Ada!`.
+BRIEF
+cat > proj/review.md <<'REVIEW'
+---
+type: trekreview
+review_version: "1.0"
+created: 2026-09-23
+task: "Eval fixture: add a greeting helper"
+slug: eval-greeting
+project_dir: proj/
+brief_path: proj/brief.md
+scope_sha_start: 0123456789abcdef0123456789abcdef01234567
+scope_sha_end: fedcba9876543210fedcba9876543210fedcba98
+reviewed_files_count: 1
+verdict: ALLOW
+findings:
+  - d2d0e27875ae9ef0d818cb08bb6f14e6d33c4232
+---
+
+# Review: add a greeting helper
+
+## Executive Summary
+
+Eval fixture. One SUGGESTION, verdict ALLOW.
+
+## Coverage
+
+One file reviewed: `src/greet.js`.
+
+## Findings (SUGGESTION)
+
+- `src/greet.js:1` - name the parameter more descriptively.
+
+## Remediation Summary
+
+Nothing blocks. The suggestion is optional.
+
+```json
+{"findings":[{"id":"d2d0e27875ae9ef0d818cb08bb6f14e6d33c4232","severity":"SUGGESTION","file":"src/greet.js","line":1,"rule_key":"naming","title":"Parameter name","detail":"Eval fixture.","recommended_action":"Rename the parameter."}]}
+```
+REVIEW