chore(cleanup): R-b R3 — profiles, phase signals and the effort dialog removed, one fixed path per command
The four profiles, the resolver, the profile validator and docs/profiles.md go; /trekbrief Phase 3.5 and the brief's phase_signals checks go (an older brief's key is ignored); MANIFEST_PROFILE_DRIFT and profile_used go. Every Profile, Composition rule, Sequencing gate and High-effort section leaves the commands. Decision 209: /trekresearch runs the full swarm with contrarian-researcher as wave 2 every run; /trekreview and the coordinator run all four passes with no high-effort normalization. Voyage agents are spawned with no model (their own model: opus applies); the two built-in spawns pass model "opus". --profile stays in the parser as a retired flag that swallows its value. §Model & effort moves to docs/architecture.md; the sequencing-gate eval case goes. Files removed: 44 (43 from the R3 list + tests/commands/trekreview.test.mjs, whose every test pinned removed text). Files edited: 43. Tests: 2467 (2466 pass / 0 fail / 1 skipped) before, 2309 (2309 pass / 0 fail / 0 skipped) after. Durable check: all three groups green, none pending. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
32621dcb63
commit
2a35d067d9
87 changed files with 225 additions and 4967 deletions
|
|
@ -1,6 +1,7 @@
|
|||
# Prøvesett — Voyage's fixed test set as a plugin eval suite
|
||||
|
||||
**Status:** v2, eight cases. v1 (seven cases) first run **6 of 7**; v2 first run
|
||||
**Status:** seven cases: v2's eight, less the sequencing-gate case (v1 case 4), which
|
||||
went with the phase signals in cleanup R-b. v1 (seven cases) first run **6 of 7**; v2 first run
|
||||
**8 of 8** (both 2026-09-23, Claude Code 2.1.280). Each version's cases and
|
||||
expectations were committed *before* its first run; the run did not change them.
|
||||
Red is data here, not a defect in the suite. What v2 changed and why: § v2 below.
|
||||
|
|
@ -53,12 +54,12 @@ promises and ends in seconds. Nothing starts the `/trekplan` exploration swarm o
|
|||
| `plan-requires-brief` | `/trekplan` without a brief prints usage and stops | "A brief is required"; no Agent, no Write | pass |
|
||||
| `plan-project-not-initialized` | `/trekplan --project` on a missing folder stops before exploring | "project directory not initialized"; no Agent, no `plan.md` | pass |
|
||||
| `plan-rejects-unknown-export` | a removed export format is named and refused | "format 'pr' is not supported"; no Agent, no Write | pass |
|
||||
| `plan-halts-without-phase-signals` | a `brief_version` 2.1 brief without `phase_signals` halts at the sequencing gate | brief-validator runs, reply names `phase_signals`; no Agent, no `plan.md` | pass |
|
||||
| `review-requires-project` | `/trekreview` without `--project` refuses | "--project <dir> is required"; no Agent, no Write | **fail** (2 of 3 graders) |
|
||||
| `review-validate-flags-bad-finding-id` | **known-positive:** `--validate` fails a review whose finding ID is not 40-char hex, and names the rule | review-validator runs; `REVIEW_BAD_FINDING_ID` in the trace; reply says FAIL; no Agent, no Write | pass |
|
||||
| `review-validate-passes-clean-review` | **known-negative:** `--validate` passes a valid review and raises nothing | review-validator runs; reply says PASS; no `REVIEW_*` error code in the trace; no Agent, no Write | pass |
|
||||
|
||||
The v1 table above is kept as it was run. v2 rows and grader changes: § v2.
|
||||
The v1 table above is kept as it was run, less case 4 (the sequencing gate, pass),
|
||||
removed in cleanup R-b with the phase signals it tested. v2 rows and grader changes: § v2.
|
||||
|
||||
### The one red case (v1; fixed in v2, see § v2)
|
||||
|
||||
|
|
@ -99,7 +100,7 @@ Expectations committed in `3892a83` before the v2 run. Offline structure tests i
|
|||
|
||||
| Change | Why |
|
||||
|--------|-----|
|
||||
| **New case `plan-halts-without-intent-approval`**: a `brief_version` 2.1 brief *with* `phase_signals` and no `intent_approved_hash`. Graders: `gate-ran` (`intent-approval.mjs --check` in a Bash call), `names-code` (the gate's own JSON `code` for `BRIEF_INTENT_NOT_APPROVED` in the trace), `no-agent`, `no-plan-file` | Veikart steg 1's gate had no case. No v1 trace reached it: case 4 halts at the sequencing gate first |
|
||||
| **New case `plan-halts-without-intent-approval`**: a `brief_version` 2.1 brief with no `intent_approved_hash` (it carried phase signals until cleanup R-b). Graders: `gate-ran` (`intent-approval.mjs --check` in a Bash call), `names-code` (the gate's own JSON `code` for `BRIEF_INTENT_NOT_APPROVED` in the trace), `no-agent`, `no-plan-file` | Veikart steg 1's gate had no case. No v1 trace reached it: case 4 halts at the sequencing gate first |
|
||||
| **`review-requires-project`**: the arg parser got a CLI that checks `--project` itself and prints `Error: --project <dir> is required.` + usage (exit 1, code `ARG_REQUIRED_MISSING`). `trekreview.md` runs it with `$ARGUMENTS` and relays its stderr. The reply-regex grader `project-required` is replaced by `parser-ran` + `names-missing-project` (the parser's code in the trace, the same form as `names-rule`) | The fix this doc recommended for the red case. Also: the prose passed `"$@"`, which the Bash tool never has. The parser never ran (0 Bash calls in the case-5 traces), so the model composed the line from prose. That failed 2 of 10 on backticks |
|
||||
| `no-error-code` → `code\W{1,12}REVIEW_[A-Z_]+`, bound to the validator's JSON output | The listed codes missed `REVIEW_WRONG_TYPE` and `REVIEW_VERSION_FORMAT` |
|
||||
| `says-pass` / `says-fail` reject a preceding `NOT` and a longer word | Raw substrings: "did not PASS" passed `says-pass` |
|
||||
|
|
@ -265,7 +266,7 @@ From the plugin root:
|
|||
claude plugin eval . --trust-plugin --ablation none --runs 1 --scaffold --no-publish --allow-tools Bash Write
|
||||
```
|
||||
|
||||
- `--scaffold` runs each case's `scaffold.sh` as you, outside the sandbox. The four
|
||||
- `--scaffold` runs each case's `scaffold.sh` as you, outside the sandbox. The three
|
||||
scaffolds in this suite only write fixture files into the run's workspace.
|
||||
- `--allow-tools Bash Write` grants the tools the commands need (Bash runs under the OS
|
||||
sandbox). Write is granted so that the `no-write` and `no-plan-file` graders measure a
|
||||
|
|
@ -291,4 +292,4 @@ will sometimes stop a change that is innocent. That is the intended price. Movin
|
|||
into code removes the instability; editing the grader does not. v2 did the move: the
|
||||
parser now writes the line, and the grader reads the parser's code. One green run does
|
||||
not show the instability is gone. That takes more runs. § Stability has them:
|
||||
`review-requires-project` 5 of 5, and case 4 15 of 15 after its own fix.
|
||||
`review-requires-project` 5 of 5, and case 4 (removed in cleanup R-b) 15 of 15 after its own fix.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue