feat(review): the review is a gate - verdict in the file, CC engine, project memory
Ordre 5 (wave 1, P11 + P5), review_version 1.1. - K7: `verdict: ALLOW | WARN | BLOCK` is required in review.md (strict error, soft warning for 1.0 reviews). New CLI lib/review/verdict-gate.mjs: 0 = ALLOW/WARN, 1 = BLOCK, 2 = could not decide (never green); a verdict greener than the file's own BLOCKERs or JSON block is exit 2. Realpath main guard (the old file:// guard exits 0 under a symlink - measured). - No finding is struck for length: succinctness trims only MINOR/SUGGESTION text (catalogue tier), keeps BLOCKER/MAJOR whole, counts trims; the coordinator lists what it declined to judge. - Claude Code's /code-review and /security-review run as the engine via the Skill tool (never --fix/--comment/--post/ultra); lib/review/engine-findings.mjs turns their findings into a reviewer payload; two catalogue keys (CODE_REVIEW_DEFECT, SECURITY_REVIEW_VULNERABILITY), catalogue 12 -> 14. An engine that did not run forbids ALLOW. - Every BLOCKER is written as a numbered `### Step R<n>` with its source finding (strict error REVIEW_MISSING_REMEDIATION_STEP). - P5: lib/review/recurrence.mjs - a finding class in two review.md files (current + git history, no new store) becomes one proposal, routed by the catalogue's memory_route; written only between Voyage markers in .claude/rules/voyage-review-lessons.md under a size budget, CLAUDE.md gets a pointer only; the operator approves every line (Phase 8.7). - Workflow substrate: catalogue enum in step; a missing reviewer or engine forbids ALLOW there too (it returned ALLOW on zero findings before). - Fixtures review-run-A/B said WARN over a BLOCKER; now BLOCK. Borrowed and credited in code: Superpowers (declined-to-judge), ai-native-sdlc (cap only nits; mistake/fix/check), Spec Kit (managed block; converge steps), Pocock (retro routing), Kiro (verdict file + gate; evidence per lesson), Cherny (engine -> brief control order; review -> memory; budget). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
2c89a6b580
commit
d88245c06e
SSH key fingerprint:
SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
28 changed files with 2052 additions and 110 deletions
|
|
@ -5,7 +5,7 @@ description: |
|
|||
review.md with severity-tagged findings (BLOCKER/MAJOR/MINOR/SUGGESTION)
|
||||
per Handover 6 (review → plan).
|
||||
argument-hint: "--project <dir> [--since <ref>] [--quick] [--validate] [--dry-run]"
|
||||
allowed-tools: Agent, Read, Glob, Grep, Write, Edit, Bash, AskUserQuestion
|
||||
allowed-tools: Agent, Read, Glob, Grep, Write, Edit, Bash, AskUserQuestion, Skill
|
||||
---
|
||||
|
||||
# Ultrareview Local v1.0
|
||||
|
|
@ -29,6 +29,17 @@ The review is **independent**: each reviewer runs without cross-feeding,
|
|||
and the coordinator applies BOUNDED operations only. Synthesis-level
|
||||
inference across files is forbidden in v1.0 (Judge Agent pattern).
|
||||
|
||||
**The review is a gate (review_version 1.1, ordre 5).** In this order:
|
||||
Claude Code's own `/code-review` and `/security-review` are the engine
|
||||
(Phase 4.7); Voyage's control against the brief runs after them and is the
|
||||
part that stops (Phase 5–6); the verdict is written in `review.md` and one
|
||||
command decides on it (`lib/review/verdict-gate.mjs`, Phase 8: 0 = ALLOW/WARN,
|
||||
1 = BLOCK, 2 = could not decide, never green). No finding is struck for
|
||||
length. A finding class seen in two reviews becomes one proposal for the
|
||||
project's memory, approved line by line (Phase 9.5). Order credited to Boris
|
||||
Cherny's public workflow (engine → brief control); the gate to Kiro and Spec
|
||||
Kit — see `docs/HANDOVER-CONTRACTS.md` § Handover 6.
|
||||
|
||||
See `agents/review-orchestrator.md` for the canonical workflow this
|
||||
command executes inline.
|
||||
|
||||
|
|
@ -150,9 +161,12 @@ The combined file list is the review scope. Note that the
|
|||
Assumptions section declares this is allowed; the review surfaces it
|
||||
explicitly in the Coverage table.
|
||||
|
||||
If the file count is `0`, write a one-line review.md noting "No diff
|
||||
between {before_sha} and {after_sha}; nothing to review." Verdict: ALLOW.
|
||||
Skip Phases 4–7. Continue to Phase 8 (validate + stats).
|
||||
If the file count is `0`, write a minimal review.md per the template:
|
||||
`verdict: ALLOW`, `findings: []` (the empty list is the one inline form the
|
||||
parser reads — a bare `findings:` is null and fails validation), and the
|
||||
three required sections, each saying "No diff between {before_sha} and
|
||||
{after_sha}; nothing to review." Skip Phases 4–7. Continue to Phase 8
|
||||
(validate + stats).
|
||||
|
||||
## Phase 4 — Triage gate (deterministic path-pattern classifier)
|
||||
|
||||
|
|
@ -253,6 +267,71 @@ prompt in Phase 5, and its summary line goes into the Coverage section of
|
|||
built by `formatCriteriaEvidence` precisely so the orchestrator cannot narrate a
|
||||
pass that never happened.
|
||||
|
||||
## Phase 4.7 — Claude Code's review engine (`/code-review`, `/security-review`)
|
||||
|
||||
**Skipped in `dry-run` and `validate`.** In `quick` mode only `/code-review`
|
||||
runs. Wrap, not a new engine: Claude Code's built-in reviews find the
|
||||
correctness and security defects; Voyage's brief control (Phase 5) runs after
|
||||
them and is what stops.
|
||||
|
||||
Verified in Claude Code's live documentation (2026-10-06):
|
||||
- Claude may start `/code-review` itself (`code-review.md`: "Claude can start
|
||||
`/code-review` on its own", unconditionally from **v2.1.246**) and
|
||||
`/security-review` (`skills.md`: "A few built-in commands are also available
|
||||
through the Skill tool, including `/init` and `/security-review`").
|
||||
- `/code-review` runs as a background subagent by default and its findings
|
||||
"arrive in your conversation when the review completes". It accepts a target
|
||||
such as a ref range.
|
||||
- `/security-review` "Reviews the diff between your branch and origin's
|
||||
default branch" and "Needs an `origin` remote". The docs do not state its
|
||||
output format or whether it runs in the background.
|
||||
|
||||
Run both through the **Skill tool**, one after the other:
|
||||
|
||||
1. `Skill("code-review", args: "high {before_sha}...{after_sha}")` — the
|
||||
review's own SHA range from Phase 3. `high` is set explicitly; note in the
|
||||
Executive Summary that this also becomes the level later `/code-review`
|
||||
runs in this session reuse.
|
||||
2. `Skill("security-review")` — no arguments; its base is fixed.
|
||||
|
||||
**Never** pass `--fix`, `--comment`, `--post` or `ultra`: a review never edits
|
||||
code, never posts anywhere, and never uploads the repo to a cloud review.
|
||||
|
||||
**Wait for the findings.** A background `/code-review` delivers its findings
|
||||
later in this conversation. Do not start Phase 6 before both engines have
|
||||
delivered or have been recorded as not run. For each engine that delivered,
|
||||
transcribe every finding — file, line (0 when none), the one-sentence summary,
|
||||
the failure scenario, the category if given, and a narrower catalogue
|
||||
`rule_key` only when one plainly fits — into
|
||||
`{project_dir}/.review-engine-{code-review|security-review}.json` as
|
||||
`{"engine": "...", "findings": [...]}`. Transcribe; never re-judge, merge or
|
||||
drop an engine finding on the way. Then convert it:
|
||||
|
||||
```bash
|
||||
git diff --name-only "$before_sha".."$after_sha" > "{project_dir}/.review-scope.txt"
|
||||
git diff --name-only HEAD >> "{project_dir}/.review-scope.txt"
|
||||
node ${CLAUDE_PLUGIN_ROOT}/lib/review/engine-findings.mjs \
|
||||
--scope "{project_dir}/.review-scope.txt" "{project_dir}/.review-engine-code-review.json"
|
||||
```
|
||||
|
||||
Exit 0 prints a reviewer payload (`claude-code-review` /
|
||||
`claude-security-review`) for Phase 6 plus `out_of_scope`: engine findings on
|
||||
files outside the review's SHA range (`/security-review` diffs against
|
||||
origin's default branch, which can be wider). Those go in the Coverage
|
||||
section, one line each — never silently dropped. Exit 1 = the transcription is
|
||||
malformed: fix it from the engine's own text and re-run. Exit 2 = could not
|
||||
run.
|
||||
|
||||
**An engine that did not run** (Claude Code older than v2.1.246, the Skill
|
||||
call refused, no `origin` remote, an `ambiguous argument` error, or no findings
|
||||
ever delivered) is recorded as not run with its reason. It stays in the
|
||||
expected reviewer set, so it forbids `ALLOW` (fail-closed); the Executive
|
||||
Summary's first sentence names it. An engine that ran and found nothing
|
||||
delivers an empty payload — that is a report, not an absence.
|
||||
|
||||
The two engine payloads are NOT shown to the Phase 5 reviewers: reviewers stay
|
||||
independent, and the coordinator is the only place the outputs meet.
|
||||
|
||||
## Phase 5 — Launch parallel reviewers
|
||||
|
||||
**Substrate routing (opt-in `--workflow`).** When `workflow_substrate == true`, run
|
||||
|
|
@ -322,8 +401,11 @@ Summary will note the brief-conformance pass was skipped.
|
|||
### Reviewer accounting — every expected reviewer MUST report
|
||||
|
||||
Write down the expected reviewer set BEFORE the spawn: both reviewers in
|
||||
default mode, `code-correctness-reviewer` alone in `quick` mode. After the
|
||||
spawn, account for each one by name.
|
||||
default mode, `code-correctness-reviewer` alone in `quick` mode — plus, in
|
||||
every mode, the Phase 4.7 engines that were due (`claude-code-review`, and
|
||||
`claude-security-review` outside `quick`). After the spawn, account for each
|
||||
one by name. An engine recorded as not run is passed on as missing, never
|
||||
left out of the set.
|
||||
|
||||
**Zero findings from a silent reviewer is indistinguishable from zero findings
|
||||
from a clean diff** — unless you check. A reviewer is *accounted for* only when
|
||||
|
|
@ -353,13 +435,18 @@ forbids `ALLOW` (`lib/review/coordinator-contract.mjs`, `missing_reviewers`).
|
|||
## Phase 6 — Coordinator dedup + verdict
|
||||
|
||||
Launch `review-coordinator` (Agent tool) with the merged findings array
|
||||
from Phase 5 plus the triage map, brief metadata, and SHA range.
|
||||
from Phase 5 AND the Phase 4.7 engine payloads, plus the triage map, brief
|
||||
metadata, SHA range, the expected reviewer set (engines included), and each
|
||||
not-run engine with its reason.
|
||||
|
||||
The coordinator runs the 4-pass process documented in
|
||||
`agents/review-coordinator.md`:
|
||||
|
||||
1. **Dedup** by `(file, line, rule_key)` triplet.
|
||||
2. **HubSpot Judge filters** — Succinctness, Accuracy, Actionability.
|
||||
Succinctness never removes a finding: only MINOR/SUGGESTION text is
|
||||
trimmed, and every trim is counted. A long BLOCKER or MAJOR keeps every
|
||||
character.
|
||||
3. **Cloudflare reasonableness** — remove speculative or catalogue-violating
|
||||
findings (skipped in `quick` mode).
|
||||
4. **Verdict** — BLOCK / WARN / ALLOW per the threshold table.
|
||||
|
|
@ -374,9 +461,10 @@ vocabulary, and the `allow_blocked_by` field: `agents/review-coordinator.md`
|
|||
§*Suppression is two-valued*, mirrored deterministically in
|
||||
`lib/review/coordinator-contract.mjs`.
|
||||
|
||||
The coordinator's output is the full review.md content — frontmatter +
|
||||
body sections + trailing JSON block. Do NOT re-run the reviewers based
|
||||
on the coordinator's output.
|
||||
The coordinator's output is the full review.md content — frontmatter
|
||||
(with the REQUIRED `verdict:`) + body sections (including `## Declined to
|
||||
Judge` and, when there is a BLOCKER, `## Remediation Steps`) + trailing JSON
|
||||
block. Do NOT re-run the reviewers based on the coordinator's output.
|
||||
|
||||
## Phase 5–6 via the Workflow substrate (opt-in `--workflow`)
|
||||
|
||||
|
|
@ -398,11 +486,16 @@ Workflow({
|
|||
briefPath: "{brief_path}",
|
||||
diffPath: "{path to the unified diff file from Phase 3}",
|
||||
triage: "{triage map as 'path → treatment' lines from Phase 4}",
|
||||
quick: {true if mode == quick, else false}
|
||||
quick: {true if mode == quick, else false},
|
||||
enginePayloads: [{the Phase 4.7 payloads from engine-findings.mjs}],
|
||||
engineNotRun: [{each engine reviewer name recorded as not run}]
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
A reviewer that returns nothing, or an engine listed in `engineNotRun`,
|
||||
forbids `ALLOW` on this path too (`missing_reviewers` in the result).
|
||||
|
||||
Contract (verified in S10 part B — follow exactly):
|
||||
- **Pass `args` as a JSON object.** The script defensively re-parses a JSON *string*, but
|
||||
the object form is the contract.
|
||||
|
|
@ -439,6 +532,18 @@ write to a temp file, then rename. The frontmatter `findings:` field
|
|||
must use **block-style YAML** (one ID per line, ` - ` prefix). The
|
||||
parser at `lib/util/frontmatter.mjs` does not support flow-style arrays.
|
||||
|
||||
**The verdict is in the file (K7).** The frontmatter carries
|
||||
`verdict: ALLOW | WARN | BLOCK`, equal to the trailing JSON block's
|
||||
`verdict`. It is required at write time; a review without it fails Phase 8.
|
||||
|
||||
**A BLOCKER is written as a plan step.** For every BLOCKER the body carries,
|
||||
under `## Remediation Steps`, one numbered step `### Step R<n>: <imperative
|
||||
action>` with `**Files:**`, `**Source finding:** <40-char id>`, `**Test
|
||||
first:**` (a test that reproduces the finding and fails today), `**Verify:**`
|
||||
(the command that proves it fixed; default: re-run `/trekreview` and the
|
||||
finding is absent) and `**On failure:** escalate`. Handover 6 can append these
|
||||
to a plan without rewording them; earlier steps are never rewritten.
|
||||
|
||||
If `mode == dry-run`: skip the write; print the would-be path and the
|
||||
first 60 lines of the rendered output.
|
||||
|
||||
|
|
@ -455,10 +560,27 @@ If validation fails:
|
|||
finding-ID, REVIEW_VERSION_FORMAT warning): repair in place — re-emit
|
||||
the missing section, recompute the finding-ID, fix the version
|
||||
string. Re-validate.
|
||||
- `REVIEW_MISSING_VERDICT` / `REVIEW_MISSING_REMEDIATION_STEP`: repair
|
||||
from the coordinator's own output — copy the JSON block's `verdict` into the
|
||||
frontmatter; write the missing `### Step R<n>` from that BLOCKER's
|
||||
`recommended_action`, `file` and `id`. Never invent a verdict the JSON block
|
||||
does not carry. Re-validate.
|
||||
- For unrepairable errors (REVIEW_WRONG_TYPE, malformed frontmatter):
|
||||
stop and ask the user to re-run; do not silently produce an invalid
|
||||
review.md.
|
||||
|
||||
**Decide on the verdict with the gate (K7).** Once the validator passes:
|
||||
|
||||
```bash
|
||||
node ${CLAUDE_PLUGIN_ROOT}/lib/review/verdict-gate.mjs "{review_path}"
|
||||
```
|
||||
|
||||
Exit 0 = ALLOW or WARN; exit 1 = BLOCK; exit 2 = the gate could not decide
|
||||
(missing verdict, a verdict the file contradicts, unreadable file) — **never
|
||||
green**. Capture the exit code as `gate_exit` and print it in Phase 9 exactly
|
||||
as returned. This is the command that later stops a merge or a release on
|
||||
BLOCK; `/trekreview` reports it and never overrides it.
|
||||
|
||||
Append a stats line to `${CLAUDE_PLUGIN_DATA}/trekreview-stats.jsonl`
|
||||
(create the file if it does not exist):
|
||||
|
||||
|
|
@ -488,8 +610,65 @@ HTML is a convenience, not a gate.
|
|||
When `mode == validate`:
|
||||
1. Skip Phases 3–7 entirely.
|
||||
2. Run the strict validator on `{project_dir}/review.md`.
|
||||
3. Print a one-line PASS/FAIL summary plus the JSON output on FAIL.
|
||||
4. Exit 0 on PASS, 1 on FAIL. Never write to disk. Never call any agent.
|
||||
3. Print a one-line PASS/FAIL summary plus the JSON output on FAIL. A
|
||||
`review_version` 1.0 review without `verdict:` FAILS here (K7 is a
|
||||
write-time rule); it still seeds a plan, because `/trekplan` reads in soft
|
||||
mode.
|
||||
4. Run `lib/review/verdict-gate.mjs` on the same file and print its line and
|
||||
exit code.
|
||||
5. Exit 0 on PASS, 1 on FAIL. Never write to disk. Never call any agent.
|
||||
Skip Phase 8.7.
|
||||
|
||||
## Phase 8.7 — The project's memory (recurring finding classes)
|
||||
|
||||
**Skipped in `dry-run` and `validate`.** A finding class (the same catalogue
|
||||
`rule_key`) that shows up in two review.md files in this repo becomes ONE
|
||||
proposal; one sighting is no proposal. The history is the review.md files the
|
||||
repo already has (every `.claude/projects/*/review.md`, plus earlier committed
|
||||
versions of them) — no new store.
|
||||
|
||||
```bash
|
||||
node ${CLAUDE_PLUGIN_ROOT}/lib/review/recurrence.mjs propose --repo "$(git rev-parse --show-toplevel)"
|
||||
```
|
||||
|
||||
It prints `{"proposals": [...]}` (exit 0) or exits 2 (could not run — say so
|
||||
in the summary; never read 2 as "nothing recurred"). Each proposal names its
|
||||
`rule_key`, its route and its evidence (the finding IDs from two reviews), and
|
||||
carries one line in the format **mistake · fix · how to check**:
|
||||
- route `check` (mechanical) — the line names the deterministic check that
|
||||
catches the class;
|
||||
- route `rule` (judgement) — the line is for the repo's `.claude/rules/`
|
||||
file, which Claude Code loads at session start;
|
||||
- review-hygiene classes are never proposed.
|
||||
|
||||
**The operator approves every line.** For each proposal, use
|
||||
`AskUserQuestion` (up to 4 proposals per call), in plain language, showing
|
||||
the line and its two pieces of evidence. Options: **"Skriv linja inn"**
|
||||
(recommended: the class has recurred) and **"Ikke nå"**; the operator may
|
||||
answer with an edited line through "Other". Never write a line the operator
|
||||
did not approve, and never treat "sounds good" or "do as you like" as a yes
|
||||
to a line it was not shown.
|
||||
|
||||
On yes:
|
||||
|
||||
```bash
|
||||
node ${CLAUDE_PLUGIN_ROOT}/lib/review/recurrence.mjs apply --repo "$(git rev-parse --show-toplevel)" \
|
||||
--rule-key <RULE_KEY> [--line "<the operator's edited line>"]
|
||||
```
|
||||
|
||||
It writes ONLY between Voyage's own markers in
|
||||
`.claude/rules/voyage-review-lessons.md` and, if the repo has a `CLAUDE.md`,
|
||||
gives it a one-line pointer once (never the lesson itself). Exit 1 with
|
||||
`over-budget` means the managed block is full: tell the operator the lessons
|
||||
belong in a skill now — that is the operator's decision, not this command's.
|
||||
Exit 1 with `invalid-line` means an edited line lacks the three parts: show
|
||||
why and ask again. Report every outcome in the summary; when no class
|
||||
recurred, say "no recurring finding class" — do not pad.
|
||||
|
||||
Borrowed: routing (Pocock `retro`), the three-part line (ai-native-sdlc),
|
||||
the managed block (Spec Kit `agent-context`), evidence per proposal and
|
||||
schema-check before write (Kiro), review → memory and the size budget
|
||||
(Cherny). Credits in `lib/review/recurrence.mjs`.
|
||||
|
||||
## Phase 9 — Present summary
|
||||
|
||||
|
|
@ -506,12 +685,20 @@ After the write succeeds, print:
|
|||
**Annotation HTML:** file://{$ANNOT_HTML}
|
||||
**Scope:** {before_sha}..{after_sha} ({reviewed_files_count} files)
|
||||
**Verdict:** {BLOCK | WARN | ALLOW}
|
||||
**Gate:** verdict-gate exit {gate_exit} ({0 = ALLOW/WARN | 1 = BLOCK | 2 = could not decide — not green})
|
||||
**Engine:** /code-review {delivered N | not run: reason} · /security-review {delivered N | not run: reason}
|
||||
|
||||
### Counts
|
||||
- BLOCKER: {N}
|
||||
- MAJOR: {N}
|
||||
- MINOR: {N}
|
||||
- SUGGESTION: {N}
|
||||
- Trimmed (MINOR/SUGGESTION text only, never removed): {N}
|
||||
- Declined to judge: {N} — {one line each: file:line rule_key — reason}
|
||||
|
||||
### Project memory
|
||||
{each proposal: rule_key → written | declined by operator | over-budget | invalid-line;
|
||||
or "no recurring finding class"; or "recurrence check could not run"}
|
||||
|
||||
### Top findings
|
||||
- [{severity}] {title} ({file}:{line})
|
||||
|
|
@ -613,7 +800,7 @@ unknown `rule_key` values from polluting downstream remediation plans
|
|||
(Handover 6), the coordinator applies its v5.1.1 high-effort
|
||||
normalization rule — substituting unknown `rule_key` values with the
|
||||
literal string `PLAN_EXECUTE_DRIFT` (the most general drift category
|
||||
in the 12-entry catalogue) and preserving the original in
|
||||
in the catalogue) and preserving the original in
|
||||
`original_rule_key`. See `agents/review-coordinator.md` § Pass 3
|
||||
"High-effort normalization (v5.1.1)" for the full normalization spec.
|
||||
|
||||
|
|
@ -652,4 +839,15 @@ Low effort: skip the brief-conformance reviewer entirely (existing
|
|||
- **Honesty.** If the diff is trivially small or all-skip, say so. Do
|
||||
not pad findings to make the review look thorough.
|
||||
- **No production code.** This command never runs production code, never
|
||||
writes to anything outside `{project_dir}` and `${CLAUDE_PLUGIN_DATA}`.
|
||||
writes to anything outside `{project_dir}` and `${CLAUDE_PLUGIN_DATA}` —
|
||||
with one exception, only after the operator approved that exact line in
|
||||
Phase 8.7: Voyage's own marked block in `.claude/rules/voyage-review-lessons.md`
|
||||
and a one-line pointer in `CLAUDE.md`.
|
||||
- **The engine never edits code.** `/code-review` and `/security-review` run
|
||||
without `--fix`, `--comment`, `--post` or `ultra`. A review that changes
|
||||
code is not a review.
|
||||
- **No finding is struck for length.** Only MINOR/SUGGESTION text is trimmed,
|
||||
and every trim is counted; what the coordinator set aside is listed under
|
||||
`## Declined to Judge`.
|
||||
- **The verdict is in the file, and a command decides on it.** `verdict:` is
|
||||
required (K7); `lib/review/verdict-gate.mjs` exit 2 is never green.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue