open/voyage
1
0
Fork 0
Code Issues Releases 13 Activity

feat(review): the security engine chosen and Claude Security converted (E 3.1)

Browse source 
lib/review/security-engine.mjs (decisions 175, 196, 219):
- choose: Claude Security when available and base..head has committed
  changes; else /security-review when origin/HEAD resolves and
  origin/HEAD...HEAD is non-empty; else no engine (no-engine, no-origin,
  empty-diff), which is "not measured", never 0. A git failure or bad
  usage is exit 2, never a reason.
- convert: reads the newest CLAUDE-SECURITY-<ts>/ report whose stamp is a
  changes scan of exactly base..head; confidence high becomes engine
  findings through toReviewerPayload, the rest go to below_threshold and
  never into findings. An unverified scan, missing researcher counts,
  fewer researchers returned than dispatched, an empty diff or scope, or
  a malformed record is exit 2.
- NO_ENGINE_REVIEWER: with no engine chosen the expected reviewer
  security-engine never reports, so the verdict gate exits 2.

ENGINES gains claude-security (its own reviewer and brief_ref, the
engines' key) with a per-engine label for the written action; it is a
security engine in isSecurityFinding, RAN_ENGINES, the coordinator
contract (out-of-scope and brief-conflict rules) and the coordinator
prose; engine:below-threshold is an unverified reason fed like
engineOutOfScope. Pins of the changed coordinator prose follow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-11-08 13:29:59 +01:00
commit cb7c13b408
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
18 changed files with 1045 additions and 44 deletions
Download patch file Download diff file Expand all files Collapse all files

45
agents/review-coordinator.md
View file

@ -32,18 +32,21 @@ This file holds only your own role: the bounded passes and the verdict.
You will receive a prompt containing:
- **Reviewer outputs** — JSON-block payloads from
`brief-conformance-reviewer` and `code-correctness-reviewer` (in `quick`
mode, only the latter), plus the engine payloads from Claude Code's own
review: `claude-code-review` (`/code-review`) and `claude-security-review`
(`/security-review`, not in `quick`). Engine findings carry the catalogue
keys `CODE_REVIEW_DEFECT` / `CODE_REVIEW_IMPROVEMENT` (MINOR: simplification,
efficiency, reuse, style, readability) / `SECURITY_REVIEW_VULNERABILITY` (or a
narrower key) and are treated exactly like any reviewer's findings.
mode, only the latter), plus the engine payloads: `claude-code-review`
(Claude Code's `/code-review`) and, not in `quick`, the one security engine
`/trekreview` chose: `claude-security` (Claude Security) or
`claude-security-review` (Claude Code's `/security-review`), or none. Engine
findings carry the catalogue keys `CODE_REVIEW_DEFECT` /
`CODE_REVIEW_IMPROVEMENT` (MINOR: simplification, efficiency, reuse, style,
readability) / `SECURITY_REVIEW_VULNERABILITY` (or a narrower key) and are
treated exactly like any reviewer's findings.
- **Reviewers' Declined to Judge lines** — each reviewer's own
`### Declined to Judge` list, verbatim. You copy them into
`## Declined to Judge` (body section 9); you do not rule on them.
- **Expected reviewers** — the names that were due, engines included, and
each engine recorded as not run with its reason. A name that did not report
forbids `ALLOW` (Pass 4).
forbids `ALLOW` (Pass 4). When no security engine could run, the set holds
`security-engine`, which never reports: security is not measured, never 0.
- **Engine findings outside the review scope** — the `out_of_scope` list from
`lib/review/engine-findings.mjs`. List each one under `## Declined to Judge`
(`engine:out-of-scope`): a real claim on a file this review did not cover.
@ -51,6 +54,12 @@ You will receive a prompt containing:
`/code-review` (given exactly this range): that one is declined but does not
hold `ALLOW` back. A MAJOR one still does: a change that breaks an
unchanged caller is cited at the caller.
- **Claude Security findings below its threshold** — the `below_threshold`
list of the `claude-security` payload (`lib/review/security-engine.mjs`):
findings Claude Security marked below confidence `high` (decision 219). List
each one under `## Declined to Judge` (`engine:below-threshold`) and in the
JSON `declined_to_judge`. It is `unverified`: not counted, never in
`findings`, and it forbids `ALLOW`.
- **Triage map** — `{file → deep-review|summary-only|skip, reason}` from
the /trekreview triage gate.
- **Brief metadata** — `task`, `slug`, `project_dir`, `brief_path` from
@ -93,8 +102,8 @@ collide:
not in the catalogue (BLOCKER > MAJOR > MINOR > SUGGESTION);
2. if the catalogue tier ties (for a catalogue key it always does,
since the triplet shares the `rule_key`), prefer a reviewer-agent
finding over an engine finding (one from `claude-code-review` /
`claude-security-review`);
finding over an engine finding (one from `claude-code-review`,
`claude-security-review` or `claude-security`);
3. among findings of one kind, the higher asserted severity;
4. among reviewer-agent findings, prefer the finding from
`brief-conformance-reviewer` (its findings are anchored to the brief);
@ -125,9 +134,10 @@ is load-bearing** — see *Suppression is two-valued* below:
| Accuracy | `file` resolves under the repo root AND `line` ≥ 0 | Path traversal escape or negative line | **dropped** (`accuracy:refuted`). A line past the end of the file (when its line count is known) is not refuted: the citation is wrong, the claim is unchecked → `unverified` (`accuracy:line-out-of-range`) |
| Actionability | `recommended_action` is non-empty AND begins with an imperative verb | Empty action, "consider …" hedges, or restating the title | `unverified` (`actionability:empty`) |
A finding that carries `engine` (`code-review` / `security-review`) came
from Claude Code's own review through `lib/review/engine-findings.mjs`, which
writes its action from the engine's failure scenario; it is
A finding that carries `engine` (`code-review` / `security-review` /
`claude-security`) came from an engine through `lib/review/engine-findings.mjs`
(Claude Security's through `lib/review/security-engine.mjs`), which writes its
action from the engine's fix or failure scenario; it is
never set aside for Actionability. (Before this rule the converter wrote
`Resolve: <title>`, and 8 of 8 engine findings were set aside in the
voyage-lekerepo trial, 2026-10-06.)
@ -177,16 +187,16 @@ Remove findings that fail ANY of these tests:
catalogue tier; this is a correction, neither a drop nor an unverified.
**The brief-conflict rule (one fixed rule).** Read the **Brief choices** input.
An engine finding (`engine: code-review` or `security-review`) whose recommended action
An engine finding (`engine: code-review`, `security-review` or `claude-security`) whose recommended action
contradicts a passed Non-Goal or stated choice gets `contradicts_brief: "<verbatim>"`
(that Non-Goal or choice, word for word) and is kept, never removed;
only a `/code-review` finding at the MAJOR tier and at severity MAJOR then drops to MINOR, with `brief_lowered_from: MAJOR`
(in place of the catalogue reset above);
a BLOCKER-tier, a MINOR-tier and any `/security-review` finding only get the mark,
a BLOCKER-tier, a MINOR-tier and any finding from a security engine (`/security-review`, Claude Security) only get the mark,
and no engine finding is otherwise written below its catalogue tier.
The rule reads engine findings alone, so only an engine finding is ever marked:
`engine` is written only on a finding whose kept Pass 1 survivor itself came from an engine payload
(`claude-code-review` → `code-review`, `claude-security-review` → `security-review`),
(`claude-code-review` → `code-review`, `claude-security-review` → `security-review`, `claude-security` → `claude-security`),
and a reviewer-agent finding (no `engine`) and a finding whose `rule_key` is not in the catalogue are never marked.
Remove any `brief_lowered_from` an incoming finding carries, and any mark on a finding
without an `engine`: only this rule writes them.
@ -330,7 +340,8 @@ You never write `ran`, `finalized`, `round`, `round_base`, `resolved` or
withheld `ALLOW`.
9. `## Declined to Judge` — REQUIRED. One line per finding you set aside
without ruling on it (every `[unverified]` one, and every
out-of-scope engine finding: `file:line rule_key — reason`), then every
out-of-scope engine finding and Claude Security finding below its
threshold: `file:line rule_key — reason`), then every
line a reviewer wrote under its own `### Declined to Judge`, copied as
`{reviewer}: {line}` (they are not findings: they stay out of the JSON
`declined_to_judge` array and do not change the verdict), then one line `Trimmed: N` (MINOR/SUGGESTION text cut, findings
@ -363,7 +374,7 @@ followed by these fields:
- `- title: {short imperative title}`
- `- detail: {what is wrong, with citation}`
- `- recommended_action: {one imperative step}`
- `- engine: {code-review | security-review, when the finding came from an engine}`
- `- engine: {code-review | security-review | claude-security, when the finding came from an engine}`
- `- contradicts_brief: {verbatim Non-Goal or choice, only when set}`
- `- brief_lowered_from: {MAJOR, only when lowered for the brief}`

27
lib/review/coordinator-contract.mjs
View file

@ -99,11 +99,15 @@ export const NOTED_REASONS = Object.freeze(new Set([
* module deliberately excludes), and `accuracy:line-out-of-range` by its
* Pass 2 Accuracy (a line past the end of the file: the citation is wrong, the
* claim is unchecked, so a line number alone never refutes); the vocabulary is
* owned here so prose and lib cannot drift.
* owned here so prose and lib cannot drift. `engine:below-threshold` (E 3.1,
* decision 219) is a Claude Security finding below confidence `high`
* (lib/review/security-engine.mjs `below_threshold`): listed, not counted, and
* it keeps the verdict from ALLOW.
*/
export const UNVERIFIED_REASONS = Object.freeze([
'actionability:empty',
'engine:out-of-scope',
'engine:below-threshold',
'unknown-rule_key',
'file-existence:indeterminate',
'accuracy:line-out-of-range',
@ -117,10 +121,11 @@ export const UNVERIFIED_REASONS = Object.freeze([
//
// The one place that decides when a finding marked as contradicting the brief
// (`contradicts_brief: "<the Non-Goal or stated choice, verbatim>"`) may drop
// to MINOR. Only an engine finding (`engine` `code-review` or `security-review`)
// is ever marked; only a `/code-review` finding at the MAJOR tier ever drops,
// to MINOR, writing `brief_lowered_from: 'MAJOR'`. A BLOCKER and any
// `/security-review` finding keep their severity and only carry the mark.
// to MINOR. Only an engine finding (`engine` `code-review`, `security-review`
// or `claude-security`) is ever marked; only a `/code-review` finding at the
// MAJOR tier ever drops, to MINOR, writing `brief_lowered_from: 'MAJOR'`. A
// BLOCKER and any finding from a security engine (`/security-review`, Claude
// Security) keep their severity and only carry the mark.
// The verdict gate (verdict-gate.mjs) imports `illegalBriefLowering`.
// `original_severity` (what the kept finding
// itself asserted before Pass 1 raised it, review 08deae68) is never read here.
@ -632,7 +637,7 @@ export function computeVerdict(findings, options = {}) {
* forbids ALLOW. Do not iterate both and count twice.
*
* @param {Array<{reviewer?: string, findings: object[]}>} reviewerPayloads
* @param {{ expectedReviewers?: string[], engineOutOfScope?: object[] }} [options]
* @param {{ expectedReviewers?: string[], engineOutOfScope?: object[], engineBelowThreshold?: object[] }} [options]
* `declined_to_judge` lists every unverified finding with its reason, and
* `trimmed_count` counts the MINOR/SUGGESTION findings whose text was trimmed
* (kept, judged, and counted — never removed for length).
@ -653,7 +658,8 @@ export function runContract(reviewerPayloads, options = {}) {
// Engine findings on files outside the review's SHA range (engine-findings.mjs
// `out_of_scope`). /security-review diffs against origin's default branch,
// wider than this review: a real claim nobody here judged, so it forbids
// ALLOW. /code-review was given exactly this range: its MINOR or SUGGESTION
// ALLOW, and so does one from Claude Security, the other security engine.
// /code-review was given exactly this range: its MINOR or SUGGESTION
// finding outside it (an improvement to code the review did not change) is
// declined and listed, never dropped, but does not hold ALLOW back
// (voyage-lekerepo trial 2, 2026-10-08). A MAJOR+ one still forbids ALLOW:
@ -664,7 +670,12 @@ export function runContract(reviewerPayloads, options = {}) {
const minorOnly = (f) => f.engine === 'code-review' && (f.severity === 'MINOR' || f.severity === 'SUGGESTION');
const outOfScopeBlocking = outOfScope.filter((f) => !minorOnly(f));
const outOfScopeNoted = outOfScope.filter(minorOnly);
const unverified = [...judgedUnverified, ...reasoned.unverified, ...outOfScopeBlocking];
// Claude Security's findings below confidence `high` (security-engine.mjs
// `below_threshold`, decision 219): never counted, listed under Declined to
// Judge, and they keep the verdict from ALLOW, as an out-of-scope one does.
const belowThreshold = (options.engineBelowThreshold ?? [])
.map((f) => ({ ...f, suppressed_reason: 'engine:below-threshold' }));
const unverified = [...judgedUnverified, ...reasoned.unverified, ...outOfScopeBlocking, ...belowThreshold];
// What the coordinator set aside without ruling on it, one line each, so a
// reader sees every finding that did not reach a verdict tier. Borrowed:
// "Declined to judge … nothing you set aside is dropped silently"

6
lib/review/engine-compare.mjs
View file

@ -15,7 +15,11 @@
//
// Only a payload's `findings` are compared. An engine finding in
// `out_of_scope` stays declined by the existing rule
// (lib/review/coordinator-contract.mjs runContract, `engine:out-of-scope`).
// (lib/review/coordinator-contract.mjs runContract, `engine:out-of-scope`),
// and so does a Claude Security finding in `below_threshold` (confidence
// below `high`, lib/review/security-engine.mjs; `engine:below-threshold`,
// decision 219): neither is ever put back. Claude Security's payload
// (`reviewer` `claude-security`) maps back through ENGINES like the others.
import { ENGINES } from './engine-findings.mjs';
import { JUDGE_TITLE_MAX, trimText } from './coordinator-contract.mjs';

35
lib/review/engine-findings.mjs
View file

@ -9,9 +9,12 @@
// orchestrator transcribes them into a small JSON list; THIS module turns that
// list into a reviewer payload the coordinator already understands, so the
// engine's findings meet Voyage's brief control in the same dedup + verdict.
// The third engine, Claude Security (E 3.1), writes its own report:
// lib/review/security-engine.mjs `convert` reads it and calls this module, so
// its findings are never transcribed by hand.
//
// Input (one engine per call):
// { "engine": "code-review" | "security-review",
// { "engine": "code-review" | "security-review" | "claude-security",
// "findings": [ { "file": "...", "line": 12, "summary": "...",
// "failure_scenario": "...", "category": "correctness",
// "fix": "<optional: the fix the engine itself proposed>",
@ -20,7 +23,8 @@
// Output: { reviewer, findings: [...catalogue-conformant...], out_of_scope: [...] }
// - rule_key: the given one when it is in the catalogue AND at least as
// severe as the engine's own key, else CODE_REVIEW_DEFECT (MAJOR) /
// SECURITY_REVIEW_VULNERABILITY (BLOCKER). A "narrower" key can never
// SECURITY_REVIEW_VULNERABILITY (BLOCKER, both security engines: `security`
// in ENGINES, SECURITY_ENGINES). A "narrower" key can never
// lower an engine finding's tier. One exception, decided by the engine's
// own category and not by the transcriber: a /code-review finding in an
// IMPROVEMENT_CATEGORIES category is CODE_REVIEW_IMPROVEMENT (MINOR), since
@ -28,8 +32,9 @@
// MAJOR defect (voyage-lekerepo trial 2, 2026-10-08).
// - recommended_action: the engine's own `fix` when the transcription
// carries one; otherwise a test-first step built from the failure
// scenario. Never the scenario dressed as a remedy (trial 2: /trekplan
// would have taken failure scenarios as step goals).
// scenario, or, without one, naming the engine by its own `label`. Never
// the scenario dressed as a remedy (trial 2: /trekplan would have taken
// failure scenarios as step goals).
// - File paths are made repo-relative (an absolute path under `repoRoot`
// loses that prefix) before they are compared with the scope.
// - A finding on a file outside the review scope is NOT dropped silently: it
@ -37,7 +42,8 @@
// Judge (`engine:out-of-scope`). It forbids ALLOW unless it is a MINOR or
// SUGGESTION from /code-review (coordinator-contract.mjs runContract);
// `/security-review` diffs against origin's default branch, which can be
// wider than the review's SHA range.
// wider than the review's SHA range. Claude Security scans exactly that
// range; one of its findings outside the scope forbids ALLOW all the same.
// - An empty scope list is an error, never "everything is out of scope".
//
// CLI: node lib/review/engine-findings.mjs [--scope <file-list.txt>] [--repo-root <dir>] <engine-findings.json>
@ -49,19 +55,36 @@ import { RULE_KEYS, getRule, SEVERITY_VALUES } from './rule-catalogue.mjs';
import { validateFindings } from './findings-schema.mjs';
import { isMain } from '../util/is-main.mjs';
// `label` names the engine in a written action; `security` marks a security
// engine (a finding from it is a security finding, review-rounds.mjs
// `isSecurityFinding`).
export const ENGINES = Object.freeze({
'code-review': Object.freeze({
reviewer: 'claude-code-review',
rule_key: 'CODE_REVIEW_DEFECT',
brief_ref: 'NFR — code correctness (Claude Code /code-review)',
label: '/code-review',
security: false,
}),
'security-review': Object.freeze({
reviewer: 'claude-security-review',
rule_key: 'SECURITY_REVIEW_VULNERABILITY',
brief_ref: 'NFR — security (Claude Code /security-review)',
label: '/security-review',
security: true,
}),
'claude-security': Object.freeze({
reviewer: 'claude-security',
rule_key: 'SECURITY_REVIEW_VULNERABILITY',
brief_ref: 'NFR — security (Claude Security)',
label: 'Claude Security',
security: true,
}),
});
/** The security engines: `/security-review` and Claude Security. */
export const SECURITY_ENGINES = Object.freeze(Object.keys(ENGINES).filter((e) => ENGINES[e].security));
function normPath(p, repoRoot) {
let out = String(p ?? '').trim().replace(/\\/g, '/');
if (repoRoot) {
@ -132,7 +155,7 @@ export function toReviewerPayload(input, opts = {}) {
recommended_action: fix
|| (scenario
? `Write a test that reproduces this at ${loc} (${scenario}), then change ${loc} until it passes`
: `Write a test that shows the defect at ${loc} that ${input.engine === 'security-review' ? '/security-review' : '/code-review'} reported, then change ${loc} until it passes`),
: `Write a test that shows the defect at ${loc} that ${spec.label} reported, then change ${loc} until it passes`),
engine: input.engine,
};
if (scope && !scope.has(file)) out_of_scope.push(out);

3
lib/review/review-answers.mjs
View file

@ -9,7 +9,8 @@
// - its `## Answers` row says `accepted` with a reason that is not hollow
// (lib/util/approval-log.mjs `isHollowAnswer`);
// - it is not a security finding (a `security` catalogue key, or a finding
// from the `security-review` engine): those are never accepted;
// from a security engine, `security-review` or `claude-security`): those
// are never accepted;
// - the latest K2 record of kind `review` with subject
// `<review path relative to the git top level>#<id>` carries
// `answerSha(...)` over the review's CURRENT trailing JSON block and

3
lib/review/review-finalize.mjs
View file

@ -16,7 +16,8 @@
// `**Source:** engine {name}, put back by the standard-path comparison`,
// the frontmatter list, counts, a verdict that is only raised, and, as a
// BLOCKER, its `### Step R<n>`. An engine finding in `out_of_scope` is
// not compared (it stays declined, `engine:out-of-scope`).
// not compared (it stays declined, `engine:out-of-scope`), nor is one in
// Claude Security's `below_threshold` (`engine:below-threshold`, E 3.1).
// - outside, right after engine (E 2.5, decisions 164, 218): the trailing
// JSON block's optional `known_outside` (Voyage's own security findings
// the coordinator listed as outside the brief's threat model) is checked

6
lib/review/review-rounds.mjs
View file

@ -84,6 +84,7 @@ import { contentOverlap, DEFAULT_THRESHOLD } from './plan-review-dedup.mjs';
import { FULL_SHA_RE, reviewedSha } from './reviewed-commit.mjs';
import { preExistingCheck, scopeShas } from './pre-existing.mjs';
import { getRule } from './rule-catalogue.mjs';
import { SECURITY_ENGINES } from './engine-findings.mjs';
import { decideStop, describeStop, roundSummary } from './stop-rule.mjs';
import { isMain } from '../util/is-main.mjs';
@ -329,11 +330,12 @@ function bestMatch(p, current, taken) {
/**
* A security finding: its catalogue category is `security`, or it came from
* the `security-review` engine. Never accepted, never left out as pre-existing.
* a security engine (`security-review`, `claude-security`: engine-findings.mjs
* SECURITY_ENGINES). Never accepted, never left out as pre-existing.
* Defined here and re-exported by review-answers.mjs, which imports this module.
*/
export function isSecurityFinding(f) {
return getRule(f?.rule_key)?.category === 'security' || f?.engine === 'security-review';
return getRule(f?.rule_key)?.category === 'security' || SECURITY_ENGINES.includes(f?.engine);
}
/** The folded finding: the current one's place and text, the higher severity, a security origin never lost. */

7
lib/review/rule-catalogue.mjs
View file

@ -18,8 +18,9 @@
// key, for a security finding by its own reviewers (plan-critic,
// scope-guardian, the code reviewer's part for command texts) that
// SECURITY_INJECTION does not cover. SECURITY_REVIEW_VULNERABILITY is the
// engines' key only: Voyage's reviewers never use it, and a finding under it
// is never Voyage's own (lib/review/threat-model.mjs).
// engines' key only (`/security-review`, and Claude Security since E 3.1):
// Voyage's reviewers never use it, and a finding under it is never Voyage's
// own (lib/review/threat-model.mjs).
//
// `memory_route` decides where a finding class that recurs across two
// review.md files is proposed as project memory (P5, lib/review/recurrence.mjs):
@ -144,7 +145,7 @@ export const RULE_CATALOGUE = Object.freeze([
rule_key: 'SECURITY_REVIEW_VULNERABILITY',
severity: 'BLOCKER',
category: 'security',
description: "A vulnerability reported by a security engine (Claude Code's /security-review) that no narrower rule_key covers. The engines' key only: Voyage's own reviewers use SECURITY_RISK.",
description: "A vulnerability reported by a security engine (Claude Security, or Claude Code's /security-review) that no narrower rule_key covers. The engines' key only: Voyage's own reviewers use SECURITY_RISK.",
memory_route: null,
}),
Object.freeze({

318
lib/review/security-engine.mjs Normal file
View file

@ -0,0 +1,318 @@
// lib/review/security-engine.mjs
// The security engine of /trekreview (E 3.1; decisions 175, 196, 219).
//
// Which engine reviews security, and Claude Security's own report turned into
// a reviewer payload. Security that no engine could look at is "not
// measured", never 0 (196; S203: `/security-review` gave 0 in 13 of 13).
//
// choose --repo <dir> --base <sha> --head <sha> --claude-security available|missing
// prints {engine, reason}. "available" is decided by the orchestrator (the
// `claude-security:claude-security` skill is listed and the Workflow tool is
// callable) and passed in; this module never probes for it.
// 1. Claude Security available and `base..head` has committed changes
// -> engine "claude-security", reason "claude-security-available".
// 2. else, when origin's HEAD resolves (`refs/remotes/origin/HEAD`):
// origin/HEAD...HEAD has changes -> "security-review" (reason
// "claude-security-missing", or "claude-security-empty-diff" when Claude
// Security was available but its range was empty); none -> null,
// "empty-diff".
// 3. else null: "empty-diff" when Claude Security was available (its range
// was empty), "no-engine" without an `origin` remote (neither engine can
// run), "no-origin" with an `origin` remote whose HEAD does not resolve.
// Engine null is security not measured: /trekreview then expects the
// reviewer NO_ENGINE_REVIEWER (`security-engine`), which never reports, so
// the review carries `missing-reviewer:security-engine` and the verdict gate
// exits 2. Only committed changes count (uncommitted edits are not in either
// engine's range).
// exit 0 = decided (the engine may be null); 2 = could not run: bad usage,
// not a git repository, a sha that does not resolve, a git failure. Never a
// reason, never a guess.
//
// convert --report-dir <dir> --base <sha> --head <sha> [--scope <file-list.txt>]
// <dir> is the scanned repo's root, where a changes scan writes its
// `CLAUDE-SECURITY-<UTC YYYYMMDD-HHMMSS>/` folder (Claude Security 0.12.0,
// scripts/render_report.py). Reads the newest such folder whose one stamp
// `CLAUDE-SECURITY-REVISION-<12 hex>[-dirty].json` has mode "changes",
// revision.commit == head and revision.merge_base == base (a `-dirty` stamp
// counts; a folder with no stamp or two is passed over), and its
// `CLAUDE-SECURITY-RESULTS.jsonl`, one finding per line. Each record becomes
// an engine finding through engine-findings.mjs `toReviewerPayload` (engine
// `claude-security`): file <- the stamp's `scan_prefix` + `file` (the record
// is relative to the scan root), summary <- title, failure_scenario <-
// exploit_scenario, category <- cwe_id, fix <- recommendation; no narrower
// rule_key. Confidence `high` -> `findings` / `out_of_scope` as for any
// engine; every other confidence -> `below_threshold`, never in `findings`,
// so engine-compare.mjs `putBack` and the verdict gate never restore one
// (219: listed under Declined to Judge as `engine:below-threshold`, an
// unverified reason: not counted, it keeps the verdict from ALLOW). No hand
// transcription.
// Prints {reviewer, findings, out_of_scope, below_threshold, report}; exit 0.
// Not measured (exit 2, the reason on stderr, nothing on stdout): no such
// stamp (an empty diff writes none), verification.status not "verified",
// researchers_returned or researchers_dispatched missing or returned <
// dispatched, run_shape.empty_diff or empty_scope true or not recorded, a
// malformed scan_prefix, a missing or unreadable JSONL, a record that does not
// parse or lacks its file, title or a confidence of low|medium|high.
// exit 1 = the scope list is empty (as engine-findings.mjs); bad usage = 2.
//
// The report folder lives in the repo under review, so its text is data: a
// stamp counts only when it names this exact head and merge base, and a
// record's text is carried, never followed.
import { spawnSync } from 'node:child_process';
import { existsSync, lstatSync, readFileSync, readdirSync } from 'node:fs';
import { join, posix } from 'node:path';
import { toReviewerPayload, ENGINES } from './engine-findings.mjs';
import { FULL_SHA_RE } from './reviewed-commit.mjs';
import { isMain } from '../util/is-main.mjs';
/** The reviewer /trekreview expects when no security engine was chosen: it never reports. */
export const NO_ENGINE_REVIEWER = 'security-engine';
const ENGINE = 'claude-security';
const REPORT_DIR_RE = /^CLAUDE-SECURITY-[0-9][0-9-]*$/;
const STAMP_RE = /^CLAUDE-SECURITY-REVISION-.+\.json$/;
const JSONL_NAME = 'CLAUDE-SECURITY-RESULTS.jsonl';
const CONFIDENCES = new Set(['low', 'medium', 'high']);
const ORIGIN_HEAD = 'refs/remotes/origin/HEAD';
const isPlainObject = (x) => x !== null && typeof x === 'object' && !Array.isArray(x);
const couldNotRun = (reason) => ({ exit: 2, reason });
const notMeasured = (why) => ({ exit: 2, reason: `Claude Security report not measured: ${why}` });
// ---- choose ------------------------------------------------------------------------
function git(repo, args) {
return spawnSync('git', ['-C', repo, ...args], { encoding: 'utf8' });
}
/** true = the two commits differ, false = no change, null = git failed. Committed trees only. */
function differs(repo, ...revs) {
const r = git(repo, ['diff', '--quiet', '--no-ext-diff', '--no-textconv', ...revs, '--']);
if (r.error) return null;
if (r.status === 0) return false;
if (r.status === 1) return true;
return null;
}
/**
* Which security engine reviews `base..head` (see the header).
* @param {{ repo: string, base: string, head: string, claudeSecurity: 'available'|'missing' }} args
* @returns {{ exit: 0, engine: string|null, reason: string } | { exit: 2, reason: string }}
*/
export function chooseEngine({ repo, base, head, claudeSecurity } = {}) {
if (typeof repo !== 'string' || repo === '') return couldNotRun('--repo is required');
if (!FULL_SHA_RE.test(String(base ?? ''))) return couldNotRun('--base must be a full 40- or 64-hex commit id');
if (!FULL_SHA_RE.test(String(head ?? ''))) return couldNotRun('--head must be a full 40- or 64-hex commit id');
if (claudeSecurity !== 'available' && claudeSecurity !== 'missing') {
return couldNotRun('--claude-security must be available or missing');
}
const top = git(repo, ['rev-parse', '--git-dir']);
if (top.error || top.status !== 0) return couldNotRun(`${repo} is not a git repository`);
for (const [name, sha] of [['--base', base], ['--head', head]]) {
const r = git(repo, ['rev-parse', '--verify', '--quiet', `${sha}^{commit}`]);
if (r.error || r.status !== 0) return couldNotRun(`${name} ${sha} does not resolve to a commit`);
}
const available = claudeSecurity === 'available';
if (available) {
const changed = differs(repo, base, head);
if (changed === null) return couldNotRun(`git diff ${base}..${head} failed`);
if (changed) return { exit: 0, engine: ENGINE, reason: 'claude-security-available' };
}
const originHead = git(repo, ['rev-parse', '--verify', '--quiet', ORIGIN_HEAD]);
if (originHead.error || (originHead.status !== 0 && originHead.status !== 1)) {
return couldNotRun(`git rev-parse ${ORIGIN_HEAD} failed`);
}
if (originHead.status === 0) {
const changed = differs(repo, `${ORIGIN_HEAD}...HEAD`);
if (changed === null) return couldNotRun(`git diff ${ORIGIN_HEAD}...HEAD failed`);
if (!changed) return { exit: 0, engine: null, reason: 'empty-diff' };
return { exit: 0, engine: 'security-review', reason: available ? 'claude-security-empty-diff' : 'claude-security-missing' };
}
if (available) return { exit: 0, engine: null, reason: 'empty-diff' };
const remotes = git(repo, ['remote']);
if (remotes.error || remotes.status !== 0) return couldNotRun('git remote failed');
const hasOrigin = remotes.stdout.split('\n').map((l) => l.trim()).includes('origin');
return { exit: 0, engine: null, reason: hasOrigin ? 'no-origin' : 'no-engine' };
}
// ---- convert -----------------------------------------------------------------------
const isDir = (p) => { try { return lstatSync(p).isDirectory(); } catch { return false; } };
const isFile = (p) => { try { return lstatSync(p).isFile(); } catch { return false; } };
/** The parsed stamp of a report folder when it holds exactly one regular stamp file, else null. */
function stampIn(folder) {
let names;
try { names = readdirSync(folder); } catch { return null; }
const stamps = names.filter((n) => STAMP_RE.test(n) && isFile(join(folder, n)));
if (stamps.length !== 1) return null;
try {
const stamp = JSON.parse(readFileSync(join(folder, stamps[0]), 'utf8'));
return isPlainObject(stamp) ? { name: stamps[0], stamp } : null;
} catch { return null; }
}
const sameSha = (a, b) => typeof a === 'string' && a.toLowerCase() === b.toLowerCase();
/** The newest report folder in `dir` whose stamp is a changes scan of exactly base..head, or null. */
export function findReport(dir, base, head) {
let names;
try { names = readdirSync(dir); } catch { return null; }
const folders = names.filter((n) => REPORT_DIR_RE.test(n) && isDir(join(dir, n))).sort().reverse();
for (const name of folders) {
const s = stampIn(join(dir, name));
if (!s) continue;
const rev = isPlainObject(s.stamp.revision) ? s.stamp.revision : {};
if (s.stamp.mode === 'changes' && sameSha(rev.commit, head) && sameSha(rev.merge_base, base)) {
return { folder: join(dir, name), name, stampName: s.name, stamp: s.stamp };
}
}
return null;
}
/** Why a matching stamp cannot be trusted as a full scan, or null. */
function stampProblem(stamp) {
const v = isPlainObject(stamp.verification) ? stamp.verification : {};
if (v.status !== 'verified') return `verification.status is ${JSON.stringify(v.status ?? null)}, not "verified"`;
const dispatched = v.researchers_dispatched;
const returned = v.researchers_returned;
if (!Number.isInteger(dispatched) || !Number.isInteger(returned)) {
return 'the stamp does not record how many researchers were dispatched and returned';
}
if (returned < dispatched) return `${returned} of ${dispatched} researchers returned`;
const shape = isPlainObject(stamp.run_shape) ? stamp.run_shape : {};
if (typeof shape.empty_diff !== 'boolean' || typeof shape.empty_scope !== 'boolean') {
return 'the stamp does not record whether the diff or the scope was empty';
}
if (shape.empty_diff) return 'the scan saw an empty diff';
if (shape.empty_scope) return 'the scan saw an empty scope';
const prefix = stamp.scan_prefix;
if (typeof prefix !== 'string' || (prefix !== '' && (!prefix.endsWith('/') || prefix.startsWith('/')
|| prefix.includes('\\') || prefix.split('/').slice(0, -1).some((seg) => seg === '' || seg === '.' || seg === '..')))) {
return `scan_prefix ${JSON.stringify(prefix ?? null)} is not a path prefix`;
}
return null;
}
/** The JSONL records as engine-findings input with their confidence, or a reason. */
function readRecords(folder, prefix) {
const p = join(folder, JSONL_NAME);
if (!isFile(p)) return { reason: `${JSONL_NAME} is missing` };
let text;
try { text = readFileSync(p, 'utf8'); } catch (e) { return { reason: `${JSONL_NAME} cannot be read: ${e.message}` }; }
const out = [];
const lines = text.split('\n');
for (let i = 0; i < lines.length; i++) {
if (lines[i].trim() === '') continue;
let r;
try { r = JSON.parse(lines[i]); } catch (e) { return { reason: `${JSONL_NAME} line ${i + 1} does not parse: ${e.message}` }; }
if (!isPlainObject(r)) return { reason: `${JSONL_NAME} line ${i + 1} is not an object` };
const confidence = String(r.confidence ?? '').trim().toLowerCase();
if (typeof r.file !== 'string' || r.file.trim() === '') return { reason: `${JSONL_NAME} line ${i + 1} has no file` };
if (typeof r.title !== 'string' || r.title.trim() === '') return { reason: `${JSONL_NAME} line ${i + 1} has no title` };
if (!CONFIDENCES.has(confidence)) {
return { reason: `${JSONL_NAME} line ${i + 1} has confidence ${JSON.stringify(r.confidence ?? null)}, not low|medium|high` };
}
const str = (v) => (typeof v === 'string' ? v : '');
out.push({
confidence,
finding: {
file: posix.normalize(prefix + r.file.trim()),
line: r.line,
summary: r.title,
failure_scenario: str(r.exploit_scenario),
category: str(r.cwe_id),
...(str(r.recommendation).trim() !== '' ? { fix: r.recommendation } : {}),
},
});
}
return { records: out };
}
/**
* Claude Security's report of `base..head` as a reviewer payload (see the header).
* @param {{ reportDir: string, base: string, head: string, scopeFiles?: string[] }} args
* @returns {{ exit: 0, payload: object } | { exit: 1|2, reason: string }}
*/
export function convertReport({ reportDir, base, head, scopeFiles } = {}) {
if (typeof reportDir !== 'string' || reportDir === '') return couldNotRun('--report-dir is required');
if (!FULL_SHA_RE.test(String(base ?? ''))) return couldNotRun('--base must be a full 40- or 64-hex commit id');
if (!FULL_SHA_RE.test(String(head ?? ''))) return couldNotRun('--head must be a full 40- or 64-hex commit id');
const found = findReport(reportDir, base, head);
if (!found) {
return notMeasured(`no CLAUDE-SECURITY-<ts>/ folder in ${reportDir} holds one stamp of a changes scan of ${base}..${head} (an empty diff writes none)`);
}
const problem = stampProblem(found.stamp);
if (problem) return notMeasured(`${found.name}/${found.stampName}: ${problem}`);
const read = readRecords(found.folder, found.stamp.scan_prefix);
if (read.reason) return notMeasured(`${found.name}: ${read.reason}`);
const high = read.records.filter((r) => r.confidence === 'high').map((r) => r.finding);
const rest = read.records.filter((r) => r.confidence !== 'high').map((r) => r.finding);
const counted = toReviewerPayload({ engine: ENGINE, findings: high }, { scopeFiles });
if (!counted.ok) return { exit: 1, reason: counted.errors.join('; ') };
const below = toReviewerPayload({ engine: ENGINE, findings: rest });
if (!below.ok) return notMeasured(`${found.name}: ${below.errors.join('; ')}`);
return {
exit: 0,
payload: {
reviewer: ENGINES[ENGINE].reviewer,
findings: counted.payload.findings,
out_of_scope: counted.out_of_scope,
below_threshold: below.payload.findings,
report: { folder: found.name, stamp: found.stampName },
},
};
}
// ---- CLI ---------------------------------------------------------------------------
const USAGE = 'Usage: security-engine.mjs choose --repo <dir> --base <sha> --head <sha> --claude-security available|missing\n'
+ ' security-engine.mjs convert --report-dir <dir> --base <sha> --head <sha> [--scope <file-list.txt>]\n';
/** `--flag value` pairs, each known flag at most once; null on anything else. */
function flags(args, known) {
const out = {};
for (let i = 0; i < args.length; i += 2) {
const k = args[i];
if (!known.includes(k) || Object.hasOwn(out, k) || i + 1 >= args.length) return null;
out[k] = args[i + 1];
}
return out;
}
function runCli(argv) {
const [cmd, ...rest] = argv;
if (cmd === 'choose') {
const f = flags(rest, ['--repo', '--base', '--head', '--claude-security']);
if (!f) return couldNotRun('bad arguments');
const r = chooseEngine({ repo: f['--repo'], base: f['--base'], head: f['--head'], claudeSecurity: f['--claude-security'] });
return r.exit === 0 ? { exit: 0, out: { engine: r.engine, reason: r.reason } } : r;
}
if (cmd === 'convert') {
const f = flags(rest, ['--report-dir', '--base', '--head', '--scope']);
if (!f) return couldNotRun('bad arguments');
let scopeFiles;
if (f['--scope'] !== undefined) {
if (!existsSync(f['--scope'])) return couldNotRun(`--scope ${f['--scope']} does not exist`);
try {
scopeFiles = readFileSync(f['--scope'], 'utf8').split('\n').map((l) => l.trim()).filter(Boolean);
} catch (e) { return couldNotRun(`cannot read --scope: ${e.message}`); }
}
const r = convertReport({ reportDir: f['--report-dir'], base: f['--base'], head: f['--head'], scopeFiles });
return r.exit === 0 ? { exit: 0, out: r.payload } : r;
}
return couldNotRun('a subcommand is required: choose or convert');
}
if (isMain(import.meta.url)) {
let r;
try { r = runCli(process.argv.slice(2)); } catch (e) { r = couldNotRun(`security-engine crashed: ${e.message}`); }
if (r.exit === 0) {
process.stdout.write(`${JSON.stringify(r.out, null, 2)}\n`);
} else {
process.stderr.write(`security-engine: ${r.reason}\n`);
if (r.reason === 'bad arguments' || r.reason.startsWith('a subcommand')) process.stderr.write(USAGE);
}
process.exit(r.exit);
}

4
lib/validators/review-validator.mjs
View file

@ -23,6 +23,7 @@ import { issue, ok, fail } from '../util/result.mjs';
import { isMain } from '../util/is-main.mjs';
import { extractFindingsBlock } from '../review/findings-schema.mjs';
import { isKnownEntry } from '../review/threat-model.mjs';
import { ENGINES } from '../review/engine-findings.mjs';
export const REVIEW_REQUIRED_FRONTMATTER = [
'type',
@ -86,7 +87,8 @@ const FULL_SHA_RE = /^(?:[0-9a-f]{40}|[0-9a-f]{64})$/;
const SHA256_RE = /^[0-9a-f]{64}$/;
const SEVERITIES = ['BLOCKER', 'MAJOR', 'MINOR', 'SUGGESTION'];
const REVIEW_POLICY_VALUES = ['absent', 'applied', 'not-applied'];
const RAN_ENGINES = ['code-review', 'security-review'];
// The engines a review's `ran` may name: engine-findings.mjs ENGINES (Claude Security since E 3.1).
const RAN_ENGINES = Object.keys(ENGINES);
const ANSWERS_HEADER = '| Finding | Answer | Reason | Record | Carried from |';
const ANSWER_FINDING_RE = /^([0-9a-f]{40})(?:#\d+)?$/;
const CARRIED_FROM_RE = /^review-round-[0-9a-f]{12}-\d+\.md#[0-9a-f]{40}$/;

95
tests/helpers/claude-security-report.mjs Normal file
View file

@ -0,0 +1,95 @@
// tests/helpers/claude-security-report.mjs
// A Claude Security report folder as a changes scan writes it (Claude Security
// 0.12.0: scripts/render_report.py, scripts/lib/finding.py, scripts/lib/plugin.py),
// for the tests of lib/review/security-engine.mjs `convert`:
//
// <dir>/CLAUDE-SECURITY-<UTC YYYYMMDD-HHMMSS>/
// .gitignore `*`
// CLAUDE-SECURITY-RESULTS.md
// CLAUDE-SECURITY-RESULTS.jsonl one finding per line
// CLAUDE-SECURITY-RESULTS.sarif
// CLAUDE-SECURITY-REVISION-<first 12 hex of the commit>[-dirty].json the stamp
//
// `writeReport` takes the commit range and lets a test change any part of the
// stamp (`stamp: (s) => ...`) or write the JSONL text itself (`jsonl`).
import { mkdirSync, writeFileSync } from 'node:fs';
import { join } from 'node:path';
/** One JSONL record in the field order the products carry, with `over` applied. */
export function record(over = {}) {
return {
id: 'F1',
title: 'Shell command built from the branch name',
impact: 'A pushed branch name runs a command on the machine that checks it out',
file: 'src/run.mjs',
line: 12,
description: 'checkout() interpolates the branch name into a shell string.',
exploit_scenario: 'A branch named `x;touch pwned` runs `touch pwned` when it is checked out',
preconditions: ['the attacker can push a branch'],
category: 'Injection',
severity: 'HIGH',
confidence: 'high',
recommendation: 'Pass the branch name as an argv element through execFile, never through a shell string',
cwe_id: 'CWE-78',
snippet: 'exec(`git checkout ${branch}`)',
symbol: 'checkout',
declared_line: 12,
via_change: null,
other_cwe_ids: [],
claudeSecurityPluginFindingId: 'a3f9c2d1e0b4',
...over,
};
}
/** The stamp of a verified changes scan of `base..head`. */
export function stampOf({ head, base, dirty = false, records = [] }) {
const count = (sev) => records.filter((r) => r.severity === sev).length;
return {
generated_at: '2026-11-08T12:00:00+00:00',
duration_s: 412,
scan_id: '6f1d2a4e-5b7c-4e1a-9d3f-2c8b0a7e6d51',
mode: 'changes',
scan_prefix: '',
scope: [],
revision: { versioned: true, commit: head, branch: 'feat/x', dirty, base, merge_base: base },
revision_source: 'git',
model: 'claude-opus-5-5',
effort: 'high',
run_shape: {
requested_effort: 'high', collapsed: null, source: 'coverage.json',
diff_files: 2, diff_lines: 30, pre_existing: 0, scope_files: 2,
empty_diff: false, empty_scope: false, researchers_dispatched: 3, verification_runs: 1,
},
findings: { total: records.length, critical: count('CRITICAL'), high: count('HIGH'), medium: count('MEDIUM'), low: count('LOW') },
verification: {
status: 'verified', candidates: records.length, candidates_deduped: records.length,
panel_votes: 3 * records.length, panel_reviewed_findings: records.length,
panel_quorum_findings: records.length, unreviewed_candidate_sites: 0,
incomplete_panel_candidates: 0, attested_findings: 0, reason: null, reason_kind: null,
researchers_dispatched: 3, researchers_returned: 3,
},
};
}
/**
* Write one report folder under `dir` and return its path.
* @param {string} dir - the scanned repo's root
* @param {{ head: string, base: string, ts?: string, dirty?: boolean, records?: object[],
* jsonl?: string|null, stamp?: (s: object) => object, stampName?: string }} opts
* `jsonl: null` writes no JSONL file; `stamp` returns the stamp to write.
*/
export function writeReport(dir, { head, base, ts = '20261108-120000', dirty = false, records = [record()], jsonl, stamp = (s) => s, stampName }) {
const folder = join(dir, `CLAUDE-SECURITY-${ts}`);
mkdirSync(folder, { recursive: true });
writeFileSync(join(folder, '.gitignore'), '*\n');
writeFileSync(join(folder, 'CLAUDE-SECURITY-RESULTS.md'), `# Claude Security results\n\n${records.length} finding(s).\n`);
writeFileSync(join(folder, 'CLAUDE-SECURITY-RESULTS.sarif'), JSON.stringify({ version: '2.1.0', runs: [] }));
if (jsonl !== null) {
writeFileSync(join(folder, 'CLAUDE-SECURITY-RESULTS.jsonl'),
jsonl ?? records.map((r) => `${JSON.stringify(r)}\n`).join(''));
}
const name = stampName ?? `CLAUDE-SECURITY-REVISION-${head.slice(0, 12)}${dirty ? '-dirty' : ''}.json`;
writeFileSync(join(folder, name), JSON.stringify(stamp(stampOf({ head, base, dirty, records })), null, 2));
return folder;
}

45
tests/lib/coordinator-contract.test.mjs
View file

@ -21,6 +21,8 @@ import {
REFUTING_REASONS,
UNVERIFIED_REASONS,
runContract,
lowerableBriefConflict,
illegalBriefLowering,
} from '../../lib/review/coordinator-contract.mjs';
const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..', '..');
@ -616,3 +618,46 @@ test('runContract — a MAJOR /code-review defect cited at an unchanged file sti
assert.notEqual(c.verdict, 'ALLOW');
assert.equal(c.declined_to_judge.length, 1);
});
// ---- E 3.1: Claude Security is a security engine; below its threshold is declined ----
const csFinding = (over) => ({
id: 'c'.repeat(40), severity: 'BLOCKER', rule_key: 'SECURITY_REVIEW_VULNERABILITY',
file: 'src/run.mjs', line: 12, brief_ref: 'NFR - security', title: 'Shell command built from the branch name',
detail: 'A crafted branch name runs a command', recommended_action: 'Pass the name as an argv element',
...over,
});
test('[E3] engine:below-threshold is an unverified reason: declined to judge, not counted, it keeps the verdict from ALLOW', () => {
assert.ok(UNVERIFIED_REASONS.includes('engine:below-threshold'));
assert.equal(classifySuppression('engine:below-threshold'), 'unverified');
const below = csFinding({ engine: 'claude-security', title: 'Token compared with ===' });
const c = runContract([{ reviewer: 'claude-security', findings: [] }],
{ expectedReviewers: ['claude-security'], engineBelowThreshold: [below] });
assert.equal(c.verdict, 'WARN', JSON.stringify(c.allow_blocked_by));
assert.equal(c.counts.BLOCKER, 0, 'a below-threshold finding is not counted');
assert.deepEqual(c.findings, []);
assert.deepEqual(c.declined_to_judge.map((d) => [d.title, d.reason]), [['Token compared with ===', 'engine:below-threshold']]);
assert.deepEqual(c.allow_blocked_by, ['unverified:engine:below-threshold (1)']);
});
test('[E3] a Claude Security payload\'s findings carry engine claude-security, read from the payload', () => {
const { findings } = ingest([{ reviewer: 'claude-security', findings: [csFinding({ engine: 'code-review' })] }]);
assert.equal(findings[0].engine, 'claude-security', 'the engine comes from the payload, never from the finding');
});
test('[E3] a Claude Security finding outside the scope forbids ALLOW, as a /security-review one does', () => {
const c = runContract([], { engineOutOfScope: [csFinding({ engine: 'claude-security', file: 'lib/old.mjs' })] });
assert.notEqual(c.verdict, 'ALLOW');
assert.deepEqual(c.declined_to_judge.map((d) => d.reason), ['engine:out-of-scope']);
});
test('[E3] the brief-conflict rule: a Claude Security finding is only ever marked, never lowered', () => {
const marked = csFinding({ engine: 'claude-security', contradicts_brief: 'No shell wrapper (Non-Goal)', rule_key: 'CODE_REVIEW_DEFECT', severity: 'MAJOR' });
assert.equal(lowerableBriefConflict(marked), false, 'only a /code-review MAJOR drops');
assert.equal(illegalBriefLowering({ ...marked, severity: 'MINOR', brief_lowered_from: 'MAJOR' }), true);
assert.equal(illegalBriefLowering(csFinding({ engine: 'claude-security', severity: 'MAJOR' })), true,
'a Claude Security finding below its catalogue tier is illegal');
assert.equal(illegalBriefLowering(csFinding({ engine: 'claude-security', contradicts_brief: 'No shell wrapper (Non-Goal)' })), false,
'marked at its tier is the legal shape');
});

69
tests/lib/engine-findings-compare.test.mjs
View file

@ -23,6 +23,7 @@ import { dirname, join } from 'node:path';
import { fileURLToPath, pathToFileURL } from 'node:url';
import { computeFindingId, computeFindingIds } from '../../lib/parsers/finding-id.mjs';
import { childEnv } from '../helpers/child-env.mjs';
import { record, writeReport } from '../helpers/claude-security-report.mjs';
const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..', '..');
const COMPARE = join(ROOT, 'lib', 'review', 'engine-compare.mjs');
@ -31,6 +32,7 @@ const FINALIZE = join(ROOT, 'lib', 'review', 'review-finalize.mjs');
const FILL = join(ROOT, 'lib', 'review', 'fill-finding-ids.mjs');
const VALIDATOR = join(ROOT, 'lib', 'validators', 'review-validator.mjs');
const GATE = join(ROOT, 'lib', 'review', 'verdict-gate.mjs');
const SECURITY_CLI = join(ROOT, 'lib', 'review', 'security-engine.mjs');
const REVIEW_REL = '.claude/projects/2026-11-02-eng/review.md';
const SOURCE_LINE = (engine) => `**Source:** engine ${engine}, put back by the standard-path comparison`;
@ -355,3 +357,70 @@ test('[ENG] a review finalized with a put-back has every id computed, no placeho
const bgate = bx.gate();
assert.equal(bgate.status, 1, `the gate reads BLOCK: ${bgate.stdout}${bgate.stderr}`);
});
// --- E 3.1: Claude Security's payload, converted from its report ------------------
/** Claude Security's report of `start..end` in the repo root, converted by the real CLI and saved as Phase 4.7 saves it. */
function claudeSecurity(fx, records) {
writeReport(fx.repo, { head: fx.end, base: fx.start, records });
const scope = join(fx.project, '.review-scope.txt');
writeFileSync(scope, 'src/a.mjs\nsrc/b.mjs\ndocs/guide/x.md\n');
const r = run([SECURITY_CLI, 'convert', '--report-dir', fx.repo, '--base', fx.start, '--head', fx.end, '--scope', scope]);
assert.equal(r.status, 0, `security-engine convert: ${r.stdout}${r.stderr}`);
writeFileSync(join(fx.project, '.review-engine-payload-claude-security.json'), r.stdout);
return JSON.parse(r.stdout);
}
const CS_HIGH = record({ file: 'src/a.mjs', line: 1, title: 'A shell command is built from the branch name' });
const CS_MEDIUM = record({ id: 'F2', file: 'src/b.mjs', line: 3, title: 'A path is joined from a query value', confidence: 'medium', cwe_id: 'CWE-22' });
test('[E3] a Claude Security finding the coordinator dropped is put back; a below-threshold one never is; ran names claude-security', () => {
const fx = engRepo();
fx.write(engReview({
start: fx.start, end: fx.end, findings: [REVIEWER_MINOR],
declined: [{ file: 'src/b.mjs', line: 3, rule_key: 'SECURITY_REVIEW_VULNERABILITY', severity: 'BLOCKER', title: CS_MEDIUM.title, reason: 'engine:below-threshold' }],
}));
const payload = claudeSecurity(fx, [CS_HIGH, CS_MEDIUM]);
assert.ok(Array.isArray(payload.below_threshold), `the payload has below_threshold: ${JSON.stringify(payload)}`);
assert.deepEqual(payload.below_threshold.map((b) => b.title), [CS_MEDIUM.title]);
const r = fx.finalize();
assert.equal(r.status, 0, r.stdout + r.stderr);
const text = fx.read();
const json = jsonOf(text);
const back = on(json.findings, 'src/a.mjs', 1, 'SECURITY_REVIEW_VULNERABILITY');
assert.equal(back.length, 1, `the dropped Claude Security finding is back: ${JSON.stringify(json.findings.map((f) => f.title))}`);
assert.equal(back[0].engine, 'claude-security');
assert.equal(back[0].severity, 'BLOCKER');
assert.equal(json.verdict, 'BLOCK');
assert.ok(entryOf(sectionOf(text, '## Findings (BLOCKER)'), back[0].id).includes(SOURCE_LINE('claude-security')),
`the entry names its source:\n${sectionOf(text, '## Findings (BLOCKER)')}`);
assert.equal(json.findings.filter((f) => f.title === CS_MEDIUM.title).length, 0, 'a below-threshold finding is never put back');
assert.deepEqual(json.ran.engines, ['claude-security'], 'ran names the engine whose payload is in the project folder');
const v = fx.validate();
assert.equal(v.status, 0, `review-validator --strict reads ran: ${v.stdout}${v.stderr}`);
const gate = fx.gate();
assert.equal(gate.status, 1, `the gate reads BLOCK: ${gate.stdout}${gate.stderr}`);
});
test('[E3] the verdict gate reads a Claude Security finding missing from the review as not measured; a below-threshold one alone is not', () => {
// Only a below-threshold finding: the review that declines it is read (exit 0, WARN).
const declined = [{ file: 'src/b.mjs', line: 3, rule_key: 'SECURITY_REVIEW_VULNERABILITY', severity: 'BLOCKER', title: CS_MEDIUM.title, reason: 'engine:below-threshold' }];
// The coordinator's review: the reviewer MINOR, the below-threshold finding declined, so WARN.
const warn = (f) => engReview({ start: f.start, end: f.end, findings: [REVIEWER_MINOR], declined })
.replace('verdict: ALLOW', 'verdict: WARN')
.replace('"verdict": "ALLOW"', '"verdict": "WARN"')
.replace('"allow_blocked_by": []', '"allow_blocked_by": ["unverified:engine:below-threshold (1)"]');
const fx = engRepo();
fx.write(warn(fx));
assert.equal(jsonOf(fx.read()).verdict, 'WARN', 'the fixture review is WARN');
claudeSecurity(fx, [CS_MEDIUM]);
const quiet = fx.gate();
assert.equal(quiet.status, 0, `a below-threshold finding is no dropped engine finding: ${quiet.stdout}${quiet.stderr}`);
// A high-confidence finding the review does not hold: not measured.
const hx = engRepo();
hx.write(warn(hx));
claudeSecurity(hx, [CS_HIGH, CS_MEDIUM]);
const loud = hx.gate();
assert.equal(loud.status, 2, loud.stdout + loud.stderr);
assert.match(String(parse(loud).reason), /engine finding not in review: BLOCKER SECURITY_REVIEW_VULNERABILITY at src\/a\.mjs:1/, loud.stdout);
});

8
tests/lib/review-rounds.test.mjs
View file

@ -1078,3 +1078,11 @@ test('[STP] a round with no readable JSON block at P = 2: prepare is not measure
}
}
});
// ---- E 3.1: a finding from Claude Security is a security finding ------------------
test('[E3] isSecurityFinding: a finding from Claude Security is a security finding whatever its key, as one from /security-review is', () => {
assert.equal(rounds.isSecurityFinding({ engine: 'claude-security', rule_key: 'CODE_REVIEW_DEFECT' }), true);
assert.equal(rounds.isSecurityFinding({ engine: 'security-review', rule_key: 'CODE_REVIEW_DEFECT' }), true);
assert.equal(rounds.isSecurityFinding({ engine: 'code-review', rule_key: 'CODE_REVIEW_DEFECT' }), false, 'control: /code-review is not a security engine');
});

13
tests/lib/review-validator-1-2.test.mjs
View file

@ -114,3 +114,16 @@ test('[V12] a malformed round, pre-existing or ran field at 1.2 fails with its c
assert.ok(codes(r.errors).includes(code), `${name}: ${code} expected, got ${JSON.stringify(r.errors)}`);
}
});
// ---- E 3.1: Claude Security is an engine that ran ---------------------------------
test('[E3] ran.engines may name claude-security; an unknown engine is still refused', () => {
const v12 = fixture('review-1-2.md');
const from = '"engines": ["code-review", "security-review"]';
assert.ok(v12.includes(from), 'the 1.2 fixture names its engines');
const r = validateReviewContent(mutate(v12, from, '"engines": ["claude-security", "code-review"]'), { strict: true });
assert.ok(!codes(r.errors).includes('REVIEW_ROUND_MALFORMED'), JSON.stringify(r.errors));
assert.equal(r.valid, true, JSON.stringify(r.errors));
const bad = validateReviewContent(mutate(v12, from, '"engines": ["claude-security-review"]'), { strict: true });
assert.ok(codes(bad.errors).includes('REVIEW_ROUND_MALFORMED'), 'control: a reviewer name is not an engine');
});

394
tests/lib/security-engine.test.mjs Normal file
View file

@ -0,0 +1,394 @@
// tests/lib/security-engine.test.mjs
// E 3.1 (decisions 175, 196, 219): the security engine of /trekreview.
// choose - which engine reviews security: Claude Security when available
// and base..head has committed changes; else /security-review when
// origin/HEAD resolves and origin/HEAD...HEAD is non-empty; else
// none (no-engine, no-origin, empty-diff) = "not measured", never 0.
// A git failure or bad usage is exit 2, never a reason.
// convert - Claude Security's report folder turned into a reviewer payload:
// confidence `high` -> engine findings, the rest -> `below_threshold`,
// never in `findings`; a scan that cannot be trusted is exit 2.
// ENGINES - claude-security is an engine of its own, and the action written
// for a finding names the engine that reported it.
// none - no engine chosen: the expected reviewer `security-engine` never
// reports, so the verdict gate exits 2, never ALLOW.
// choose runs the real CLI in throwaway git repos with no hooks and no global
// or system git config.
import { after, test } from 'node:test';
import { strict as assert } from 'node:assert';
import { spawnSync } from 'node:child_process';
import { appendFileSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { dirname, join } from 'node:path';
import { fileURLToPath } from 'node:url';
import { childEnv } from '../helpers/child-env.mjs';
import { record, writeReport } from '../helpers/claude-security-report.mjs';
import { convertReport, NO_ENGINE_REVIEWER } from '../../lib/review/security-engine.mjs';
import { ENGINES, toReviewerPayload } from '../../lib/review/engine-findings.mjs';
import { runContract } from '../../lib/review/coordinator-contract.mjs';
import { decideVerdict } from '../../lib/review/verdict-gate.mjs';
const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..', '..');
const CLI = join(ROOT, 'lib', 'review', 'security-engine.mjs');
const GIT_ENV = { GIT_CONFIG_GLOBAL: '/dev/null', GIT_CONFIG_NOSYSTEM: '1' };
const HEX40 = /^[0-9a-f]{40}$/;
const made = [];
after(() => { for (const b of made) rmSync(b, { recursive: true, force: true }); });
const run = (args) => spawnSync(process.execPath, [CLI, ...args], { encoding: 'utf8', env: childEnv(GIT_ENV) });
const parse = (r) => { try { return JSON.parse(r.stdout); } catch { return {}; } };
function g(cwd, ...args) {
const r = spawnSync('git', args, { cwd, encoding: 'utf8', env: childEnv(GIT_ENV) });
if (r.status !== 0) throw new Error(`git ${args.join(' ')}: ${r.stderr}`);
return r.stdout.trim();
}
/** A throwaway repo with two commits (start, end) that change a.mjs; no origin remote. */
function repo() {
const base = mkdtempSync(join(tmpdir(), 'security-engine-'));
made.push(base);
const dir = join(base, 'repo');
mkdirSync(dir, { recursive: true });
mkdirSync(join(base, 'hooks-none'));
g(dir, 'init', '-q', '-b', 'main');
appendFileSync(join(dir, '.git', 'config'), '[user]\n\tname = Test\n\temail = test@example.invalid\n'
+ `[commit]\n\tgpgsign = false\n[core]\n\thooksPath = "${join(base, 'hooks-none')}"\n`);
writeFileSync(join(dir, 'a.mjs'), 'export const a = 1;\n');
g(dir, 'add', 'a.mjs');
g(dir, 'commit', '-q', '-m', 'chore: start');
const start = g(dir, 'rev-parse', 'HEAD');
writeFileSync(join(dir, 'a.mjs'), 'export const a = 2;\n');
g(dir, 'add', 'a.mjs');
g(dir, 'commit', '-q', '-m', 'feat: the work');
const end = g(dir, 'rev-parse', 'HEAD');
return {
base, dir, start, end,
/** An `origin` remote (never fetched) whose HEAD points at `sha`. */
originAt(sha) {
g(dir, 'remote', 'add', 'origin', join(base, 'nowhere.git'));
g(dir, 'update-ref', 'refs/remotes/origin/main', sha);
g(dir, 'symbolic-ref', 'refs/remotes/origin/HEAD', 'refs/remotes/origin/main');
},
/** An `origin` remote with no remote-tracking refs at all. */
originWithoutHead() {
g(dir, 'remote', 'add', 'origin', join(base, 'nowhere.git'));
},
};
}
const choose = (fx, { base = fx.start, head = fx.end, cs = 'missing' } = {}) =>
run(['choose', '--repo', fx.dir, '--base', base, '--head', head, '--claude-security', cs]);
function decided(r) {
assert.equal(r.status, 0, `choose decides: ${r.stdout}${r.stderr}`);
const j = parse(r);
assert.deepEqual(Object.keys(j).sort(), ['engine', 'reason'], `the output is {engine, reason}: ${r.stdout}`);
return j;
}
// ---- choose ----------------------------------------------------------------------
test('[CHS] Claude Security available and base..head has committed changes: claude-security', () => {
const fx = repo();
assert.deepEqual(decided(choose(fx, { cs: 'available' })), { engine: 'claude-security', reason: 'claude-security-available' });
});
test('[CHS] Claude Security missing, origin/HEAD behind HEAD: security-review', () => {
const fx = repo();
fx.originAt(fx.start);
assert.deepEqual(decided(choose(fx)), { engine: 'security-review', reason: 'claude-security-missing' });
});
test('[CHS] Claude Security available but base..head empty, origin/HEAD behind HEAD: security-review', () => {
const fx = repo();
fx.originAt(fx.start);
assert.deepEqual(decided(choose(fx, { base: fx.end, cs: 'available' })), { engine: 'security-review', reason: 'claude-security-empty-diff' });
});
test('[CHS] origin/HEAD equals HEAD and Claude Security is missing: no engine, empty-diff (a pushed main is not measured, never 0)', () => {
const fx = repo();
fx.originAt(fx.end);
assert.deepEqual(decided(choose(fx)), { engine: null, reason: 'empty-diff' });
});
test('[CHS] Claude Security available, base..head empty, no origin/HEAD: no engine, empty-diff', () => {
const fx = repo();
assert.deepEqual(decided(choose(fx, { base: fx.end, cs: 'available' })), { engine: null, reason: 'empty-diff' });
});
test('[CHS] only committed changes count: an uncommitted edit on top of an empty range is still empty-diff', () => {
const fx = repo();
writeFileSync(join(fx.dir, 'a.mjs'), 'export const a = 3;\n');
assert.deepEqual(decided(choose(fx, { base: fx.end, cs: 'available' })), { engine: null, reason: 'empty-diff' });
});
test('[CHS] no origin remote and Claude Security missing: no engine, no-engine', () => {
const fx = repo();
assert.deepEqual(decided(choose(fx)), { engine: null, reason: 'no-engine' });
});
test('[CHS] an origin remote whose HEAD does not resolve, Claude Security missing: no engine, no-origin', () => {
const fx = repo();
fx.originWithoutHead();
assert.deepEqual(decided(choose(fx)), { engine: null, reason: 'no-origin' });
});
test('[CHS] a git failure is exit 2 (could not run), never a reason', () => {
const fx = repo();
const plain = join(fx.base, 'plain');
mkdirSync(plain);
const notRepo = run(['choose', '--repo', plain, '--base', fx.start, '--head', fx.end, '--claude-security', 'available']);
assert.equal(notRepo.status, 2, `not a git repository: ${notRepo.stdout}${notRepo.stderr}`);
assert.equal(notRepo.stdout, '', 'nothing on stdout');
const unknown = 'f'.repeat(40);
for (const [base, head] of [[unknown, fx.end], [fx.start, unknown]]) {
const r = choose(fx, { base, head, cs: 'available' });
assert.equal(r.status, 2, `a sha that does not resolve: ${r.stdout}${r.stderr}`);
assert.equal(r.stdout, '');
}
});
test('[CHS] bad usage is exit 2', () => {
const fx = repo();
const cases = [
['no subcommand', []],
['an unknown subcommand', ['pick', '--repo', fx.dir]],
['--claude-security missing', ['choose', '--repo', fx.dir, '--base', fx.start, '--head', fx.end]],
['--claude-security maybe', ['choose', '--repo', fx.dir, '--base', fx.start, '--head', fx.end, '--claude-security', 'maybe']],
['--base a ref name', ['choose', '--repo', fx.dir, '--base', 'HEAD~1', '--head', fx.end, '--claude-security', 'available']],
['--base an option', ['choose', '--repo', fx.dir, '--base', '--output=/tmp/x', '--head', fx.end, '--claude-security', 'available']],
['--repo missing', ['choose', '--base', fx.start, '--head', fx.end, '--claude-security', 'available']],
['an unknown flag', ['choose', '--repo', fx.dir, '--base', fx.start, '--head', fx.end, '--claude-security', 'available', '--force']],
];
for (const [name, args] of cases) {
const r = run(args);
assert.equal(r.status, 2, `${name}: exit 2, got ${r.status}: ${r.stdout}${r.stderr}`);
assert.equal(r.stdout, '', `${name}: nothing on stdout`);
}
});
// ---- convert ---------------------------------------------------------------------
const BASE = 'a'.repeat(40);
const HEAD = 'b'.repeat(40);
function reportDir() {
const d = mkdtempSync(join(tmpdir(), 'security-engine-report-'));
made.push(d);
return d;
}
const MEDIUM = record({ id: 'F3', title: 'Path joined from a query value', file: 'src/run.mjs', line: 30, confidence: 'medium', cwe_id: 'CWE-22', category: 'Path Traversal' });
const LOW = record({ id: 'F4', title: 'Token compared with ===', file: 'src/run.mjs', line: 44, confidence: 'low', severity: 'LOW', cwe_id: 'CWE-208' });
const OUTSIDE = record({ id: 'F2', title: 'Old helper leaks a handle path', file: 'lib/old.mjs', line: 4 });
test('[CNV] a verified changes scan of base..head: confidence high becomes an engine finding, the rest go to below_threshold and never into findings', () => {
const dir = reportDir();
writeReport(dir, { head: HEAD, base: BASE, records: [record(), OUTSIDE, MEDIUM, LOW] });
const r = convertReport({ reportDir: dir, base: BASE, head: HEAD, scopeFiles: ['src/run.mjs', 'src/other.mjs'] });
assert.equal(r.exit, 0, r.reason);
const p = r.payload;
assert.equal(p.reviewer, 'claude-security');
assert.equal(p.findings.length, 1, JSON.stringify(p.findings.map((f) => f.title)));
const [f] = p.findings;
assert.deepEqual([f.rule_key, f.severity, f.engine, f.file, f.line], ['SECURITY_REVIEW_VULNERABILITY', 'BLOCKER', 'claude-security', 'src/run.mjs', 12]);
assert.equal(f.title, record().title, 'summary <- title');
assert.ok(f.detail.includes(record().exploit_scenario), `failure_scenario <- exploit_scenario: ${f.detail}`);
assert.ok(f.detail.includes('CWE-78'), `category <- cwe_id: ${f.detail}`);
assert.equal(f.recommended_action, record().recommendation, 'fix <- recommendation');
assert.equal(f.brief_ref, ENGINES['claude-security'].brief_ref);
assert.match(f.id, HEX40);
assert.ok(!('rule_key' in record()), 'the record carries no narrower key');
assert.deepEqual(p.out_of_scope.map((o) => [o.file, o.engine]), [['lib/old.mjs', 'claude-security']], 'a high finding outside the scope is listed, never dropped');
assert.deepEqual(p.below_threshold.map((b) => b.title).sort(), [LOW.title, MEDIUM.title].sort(), 'medium and low go to below_threshold');
for (const b of p.below_threshold) assert.equal(b.engine, 'claude-security');
const inFindings = new Set([...p.findings, ...p.out_of_scope].map((x) => x.title));
for (const b of p.below_threshold) assert.ok(!inFindings.has(b.title), `${b.title} is never in findings or out_of_scope`);
});
test('[CNV] the CLI prints the payload and exits 0', () => {
const dir = reportDir();
writeReport(dir, { head: HEAD, base: BASE, records: [record(), MEDIUM] });
const scope = join(dir, 'scope.txt');
writeFileSync(scope, 'src/run.mjs\n');
const r = run(['convert', '--report-dir', dir, '--base', BASE, '--head', HEAD, '--scope', scope]);
assert.equal(r.status, 0, r.stdout + r.stderr);
const p = parse(r);
assert.equal(p.reviewer, 'claude-security');
assert.equal(p.findings.length, 1);
assert.equal(p.below_threshold.length, 1);
});
test('[CNV] a -dirty stamp counts', () => {
const dir = reportDir();
writeReport(dir, { head: HEAD, base: BASE, dirty: true });
const r = convertReport({ reportDir: dir, base: BASE, head: HEAD });
assert.equal(r.exit, 0, r.reason);
assert.equal(r.payload.findings.length, 1);
});
test('[CNV] the newest folder whose stamp matches is read; a newer folder of another commit is passed over', () => {
const dir = reportDir();
writeReport(dir, { head: HEAD, base: BASE, ts: '20261108-100000', records: [record({ title: 'older scan' })] });
writeReport(dir, { head: HEAD, base: BASE, ts: '20261108-110000', records: [record({ title: 'newest matching scan' })] });
writeReport(dir, { head: 'c'.repeat(40), base: BASE, ts: '20261108-120000', records: [record({ title: 'another commit' })] });
const r = convertReport({ reportDir: dir, base: BASE, head: HEAD });
assert.equal(r.exit, 0, r.reason);
assert.deepEqual(r.payload.findings.map((f) => f.title), ['newest matching scan']);
});
test('[CNV] a finding file relative to the scan root is made repo-relative with the stamp\'s scan_prefix', () => {
const dir = reportDir();
writeReport(dir, { head: HEAD, base: BASE, records: [record({ file: 'run.mjs' })], stamp: (s) => ({ ...s, scan_prefix: 'src/' }) });
const r = convertReport({ reportDir: dir, base: BASE, head: HEAD, scopeFiles: ['src/run.mjs'] });
assert.equal(r.exit, 0, r.reason);
assert.deepEqual(r.payload.findings.map((f) => f.file), ['src/run.mjs']);
});
test('[CNV] a scan that found nothing is a report: exit 0, no findings', () => {
const dir = reportDir();
writeReport(dir, { head: HEAD, base: BASE, records: [] });
const r = convertReport({ reportDir: dir, base: BASE, head: HEAD });
assert.equal(r.exit, 0, r.reason);
assert.deepEqual([r.payload.findings, r.payload.out_of_scope, r.payload.below_threshold], [[], [], []]);
});
test('[CNV] not measured: exit 2, the reason on stderr, nothing on stdout', () => {
const setV = (key, value) => (s) => ({ ...s, verification: { ...s.verification, [key]: value } });
const dropV = (key) => (s) => { const v = { ...s.verification }; delete v[key]; return { ...s, verification: v }; };
const setShape = (key, value) => (s) => ({ ...s, run_shape: { ...s.run_shape, [key]: value } });
const cases = [
['no report folder', null],
['a stamp of another commit', { head: 'c'.repeat(40) }],
['a stamp of another merge base', { stamp: (s) => ({ ...s, revision: { ...s.revision, merge_base: 'd'.repeat(40) } }) }],
['a commit scan, not a changes scan', { stamp: (s) => ({ ...s, mode: 'commit' }) }],
['verification unverified', { stamp: setV('status', 'unverified') }],
['fewer researchers returned than dispatched', { stamp: setV('researchers_returned', 2) }],
['researchers_returned missing', { stamp: dropV('researchers_returned') }],
['researchers_dispatched missing', { stamp: dropV('researchers_dispatched') }],
['the scan saw an empty diff', { stamp: setShape('empty_diff', true) }],
['the scan saw an empty scope', { stamp: setShape('empty_scope', true) }],
['no JSONL', { jsonl: null }],
['a JSONL line that does not parse', { jsonl: `${JSON.stringify(record())}\n{"title": \n` }],
['a record with an unknown confidence', { records: [record({ confidence: 'certain' })] }],
['a record with no title', { records: [record({ title: '' })] }],
];
for (const [name, opts] of cases) {
const dir = reportDir();
if (opts !== null) writeReport(dir, { head: HEAD, base: BASE, ...opts });
const r = run(['convert', '--report-dir', dir, '--base', BASE, '--head', HEAD]);
assert.equal(r.status, 2, `${name}: exit 2, got ${r.status}: ${r.stdout}${r.stderr}`);
assert.equal(r.stdout, '', `${name}: nothing on stdout`);
assert.match(r.stderr, /not measured/, `${name}: the reason says not measured: ${r.stderr}`);
}
});
test('[CNV] an empty scope list is exit 1 (the review scope is unknown), as engine-findings.mjs', () => {
const dir = reportDir();
writeReport(dir, { head: HEAD, base: BASE });
const scope = join(dir, 'scope.txt');
writeFileSync(scope, '\n');
const r = run(['convert', '--report-dir', dir, '--base', BASE, '--head', HEAD, '--scope', scope]);
assert.equal(r.status, 1, r.stdout + r.stderr);
assert.equal(r.stdout, '');
});
test('[CNV] bad usage is exit 2', () => {
const dir = reportDir();
writeReport(dir, { head: HEAD, base: BASE });
for (const [name, args] of [
['--head missing', ['convert', '--report-dir', dir, '--base', BASE]],
['--base not a full sha', ['convert', '--report-dir', dir, '--base', 'main', '--head', HEAD]],
['--report-dir missing', ['convert', '--base', BASE, '--head', HEAD]],
['a scope file that does not exist', ['convert', '--report-dir', dir, '--base', BASE, '--head', HEAD, '--scope', join(dir, 'none.txt')]],
]) {
const r = run(args);
assert.equal(r.status, 2, `${name}: exit 2, got ${r.status}: ${r.stdout}${r.stderr}`);
assert.equal(r.stdout, '', `${name}: nothing on stdout`);
}
});
// ---- ENGINES --------------------------------------------------------------------
test('[ENG3] ENGINES has claude-security: reviewer claude-security, the engines\' key, its own brief_ref', () => {
const s = ENGINES['claude-security'];
assert.ok(s, 'ENGINES has claude-security');
assert.equal(s.reviewer, 'claude-security');
assert.equal(s.rule_key, 'SECURITY_REVIEW_VULNERABILITY');
assert.equal(typeof s.brief_ref, 'string');
assert.notEqual(s.brief_ref, ENGINES['security-review'].brief_ref, 'its own brief_ref');
assert.match(s.brief_ref, /Claude Security/);
});
test('[ENG3] the action written for a finding without a fix names the engine that reported it', () => {
const bare = [{ file: 'src/run.mjs', line: 3, summary: 's' }];
const action = (engine) => {
const r = toReviewerPayload({ engine, findings: bare });
assert.equal(r.ok, true, `${engine}: ${r.errors.join('; ')}`);
return r.payload.findings[0].recommended_action;
};
assert.match(action('code-review'), /\/code-review reported/);
assert.match(action('security-review'), /\/security-review reported/);
assert.match(action('claude-security'), /Claude Security reported/);
assert.doesNotMatch(action('claude-security'), /\/code-review/, 'another engine is never called /code-review');
});
// ---- no engine: not measured, never ALLOW -----------------------------------------
// Copied from tests/lib/verdict-gate.test.mjs:29-74 (review), never imported. Changed:
// allow_blocked_by is the one argument.
function review(allowBlockedBy) {
const json = JSON.stringify({ verdict: 'WARN', counts: { BLOCKER: 0, MAJOR: 0, MINOR: 0, SUGGESTION: 0 }, allow_blocked_by: allowBlockedBy, findings: [] }, null, 2);
return `---
type: trekreview
review_version: "1.1"
task: "t"
slug: s
project_dir: .claude/projects/2026-11-08-s/
brief_path: .claude/projects/2026-11-08-s/brief.md
scope_sha_end: abc
reviewed_files_count: 1
verdict: WARN
findings: []
---
# Review: t
## Executive Summary
x
## Coverage
| File | Treatment | Reason |
|------|-----------|--------|
| a.mjs | deep-review | default |
## Remediation Summary
x
\`\`\`json
${json}
\`\`\`
`;
}
test('[NOE] no engine chosen: the expected reviewer security-engine never reports, so the review is not measured and the verdict gate exits 2, never ALLOW', () => {
assert.equal(NO_ENGINE_REVIEWER, 'security-engine');
const expected = ['brief-conformance-reviewer', 'code-correctness-reviewer', ENGINES['code-review'].reviewer, NO_ENGINE_REVIEWER];
const c = runContract([
{ reviewer: 'brief-conformance-reviewer', findings: [] },
{ reviewer: 'code-correctness-reviewer', findings: [] },
{ reviewer: ENGINES['code-review'].reviewer, findings: [] },
], { expectedReviewers: expected });
assert.deepEqual(c.missing_reviewers, ['security-engine']);
assert.ok(c.allow_blocked_by.includes('missing-reviewer:security-engine'), JSON.stringify(c.allow_blocked_by));
assert.notEqual(c.verdict, 'ALLOW');
const d = decideVerdict(review(c.allow_blocked_by));
assert.equal(d.exit, 2, d.reason);
assert.match(d.reason, /missing-reviewer:security-engine/);
});

3
tests/lib/threat-model.test.mjs
View file

@ -210,6 +210,9 @@ test('backedLine: only a security finding not from an engine can be outside (218
const engine = own({ engine: 'security-review' });
assert.equal(isOwnSecurityFinding(engine), false);
assert.equal(backedLine(engine, tm), null, 'an engine finding blocks as today');
const claudeSecurity = own({ engine: 'claude-security' });
assert.equal(isOwnSecurityFinding(claudeSecurity), false);
assert.equal(backedLine(claudeSecurity, tm), null, 'a Claude Security finding blocks as today (E 3.1)');
const enginesKey = own({ rule_key: 'SECURITY_REVIEW_VULNERABILITY' });
assert.equal(backedLine(enginesKey, tm), null, "the engines' key is never Voyage's own");
const notSecurity = own({ rule_key: 'MISSING_TEST' });

8
tests/review/brief-conflict-findings.test.mjs
View file

@ -372,11 +372,11 @@ test('review-coordinator Pass 3 — only an engine finding is marked, and the on
const coord = readDoc('agents', 'review-coordinator.md');
const p3 = cut(coord, '### Pass 3', 'In `quick` mode', 'Pass 3');
assert.match(p3, /`contradicts_brief: "<verbatim>"`/, 'the mark carries the Non-Goal or choice verbatim');
assert.match(p3, /engine finding \(`engine: code-review` or `security-review`\)/, 'only an engine finding is marked');
assert.match(p3, /engine finding \(`engine: code-review`, `security-review` or `claude-security`\)/, 'only an engine finding is marked');
assert.match(p3, /only an engine finding is ever marked/);
assert.match(p3, /only a `\/code-review` finding at the MAJOR tier and at severity MAJOR then drops to MINOR, with `brief_lowered_from: MAJOR`/,
'the one lowering, severity MAJOR required');
assert.match(p3, /a BLOCKER-tier, a MINOR-tier and any `\/security-review` finding only get the mark/);
assert.match(p3, /a BLOCKER-tier, a MINOR-tier and any finding from a security engine \(`\/security-review`, Claude Security\) only get the mark/);
assert.match(p3, /no engine finding is otherwise written below its catalogue tier/);
assert.match(p3, /`engine` is written only on a finding whose kept Pass 1 survivor itself came from an engine payload/);
assert.match(p3, /a reviewer-agent finding \(no `engine`\) and a finding whose `rule_key` is not in the catalogue are never marked/);
@ -412,7 +412,7 @@ test('review-coordinator hard rules — the brief re-evaluation, the severity fl
test('review-coordinator output — the field list and the trailing JSON example carry engine, contradicts_brief, brief_lowered_from', () => {
const coord = readDoc('agents', 'review-coordinator.md');
const fields = cut(coord, 'Each Findings subsection uses the', '### Trailing JSON block', 'field list');
assert.ok(fields.includes('- `- engine: {code-review | security-review, when the finding came from an engine}`'));
assert.ok(fields.includes('- `- engine: {code-review | security-review | claude-security, when the finding came from an engine}`'));
assert.ok(fields.includes('- `- contradicts_brief: {verbatim Non-Goal or choice, only when set}`'));
assert.ok(fields.includes('- `- brief_lowered_from: {MAJOR, only when lowered for the brief}`'));
assert.ok(!fields.includes('original_severity'), 'the brief rule does not read original_severity');
@ -605,7 +605,7 @@ test('own property — toReviewerPayload refuses engine toString with the "engin
const r = toReviewerPayload({ engine: 'toString', findings: [] });
assert.equal(r.ok, false, 'toString is not an engine');
assert.equal(r.errors.length, 1);
assert.match(r.errors[0], /^engine must be one of code-review\|security-review, got "toString"$/);
assert.match(r.errors[0], /^engine must be one of code-review\|security-review\|claude-security, got "toString"$/);
});
// ---- original_severity names what the kept finding asserted (review 08deae68) --