fix(review): a review with a control that did not run stops a merge

Operator decision 2026-10-06. Before, a WARN that existed only because a
reviewer or engine never reported (e.g. /security-review without an origin
remote) mapped to verdict-gate exit 0, so a review nobody performed in full
was green. Now `missing-reviewer:` / `unattributable-payload` in the JSON
block's allow_blocked_by is exit 2 (could not decide, never green); a WARN
for a MAJOR or a set-aside finding still passes with its reasons. A 1.1
review must carry allow_blocked_by.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-10-06 20:06:29 +02:00
commit 22e792ce19
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
7 changed files with 72 additions and 14 deletions

View file

@ -421,10 +421,14 @@ node lib/review/verdict-gate.mjs <dir>/review.md
# 0 = ALLOW or WARN · 1 = BLOCK · 2 = could not decide (never green)
```
A WARN that exists only because a check did not run is a 2, not a 0: a
review that was not performed in full stops a merge.
No finding is struck for length: only MINOR/SUGGESTION text is trimmed, and
counted; what the coordinator set aside is listed under `## Declined to
Judge`. An engine that could not run (older Claude Code, no `origin` remote)
keeps the review from coming back `ALLOW`, and the summary says why.
keeps the review from coming back `ALLOW` — and from passing the gate — and
the summary says why.
### Modes

View file

@ -214,7 +214,9 @@ Verdict is mechanical — never override. The verdict goes into the
frontmatter `verdict:` field (REQUIRED, K7), the trailing JSON block AND the
Executive Summary's first sentence — the same value in all three.
`lib/review/verdict-gate.mjs` reads the frontmatter and refuses (exit 2,
never green) a file whose JSON block or BLOCKER findings contradict it.
never green) a file whose JSON block or BLOCKER findings contradict it, and
a WARN whose `allow_blocked_by` names a reviewer or engine that did not run.
Always write `allow_blocked_by` (an empty list when nothing withheld ALLOW).
## Output: review.md content

View file

@ -168,7 +168,8 @@ required sections (`## Executive Summary`, `## Coverage`, `## Declined to
Judge` with `None.` and `Trimmed: 0`, `## Remediation Summary`), each saying
"No diff between {before_sha} and {after_sha}; nothing to review.", and the
trailing block `{"verdict": "ALLOW", "counts": {"BLOCKER": 0, "MAJOR": 0,
"MINOR": 0, "SUGGESTION": 0}, "findings": []}` — without it the verdict gate
"MINOR": 0, "SUGGESTION": 0}, "allow_blocked_by": [], "findings": []}` —
without it the verdict gate
cannot back the ALLOW and exits 2. There is nothing for the Phase 4.7 engine
to review, so its absence does not withhold ALLOW here. Skip Phases 4–7.
Continue to Phase 8 (validate + stats).
@ -594,9 +595,12 @@ Exit 0 = ALLOW or WARN; exit 1 = BLOCK; exit 2 = the gate could not decide
(missing verdict, a verdict the file contradicts — in its JSON block, its
`## Findings (BLOCKER)` section or its finding-id list — a JSON block that is
not the last fence or has the wrong shape, two verdict lines, unreadable
file) — **never green**. It takes exactly one file. On exit 0 with WARN,
`--json` also returns `warn_reasons` (the JSON block's `allow_blocked_by`);
print them. Capture the exit code as `gate_exit` and print it in Phase 9 exactly
file) — **never green**. A WARN that exists because a reviewer or engine
did not run (`missing-reviewer:` / `unattributable-payload` in
`allow_blocked_by`) is also exit 2: a review not performed in full stops a
merge (operator decision 2026-10-06). It takes exactly one file. On exit 0
with WARN, `--json` also returns `warn_reasons` (the JSON block's
`allow_blocked_by`); print them. Capture the exit code as `gate_exit` and print it in Phase 9 exactly
as returned. This is the command that later stops a merge or a release on
BLOCK; `/trekreview` reports it and never overrides it.
@ -715,7 +719,7 @@ After the write succeeds, print:
**Annotation HTML:** file://{$ANNOT_HTML}
**Scope:** {before_sha}..{after_sha} ({reviewed_files_count} files)
**Verdict:** {BLOCK | WARN | ALLOW}
**Gate:** verdict-gate exit {gate_exit} ({0 = ALLOW/WARN | 1 = BLOCK | 2 = could not decide — not green})
**Gate:** verdict-gate exit {gate_exit} ({0 = ALLOW/WARN | 1 = BLOCK | 2 = could not decide, or a control did not run — not green})
**Engine:** /code-review {delivered N | not run: reason} · /security-review {delivered N | not run: reason}
### Counts

View file

@ -336,7 +336,7 @@ The strongest validation in the entire pipeline. Phase 5.5 (planning-orchestrato
**Consumer:** `/trekplan` Phase 1 when `--brief review.md` is supplied and the consumer detects `type: trekreview` in frontmatter. The plan command branches into a remediation-plan path: BLOCKER + MAJOR findings become plan goals, the produced `plan.md` carries a `source_findings: [<id>, ...]` frontmatter list as the audit trail back to the consumed findings. MINOR + SUGGESTION are skipped for plan-input. Since 1.1 every BLOCKER is also written as a numbered `### Step R<n>` under `## Remediation Steps` (Files · Source finding · Test first · Verify · On failure), so a consumer can append it as a plan step without rewording it.
**Second consumer — the verdict gate (K7, 1.1):** `lib/review/verdict-gate.mjs <review.md>` reads the frontmatter `verdict` and exits 0 = ALLOW/WARN, 1 = BLOCK, 2 = could not decide. Exit 2 is never green: a missing file, a missing or unknown verdict, two verdict lines, a trailing JSON block that is not the last fence or lacks `verdict`/`counts`/`findings`, and a file whose JSON block, `## Findings (BLOCKER)` section or finding-id list contradicts a greener verdict all land there. Exactly one file per call. Whatever stops a merge or a release on a review reads this exit code, never a summary. Borrowed: the verdict goes to a file and a command gates on it (Kiro `feature-pipeline`); the executor reads the verdict but never writes it (Spec Kit).
**Second consumer — the verdict gate (K7, 1.1):** `lib/review/verdict-gate.mjs <review.md>` reads the frontmatter `verdict` and exits 0 = ALLOW/WARN, 1 = BLOCK, 2 = could not decide. Exit 2 is never green: a missing file, a missing or unknown verdict, two verdict lines, a trailing JSON block that is not the last fence or lacks `verdict`/`counts`/`findings`, and a file whose JSON block, `## Findings (BLOCKER)` section or finding-id list contradicts a greener verdict all land there — and so does a WARN that exists because a reviewer or engine did not run (`missing-reviewer:` / `unattributable-payload` in the JSON block's `allow_blocked_by`, which a 1.1 review must carry): a review not performed in full stops a merge (operator decision 2026-10-06). Exactly one file per call. Whatever stops a merge or a release on a review reads this exit code, never a summary. Borrowed: the verdict goes to a file and a command gates on it (Kiro `feature-pipeline`); the executor reads the verdict but never writes it (Spec Kit).
**Path conventions:**
- Project-dir mode (recommended): `{project_dir}/review.md` (one per review iteration; subsequent runs overwrite atomically).

View file

@ -8,7 +8,10 @@
// 0 = ALLOW or WARN
// 1 = BLOCK
// 2 = could not decide: missing/unreadable file, not a review, no verdict,
// an unknown verdict, or a file that contradicts its own verdict.
// an unknown verdict, a file that contradicts its own verdict, or a
// WARN because a reviewer or engine did not run (`missing-reviewer:` /
// `unattributable-payload` in allow_blocked_by — operator decision
// 2026-10-06: a review not performed in full stops a merge).
// Exit 2 is NEVER green.
//
// Fail-closed reading: a missing `verdict` is exit 2, never ALLOW. A verdict
@ -96,10 +99,25 @@ export function decideVerdict(text) {
if (fmIds.size !== jsonIds.size || [...fmIds].some((id) => !jsonIds.has(id))) {
return undecided('frontmatter findings: list and the JSON block finding ids differ', verdict);
}
const warnReasons = Array.isArray(payload.allow_blocked_by) ? payload.allow_blocked_by : [];
// A review in which a reviewer or engine never reported was not performed in
// full: that is "could not decide", not a passable warning (operator
// decision 2026-10-06, ordre 5). A 1.1 review must carry the list.
if (atLeastVersion(fm.review_version, 1, 1) && !Array.isArray(payload.allow_blocked_by)) {
return undecided('trailing JSON block lacks allow_blocked_by (required from review_version 1.1)', verdict);
}
const warnReasons = Array.isArray(payload.allow_blocked_by) ? payload.allow_blocked_by.map(String) : [];
const notRun = warnReasons.filter((w) => w.startsWith('missing-reviewer:') || w.startsWith('unattributable-payload'));
if (notRun.length > 0) {
return undecided(`a control did not run: ${notRun.join(', ')}`, verdict);
}
return { exit: EXIT.PASS, verdict, reason: `verdict ${verdict}`, warn_reasons: warnReasons };
}
function atLeastVersion(v, major, minor) {
const m = typeof v === 'string' ? v.match(/^(\d+)\.(\d+)$/) : null;
return Boolean(m) && (Number(m[1]) > major || (Number(m[1]) === major && Number(m[2]) >= minor));
}
const FENCE_RE = /^```[ \t]*([A-Za-z0-9_+-]*)[^\n]*\n[\s\S]*?^```[ \t]*$/gm;
/** Language tag of the LAST fenced block in `body` (lower-cased), '' for none, null when no fence. */

View file

@ -86,6 +86,11 @@ test('trekreview — Phase 8 decides with lib/review/verdict-gate.mjs and calls
assert.match(p, /never\s+green/);
});
test('gate — a control that did not run stops a merge (operator decision 2026-10-06) is written down', () => {
assert.match(section(CMD, '## Phase 8 —'), /a review not performed in full stops a\s+merge/);
assert.match(section(read('docs/HANDOVER-CONTRACTS.md'), '## Handover 6'), /did not run/);
});
test('trekreview — a BLOCKER is written as a numbered Step R<n> with Source finding (Handover 6)', () => {
assert.ok(CMD.includes('### Step R<n>'));
assert.ok(COORD.includes('**Source finding:**'));
@ -107,7 +112,8 @@ test('template — templates/trekreview-template.md validates strictly and the g
test('trekreview — the zero-diff review the prose prescribes passes the gate as ALLOW', async () => {
const { decideVerdict } = await import('../../lib/review/verdict-gate.mjs');
const p3 = section(CMD, '## Phase 3');
assert.ok(p3.includes('findings: []') && p3.includes('"findings": []'), 'Phase 3 prescribes both');
assert.ok(p3.includes('findings: []') && p3.includes('"findings": []') && p3.includes('"allow_blocked_by": []'),
'Phase 3 prescribes the empty list, the JSON block and allow_blocked_by');
const text = `---
type: trekreview
review_version: "1.1"
@ -140,7 +146,7 @@ Trimmed: 0
No diff.
\`\`\`json
{"verdict": "ALLOW", "counts": {"BLOCKER": 0, "MAJOR": 0, "MINOR": 0, "SUGGESTION": 0}, "findings": []}
{"verdict": "ALLOW", "counts": {"BLOCKER": 0, "MAJOR": 0, "MINOR": 0, "SUGGESTION": 0}, "allow_blocked_by": [], "findings": []}
\`\`\`
`;
assert.equal(decideVerdict(text).exit, 0, decideVerdict(text).reason);

View file

@ -16,7 +16,7 @@ const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..', '..');
const GATE = join(ROOT, 'lib', 'review', 'verdict-gate.mjs');
const { decideVerdict } = existsSync(GATE) ? await import('../../lib/review/verdict-gate.mjs') : {};
function review({ verdict, jsonVerdict, severities = [] }) {
function review({ verdict, jsonVerdict, severities = [], allowBlockedBy = [] }) {
const ids = severities.map((_, i) => String(i + 1).padStart(40, 'a'));
const fmVerdict = verdict === undefined ? '' : `verdict: ${verdict}\n`;
const findings = severities.map((s, i) => ({
@ -24,7 +24,7 @@ function review({ verdict, jsonVerdict, severities = [] }) {
}));
const counts = { BLOCKER: 0, MAJOR: 0, MINOR: 0, SUGGESTION: 0 };
for (const s of severities) counts[s] += 1;
const json = JSON.stringify({ verdict: jsonVerdict ?? verdict, counts, findings }, null, 2);
const json = JSON.stringify({ verdict: jsonVerdict ?? verdict, counts, allow_blocked_by: allowBlockedBy, findings }, null, 2);
return `---
type: trekreview
review_version: "1.1"
@ -182,6 +182,30 @@ test('verdict-gate — frontmatter finding ids that differ from the JSON block a
assert.equal(decideVerdict(t).exit, 2);
});
// ---- Operator decision 2026-10-06: a control that did not run stops a merge ----
test('verdict-gate — WARN because a reviewer or engine did not run is exit 2, not a passable warning', () => {
for (const reason of ['missing-reviewer:claude-security-review', 'unattributable-payload (1)']) {
const t = review({ verdict: 'WARN', severities: ['MINOR'], allowBlockedBy: [reason] });
const d = decideVerdict(t);
assert.equal(d.exit, 2, reason);
assert.match(d.reason, /did not run/);
}
});
test('verdict-gate — WARN for a set-aside finding or a MAJOR still passes (exit 0) with its reasons', () => {
const d = decideVerdict(review({ verdict: 'WARN', severities: ['MAJOR'], allowBlockedBy: ['unverified:actionability:empty (1)'] }));
assert.equal(d.exit, 0);
assert.deepEqual(d.warn_reasons, ['unverified:actionability:empty (1)']);
});
test('verdict-gate — a 1.1 review whose JSON block lacks allow_blocked_by cannot pass', () => {
const base = review({ verdict: 'WARN', severities: ['MAJOR'] });
const t = base.replace(/\s*"allow_blocked_by": \[\],?/, '');
assert.notEqual(t, base, 'fixture control');
assert.equal(decideVerdict(t).exit, 2);
});
test('verdict-gate — more than one file argument is refused (exit 2), never a first-match decision', () => {
const dir = mkdtempSync(join(tmpdir(), 'verdict-gate-'));
try {