fix(review): a review with a control that did not run stops a merge
Operator decision 2026-10-06. Before, a WARN that existed only because a reviewer or engine never reported (e.g. /security-review without an origin remote) mapped to verdict-gate exit 0, so a review nobody performed in full was green. Now `missing-reviewer:` / `unattributable-payload` in the JSON block's allow_blocked_by is exit 2 (could not decide, never green); a WARN for a MAJOR or a set-aside finding still passes with its reasons. A 1.1 review must carry allow_blocked_by. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
40f6ae1177
commit
22e792ce19
7 changed files with 72 additions and 14 deletions
|
|
@ -421,10 +421,14 @@ node lib/review/verdict-gate.mjs <dir>/review.md
|
|||
# 0 = ALLOW or WARN · 1 = BLOCK · 2 = could not decide (never green)
|
||||
```
|
||||
|
||||
A WARN that exists only because a check did not run is a 2, not a 0: a
|
||||
review that was not performed in full stops a merge.
|
||||
|
||||
No finding is struck for length: only MINOR/SUGGESTION text is trimmed, and
|
||||
counted; what the coordinator set aside is listed under `## Declined to
|
||||
Judge`. An engine that could not run (older Claude Code, no `origin` remote)
|
||||
keeps the review from coming back `ALLOW`, and the summary says why.
|
||||
keeps the review from coming back `ALLOW` — and from passing the gate — and
|
||||
the summary says why.
|
||||
|
||||
### Modes
|
||||
|
||||
|
|
|
|||
|
|
@ -214,7 +214,9 @@ Verdict is mechanical — never override. The verdict goes into the
|
|||
frontmatter `verdict:` field (REQUIRED, K7), the trailing JSON block AND the
|
||||
Executive Summary's first sentence — the same value in all three.
|
||||
`lib/review/verdict-gate.mjs` reads the frontmatter and refuses (exit 2,
|
||||
never green) a file whose JSON block or BLOCKER findings contradict it.
|
||||
never green) a file whose JSON block or BLOCKER findings contradict it, and
|
||||
a WARN whose `allow_blocked_by` names a reviewer or engine that did not run.
|
||||
Always write `allow_blocked_by` (an empty list when nothing withheld ALLOW).
|
||||
|
||||
## Output: review.md content
|
||||
|
||||
|
|
|
|||
|
|
@ -168,7 +168,8 @@ required sections (`## Executive Summary`, `## Coverage`, `## Declined to
|
|||
Judge` with `None.` and `Trimmed: 0`, `## Remediation Summary`), each saying
|
||||
"No diff between {before_sha} and {after_sha}; nothing to review.", and the
|
||||
trailing block `{"verdict": "ALLOW", "counts": {"BLOCKER": 0, "MAJOR": 0,
|
||||
"MINOR": 0, "SUGGESTION": 0}, "findings": []}` — without it the verdict gate
|
||||
"MINOR": 0, "SUGGESTION": 0}, "allow_blocked_by": [], "findings": []}` —
|
||||
without it the verdict gate
|
||||
cannot back the ALLOW and exits 2. There is nothing for the Phase 4.7 engine
|
||||
to review, so its absence does not withhold ALLOW here. Skip Phases 4–7.
|
||||
Continue to Phase 8 (validate + stats).
|
||||
|
|
@ -594,9 +595,12 @@ Exit 0 = ALLOW or WARN; exit 1 = BLOCK; exit 2 = the gate could not decide
|
|||
(missing verdict, a verdict the file contradicts — in its JSON block, its
|
||||
`## Findings (BLOCKER)` section or its finding-id list — a JSON block that is
|
||||
not the last fence or has the wrong shape, two verdict lines, unreadable
|
||||
file) — **never green**. It takes exactly one file. On exit 0 with WARN,
|
||||
`--json` also returns `warn_reasons` (the JSON block's `allow_blocked_by`);
|
||||
print them. Capture the exit code as `gate_exit` and print it in Phase 9 exactly
|
||||
file) — **never green**. A WARN that exists because a reviewer or engine
|
||||
did not run (`missing-reviewer:` / `unattributable-payload` in
|
||||
`allow_blocked_by`) is also exit 2: a review not performed in full stops a
|
||||
merge (operator decision 2026-10-06). It takes exactly one file. On exit 0
|
||||
with WARN, `--json` also returns `warn_reasons` (the JSON block's
|
||||
`allow_blocked_by`); print them. Capture the exit code as `gate_exit` and print it in Phase 9 exactly
|
||||
as returned. This is the command that later stops a merge or a release on
|
||||
BLOCK; `/trekreview` reports it and never overrides it.
|
||||
|
||||
|
|
@ -715,7 +719,7 @@ After the write succeeds, print:
|
|||
**Annotation HTML:** file://{$ANNOT_HTML}
|
||||
**Scope:** {before_sha}..{after_sha} ({reviewed_files_count} files)
|
||||
**Verdict:** {BLOCK | WARN | ALLOW}
|
||||
**Gate:** verdict-gate exit {gate_exit} ({0 = ALLOW/WARN | 1 = BLOCK | 2 = could not decide — not green})
|
||||
**Gate:** verdict-gate exit {gate_exit} ({0 = ALLOW/WARN | 1 = BLOCK | 2 = could not decide, or a control did not run — not green})
|
||||
**Engine:** /code-review {delivered N | not run: reason} · /security-review {delivered N | not run: reason}
|
||||
|
||||
### Counts
|
||||
|
|
|
|||
|
|
@ -336,7 +336,7 @@ The strongest validation in the entire pipeline. Phase 5.5 (planning-orchestrato
|
|||
|
||||
**Consumer:** `/trekplan` Phase 1 when `--brief review.md` is supplied and the consumer detects `type: trekreview` in frontmatter. The plan command branches into a remediation-plan path: BLOCKER + MAJOR findings become plan goals, the produced `plan.md` carries a `source_findings: [<id>, ...]` frontmatter list as the audit trail back to the consumed findings. MINOR + SUGGESTION are skipped for plan-input. Since 1.1 every BLOCKER is also written as a numbered `### Step R<n>` under `## Remediation Steps` (Files · Source finding · Test first · Verify · On failure), so a consumer can append it as a plan step without rewording it.
|
||||
|
||||
**Second consumer — the verdict gate (K7, 1.1):** `lib/review/verdict-gate.mjs <review.md>` reads the frontmatter `verdict` and exits 0 = ALLOW/WARN, 1 = BLOCK, 2 = could not decide. Exit 2 is never green: a missing file, a missing or unknown verdict, two verdict lines, a trailing JSON block that is not the last fence or lacks `verdict`/`counts`/`findings`, and a file whose JSON block, `## Findings (BLOCKER)` section or finding-id list contradicts a greener verdict all land there. Exactly one file per call. Whatever stops a merge or a release on a review reads this exit code, never a summary. Borrowed: the verdict goes to a file and a command gates on it (Kiro `feature-pipeline`); the executor reads the verdict but never writes it (Spec Kit).
|
||||
**Second consumer — the verdict gate (K7, 1.1):** `lib/review/verdict-gate.mjs <review.md>` reads the frontmatter `verdict` and exits 0 = ALLOW/WARN, 1 = BLOCK, 2 = could not decide. Exit 2 is never green: a missing file, a missing or unknown verdict, two verdict lines, a trailing JSON block that is not the last fence or lacks `verdict`/`counts`/`findings`, and a file whose JSON block, `## Findings (BLOCKER)` section or finding-id list contradicts a greener verdict all land there — and so does a WARN that exists because a reviewer or engine did not run (`missing-reviewer:` / `unattributable-payload` in the JSON block's `allow_blocked_by`, which a 1.1 review must carry): a review not performed in full stops a merge (operator decision 2026-10-06). Exactly one file per call. Whatever stops a merge or a release on a review reads this exit code, never a summary. Borrowed: the verdict goes to a file and a command gates on it (Kiro `feature-pipeline`); the executor reads the verdict but never writes it (Spec Kit).
|
||||
|
||||
**Path conventions:**
|
||||
- Project-dir mode (recommended): `{project_dir}/review.md` (one per review iteration; subsequent runs overwrite atomically).
|
||||
|
|
|
|||
|
|
@ -8,7 +8,10 @@
|
|||
// 0 = ALLOW or WARN
|
||||
// 1 = BLOCK
|
||||
// 2 = could not decide: missing/unreadable file, not a review, no verdict,
|
||||
// an unknown verdict, or a file that contradicts its own verdict.
|
||||
// an unknown verdict, a file that contradicts its own verdict, or a
|
||||
// WARN because a reviewer or engine did not run (`missing-reviewer:` /
|
||||
// `unattributable-payload` in allow_blocked_by — operator decision
|
||||
// 2026-10-06: a review not performed in full stops a merge).
|
||||
// Exit 2 is NEVER green.
|
||||
//
|
||||
// Fail-closed reading: a missing `verdict` is exit 2, never ALLOW. A verdict
|
||||
|
|
@ -96,10 +99,25 @@ export function decideVerdict(text) {
|
|||
if (fmIds.size !== jsonIds.size || [...fmIds].some((id) => !jsonIds.has(id))) {
|
||||
return undecided('frontmatter findings: list and the JSON block finding ids differ', verdict);
|
||||
}
|
||||
const warnReasons = Array.isArray(payload.allow_blocked_by) ? payload.allow_blocked_by : [];
|
||||
// A review in which a reviewer or engine never reported was not performed in
|
||||
// full: that is "could not decide", not a passable warning (operator
|
||||
// decision 2026-10-06, ordre 5). A 1.1 review must carry the list.
|
||||
if (atLeastVersion(fm.review_version, 1, 1) && !Array.isArray(payload.allow_blocked_by)) {
|
||||
return undecided('trailing JSON block lacks allow_blocked_by (required from review_version 1.1)', verdict);
|
||||
}
|
||||
const warnReasons = Array.isArray(payload.allow_blocked_by) ? payload.allow_blocked_by.map(String) : [];
|
||||
const notRun = warnReasons.filter((w) => w.startsWith('missing-reviewer:') || w.startsWith('unattributable-payload'));
|
||||
if (notRun.length > 0) {
|
||||
return undecided(`a control did not run: ${notRun.join(', ')}`, verdict);
|
||||
}
|
||||
return { exit: EXIT.PASS, verdict, reason: `verdict ${verdict}`, warn_reasons: warnReasons };
|
||||
}
|
||||
|
||||
function atLeastVersion(v, major, minor) {
|
||||
const m = typeof v === 'string' ? v.match(/^(\d+)\.(\d+)$/) : null;
|
||||
return Boolean(m) && (Number(m[1]) > major || (Number(m[1]) === major && Number(m[2]) >= minor));
|
||||
}
|
||||
|
||||
const FENCE_RE = /^```[ \t]*([A-Za-z0-9_+-]*)[^\n]*\n[\s\S]*?^```[ \t]*$/gm;
|
||||
|
||||
/** Language tag of the LAST fenced block in `body` (lower-cased), '' for none, null when no fence. */
|
||||
|
|
|
|||
|
|
@ -86,6 +86,11 @@ test('trekreview — Phase 8 decides with lib/review/verdict-gate.mjs and calls
|
|||
assert.match(p, /never\s+green/);
|
||||
});
|
||||
|
||||
test('gate — a control that did not run stops a merge (operator decision 2026-10-06) is written down', () => {
|
||||
assert.match(section(CMD, '## Phase 8 —'), /a review not performed in full stops a\s+merge/);
|
||||
assert.match(section(read('docs/HANDOVER-CONTRACTS.md'), '## Handover 6'), /did not run/);
|
||||
});
|
||||
|
||||
test('trekreview — a BLOCKER is written as a numbered Step R<n> with Source finding (Handover 6)', () => {
|
||||
assert.ok(CMD.includes('### Step R<n>'));
|
||||
assert.ok(COORD.includes('**Source finding:**'));
|
||||
|
|
@ -107,7 +112,8 @@ test('template — templates/trekreview-template.md validates strictly and the g
|
|||
test('trekreview — the zero-diff review the prose prescribes passes the gate as ALLOW', async () => {
|
||||
const { decideVerdict } = await import('../../lib/review/verdict-gate.mjs');
|
||||
const p3 = section(CMD, '## Phase 3');
|
||||
assert.ok(p3.includes('findings: []') && p3.includes('"findings": []'), 'Phase 3 prescribes both');
|
||||
assert.ok(p3.includes('findings: []') && p3.includes('"findings": []') && p3.includes('"allow_blocked_by": []'),
|
||||
'Phase 3 prescribes the empty list, the JSON block and allow_blocked_by');
|
||||
const text = `---
|
||||
type: trekreview
|
||||
review_version: "1.1"
|
||||
|
|
@ -140,7 +146,7 @@ Trimmed: 0
|
|||
No diff.
|
||||
|
||||
\`\`\`json
|
||||
{"verdict": "ALLOW", "counts": {"BLOCKER": 0, "MAJOR": 0, "MINOR": 0, "SUGGESTION": 0}, "findings": []}
|
||||
{"verdict": "ALLOW", "counts": {"BLOCKER": 0, "MAJOR": 0, "MINOR": 0, "SUGGESTION": 0}, "allow_blocked_by": [], "findings": []}
|
||||
\`\`\`
|
||||
`;
|
||||
assert.equal(decideVerdict(text).exit, 0, decideVerdict(text).reason);
|
||||
|
|
|
|||
|
|
@ -16,7 +16,7 @@ const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..', '..');
|
|||
const GATE = join(ROOT, 'lib', 'review', 'verdict-gate.mjs');
|
||||
const { decideVerdict } = existsSync(GATE) ? await import('../../lib/review/verdict-gate.mjs') : {};
|
||||
|
||||
function review({ verdict, jsonVerdict, severities = [] }) {
|
||||
function review({ verdict, jsonVerdict, severities = [], allowBlockedBy = [] }) {
|
||||
const ids = severities.map((_, i) => String(i + 1).padStart(40, 'a'));
|
||||
const fmVerdict = verdict === undefined ? '' : `verdict: ${verdict}\n`;
|
||||
const findings = severities.map((s, i) => ({
|
||||
|
|
@ -24,7 +24,7 @@ function review({ verdict, jsonVerdict, severities = [] }) {
|
|||
}));
|
||||
const counts = { BLOCKER: 0, MAJOR: 0, MINOR: 0, SUGGESTION: 0 };
|
||||
for (const s of severities) counts[s] += 1;
|
||||
const json = JSON.stringify({ verdict: jsonVerdict ?? verdict, counts, findings }, null, 2);
|
||||
const json = JSON.stringify({ verdict: jsonVerdict ?? verdict, counts, allow_blocked_by: allowBlockedBy, findings }, null, 2);
|
||||
return `---
|
||||
type: trekreview
|
||||
review_version: "1.1"
|
||||
|
|
@ -182,6 +182,30 @@ test('verdict-gate — frontmatter finding ids that differ from the JSON block a
|
|||
assert.equal(decideVerdict(t).exit, 2);
|
||||
});
|
||||
|
||||
// ---- Operator decision 2026-10-06: a control that did not run stops a merge ----
|
||||
|
||||
test('verdict-gate — WARN because a reviewer or engine did not run is exit 2, not a passable warning', () => {
|
||||
for (const reason of ['missing-reviewer:claude-security-review', 'unattributable-payload (1)']) {
|
||||
const t = review({ verdict: 'WARN', severities: ['MINOR'], allowBlockedBy: [reason] });
|
||||
const d = decideVerdict(t);
|
||||
assert.equal(d.exit, 2, reason);
|
||||
assert.match(d.reason, /did not run/);
|
||||
}
|
||||
});
|
||||
|
||||
test('verdict-gate — WARN for a set-aside finding or a MAJOR still passes (exit 0) with its reasons', () => {
|
||||
const d = decideVerdict(review({ verdict: 'WARN', severities: ['MAJOR'], allowBlockedBy: ['unverified:actionability:empty (1)'] }));
|
||||
assert.equal(d.exit, 0);
|
||||
assert.deepEqual(d.warn_reasons, ['unverified:actionability:empty (1)']);
|
||||
});
|
||||
|
||||
test('verdict-gate — a 1.1 review whose JSON block lacks allow_blocked_by cannot pass', () => {
|
||||
const base = review({ verdict: 'WARN', severities: ['MAJOR'] });
|
||||
const t = base.replace(/\s*"allow_blocked_by": \[\],?/, '');
|
||||
assert.notEqual(t, base, 'fixture control');
|
||||
assert.equal(decideVerdict(t).exit, 2);
|
||||
});
|
||||
|
||||
test('verdict-gate — more than one file argument is refused (exit 2), never a first-match decision', () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'verdict-gate-'));
|
||||
try {
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue