fix(review): the workflow review fails closed when its coordinator returns nothing

Review finding edf2ec56; every reviewer that did not run is named in allow_blocked_by.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-10-14 11:20:28 +02:00
commit 25856f2e9c
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
4 changed files with 166 additions and 8 deletions

View file

@ -544,8 +544,8 @@ Workflow({
})
```
A reviewer that returns nothing, or an engine listed in `engineNotRun`,
forbids `ALLOW` on this path too (`missing_reviewers` in the result).
A reviewer that returns nothing, an engine listed in `engineNotRun`, or a coordinator that returns no verdict and findings forbids `ALLOW` on this path too (`missing_reviewers` in the result), and the result's `allow_blocked_by` names each one (`missing-reviewer:<name>`, the coordinator as `missing-reviewer:review-coordinator`);
Phase 7 writes that list as the trailing JSON block's `allow_blocked_by`, so the verdict gate reads such a WARN review as not performed in full (exit 2; a BLOCK stays exit 1).
On this path dedup ranks as Pass 1 does (both paths), every engine finding the
coordinator did not return at its exact `(file, line, rule_key)` is put back at its own
severity (operator decision 23), and a returned finding is never below the lower of its
@ -567,7 +567,7 @@ Contract (verified in S10 part B — follow exactly):
Then continue at **Phase 7** exactly as the prose path does — Phase 7 rendering, Phase 8
validation, and the operator gate are **shared** and substrate-independent (both paths
return the same `{verdict, findings}` shape). The workflow's returned findings keep
return the same `{verdict, findings}` shape, and the workflow's `allow_blocked_by` goes into the trailing JSON block). The workflow's returned findings keep
`engine` (set by the script from the payload a finding came from, never from a label),
`contradicts_brief` and `brief_lowered_from`, and Phase 7 writes them into each
finding's field list and the trailing JSON block of review.md exactly as the prose path

View file

@ -478,7 +478,7 @@ The gate cannot check an `engine` label on the prose path: there the coordinator
the workflow path (`--workflow`) takes `engine` from the payload source and writes it into the returned finding.
`runContract` (`lib/review/coordinator-contract.mjs`) likewise takes `engine` from the payload a finding arrived in, never from the finding's own field,
and in dedup the survivor is ranked by catalogue tier (for a catalogue key it always ties inside one triplet), so a reviewer-agent finding survives over an engine finding and carries the highest severity the group asserted; an engine's mark never lowers what a reviewer agent asserted.
On the workflow path dedup ranks as Pass 1 does (both paths), every engine finding the coordinator did not return at its exact `(file, line, rule_key)` is put back at its own severity (operator decision 23), and a returned finding is never below the lower of its reviewer counterpart's severity and its catalogue tier (floor (b)), so the workflow path is stricter than the prose path.
On the workflow path dedup ranks as Pass 1 does (both paths), every engine finding the coordinator did not return at its exact `(file, line, rule_key)` is put back at its own severity (operator decision 23), and a returned finding is never below the lower of its reviewer counterpart's severity and its catalogue tier (floor (b)), so the workflow path is stricter than the prose path; a coordinator that returns no verdict and findings counts as a missing reviewer there (`missing-reviewer:review-coordinator` in `allow_blocked_by`), and its findings are read at their catalogue tier or higher, so the review is at least WARN and the verdict gate exits 2 on such a WARN (a BLOCK stays exit 1).
A plan made from the review skips a lowered finding: it is MINOR, and `/trekplan` takes BLOCKER and MAJOR findings only.
**Validation strategy:**

View file

@ -21,7 +21,9 @@
// Ordre 5: the Claude Code engine payloads (Phase 4.7, run in prose before this
// script) are merged with the reviewers' findings; a reviewer or engine that
// returned nothing forbids ALLOW (fail-closed, as on the prose path); and no
// finding is struck for length.
// finding is struck for length. A coordinator that returns no verdict and
// findings counts as a missing reviewer, and every reviewer, engine and
// coordinator that did not deliver is named in `allow_blocked_by` (review edf2ec56).
//
// Run via the Workflow tool: Workflow({ scriptPath: this file, args }).
// The structured result is logged as a single JSON line (prefix RESULT_JSON:)
@ -217,9 +219,12 @@ const reviewerThunks = [
agentType: 'voyage:code-correctness-reviewer', schema: FINDINGS_SCHEMA, label: 'correctness', phase: 'Review',
}),
];
// The reviewer agent names in run order, so a reviewer that returned nothing is named.
const reviewerNames = a.quick ? ['code-correctness-reviewer'] : ['brief-conformance-reviewer', 'code-correctness-reviewer'];
const toRun = a.quick ? [reviewerThunks[1]] : reviewerThunks;
const rawResults = await parallel(toRun);
const reviewerResults = rawResults.filter(Boolean);
const missingReviewerNames = reviewerNames.filter((_, i) => !rawResults[i]);
// A reviewer that returned nothing is an absent one, not a clean one. The
// engines that were due are counted from what actually arrived, never from
// what the caller remembered to list as not run.
@ -227,7 +232,7 @@ const enginePayloads = Array.isArray(a.enginePayloads) ? a.enginePayloads.filter
const expectedEngines = a.quick ? ['claude-code-review'] : ['claude-code-review', 'claude-security-review'];
const arrivedEngines = new Set(enginePayloads.map((p) => p.reviewer));
const missingEngines = expectedEngines.filter((name) => !arrivedEngines.has(name));
const missingReviewers = (rawResults.length - reviewerResults.length) + missingEngines.length;
let missingReviewers = (rawResults.length - reviewerResults.length) + missingEngines.length;
// Provenance, before dedup: "came from an engine" is read from the source,
// never from an `engine` field. A reviewer finding loses any `engine` it
// carries; an engine payload's findings get `engine` from its `reviewer`
@ -260,10 +265,14 @@ if (merged.length === 0) {
agentType: 'voyage:review-coordinator', schema: VERDICT_SCHEMA, label: 'coordinator', phase: 'Coordinate',
});
}
// A coordinator that returns no verdict and findings is a missing reviewer, not
// a clean one (the zero-findings skip above always sets a valid verdict).
const coordinatorMissing = merged.length > 0 && !(coord && ['BLOCK', 'WARN', 'ALLOW'].includes(coord.verdict) && Array.isArray(coord.findings));
if (coordinatorMissing) missingReviewers += 1;
// The verdict is mechanical: recompute its floor from the findings the
// coordinator returned, so an LLM-written ALLOW over a BLOCKER cannot pass.
const returned = Array.isArray(coord?.findings) ? coord.findings : merged;
const returned = coordinatorMissing ? merged : coord.findings;
// The floor reads two inputs, never the labels the coordinator wrote:
// - `merged` (the coordinator's input, same file/line/rule_key) for each
// returned finding's counterpart: (a) an engine counterpart raises it to the
@ -280,12 +289,23 @@ const returned = Array.isArray(coord?.findings) ? coord.findings : merged;
// move or relabel of an engine finding no longer lowers the verdict.
// The raises and put-backs are written into `returned`, the marks kept, so the
// rendered review.md carries them.
// A coordinator that failed (no verdict and findings) is not trusted with a
// partial list: `returned` is then every merged finding, each first raised to
// its catalogue tier when the key is known (never lowered), as the default
// path's Pass 3 would reset it; that raise stands after `raiseTo` is declared.
const counterparts = new Map(merged
.filter((m) => m && typeof m === 'object')
.map((m) => [tripletOf(m), m]));
const present = (f, field) => f[field] !== undefined && f[field] !== null;
const raiseTo = (f, sev) => { if (sevRank(sev) < sevRank(f.severity)) f.severity = sev; };
const lowerOf = (s1, s2) => (sevRank(s1) >= sevRank(s2) ? s1 : s2);
if (coordinatorMissing) {
for (const f of returned) {
if (!f || typeof f !== 'object') continue;
const tier = tierOf(f.rule_key);
if (tier !== null) raiseTo(f, tier);
}
}
for (const f of returned) {
if (!f || typeof f !== 'object') continue;
const c = counterparts.get(tripletOf(f));
@ -321,15 +341,23 @@ for (const f of putBack) {
returned.push(copy);
}
const has = (sev) => returned.some((f) => f && f.severity === sev);
let verdict = coord?.verdict ?? null;
let verdict = coordinatorMissing ? (coord?.verdict === 'BLOCK' ? 'BLOCK' : 'WARN') : coord.verdict;
if (has('BLOCKER')) verdict = 'BLOCK';
else if (has('MAJOR') && verdict !== 'BLOCK') verdict = 'WARN';
if (verdict === 'ALLOW' && missingReviewers > 0) verdict = 'WARN';
// Every reviewer, engine and coordinator that did not deliver, in the form the
// default path writes and lib/review/verdict-gate.mjs counts as not run.
const allowBlockedBy = [
...missingReviewerNames.map((name) => `missing-reviewer:${name}`),
...missingEngines.map((name) => `missing-reviewer:${name}`),
...(coordinatorMissing ? ['missing-reviewer:review-coordinator'] : []),
];
const result = {
verdict,
missing_reviewers: missingReviewers,
missing_engines: missingEngines,
allow_blocked_by: allowBlockedBy,
findings: returned,
raw_reviewer_count: reviewerResults.length,
raw_finding_count: allFindings.length,

View file

@ -82,6 +82,136 @@ test('armB — quick mode expects only the /code-review engine', async () => {
assert.deepEqual(r.missing_engines, []);
});
// ---- Review edf2ec56: a coordinator that returns nothing fails closed ----------------
//
// A coordinator that returns no verdict and findings counts as a missing reviewer:
// its partial list is not trusted (the floors read every merged finding, each raised
// to its catalogue tier, never lowered), ALLOW is forbidden, a BLOCK is never lowered,
// and every reviewer, engine and coordinator that did not deliver is named in
// `allow_blocked_by` as `missing-reviewer:<name>`.
const COORD_MISSING = 'missing-reviewer:review-coordinator';
const MEH_MINOR = () => ({ severity: 'MINOR', rule_key: 'MISSING_ERROR_HANDLING', file: 'src/r.mjs', line: 4, title: 't', detail: 'd' });
const CRI_MINOR = () => ({ severity: 'MINOR', rule_key: 'CODE_REVIEW_IMPROVEMENT', file: 'src/e.mjs', line: 8, title: 't', detail: 'd' });
const BOTH_ENGINES = (code = [], security = []) => [engine('claude-code-review', code), engine('claude-security-review', security)];
async function coordCase({ coordinator, reviewerFindings = [MEH_MINOR()], codeFindings = [CRI_MINOR()] }) {
const s = stubs({
reviewers: [{ reviewer: 'brief-conformance', findings: reviewerFindings }, { reviewer: 'code-correctness', findings: [] }],
coordinator,
});
const r = await exec({ ...BASE, enginePayloads: BOTH_ENGINES(codeFindings) }, s);
return { r, s };
}
const failedCoordinators = [
['returns null', () => null],
['returns findings (a partial list) and no verdict', () => ({ findings: [CRI_MINOR()] })],
['returns ALLOW with no findings array', () => ({ verdict: 'ALLOW' })],
];
for (const [label, make] of failedCoordinators) {
test(`edf2ec56 — a coordinator that ${label} → WARN, every merged finding still MINOR, the coordinator named`, async () => {
const { r, s } = await coordCase({ coordinator: make() });
assert.ok(typeof s.calls.coordinatorPrompt === 'string', 'the coordinator was called');
assert.equal(r.verdict, 'WARN');
assert.deepEqual(r.findings.map((f) => f.rule_key).sort(), ['CODE_REVIEW_IMPROVEMENT', 'MISSING_ERROR_HANDLING'],
'the floors read every merged finding, not a partial list');
for (const f of r.findings) assert.equal(f.severity, 'MINOR');
assert.equal(r.missing_reviewers, 1);
assert.deepEqual(r.allow_blocked_by, [COORD_MISSING]);
});
}
test('edf2ec56 — a coordinator returning an empty findings array and no verdict over a reviewer BLOCKER → BLOCK, the BLOCKER in findings', async () => {
const blocker = { severity: 'BLOCKER', rule_key: 'UNIMPLEMENTED_CRITERION', file: 'src/r.mjs', line: 2, title: 't', detail: 'd' };
const { r } = await coordCase({ coordinator: { findings: [] }, reviewerFindings: [blocker], codeFindings: [] });
assert.equal(r.verdict, 'BLOCK');
assert.equal(r.findings.filter((f) => f.rule_key === 'UNIMPLEMENTED_CRITERION' && f.severity === 'BLOCKER').length, 1);
assert.deepEqual(r.allow_blocked_by, [COORD_MISSING]);
});
test('edf2ec56 — a coordinator returning BLOCK and no findings array → BLOCK (never lowered)', async () => {
const { r } = await coordCase({ coordinator: { verdict: 'BLOCK' } });
assert.equal(r.verdict, 'BLOCK');
assert.deepEqual(r.allow_blocked_by, [COORD_MISSING]);
});
test('edf2ec56 — a coordinator returning the verdict PASS (outside BLOCK/WARN/ALLOW) → WARN, the coordinator named', async () => {
const { r } = await coordCase({ coordinator: { verdict: 'PASS', findings: [MEH_MINOR(), CRI_MINOR()] } });
assert.equal(r.verdict, 'WARN');
assert.ok(r.allow_blocked_by.includes(COORD_MISSING));
});
test('edf2ec56 — no engine payloads: both engines are named in allow_blocked_by, WARN', async () => {
const s = stubs({
reviewers: [{ reviewer: 'brief-conformance', findings: [MEH_MINOR()] }, { reviewer: 'code-correctness', findings: [] }],
coordinator: { verdict: 'ALLOW', findings: [MEH_MINOR()] },
});
const r = await exec({ ...BASE }, s);
assert.equal(r.verdict, 'WARN');
assert.deepEqual(r.allow_blocked_by, ['missing-reviewer:claude-code-review', 'missing-reviewer:claude-security-review']);
});
test('edf2ec56 — a first reviewer that returns null is named missing-reviewer:brief-conformance-reviewer', async () => {
const s = stubs({
reviewers: [null, { reviewer: 'code-correctness', findings: [MEH_MINOR()] }],
coordinator: { verdict: 'ALLOW', findings: [MEH_MINOR()] },
});
const r = await exec({ ...BASE, enginePayloads: BOTH_ENGINES() }, s);
assert.equal(r.verdict, 'WARN');
assert.deepEqual(r.allow_blocked_by, ['missing-reviewer:brief-conformance-reviewer']);
});
test('edf2ec56 — quick mode: a correctness reviewer that returns null is named missing-reviewer:code-correctness-reviewer, no brief-conformance-reviewer', async () => {
const s = stubs({ reviewers: [null], coordinator: { verdict: 'ALLOW', findings: [CRI_MINOR()] } });
const r = await exec({ ...BASE, quick: true, enginePayloads: [engine('claude-code-review', [CRI_MINOR()])] }, s);
assert.equal(r.verdict, 'WARN');
assert.deepEqual(r.allow_blocked_by, ['missing-reviewer:code-correctness-reviewer']);
assert.ok(!r.allow_blocked_by.includes('missing-reviewer:brief-conformance-reviewer'));
});
test('edf2ec56 — a null coordinator: a MISSING_TEST asserted MINOR is read at MAJOR (its tier), a MISSING_ERROR_HANDLING asserted BLOCKER stays BLOCKER → BLOCK', async () => {
const mt = { severity: 'MINOR', rule_key: 'MISSING_TEST', file: 'src/r.mjs', line: 5, title: 't', detail: 'd' };
const mehBlocker = { severity: 'BLOCKER', rule_key: 'MISSING_ERROR_HANDLING', file: 'src/r.mjs', line: 6, title: 't', detail: 'd' };
const { r } = await coordCase({ coordinator: null, reviewerFindings: [mt, mehBlocker], codeFindings: [] });
const sevOf = (key) => r.findings.find((f) => f.rule_key === key)?.severity;
assert.equal(sevOf('MISSING_TEST'), 'MAJOR', 'raised to its catalogue tier');
assert.equal(sevOf('MISSING_ERROR_HANDLING'), 'BLOCKER', 'a raise, never a reset');
assert.equal(r.verdict, 'BLOCK');
});
test('edf2ec56 control — a coordinator returning ALLOW with the MINOR findings and both engines → ALLOW, nothing named', async () => {
const { r } = await coordCase({ coordinator: { verdict: 'ALLOW', findings: [MEH_MINOR(), CRI_MINOR()] } });
assert.equal(r.verdict, 'ALLOW');
assert.equal(r.missing_reviewers, 0);
assert.deepEqual(r.allow_blocked_by, []);
});
test('edf2ec56 control — no findings at all → ALLOW, the coordinator skipped and not named', async () => {
const { r, s } = await coordCase({ coordinator: null, reviewerFindings: [], codeFindings: [] });
assert.equal(s.calls.coordinatorPrompt, null, 'the coordinator was skipped');
assert.equal(r.verdict, 'ALLOW');
assert.deepEqual(r.allow_blocked_by, []);
assert.ok(!r.allow_blocked_by.includes(COORD_MISSING));
});
test('edf2ec56 control — a BLOCKER with a null coordinator → BLOCK', async () => {
const blocker = { severity: 'BLOCKER', rule_key: 'UNIMPLEMENTED_CRITERION', file: 'src/r.mjs', line: 2, title: 't', detail: 'd' };
const { r } = await coordCase({ coordinator: null, reviewerFindings: [blocker], codeFindings: [] });
assert.equal(r.verdict, 'BLOCK');
assert.deepEqual(r.allow_blocked_by, [COORD_MISSING]);
});
test('edf2ec56 — the workflow section of commands/trekreview.md and Handover 6 name the coordinator; the section names allow_blocked_by', () => {
const cmd = readFileSync(join(ROOT, 'commands', 'trekreview.md'), 'utf-8');
const handover = readFileSync(join(ROOT, 'docs', 'HANDOVER-CONTRACTS.md'), 'utf-8');
const wf = section(cmd, '## Phase 5–6 via the Workflow substrate', '## Phase 7');
assert.match(wf, /missing-reviewer:review-coordinator/);
assert.match(wf, /allow_blocked_by/);
const h6 = section(handover, '## Handover 6 — `review.md` → plan', '\n## Handover 7');
assert.match(h6, /missing-reviewer:review-coordinator/);
});
// ---- Step 9: the brief-conflict rule on the Workflow path ---------------------------
//
// Each case stubs the reviewers' and engines' payloads (so `merged` holds the