fix(review): the workflow review fails closed when its coordinator returns nothing
Review finding edf2ec56; every reviewer that did not run is named in allow_blocked_by. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
9529df7665
commit
25856f2e9c
4 changed files with 166 additions and 8 deletions
|
|
@ -544,8 +544,8 @@ Workflow({
|
|||
})
|
||||
```
|
||||
|
||||
A reviewer that returns nothing, or an engine listed in `engineNotRun`,
|
||||
forbids `ALLOW` on this path too (`missing_reviewers` in the result).
|
||||
A reviewer that returns nothing, an engine listed in `engineNotRun`, or a coordinator that returns no verdict and findings forbids `ALLOW` on this path too (`missing_reviewers` in the result), and the result's `allow_blocked_by` names each one (`missing-reviewer:<name>`, the coordinator as `missing-reviewer:review-coordinator`);
|
||||
Phase 7 writes that list as the trailing JSON block's `allow_blocked_by`, so the verdict gate reads such a WARN review as not performed in full (exit 2; a BLOCK stays exit 1).
|
||||
On this path dedup ranks as Pass 1 does (both paths), every engine finding the
|
||||
coordinator did not return at its exact `(file, line, rule_key)` is put back at its own
|
||||
severity (operator decision 23), and a returned finding is never below the lower of its
|
||||
|
|
@ -567,7 +567,7 @@ Contract (verified in S10 part B — follow exactly):
|
|||
|
||||
Then continue at **Phase 7** exactly as the prose path does — Phase 7 rendering, Phase 8
|
||||
validation, and the operator gate are **shared** and substrate-independent (both paths
|
||||
return the same `{verdict, findings}` shape). The workflow's returned findings keep
|
||||
return the same `{verdict, findings}` shape, and the workflow's `allow_blocked_by` goes into the trailing JSON block). The workflow's returned findings keep
|
||||
`engine` (set by the script from the payload a finding came from, never from a label),
|
||||
`contradicts_brief` and `brief_lowered_from`, and Phase 7 writes them into each
|
||||
finding's field list and the trailing JSON block of review.md exactly as the prose path
|
||||
|
|
|
|||
|
|
@ -478,7 +478,7 @@ The gate cannot check an `engine` label on the prose path: there the coordinator
|
|||
the workflow path (`--workflow`) takes `engine` from the payload source and writes it into the returned finding.
|
||||
`runContract` (`lib/review/coordinator-contract.mjs`) likewise takes `engine` from the payload a finding arrived in, never from the finding's own field,
|
||||
and in dedup the survivor is ranked by catalogue tier (for a catalogue key it always ties inside one triplet), so a reviewer-agent finding survives over an engine finding and carries the highest severity the group asserted; an engine's mark never lowers what a reviewer agent asserted.
|
||||
On the workflow path dedup ranks as Pass 1 does (both paths), every engine finding the coordinator did not return at its exact `(file, line, rule_key)` is put back at its own severity (operator decision 23), and a returned finding is never below the lower of its reviewer counterpart's severity and its catalogue tier (floor (b)), so the workflow path is stricter than the prose path.
|
||||
On the workflow path dedup ranks as Pass 1 does (both paths), every engine finding the coordinator did not return at its exact `(file, line, rule_key)` is put back at its own severity (operator decision 23), and a returned finding is never below the lower of its reviewer counterpart's severity and its catalogue tier (floor (b)), so the workflow path is stricter than the prose path; a coordinator that returns no verdict and findings counts as a missing reviewer there (`missing-reviewer:review-coordinator` in `allow_blocked_by`), and its findings are read at their catalogue tier or higher, so the review is at least WARN and the verdict gate exits 2 on such a WARN (a BLOCK stays exit 1).
|
||||
A plan made from the review skips a lowered finding: it is MINOR, and `/trekplan` takes BLOCKER and MAJOR findings only.
|
||||
|
||||
**Validation strategy:**
|
||||
|
|
|
|||
|
|
@ -21,7 +21,9 @@
|
|||
// Ordre 5: the Claude Code engine payloads (Phase 4.7, run in prose before this
|
||||
// script) are merged with the reviewers' findings; a reviewer or engine that
|
||||
// returned nothing forbids ALLOW (fail-closed, as on the prose path); and no
|
||||
// finding is struck for length.
|
||||
// finding is struck for length. A coordinator that returns no verdict and
|
||||
// findings counts as a missing reviewer, and every reviewer, engine and
|
||||
// coordinator that did not deliver is named in `allow_blocked_by` (review edf2ec56).
|
||||
//
|
||||
// Run via the Workflow tool: Workflow({ scriptPath: this file, args }).
|
||||
// The structured result is logged as a single JSON line (prefix RESULT_JSON:)
|
||||
|
|
@ -217,9 +219,12 @@ const reviewerThunks = [
|
|||
agentType: 'voyage:code-correctness-reviewer', schema: FINDINGS_SCHEMA, label: 'correctness', phase: 'Review',
|
||||
}),
|
||||
];
|
||||
// The reviewer agent names in run order, so a reviewer that returned nothing is named.
|
||||
const reviewerNames = a.quick ? ['code-correctness-reviewer'] : ['brief-conformance-reviewer', 'code-correctness-reviewer'];
|
||||
const toRun = a.quick ? [reviewerThunks[1]] : reviewerThunks;
|
||||
const rawResults = await parallel(toRun);
|
||||
const reviewerResults = rawResults.filter(Boolean);
|
||||
const missingReviewerNames = reviewerNames.filter((_, i) => !rawResults[i]);
|
||||
// A reviewer that returned nothing is an absent one, not a clean one. The
|
||||
// engines that were due are counted from what actually arrived, never from
|
||||
// what the caller remembered to list as not run.
|
||||
|
|
@ -227,7 +232,7 @@ const enginePayloads = Array.isArray(a.enginePayloads) ? a.enginePayloads.filter
|
|||
const expectedEngines = a.quick ? ['claude-code-review'] : ['claude-code-review', 'claude-security-review'];
|
||||
const arrivedEngines = new Set(enginePayloads.map((p) => p.reviewer));
|
||||
const missingEngines = expectedEngines.filter((name) => !arrivedEngines.has(name));
|
||||
const missingReviewers = (rawResults.length - reviewerResults.length) + missingEngines.length;
|
||||
let missingReviewers = (rawResults.length - reviewerResults.length) + missingEngines.length;
|
||||
// Provenance, before dedup: "came from an engine" is read from the source,
|
||||
// never from an `engine` field. A reviewer finding loses any `engine` it
|
||||
// carries; an engine payload's findings get `engine` from its `reviewer`
|
||||
|
|
@ -260,10 +265,14 @@ if (merged.length === 0) {
|
|||
agentType: 'voyage:review-coordinator', schema: VERDICT_SCHEMA, label: 'coordinator', phase: 'Coordinate',
|
||||
});
|
||||
}
|
||||
// A coordinator that returns no verdict and findings is a missing reviewer, not
|
||||
// a clean one (the zero-findings skip above always sets a valid verdict).
|
||||
const coordinatorMissing = merged.length > 0 && !(coord && ['BLOCK', 'WARN', 'ALLOW'].includes(coord.verdict) && Array.isArray(coord.findings));
|
||||
if (coordinatorMissing) missingReviewers += 1;
|
||||
|
||||
// The verdict is mechanical: recompute its floor from the findings the
|
||||
// coordinator returned, so an LLM-written ALLOW over a BLOCKER cannot pass.
|
||||
const returned = Array.isArray(coord?.findings) ? coord.findings : merged;
|
||||
const returned = coordinatorMissing ? merged : coord.findings;
|
||||
// The floor reads two inputs, never the labels the coordinator wrote:
|
||||
// - `merged` (the coordinator's input, same file/line/rule_key) for each
|
||||
// returned finding's counterpart: (a) an engine counterpart raises it to the
|
||||
|
|
@ -280,12 +289,23 @@ const returned = Array.isArray(coord?.findings) ? coord.findings : merged;
|
|||
// move or relabel of an engine finding no longer lowers the verdict.
|
||||
// The raises and put-backs are written into `returned`, the marks kept, so the
|
||||
// rendered review.md carries them.
|
||||
// A coordinator that failed (no verdict and findings) is not trusted with a
|
||||
// partial list: `returned` is then every merged finding, each first raised to
|
||||
// its catalogue tier when the key is known (never lowered), as the default
|
||||
// path's Pass 3 would reset it; that raise stands after `raiseTo` is declared.
|
||||
const counterparts = new Map(merged
|
||||
.filter((m) => m && typeof m === 'object')
|
||||
.map((m) => [tripletOf(m), m]));
|
||||
const present = (f, field) => f[field] !== undefined && f[field] !== null;
|
||||
const raiseTo = (f, sev) => { if (sevRank(sev) < sevRank(f.severity)) f.severity = sev; };
|
||||
const lowerOf = (s1, s2) => (sevRank(s1) >= sevRank(s2) ? s1 : s2);
|
||||
if (coordinatorMissing) {
|
||||
for (const f of returned) {
|
||||
if (!f || typeof f !== 'object') continue;
|
||||
const tier = tierOf(f.rule_key);
|
||||
if (tier !== null) raiseTo(f, tier);
|
||||
}
|
||||
}
|
||||
for (const f of returned) {
|
||||
if (!f || typeof f !== 'object') continue;
|
||||
const c = counterparts.get(tripletOf(f));
|
||||
|
|
@ -321,15 +341,23 @@ for (const f of putBack) {
|
|||
returned.push(copy);
|
||||
}
|
||||
const has = (sev) => returned.some((f) => f && f.severity === sev);
|
||||
let verdict = coord?.verdict ?? null;
|
||||
let verdict = coordinatorMissing ? (coord?.verdict === 'BLOCK' ? 'BLOCK' : 'WARN') : coord.verdict;
|
||||
if (has('BLOCKER')) verdict = 'BLOCK';
|
||||
else if (has('MAJOR') && verdict !== 'BLOCK') verdict = 'WARN';
|
||||
if (verdict === 'ALLOW' && missingReviewers > 0) verdict = 'WARN';
|
||||
|
||||
// Every reviewer, engine and coordinator that did not deliver, in the form the
|
||||
// default path writes and lib/review/verdict-gate.mjs counts as not run.
|
||||
const allowBlockedBy = [
|
||||
...missingReviewerNames.map((name) => `missing-reviewer:${name}`),
|
||||
...missingEngines.map((name) => `missing-reviewer:${name}`),
|
||||
...(coordinatorMissing ? ['missing-reviewer:review-coordinator'] : []),
|
||||
];
|
||||
const result = {
|
||||
verdict,
|
||||
missing_reviewers: missingReviewers,
|
||||
missing_engines: missingEngines,
|
||||
allow_blocked_by: allowBlockedBy,
|
||||
findings: returned,
|
||||
raw_reviewer_count: reviewerResults.length,
|
||||
raw_finding_count: allFindings.length,
|
||||
|
|
|
|||
|
|
@ -82,6 +82,136 @@ test('armB — quick mode expects only the /code-review engine', async () => {
|
|||
assert.deepEqual(r.missing_engines, []);
|
||||
});
|
||||
|
||||
// ---- Review edf2ec56: a coordinator that returns nothing fails closed ----------------
|
||||
//
|
||||
// A coordinator that returns no verdict and findings counts as a missing reviewer:
|
||||
// its partial list is not trusted (the floors read every merged finding, each raised
|
||||
// to its catalogue tier, never lowered), ALLOW is forbidden, a BLOCK is never lowered,
|
||||
// and every reviewer, engine and coordinator that did not deliver is named in
|
||||
// `allow_blocked_by` as `missing-reviewer:<name>`.
|
||||
|
||||
const COORD_MISSING = 'missing-reviewer:review-coordinator';
|
||||
const MEH_MINOR = () => ({ severity: 'MINOR', rule_key: 'MISSING_ERROR_HANDLING', file: 'src/r.mjs', line: 4, title: 't', detail: 'd' });
|
||||
const CRI_MINOR = () => ({ severity: 'MINOR', rule_key: 'CODE_REVIEW_IMPROVEMENT', file: 'src/e.mjs', line: 8, title: 't', detail: 'd' });
|
||||
const BOTH_ENGINES = (code = [], security = []) => [engine('claude-code-review', code), engine('claude-security-review', security)];
|
||||
|
||||
async function coordCase({ coordinator, reviewerFindings = [MEH_MINOR()], codeFindings = [CRI_MINOR()] }) {
|
||||
const s = stubs({
|
||||
reviewers: [{ reviewer: 'brief-conformance', findings: reviewerFindings }, { reviewer: 'code-correctness', findings: [] }],
|
||||
coordinator,
|
||||
});
|
||||
const r = await exec({ ...BASE, enginePayloads: BOTH_ENGINES(codeFindings) }, s);
|
||||
return { r, s };
|
||||
}
|
||||
|
||||
const failedCoordinators = [
|
||||
['returns null', () => null],
|
||||
['returns findings (a partial list) and no verdict', () => ({ findings: [CRI_MINOR()] })],
|
||||
['returns ALLOW with no findings array', () => ({ verdict: 'ALLOW' })],
|
||||
];
|
||||
for (const [label, make] of failedCoordinators) {
|
||||
test(`edf2ec56 — a coordinator that ${label} → WARN, every merged finding still MINOR, the coordinator named`, async () => {
|
||||
const { r, s } = await coordCase({ coordinator: make() });
|
||||
assert.ok(typeof s.calls.coordinatorPrompt === 'string', 'the coordinator was called');
|
||||
assert.equal(r.verdict, 'WARN');
|
||||
assert.deepEqual(r.findings.map((f) => f.rule_key).sort(), ['CODE_REVIEW_IMPROVEMENT', 'MISSING_ERROR_HANDLING'],
|
||||
'the floors read every merged finding, not a partial list');
|
||||
for (const f of r.findings) assert.equal(f.severity, 'MINOR');
|
||||
assert.equal(r.missing_reviewers, 1);
|
||||
assert.deepEqual(r.allow_blocked_by, [COORD_MISSING]);
|
||||
});
|
||||
}
|
||||
|
||||
test('edf2ec56 — a coordinator returning an empty findings array and no verdict over a reviewer BLOCKER → BLOCK, the BLOCKER in findings', async () => {
|
||||
const blocker = { severity: 'BLOCKER', rule_key: 'UNIMPLEMENTED_CRITERION', file: 'src/r.mjs', line: 2, title: 't', detail: 'd' };
|
||||
const { r } = await coordCase({ coordinator: { findings: [] }, reviewerFindings: [blocker], codeFindings: [] });
|
||||
assert.equal(r.verdict, 'BLOCK');
|
||||
assert.equal(r.findings.filter((f) => f.rule_key === 'UNIMPLEMENTED_CRITERION' && f.severity === 'BLOCKER').length, 1);
|
||||
assert.deepEqual(r.allow_blocked_by, [COORD_MISSING]);
|
||||
});
|
||||
|
||||
test('edf2ec56 — a coordinator returning BLOCK and no findings array → BLOCK (never lowered)', async () => {
|
||||
const { r } = await coordCase({ coordinator: { verdict: 'BLOCK' } });
|
||||
assert.equal(r.verdict, 'BLOCK');
|
||||
assert.deepEqual(r.allow_blocked_by, [COORD_MISSING]);
|
||||
});
|
||||
|
||||
test('edf2ec56 — a coordinator returning the verdict PASS (outside BLOCK/WARN/ALLOW) → WARN, the coordinator named', async () => {
|
||||
const { r } = await coordCase({ coordinator: { verdict: 'PASS', findings: [MEH_MINOR(), CRI_MINOR()] } });
|
||||
assert.equal(r.verdict, 'WARN');
|
||||
assert.ok(r.allow_blocked_by.includes(COORD_MISSING));
|
||||
});
|
||||
|
||||
test('edf2ec56 — no engine payloads: both engines are named in allow_blocked_by, WARN', async () => {
|
||||
const s = stubs({
|
||||
reviewers: [{ reviewer: 'brief-conformance', findings: [MEH_MINOR()] }, { reviewer: 'code-correctness', findings: [] }],
|
||||
coordinator: { verdict: 'ALLOW', findings: [MEH_MINOR()] },
|
||||
});
|
||||
const r = await exec({ ...BASE }, s);
|
||||
assert.equal(r.verdict, 'WARN');
|
||||
assert.deepEqual(r.allow_blocked_by, ['missing-reviewer:claude-code-review', 'missing-reviewer:claude-security-review']);
|
||||
});
|
||||
|
||||
test('edf2ec56 — a first reviewer that returns null is named missing-reviewer:brief-conformance-reviewer', async () => {
|
||||
const s = stubs({
|
||||
reviewers: [null, { reviewer: 'code-correctness', findings: [MEH_MINOR()] }],
|
||||
coordinator: { verdict: 'ALLOW', findings: [MEH_MINOR()] },
|
||||
});
|
||||
const r = await exec({ ...BASE, enginePayloads: BOTH_ENGINES() }, s);
|
||||
assert.equal(r.verdict, 'WARN');
|
||||
assert.deepEqual(r.allow_blocked_by, ['missing-reviewer:brief-conformance-reviewer']);
|
||||
});
|
||||
|
||||
test('edf2ec56 — quick mode: a correctness reviewer that returns null is named missing-reviewer:code-correctness-reviewer, no brief-conformance-reviewer', async () => {
|
||||
const s = stubs({ reviewers: [null], coordinator: { verdict: 'ALLOW', findings: [CRI_MINOR()] } });
|
||||
const r = await exec({ ...BASE, quick: true, enginePayloads: [engine('claude-code-review', [CRI_MINOR()])] }, s);
|
||||
assert.equal(r.verdict, 'WARN');
|
||||
assert.deepEqual(r.allow_blocked_by, ['missing-reviewer:code-correctness-reviewer']);
|
||||
assert.ok(!r.allow_blocked_by.includes('missing-reviewer:brief-conformance-reviewer'));
|
||||
});
|
||||
|
||||
test('edf2ec56 — a null coordinator: a MISSING_TEST asserted MINOR is read at MAJOR (its tier), a MISSING_ERROR_HANDLING asserted BLOCKER stays BLOCKER → BLOCK', async () => {
|
||||
const mt = { severity: 'MINOR', rule_key: 'MISSING_TEST', file: 'src/r.mjs', line: 5, title: 't', detail: 'd' };
|
||||
const mehBlocker = { severity: 'BLOCKER', rule_key: 'MISSING_ERROR_HANDLING', file: 'src/r.mjs', line: 6, title: 't', detail: 'd' };
|
||||
const { r } = await coordCase({ coordinator: null, reviewerFindings: [mt, mehBlocker], codeFindings: [] });
|
||||
const sevOf = (key) => r.findings.find((f) => f.rule_key === key)?.severity;
|
||||
assert.equal(sevOf('MISSING_TEST'), 'MAJOR', 'raised to its catalogue tier');
|
||||
assert.equal(sevOf('MISSING_ERROR_HANDLING'), 'BLOCKER', 'a raise, never a reset');
|
||||
assert.equal(r.verdict, 'BLOCK');
|
||||
});
|
||||
|
||||
test('edf2ec56 control — a coordinator returning ALLOW with the MINOR findings and both engines → ALLOW, nothing named', async () => {
|
||||
const { r } = await coordCase({ coordinator: { verdict: 'ALLOW', findings: [MEH_MINOR(), CRI_MINOR()] } });
|
||||
assert.equal(r.verdict, 'ALLOW');
|
||||
assert.equal(r.missing_reviewers, 0);
|
||||
assert.deepEqual(r.allow_blocked_by, []);
|
||||
});
|
||||
|
||||
test('edf2ec56 control — no findings at all → ALLOW, the coordinator skipped and not named', async () => {
|
||||
const { r, s } = await coordCase({ coordinator: null, reviewerFindings: [], codeFindings: [] });
|
||||
assert.equal(s.calls.coordinatorPrompt, null, 'the coordinator was skipped');
|
||||
assert.equal(r.verdict, 'ALLOW');
|
||||
assert.deepEqual(r.allow_blocked_by, []);
|
||||
assert.ok(!r.allow_blocked_by.includes(COORD_MISSING));
|
||||
});
|
||||
|
||||
test('edf2ec56 control — a BLOCKER with a null coordinator → BLOCK', async () => {
|
||||
const blocker = { severity: 'BLOCKER', rule_key: 'UNIMPLEMENTED_CRITERION', file: 'src/r.mjs', line: 2, title: 't', detail: 'd' };
|
||||
const { r } = await coordCase({ coordinator: null, reviewerFindings: [blocker], codeFindings: [] });
|
||||
assert.equal(r.verdict, 'BLOCK');
|
||||
assert.deepEqual(r.allow_blocked_by, [COORD_MISSING]);
|
||||
});
|
||||
|
||||
test('edf2ec56 — the workflow section of commands/trekreview.md and Handover 6 name the coordinator; the section names allow_blocked_by', () => {
|
||||
const cmd = readFileSync(join(ROOT, 'commands', 'trekreview.md'), 'utf-8');
|
||||
const handover = readFileSync(join(ROOT, 'docs', 'HANDOVER-CONTRACTS.md'), 'utf-8');
|
||||
const wf = section(cmd, '## Phase 5–6 via the Workflow substrate', '## Phase 7');
|
||||
assert.match(wf, /missing-reviewer:review-coordinator/);
|
||||
assert.match(wf, /allow_blocked_by/);
|
||||
const h6 = section(handover, '## Handover 6 — `review.md` → plan', '\n## Handover 7');
|
||||
assert.match(h6, /missing-reviewer:review-coordinator/);
|
||||
});
|
||||
|
||||
// ---- Step 9: the brief-conflict rule on the Workflow path ---------------------------
|
||||
//
|
||||
// Each case stubs the reviewers' and engines' payloads (so `merged` holds the
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue