open/voyage
1
0
Fork 0
Code Issues Releases 2 Activity

fix(review): fail-closed verdicts - an unsubstantiated finding can no longer yield ALLOW

Browse source 
computeVerdict counted only the findings handed to it (reasoned.kept), so a
finding removed by Pass 2 or Pass 3, and a reviewer whose payload was thrown
away or never arrived, were arithmetically identical to a finding that never
existed. All three pushed the verdict toward ALLOW.

Measured before the fix (probes, 2026-09-01):
  - a BLOCKER with a 101-character title -> ALLOW (Pass 2 succinctness)
  - a payload with one ad-hoc rule_key is skipped WHOLE at ingest, taking a
    valid BLOCKER sibling with it -> ALLOW
  - a reviewer that never reported -> ALLOW
Pass 3's own no-citation / unknown-rule_key branches turned out unreachable
through runContract (validateFindings rejects those payloads first), so the
reachable exposure was Pass 2 plus the skipped/absent reviewer.

THE OPEN DESIGN DECISION, and why it went against the order's default.
The order proposed: indeterminate file-existence YES, plain succinctness NO
("a too-long finding is not an uncertain finding"). I kept the first and
overrode the second, on one principle:

  A removal is `dropped` only when the test REFUTED the finding as a claim
  about this codebase. Every other removal is `unverified`.

Succinctness and actionability read a `.length`. They never examine the claim,
so they cannot establish the finding is unreal - and dropping a BLOCKER for a
101-character title is precisely the fail-open shape being fixed. Three things
settled it:

1. Under the order's default the fix would have been almost inert. Pass 3's
   drop branches are unreachable via runContract, so leaving Pass 2 out would
   have left the only reachable finding-level exposure open.
2. Cost asymmetry, priced rather than asserted: the verdict is not a gate.
   Handover 6 feeds `findings` filtered to BLOCKER+MAJOR into /trekplan
   (commands/trekplan.md:218); `verdict` is optional metadata
   (docs/HANDOVER-CONTRACTS.md:353). Nothing loops or re-plans on WARN. So a
   false `unverified` costs WARN plus a printed reason; a false drop costs a
   silent ALLOW over a live BLOCKER.
3. unknown-rule_key joins them for the same reason: an ad-hoc key is a real
   defect wearing the wrong label, and v5.1.1 high-effort mode already KEEPS
   those, normalised to PLAN_EXECUTE_DRIFT. Refuting them at normal effort
   while keeping them at high effort would be incoherent.

no-citation stays a drop: a finding whose file is empty or whose line is
negative names no location, so it makes no checkable claim at all - the one
deterministic refutation, and what the Pass 3 prose already said it was.

Iron Law: tests/lib/coordinator-contract.test.mjs first, red (missing export +
the three measured ALLOWs), then production code. Two existing assertions were
updated AFTER implementation as contract changes, not to make the red pass.
A known-positive control pins that ALLOW is still reachable - without it,
"no ALLOW" is not a fail-closed contract, only a broken one.

lib/review/coordinator-contract.mjs
  + classifySuppression / REFUTING_REASONS / UNVERIFIED_REASONS - one
    vocabulary owned by the lib, including the tokens only the LLM
    coordinator emits (accuracy:refuted, file-existence:refuted/indeterminate),
    so prose and lib cannot drift. Unclassified reasons default to unverified:
    the default fails closed.
  ~ judgeFilter / reasonablenessFilter return {kept, dropped, unverified}
  ~ computeVerdict(findings, {unverified, missingReviewers}) -> + allow_blocked_by.
    Never raises a verdict, only withholds ALLOW. Unverified findings are NOT
    counted into a severity tier: their severity was never substantiated, and
    counting it would be invention.
  ~ runContract(payloads, {expectedReviewers}) -> + unverified,
    missing_reviewers, allow_blocked_by. `suppressed` stays the union of
    dropped + unverified, so existing consumers (gold-eval) keep their meaning.

agents/review-coordinator.md - Pass 2/3 tables gain a fate column, new
  "Suppression is two-valued" section, Pass 4 threshold table gains the two
  fail-closed rows, Executive Summary must state a withheld ALLOW, Suppressed
  Findings tags each line [dropped]/[unverified]. Pass 3's unknown-rule_key
  bullet explicitly says high-effort does not reach that branch, so the same
  input never has two documented fates.

commands/trekreview.md - Phase 5 "Reviewer accounting": the expected set is
  written down before the spawn, a silent reviewer gets one re-ask and then
  STOP. That extends the pattern already in the file (schema failure -> 2
  bounded re-asks -> "do not feed unvalidated findings to the coordinator") to
  the other two ways a reviewer goes missing, rather than softening it to WARN.
  The lib's missing_reviewers stays as belt-and-braces for direct callers.

docs/agent-return-channel-defect.md - the "inferred, not observed" caveat on
  the unnamed arm above 66 lines is struck: akashic-intelligence S27
  (f168630) measured 2/2 unnamed agents returning against a 4370-line plan,
  30449 B and 10989 B, both valid JSON. Recorded with akashic's own two
  caveats intact - the measurer owns the finding, and byte-identity between
  the returned string and the file on disk was not proven. The separate S25
  named-arm figures are left standing; these are two measurements, not a
  correction of one by the other.

No release, no version bump, no tag, no catalogue ref, no Workflow port.
Suite 1025 (1023/0/2) -> 1034 (1032/0/2), 0 failures.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-01 22:46:39 +02:00
commit e2aec019ac
6 changed files with 469 additions and 50 deletions
Download patch file Download diff file Expand all files Collapse all files

171
lib/review/coordinator-contract.mjs
View file

@ -19,8 +19,10 @@
// What IS implemented, purely: Pass 1 (triplet dedup → highest-severity-wins
// survivor + conformance tiebreak + detail concat + raised_by provenance),
// Pass 2 succinctness + actionability-presence, Pass 3 reasonableness
// (citation / unknown-rule_key drop, severity-mismatch correction), Pass 4
// verdict thresholds. No LLM, no network, no time, no randomness.
// (citation / unknown-rule_key suppression, severity-mismatch correction),
// Pass 4 verdict thresholds — fail-closed: a suppression that did not REFUTE
// the finding, and a reviewer that never reported, forbid ALLOW (see
// classifySuppression). No LLM, no network, no time, no randomness.
//
// Reuses: SEVERITY_VALUES / RULE_KEYS / getRule (rule-catalogue.mjs),
// computeFindingId (finding-id.mjs, triplet), validateFindings
@ -35,6 +37,71 @@ import { validateFindings } from './findings-schema.mjs';
export const JUDGE_TITLE_MAX = 100;
export const JUDGE_DETAIL_MAX = 800;
// ---- Suppression classification (fail-closed) --------------------------------
//
// A removal is `dropped` ONLY when the test refuted the finding as a claim
// about this codebase. Every other removal is `unverified`: the coordinator
// took the finding out of the count without ever establishing it was unreal,
// so it may not be spent as evidence of a clean review.
/**
* Reasons that REFUTE. `no-citation` is the only one this deterministic subset
* can emit: a finding whose `file` is empty or whose `line` is negative names
* no location, so it makes no checkable claim at all
* (agents/review-coordinator.md Pass 3 — "Speculative 'code might break
* somewhere' findings have no anchor").
*
* `accuracy:refuted` (Pass 2 Accuracy — a citation escaping the repo root) and
* `file-existence:refuted` (Pass 3 — absent from both working tree and diff)
* are emitted by the LLM coordinator, whose fs/judgement branches this module
* excludes. They are declared here anyway: the vocabulary is owned in one
* place so prose and lib cannot drift.
*/
export const REFUTING_REASONS = Object.freeze(new Set([
'no-citation',
'accuracy:refuted',
'file-existence:refuted',
]));
/**
* The reason vocabulary on the unverified side. `file-existence:indeterminate`
* is emitted by the LLM coordinator's Pass 3 (which runs the fs Glob this
* module deliberately excludes); the vocabulary is owned here so prose and lib
* cannot drift.
*/
export const UNVERIFIED_REASONS = Object.freeze([
'succinctness:title',
'succinctness:detail',
'actionability:empty',
'unknown-rule_key',
'file-existence:indeterminate',
]);
/**
* Classify a suppression reason. Anything not declared refuting is
* `unverified` — the default is fail-CLOSED, so a reason introduced later
* without a decision cannot silently move the verdict toward ALLOW.
* @param {string} reason
* @returns {'refuted'|'unverified'}
*/
export function classifySuppression(reason) {
return REFUTING_REASONS.has(reason) ? 'refuted' : 'unverified';
}
/**
* Tag a finding with its suppression reason and route it to the refuted
* (`dropped`) or the `unverified` bucket.
* @param {object} finding
* @param {string} reason
* @param {object[]} dropped
* @param {object[]} unverified
*/
function suppress(finding, reason, dropped, unverified) {
const tagged = { ...finding, suppressed_reason: reason };
if (classifySuppression(reason) === 'refuted') dropped.push(tagged);
else unverified.push(tagged);
}
/**
* Catalogue-tier rank of a severity: lower number = higher severity.
* BLOCKER=0 … SUGGESTION=3; an unknown severity ranks last.
@ -122,12 +189,18 @@ export function dedupByTriplet(findings) {
* (title > 100 or detail > 800 chars) and actionability (recommended_action,
* when present, must be a non-empty string). The imperative-verb test is
* excluded (LLM judgement).
*
* Both tests read a `.length`; neither examines the claim, so neither can
* establish the finding is unreal. Both therefore route to `unverified`.
* `dropped` stays in the signature for the refuting Pass-2 filter this subset
* excludes (Accuracy: a path-traversal escape IS a refutation).
* @param {object[]} findings
* @returns {{ kept: object[], dropped: object[] }}
* @returns {{ kept: object[], dropped: object[], unverified: object[] }}
*/
export function judgeFilter(findings) {
const kept = [];
const dropped = [];
const unverified = [];
for (const f of findings) {
const titleLen = (f.title ?? '').length;
const detailLen = (f.detail ?? '').length;
@ -138,31 +211,38 @@ export function judgeFilter(findings) {
(typeof f.recommended_action !== 'string' || f.recommended_action.trim().length === 0)) {
reason = 'actionability:empty';
}
if (reason) dropped.push({ ...f, suppressed_reason: reason });
if (reason) suppress(f, reason, dropped, unverified);
else kept.push(f);
}
return { kept, dropped };
return { kept, dropped, unverified };
}
/**
* Pass 3 — Cloudflare reasonableness (deterministic subset): drop findings
* with no citation (empty file / line < 0) or an unknown rule_key; CORRECT a
* severity that does not match the catalogue tier (a correction, not a drop).
* The fs file-existence glob is excluded (I/O).
* The fs file-existence glob is excluded (I/O) — its indeterminate branch is
* prose-side, tokenised as `file-existence:indeterminate`.
*
* `no-citation` REFUTES (the finding names no location, so it makes no
* checkable claim) and is dropped. `unknown-rule_key` does not: an ad-hoc key
* is a real defect wearing the wrong label — v5.1.1 high-effort mode already
* KEEPS these, normalised to PLAN_EXECUTE_DRIFT — so it routes to `unverified`.
* @param {object[]} findings
* @returns {{ kept: object[], dropped: object[] }}
* @returns {{ kept: object[], dropped: object[], unverified: object[] }}
*/
export function reasonablenessFilter(findings) {
const kept = [];
const dropped = [];
const unverified = [];
for (const f of findings) {
if (typeof f.file !== 'string' || f.file.length === 0 ||
(typeof f.line === 'number' && f.line < 0)) {
dropped.push({ ...f, suppressed_reason: 'no-citation' });
suppress(f, 'no-citation', dropped, unverified);
continue;
}
if (!RULE_KEYS.has(f.rule_key)) {
dropped.push({ ...f, suppressed_reason: 'unknown-rule_key' });
suppress(f, 'unknown-rule_key', dropped, unverified);
continue;
}
const rule = getRule(f.rule_key);
@ -172,44 +252,101 @@ export function reasonablenessFilter(findings) {
kept.push(f);
}
}
return { kept, dropped };
return { kept, dropped, unverified };
}
/**
* Pass 4 — compute the verdict from severity counts (after dedup + filtering).
* BLOCKER ≥ 1 → BLOCK; else MAJOR ≥ 1 → WARN; else ALLOW.
*
* FAIL-CLOSED: ALLOW additionally requires that nothing is `unverified` and
* that every expected reviewer reported. Neither ever RAISES a verdict — the
* severity thresholds are untouched — they only forbid the clean one, so the
* worst case of a false unverified is WARN plus a stated reason, never a
* silent pass. Unverified findings are NOT counted into a severity tier: their
* severity is reviewer-asserted and was never substantiated.
*
* @param {object[]} findings
* @returns {{ verdict: 'BLOCK'|'WARN'|'ALLOW', counts: Record<string, number> }}
* @param {{ unverified?: object[], missingReviewers?: string[] }} [options]
* @returns {{ verdict: 'BLOCK'|'WARN'|'ALLOW', counts: Record<string, number>, allow_blocked_by: string[] }}
*/
export function computeVerdict(findings) {
export function computeVerdict(findings, options = {}) {
const counts = { BLOCKER: 0, MAJOR: 0, MINOR: 0, SUGGESTION: 0 };
for (const f of findings) {
if (counts[f.severity] !== undefined) counts[f.severity] += 1;
}
const unverified = options.unverified ?? [];
const missingReviewers = options.missingReviewers ?? [];
const allow_blocked_by = [];
const byReason = new Map();
for (const f of unverified) {
const reason = f?.suppressed_reason ?? 'unspecified';
byReason.set(reason, (byReason.get(reason) ?? 0) + 1);
}
for (const [reason, n] of byReason) allow_blocked_by.push(`unverified:${reason} (${n})`);
for (const r of missingReviewers) allow_blocked_by.push(`missing-reviewer:${r}`);
let verdict;
if (counts.BLOCKER >= 1) verdict = 'BLOCK';
else if (counts.MAJOR >= 1) verdict = 'WARN';
else if (allow_blocked_by.length > 0) verdict = 'WARN';
else verdict = 'ALLOW';
return { verdict, counts };
return { verdict, counts, allow_blocked_by };
}
/**
* Run the full deterministic contract: ingest → Pass 1 → Pass 2 → Pass 3 → Pass 4.
*
* `options.expectedReviewers` names the reviewers this review was supposed to
* hear from. A reviewer that is absent from the payloads, or whose payload
* failed schema validation and was thrown away at ingest, lands in
* `missing_reviewers` and forbids ALLOW: an unread reviewer is an absent one,
* and zero findings from a silent reviewer must not read like zero findings
* from a clean diff.
*
* `suppressed` stays the UNION of `dropped` (refuted) and `unverified` so
* existing consumers keep their meaning; `unverified` is the subset that
* forbids ALLOW. Do not iterate both and count twice.
*
* @param {Array<{reviewer?: string, findings: object[]}>} reviewerPayloads
* @returns {{ verdict: string, counts: Record<string, number>, findings: object[], suppressed: object[], skipped: object[] }}
* @param {{ expectedReviewers?: string[] }} [options]
* @returns {{ verdict: string, counts: Record<string, number>, findings: object[], suppressed: object[], unverified: object[], skipped: object[], missing_reviewers: string[], allow_blocked_by: string[] }}
*/
export function runContract(reviewerPayloads) {
export function runContract(reviewerPayloads, options = {}) {
const { findings: ingested, skipped } = ingest(reviewerPayloads);
const deduped = dedupByTriplet(ingested);
const judged = judgeFilter(deduped);
const reasoned = reasonablenessFilter(judged.kept);
const { verdict, counts } = computeVerdict(reasoned.kept);
const unverified = [...judged.unverified, ...reasoned.unverified];
const reported = new Set();
for (const payload of reviewerPayloads) {
if (typeof payload?.reviewer === 'string' && payload.reviewer.length > 0) reported.add(payload.reviewer);
}
for (const s of skipped) reported.delete(s.reviewer);
const missing_reviewers = [];
for (const s of skipped) {
const name = s.reviewer ?? 'unnamed reviewer';
if (!missing_reviewers.includes(name)) missing_reviewers.push(name);
}
for (const r of options.expectedReviewers ?? []) {
if (!reported.has(r) && !missing_reviewers.includes(r)) missing_reviewers.push(r);
}
const { verdict, counts, allow_blocked_by } = computeVerdict(reasoned.kept, {
unverified,
missingReviewers: missing_reviewers,
});
return {
verdict,
counts,
findings: reasoned.kept,
suppressed: [...judged.dropped, ...reasoned.dropped],
suppressed: [...judged.dropped, ...judged.unverified, ...reasoned.dropped, ...reasoned.unverified],
unverified,
skipped,
missing_reviewers,
allow_blocked_by,
};
}