open/voyage
1
0
Fork 0
Code Issues Releases 2 Activity

fix(review): fail-closed verdicts - an unsubstantiated finding can no longer yield ALLOW

Browse source 
computeVerdict counted only the findings handed to it (reasoned.kept), so a
finding removed by Pass 2 or Pass 3, and a reviewer whose payload was thrown
away or never arrived, were arithmetically identical to a finding that never
existed. All three pushed the verdict toward ALLOW.

Measured before the fix (probes, 2026-09-01):
  - a BLOCKER with a 101-character title -> ALLOW (Pass 2 succinctness)
  - a payload with one ad-hoc rule_key is skipped WHOLE at ingest, taking a
    valid BLOCKER sibling with it -> ALLOW
  - a reviewer that never reported -> ALLOW
Pass 3's own no-citation / unknown-rule_key branches turned out unreachable
through runContract (validateFindings rejects those payloads first), so the
reachable exposure was Pass 2 plus the skipped/absent reviewer.

THE OPEN DESIGN DECISION, and why it went against the order's default.
The order proposed: indeterminate file-existence YES, plain succinctness NO
("a too-long finding is not an uncertain finding"). I kept the first and
overrode the second, on one principle:

  A removal is `dropped` only when the test REFUTED the finding as a claim
  about this codebase. Every other removal is `unverified`.

Succinctness and actionability read a `.length`. They never examine the claim,
so they cannot establish the finding is unreal - and dropping a BLOCKER for a
101-character title is precisely the fail-open shape being fixed. Three things
settled it:

1. Under the order's default the fix would have been almost inert. Pass 3's
   drop branches are unreachable via runContract, so leaving Pass 2 out would
   have left the only reachable finding-level exposure open.
2. Cost asymmetry, priced rather than asserted: the verdict is not a gate.
   Handover 6 feeds `findings` filtered to BLOCKER+MAJOR into /trekplan
   (commands/trekplan.md:218); `verdict` is optional metadata
   (docs/HANDOVER-CONTRACTS.md:353). Nothing loops or re-plans on WARN. So a
   false `unverified` costs WARN plus a printed reason; a false drop costs a
   silent ALLOW over a live BLOCKER.
3. unknown-rule_key joins them for the same reason: an ad-hoc key is a real
   defect wearing the wrong label, and v5.1.1 high-effort mode already KEEPS
   those, normalised to PLAN_EXECUTE_DRIFT. Refuting them at normal effort
   while keeping them at high effort would be incoherent.

no-citation stays a drop: a finding whose file is empty or whose line is
negative names no location, so it makes no checkable claim at all - the one
deterministic refutation, and what the Pass 3 prose already said it was.

Iron Law: tests/lib/coordinator-contract.test.mjs first, red (missing export +
the three measured ALLOWs), then production code. Two existing assertions were
updated AFTER implementation as contract changes, not to make the red pass.
A known-positive control pins that ALLOW is still reachable - without it,
"no ALLOW" is not a fail-closed contract, only a broken one.

lib/review/coordinator-contract.mjs
  + classifySuppression / REFUTING_REASONS / UNVERIFIED_REASONS - one
    vocabulary owned by the lib, including the tokens only the LLM
    coordinator emits (accuracy:refuted, file-existence:refuted/indeterminate),
    so prose and lib cannot drift. Unclassified reasons default to unverified:
    the default fails closed.
  ~ judgeFilter / reasonablenessFilter return {kept, dropped, unverified}
  ~ computeVerdict(findings, {unverified, missingReviewers}) -> + allow_blocked_by.
    Never raises a verdict, only withholds ALLOW. Unverified findings are NOT
    counted into a severity tier: their severity was never substantiated, and
    counting it would be invention.
  ~ runContract(payloads, {expectedReviewers}) -> + unverified,
    missing_reviewers, allow_blocked_by. `suppressed` stays the union of
    dropped + unverified, so existing consumers (gold-eval) keep their meaning.

agents/review-coordinator.md - Pass 2/3 tables gain a fate column, new
  "Suppression is two-valued" section, Pass 4 threshold table gains the two
  fail-closed rows, Executive Summary must state a withheld ALLOW, Suppressed
  Findings tags each line [dropped]/[unverified]. Pass 3's unknown-rule_key
  bullet explicitly says high-effort does not reach that branch, so the same
  input never has two documented fates.

commands/trekreview.md - Phase 5 "Reviewer accounting": the expected set is
  written down before the spawn, a silent reviewer gets one re-ask and then
  STOP. That extends the pattern already in the file (schema failure -> 2
  bounded re-asks -> "do not feed unvalidated findings to the coordinator") to
  the other two ways a reviewer goes missing, rather than softening it to WARN.
  The lib's missing_reviewers stays as belt-and-braces for direct callers.

docs/agent-return-channel-defect.md - the "inferred, not observed" caveat on
  the unnamed arm above 66 lines is struck: akashic-intelligence S27
  (f168630) measured 2/2 unnamed agents returning against a 4370-line plan,
  30449 B and 10989 B, both valid JSON. Recorded with akashic's own two
  caveats intact - the measurer owns the finding, and byte-identity between
  the returned string and the file on disk was not proven. The separate S25
  named-arm figures are left standing; these are two measurements, not a
  correction of one by the other.

No release, no version bump, no tag, no catalogue ref, no Workflow port.
Suite 1025 (1023/0/2) -> 1034 (1032/0/2), 0 failures.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-01 22:46:39 +02:00
commit e2aec019ac
6 changed files with 469 additions and 50 deletions
Download patch file Download diff file Expand all files Collapse all files

23
CHANGELOG.md
View file

@ -6,6 +6,29 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
## Unreleased
### Fixed
- **`/trekreview` verdicts are fail-closed.** A finding removed by Pass 2 or
Pass 3, and a reviewer whose payload was thrown away or never arrived, were
arithmetically identical to a finding that never existed — all three pushed
the verdict toward `ALLOW`. Measured before the fix: a BLOCKER with a
101-character title → `ALLOW`; a payload carrying one ad-hoc `rule_key` was
skipped whole at ingest, taking a valid BLOCKER sibling with it → `ALLOW`.
`lib/review/coordinator-contract.mjs` now splits every removal two ways —
**dropped** when the test refuted the finding as a claim about this codebase,
**unverified** otherwise — and a non-empty `unverified` bucket, or a reviewer
in `expectedReviewers` that did not report, forbids `ALLOW` (verdict `WARN`,
with `allow_blocked_by` naming why). The rule never *raises* a verdict: the
severity catalogue and the BLOCKER/MAJOR thresholds are untouched. New
exports: `classifySuppression`, `REFUTING_REASONS`, `UNVERIFIED_REASONS`;
`runContract` gains `unverified`, `missing_reviewers` and `allow_blocked_by`
(`suppressed` stays the union, so existing consumers keep their meaning).
Mirrored in `agents/review-coordinator.md` (Pass 2/3 fate columns, the new
§*Suppression is two-valued*, Pass 4 threshold table) and
`commands/trekreview.md` (Phase 5 reviewer accounting → STOP; Phase 6).
Driven test-first: 9 new tests in `tests/lib/coordinator-contract.test.mjs`,
incl. a known-positive control proving `ALLOW` is still reachable.
### Docs
- **`/trekresearch --engine deep-research`: document the real version window.**

103
agents/review-coordinator.md
View file

@ -84,31 +84,48 @@ identical `(file, rule_key)` and `line == 0` collide.
### Pass 2 — HubSpot Judge filters (3 criteria)
Drop findings that fail ANY of these filters:
Remove findings that fail ANY of these filters. **The `Removed as` column
is load-bearing** — see *Suppression is two-valued* below:
| Filter | Test | Drop if |
|--------|------|---------|
| Succinctness | `title.length ≤ 100` and `detail.length ≤ 800` chars | Title is a paragraph or detail is a wall of text |
| Accuracy | `file` resolves under the repo root AND `line` is plausible (≥ 0; ≤ file line count when known) | Path traversal escape, negative line, or impossibly large line number |
| Actionability | `recommended_action` is non-empty AND begins with an imperative verb | Empty action, "consider …" hedges, or restating the title |
| Filter | Test | Fails if | Removed as |
|--------|------|----------|------------|
| Succinctness | `title.length ≤ 100` and `detail.length ≤ 800` chars | Title is a paragraph or detail is a wall of text | `unverified` (`succinctness:title` / `succinctness:detail`) |
| Accuracy | `file` resolves under the repo root AND `line` is plausible (≥ 0; ≤ file line count when known) | Path traversal escape, negative line, or impossibly large line number | **dropped** (`accuracy:refuted`) |
| Actionability | `recommended_action` is non-empty AND begins with an imperative verb | Empty action, "consider …" hedges, or restating the title | `unverified` (`actionability:empty`) |
When dropping a finding, preserve a one-line note in the
Succinctness and Actionability read the finding's *packaging*; neither
examines the claim, so neither can establish the finding is unreal. Accuracy
does: a citation that escapes the repo root refutes the finding as a claim
about this codebase.
When removing a finding, preserve a one-line note in the
`Suppressed Findings` body section so the user knows why the count
shrank.
### Pass 3 — Cloudflare reasonableness (skipped in quick mode)
Drop findings that fail ANY of these tests:
Remove findings that fail ANY of these tests:
- **No file:line citation.** `file` is empty, or `line < 0`. Speculative
"code might break somewhere" findings have no anchor and are dropped.
- **Unknown rule_key.** `rule_key` is not in `RULE_CATALOGUE`. Reviewers
occasionally emit ad-hoc rule keys; the catalogue is the contract.
- **No file:line citation** → **dropped** (`no-citation`). `file` is empty,
or `line < 0`. Speculative "code might break somewhere" findings name no
location, so they make no checkable claim at all.
- **Unknown rule_key** → `unverified` (`unknown-rule_key`). `rule_key` is not
in `RULE_CATALOGUE`. Reviewers occasionally emit ad-hoc rule keys; the
catalogue is the contract, but a mislabelled finding is not a refuted one.
*(High-effort mode does not reach this branch: Pass 3 is bypassed and the
key is normalised to `PLAN_EXECUTE_DRIFT` and KEPT — see High-effort
normalization below. The two fates never apply to the same input.)*
- **Non-existent file.** `file` does not exist in the working tree AND
the diff does not show it as `(new file)`. Use Glob to verify.
the diff does not show it as `(new file)`. Use Glob to verify. **This test
has three outcomes, not two:** Glob resolves and the file is absent from
both tree and diff → **dropped** (`file-existence:refuted`); Glob resolves
and the file is present → keep; **Glob cannot decide** (path outside the
working tree, unreadable, or the tool errored) → `unverified`
(`file-existence:indeterminate`). Never collapse *unresolvable* into
*refuted*.
- **Catalogue severity mismatch.** `severity` does not match the rule's
catalogue tier (e.g., `MISSING_TEST` emitted as MINOR). Reset to the
catalogue tier; this is a correction, not a drop.
catalogue tier; this is a correction, neither a drop nor an unverified.
In `quick` mode, skip this pass entirely. Note the skip in the
Executive Summary so the reader knows reasonableness was not applied.
@ -125,6 +142,32 @@ purposes. This normalization happens BEFORE writing review.md,
ensuring all `rule_key` values in the final review match the
catalogue.
### Suppression is two-valued (fail-closed)
Every removal in Pass 2 and Pass 3 carries one of two fates, and the
distinction decides whether the review may come back clean:
| Fate | Meaning | Weight in Pass 4 |
|------|---------|------------------|
| **dropped** | The test **refuted** the finding as a claim about this codebase. | None. It weighs nothing, correctly. |
| **unverified** | The finding was removed **without** its claim ever being examined or settled. | Forbids `ALLOW`. |
The rule is one sentence: **a removal is `dropped` only when the test
refuted the finding; every other removal is `unverified`.** A reason you
cannot place is `unverified` — the default fails closed.
Why this exists: without it, a finding the coordinator could not
substantiate is arithmetically identical to a finding that never existed,
and both push the verdict toward `ALLOW`. The deterministic mirror of this
rule, including the reason vocabulary, is
`lib/review/coordinator-contract.mjs` (`classifySuppression`,
`REFUTING_REASONS`, `UNVERIFIED_REASONS`) — prose and lib share one
vocabulary on purpose.
**Unverified findings are not counted into a severity tier.** Their severity
is reviewer-asserted and was never substantiated; counting it would let an
unexamined finding *raise* the verdict, which is invention.
### Pass 4 — Compute verdict
Count findings by severity AFTER dedup and filtering. Verdict thresholds:
@ -133,7 +176,19 @@ Count findings by severity AFTER dedup and filtering. Verdict thresholds:
|--------|---------|
| `BLOCKER ≥ 1` | `BLOCK` |
| `BLOCKER == 0` AND `MAJOR ≥ 1` | `WARN` |
| `BLOCKER == 0` AND `MAJOR == 0` | `ALLOW` |
| `BLOCKER == 0` AND `MAJOR == 0` AND nothing `unverified` AND every reviewer reported | `ALLOW` |
| `BLOCKER == 0` AND `MAJOR == 0` AND (`unverified` non-empty OR a reviewer did not report) | `WARN` |
The fail-closed row never RAISES a verdict — it only withholds the clean
one. The worst case of a false `unverified` is `WARN` plus a stated reason;
the worst case of the old behaviour was a silent `ALLOW` over a live
BLOCKER.
**When `ALLOW` is withheld, the Executive Summary's FIRST sentence must say
so and name why** — e.g. "WARN: no blocking findings survived, but 1 finding
could not be verified (succinctness:title) and brief-conformance-reviewer did
not report." A withheld ALLOW that the reader cannot see is the same defect
in a new place.
Verdict is mechanical — never override. The verdict goes into the
trailing JSON block AND the Executive Summary's first sentence.
@ -181,8 +236,10 @@ prefix). Flow-style `findings: [a, b]` breaks the frontmatter parser.
5. `## Findings (MAJOR)` — one subsection per MAJOR finding.
6. `## Findings (MINOR)` — one subsection per MINOR finding.
7. `## Findings (SUGGESTION)` — one subsection per SUGGESTION finding.
8. `## Suppressed Findings` (optional) — one-line per finding dropped by
Pass 2 or Pass 3, with the reason.
8. `## Suppressed Findings` (optional) — one line per finding removed by
Pass 2 or Pass 3, with the reason AND its fate, tagged `[dropped]` or
`[unverified]`. Unverified lines come first: they are the ones that
withheld `ALLOW`.
9. `## Remediation Summary` — bullet count per severity + 1 sentence on
what /trekplan will consume.
@ -204,6 +261,7 @@ The LAST fenced block in the file is a `json` block:
{
"verdict": "BLOCK | WARN | ALLOW",
"counts": { "BLOCKER": N, "MAJOR": N, "MINOR": N, "SUGGESTION": N },
"allow_blocked_by": ["unverified:succinctness:title (1)", "missing-reviewer:brief-conformance-reviewer"],
"findings": [
{
"id": "<40-char-hex>",
@ -243,9 +301,14 @@ for the ID list.
the canonical 40-char SHA1 from `(file, line, rule_key, title)` using
the algorithm in `lib/parsers/finding-id.mjs`. The frontmatter
`findings:` list and the JSON block IDs must match.
- **Suppressed findings are accountable.** When you drop a finding via
Pass 2 or Pass 3, log it in `## Suppressed Findings` with the reason.
Silent drops break the audit trail.
- **Suppressed findings are accountable.** When you remove a finding via
Pass 2 or Pass 3, log it in `## Suppressed Findings` with the reason and
its fate (`[dropped]` / `[unverified]`). Silent drops break the audit
trail.
- **Never spend an unexamined finding as evidence of a clean review.** If a
removal did not refute the finding, it is `unverified` and `ALLOW` is off
the table. This is the one place where you may not be minimal: when in
doubt about a reason's fate, it is `unverified`.
- **No invention.** Never add a finding that did not appear in the
reviewer outputs. Never escalate a finding's severity beyond what the
catalogue specifies.

43
commands/trekreview.md
View file

@ -249,6 +249,37 @@ do not feed unvalidated findings to the coordinator.
In `quick` mode, launch only `code-correctness-reviewer`. The Executive
Summary will note the brief-conformance pass was skipped.
### Reviewer accounting — every expected reviewer MUST report
Write down the expected reviewer set BEFORE the spawn: both reviewers in
default mode, `code-correctness-reviewer` alone in `quick` mode. After the
spawn, account for each one by name.
**Zero findings from a silent reviewer is indistinguishable from zero findings
from a clean diff** — unless you check. A reviewer is *accounted for* only when
it returned a payload that validated. Three ways it fails to:
| Failure | Handling |
|---------|----------|
| Output fails the schema after the 2 bounded re-asks | STOP (already specified above) |
| Returned no final message at all | Re-ask that reviewer **once**. Still nothing → STOP. |
| Was never launched (spawn error, wrong mode) | STOP. |
**On STOP: name the reviewer and the failure, and do not proceed to Phase 6.**
Do not let the coordinator compute a verdict over a review one of whose
reviewers never spoke — the count would be complete-looking and wrong. This is
the same shape as the schema branch above ("do not feed unvalidated findings to
the coordinator"), applied to the other two ways a reviewer can go missing.
A reviewer that ran but never delivered is most often the return-channel
defect: check `~/.claude/projects/<proj>/<session>/subagents/agent-*.jsonl` for
its final assistant block before re-asking, and confirm no `name` parameter was
passed at the spawn (see the warning at the top of this phase).
If you proceed anyway under an explicit operator instruction, pass the expected
set to the coordinator as `expectedReviewers` so the missing reviewer at least
forbids `ALLOW` (`lib/review/coordinator-contract.mjs`, `missing_reviewers`).
## Phase 6 — Coordinator dedup + verdict
Launch `review-coordinator` (Agent tool) with the merged findings array
@ -259,10 +290,20 @@ The coordinator runs the 4-pass process documented in
1. **Dedup** by `(file, line, rule_key)` triplet.
2. **HubSpot Judge filters** — Succinctness, Accuracy, Actionability.
3. **Cloudflare reasonableness** — drop speculative or catalogue-violating
3. **Cloudflare reasonableness** — remove speculative or catalogue-violating
findings (skipped in `quick` mode).
4. **Verdict** — BLOCK / WARN / ALLOW per the threshold table.
**Fail-closed.** Every removal in Pass 2 and Pass 3 is either
**dropped** (the test refuted the finding as a claim about this codebase) or
**unverified** (the finding was removed without its claim ever being settled).
A non-empty `unverified` bucket forbids `ALLOW`; the verdict becomes `WARN` and
the Executive Summary's first sentence must say why. The fail-closed rule never
raises a verdict — it only withholds the clean one. Fate table, reason
vocabulary, and the `allow_blocked_by` field: `agents/review-coordinator.md`
§*Suppression is two-valued*, mirrored deterministically in
`lib/review/coordinator-contract.mjs`.
The coordinator's output is the full review.md content — frontmatter +
body sections + trailing JSON block. Do NOT re-run the reviewers based
on the coordinator's output.

16
docs/agent-return-channel-defect.md
View file

@ -167,10 +167,18 @@ arm produces correct final text at 66 lines and at 3730 lines alike, and only
delivery fails, identically at both. The 38 recovered findings were re-used
instead of re-run.
What it does **not** close: their four cells were all named, so the *returning*
(unnamed) arm still has no measurement above 66 lines. That a plain subagent
returns a 21 KB result at that scale is inferred, not observed. Stated as
inferred.
**A second external measurement closes the other half (akashic-intelligence
S27, commit `f168630`).** The gap left above was that all four S25 cells were
named, so the *returning* (unnamed) arm had no measurement above 66 lines.
S27 supplies one: denominator **2 of 2 unnamed agents**, against a 4370-line
plan; both returned, 30449 B and 10989 B, both valid JSON. The unnamed arm
therefore returns at full scale as observed fact, not as inference.
Two caveats, kept at the strength akashic itself stated them. The measurement
was taken by the repo that owns the finding, not by an independent third party.
And byte-identity between the returned string and the file on disk was not
proven — what is established is that a well-formed result of that size arrived,
not that it arrived unaltered.
Their PONG control also carries the same lesson as cell 1 above, in a third
repo: S25 reported that agent as having "gone idle without sending PONG". It

171
lib/review/coordinator-contract.mjs
View file

@ -19,8 +19,10 @@
// What IS implemented, purely: Pass 1 (triplet dedup → highest-severity-wins
// survivor + conformance tiebreak + detail concat + raised_by provenance),
// Pass 2 succinctness + actionability-presence, Pass 3 reasonableness
// (citation / unknown-rule_key drop, severity-mismatch correction), Pass 4
// verdict thresholds. No LLM, no network, no time, no randomness.
// (citation / unknown-rule_key suppression, severity-mismatch correction),
// Pass 4 verdict thresholds — fail-closed: a suppression that did not REFUTE
// the finding, and a reviewer that never reported, forbid ALLOW (see
// classifySuppression). No LLM, no network, no time, no randomness.
//
// Reuses: SEVERITY_VALUES / RULE_KEYS / getRule (rule-catalogue.mjs),
// computeFindingId (finding-id.mjs, triplet), validateFindings
@ -35,6 +37,71 @@ import { validateFindings } from './findings-schema.mjs';
export const JUDGE_TITLE_MAX = 100;
export const JUDGE_DETAIL_MAX = 800;
// ---- Suppression classification (fail-closed) --------------------------------
//
// A removal is `dropped` ONLY when the test refuted the finding as a claim
// about this codebase. Every other removal is `unverified`: the coordinator
// took the finding out of the count without ever establishing it was unreal,
// so it may not be spent as evidence of a clean review.
/**
* Reasons that REFUTE. `no-citation` is the only one this deterministic subset
* can emit: a finding whose `file` is empty or whose `line` is negative names
* no location, so it makes no checkable claim at all
* (agents/review-coordinator.md Pass 3 — "Speculative 'code might break
* somewhere' findings have no anchor").
*
* `accuracy:refuted` (Pass 2 Accuracy — a citation escaping the repo root) and
* `file-existence:refuted` (Pass 3 — absent from both working tree and diff)
* are emitted by the LLM coordinator, whose fs/judgement branches this module
* excludes. They are declared here anyway: the vocabulary is owned in one
* place so prose and lib cannot drift.
*/
export const REFUTING_REASONS = Object.freeze(new Set([
'no-citation',
'accuracy:refuted',
'file-existence:refuted',
]));
/**
* The reason vocabulary on the unverified side. `file-existence:indeterminate`
* is emitted by the LLM coordinator's Pass 3 (which runs the fs Glob this
* module deliberately excludes); the vocabulary is owned here so prose and lib
* cannot drift.
*/
export const UNVERIFIED_REASONS = Object.freeze([
'succinctness:title',
'succinctness:detail',
'actionability:empty',
'unknown-rule_key',
'file-existence:indeterminate',
]);
/**
* Classify a suppression reason. Anything not declared refuting is
* `unverified` — the default is fail-CLOSED, so a reason introduced later
* without a decision cannot silently move the verdict toward ALLOW.
* @param {string} reason
* @returns {'refuted'|'unverified'}
*/
export function classifySuppression(reason) {
return REFUTING_REASONS.has(reason) ? 'refuted' : 'unverified';
}
/**
* Tag a finding with its suppression reason and route it to the refuted
* (`dropped`) or the `unverified` bucket.
* @param {object} finding
* @param {string} reason
* @param {object[]} dropped
* @param {object[]} unverified
*/
function suppress(finding, reason, dropped, unverified) {
const tagged = { ...finding, suppressed_reason: reason };
if (classifySuppression(reason) === 'refuted') dropped.push(tagged);
else unverified.push(tagged);
}
/**
* Catalogue-tier rank of a severity: lower number = higher severity.
* BLOCKER=0 … SUGGESTION=3; an unknown severity ranks last.
@ -122,12 +189,18 @@ export function dedupByTriplet(findings) {
* (title > 100 or detail > 800 chars) and actionability (recommended_action,
* when present, must be a non-empty string). The imperative-verb test is
* excluded (LLM judgement).
*
* Both tests read a `.length`; neither examines the claim, so neither can
* establish the finding is unreal. Both therefore route to `unverified`.
* `dropped` stays in the signature for the refuting Pass-2 filter this subset
* excludes (Accuracy: a path-traversal escape IS a refutation).
* @param {object[]} findings
* @returns {{ kept: object[], dropped: object[] }}
* @returns {{ kept: object[], dropped: object[], unverified: object[] }}
*/
export function judgeFilter(findings) {
const kept = [];
const dropped = [];
const unverified = [];
for (const f of findings) {
const titleLen = (f.title ?? '').length;
const detailLen = (f.detail ?? '').length;
@ -138,31 +211,38 @@ export function judgeFilter(findings) {
(typeof f.recommended_action !== 'string' || f.recommended_action.trim().length === 0)) {
reason = 'actionability:empty';
}
if (reason) dropped.push({ ...f, suppressed_reason: reason });
if (reason) suppress(f, reason, dropped, unverified);
else kept.push(f);
}
return { kept, dropped };
return { kept, dropped, unverified };
}
/**
* Pass 3 — Cloudflare reasonableness (deterministic subset): drop findings
* with no citation (empty file / line < 0) or an unknown rule_key; CORRECT a
* severity that does not match the catalogue tier (a correction, not a drop).
* The fs file-existence glob is excluded (I/O).
* The fs file-existence glob is excluded (I/O) — its indeterminate branch is
* prose-side, tokenised as `file-existence:indeterminate`.
*
* `no-citation` REFUTES (the finding names no location, so it makes no
* checkable claim) and is dropped. `unknown-rule_key` does not: an ad-hoc key
* is a real defect wearing the wrong label — v5.1.1 high-effort mode already
* KEEPS these, normalised to PLAN_EXECUTE_DRIFT — so it routes to `unverified`.
* @param {object[]} findings
* @returns {{ kept: object[], dropped: object[] }}
* @returns {{ kept: object[], dropped: object[], unverified: object[] }}
*/
export function reasonablenessFilter(findings) {
const kept = [];
const dropped = [];
const unverified = [];
for (const f of findings) {
if (typeof f.file !== 'string' || f.file.length === 0 ||
(typeof f.line === 'number' && f.line < 0)) {
dropped.push({ ...f, suppressed_reason: 'no-citation' });
suppress(f, 'no-citation', dropped, unverified);
continue;
}
if (!RULE_KEYS.has(f.rule_key)) {
dropped.push({ ...f, suppressed_reason: 'unknown-rule_key' });
suppress(f, 'unknown-rule_key', dropped, unverified);
continue;
}
const rule = getRule(f.rule_key);
@ -172,44 +252,101 @@ export function reasonablenessFilter(findings) {
kept.push(f);
}
}
return { kept, dropped };
return { kept, dropped, unverified };
}
/**
* Pass 4 — compute the verdict from severity counts (after dedup + filtering).
* BLOCKER ≥ 1 → BLOCK; else MAJOR ≥ 1 → WARN; else ALLOW.
*
* FAIL-CLOSED: ALLOW additionally requires that nothing is `unverified` and
* that every expected reviewer reported. Neither ever RAISES a verdict — the
* severity thresholds are untouched — they only forbid the clean one, so the
* worst case of a false unverified is WARN plus a stated reason, never a
* silent pass. Unverified findings are NOT counted into a severity tier: their
* severity is reviewer-asserted and was never substantiated.
*
* @param {object[]} findings
* @returns {{ verdict: 'BLOCK'|'WARN'|'ALLOW', counts: Record<string, number> }}
* @param {{ unverified?: object[], missingReviewers?: string[] }} [options]
* @returns {{ verdict: 'BLOCK'|'WARN'|'ALLOW', counts: Record<string, number>, allow_blocked_by: string[] }}
*/
export function computeVerdict(findings) {
export function computeVerdict(findings, options = {}) {
const counts = { BLOCKER: 0, MAJOR: 0, MINOR: 0, SUGGESTION: 0 };
for (const f of findings) {
if (counts[f.severity] !== undefined) counts[f.severity] += 1;
}
const unverified = options.unverified ?? [];
const missingReviewers = options.missingReviewers ?? [];
const allow_blocked_by = [];
const byReason = new Map();
for (const f of unverified) {
const reason = f?.suppressed_reason ?? 'unspecified';
byReason.set(reason, (byReason.get(reason) ?? 0) + 1);
}
for (const [reason, n] of byReason) allow_blocked_by.push(`unverified:${reason} (${n})`);
for (const r of missingReviewers) allow_blocked_by.push(`missing-reviewer:${r}`);
let verdict;
if (counts.BLOCKER >= 1) verdict = 'BLOCK';
else if (counts.MAJOR >= 1) verdict = 'WARN';
else if (allow_blocked_by.length > 0) verdict = 'WARN';
else verdict = 'ALLOW';
return { verdict, counts };
return { verdict, counts, allow_blocked_by };
}
/**
* Run the full deterministic contract: ingest → Pass 1 → Pass 2 → Pass 3 → Pass 4.
*
* `options.expectedReviewers` names the reviewers this review was supposed to
* hear from. A reviewer that is absent from the payloads, or whose payload
* failed schema validation and was thrown away at ingest, lands in
* `missing_reviewers` and forbids ALLOW: an unread reviewer is an absent one,
* and zero findings from a silent reviewer must not read like zero findings
* from a clean diff.
*
* `suppressed` stays the UNION of `dropped` (refuted) and `unverified` so
* existing consumers keep their meaning; `unverified` is the subset that
* forbids ALLOW. Do not iterate both and count twice.
*
* @param {Array<{reviewer?: string, findings: object[]}>} reviewerPayloads
* @returns {{ verdict: string, counts: Record<string, number>, findings: object[], suppressed: object[], skipped: object[] }}
* @param {{ expectedReviewers?: string[] }} [options]
* @returns {{ verdict: string, counts: Record<string, number>, findings: object[], suppressed: object[], unverified: object[], skipped: object[], missing_reviewers: string[], allow_blocked_by: string[] }}
*/
export function runContract(reviewerPayloads) {
export function runContract(reviewerPayloads, options = {}) {
const { findings: ingested, skipped } = ingest(reviewerPayloads);
const deduped = dedupByTriplet(ingested);
const judged = judgeFilter(deduped);
const reasoned = reasonablenessFilter(judged.kept);
const { verdict, counts } = computeVerdict(reasoned.kept);
const unverified = [...judged.unverified, ...reasoned.unverified];
const reported = new Set();
for (const payload of reviewerPayloads) {
if (typeof payload?.reviewer === 'string' && payload.reviewer.length > 0) reported.add(payload.reviewer);
}
for (const s of skipped) reported.delete(s.reviewer);
const missing_reviewers = [];
for (const s of skipped) {
const name = s.reviewer ?? 'unnamed reviewer';
if (!missing_reviewers.includes(name)) missing_reviewers.push(name);
}
for (const r of options.expectedReviewers ?? []) {
if (!reported.has(r) && !missing_reviewers.includes(r)) missing_reviewers.push(r);
}
const { verdict, counts, allow_blocked_by } = computeVerdict(reasoned.kept, {
unverified,
missingReviewers: missing_reviewers,
});
return {
verdict,
counts,
findings: reasoned.kept,
suppressed: [...judged.dropped, ...reasoned.dropped],
suppressed: [...judged.dropped, ...judged.unverified, ...reasoned.dropped, ...reasoned.unverified],
unverified,
skipped,
missing_reviewers,
allow_blocked_by,
};
}

161
tests/lib/coordinator-contract.test.mjs
View file

@ -6,6 +6,9 @@
import { test } from 'node:test';
import { strict as assert } from 'node:assert';
import { readFileSync } from 'node:fs';
import { join, dirname } from 'node:path';
import { fileURLToPath } from 'node:url';
import {
severityRank,
ingest,
@ -13,9 +16,14 @@ import {
judgeFilter,
reasonablenessFilter,
computeVerdict,
classifySuppression,
REFUTING_REASONS,
UNVERIFIED_REASONS,
runContract,
} from '../../lib/review/coordinator-contract.mjs';
const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..', '..');
// ---- Pass 1 — dedup --------------------------------------------------------
test('dedupByTriplet — genuine cross-reviewer collapse (identical triplet) → 1, raised_by both', () => {
@ -98,29 +106,41 @@ test('computeVerdict — counts each severity tier', () => {
// ---- Pass 3 — reasonableness -----------------------------------------------
test('reasonablenessFilter — drops unknown rule_key + citation-less, corrects severity mismatch', () => {
test('reasonablenessFilter — citation-less is REFUTED, unknown rule_key is UNVERIFIED, severity mismatch corrected', () => {
// Contract change (fail-closed): only `no-citation` refutes. An ad-hoc
// rule_key is a real defect wearing the wrong label — v5.1.1 high-effort mode
// already keeps those, normalised to PLAN_EXECUTE_DRIFT.
const r = reasonablenessFilter([
{ file: 'x.mjs', line: 1, rule_key: 'NOPE_KEY', severity: 'BLOCKER' }, // unknown → drop
{ file: 'x.mjs', line: 1, rule_key: 'NOPE_KEY', severity: 'BLOCKER' }, // unknown → unverified
{ file: '', line: 1, rule_key: 'MISSING_TEST', severity: 'MAJOR' }, // no file → drop
{ file: 'x.mjs', line: -1, rule_key: 'MISSING_TEST', severity: 'MAJOR' }, // line < 0 → drop
{ file: 'x.mjs', line: 1, rule_key: 'MISSING_TEST', severity: 'MINOR' }, // catalogue is MAJOR → correct, keep
]);
assert.equal(r.kept.length, 1);
assert.equal(r.dropped.length, 3);
assert.equal(r.dropped.length, 2);
assert.deepEqual(r.dropped.map((f) => f.suppressed_reason), ['no-citation', 'no-citation']);
assert.equal(r.unverified.length, 1);
assert.equal(r.unverified[0].suppressed_reason, 'unknown-rule_key');
assert.equal(r.kept[0].severity, 'MAJOR');
assert.equal(r.kept[0].original_severity, 'MINOR');
});
// ---- Pass 2 — judge --------------------------------------------------------
test('judgeFilter — drops over-long title and empty recommended_action', () => {
test('judgeFilter — over-long title and empty recommended_action are UNVERIFIED, not dropped', () => {
// Contract change (fail-closed): both implemented Pass 2 tests read a
// `.length` and never examine the claim, so neither refutes the finding.
// `dropped` is empty here on purpose — the refuting Pass 2 filter (Accuracy)
// is the one this deterministic subset excludes.
const j = judgeFilter([
{ file: 'x.mjs', line: 1, rule_key: 'MISSING_TEST', severity: 'MAJOR', title: 'x'.repeat(101) }, // too long → drop
{ file: 'x.mjs', line: 2, rule_key: 'MISSING_TEST', severity: 'MAJOR', title: 'ok', recommended_action: ' ' }, // empty action → drop
{ file: 'x.mjs', line: 1, rule_key: 'MISSING_TEST', severity: 'MAJOR', title: 'x'.repeat(101) }, // too long → unverified
{ file: 'x.mjs', line: 2, rule_key: 'MISSING_TEST', severity: 'MAJOR', title: 'ok', recommended_action: ' ' }, // empty action → unverified
{ file: 'x.mjs', line: 3, rule_key: 'MISSING_TEST', severity: 'MAJOR', title: 'ok' }, // keep (no action field is fine)
]);
assert.equal(j.kept.length, 1);
assert.equal(j.dropped.length, 2);
assert.equal(j.dropped.length, 0);
assert.equal(j.unverified.length, 2);
assert.deepEqual(j.unverified.map((f) => f.suppressed_reason), ['succinctness:title', 'actionability:empty']);
});
// ---- ingest ----------------------------------------------------------------
@ -161,3 +181,130 @@ test('runContract — deterministic: identical input yields identical output', (
];
assert.deepEqual(runContract(input), runContract(input));
});
// ---- Fail-closed: the `unverified` bucket (ORDRE 834432937) -----------------
//
// The defect: a finding REMOVED by Pass 2/Pass 3, and a reviewer whose payload
// was thrown away or never arrived, are all arithmetically identical to a
// finding that never existed -- they push the verdict toward ALLOW. Measured
// before the fix (probe, 2026-09-01): an over-long-title BLOCKER -> ALLOW; a
// payload with one ad-hoc rule_key -> the whole payload skipped, its valid
// BLOCKER sibling gone -> ALLOW.
//
// The rule under test: a removal is `dropped` ONLY when the test refutes the
// finding as a claim about this codebase. Every other removal is `unverified`,
// and a non-empty `unverified` -- or a reviewer that did not report -- forbids
// ALLOW.
test('classifySuppression — only no-citation refutes; form and taxonomy failures are unverified', () => {
assert.equal(classifySuppression('no-citation'), 'refuted',
'a finding that names no location makes no checkable claim');
assert.equal(classifySuppression('succinctness:title'), 'unverified');
assert.equal(classifySuppression('succinctness:detail'), 'unverified');
assert.equal(classifySuppression('actionability:empty'), 'unverified');
assert.equal(classifySuppression('unknown-rule_key'), 'unverified');
assert.equal(classifySuppression('file-existence:indeterminate'), 'unverified');
assert.equal(classifySuppression('something-nobody-declared'), 'unverified',
'an unclassified reason must fail CLOSED, not open');
});
test('computeVerdict — non-empty unverified forbids ALLOW but never downgrades BLOCK or WARN', () => {
const u = [{ file: 'x.mjs', line: 1, rule_key: 'MISSING_TEST', severity: 'BLOCKER' }];
const withUnverified = computeVerdict([], { unverified: u });
assert.equal(withUnverified.verdict, 'WARN', 'ALLOW is forbidden while anything is unverified');
assert.deepEqual(withUnverified.counts, { BLOCKER: 0, MAJOR: 0, MINOR: 0, SUGGESTION: 0 },
'the unverified finding is NOT counted into a severity tier');
assert.ok(withUnverified.allow_blocked_by.length > 0);
assert.equal(computeVerdict([{ severity: 'BLOCKER' }], { unverified: u }).verdict, 'BLOCK',
'BLOCK stands regardless of the unverified bucket');
assert.equal(computeVerdict([{ severity: 'MAJOR' }], { unverified: u }).verdict, 'WARN');
assert.equal(computeVerdict([], { unverified: [] }).verdict, 'ALLOW',
'known-positive control: an empty unverified bucket still allows ALLOW');
});
test('computeVerdict — a reviewer that did not report forbids ALLOW', () => {
const r = computeVerdict([], { missingReviewers: ['brief-conformance-reviewer'] });
assert.equal(r.verdict, 'WARN');
assert.ok(r.allow_blocked_by.some((x) => x.includes('brief-conformance-reviewer')));
});
test('runContract — a BLOCKER dropped for an over-long title cannot yield ALLOW', () => {
// Pass 2 succinctness reads `.length`. It never examines the claim, so it
// cannot establish the finding is unreal -- it is unverified, not refuted.
const result = runContract([
{ reviewer: 'code-correctness-reviewer', findings: [
{ file: 'lib/auth/jwt.mjs', line: 19, rule_key: 'SECURITY_INJECTION', severity: 'BLOCKER', title: 'x'.repeat(101), detail: 'algo taken from the JWT header' },
] },
]);
assert.notEqual(result.verdict, 'ALLOW', 'an unsubstantiated BLOCKER must never clear the review');
assert.equal(result.findings.length, 0, 'it is still not a kept finding');
assert.equal(result.unverified.length, 1);
assert.equal(result.unverified[0].suppressed_reason, 'succinctness:title');
assert.equal(result.suppressed.length, 1, 'suppressed stays the union of dropped + unverified');
});
test('runContract — a schema-invalid payload cannot yield ALLOW (an unread reviewer is an absent one)', () => {
// Measured: one ad-hoc rule_key invalidates the WHOLE payload at ingest, so a
// valid BLOCKER sibling disappears with it. That must not read as "clean".
const result = runContract([
{ reviewer: 'code-correctness-reviewer', findings: [
{ file: 'lib/auth/jwt.mjs', line: 19, rule_key: 'SECURITY_INJECTION', severity: 'BLOCKER', title: 'real' },
{ file: 'x.mjs', line: 1, rule_key: 'NOPE_KEY', severity: 'MINOR', title: 'ad-hoc key' },
] },
]);
assert.equal(result.skipped.length, 1);
assert.notEqual(result.verdict, 'ALLOW');
assert.ok(result.allow_blocked_by.some((x) => x.includes('code-correctness-reviewer')));
});
test('runContract — a reviewer named in expectedReviewers that never reported cannot yield ALLOW', () => {
const result = runContract(
[{ reviewer: 'code-correctness-reviewer', findings: [] }],
{ expectedReviewers: ['code-correctness-reviewer', 'brief-conformance-reviewer'] },
);
assert.deepEqual(result.missing_reviewers, ['brief-conformance-reviewer']);
assert.notEqual(result.verdict, 'ALLOW');
});
test('runContract — known-positive control: every reviewer reported, nothing suppressed → ALLOW', () => {
// Proves ALLOW is still REACHABLE. Without this, "no ALLOW" is not a
// fail-closed contract, only a broken one.
const result = runContract(
[
{ reviewer: 'code-correctness-reviewer', findings: [
{ file: 'a.mjs', line: 1, rule_key: 'MISSING_ERROR_HANDLING', severity: 'MINOR', title: 'unguarded await', recommended_action: 'Wrap the await in a try/catch.' },
] },
{ reviewer: 'brief-conformance-reviewer', findings: [] },
],
{ expectedReviewers: ['code-correctness-reviewer', 'brief-conformance-reviewer'] },
);
assert.equal(result.verdict, 'ALLOW');
assert.equal(result.unverified.length, 0);
assert.deepEqual(result.missing_reviewers, []);
assert.deepEqual(result.allow_blocked_by, []);
});
test('classifySuppression — the refuting reasons the LLM coordinator emits are declared here too', () => {
// agents/review-coordinator.md Pass 2 "Accuracy" and Pass 3 "Non-existent
// file" DO refute (a citation outside the repo root, a file absent from both
// tree and diff). Both are fs/judgement branches this deterministic subset
// excludes, but the vocabulary is owned here so prose and lib cannot drift.
assert.equal(classifySuppression('accuracy:refuted'), 'refuted');
assert.equal(classifySuppression('file-existence:refuted'), 'refuted');
assert.equal(classifySuppression('file-existence:indeterminate'), 'unverified',
'unresolvable must never collapse into refuted');
});
test('suppression vocabulary — the two sets are disjoint and every reason is documented in the prose', () => {
const refuting = [...REFUTING_REASONS];
const overlap = refuting.filter((r) => UNVERIFIED_REASONS.includes(r));
assert.deepEqual(overlap, [], 'a reason cannot be both refuting and unverified');
const prose = readFileSync(join(ROOT, 'agents/review-coordinator.md'), 'utf-8');
assert.ok(prose.includes('review-coordinator'), 'known-positive control: the prose file loaded');
for (const reason of [...refuting, ...UNVERIFIED_REASONS]) {
assert.ok(prose.includes(reason),
`reason "${reason}" is declared in the lib but never documented in agents/review-coordinator.md`);
}
});