fix(review): fail-closed verdicts - an unsubstantiated finding can no longer yield ALLOW
computeVerdict counted only the findings handed to it (reasoned.kept), so a
finding removed by Pass 2 or Pass 3, and a reviewer whose payload was thrown
away or never arrived, were arithmetically identical to a finding that never
existed. All three pushed the verdict toward ALLOW.
Measured before the fix (probes, 2026-09-01):
- a BLOCKER with a 101-character title -> ALLOW (Pass 2 succinctness)
- a payload with one ad-hoc rule_key is skipped WHOLE at ingest, taking a
valid BLOCKER sibling with it -> ALLOW
- a reviewer that never reported -> ALLOW
Pass 3's own no-citation / unknown-rule_key branches turned out unreachable
through runContract (validateFindings rejects those payloads first), so the
reachable exposure was Pass 2 plus the skipped/absent reviewer.
THE OPEN DESIGN DECISION, and why it went against the order's default.
The order proposed: indeterminate file-existence YES, plain succinctness NO
("a too-long finding is not an uncertain finding"). I kept the first and
overrode the second, on one principle:
A removal is `dropped` only when the test REFUTED the finding as a claim
about this codebase. Every other removal is `unverified`.
Succinctness and actionability read a `.length`. They never examine the claim,
so they cannot establish the finding is unreal - and dropping a BLOCKER for a
101-character title is precisely the fail-open shape being fixed. Three things
settled it:
1. Under the order's default the fix would have been almost inert. Pass 3's
drop branches are unreachable via runContract, so leaving Pass 2 out would
have left the only reachable finding-level exposure open.
2. Cost asymmetry, priced rather than asserted: the verdict is not a gate.
Handover 6 feeds `findings` filtered to BLOCKER+MAJOR into /trekplan
(commands/trekplan.md:218); `verdict` is optional metadata
(docs/HANDOVER-CONTRACTS.md:353). Nothing loops or re-plans on WARN. So a
false `unverified` costs WARN plus a printed reason; a false drop costs a
silent ALLOW over a live BLOCKER.
3. unknown-rule_key joins them for the same reason: an ad-hoc key is a real
defect wearing the wrong label, and v5.1.1 high-effort mode already KEEPS
those, normalised to PLAN_EXECUTE_DRIFT. Refuting them at normal effort
while keeping them at high effort would be incoherent.
no-citation stays a drop: a finding whose file is empty or whose line is
negative names no location, so it makes no checkable claim at all - the one
deterministic refutation, and what the Pass 3 prose already said it was.
Iron Law: tests/lib/coordinator-contract.test.mjs first, red (missing export +
the three measured ALLOWs), then production code. Two existing assertions were
updated AFTER implementation as contract changes, not to make the red pass.
A known-positive control pins that ALLOW is still reachable - without it,
"no ALLOW" is not a fail-closed contract, only a broken one.
lib/review/coordinator-contract.mjs
+ classifySuppression / REFUTING_REASONS / UNVERIFIED_REASONS - one
vocabulary owned by the lib, including the tokens only the LLM
coordinator emits (accuracy:refuted, file-existence:refuted/indeterminate),
so prose and lib cannot drift. Unclassified reasons default to unverified:
the default fails closed.
~ judgeFilter / reasonablenessFilter return {kept, dropped, unverified}
~ computeVerdict(findings, {unverified, missingReviewers}) -> + allow_blocked_by.
Never raises a verdict, only withholds ALLOW. Unverified findings are NOT
counted into a severity tier: their severity was never substantiated, and
counting it would be invention.
~ runContract(payloads, {expectedReviewers}) -> + unverified,
missing_reviewers, allow_blocked_by. `suppressed` stays the union of
dropped + unverified, so existing consumers (gold-eval) keep their meaning.
agents/review-coordinator.md - Pass 2/3 tables gain a fate column, new
"Suppression is two-valued" section, Pass 4 threshold table gains the two
fail-closed rows, Executive Summary must state a withheld ALLOW, Suppressed
Findings tags each line [dropped]/[unverified]. Pass 3's unknown-rule_key
bullet explicitly says high-effort does not reach that branch, so the same
input never has two documented fates.
commands/trekreview.md - Phase 5 "Reviewer accounting": the expected set is
written down before the spawn, a silent reviewer gets one re-ask and then
STOP. That extends the pattern already in the file (schema failure -> 2
bounded re-asks -> "do not feed unvalidated findings to the coordinator") to
the other two ways a reviewer goes missing, rather than softening it to WARN.
The lib's missing_reviewers stays as belt-and-braces for direct callers.
docs/agent-return-channel-defect.md - the "inferred, not observed" caveat on
the unnamed arm above 66 lines is struck: akashic-intelligence S27
(f168630) measured 2/2 unnamed agents returning against a 4370-line plan,
30449 B and 10989 B, both valid JSON. Recorded with akashic's own two
caveats intact - the measurer owns the finding, and byte-identity between
the returned string and the file on disk was not proven. The separate S25
named-arm figures are left standing; these are two measurements, not a
correction of one by the other.
No release, no version bump, no tag, no catalogue ref, no Workflow port.
Suite 1025 (1023/0/2) -> 1034 (1032/0/2), 0 failures.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
d650ff3bac
commit
e2aec019ac
6 changed files with 469 additions and 50 deletions
|
|
@ -84,31 +84,48 @@ identical `(file, rule_key)` and `line == 0` collide.
|
|||
|
||||
### Pass 2 — HubSpot Judge filters (3 criteria)
|
||||
|
||||
Drop findings that fail ANY of these filters:
|
||||
Remove findings that fail ANY of these filters. **The `Removed as` column
|
||||
is load-bearing** — see *Suppression is two-valued* below:
|
||||
|
||||
| Filter | Test | Drop if |
|
||||
|--------|------|---------|
|
||||
| Succinctness | `title.length ≤ 100` and `detail.length ≤ 800` chars | Title is a paragraph or detail is a wall of text |
|
||||
| Accuracy | `file` resolves under the repo root AND `line` is plausible (≥ 0; ≤ file line count when known) | Path traversal escape, negative line, or impossibly large line number |
|
||||
| Actionability | `recommended_action` is non-empty AND begins with an imperative verb | Empty action, "consider …" hedges, or restating the title |
|
||||
| Filter | Test | Fails if | Removed as |
|
||||
|--------|------|----------|------------|
|
||||
| Succinctness | `title.length ≤ 100` and `detail.length ≤ 800` chars | Title is a paragraph or detail is a wall of text | `unverified` (`succinctness:title` / `succinctness:detail`) |
|
||||
| Accuracy | `file` resolves under the repo root AND `line` is plausible (≥ 0; ≤ file line count when known) | Path traversal escape, negative line, or impossibly large line number | **dropped** (`accuracy:refuted`) |
|
||||
| Actionability | `recommended_action` is non-empty AND begins with an imperative verb | Empty action, "consider …" hedges, or restating the title | `unverified` (`actionability:empty`) |
|
||||
|
||||
When dropping a finding, preserve a one-line note in the
|
||||
Succinctness and Actionability read the finding's *packaging*; neither
|
||||
examines the claim, so neither can establish the finding is unreal. Accuracy
|
||||
does: a citation that escapes the repo root refutes the finding as a claim
|
||||
about this codebase.
|
||||
|
||||
When removing a finding, preserve a one-line note in the
|
||||
`Suppressed Findings` body section so the user knows why the count
|
||||
shrank.
|
||||
|
||||
### Pass 3 — Cloudflare reasonableness (skipped in quick mode)
|
||||
|
||||
Drop findings that fail ANY of these tests:
|
||||
Remove findings that fail ANY of these tests:
|
||||
|
||||
- **No file:line citation.** `file` is empty, or `line < 0`. Speculative
|
||||
"code might break somewhere" findings have no anchor and are dropped.
|
||||
- **Unknown rule_key.** `rule_key` is not in `RULE_CATALOGUE`. Reviewers
|
||||
occasionally emit ad-hoc rule keys; the catalogue is the contract.
|
||||
- **No file:line citation** → **dropped** (`no-citation`). `file` is empty,
|
||||
or `line < 0`. Speculative "code might break somewhere" findings name no
|
||||
location, so they make no checkable claim at all.
|
||||
- **Unknown rule_key** → `unverified` (`unknown-rule_key`). `rule_key` is not
|
||||
in `RULE_CATALOGUE`. Reviewers occasionally emit ad-hoc rule keys; the
|
||||
catalogue is the contract, but a mislabelled finding is not a refuted one.
|
||||
*(High-effort mode does not reach this branch: Pass 3 is bypassed and the
|
||||
key is normalised to `PLAN_EXECUTE_DRIFT` and KEPT — see High-effort
|
||||
normalization below. The two fates never apply to the same input.)*
|
||||
- **Non-existent file.** `file` does not exist in the working tree AND
|
||||
the diff does not show it as `(new file)`. Use Glob to verify.
|
||||
the diff does not show it as `(new file)`. Use Glob to verify. **This test
|
||||
has three outcomes, not two:** Glob resolves and the file is absent from
|
||||
both tree and diff → **dropped** (`file-existence:refuted`); Glob resolves
|
||||
and the file is present → keep; **Glob cannot decide** (path outside the
|
||||
working tree, unreadable, or the tool errored) → `unverified`
|
||||
(`file-existence:indeterminate`). Never collapse *unresolvable* into
|
||||
*refuted*.
|
||||
- **Catalogue severity mismatch.** `severity` does not match the rule's
|
||||
catalogue tier (e.g., `MISSING_TEST` emitted as MINOR). Reset to the
|
||||
catalogue tier; this is a correction, not a drop.
|
||||
catalogue tier; this is a correction, neither a drop nor an unverified.
|
||||
|
||||
In `quick` mode, skip this pass entirely. Note the skip in the
|
||||
Executive Summary so the reader knows reasonableness was not applied.
|
||||
|
|
@ -125,6 +142,32 @@ purposes. This normalization happens BEFORE writing review.md,
|
|||
ensuring all `rule_key` values in the final review match the
|
||||
catalogue.
|
||||
|
||||
### Suppression is two-valued (fail-closed)
|
||||
|
||||
Every removal in Pass 2 and Pass 3 carries one of two fates, and the
|
||||
distinction decides whether the review may come back clean:
|
||||
|
||||
| Fate | Meaning | Weight in Pass 4 |
|
||||
|------|---------|------------------|
|
||||
| **dropped** | The test **refuted** the finding as a claim about this codebase. | None. It weighs nothing, correctly. |
|
||||
| **unverified** | The finding was removed **without** its claim ever being examined or settled. | Forbids `ALLOW`. |
|
||||
|
||||
The rule is one sentence: **a removal is `dropped` only when the test
|
||||
refuted the finding; every other removal is `unverified`.** A reason you
|
||||
cannot place is `unverified` — the default fails closed.
|
||||
|
||||
Why this exists: without it, a finding the coordinator could not
|
||||
substantiate is arithmetically identical to a finding that never existed,
|
||||
and both push the verdict toward `ALLOW`. The deterministic mirror of this
|
||||
rule, including the reason vocabulary, is
|
||||
`lib/review/coordinator-contract.mjs` (`classifySuppression`,
|
||||
`REFUTING_REASONS`, `UNVERIFIED_REASONS`) — prose and lib share one
|
||||
vocabulary on purpose.
|
||||
|
||||
**Unverified findings are not counted into a severity tier.** Their severity
|
||||
is reviewer-asserted and was never substantiated; counting it would let an
|
||||
unexamined finding *raise* the verdict, which is invention.
|
||||
|
||||
### Pass 4 — Compute verdict
|
||||
|
||||
Count findings by severity AFTER dedup and filtering. Verdict thresholds:
|
||||
|
|
@ -133,7 +176,19 @@ Count findings by severity AFTER dedup and filtering. Verdict thresholds:
|
|||
|--------|---------|
|
||||
| `BLOCKER ≥ 1` | `BLOCK` |
|
||||
| `BLOCKER == 0` AND `MAJOR ≥ 1` | `WARN` |
|
||||
| `BLOCKER == 0` AND `MAJOR == 0` | `ALLOW` |
|
||||
| `BLOCKER == 0` AND `MAJOR == 0` AND nothing `unverified` AND every reviewer reported | `ALLOW` |
|
||||
| `BLOCKER == 0` AND `MAJOR == 0` AND (`unverified` non-empty OR a reviewer did not report) | `WARN` |
|
||||
|
||||
The fail-closed row never RAISES a verdict — it only withholds the clean
|
||||
one. The worst case of a false `unverified` is `WARN` plus a stated reason;
|
||||
the worst case of the old behaviour was a silent `ALLOW` over a live
|
||||
BLOCKER.
|
||||
|
||||
**When `ALLOW` is withheld, the Executive Summary's FIRST sentence must say
|
||||
so and name why** — e.g. "WARN: no blocking findings survived, but 1 finding
|
||||
could not be verified (succinctness:title) and brief-conformance-reviewer did
|
||||
not report." A withheld ALLOW that the reader cannot see is the same defect
|
||||
in a new place.
|
||||
|
||||
Verdict is mechanical — never override. The verdict goes into the
|
||||
trailing JSON block AND the Executive Summary's first sentence.
|
||||
|
|
@ -181,8 +236,10 @@ prefix). Flow-style `findings: [a, b]` breaks the frontmatter parser.
|
|||
5. `## Findings (MAJOR)` — one subsection per MAJOR finding.
|
||||
6. `## Findings (MINOR)` — one subsection per MINOR finding.
|
||||
7. `## Findings (SUGGESTION)` — one subsection per SUGGESTION finding.
|
||||
8. `## Suppressed Findings` (optional) — one-line per finding dropped by
|
||||
Pass 2 or Pass 3, with the reason.
|
||||
8. `## Suppressed Findings` (optional) — one line per finding removed by
|
||||
Pass 2 or Pass 3, with the reason AND its fate, tagged `[dropped]` or
|
||||
`[unverified]`. Unverified lines come first: they are the ones that
|
||||
withheld `ALLOW`.
|
||||
9. `## Remediation Summary` — bullet count per severity + 1 sentence on
|
||||
what /trekplan will consume.
|
||||
|
||||
|
|
@ -204,6 +261,7 @@ The LAST fenced block in the file is a `json` block:
|
|||
{
|
||||
"verdict": "BLOCK | WARN | ALLOW",
|
||||
"counts": { "BLOCKER": N, "MAJOR": N, "MINOR": N, "SUGGESTION": N },
|
||||
"allow_blocked_by": ["unverified:succinctness:title (1)", "missing-reviewer:brief-conformance-reviewer"],
|
||||
"findings": [
|
||||
{
|
||||
"id": "<40-char-hex>",
|
||||
|
|
@ -243,9 +301,14 @@ for the ID list.
|
|||
the canonical 40-char SHA1 from `(file, line, rule_key, title)` using
|
||||
the algorithm in `lib/parsers/finding-id.mjs`. The frontmatter
|
||||
`findings:` list and the JSON block IDs must match.
|
||||
- **Suppressed findings are accountable.** When you drop a finding via
|
||||
Pass 2 or Pass 3, log it in `## Suppressed Findings` with the reason.
|
||||
Silent drops break the audit trail.
|
||||
- **Suppressed findings are accountable.** When you remove a finding via
|
||||
Pass 2 or Pass 3, log it in `## Suppressed Findings` with the reason and
|
||||
its fate (`[dropped]` / `[unverified]`). Silent drops break the audit
|
||||
trail.
|
||||
- **Never spend an unexamined finding as evidence of a clean review.** If a
|
||||
removal did not refute the finding, it is `unverified` and `ALLOW` is off
|
||||
the table. This is the one place where you may not be minimal: when in
|
||||
doubt about a reason's fate, it is `unverified`.
|
||||
- **No invention.** Never add a finding that did not appear in the
|
||||
reviewer outputs. Never escalate a finding's severity beyond what the
|
||||
catalogue specifies.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue