fix(review): fail-closed verdicts - an unsubstantiated finding can no longer yield ALLOW
computeVerdict counted only the findings handed to it (reasoned.kept), so a
finding removed by Pass 2 or Pass 3, and a reviewer whose payload was thrown
away or never arrived, were arithmetically identical to a finding that never
existed. All three pushed the verdict toward ALLOW.
Measured before the fix (probes, 2026-09-01):
- a BLOCKER with a 101-character title -> ALLOW (Pass 2 succinctness)
- a payload with one ad-hoc rule_key is skipped WHOLE at ingest, taking a
valid BLOCKER sibling with it -> ALLOW
- a reviewer that never reported -> ALLOW
Pass 3's own no-citation / unknown-rule_key branches turned out unreachable
through runContract (validateFindings rejects those payloads first), so the
reachable exposure was Pass 2 plus the skipped/absent reviewer.
THE OPEN DESIGN DECISION, and why it went against the order's default.
The order proposed: indeterminate file-existence YES, plain succinctness NO
("a too-long finding is not an uncertain finding"). I kept the first and
overrode the second, on one principle:
A removal is `dropped` only when the test REFUTED the finding as a claim
about this codebase. Every other removal is `unverified`.
Succinctness and actionability read a `.length`. They never examine the claim,
so they cannot establish the finding is unreal - and dropping a BLOCKER for a
101-character title is precisely the fail-open shape being fixed. Three things
settled it:
1. Under the order's default the fix would have been almost inert. Pass 3's
drop branches are unreachable via runContract, so leaving Pass 2 out would
have left the only reachable finding-level exposure open.
2. Cost asymmetry, priced rather than asserted: the verdict is not a gate.
Handover 6 feeds `findings` filtered to BLOCKER+MAJOR into /trekplan
(commands/trekplan.md:218); `verdict` is optional metadata
(docs/HANDOVER-CONTRACTS.md:353). Nothing loops or re-plans on WARN. So a
false `unverified` costs WARN plus a printed reason; a false drop costs a
silent ALLOW over a live BLOCKER.
3. unknown-rule_key joins them for the same reason: an ad-hoc key is a real
defect wearing the wrong label, and v5.1.1 high-effort mode already KEEPS
those, normalised to PLAN_EXECUTE_DRIFT. Refuting them at normal effort
while keeping them at high effort would be incoherent.
no-citation stays a drop: a finding whose file is empty or whose line is
negative names no location, so it makes no checkable claim at all - the one
deterministic refutation, and what the Pass 3 prose already said it was.
Iron Law: tests/lib/coordinator-contract.test.mjs first, red (missing export +
the three measured ALLOWs), then production code. Two existing assertions were
updated AFTER implementation as contract changes, not to make the red pass.
A known-positive control pins that ALLOW is still reachable - without it,
"no ALLOW" is not a fail-closed contract, only a broken one.
lib/review/coordinator-contract.mjs
+ classifySuppression / REFUTING_REASONS / UNVERIFIED_REASONS - one
vocabulary owned by the lib, including the tokens only the LLM
coordinator emits (accuracy:refuted, file-existence:refuted/indeterminate),
so prose and lib cannot drift. Unclassified reasons default to unverified:
the default fails closed.
~ judgeFilter / reasonablenessFilter return {kept, dropped, unverified}
~ computeVerdict(findings, {unverified, missingReviewers}) -> + allow_blocked_by.
Never raises a verdict, only withholds ALLOW. Unverified findings are NOT
counted into a severity tier: their severity was never substantiated, and
counting it would be invention.
~ runContract(payloads, {expectedReviewers}) -> + unverified,
missing_reviewers, allow_blocked_by. `suppressed` stays the union of
dropped + unverified, so existing consumers (gold-eval) keep their meaning.
agents/review-coordinator.md - Pass 2/3 tables gain a fate column, new
"Suppression is two-valued" section, Pass 4 threshold table gains the two
fail-closed rows, Executive Summary must state a withheld ALLOW, Suppressed
Findings tags each line [dropped]/[unverified]. Pass 3's unknown-rule_key
bullet explicitly says high-effort does not reach that branch, so the same
input never has two documented fates.
commands/trekreview.md - Phase 5 "Reviewer accounting": the expected set is
written down before the spawn, a silent reviewer gets one re-ask and then
STOP. That extends the pattern already in the file (schema failure -> 2
bounded re-asks -> "do not feed unvalidated findings to the coordinator") to
the other two ways a reviewer goes missing, rather than softening it to WARN.
The lib's missing_reviewers stays as belt-and-braces for direct callers.
docs/agent-return-channel-defect.md - the "inferred, not observed" caveat on
the unnamed arm above 66 lines is struck: akashic-intelligence S27
(f168630) measured 2/2 unnamed agents returning against a 4370-line plan,
30449 B and 10989 B, both valid JSON. Recorded with akashic's own two
caveats intact - the measurer owns the finding, and byte-identity between
the returned string and the file on disk was not proven. The separate S25
named-arm figures are left standing; these are two measurements, not a
correction of one by the other.
No release, no version bump, no tag, no catalogue ref, no Workflow port.
Suite 1025 (1023/0/2) -> 1034 (1032/0/2), 0 failures.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
d650ff3bac
commit
e2aec019ac
6 changed files with 469 additions and 50 deletions
23
CHANGELOG.md
23
CHANGELOG.md
|
|
@ -6,6 +6,29 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||||
|
|
||||||
## Unreleased
|
## Unreleased
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- **`/trekreview` verdicts are fail-closed.** A finding removed by Pass 2 or
|
||||||
|
Pass 3, and a reviewer whose payload was thrown away or never arrived, were
|
||||||
|
arithmetically identical to a finding that never existed — all three pushed
|
||||||
|
the verdict toward `ALLOW`. Measured before the fix: a BLOCKER with a
|
||||||
|
101-character title → `ALLOW`; a payload carrying one ad-hoc `rule_key` was
|
||||||
|
skipped whole at ingest, taking a valid BLOCKER sibling with it → `ALLOW`.
|
||||||
|
`lib/review/coordinator-contract.mjs` now splits every removal two ways —
|
||||||
|
**dropped** when the test refuted the finding as a claim about this codebase,
|
||||||
|
**unverified** otherwise — and a non-empty `unverified` bucket, or a reviewer
|
||||||
|
in `expectedReviewers` that did not report, forbids `ALLOW` (verdict `WARN`,
|
||||||
|
with `allow_blocked_by` naming why). The rule never *raises* a verdict: the
|
||||||
|
severity catalogue and the BLOCKER/MAJOR thresholds are untouched. New
|
||||||
|
exports: `classifySuppression`, `REFUTING_REASONS`, `UNVERIFIED_REASONS`;
|
||||||
|
`runContract` gains `unverified`, `missing_reviewers` and `allow_blocked_by`
|
||||||
|
(`suppressed` stays the union, so existing consumers keep their meaning).
|
||||||
|
Mirrored in `agents/review-coordinator.md` (Pass 2/3 fate columns, the new
|
||||||
|
§*Suppression is two-valued*, Pass 4 threshold table) and
|
||||||
|
`commands/trekreview.md` (Phase 5 reviewer accounting → STOP; Phase 6).
|
||||||
|
Driven test-first: 9 new tests in `tests/lib/coordinator-contract.test.mjs`,
|
||||||
|
incl. a known-positive control proving `ALLOW` is still reachable.
|
||||||
|
|
||||||
### Docs
|
### Docs
|
||||||
|
|
||||||
- **`/trekresearch --engine deep-research`: document the real version window.**
|
- **`/trekresearch --engine deep-research`: document the real version window.**
|
||||||
|
|
|
||||||
|
|
@ -84,31 +84,48 @@ identical `(file, rule_key)` and `line == 0` collide.
|
||||||
|
|
||||||
### Pass 2 — HubSpot Judge filters (3 criteria)
|
### Pass 2 — HubSpot Judge filters (3 criteria)
|
||||||
|
|
||||||
Drop findings that fail ANY of these filters:
|
Remove findings that fail ANY of these filters. **The `Removed as` column
|
||||||
|
is load-bearing** — see *Suppression is two-valued* below:
|
||||||
|
|
||||||
| Filter | Test | Drop if |
|
| Filter | Test | Fails if | Removed as |
|
||||||
|--------|------|---------|
|
|--------|------|----------|------------|
|
||||||
| Succinctness | `title.length ≤ 100` and `detail.length ≤ 800` chars | Title is a paragraph or detail is a wall of text |
|
| Succinctness | `title.length ≤ 100` and `detail.length ≤ 800` chars | Title is a paragraph or detail is a wall of text | `unverified` (`succinctness:title` / `succinctness:detail`) |
|
||||||
| Accuracy | `file` resolves under the repo root AND `line` is plausible (≥ 0; ≤ file line count when known) | Path traversal escape, negative line, or impossibly large line number |
|
| Accuracy | `file` resolves under the repo root AND `line` is plausible (≥ 0; ≤ file line count when known) | Path traversal escape, negative line, or impossibly large line number | **dropped** (`accuracy:refuted`) |
|
||||||
| Actionability | `recommended_action` is non-empty AND begins with an imperative verb | Empty action, "consider …" hedges, or restating the title |
|
| Actionability | `recommended_action` is non-empty AND begins with an imperative verb | Empty action, "consider …" hedges, or restating the title | `unverified` (`actionability:empty`) |
|
||||||
|
|
||||||
When dropping a finding, preserve a one-line note in the
|
Succinctness and Actionability read the finding's *packaging*; neither
|
||||||
|
examines the claim, so neither can establish the finding is unreal. Accuracy
|
||||||
|
does: a citation that escapes the repo root refutes the finding as a claim
|
||||||
|
about this codebase.
|
||||||
|
|
||||||
|
When removing a finding, preserve a one-line note in the
|
||||||
`Suppressed Findings` body section so the user knows why the count
|
`Suppressed Findings` body section so the user knows why the count
|
||||||
shrank.
|
shrank.
|
||||||
|
|
||||||
### Pass 3 — Cloudflare reasonableness (skipped in quick mode)
|
### Pass 3 — Cloudflare reasonableness (skipped in quick mode)
|
||||||
|
|
||||||
Drop findings that fail ANY of these tests:
|
Remove findings that fail ANY of these tests:
|
||||||
|
|
||||||
- **No file:line citation.** `file` is empty, or `line < 0`. Speculative
|
- **No file:line citation** → **dropped** (`no-citation`). `file` is empty,
|
||||||
"code might break somewhere" findings have no anchor and are dropped.
|
or `line < 0`. Speculative "code might break somewhere" findings name no
|
||||||
- **Unknown rule_key.** `rule_key` is not in `RULE_CATALOGUE`. Reviewers
|
location, so they make no checkable claim at all.
|
||||||
occasionally emit ad-hoc rule keys; the catalogue is the contract.
|
- **Unknown rule_key** → `unverified` (`unknown-rule_key`). `rule_key` is not
|
||||||
|
in `RULE_CATALOGUE`. Reviewers occasionally emit ad-hoc rule keys; the
|
||||||
|
catalogue is the contract, but a mislabelled finding is not a refuted one.
|
||||||
|
*(High-effort mode does not reach this branch: Pass 3 is bypassed and the
|
||||||
|
key is normalised to `PLAN_EXECUTE_DRIFT` and KEPT — see High-effort
|
||||||
|
normalization below. The two fates never apply to the same input.)*
|
||||||
- **Non-existent file.** `file` does not exist in the working tree AND
|
- **Non-existent file.** `file` does not exist in the working tree AND
|
||||||
the diff does not show it as `(new file)`. Use Glob to verify.
|
the diff does not show it as `(new file)`. Use Glob to verify. **This test
|
||||||
|
has three outcomes, not two:** Glob resolves and the file is absent from
|
||||||
|
both tree and diff → **dropped** (`file-existence:refuted`); Glob resolves
|
||||||
|
and the file is present → keep; **Glob cannot decide** (path outside the
|
||||||
|
working tree, unreadable, or the tool errored) → `unverified`
|
||||||
|
(`file-existence:indeterminate`). Never collapse *unresolvable* into
|
||||||
|
*refuted*.
|
||||||
- **Catalogue severity mismatch.** `severity` does not match the rule's
|
- **Catalogue severity mismatch.** `severity` does not match the rule's
|
||||||
catalogue tier (e.g., `MISSING_TEST` emitted as MINOR). Reset to the
|
catalogue tier (e.g., `MISSING_TEST` emitted as MINOR). Reset to the
|
||||||
catalogue tier; this is a correction, not a drop.
|
catalogue tier; this is a correction, neither a drop nor an unverified.
|
||||||
|
|
||||||
In `quick` mode, skip this pass entirely. Note the skip in the
|
In `quick` mode, skip this pass entirely. Note the skip in the
|
||||||
Executive Summary so the reader knows reasonableness was not applied.
|
Executive Summary so the reader knows reasonableness was not applied.
|
||||||
|
|
@ -125,6 +142,32 @@ purposes. This normalization happens BEFORE writing review.md,
|
||||||
ensuring all `rule_key` values in the final review match the
|
ensuring all `rule_key` values in the final review match the
|
||||||
catalogue.
|
catalogue.
|
||||||
|
|
||||||
|
### Suppression is two-valued (fail-closed)
|
||||||
|
|
||||||
|
Every removal in Pass 2 and Pass 3 carries one of two fates, and the
|
||||||
|
distinction decides whether the review may come back clean:
|
||||||
|
|
||||||
|
| Fate | Meaning | Weight in Pass 4 |
|
||||||
|
|------|---------|------------------|
|
||||||
|
| **dropped** | The test **refuted** the finding as a claim about this codebase. | None. It weighs nothing, correctly. |
|
||||||
|
| **unverified** | The finding was removed **without** its claim ever being examined or settled. | Forbids `ALLOW`. |
|
||||||
|
|
||||||
|
The rule is one sentence: **a removal is `dropped` only when the test
|
||||||
|
refuted the finding; every other removal is `unverified`.** A reason you
|
||||||
|
cannot place is `unverified` — the default fails closed.
|
||||||
|
|
||||||
|
Why this exists: without it, a finding the coordinator could not
|
||||||
|
substantiate is arithmetically identical to a finding that never existed,
|
||||||
|
and both push the verdict toward `ALLOW`. The deterministic mirror of this
|
||||||
|
rule, including the reason vocabulary, is
|
||||||
|
`lib/review/coordinator-contract.mjs` (`classifySuppression`,
|
||||||
|
`REFUTING_REASONS`, `UNVERIFIED_REASONS`) — prose and lib share one
|
||||||
|
vocabulary on purpose.
|
||||||
|
|
||||||
|
**Unverified findings are not counted into a severity tier.** Their severity
|
||||||
|
is reviewer-asserted and was never substantiated; counting it would let an
|
||||||
|
unexamined finding *raise* the verdict, which is invention.
|
||||||
|
|
||||||
### Pass 4 — Compute verdict
|
### Pass 4 — Compute verdict
|
||||||
|
|
||||||
Count findings by severity AFTER dedup and filtering. Verdict thresholds:
|
Count findings by severity AFTER dedup and filtering. Verdict thresholds:
|
||||||
|
|
@ -133,7 +176,19 @@ Count findings by severity AFTER dedup and filtering. Verdict thresholds:
|
||||||
|--------|---------|
|
|--------|---------|
|
||||||
| `BLOCKER ≥ 1` | `BLOCK` |
|
| `BLOCKER ≥ 1` | `BLOCK` |
|
||||||
| `BLOCKER == 0` AND `MAJOR ≥ 1` | `WARN` |
|
| `BLOCKER == 0` AND `MAJOR ≥ 1` | `WARN` |
|
||||||
| `BLOCKER == 0` AND `MAJOR == 0` | `ALLOW` |
|
| `BLOCKER == 0` AND `MAJOR == 0` AND nothing `unverified` AND every reviewer reported | `ALLOW` |
|
||||||
|
| `BLOCKER == 0` AND `MAJOR == 0` AND (`unverified` non-empty OR a reviewer did not report) | `WARN` |
|
||||||
|
|
||||||
|
The fail-closed row never RAISES a verdict — it only withholds the clean
|
||||||
|
one. The worst case of a false `unverified` is `WARN` plus a stated reason;
|
||||||
|
the worst case of the old behaviour was a silent `ALLOW` over a live
|
||||||
|
BLOCKER.
|
||||||
|
|
||||||
|
**When `ALLOW` is withheld, the Executive Summary's FIRST sentence must say
|
||||||
|
so and name why** — e.g. "WARN: no blocking findings survived, but 1 finding
|
||||||
|
could not be verified (succinctness:title) and brief-conformance-reviewer did
|
||||||
|
not report." A withheld ALLOW that the reader cannot see is the same defect
|
||||||
|
in a new place.
|
||||||
|
|
||||||
Verdict is mechanical — never override. The verdict goes into the
|
Verdict is mechanical — never override. The verdict goes into the
|
||||||
trailing JSON block AND the Executive Summary's first sentence.
|
trailing JSON block AND the Executive Summary's first sentence.
|
||||||
|
|
@ -181,8 +236,10 @@ prefix). Flow-style `findings: [a, b]` breaks the frontmatter parser.
|
||||||
5. `## Findings (MAJOR)` — one subsection per MAJOR finding.
|
5. `## Findings (MAJOR)` — one subsection per MAJOR finding.
|
||||||
6. `## Findings (MINOR)` — one subsection per MINOR finding.
|
6. `## Findings (MINOR)` — one subsection per MINOR finding.
|
||||||
7. `## Findings (SUGGESTION)` — one subsection per SUGGESTION finding.
|
7. `## Findings (SUGGESTION)` — one subsection per SUGGESTION finding.
|
||||||
8. `## Suppressed Findings` (optional) — one-line per finding dropped by
|
8. `## Suppressed Findings` (optional) — one line per finding removed by
|
||||||
Pass 2 or Pass 3, with the reason.
|
Pass 2 or Pass 3, with the reason AND its fate, tagged `[dropped]` or
|
||||||
|
`[unverified]`. Unverified lines come first: they are the ones that
|
||||||
|
withheld `ALLOW`.
|
||||||
9. `## Remediation Summary` — bullet count per severity + 1 sentence on
|
9. `## Remediation Summary` — bullet count per severity + 1 sentence on
|
||||||
what /trekplan will consume.
|
what /trekplan will consume.
|
||||||
|
|
||||||
|
|
@ -204,6 +261,7 @@ The LAST fenced block in the file is a `json` block:
|
||||||
{
|
{
|
||||||
"verdict": "BLOCK | WARN | ALLOW",
|
"verdict": "BLOCK | WARN | ALLOW",
|
||||||
"counts": { "BLOCKER": N, "MAJOR": N, "MINOR": N, "SUGGESTION": N },
|
"counts": { "BLOCKER": N, "MAJOR": N, "MINOR": N, "SUGGESTION": N },
|
||||||
|
"allow_blocked_by": ["unverified:succinctness:title (1)", "missing-reviewer:brief-conformance-reviewer"],
|
||||||
"findings": [
|
"findings": [
|
||||||
{
|
{
|
||||||
"id": "<40-char-hex>",
|
"id": "<40-char-hex>",
|
||||||
|
|
@ -243,9 +301,14 @@ for the ID list.
|
||||||
the canonical 40-char SHA1 from `(file, line, rule_key, title)` using
|
the canonical 40-char SHA1 from `(file, line, rule_key, title)` using
|
||||||
the algorithm in `lib/parsers/finding-id.mjs`. The frontmatter
|
the algorithm in `lib/parsers/finding-id.mjs`. The frontmatter
|
||||||
`findings:` list and the JSON block IDs must match.
|
`findings:` list and the JSON block IDs must match.
|
||||||
- **Suppressed findings are accountable.** When you drop a finding via
|
- **Suppressed findings are accountable.** When you remove a finding via
|
||||||
Pass 2 or Pass 3, log it in `## Suppressed Findings` with the reason.
|
Pass 2 or Pass 3, log it in `## Suppressed Findings` with the reason and
|
||||||
Silent drops break the audit trail.
|
its fate (`[dropped]` / `[unverified]`). Silent drops break the audit
|
||||||
|
trail.
|
||||||
|
- **Never spend an unexamined finding as evidence of a clean review.** If a
|
||||||
|
removal did not refute the finding, it is `unverified` and `ALLOW` is off
|
||||||
|
the table. This is the one place where you may not be minimal: when in
|
||||||
|
doubt about a reason's fate, it is `unverified`.
|
||||||
- **No invention.** Never add a finding that did not appear in the
|
- **No invention.** Never add a finding that did not appear in the
|
||||||
reviewer outputs. Never escalate a finding's severity beyond what the
|
reviewer outputs. Never escalate a finding's severity beyond what the
|
||||||
catalogue specifies.
|
catalogue specifies.
|
||||||
|
|
|
||||||
|
|
@ -249,6 +249,37 @@ do not feed unvalidated findings to the coordinator.
|
||||||
In `quick` mode, launch only `code-correctness-reviewer`. The Executive
|
In `quick` mode, launch only `code-correctness-reviewer`. The Executive
|
||||||
Summary will note the brief-conformance pass was skipped.
|
Summary will note the brief-conformance pass was skipped.
|
||||||
|
|
||||||
|
### Reviewer accounting — every expected reviewer MUST report
|
||||||
|
|
||||||
|
Write down the expected reviewer set BEFORE the spawn: both reviewers in
|
||||||
|
default mode, `code-correctness-reviewer` alone in `quick` mode. After the
|
||||||
|
spawn, account for each one by name.
|
||||||
|
|
||||||
|
**Zero findings from a silent reviewer is indistinguishable from zero findings
|
||||||
|
from a clean diff** — unless you check. A reviewer is *accounted for* only when
|
||||||
|
it returned a payload that validated. Three ways it fails to:
|
||||||
|
|
||||||
|
| Failure | Handling |
|
||||||
|
|---------|----------|
|
||||||
|
| Output fails the schema after the 2 bounded re-asks | STOP (already specified above) |
|
||||||
|
| Returned no final message at all | Re-ask that reviewer **once**. Still nothing → STOP. |
|
||||||
|
| Was never launched (spawn error, wrong mode) | STOP. |
|
||||||
|
|
||||||
|
**On STOP: name the reviewer and the failure, and do not proceed to Phase 6.**
|
||||||
|
Do not let the coordinator compute a verdict over a review one of whose
|
||||||
|
reviewers never spoke — the count would be complete-looking and wrong. This is
|
||||||
|
the same shape as the schema branch above ("do not feed unvalidated findings to
|
||||||
|
the coordinator"), applied to the other two ways a reviewer can go missing.
|
||||||
|
|
||||||
|
A reviewer that ran but never delivered is most often the return-channel
|
||||||
|
defect: check `~/.claude/projects/<proj>/<session>/subagents/agent-*.jsonl` for
|
||||||
|
its final assistant block before re-asking, and confirm no `name` parameter was
|
||||||
|
passed at the spawn (see the warning at the top of this phase).
|
||||||
|
|
||||||
|
If you proceed anyway under an explicit operator instruction, pass the expected
|
||||||
|
set to the coordinator as `expectedReviewers` so the missing reviewer at least
|
||||||
|
forbids `ALLOW` (`lib/review/coordinator-contract.mjs`, `missing_reviewers`).
|
||||||
|
|
||||||
## Phase 6 — Coordinator dedup + verdict
|
## Phase 6 — Coordinator dedup + verdict
|
||||||
|
|
||||||
Launch `review-coordinator` (Agent tool) with the merged findings array
|
Launch `review-coordinator` (Agent tool) with the merged findings array
|
||||||
|
|
@ -259,10 +290,20 @@ The coordinator runs the 4-pass process documented in
|
||||||
|
|
||||||
1. **Dedup** by `(file, line, rule_key)` triplet.
|
1. **Dedup** by `(file, line, rule_key)` triplet.
|
||||||
2. **HubSpot Judge filters** — Succinctness, Accuracy, Actionability.
|
2. **HubSpot Judge filters** — Succinctness, Accuracy, Actionability.
|
||||||
3. **Cloudflare reasonableness** — drop speculative or catalogue-violating
|
3. **Cloudflare reasonableness** — remove speculative or catalogue-violating
|
||||||
findings (skipped in `quick` mode).
|
findings (skipped in `quick` mode).
|
||||||
4. **Verdict** — BLOCK / WARN / ALLOW per the threshold table.
|
4. **Verdict** — BLOCK / WARN / ALLOW per the threshold table.
|
||||||
|
|
||||||
|
**Fail-closed.** Every removal in Pass 2 and Pass 3 is either
|
||||||
|
**dropped** (the test refuted the finding as a claim about this codebase) or
|
||||||
|
**unverified** (the finding was removed without its claim ever being settled).
|
||||||
|
A non-empty `unverified` bucket forbids `ALLOW`; the verdict becomes `WARN` and
|
||||||
|
the Executive Summary's first sentence must say why. The fail-closed rule never
|
||||||
|
raises a verdict — it only withholds the clean one. Fate table, reason
|
||||||
|
vocabulary, and the `allow_blocked_by` field: `agents/review-coordinator.md`
|
||||||
|
§*Suppression is two-valued*, mirrored deterministically in
|
||||||
|
`lib/review/coordinator-contract.mjs`.
|
||||||
|
|
||||||
The coordinator's output is the full review.md content — frontmatter +
|
The coordinator's output is the full review.md content — frontmatter +
|
||||||
body sections + trailing JSON block. Do NOT re-run the reviewers based
|
body sections + trailing JSON block. Do NOT re-run the reviewers based
|
||||||
on the coordinator's output.
|
on the coordinator's output.
|
||||||
|
|
|
||||||
|
|
@ -167,10 +167,18 @@ arm produces correct final text at 66 lines and at 3730 lines alike, and only
|
||||||
delivery fails, identically at both. The 38 recovered findings were re-used
|
delivery fails, identically at both. The 38 recovered findings were re-used
|
||||||
instead of re-run.
|
instead of re-run.
|
||||||
|
|
||||||
What it does **not** close: their four cells were all named, so the *returning*
|
**A second external measurement closes the other half (akashic-intelligence
|
||||||
(unnamed) arm still has no measurement above 66 lines. That a plain subagent
|
S27, commit `f168630`).** The gap left above was that all four S25 cells were
|
||||||
returns a 21 KB result at that scale is inferred, not observed. Stated as
|
named, so the *returning* (unnamed) arm had no measurement above 66 lines.
|
||||||
inferred.
|
S27 supplies one: denominator **2 of 2 unnamed agents**, against a 4370-line
|
||||||
|
plan; both returned, 30449 B and 10989 B, both valid JSON. The unnamed arm
|
||||||
|
therefore returns at full scale as observed fact, not as inference.
|
||||||
|
|
||||||
|
Two caveats, kept at the strength akashic itself stated them. The measurement
|
||||||
|
was taken by the repo that owns the finding, not by an independent third party.
|
||||||
|
And byte-identity between the returned string and the file on disk was not
|
||||||
|
proven — what is established is that a well-formed result of that size arrived,
|
||||||
|
not that it arrived unaltered.
|
||||||
|
|
||||||
Their PONG control also carries the same lesson as cell 1 above, in a third
|
Their PONG control also carries the same lesson as cell 1 above, in a third
|
||||||
repo: S25 reported that agent as having "gone idle without sending PONG". It
|
repo: S25 reported that agent as having "gone idle without sending PONG". It
|
||||||
|
|
|
||||||
|
|
@ -19,8 +19,10 @@
|
||||||
// What IS implemented, purely: Pass 1 (triplet dedup → highest-severity-wins
|
// What IS implemented, purely: Pass 1 (triplet dedup → highest-severity-wins
|
||||||
// survivor + conformance tiebreak + detail concat + raised_by provenance),
|
// survivor + conformance tiebreak + detail concat + raised_by provenance),
|
||||||
// Pass 2 succinctness + actionability-presence, Pass 3 reasonableness
|
// Pass 2 succinctness + actionability-presence, Pass 3 reasonableness
|
||||||
// (citation / unknown-rule_key drop, severity-mismatch correction), Pass 4
|
// (citation / unknown-rule_key suppression, severity-mismatch correction),
|
||||||
// verdict thresholds. No LLM, no network, no time, no randomness.
|
// Pass 4 verdict thresholds — fail-closed: a suppression that did not REFUTE
|
||||||
|
// the finding, and a reviewer that never reported, forbid ALLOW (see
|
||||||
|
// classifySuppression). No LLM, no network, no time, no randomness.
|
||||||
//
|
//
|
||||||
// Reuses: SEVERITY_VALUES / RULE_KEYS / getRule (rule-catalogue.mjs),
|
// Reuses: SEVERITY_VALUES / RULE_KEYS / getRule (rule-catalogue.mjs),
|
||||||
// computeFindingId (finding-id.mjs, triplet), validateFindings
|
// computeFindingId (finding-id.mjs, triplet), validateFindings
|
||||||
|
|
@ -35,6 +37,71 @@ import { validateFindings } from './findings-schema.mjs';
|
||||||
export const JUDGE_TITLE_MAX = 100;
|
export const JUDGE_TITLE_MAX = 100;
|
||||||
export const JUDGE_DETAIL_MAX = 800;
|
export const JUDGE_DETAIL_MAX = 800;
|
||||||
|
|
||||||
|
// ---- Suppression classification (fail-closed) --------------------------------
|
||||||
|
//
|
||||||
|
// A removal is `dropped` ONLY when the test refuted the finding as a claim
|
||||||
|
// about this codebase. Every other removal is `unverified`: the coordinator
|
||||||
|
// took the finding out of the count without ever establishing it was unreal,
|
||||||
|
// so it may not be spent as evidence of a clean review.
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Reasons that REFUTE. `no-citation` is the only one this deterministic subset
|
||||||
|
* can emit: a finding whose `file` is empty or whose `line` is negative names
|
||||||
|
* no location, so it makes no checkable claim at all
|
||||||
|
* (agents/review-coordinator.md Pass 3 — "Speculative 'code might break
|
||||||
|
* somewhere' findings have no anchor").
|
||||||
|
*
|
||||||
|
* `accuracy:refuted` (Pass 2 Accuracy — a citation escaping the repo root) and
|
||||||
|
* `file-existence:refuted` (Pass 3 — absent from both working tree and diff)
|
||||||
|
* are emitted by the LLM coordinator, whose fs/judgement branches this module
|
||||||
|
* excludes. They are declared here anyway: the vocabulary is owned in one
|
||||||
|
* place so prose and lib cannot drift.
|
||||||
|
*/
|
||||||
|
export const REFUTING_REASONS = Object.freeze(new Set([
|
||||||
|
'no-citation',
|
||||||
|
'accuracy:refuted',
|
||||||
|
'file-existence:refuted',
|
||||||
|
]));
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The reason vocabulary on the unverified side. `file-existence:indeterminate`
|
||||||
|
* is emitted by the LLM coordinator's Pass 3 (which runs the fs Glob this
|
||||||
|
* module deliberately excludes); the vocabulary is owned here so prose and lib
|
||||||
|
* cannot drift.
|
||||||
|
*/
|
||||||
|
export const UNVERIFIED_REASONS = Object.freeze([
|
||||||
|
'succinctness:title',
|
||||||
|
'succinctness:detail',
|
||||||
|
'actionability:empty',
|
||||||
|
'unknown-rule_key',
|
||||||
|
'file-existence:indeterminate',
|
||||||
|
]);
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Classify a suppression reason. Anything not declared refuting is
|
||||||
|
* `unverified` — the default is fail-CLOSED, so a reason introduced later
|
||||||
|
* without a decision cannot silently move the verdict toward ALLOW.
|
||||||
|
* @param {string} reason
|
||||||
|
* @returns {'refuted'|'unverified'}
|
||||||
|
*/
|
||||||
|
export function classifySuppression(reason) {
|
||||||
|
return REFUTING_REASONS.has(reason) ? 'refuted' : 'unverified';
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Tag a finding with its suppression reason and route it to the refuted
|
||||||
|
* (`dropped`) or the `unverified` bucket.
|
||||||
|
* @param {object} finding
|
||||||
|
* @param {string} reason
|
||||||
|
* @param {object[]} dropped
|
||||||
|
* @param {object[]} unverified
|
||||||
|
*/
|
||||||
|
function suppress(finding, reason, dropped, unverified) {
|
||||||
|
const tagged = { ...finding, suppressed_reason: reason };
|
||||||
|
if (classifySuppression(reason) === 'refuted') dropped.push(tagged);
|
||||||
|
else unverified.push(tagged);
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Catalogue-tier rank of a severity: lower number = higher severity.
|
* Catalogue-tier rank of a severity: lower number = higher severity.
|
||||||
* BLOCKER=0 … SUGGESTION=3; an unknown severity ranks last.
|
* BLOCKER=0 … SUGGESTION=3; an unknown severity ranks last.
|
||||||
|
|
@ -122,12 +189,18 @@ export function dedupByTriplet(findings) {
|
||||||
* (title > 100 or detail > 800 chars) and actionability (recommended_action,
|
* (title > 100 or detail > 800 chars) and actionability (recommended_action,
|
||||||
* when present, must be a non-empty string). The imperative-verb test is
|
* when present, must be a non-empty string). The imperative-verb test is
|
||||||
* excluded (LLM judgement).
|
* excluded (LLM judgement).
|
||||||
|
*
|
||||||
|
* Both tests read a `.length`; neither examines the claim, so neither can
|
||||||
|
* establish the finding is unreal. Both therefore route to `unverified`.
|
||||||
|
* `dropped` stays in the signature for the refuting Pass-2 filter this subset
|
||||||
|
* excludes (Accuracy: a path-traversal escape IS a refutation).
|
||||||
* @param {object[]} findings
|
* @param {object[]} findings
|
||||||
* @returns {{ kept: object[], dropped: object[] }}
|
* @returns {{ kept: object[], dropped: object[], unverified: object[] }}
|
||||||
*/
|
*/
|
||||||
export function judgeFilter(findings) {
|
export function judgeFilter(findings) {
|
||||||
const kept = [];
|
const kept = [];
|
||||||
const dropped = [];
|
const dropped = [];
|
||||||
|
const unverified = [];
|
||||||
for (const f of findings) {
|
for (const f of findings) {
|
||||||
const titleLen = (f.title ?? '').length;
|
const titleLen = (f.title ?? '').length;
|
||||||
const detailLen = (f.detail ?? '').length;
|
const detailLen = (f.detail ?? '').length;
|
||||||
|
|
@ -138,31 +211,38 @@ export function judgeFilter(findings) {
|
||||||
(typeof f.recommended_action !== 'string' || f.recommended_action.trim().length === 0)) {
|
(typeof f.recommended_action !== 'string' || f.recommended_action.trim().length === 0)) {
|
||||||
reason = 'actionability:empty';
|
reason = 'actionability:empty';
|
||||||
}
|
}
|
||||||
if (reason) dropped.push({ ...f, suppressed_reason: reason });
|
if (reason) suppress(f, reason, dropped, unverified);
|
||||||
else kept.push(f);
|
else kept.push(f);
|
||||||
}
|
}
|
||||||
return { kept, dropped };
|
return { kept, dropped, unverified };
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Pass 3 — Cloudflare reasonableness (deterministic subset): drop findings
|
* Pass 3 — Cloudflare reasonableness (deterministic subset): drop findings
|
||||||
* with no citation (empty file / line < 0) or an unknown rule_key; CORRECT a
|
* with no citation (empty file / line < 0) or an unknown rule_key; CORRECT a
|
||||||
* severity that does not match the catalogue tier (a correction, not a drop).
|
* severity that does not match the catalogue tier (a correction, not a drop).
|
||||||
* The fs file-existence glob is excluded (I/O).
|
* The fs file-existence glob is excluded (I/O) — its indeterminate branch is
|
||||||
|
* prose-side, tokenised as `file-existence:indeterminate`.
|
||||||
|
*
|
||||||
|
* `no-citation` REFUTES (the finding names no location, so it makes no
|
||||||
|
* checkable claim) and is dropped. `unknown-rule_key` does not: an ad-hoc key
|
||||||
|
* is a real defect wearing the wrong label — v5.1.1 high-effort mode already
|
||||||
|
* KEEPS these, normalised to PLAN_EXECUTE_DRIFT — so it routes to `unverified`.
|
||||||
* @param {object[]} findings
|
* @param {object[]} findings
|
||||||
* @returns {{ kept: object[], dropped: object[] }}
|
* @returns {{ kept: object[], dropped: object[], unverified: object[] }}
|
||||||
*/
|
*/
|
||||||
export function reasonablenessFilter(findings) {
|
export function reasonablenessFilter(findings) {
|
||||||
const kept = [];
|
const kept = [];
|
||||||
const dropped = [];
|
const dropped = [];
|
||||||
|
const unverified = [];
|
||||||
for (const f of findings) {
|
for (const f of findings) {
|
||||||
if (typeof f.file !== 'string' || f.file.length === 0 ||
|
if (typeof f.file !== 'string' || f.file.length === 0 ||
|
||||||
(typeof f.line === 'number' && f.line < 0)) {
|
(typeof f.line === 'number' && f.line < 0)) {
|
||||||
dropped.push({ ...f, suppressed_reason: 'no-citation' });
|
suppress(f, 'no-citation', dropped, unverified);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
if (!RULE_KEYS.has(f.rule_key)) {
|
if (!RULE_KEYS.has(f.rule_key)) {
|
||||||
dropped.push({ ...f, suppressed_reason: 'unknown-rule_key' });
|
suppress(f, 'unknown-rule_key', dropped, unverified);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
const rule = getRule(f.rule_key);
|
const rule = getRule(f.rule_key);
|
||||||
|
|
@ -172,44 +252,101 @@ export function reasonablenessFilter(findings) {
|
||||||
kept.push(f);
|
kept.push(f);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return { kept, dropped };
|
return { kept, dropped, unverified };
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Pass 4 — compute the verdict from severity counts (after dedup + filtering).
|
* Pass 4 — compute the verdict from severity counts (after dedup + filtering).
|
||||||
* BLOCKER ≥ 1 → BLOCK; else MAJOR ≥ 1 → WARN; else ALLOW.
|
* BLOCKER ≥ 1 → BLOCK; else MAJOR ≥ 1 → WARN; else ALLOW.
|
||||||
|
*
|
||||||
|
* FAIL-CLOSED: ALLOW additionally requires that nothing is `unverified` and
|
||||||
|
* that every expected reviewer reported. Neither ever RAISES a verdict — the
|
||||||
|
* severity thresholds are untouched — they only forbid the clean one, so the
|
||||||
|
* worst case of a false unverified is WARN plus a stated reason, never a
|
||||||
|
* silent pass. Unverified findings are NOT counted into a severity tier: their
|
||||||
|
* severity is reviewer-asserted and was never substantiated.
|
||||||
|
*
|
||||||
* @param {object[]} findings
|
* @param {object[]} findings
|
||||||
* @returns {{ verdict: 'BLOCK'|'WARN'|'ALLOW', counts: Record<string, number> }}
|
* @param {{ unverified?: object[], missingReviewers?: string[] }} [options]
|
||||||
|
* @returns {{ verdict: 'BLOCK'|'WARN'|'ALLOW', counts: Record<string, number>, allow_blocked_by: string[] }}
|
||||||
*/
|
*/
|
||||||
export function computeVerdict(findings) {
|
export function computeVerdict(findings, options = {}) {
|
||||||
const counts = { BLOCKER: 0, MAJOR: 0, MINOR: 0, SUGGESTION: 0 };
|
const counts = { BLOCKER: 0, MAJOR: 0, MINOR: 0, SUGGESTION: 0 };
|
||||||
for (const f of findings) {
|
for (const f of findings) {
|
||||||
if (counts[f.severity] !== undefined) counts[f.severity] += 1;
|
if (counts[f.severity] !== undefined) counts[f.severity] += 1;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const unverified = options.unverified ?? [];
|
||||||
|
const missingReviewers = options.missingReviewers ?? [];
|
||||||
|
const allow_blocked_by = [];
|
||||||
|
const byReason = new Map();
|
||||||
|
for (const f of unverified) {
|
||||||
|
const reason = f?.suppressed_reason ?? 'unspecified';
|
||||||
|
byReason.set(reason, (byReason.get(reason) ?? 0) + 1);
|
||||||
|
}
|
||||||
|
for (const [reason, n] of byReason) allow_blocked_by.push(`unverified:${reason} (${n})`);
|
||||||
|
for (const r of missingReviewers) allow_blocked_by.push(`missing-reviewer:${r}`);
|
||||||
|
|
||||||
let verdict;
|
let verdict;
|
||||||
if (counts.BLOCKER >= 1) verdict = 'BLOCK';
|
if (counts.BLOCKER >= 1) verdict = 'BLOCK';
|
||||||
else if (counts.MAJOR >= 1) verdict = 'WARN';
|
else if (counts.MAJOR >= 1) verdict = 'WARN';
|
||||||
|
else if (allow_blocked_by.length > 0) verdict = 'WARN';
|
||||||
else verdict = 'ALLOW';
|
else verdict = 'ALLOW';
|
||||||
return { verdict, counts };
|
return { verdict, counts, allow_blocked_by };
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Run the full deterministic contract: ingest → Pass 1 → Pass 2 → Pass 3 → Pass 4.
|
* Run the full deterministic contract: ingest → Pass 1 → Pass 2 → Pass 3 → Pass 4.
|
||||||
|
*
|
||||||
|
* `options.expectedReviewers` names the reviewers this review was supposed to
|
||||||
|
* hear from. A reviewer that is absent from the payloads, or whose payload
|
||||||
|
* failed schema validation and was thrown away at ingest, lands in
|
||||||
|
* `missing_reviewers` and forbids ALLOW: an unread reviewer is an absent one,
|
||||||
|
* and zero findings from a silent reviewer must not read like zero findings
|
||||||
|
* from a clean diff.
|
||||||
|
*
|
||||||
|
* `suppressed` stays the UNION of `dropped` (refuted) and `unverified` so
|
||||||
|
* existing consumers keep their meaning; `unverified` is the subset that
|
||||||
|
* forbids ALLOW. Do not iterate both and count twice.
|
||||||
|
*
|
||||||
* @param {Array<{reviewer?: string, findings: object[]}>} reviewerPayloads
|
* @param {Array<{reviewer?: string, findings: object[]}>} reviewerPayloads
|
||||||
* @returns {{ verdict: string, counts: Record<string, number>, findings: object[], suppressed: object[], skipped: object[] }}
|
* @param {{ expectedReviewers?: string[] }} [options]
|
||||||
|
* @returns {{ verdict: string, counts: Record<string, number>, findings: object[], suppressed: object[], unverified: object[], skipped: object[], missing_reviewers: string[], allow_blocked_by: string[] }}
|
||||||
*/
|
*/
|
||||||
export function runContract(reviewerPayloads) {
|
export function runContract(reviewerPayloads, options = {}) {
|
||||||
const { findings: ingested, skipped } = ingest(reviewerPayloads);
|
const { findings: ingested, skipped } = ingest(reviewerPayloads);
|
||||||
const deduped = dedupByTriplet(ingested);
|
const deduped = dedupByTriplet(ingested);
|
||||||
const judged = judgeFilter(deduped);
|
const judged = judgeFilter(deduped);
|
||||||
const reasoned = reasonablenessFilter(judged.kept);
|
const reasoned = reasonablenessFilter(judged.kept);
|
||||||
const { verdict, counts } = computeVerdict(reasoned.kept);
|
const unverified = [...judged.unverified, ...reasoned.unverified];
|
||||||
|
|
||||||
|
const reported = new Set();
|
||||||
|
for (const payload of reviewerPayloads) {
|
||||||
|
if (typeof payload?.reviewer === 'string' && payload.reviewer.length > 0) reported.add(payload.reviewer);
|
||||||
|
}
|
||||||
|
for (const s of skipped) reported.delete(s.reviewer);
|
||||||
|
const missing_reviewers = [];
|
||||||
|
for (const s of skipped) {
|
||||||
|
const name = s.reviewer ?? 'unnamed reviewer';
|
||||||
|
if (!missing_reviewers.includes(name)) missing_reviewers.push(name);
|
||||||
|
}
|
||||||
|
for (const r of options.expectedReviewers ?? []) {
|
||||||
|
if (!reported.has(r) && !missing_reviewers.includes(r)) missing_reviewers.push(r);
|
||||||
|
}
|
||||||
|
|
||||||
|
const { verdict, counts, allow_blocked_by } = computeVerdict(reasoned.kept, {
|
||||||
|
unverified,
|
||||||
|
missingReviewers: missing_reviewers,
|
||||||
|
});
|
||||||
return {
|
return {
|
||||||
verdict,
|
verdict,
|
||||||
counts,
|
counts,
|
||||||
findings: reasoned.kept,
|
findings: reasoned.kept,
|
||||||
suppressed: [...judged.dropped, ...reasoned.dropped],
|
suppressed: [...judged.dropped, ...judged.unverified, ...reasoned.dropped, ...reasoned.unverified],
|
||||||
|
unverified,
|
||||||
skipped,
|
skipped,
|
||||||
|
missing_reviewers,
|
||||||
|
allow_blocked_by,
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -6,6 +6,9 @@
|
||||||
|
|
||||||
import { test } from 'node:test';
|
import { test } from 'node:test';
|
||||||
import { strict as assert } from 'node:assert';
|
import { strict as assert } from 'node:assert';
|
||||||
|
import { readFileSync } from 'node:fs';
|
||||||
|
import { join, dirname } from 'node:path';
|
||||||
|
import { fileURLToPath } from 'node:url';
|
||||||
import {
|
import {
|
||||||
severityRank,
|
severityRank,
|
||||||
ingest,
|
ingest,
|
||||||
|
|
@ -13,9 +16,14 @@ import {
|
||||||
judgeFilter,
|
judgeFilter,
|
||||||
reasonablenessFilter,
|
reasonablenessFilter,
|
||||||
computeVerdict,
|
computeVerdict,
|
||||||
|
classifySuppression,
|
||||||
|
REFUTING_REASONS,
|
||||||
|
UNVERIFIED_REASONS,
|
||||||
runContract,
|
runContract,
|
||||||
} from '../../lib/review/coordinator-contract.mjs';
|
} from '../../lib/review/coordinator-contract.mjs';
|
||||||
|
|
||||||
|
const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..', '..');
|
||||||
|
|
||||||
// ---- Pass 1 — dedup --------------------------------------------------------
|
// ---- Pass 1 — dedup --------------------------------------------------------
|
||||||
|
|
||||||
test('dedupByTriplet — genuine cross-reviewer collapse (identical triplet) → 1, raised_by both', () => {
|
test('dedupByTriplet — genuine cross-reviewer collapse (identical triplet) → 1, raised_by both', () => {
|
||||||
|
|
@ -98,29 +106,41 @@ test('computeVerdict — counts each severity tier', () => {
|
||||||
|
|
||||||
// ---- Pass 3 — reasonableness -----------------------------------------------
|
// ---- Pass 3 — reasonableness -----------------------------------------------
|
||||||
|
|
||||||
test('reasonablenessFilter — drops unknown rule_key + citation-less, corrects severity mismatch', () => {
|
test('reasonablenessFilter — citation-less is REFUTED, unknown rule_key is UNVERIFIED, severity mismatch corrected', () => {
|
||||||
|
// Contract change (fail-closed): only `no-citation` refutes. An ad-hoc
|
||||||
|
// rule_key is a real defect wearing the wrong label — v5.1.1 high-effort mode
|
||||||
|
// already keeps those, normalised to PLAN_EXECUTE_DRIFT.
|
||||||
const r = reasonablenessFilter([
|
const r = reasonablenessFilter([
|
||||||
{ file: 'x.mjs', line: 1, rule_key: 'NOPE_KEY', severity: 'BLOCKER' }, // unknown → drop
|
{ file: 'x.mjs', line: 1, rule_key: 'NOPE_KEY', severity: 'BLOCKER' }, // unknown → unverified
|
||||||
{ file: '', line: 1, rule_key: 'MISSING_TEST', severity: 'MAJOR' }, // no file → drop
|
{ file: '', line: 1, rule_key: 'MISSING_TEST', severity: 'MAJOR' }, // no file → drop
|
||||||
{ file: 'x.mjs', line: -1, rule_key: 'MISSING_TEST', severity: 'MAJOR' }, // line < 0 → drop
|
{ file: 'x.mjs', line: -1, rule_key: 'MISSING_TEST', severity: 'MAJOR' }, // line < 0 → drop
|
||||||
{ file: 'x.mjs', line: 1, rule_key: 'MISSING_TEST', severity: 'MINOR' }, // catalogue is MAJOR → correct, keep
|
{ file: 'x.mjs', line: 1, rule_key: 'MISSING_TEST', severity: 'MINOR' }, // catalogue is MAJOR → correct, keep
|
||||||
]);
|
]);
|
||||||
assert.equal(r.kept.length, 1);
|
assert.equal(r.kept.length, 1);
|
||||||
assert.equal(r.dropped.length, 3);
|
assert.equal(r.dropped.length, 2);
|
||||||
|
assert.deepEqual(r.dropped.map((f) => f.suppressed_reason), ['no-citation', 'no-citation']);
|
||||||
|
assert.equal(r.unverified.length, 1);
|
||||||
|
assert.equal(r.unverified[0].suppressed_reason, 'unknown-rule_key');
|
||||||
assert.equal(r.kept[0].severity, 'MAJOR');
|
assert.equal(r.kept[0].severity, 'MAJOR');
|
||||||
assert.equal(r.kept[0].original_severity, 'MINOR');
|
assert.equal(r.kept[0].original_severity, 'MINOR');
|
||||||
});
|
});
|
||||||
|
|
||||||
// ---- Pass 2 — judge --------------------------------------------------------
|
// ---- Pass 2 — judge --------------------------------------------------------
|
||||||
|
|
||||||
test('judgeFilter — drops over-long title and empty recommended_action', () => {
|
test('judgeFilter — over-long title and empty recommended_action are UNVERIFIED, not dropped', () => {
|
||||||
|
// Contract change (fail-closed): both implemented Pass 2 tests read a
|
||||||
|
// `.length` and never examine the claim, so neither refutes the finding.
|
||||||
|
// `dropped` is empty here on purpose — the refuting Pass 2 filter (Accuracy)
|
||||||
|
// is the one this deterministic subset excludes.
|
||||||
const j = judgeFilter([
|
const j = judgeFilter([
|
||||||
{ file: 'x.mjs', line: 1, rule_key: 'MISSING_TEST', severity: 'MAJOR', title: 'x'.repeat(101) }, // too long → drop
|
{ file: 'x.mjs', line: 1, rule_key: 'MISSING_TEST', severity: 'MAJOR', title: 'x'.repeat(101) }, // too long → unverified
|
||||||
{ file: 'x.mjs', line: 2, rule_key: 'MISSING_TEST', severity: 'MAJOR', title: 'ok', recommended_action: ' ' }, // empty action → drop
|
{ file: 'x.mjs', line: 2, rule_key: 'MISSING_TEST', severity: 'MAJOR', title: 'ok', recommended_action: ' ' }, // empty action → unverified
|
||||||
{ file: 'x.mjs', line: 3, rule_key: 'MISSING_TEST', severity: 'MAJOR', title: 'ok' }, // keep (no action field is fine)
|
{ file: 'x.mjs', line: 3, rule_key: 'MISSING_TEST', severity: 'MAJOR', title: 'ok' }, // keep (no action field is fine)
|
||||||
]);
|
]);
|
||||||
assert.equal(j.kept.length, 1);
|
assert.equal(j.kept.length, 1);
|
||||||
assert.equal(j.dropped.length, 2);
|
assert.equal(j.dropped.length, 0);
|
||||||
|
assert.equal(j.unverified.length, 2);
|
||||||
|
assert.deepEqual(j.unverified.map((f) => f.suppressed_reason), ['succinctness:title', 'actionability:empty']);
|
||||||
});
|
});
|
||||||
|
|
||||||
// ---- ingest ----------------------------------------------------------------
|
// ---- ingest ----------------------------------------------------------------
|
||||||
|
|
@ -161,3 +181,130 @@ test('runContract — deterministic: identical input yields identical output', (
|
||||||
];
|
];
|
||||||
assert.deepEqual(runContract(input), runContract(input));
|
assert.deepEqual(runContract(input), runContract(input));
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// ---- Fail-closed: the `unverified` bucket (ORDRE 834432937) -----------------
|
||||||
|
//
|
||||||
|
// The defect: a finding REMOVED by Pass 2/Pass 3, and a reviewer whose payload
|
||||||
|
// was thrown away or never arrived, are all arithmetically identical to a
|
||||||
|
// finding that never existed -- they push the verdict toward ALLOW. Measured
|
||||||
|
// before the fix (probe, 2026-09-01): an over-long-title BLOCKER -> ALLOW; a
|
||||||
|
// payload with one ad-hoc rule_key -> the whole payload skipped, its valid
|
||||||
|
// BLOCKER sibling gone -> ALLOW.
|
||||||
|
//
|
||||||
|
// The rule under test: a removal is `dropped` ONLY when the test refutes the
|
||||||
|
// finding as a claim about this codebase. Every other removal is `unverified`,
|
||||||
|
// and a non-empty `unverified` -- or a reviewer that did not report -- forbids
|
||||||
|
// ALLOW.
|
||||||
|
|
||||||
|
test('classifySuppression — only no-citation refutes; form and taxonomy failures are unverified', () => {
|
||||||
|
assert.equal(classifySuppression('no-citation'), 'refuted',
|
||||||
|
'a finding that names no location makes no checkable claim');
|
||||||
|
assert.equal(classifySuppression('succinctness:title'), 'unverified');
|
||||||
|
assert.equal(classifySuppression('succinctness:detail'), 'unverified');
|
||||||
|
assert.equal(classifySuppression('actionability:empty'), 'unverified');
|
||||||
|
assert.equal(classifySuppression('unknown-rule_key'), 'unverified');
|
||||||
|
assert.equal(classifySuppression('file-existence:indeterminate'), 'unverified');
|
||||||
|
assert.equal(classifySuppression('something-nobody-declared'), 'unverified',
|
||||||
|
'an unclassified reason must fail CLOSED, not open');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('computeVerdict — non-empty unverified forbids ALLOW but never downgrades BLOCK or WARN', () => {
|
||||||
|
const u = [{ file: 'x.mjs', line: 1, rule_key: 'MISSING_TEST', severity: 'BLOCKER' }];
|
||||||
|
const withUnverified = computeVerdict([], { unverified: u });
|
||||||
|
assert.equal(withUnverified.verdict, 'WARN', 'ALLOW is forbidden while anything is unverified');
|
||||||
|
assert.deepEqual(withUnverified.counts, { BLOCKER: 0, MAJOR: 0, MINOR: 0, SUGGESTION: 0 },
|
||||||
|
'the unverified finding is NOT counted into a severity tier');
|
||||||
|
assert.ok(withUnverified.allow_blocked_by.length > 0);
|
||||||
|
|
||||||
|
assert.equal(computeVerdict([{ severity: 'BLOCKER' }], { unverified: u }).verdict, 'BLOCK',
|
||||||
|
'BLOCK stands regardless of the unverified bucket');
|
||||||
|
assert.equal(computeVerdict([{ severity: 'MAJOR' }], { unverified: u }).verdict, 'WARN');
|
||||||
|
assert.equal(computeVerdict([], { unverified: [] }).verdict, 'ALLOW',
|
||||||
|
'known-positive control: an empty unverified bucket still allows ALLOW');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('computeVerdict — a reviewer that did not report forbids ALLOW', () => {
|
||||||
|
const r = computeVerdict([], { missingReviewers: ['brief-conformance-reviewer'] });
|
||||||
|
assert.equal(r.verdict, 'WARN');
|
||||||
|
assert.ok(r.allow_blocked_by.some((x) => x.includes('brief-conformance-reviewer')));
|
||||||
|
});
|
||||||
|
|
||||||
|
test('runContract — a BLOCKER dropped for an over-long title cannot yield ALLOW', () => {
|
||||||
|
// Pass 2 succinctness reads `.length`. It never examines the claim, so it
|
||||||
|
// cannot establish the finding is unreal -- it is unverified, not refuted.
|
||||||
|
const result = runContract([
|
||||||
|
{ reviewer: 'code-correctness-reviewer', findings: [
|
||||||
|
{ file: 'lib/auth/jwt.mjs', line: 19, rule_key: 'SECURITY_INJECTION', severity: 'BLOCKER', title: 'x'.repeat(101), detail: 'algo taken from the JWT header' },
|
||||||
|
] },
|
||||||
|
]);
|
||||||
|
assert.notEqual(result.verdict, 'ALLOW', 'an unsubstantiated BLOCKER must never clear the review');
|
||||||
|
assert.equal(result.findings.length, 0, 'it is still not a kept finding');
|
||||||
|
assert.equal(result.unverified.length, 1);
|
||||||
|
assert.equal(result.unverified[0].suppressed_reason, 'succinctness:title');
|
||||||
|
assert.equal(result.suppressed.length, 1, 'suppressed stays the union of dropped + unverified');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('runContract — a schema-invalid payload cannot yield ALLOW (an unread reviewer is an absent one)', () => {
|
||||||
|
// Measured: one ad-hoc rule_key invalidates the WHOLE payload at ingest, so a
|
||||||
|
// valid BLOCKER sibling disappears with it. That must not read as "clean".
|
||||||
|
const result = runContract([
|
||||||
|
{ reviewer: 'code-correctness-reviewer', findings: [
|
||||||
|
{ file: 'lib/auth/jwt.mjs', line: 19, rule_key: 'SECURITY_INJECTION', severity: 'BLOCKER', title: 'real' },
|
||||||
|
{ file: 'x.mjs', line: 1, rule_key: 'NOPE_KEY', severity: 'MINOR', title: 'ad-hoc key' },
|
||||||
|
] },
|
||||||
|
]);
|
||||||
|
assert.equal(result.skipped.length, 1);
|
||||||
|
assert.notEqual(result.verdict, 'ALLOW');
|
||||||
|
assert.ok(result.allow_blocked_by.some((x) => x.includes('code-correctness-reviewer')));
|
||||||
|
});
|
||||||
|
|
||||||
|
test('runContract — a reviewer named in expectedReviewers that never reported cannot yield ALLOW', () => {
|
||||||
|
const result = runContract(
|
||||||
|
[{ reviewer: 'code-correctness-reviewer', findings: [] }],
|
||||||
|
{ expectedReviewers: ['code-correctness-reviewer', 'brief-conformance-reviewer'] },
|
||||||
|
);
|
||||||
|
assert.deepEqual(result.missing_reviewers, ['brief-conformance-reviewer']);
|
||||||
|
assert.notEqual(result.verdict, 'ALLOW');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('runContract — known-positive control: every reviewer reported, nothing suppressed → ALLOW', () => {
|
||||||
|
// Proves ALLOW is still REACHABLE. Without this, "no ALLOW" is not a
|
||||||
|
// fail-closed contract, only a broken one.
|
||||||
|
const result = runContract(
|
||||||
|
[
|
||||||
|
{ reviewer: 'code-correctness-reviewer', findings: [
|
||||||
|
{ file: 'a.mjs', line: 1, rule_key: 'MISSING_ERROR_HANDLING', severity: 'MINOR', title: 'unguarded await', recommended_action: 'Wrap the await in a try/catch.' },
|
||||||
|
] },
|
||||||
|
{ reviewer: 'brief-conformance-reviewer', findings: [] },
|
||||||
|
],
|
||||||
|
{ expectedReviewers: ['code-correctness-reviewer', 'brief-conformance-reviewer'] },
|
||||||
|
);
|
||||||
|
assert.equal(result.verdict, 'ALLOW');
|
||||||
|
assert.equal(result.unverified.length, 0);
|
||||||
|
assert.deepEqual(result.missing_reviewers, []);
|
||||||
|
assert.deepEqual(result.allow_blocked_by, []);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('classifySuppression — the refuting reasons the LLM coordinator emits are declared here too', () => {
|
||||||
|
// agents/review-coordinator.md Pass 2 "Accuracy" and Pass 3 "Non-existent
|
||||||
|
// file" DO refute (a citation outside the repo root, a file absent from both
|
||||||
|
// tree and diff). Both are fs/judgement branches this deterministic subset
|
||||||
|
// excludes, but the vocabulary is owned here so prose and lib cannot drift.
|
||||||
|
assert.equal(classifySuppression('accuracy:refuted'), 'refuted');
|
||||||
|
assert.equal(classifySuppression('file-existence:refuted'), 'refuted');
|
||||||
|
assert.equal(classifySuppression('file-existence:indeterminate'), 'unverified',
|
||||||
|
'unresolvable must never collapse into refuted');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('suppression vocabulary — the two sets are disjoint and every reason is documented in the prose', () => {
|
||||||
|
const refuting = [...REFUTING_REASONS];
|
||||||
|
const overlap = refuting.filter((r) => UNVERIFIED_REASONS.includes(r));
|
||||||
|
assert.deepEqual(overlap, [], 'a reason cannot be both refuting and unverified');
|
||||||
|
|
||||||
|
const prose = readFileSync(join(ROOT, 'agents/review-coordinator.md'), 'utf-8');
|
||||||
|
assert.ok(prose.includes('review-coordinator'), 'known-positive control: the prose file loaded');
|
||||||
|
for (const reason of [...refuting, ...UNVERIFIED_REASONS]) {
|
||||||
|
assert.ok(prose.includes(reason),
|
||||||
|
`reason "${reason}" is declared in the lib but never documented in agents/review-coordinator.md`);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue