feat(review): the standard path puts back engine findings the coordinator dropped
This commit is contained in:
parent
8d399fddfb
commit
349a4fc789
SSH key fingerprint:
SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
6 changed files with 559 additions and 21 deletions
|
|
@ -351,9 +351,14 @@ git diff --name-only "$before_sha".."$after_sha" > "{project_dir}/.review-scope.
|
|||
git diff --name-only HEAD >> "{project_dir}/.review-scope.txt"
|
||||
node ${CLAUDE_PLUGIN_ROOT}/lib/review/engine-findings.mjs \
|
||||
--scope "{project_dir}/.review-scope.txt" --repo-root "$(git rev-parse --show-toplevel)" \
|
||||
"{project_dir}/.review-engine-code-review.json"
|
||||
"{project_dir}/.review-engine-code-review.json" \
|
||||
> "{project_dir}/.review-engine-payload-code-review.json" \
|
||||
&& cat "{project_dir}/.review-engine-payload-code-review.json"
|
||||
```
|
||||
|
||||
The converted payload is saved to `{project_dir}/.review-engine-payload-{engine}.json`
|
||||
(`code-review` or `security-review`), the same stdout, also written: Phase 7's finalize
|
||||
reads it from there. Never edit that file by hand.
|
||||
Exit 0 prints a reviewer payload (`claude-code-review` /
|
||||
`claude-security-review`) for Phase 6 plus `out_of_scope`: engine findings on
|
||||
files outside the review's SHA range (`/security-review` diffs against
|
||||
|
|
@ -610,7 +615,7 @@ Workflow({
|
|||
A reviewer that returns nothing, an engine listed in `engineNotRun`, or a coordinator that returns no verdict and findings forbids `ALLOW` on this path too (`missing_reviewers` in the result), and the result's `allow_blocked_by` names each one (`missing-reviewer:<name>`, the coordinator as `missing-reviewer:review-coordinator`);
|
||||
Phase 7 writes that list as the trailing JSON block's `allow_blocked_by`, so the verdict gate reads such a WARN review as not performed in full (exit 2; a BLOCK stays exit 1).
|
||||
On this path dedup ranks as Pass 1 does (both paths), and after the coordinator each finding is matched to its own row, its `(file, line, rule_key)` triplet plus its title, trimmed and in lower case: of the engine findings in a row no returned finding is on, the one with the highest severity (on a tie, the first seen) is put back at its own severity (operator decision 23), and a returned finding is never below the lower of its reviewer counterpart's severity and its catalogue tier (floor (b)); its counterpart is its own row, else the strictest row on its triplet.
|
||||
So the workflow path is stricter than the prose path.
|
||||
Both paths put back engine findings; the workflow path also applies floor (b).
|
||||
|
||||
Contract (verified in S10 part B — follow exactly):
|
||||
- **Pass `args` as a JSON object.** The script defensively re-parses a JSON *string*, but
|
||||
|
|
@ -685,7 +690,13 @@ findings reach review.md only through finalize's carry, so the verdict gate read
|
|||
`finalized: true` as exit 2 (`review-not-measured: round 2+ review not finalized`), never green.
|
||||
Finalize writes `round`, `round_base`, `finalized: true`, `ran` (what ran this round, read from the
|
||||
`.review-out-*.md` and `.review-engine-payload-*.json` files) and, from round 2, `resolved` with a
|
||||
`## Resolved since round <n>` section; a carried BLOCKER gets its `### Step R<n>`. `review-answers.mjs carry`
|
||||
`## Resolved since round <n>` section; a carried BLOCKER gets its `### Step R<n>`.
|
||||
The standard path now puts back engine findings at MAJOR or above the way the workflow path does: finalize's
|
||||
first pass reads the Phase 4.7 payloads itself from `{project_dir}` (`.review-engine-payload-*.json`, no new
|
||||
argument), and each engine finding at MAJOR or BLOCKER whose row no finding of review.md carries is put back at its
|
||||
own severity, with the line `**Source:** engine {name}, put back by the standard-path comparison` in its entry; the
|
||||
verdict is only raised, and a put-back BLOCKER gets its `### Step R<n>`. An engine finding in `out_of_scope` stays
|
||||
declined (`engine:out-of-scope`). `review-answers.mjs carry`
|
||||
writes an `## Answers` row with `Carried from` for each carried MAJOR the operator accepted in an
|
||||
earlier round; on exit 1 or 2 print the reason in one line and continue: no answer is carried.
|
||||
A finding is pre-existing when it sits on a line the range `scope_sha_start..scope_sha_end` did not add or
|
||||
|
|
|
|||
|
|
@ -458,7 +458,7 @@ missing `exit_code` (C15, `PROGRESS_FAILED_CRITERIA_KEY`).
|
|||
| `round_base` | string \| null | optional (1.2) | full 40- or 64-hex commit id, or `null` at round 1 | The commit the previous round reviewed; written by `review-finalize.mjs` |
|
||||
| `review_policy` | string | optional (1.2) | `absent \| applied \| not-applied` | Whether a `REVIEW.md` policy block was applied; written by `review-finalize.mjs` |
|
||||
| `review_policy_note` | string | optional (1.2) | one line | Why a policy was not applied, or what it did |
|
||||
| `review_policy_sha256` | string | optional (1.2) | 64-hex, or `none` | sha256 of the policy block read at `scope_sha_start` |
|
||||
| `review_policy_sha256` | string | optional (1.2) | 64-hex, or `none` | sha256 of the `REVIEW.md` blob read at `scope_sha_start` |
|
||||
| `finalized` | boolean | optional (1.2) | `true \| false` | `true` is set by `review-finalize.mjs` when it has written `ran` |
|
||||
|
||||
`findings:` is a flat array of finding-IDs (40-char hex from `lib/parsers/finding-id.mjs`). The full finding objects (severity, location, message, evidence, fix) live in the body as `### <id>` subsections under per-severity `## Findings (...)` headings — same pattern as brief-reviewer to avoid frontmatter-parser fragility on lists of dicts.
|
||||
|
|
@ -474,6 +474,7 @@ Since 1.1 (ordre 5):
|
|||
- `## Remediation Steps` — required at `review_version` ≥ 1.1 when there is a BLOCKER: one `### Step R<n>` per BLOCKER carrying `Source finding: <id>` (strict error `REVIEW_MISSING_REMEDIATION_STEP`, soft warning). Steps are appended and numbered, never reworded across iterations (Spec Kit `/converge`).
|
||||
- `## Declined to Judge` — required at `review_version` ≥ 1.1 (strict error, soft warning; 1.0 reviews exempt): every finding set aside without a ruling, with its reason (including engine findings outside the review scope, `engine:out-of-scope`), plus `Trimmed: N`. **No finding is struck for length**: only MINOR/SUGGESTION text is trimmed, and counted (`trimmed` in the JSON block); a long BLOCKER/MAJOR keeps every character.
|
||||
- Findings from Claude Code's own `/code-review` and `/security-review` enter under the catalogue keys `CODE_REVIEW_DEFECT` (MAJOR), `CODE_REVIEW_IMPROVEMENT` (MINOR: a `/code-review` finding whose category is simplification, efficiency, reuse, style or readability) and `SECURITY_REVIEW_VULNERABILITY` (BLOCKER), or a narrower key, with `reviewer` `claude-code-review` / `claude-security-review`.
|
||||
- Working files of one review run, in the project folder (not part of `review.md`): the engine transcriptions `.review-engine-{code-review|security-review}.json` (`/trekreview` Phase 4.7) and, beside them, the converted payloads `.review-engine-payload-{engine}.json` (the `engine-findings.mjs` output, `{reviewer, findings, out_of_scope}`). `review-finalize.mjs` reads the payloads to record which engines ran and to put back, at its own severity, an engine finding at MAJOR or BLOCKER that no finding of `review.md` carries (the standard-path comparison); `review-rounds.mjs prepare` deletes the payloads at the start of each round.
|
||||
|
||||
Since 1.2 (every field and section optional; a malformed one present is a strict error, a soft warning):
|
||||
- Trailing JSON block, additive: per finding `pre_existing` (boolean: the range did not add or change the cited line), `carried_from` (the previous round's finding id) and `instruction_ref` (the repo path of the instruction file a CLAUDE.md-key finding cites); top level `pre_existing_counts` (`{SEVERITY: n}`), `resolved` (the previous round's finding ids no longer raised) and `ran` (`{engines: [code-review|security-review], reviewers: [string]}`, written with `finalized: true`). Codes `REVIEW_PRE_EXISTING_MALFORMED`, `REVIEW_ROUND_MALFORMED`, `REVIEW_POLICY_MALFORMED`.
|
||||
|
|
|
|||
89
lib/review/engine-compare.mjs
Normal file
89
lib/review/engine-compare.mjs
Normal file
|
|
@ -0,0 +1,89 @@
|
|||
// lib/review/engine-compare.mjs
|
||||
// The standard-path put-back (fix round 4.5b2 part B, Step 9): on the prose
|
||||
// path, an engine finding at MAJOR or BLOCKER that the coordinator dropped is
|
||||
// put back at its own severity, the way the workflow path does
|
||||
// (scripts/trekreview-armB.workflow.mjs, operator decision 23).
|
||||
//
|
||||
// A finding is matched on its row: its (file, line, rule_key) triplet plus
|
||||
// its title, trimmed and in lower case. An engine finding's row is returned
|
||||
// when a finding of review.md carries its key, or the key of its title as the
|
||||
// coordinator trims it, so a trimmed title puts back no copy. Each row not
|
||||
// returned is put back once, its highest severity first (on a tie, the first
|
||||
// seen), with `engine` taken from the payload it came from (the `reviewer`
|
||||
// field mapped back through ENGINES, never the finding's own label) and no id
|
||||
// (lib/review/review-finalize.mjs recomputes every id). The verdict is only
|
||||
// raised: a BLOCKER gives BLOCK, a MAJOR at least WARN.
|
||||
//
|
||||
// Only a payload's `findings` are compared. An engine finding in
|
||||
// `out_of_scope` stays declined by the existing rule
|
||||
// (lib/review/coordinator-contract.mjs runContract, `engine:out-of-scope`).
|
||||
//
|
||||
// The row rule below is a lib copy of the armB workflow's (scripts cannot be
|
||||
// imported); tests/lib/engine-findings-compare.test.mjs pins both equal.
|
||||
|
||||
import { ENGINES } from './engine-findings.mjs';
|
||||
|
||||
// Copied from scripts/trekreview-armB.workflow.mjs:60-73 (rowOf, trimmedTitle).
|
||||
// The (file,line,rule_key) triplet a finding is deduped and matched on.
|
||||
const tripletOf = (f) => `${f.file}\u0000${f.line}\u0000${f.rule_key}`;
|
||||
// A finding's title as the grouping compares it: trimmed, lower case.
|
||||
const normTitle = (f) => String(f.title).trim().toLowerCase();
|
||||
// The row a finding is matched to after the coordinator: its triplet plus its
|
||||
// `normTitle`. An untitled finding gets a key no titled row has.
|
||||
const rowOf = (f) => `${tripletOf(f)}\u0000${f.title === undefined ? '\u0001' : normTitle(f)}`;
|
||||
// The coordinator trims an over-long MINOR/SUGGESTION title as
|
||||
// lib/review/coordinator-contract.mjs judgeFilter does; its marker and limit
|
||||
// are inlined (scripts cannot import) and pinned equal by the workflow test.
|
||||
const TRIM_MARKER = ' […trimmed]';
|
||||
const JUDGE_TITLE_MAX = 100;
|
||||
// A title as that trim writes it: over JUDGE_TITLE_MAX characters, cut to
|
||||
// leave room for the marker, then the marker; any other title unchanged.
|
||||
const trimmedTitle = (t) => (t.length > JUDGE_TITLE_MAX ? t.slice(0, JUDGE_TITLE_MAX - TRIM_MARKER.length) + TRIM_MARKER : t);
|
||||
|
||||
export { tripletOf, normTitle, rowOf, trimmedTitle };
|
||||
|
||||
const PUT_BACK_SEVERITIES = ['BLOCKER', 'MAJOR'];
|
||||
const VERDICT_RANK = { ALLOW: 0, WARN: 1, BLOCK: 2 };
|
||||
const ENGINE_OF_REVIEWER = Object.freeze(Object.fromEntries(Object.entries(ENGINES).map(([engine, s]) => [s.reviewer, engine])));
|
||||
const isPlainObject = (x) => x !== null && typeof x === 'object' && !Array.isArray(x);
|
||||
|
||||
/**
|
||||
* The engine findings at MAJOR or BLOCKER whose row no finding of `payload`
|
||||
* carries, as copies to append, and the verdict raised by them.
|
||||
* @param {{ verdict?: string, findings: object[] }} payload - review.md's trailing JSON block
|
||||
* @param {Array<{ reviewer: string, findings: object[], out_of_scope?: object[] }>} enginePayloads
|
||||
* @returns {{ added: object[], raisedVerdict: string|undefined }}
|
||||
*/
|
||||
export function putBack(payload, enginePayloads) {
|
||||
const returned = (Array.isArray(payload?.findings) ? payload.findings : []).filter(isPlainObject);
|
||||
const rows = new Set(returned.map(rowOf));
|
||||
const rowReturned = (m) => rows.has(rowOf(m)) ||
|
||||
(typeof m.title === 'string' && rows.has(rowOf({ ...m, title: trimmedTitle(m.title) })));
|
||||
const candidates = [];
|
||||
for (const p of Array.isArray(enginePayloads) ? enginePayloads : []) {
|
||||
if (!isPlainObject(p) || !Array.isArray(p.findings)) continue;
|
||||
const engine = typeof p.reviewer === 'string' && Object.hasOwn(ENGINE_OF_REVIEWER, p.reviewer) ? ENGINE_OF_REVIEWER[p.reviewer] : null;
|
||||
if (!engine) continue;
|
||||
for (const f of p.findings) {
|
||||
if (!isPlainObject(f) || !PUT_BACK_SEVERITIES.includes(f.severity)) continue;
|
||||
candidates.push({ f, engine, rank: PUT_BACK_SEVERITIES.indexOf(f.severity) });
|
||||
}
|
||||
}
|
||||
// Highest severity first; Array.prototype.sort is stable, so a tie keeps the first seen.
|
||||
candidates.sort((a, b) => a.rank - b.rank);
|
||||
const added = [];
|
||||
for (const { f, engine } of candidates) {
|
||||
if (rowReturned(f)) continue;
|
||||
const { id: _id, brief_lowered_from: _lowered, contradicts_brief: _contradicts, pre_existing: _pre, carried_from: _cf, ...rest } = f;
|
||||
const copy = { id: null, ...rest, engine };
|
||||
added.push(copy);
|
||||
rows.add(rowOf(copy));
|
||||
}
|
||||
const current = payload?.verdict;
|
||||
let raised = current;
|
||||
if (Object.hasOwn(VERDICT_RANK, current)) {
|
||||
const need = added.some((f) => f.severity === 'BLOCKER') ? 'BLOCK' : added.length > 0 ? 'WARN' : 'ALLOW';
|
||||
if (VERDICT_RANK[need] > VERDICT_RANK[current]) raised = need;
|
||||
}
|
||||
return { added, raisedVerdict: raised };
|
||||
}
|
||||
|
|
@ -3,8 +3,21 @@
|
|||
// lib/review/fill-finding-ids.mjs, on both paths (review_version 1.2, fix
|
||||
// round 4.5b2 part B). Its passes run in the Architecture order (engine,
|
||||
// instruction, carry, pre-existing, policy); later steps insert theirs at
|
||||
// their places in PASSES. Today it runs carry, pre-existing and policy:
|
||||
// their places in PASSES. Today it runs engine, carry, pre-existing and policy:
|
||||
//
|
||||
// - engine, always the first pass: `putBack` (lib/review/engine-compare.mjs)
|
||||
// compares the engine payloads Phase 4.7 saved in the project folder
|
||||
// (each regular `.review-engine-payload-*.json` whose top-level `reviewer`
|
||||
// maps back through `ENGINES`, read with readdirSync) with the review's
|
||||
// findings. Each engine finding at MAJOR or BLOCKER whose row no finding
|
||||
// carries (its own row, or the row of its trimmed title) is appended at
|
||||
// its own severity with `engine` kept and no id (the id recomputation
|
||||
// gives it its computed id). It gets a `### ` entry under its
|
||||
// `## Findings (<SEV>)` section with the line
|
||||
// `**Source:** engine {name}, put back by the standard-path comparison`,
|
||||
// the frontmatter list, counts, a verdict that is only raised, and, as a
|
||||
// BLOCKER, its `### Step R<n>`. An engine finding in `out_of_scope` is
|
||||
// not compared (it stays declined, `engine:out-of-scope`).
|
||||
// - carry (round 2 and later, `--previous <archive>`): `carry`
|
||||
// (lib/review/review-rounds.mjs) keeps every previous finding that this
|
||||
// round gives no positive evidence of being resolved, with `carried_from`;
|
||||
|
|
@ -80,6 +93,7 @@ import { markFindings, scopeShas } from './pre-existing.mjs';
|
|||
import { ENGINES } from './engine-findings.mjs';
|
||||
import { carry, frontmatterField, MAX_REVIEW_ROUNDS, runId } from './review-rounds.mjs';
|
||||
import { applyPolicy, readPolicy } from './review-policy.mjs';
|
||||
import { putBack } from './engine-compare.mjs';
|
||||
import { isMain } from '../util/is-main.mjs';
|
||||
|
||||
const SEVERITIES = ['BLOCKER', 'MAJOR', 'MINOR', 'SUGGESTION'];
|
||||
|
|
@ -185,6 +199,27 @@ function appendToSection(lines, headingRe, heading, block, before) {
|
|||
return lines;
|
||||
}
|
||||
|
||||
// ---- pass: engine ------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Pass `engine`, always the first pass: the engine findings at MAJOR or
|
||||
* BLOCKER the coordinator dropped are put back (lib/review/engine-compare.mjs).
|
||||
*/
|
||||
function passEngine(payload, ctx) {
|
||||
const r = putBack(payload, ctx.evidence.payloads ?? []);
|
||||
for (const f of r.added) {
|
||||
payload.findings.push(f);
|
||||
ctx.added.add(f);
|
||||
ctx.putBack.add(f);
|
||||
}
|
||||
ctx.engine = { put_back: r.added.length };
|
||||
if (r.added.length === 0) return {};
|
||||
ctx.recount = true;
|
||||
if (r.raisedVerdict === payload.verdict) return {};
|
||||
payload.verdict = r.raisedVerdict;
|
||||
return { headEdit: (head) => withVerdict(head, r.raisedVerdict) };
|
||||
}
|
||||
|
||||
// ---- pass: carry -------------------------------------------------------------------
|
||||
|
||||
const LATER_SECTIONS = [/^##\s+Suppressed Findings\b/, /^##\s+Pre-existing\s*$/, /^##\s+Resolved since round\b/,
|
||||
|
|
@ -201,9 +236,20 @@ function resolvedSection(n, list) {
|
|||
function passCarry(payload, ctx) {
|
||||
if (!ctx.previous) return {};
|
||||
const before = payload.findings.length;
|
||||
const r = carry(ctx.previous.findings, payload.findings, {
|
||||
const current = payload.findings;
|
||||
const r = carry(ctx.previous.findings, current, {
|
||||
marks: ctx.evidence.marks, ran: ctx.evidence.ran, previousRan: ctx.previous.ran,
|
||||
});
|
||||
// carry returns a copy of each current finding at its index: a finding an
|
||||
// earlier pass added (an engine put-back) stays added under its copy.
|
||||
for (let i = 0; i < before; i++) {
|
||||
for (const set of [ctx.added, ctx.putBack]) {
|
||||
if (set.has(current[i])) {
|
||||
set.delete(current[i]);
|
||||
set.add(r.findings[i]);
|
||||
}
|
||||
}
|
||||
}
|
||||
payload.findings = r.findings;
|
||||
for (const f of r.findings.slice(before)) ctx.added.add(f);
|
||||
payload.resolved = r.resolved;
|
||||
|
|
@ -378,6 +424,7 @@ function passPolicy(payload, ctx) {
|
|||
|
||||
/** The passes, in the Architecture order; the id recomputation runs after them, always last. */
|
||||
export const PASSES = Object.freeze([
|
||||
{ name: 'engine', run: passEngine },
|
||||
{ name: 'carry', run: passCarry },
|
||||
{ name: 'pre-existing', run: passPreExisting },
|
||||
{ name: 'policy', run: passPolicy },
|
||||
|
|
@ -433,9 +480,14 @@ function editFrontmatter(head, ids, fields) {
|
|||
return [lines[0], ...out, ...lines.slice(fmEnd)].join('\n');
|
||||
}
|
||||
|
||||
/** A body entry for a finding finalize added, in the coordinator's form. */
|
||||
function entryLines(f) {
|
||||
const lines = [`### ${f.id}`, '', `- file: ${oneLine(f.file)}`, `- line: ${f.line}`, `- rule_key: ${oneLine(f.rule_key)}`];
|
||||
/**
|
||||
* A body entry for a finding finalize added, in the coordinator's form; an
|
||||
* engine put-back names its source first.
|
||||
*/
|
||||
function entryLines(f, putBackSet) {
|
||||
const lines = [`### ${f.id}`, ''];
|
||||
if (putBackSet.has(f)) lines.push(`**Source:** engine ${oneLine(f.engine)}, put back by the standard-path comparison`, '');
|
||||
lines.push(`- file: ${oneLine(f.file)}`, `- line: ${f.line}`, `- rule_key: ${oneLine(f.rule_key)}`);
|
||||
for (const k of ['brief_ref', 'title', 'detail', 'recommended_action', 'engine', 'carried_from']) {
|
||||
if (f[k] !== undefined && f[k] !== null && oneLine(f[k]) !== '') lines.push(`- ${k}: ${oneLine(f[k])}`);
|
||||
}
|
||||
|
|
@ -453,14 +505,14 @@ function findingsBefore(sev) {
|
|||
* The body entries: one added for each finding in `added`, and a `### <id>`
|
||||
* entry that sits under another severity's `## Findings` section moved to its own.
|
||||
*/
|
||||
function placeEntries(head, findings, added) {
|
||||
function placeEntries(head, findings, added, putBackSet) {
|
||||
let lines = head.split('\n');
|
||||
const fmEnd = frontmatterEnd(lines);
|
||||
for (const f of findings) {
|
||||
const sev = sevOf(f);
|
||||
if (!SEVERITIES.includes(sev) || typeof f.id !== 'string') continue;
|
||||
if (added.has(f)) {
|
||||
lines = appendToSection(lines, findingsHeading(sev), `## Findings (${sev})`, entryLines(f), findingsBefore(sev));
|
||||
lines = appendToSection(lines, findingsHeading(sev), `## Findings (${sev})`, entryLines(f, putBackSet), findingsBefore(sev));
|
||||
continue;
|
||||
}
|
||||
const at = lines.findIndex((l, i) => i > fmEnd && new RegExp(`^###\\s+${f.id}\\s*$`).test(l));
|
||||
|
|
@ -566,8 +618,8 @@ export function finalizeReview(text, { repo, round, roundBase = null, previous =
|
|||
const dup = before.find((id, i) => before.indexOf(id) !== i);
|
||||
if (dup !== undefined) return { exit: 1, reason: `two findings carry the id ${JSON.stringify(dup)}` };
|
||||
|
||||
const ev = evidence ?? { ran: { engines: [], reviewers: [] }, marks: [] };
|
||||
const ctx = { repo, text, round, previous, evidence: ev, added: new Set(), carried: false };
|
||||
const ev = evidence ?? { ran: { engines: [], reviewers: [] }, marks: [], payloads: [] };
|
||||
const ctx = { repo, text, round, previous, evidence: ev, added: new Set(), putBack: new Set(), carried: false };
|
||||
let head = parts.head;
|
||||
const out = { passes: [] };
|
||||
for (const pass of PASSES) {
|
||||
|
|
@ -590,7 +642,7 @@ export function finalizeReview(text, { repo, round, roundBase = null, previous =
|
|||
head = rewriteIds(head, map);
|
||||
out.passes.push('ids');
|
||||
|
||||
head = placeEntries(head, payload.findings, ctx.added);
|
||||
head = placeEntries(head, payload.findings, ctx.added, ctx.putBack);
|
||||
if (ctx.carried || ctx.recount) {
|
||||
payload.counts = Object.fromEntries(SEVERITIES.map((s) => [s, payload.findings.filter((f) => sevOf(f) === s).length]));
|
||||
}
|
||||
|
|
@ -617,22 +669,24 @@ export function finalizeReview(text, { repo, round, roundBase = null, previous =
|
|||
return {
|
||||
exit: 0, text: `${head}${parts.fence}${body}${parts.tail}`, ...out, ids_rewritten: map.size,
|
||||
carried: ctx.added.size, resolved: payload.resolved ?? [], ran: payload.ran, r_steps_added: steps.added,
|
||||
policy: ctx.policy ?? null,
|
||||
policy: ctx.policy ?? null, put_back: ctx.engine?.put_back ?? 0,
|
||||
};
|
||||
}
|
||||
|
||||
// ---- evidence and the previous review ------------------------------------------------
|
||||
|
||||
/**
|
||||
* What ran this round, from the project folder (positive evidence only), and
|
||||
* the reviewers' marks of previous findings.
|
||||
* What ran this round, from the project folder (positive evidence only), the
|
||||
* reviewers' marks of previous findings, and the engine payloads that prove
|
||||
* an engine ran (for the engine pass).
|
||||
* @param {string} dir
|
||||
* @returns {{ ran: {engines: string[], reviewers: string[]}, marks: Array<{id, resolved, reviewer}> }}
|
||||
* @returns {{ ran: {engines: string[], reviewers: string[]}, marks: Array<{id, resolved, reviewer}>, payloads: object[] }}
|
||||
*/
|
||||
export function roundEvidence(dir) {
|
||||
const engines = new Set();
|
||||
const reviewers = new Set();
|
||||
const marks = [];
|
||||
const payloads = [];
|
||||
let names = [];
|
||||
try { names = readdirSync(dir).sort(); } catch { names = []; }
|
||||
for (const name of names) {
|
||||
|
|
@ -643,7 +697,10 @@ export function roundEvidence(dir) {
|
|||
try { payload = JSON.parse(readFileSync(p, 'utf8')); } catch { continue; }
|
||||
const engine = isPlainObject(payload) && typeof payload.reviewer === 'string' && Object.hasOwn(ENGINE_OF_REVIEWER, payload.reviewer)
|
||||
? ENGINE_OF_REVIEWER[payload.reviewer] : null;
|
||||
if (engine && Array.isArray(payload.findings)) engines.add(engine);
|
||||
if (engine && Array.isArray(payload.findings)) {
|
||||
engines.add(engine);
|
||||
payloads.push(payload);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
const m = OUT_RE.exec(name);
|
||||
|
|
@ -659,7 +716,7 @@ export function roundEvidence(dir) {
|
|||
}
|
||||
}
|
||||
}
|
||||
return { ran: { engines: [...engines].sort(), reviewers: [...reviewers].sort() }, marks };
|
||||
return { ran: { engines: [...engines].sort(), reviewers: [...reviewers].sort() }, marks, payloads };
|
||||
}
|
||||
|
||||
/** A `ran` object of the 1.2 shape, else null. */
|
||||
|
|
|
|||
2
tests/fixtures/jsonl-schemas.md
vendored
2
tests/fixtures/jsonl-schemas.md
vendored
|
|
@ -25,7 +25,7 @@
|
|||
| trekexecute-stats (Phase 9 record) | ts, plan, plan_type, mode, result, steps_total, steps_passed, steps_failed, steps_skipped, failed_at_step | lib/stats/stats-append.mjs (trekexecute.md Phase 9) | trekexecute.md:3005-3027 | profile, phase_models, profile_source | none |
|
||||
| trekexecute-stats (autonomy events) | ts, event, known_event, payload | lib/stats/event-emit.mjs `emit()` | event-emit.mjs:78-99 | payload.profile, payload.phase_models, payload.profile_source | none |
|
||||
| trekexecute-stats (PostToolUse Bash) | ts, session_id, command_excerpt, duration_ms, success | hooks/scripts/post-bash-stats.mjs (Bash PostToolUse) | post-bash-stats.mjs:42-54 | none (hook is plugin-level, not profile-aware) | command_excerpt (CWE-212) |
|
||||
| trekreview-stats | ts, slug, verdict, counts (BLOCKER/MAJOR/MINOR/SUGGESTION), reviewed_files_count, mode, duration_ms | lib/stats/stats-append.mjs (trekreview.md Phase 8) | trekreview.md:723-776 | profile, phase_models, profile_source | none |
|
||||
| trekreview-stats | ts, slug, verdict, counts (BLOCKER/MAJOR/MINOR/SUGGESTION), reviewed_files_count, mode, duration_ms | lib/stats/stats-append.mjs (trekreview.md Phase 8) | trekreview.md:734-787 | profile, phase_models, profile_source | none |
|
||||
| trekcontinue-stats | ts, project, next_session_label, status | lib/stats/stats-append.mjs (trekcontinue.md Phase 5) | trekcontinue.md:304-308 | profile, profile_source | none |
|
||||
| token-usage | ts, session_id, scope, model, tokens_input, tokens_output, tokens_cache_creation, tokens_cache_read, cost_usd, is_estimate, price_table_version | lib/stats/token-usage.mjs `captureTokenUsage` (via hooks/scripts/otel-export.mjs Stop hook, opt-in `VOYAGE_TOKEN_METER`) | token-usage.mjs:buildRecord | n/a (SKAL-2 schema) | session_id (stripped at export) |
|
||||
|
||||
|
|
|
|||
380
tests/lib/engine-findings-compare.test.mjs
Normal file
380
tests/lib/engine-findings-compare.test.mjs
Normal file
|
|
@ -0,0 +1,380 @@
|
|||
// tests/lib/engine-findings-compare.test.mjs
|
||||
// [ENG] fix round 4.5b2 part B, Step 9: on the standard path an engine finding
|
||||
// the coordinator dropped is put back at its severity. /trekreview Phase 4.7
|
||||
// saves each converted engine payload to
|
||||
// `{project_dir}/.review-engine-payload-{engine}.json`;
|
||||
// lib/review/review-finalize.mjs runs its `engine` pass first: each engine
|
||||
// finding at MAJOR or BLOCKER whose row (lib/review/engine-compare.mjs
|
||||
// `rowOf`, the armB workflow's row rule, or the row of its trimmed title) no
|
||||
// finding of review.md carries is appended at its own severity with `engine`
|
||||
// kept, a `### ` entry with the `**Source:**` line, the JSON, the frontmatter
|
||||
// list and counts, and a verdict that is only raised. A put-back BLOCKER gets
|
||||
// its `### Step R<n>`. An engine finding in `out_of_scope` is not compared.
|
||||
// Tests run the real CLIs (engine-findings, finalize, fill-finding-ids,
|
||||
// validator, gate) in throwaway git repos with no hooks; engine-compare.mjs
|
||||
// is also read directly and pinned equal to the armB workflow's copy.
|
||||
|
||||
import { after, test } from 'node:test';
|
||||
import { strict as assert } from 'node:assert';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { appendFileSync, existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { dirname, join } from 'node:path';
|
||||
import { fileURLToPath, pathToFileURL } from 'node:url';
|
||||
import { computeFindingId, computeFindingIds } from '../../lib/parsers/finding-id.mjs';
|
||||
import { childEnv } from '../helpers/child-env.mjs';
|
||||
|
||||
const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..', '..');
|
||||
const COMPARE = join(ROOT, 'lib', 'review', 'engine-compare.mjs');
|
||||
const ENGINE_CLI = join(ROOT, 'lib', 'review', 'engine-findings.mjs');
|
||||
const FINALIZE = join(ROOT, 'lib', 'review', 'review-finalize.mjs');
|
||||
const FILL = join(ROOT, 'lib', 'review', 'fill-finding-ids.mjs');
|
||||
const VALIDATOR = join(ROOT, 'lib', 'validators', 'review-validator.mjs');
|
||||
const GATE = join(ROOT, 'lib', 'review', 'verdict-gate.mjs');
|
||||
const ARMB = join(ROOT, 'scripts', 'trekreview-armB.workflow.mjs');
|
||||
const REVIEW_REL = '.claude/projects/2026-11-02-eng/review.md';
|
||||
const SOURCE_LINE = (engine) => `**Source:** engine ${engine}, put back by the standard-path comparison`;
|
||||
|
||||
// Loaded so that each new assertion fails on its own while the module is absent.
|
||||
let compareMod = {};
|
||||
try {
|
||||
compareMod = await import(pathToFileURL(COMPARE).href);
|
||||
} catch (e) {
|
||||
if (existsSync(COMPARE)) throw e;
|
||||
}
|
||||
|
||||
const made = [];
|
||||
after(() => { for (const b of made) rmSync(b, { recursive: true, force: true }); });
|
||||
|
||||
function g(cwd, ...args) {
|
||||
const r = spawnSync('git', args, { cwd, encoding: 'utf8', env: childEnv() });
|
||||
if (r.status !== 0) throw new Error(`git ${args.join(' ')}: ${r.stderr}`);
|
||||
return r.stdout.trim();
|
||||
}
|
||||
|
||||
function putIn(dir, rel, text) {
|
||||
mkdirSync(dirname(join(dir, rel)), { recursive: true });
|
||||
writeFileSync(join(dir, rel), text);
|
||||
}
|
||||
|
||||
const run = (args) => spawnSync(process.execPath, args, { encoding: 'utf8', env: childEnv() });
|
||||
const parse = (r) => { try { return JSON.parse(r.stdout); } catch { return {}; } };
|
||||
const ten = (tag) => Array.from({ length: 10 }, (_, i) => `export const ${tag}${i + 1} = ${i + 1};`).join('\n') + '\n';
|
||||
|
||||
/**
|
||||
* A throwaway repo: the start commit holds src/a.mjs, src/b.mjs and
|
||||
* docs/guide/x.md; the end commit changes line 1 of src/a.mjs and line 3 of
|
||||
* src/b.mjs. No REVIEW.md, so no policy.
|
||||
*/
|
||||
function engRepo() {
|
||||
const base = mkdtempSync(join(tmpdir(), 'engine-compare-eng-'));
|
||||
made.push(base);
|
||||
const repo = join(base, 'repo');
|
||||
mkdirSync(repo, { recursive: true });
|
||||
mkdirSync(join(base, 'hooks-none'));
|
||||
g(repo, 'init', '-q', '-b', 'main');
|
||||
appendFileSync(join(repo, '.git', 'config'), '[user]\n\tname = Test\n\temail = test@example.invalid\n'
|
||||
+ `[commit]\n\tgpgsign = false\n[core]\n\thooksPath = "${join(base, 'hooks-none')}"\n`);
|
||||
appendFileSync(join(repo, '.git', 'info', 'exclude'), '.claude/projects/\n');
|
||||
putIn(repo, 'src/a.mjs', ten('a'));
|
||||
putIn(repo, 'src/b.mjs', ten('b'));
|
||||
putIn(repo, 'docs/guide/x.md', '# Guide\n\none\ntwo\nthree\n');
|
||||
g(repo, 'add', '-A');
|
||||
g(repo, 'commit', '-q', '-m', 'chore: start');
|
||||
const start = g(repo, 'rev-parse', 'HEAD');
|
||||
putIn(repo, 'src/a.mjs', ten('a').replace('export const a1 = 1;', 'export const a1 = 11;'));
|
||||
putIn(repo, 'src/b.mjs', ten('b').replace('export const b3 = 3;', 'export const b3 = 33;'));
|
||||
g(repo, 'add', '-A');
|
||||
g(repo, 'commit', '-q', '-m', 'feat: the work');
|
||||
const end = g(repo, 'rev-parse', 'HEAD');
|
||||
const reviewPath = join(repo, ...REVIEW_REL.split('/'));
|
||||
const project = dirname(reviewPath);
|
||||
mkdirSync(project, { recursive: true });
|
||||
const scope = join(base, 'scope.txt');
|
||||
writeFileSync(scope, 'src/a.mjs\nsrc/b.mjs\ndocs/guide/x.md\n');
|
||||
return {
|
||||
repo, start, end, reviewPath, project,
|
||||
write: (text) => writeFileSync(reviewPath, text),
|
||||
read: () => readFileSync(reviewPath, 'utf8'),
|
||||
/** An engine's transcription converted by the real engine-findings.mjs CLI and saved as Phase 4.7 saves it. */
|
||||
engine: (engine, findings) => {
|
||||
const input = join(base, `.review-engine-${engine}.json`);
|
||||
writeFileSync(input, JSON.stringify({ engine, findings }));
|
||||
const r = run([ENGINE_CLI, '--scope', scope, input]);
|
||||
assert.equal(r.status, 0, `engine-findings: ${r.stdout}${r.stderr}`);
|
||||
writeFileSync(join(project, `.review-engine-payload-${engine}.json`), r.stdout);
|
||||
return JSON.parse(r.stdout);
|
||||
},
|
||||
finalize: () => run([FINALIZE, '--review', reviewPath, '--repo', repo, '--project', project, '--round', '1']),
|
||||
fill: () => run([FILL, reviewPath]),
|
||||
validate: () => run([VALIDATOR, '--strict', reviewPath]),
|
||||
gate: () => run([GATE, '--json', reviewPath, '--repo', repo]),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* A review_version 1.2 review of `start..end` as the coordinator wrote it and
|
||||
* fill-finding-ids filled it. `findings` are [{severity, file, line, rule_key,
|
||||
* engine?, title}]; ids computed over (file, line, rule_key). `declined` goes
|
||||
* into the JSON `declined_to_judge`.
|
||||
*/
|
||||
function engReview({ start, end, findings, declined = [] }) {
|
||||
const list = findings.map((f, i) => ({
|
||||
severity: f.severity, rule_key: f.rule_key, ...(f.engine ? { engine: f.engine } : {}),
|
||||
file: f.file, line: f.line, title: f.title ?? `t${i + 1}`, detail: `d${i + 1}`, recommended_action: 'Add a test.',
|
||||
}));
|
||||
const ids = computeFindingIds(list);
|
||||
const ordered = list.map((f, i) => ({ id: ids[i], ...f }));
|
||||
const counts = { BLOCKER: 0, MAJOR: 0, MINOR: 0, SUGGESTION: 0 };
|
||||
for (const f of ordered) counts[f.severity] += 1;
|
||||
const v = counts.BLOCKER > 0 ? 'BLOCK' : counts.MAJOR > 0 ? 'WARN' : 'ALLOW';
|
||||
const json = JSON.stringify({ verdict: v, counts, allow_blocked_by: [], declined_to_judge: declined, findings: ordered }, null, 2);
|
||||
const sections = ['BLOCKER', 'MAJOR', 'MINOR', 'SUGGESTION'].map((sev) => {
|
||||
const of = ordered.filter((f) => f.severity === sev);
|
||||
if (of.length === 0) return '';
|
||||
return `## Findings (${sev})\n\n${of.map((f) => `### ${f.id}\n\n- **Rule:** \`${f.rule_key}\`\n- **Location:** \`${f.file}:${f.line}\`\n- **Title:** ${f.title}\n`).join('\n')}\n`;
|
||||
}).join('');
|
||||
const steps = ordered.filter((f) => f.severity === 'BLOCKER')
|
||||
.map((f, i) => `### Step R${i + 1}: fix ${f.title}\n\n- **Files:** \`${f.file}\`\n- **Source finding:** \`${f.id}\`\n- **Test first:** a test that fails today\n- **Verify:** re-run /trekreview\n- **On failure:** escalate\n`).join('\n');
|
||||
const declinedLines = declined.length === 0 ? 'None.'
|
||||
: declined.map((d) => `- ${d.file}:${d.line} ${d.rule_key} — ${d.reason} (${d.title})`).join('\n');
|
||||
return `---
|
||||
type: trekreview
|
||||
review_version: "1.2"
|
||||
task: "t"
|
||||
slug: eng
|
||||
project_dir: .claude/projects/2026-11-02-eng/
|
||||
brief_path: .claude/projects/2026-11-02-eng/brief.md
|
||||
scope_sha_start: ${start}
|
||||
scope_sha_end: ${end}
|
||||
reviewed_files_count: 3
|
||||
verdict: ${v}
|
||||
findings:
|
||||
${ordered.map((f) => ` - ${f.id}`).join('\n')}
|
||||
---
|
||||
|
||||
# Review: t
|
||||
|
||||
## Executive Summary
|
||||
|
||||
x
|
||||
|
||||
## Coverage
|
||||
|
||||
| File | Treatment | Reason |
|
||||
|------|-----------|--------|
|
||||
| src/a.mjs | deep-review | default |
|
||||
| src/b.mjs | deep-review | default |
|
||||
| docs/guide/x.md | deep-review | default |
|
||||
|
||||
${sections}## Declined to Judge
|
||||
|
||||
${declinedLines}
|
||||
|
||||
Trimmed: 0
|
||||
|
||||
## Remediation Summary
|
||||
|
||||
x
|
||||
|
||||
## Remediation Steps
|
||||
|
||||
${steps || 'None.\n'}
|
||||
\`\`\`json
|
||||
${json}
|
||||
\`\`\`
|
||||
`;
|
||||
}
|
||||
|
||||
const jsonOf = (text) => JSON.parse(text.slice(text.lastIndexOf('```json\n') + 8, text.lastIndexOf('```')));
|
||||
const frontmatter = (text) => /^---\n([\s\S]*?)\n---\n/.exec(text)?.[1] ?? '';
|
||||
const fmField = (text, key) => {
|
||||
const m = new RegExp(`^${key}:[ \\t]*(.*)$`, 'm').exec(frontmatter(text));
|
||||
return m ? m[1].replace(/^"(.*)"$/, '$1') : undefined;
|
||||
};
|
||||
const fmIds = (text) => frontmatter(text).split('\n').filter((l) => /^ {2}- [0-9a-f]{40}$/.test(l)).map((l) => l.trim().slice(2));
|
||||
/** The text from `heading` up to the next `## ` heading ('' when absent). */
|
||||
function sectionOf(text, heading) {
|
||||
const start = text.indexOf(`\n${heading}`);
|
||||
if (start === -1) return '';
|
||||
const next = text.indexOf('\n## ', start + heading.length + 1);
|
||||
return text.slice(start + 1, next === -1 ? undefined : next);
|
||||
}
|
||||
/** The `### <id>` entry of `id` in `section`, up to the next `### ` or end ('' when absent). */
|
||||
function entryOf(section, id) {
|
||||
const at = section.indexOf(`### ${id}\n`);
|
||||
if (at === -1) return '';
|
||||
const next = section.indexOf('\n### ', at + 4);
|
||||
return section.slice(at, next === -1 ? undefined : next);
|
||||
}
|
||||
const on = (findings, file, line, rule_key) => findings.filter((f) => f.file === file && f.line === line && f.rule_key === rule_key);
|
||||
|
||||
/** A MINOR the reviewers raised: the coordinator's review holds only it, so the review is ALLOW. */
|
||||
const REVIEWER_MINOR = { severity: 'MINOR', file: 'src/b.mjs', line: 3, rule_key: 'MISSING_ERROR_HANDLING', title: 'unchecked return' };
|
||||
|
||||
// --- put back --------------------------------------------------------------------
|
||||
|
||||
test('[ENG] an engine MAJOR the coordinator\'s review.md does not hold is put back at its severity and the review is at least WARN', () => {
|
||||
const fx = engRepo();
|
||||
fx.write(engReview({ start: fx.start, end: fx.end, findings: [REVIEWER_MINOR] }));
|
||||
fx.engine('code-review', [{ file: 'src/a.mjs', line: 1, summary: 'The counter overflows past 2^31', failure_scenario: 'a long run wraps the count' }]);
|
||||
const r = fx.finalize();
|
||||
assert.equal(r.status, 0, r.stdout + r.stderr);
|
||||
const text = fx.read();
|
||||
const json = jsonOf(text);
|
||||
const back = on(json.findings, 'src/a.mjs', 1, 'CODE_REVIEW_DEFECT');
|
||||
assert.equal(back.length, 1, `the dropped engine MAJOR is back in the JSON findings: ${JSON.stringify(json.findings.map((f) => f.title))}`);
|
||||
assert.equal(back[0].severity, 'MAJOR', 'at its own severity');
|
||||
assert.equal(back[0].engine, 'code-review', 'engine is kept');
|
||||
assert.equal(back[0].title, 'The counter overflows past 2^31');
|
||||
assert.equal(json.verdict, 'WARN', 'the review is at least WARN');
|
||||
assert.equal(fmField(text, 'verdict'), 'WARN', 'the frontmatter verdict is raised with it');
|
||||
assert.deepEqual(json.counts, { BLOCKER: 0, MAJOR: 1, MINOR: 1, SUGGESTION: 0 }, 'counts hold the put-back');
|
||||
assert.deepEqual(fmIds(text), json.findings.map((f) => f.id), 'the frontmatter list is the JSON ids');
|
||||
const entry = entryOf(sectionOf(text, '## Findings (MAJOR)'), back[0].id);
|
||||
assert.ok(entry !== '', `a ### entry under ## Findings (MAJOR):\n${sectionOf(text, '## Findings (MAJOR)')}`);
|
||||
assert.ok(entry.includes(SOURCE_LINE('code-review')), `the entry names its source:\n${entry}`);
|
||||
// Control: the reviewer MINOR stays as it was.
|
||||
assert.equal(on(json.findings, 'src/b.mjs', 3, 'MISSING_ERROR_HANDLING').length, 1);
|
||||
});
|
||||
|
||||
test('[ENG] a dropped engine BLOCKER is put back and the verdict is BLOCK', () => {
|
||||
const fx = engRepo();
|
||||
fx.write(engReview({ start: fx.start, end: fx.end, findings: [REVIEWER_MINOR] }));
|
||||
fx.engine('security-review', [{ file: 'src/a.mjs', line: 1, summary: 'A shell command is built from request input', failure_scenario: 'a crafted name runs a command' }]);
|
||||
const r = fx.finalize();
|
||||
assert.equal(r.status, 0, r.stdout + r.stderr);
|
||||
const text = fx.read();
|
||||
const json = jsonOf(text);
|
||||
const back = on(json.findings, 'src/a.mjs', 1, 'SECURITY_REVIEW_VULNERABILITY');
|
||||
assert.equal(back.length, 1, `the dropped engine BLOCKER is back: ${JSON.stringify(json.findings.map((f) => f.title))}`);
|
||||
assert.equal(back[0].severity, 'BLOCKER');
|
||||
assert.equal(back[0].engine, 'security-review');
|
||||
assert.equal(json.verdict, 'BLOCK');
|
||||
assert.equal(fmField(text, 'verdict'), 'BLOCK');
|
||||
assert.ok(entryOf(sectionOf(text, '## Findings (BLOCKER)'), back[0].id).includes(SOURCE_LINE('security-review')),
|
||||
`a ### entry under ## Findings (BLOCKER) names its source:\n${sectionOf(text, '## Findings (BLOCKER)')}`);
|
||||
const gate = fx.gate();
|
||||
assert.equal(gate.status, 1, `the gate reads BLOCK: ${gate.stdout}${gate.stderr}`);
|
||||
});
|
||||
|
||||
// --- not duplicated --------------------------------------------------------------
|
||||
|
||||
test('[ENG] a finding returned under a trimmed title is not put back twice; an engine finding in out_of_scope is left to the existing declined rule', () => {
|
||||
const fx = engRepo();
|
||||
const long = `The retry loop never stops when the server answers 503 ${'and keeps the connection open '.repeat(4)}forever`;
|
||||
assert.ok(long.length > 100, 'the title is over the coordinator\'s 100-character limit');
|
||||
// The coordinator's trim (lib/review/coordinator-contract.mjs judgeFilter): cut to leave room for the marker, then the marker.
|
||||
const trimmed = `${long.slice(0, 100 - ' […trimmed]'.length)} […trimmed]`;
|
||||
const payload = fx.engine('code-review', [
|
||||
{ file: 'src/a.mjs', line: 1, summary: long, failure_scenario: 'a 503 hangs the run' },
|
||||
{ file: 'src/b.mjs', line: 3, summary: 'The parser drops the last line', failure_scenario: 'no newline at end of file' },
|
||||
{ file: 'lib/outside.mjs', line: 4, summary: 'An unchanged helper leaks a handle', failure_scenario: 'many calls exhaust handles' },
|
||||
]);
|
||||
assert.equal(payload.out_of_scope.length, 1, 'the CLI placed the lib/outside.mjs finding in out_of_scope');
|
||||
const out = payload.out_of_scope[0];
|
||||
fx.write(engReview({
|
||||
start: fx.start, end: fx.end,
|
||||
findings: [{ severity: 'MAJOR', file: 'src/a.mjs', line: 1, rule_key: 'CODE_REVIEW_DEFECT', engine: 'code-review', title: trimmed }],
|
||||
declined: [{ file: out.file, line: out.line, rule_key: out.rule_key, severity: out.severity, title: out.title, engine: 'code-review', reason: 'engine:out-of-scope' }],
|
||||
}));
|
||||
const r = fx.finalize();
|
||||
assert.equal(r.status, 0, r.stdout + r.stderr);
|
||||
const json = jsonOf(fx.read());
|
||||
const engineTitles = json.findings.filter((f) => f.engine === 'code-review').map((f) => f.title).sort();
|
||||
assert.deepEqual(engineTitles, ['The parser drops the last line', trimmed].sort(),
|
||||
'only the dropped finding is put back: the trimmed one is not put back twice');
|
||||
assert.equal(on(json.findings, 'src/a.mjs', 1, 'CODE_REVIEW_DEFECT').length, 1, 'one row on the trimmed finding\'s triplet');
|
||||
assert.equal(json.findings.filter((f) => f.file === 'lib/outside.mjs').length, 0, 'the out_of_scope finding is not put back');
|
||||
assert.deepEqual(json.declined_to_judge.map((d) => [d.file, d.reason]), [['lib/outside.mjs', 'engine:out-of-scope']],
|
||||
'it stays declined by the existing rule');
|
||||
});
|
||||
|
||||
// --- the lib row rule equals armB's ------------------------------------------------
|
||||
|
||||
test('[ENG] lib rowOf and trimmedTitle equal the armB workflow\'s on a table of titles', () => {
|
||||
assert.equal(typeof compareMod.rowOf, 'function', 'engine-compare.mjs exports rowOf');
|
||||
assert.equal(typeof compareMod.trimmedTitle, 'function', 'engine-compare.mjs exports trimmedTitle');
|
||||
// The armB definitions, read and evaluated as text (scripts/trekreview-armB.workflow.mjs).
|
||||
const src = readFileSync(ARMB, 'utf8');
|
||||
const from = src.indexOf('const tripletOf = ');
|
||||
const endMark = 'const trimmedTitle = ';
|
||||
const to = src.indexOf('\n', src.indexOf(endMark));
|
||||
assert.ok(from !== -1 && src.indexOf(endMark) > from && to > from, 'the armB workflow carries tripletOf .. trimmedTitle in one block');
|
||||
const armb = new Function(`${src.slice(from, to)}\nreturn { rowOf, trimmedTitle };`)();
|
||||
const titles = ['', 'x', ' Padded Title ', 'MiXeD Case', 'æøå … ünïcode', 'a'.repeat(99), 'b'.repeat(100), 'c'.repeat(101),
|
||||
'd'.repeat(250), `${'e'.repeat(95)} […trimmed]`, 'tab\there', 'line\nbreak'];
|
||||
for (const t of titles) {
|
||||
assert.equal(compareMod.trimmedTitle(t), armb.trimmedTitle(t), `trimmedTitle(${JSON.stringify(t.slice(0, 20))}…, ${t.length})`);
|
||||
for (const f of [
|
||||
{ file: 'src/a.mjs', line: 1, rule_key: 'CODE_REVIEW_DEFECT', title: t },
|
||||
{ file: 'src/a.mjs', line: '1', rule_key: 'SECURITY_REVIEW_VULNERABILITY', title: t },
|
||||
{ file: 'docs/x.md', line: 0, rule_key: undefined, title: t },
|
||||
]) {
|
||||
assert.equal(compareMod.rowOf(f), armb.rowOf(f), `rowOf on ${JSON.stringify(f).slice(0, 80)}`);
|
||||
}
|
||||
}
|
||||
const untitled = { file: 'src/a.mjs', line: 1, rule_key: 'CODE_REVIEW_DEFECT' };
|
||||
assert.equal(compareMod.rowOf(untitled), armb.rowOf(untitled), 'rowOf of an untitled finding');
|
||||
assert.notEqual(compareMod.rowOf(untitled), compareMod.rowOf({ ...untitled, title: '' }), 'an untitled finding gets a key no titled row has');
|
||||
});
|
||||
|
||||
// --- the gate reads the finalized file ------------------------------------------
|
||||
|
||||
test('[ENG] a review finalized with a put-back has every id computed, no placeholder left, and the gate reads it', () => {
|
||||
// A MAJOR: the engine's finding shares its triplet with a reviewer finding under another title, so it is a row of its own.
|
||||
const fx = engRepo();
|
||||
const reviewerMajor = { severity: 'MAJOR', file: 'src/a.mjs', line: 1, rule_key: 'CODE_REVIEW_DEFECT', title: 'The reviewer saw a wrong bound' };
|
||||
fx.write(engReview({ start: fx.start, end: fx.end, findings: [reviewerMajor, REVIEWER_MINOR] }));
|
||||
fx.engine('code-review', [{ file: 'src/a.mjs', line: 1, summary: 'The counter overflows past 2^31', failure_scenario: 'a long run wraps the count' }]);
|
||||
const r = fx.finalize();
|
||||
assert.equal(r.status, 0, r.stdout + r.stderr);
|
||||
const text = fx.read();
|
||||
const json = jsonOf(text);
|
||||
const back = json.findings.filter((f) => f.title === 'The counter overflows past 2^31');
|
||||
assert.equal(back.length, 1, `the engine MAJOR is put back: ${JSON.stringify(json.findings.map((f) => f.title))}`);
|
||||
// The second finding on one triplet gets the id over `rule_key#2`; the reviewer's keeps the plain one.
|
||||
assert.equal(back[0].id, computeFindingId('src/a.mjs', 1, 'CODE_REVIEW_DEFECT#2'), 'the put-back has its computed id');
|
||||
const kept = json.findings.find((f) => f.title === reviewerMajor.title);
|
||||
assert.equal(kept.id, computeFindingId('src/a.mjs', 1, 'CODE_REVIEW_DEFECT'), 'the reviewer finding keeps its id');
|
||||
assert.doesNotMatch(text, /@@ID_\d+@@/, 'no placeholder is left');
|
||||
assert.deepEqual(json.findings.map((f) => f.id), computeFindingIds(json.findings), 'every JSON id is the computed one');
|
||||
for (const f of json.findings) {
|
||||
assert.equal(text.split('\n').filter((l) => l === `### ${f.id}`).length, 1, `one ### entry for ${f.title}`);
|
||||
}
|
||||
assert.ok(entryOf(sectionOf(text, '## Findings (MAJOR)'), kept.id).includes(reviewerMajor.title), 'the reviewer entry keeps its own text');
|
||||
assert.deepEqual(fmIds(text), json.findings.map((f) => f.id), 'the frontmatter list is the JSON ids');
|
||||
const fill = fx.fill();
|
||||
assert.equal(fill.status, 0, `fill-finding-ids finds nothing to fill: ${fill.stdout}${fill.stderr}`);
|
||||
const v = fx.validate();
|
||||
assert.equal(v.status, 0, `review-validator --strict: ${v.stdout}${v.stderr}`);
|
||||
// The gate passes its frontmatter-vs-JSON id check and reaches the answers: the put-back MAJOR is unanswered.
|
||||
const gate = fx.gate();
|
||||
const verdict = parse(gate);
|
||||
assert.equal(gate.status, 2, gate.stdout + gate.stderr);
|
||||
assert.match(String(verdict.reason), /^review-unanswered:/, `the gate reads the file up to the answers: ${gate.stdout}`);
|
||||
assert.ok(Array.isArray(verdict.unanswered) && verdict.unanswered.includes(back[0].id), `the put-back MAJOR counts: ${gate.stdout}`);
|
||||
|
||||
// A BLOCKER: the put-back gets its ### Step R<n>, and the strict validator finds it.
|
||||
const bx = engRepo();
|
||||
bx.write(engReview({ start: bx.start, end: bx.end, findings: [REVIEWER_MINOR] }));
|
||||
bx.engine('security-review', [{ file: 'src/a.mjs', line: 1, summary: 'A shell command is built from request input', failure_scenario: 'a crafted name runs a command' }]);
|
||||
const rb = bx.finalize();
|
||||
assert.equal(rb.status, 0, rb.stdout + rb.stderr);
|
||||
const btext = bx.read();
|
||||
const bjson = jsonOf(btext);
|
||||
const blocker = bjson.findings.find((f) => f.severity === 'BLOCKER');
|
||||
assert.ok(blocker, `the engine BLOCKER is put back: ${JSON.stringify(bjson.findings.map((f) => f.title))}`);
|
||||
assert.equal(blocker.id, computeFindingId('src/a.mjs', 1, 'SECURITY_REVIEW_VULNERABILITY'), 'it has its computed id');
|
||||
assert.doesNotMatch(btext, /@@ID_\d+@@/, 'no placeholder is left');
|
||||
const steps = sectionOf(btext, '## Remediation Steps');
|
||||
assert.match(steps, /^### Step R1\b/m, `a ### Step R<n> for the put-back BLOCKER:\n${steps}`);
|
||||
assert.ok(steps.includes(`**Source finding:** \`${blocker.id}\``), `the step names the put-back:\n${steps}`);
|
||||
assert.deepEqual(fmIds(btext), bjson.findings.map((f) => f.id), 'the frontmatter list is the JSON ids');
|
||||
const bfill = bx.fill();
|
||||
assert.equal(bfill.status, 0, `fill-finding-ids finds nothing to fill: ${bfill.stdout}${bfill.stderr}`);
|
||||
const bv = bx.validate();
|
||||
assert.equal(bv.status, 0, `review-validator --strict finds the R-step: ${bv.stdout}${bv.stderr}`);
|
||||
const bgate = bx.gate();
|
||||
assert.equal(bgate.status, 1, `the gate reads BLOCK: ${bgate.stdout}${bgate.stderr}`);
|
||||
});
|
||||
Loading…
Add table
Add a link
Reference in a new issue