Scores committed agent-run fixtures against the golden corpus at
(file, rule_key) granularity, building on the deterministic coordinator
contract (4a). Offline: committed reviewer payloads, no live agent spawn,
no LLM, no network (the LLM-in-the-loop grading is the separate 4c tier).
- lib/review/gold-scorer.mjs: scoreFindings (precision/recall/f1 at
(file,rule_key) granularity, line+severity ignored) + scoreVerdict; pure,
with documented vacuous-set conventions.
- tests/fixtures/bakeoff-rich/runs/run-perfect.json: committed run that
reproduces all 5 seeded gold findings through runContract.
- tests/lib/gold-eval.test.mjs: the scoring RUN (precision/recall/f1 = 1.0,
verdict == expected_verdict BLOCK, nothing suppressed/skipped).
- lib/util/test-census.mjs: third census category (goldEval) — a scoring run
is neither behavior coverage nor a doc-pin; honest-count invariant now 3-way.
- docs/eval-corpus/README.md: 4b moved from Future hardening to implemented.
Suite 809 -> 822 (820/0/2). gold-scorer covers TP+FP+FN+degenerate paths.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BJQYC5vpkJWxndS55vQQZ6
Closes devils-advocate audit §Top changes #8 (MINOR×MED). Reports the
behavior-test count separately from the doc-consistency prose-pin count so the
cited total no longer oversells behavior coverage, and removes the clearest
reword-fragile redundancies. Conservative scope (operator-chosen): no genuine
regression guard dropped.
1. Suite census (report separately). lib/util/test-census.mjs +
tests/lib/test-census.test.mjs walk tests/**/*.test.mjs and bucket top-level
test() declarations into behavior vs doc-consistency-pins (bucket per file,
regex PIN_FILE_RE), asserting the two sum to the total so neither can drift
silently. Split emitted as a t.diagnostic: behavior=601 doc-consistency-pins=69.
Metric = top-level declarations; node:test's runtime total counts subtests
too and is therefore >= it.
2. Conservative prune (3 tests, each provably subsumed):
- "trekexecute.md still parses v1.7 plan schema" — tautological OR-chain;
real coverage = the plan_version:1.7 template pin + plan-validator /
plan-schema behavior tests.
- "CLAUDE.md mentions all six pipeline commands" — hardcoded six-string list
subsumed by the filesystem-driven "commands table mentions every
commands/*.md file" structural pin.
- "CLAUDE.md mentions /trekcontinue command" — same subsumption.
Each removal leaves an in-place note (why + where coverage lives).
3. doc-consistency.test.mjs header now documents the structural-invariant vs
prose/existence-pin distinction so new pins land in the right kind.
TDD: census test written failing-first (ERR_MODULE_NOT_FOUND before
lib/util/test-census.mjs existed). Suite 699->698 (696 pass / 2 skip / 0 fail):
-3 prune +2 census. plugin validate passes (1 accepted warning).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqBYc8Ltrk7LipyJmGxXiB
