Kjell Tore Guttormsen
440594f1b2
Scores committed agent-run fixtures against the golden corpus at (file, rule_key) granularity, building on the deterministic coordinator contract (4a). Offline: committed reviewer payloads, no live agent spawn, no LLM, no network (the LLM-in-the-loop grading is the separate 4c tier). - lib/review/gold-scorer.mjs: scoreFindings (precision/recall/f1 at (file,rule_key) granularity, line+severity ignored) + scoreVerdict; pure, with documented vacuous-set conventions. - tests/fixtures/bakeoff-rich/runs/run-perfect.json: committed run that reproduces all 5 seeded gold findings through runContract. - tests/lib/gold-eval.test.mjs: the scoring RUN (precision/recall/f1 = 1.0, verdict == expected_verdict BLOCK, nothing suppressed/skipped). - lib/util/test-census.mjs: third census category (goldEval) — a scoring run is neither behavior coverage nor a doc-pin; honest-count invariant now 3-way. - docs/eval-corpus/README.md: 4b moved from Future hardening to implemented. Suite 809 -> 822 (820/0/2). gold-scorer covers TP+FP+FN+degenerate paths. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BJQYC5vpkJWxndS55vQQZ6
17 lines
709 B
JSON
17 lines
709 B
JSON
[
|
|
{
|
|
"reviewer": "brief-conformance-reviewer",
|
|
"findings": [
|
|
{ "severity": "BLOCKER", "rule_key": "UNIMPLEMENTED_CRITERION", "file": "lib/handlers/login.mjs", "line": 17 },
|
|
{ "severity": "MAJOR", "rule_key": "PLAN_EXECUTE_DRIFT", "file": "lib/handlers/login.mjs", "line": 13 }
|
|
]
|
|
},
|
|
{
|
|
"reviewer": "code-correctness-reviewer",
|
|
"findings": [
|
|
{ "severity": "BLOCKER", "rule_key": "SECURITY_INJECTION", "file": "lib/auth/jwt.mjs", "line": 19 },
|
|
{ "severity": "MAJOR", "rule_key": "MISSING_TEST", "file": "lib/auth/refresh.mjs", "line": 0 },
|
|
{ "severity": "MINOR", "rule_key": "MISSING_ERROR_HANDLING", "file": "lib/auth/refresh.mjs", "line": 10 }
|
|
]
|
|
}
|
|
]
|