feat(eval): SKAL-1·4b offline gold-scored output eval
Scores committed agent-run fixtures against the golden corpus at (file, rule_key) granularity, building on the deterministic coordinator contract (4a). Offline: committed reviewer payloads, no live agent spawn, no LLM, no network (the LLM-in-the-loop grading is the separate 4c tier). - lib/review/gold-scorer.mjs: scoreFindings (precision/recall/f1 at (file,rule_key) granularity, line+severity ignored) + scoreVerdict; pure, with documented vacuous-set conventions. - tests/fixtures/bakeoff-rich/runs/run-perfect.json: committed run that reproduces all 5 seeded gold findings through runContract. - tests/lib/gold-eval.test.mjs: the scoring RUN (precision/recall/f1 = 1.0, verdict == expected_verdict BLOCK, nothing suppressed/skipped). - lib/util/test-census.mjs: third census category (goldEval) — a scoring run is neither behavior coverage nor a doc-pin; honest-count invariant now 3-way. - docs/eval-corpus/README.md: 4b moved from Future hardening to implemented. Suite 809 -> 822 (820/0/2). gold-scorer covers TP+FP+FN+degenerate paths. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BJQYC5vpkJWxndS55vQQZ6
This commit is contained in:
parent
da418e653d
commit
440594f1b2
7 changed files with 255 additions and 19 deletions
|
|
@ -7,9 +7,11 @@ review/coordinator agent **misfires** on a real task, the case is distilled into
|
|||
a machine-readable record and added here, so the failure can never silently
|
||||
regress.
|
||||
|
||||
This tier is the **staging ground** for the offline gold-scored output eval
|
||||
(SKAL-1·4b). It is deliberately **not wired into CI** yet — these are committed
|
||||
fixtures + a schema, not a scoring run.
|
||||
This corpus is the gold that the **offline gold-scored output eval (SKAL-1·4b)**
|
||||
scores against — that eval is now implemented and wired into `node --test`
|
||||
(see [§Gold-scored output eval](#gold-scored-output-eval-skal-14b) below). The
|
||||
corpus records here are committed fixtures + a schema; the scoring run is the
|
||||
separate piece that consumes them.
|
||||
|
||||
## Seed example
|
||||
|
||||
|
|
@ -65,10 +67,38 @@ A corpus file is one JSON object:
|
|||
3. Add (or extend) a loader test in the `tests/lib/gold-corpus.test.mjs` shape so
|
||||
the record's shape + catalogue membership are pinned under `node --test`.
|
||||
|
||||
## Gold-scored output eval (SKAL-1·4b)
|
||||
|
||||
The offline scoring run that grades a recorded agent run against this corpus.
|
||||
**Offline** = committed reviewer payloads, no live agent spawn, no LLM, no
|
||||
network (the LLM-in-the-loop grading is the separate 4c tier).
|
||||
|
||||
- **Committed runs** live under `tests/fixtures/bakeoff-rich/runs/`. Each file is
|
||||
the JSON **reviewer payloads** (one object per reviewer, `{ reviewer, findings }`)
|
||||
that a recorded run produced. `run-perfect.json` is the regression guard: fed
|
||||
through the coordinator contract it must reproduce every seeded gold finding.
|
||||
- **The contract** `lib/review/coordinator-contract.mjs::runContract(payloads)`
|
||||
turns those payloads into the deterministic coordinator output (4a).
|
||||
- **The scorer** `lib/review/gold-scorer.mjs`:
|
||||
- `scoreFindings(runFindings, goldFindings)` matches at **`(file, rule_key)`
|
||||
granularity** (line + severity ignored) → `{ tp, fp, fn, precision, recall,
|
||||
f1, matched, missed, spurious }`. Vacuous-set conventions (empty run →
|
||||
recall 0; empty gold → precision 0; f1 collapses to 0) are documented in the
|
||||
module header.
|
||||
- `scoreVerdict(runVerdict, goldVerdict)` → exact verdict match.
|
||||
- **The scoring run** `tests/lib/gold-eval.test.mjs` asserts `run-perfect`
|
||||
reproduces gold at precision/recall/f1 = 1.0 and `verdict === expected_verdict`.
|
||||
- **Third test-census category.** `lib/util/test-census.mjs` now reports a
|
||||
`goldEval` bucket (matched by `GOLD_EVAL_FILE_RE`) separately from `behavior`
|
||||
and `docPins` — a scoring run is neither behavior coverage nor a prose pin, so
|
||||
the honest-count invariant is now a 3-way sum.
|
||||
|
||||
## Future hardening (not in this tier)
|
||||
|
||||
- SKAL-1·4b: an offline gold-scored output eval that scores recorded agent runs
|
||||
against these records at `(file, rule_key)` granularity, with committed runs
|
||||
and a third test-census category.
|
||||
- A prose↔JSON cross-assertion (parse the fixture README table, diff against
|
||||
`gold.json`) to mechanically bound the two-source-of-truth drift.
|
||||
- Degraded-run fixtures (a run that misses or invents findings) to exercise the
|
||||
scorer's discriminating path end-to-end; today that path is covered by
|
||||
`tests/lib/gold-scorer.test.mjs` with inline synthetic findings.
|
||||
- SKAL-1·4c: the LLM-in-the-loop eval that grades live agent runs (needs a
|
||||
filesystem + model judgement, deliberately excluded from this deterministic tier).
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue