voyage/docs/eval-corpus/README.md

# Eval corpus — frozen failures for Voyage's own agents

This directory is the home for the **golden / frozen-failure corpus** that
grounds Voyage's self-evaluation (SKAL-1·4a, the eval-foundation tier). It
follows the Anthropic "collect 20–50 real cases" practice: every time a Voyage
review/coordinator agent **misfires** on a real task, the case is distilled into
a machine-readable record and added here, so the failure can never silently
regress.

This corpus is the gold that the **offline gold-scored output eval (SKAL-1·4b)**
scores against — that eval is now implemented and wired into `node --test`
(see [§Gold-scored output eval](#gold-scored-output-eval-skal-14b) below). The
corpus records here are committed fixtures + a schema; the scoring run is the
separate piece that consumes them.

## Seed example

The first corpus entry is
[`tests/fixtures/bakeoff-rich/gold.json`](../../tests/fixtures/bakeoff-rich/gold.json)
— the 5 brief-traceable seeded findings of the bakeoff-rich JWT-auth fixture.
`gold.json` is the **canonical machine-readable form**; the prose table in
`tests/fixtures/bakeoff-rich/README.md` is illustrative. When they disagree,
`gold.json` wins.

## Record schema (`voyage-eval-gold/1`)

A corpus file is one JSON object:

```jsonc
{
  "schema": "voyage-eval-gold/1",
  "source": "<path to the prose/fixture this was distilled from>",
  "description": "<one line>",
  "expected_verdict": "BLOCK | WARN | ALLOW",   // review-coordinator Pass-4 outcome
  "findings": [
    {
      "file": "<repo-relative path in the reviewed diff>",
      "line": 0,                                  // integer >= 0; 0 = file-scoped
      "rule_key": "<member of lib/review/rule-catalogue.mjs RULE_CATALOGUE>",
      "severity": "BLOCKER | MAJOR | MINOR | SUGGESTION",
      "owner_reviewer": "conformance | correctness"
      // optional eval-extension fields are permitted, e.g.:
      // "dual_flaggable": "<a second rule_key the same issue could carry>"
    }
  ]
}
```

### Hard constraints

- **`rule_key` must be a member of `RULE_CATALOGUE`** (`lib/review/rule-catalogue.mjs`).
  The catalogue is the contract; an invented rule_key is a corpus bug.
- **`severity` must be one of `SEVERITY_VALUES`** (`BLOCKER`, `MAJOR`, `MINOR`, `SUGGESTION`).
- **`expected_verdict`** is the deterministic Pass-4 outcome of
  `lib/review/coordinator-contract.mjs::computeVerdict`: `BLOCKER ≥ 1 → BLOCK`,
  else `MAJOR ≥ 1 → WARN`, else `ALLOW`.
- Finding-level fields (`file`, `line`, `rule_key`, `severity`) mirror
  `FINDING_REQUIRED_FIELDS` (`lib/review/findings-schema.mjs`); `owner_reviewer`
  and any extension fields are eval-specific additions (superset).

## Adding a new frozen failure

1. Distil the misfire into a `voyage-eval-gold/1` record (one `.json` file here,
   or a new entry in an existing corpus file).
2. Confirm every `rule_key` is in the catalogue and `expected_verdict` matches
   the Pass-4 computation.
3. Add (or extend) a loader test in the `tests/lib/gold-corpus.test.mjs` shape so
   the record's shape + catalogue membership are pinned under `node --test`.

## Gold-scored output eval (SKAL-1·4b)

The offline scoring run that grades a recorded agent run against this corpus.
**Offline** = committed reviewer payloads, no live agent spawn, no LLM, no
network (the LLM-in-the-loop grading is the separate 4c tier).

- **Committed runs** live under `tests/fixtures/bakeoff-rich/runs/`. Each file is
  the JSON **reviewer payloads** (one object per reviewer, `{ reviewer, findings }`)
  that a recorded run produced. `run-perfect.json` is the regression guard: fed
  through the coordinator contract it must reproduce every seeded gold finding.
- **The contract** `lib/review/coordinator-contract.mjs::runContract(payloads)`
  turns those payloads into the deterministic coordinator output (4a).
- **The scorer** `lib/review/gold-scorer.mjs`:
  - `scoreFindings(runFindings, goldFindings)` matches at **`(file, rule_key)`
    granularity** (line + severity ignored) → `{ tp, fp, fn, precision, recall,
    f1, matched, missed, spurious }`. Vacuous-set conventions (empty run →
    recall 0; empty gold → precision 0; f1 collapses to 0) are documented in the
    module header.
  - `scoreVerdict(runVerdict, goldVerdict)` → exact verdict match.
- **The scoring run** `tests/lib/gold-eval.test.mjs` asserts `run-perfect`
  reproduces gold at precision/recall/f1 = 1.0 and `verdict === expected_verdict`.
- **Third test-census category.** `lib/util/test-census.mjs` now reports a
  `goldEval` bucket (matched by `GOLD_EVAL_FILE_RE`) separately from `behavior`
  and `docPins` — a scoring run is neither behavior coverage nor a prose pin, so
  the honest-count invariant is now a 3-way sum.

## Future hardening (not in this tier)

- A prose↔JSON cross-assertion (parse the fixture README table, diff against
  `gold.json`) to mechanically bound the two-source-of-truth drift.
- Degraded-run fixtures (a run that misses or invents findings) to exercise the
  scorer's discriminating path end-to-end; today that path is covered by
  `tests/lib/gold-scorer.test.mjs` with inline synthetic findings.
- SKAL-1·4c: the LLM-in-the-loop eval that grades live agent runs (needs a
  filesystem + model judgement, deliberately excluded from this deterministic tier).