voyage/docs/eval-corpus/README.md

Eval corpus — frozen failures for Voyage's own agents

This directory is the home for the golden / frozen-failure corpus that grounds Voyage's self-evaluation (SKAL-1·4a, the eval-foundation tier). It follows the Anthropic "collect 20–50 real cases" practice: every time a Voyage review/coordinator agent misfires on a real task, the case is distilled into a machine-readable record and added here, so the failure can never silently regress.

This corpus is the gold that the offline gold-scored output eval (SKAL-1·4b) scores against — that eval is now implemented and wired into node --test (see §Gold-scored output eval below). The corpus records here are committed fixtures + a schema; the scoring run is the separate piece that consumes them.

Seed example

The first corpus entry is tests/fixtures/bakeoff-rich/gold.json — the 5 brief-traceable seeded findings of the bakeoff-rich JWT-auth fixture. gold.json is the canonical machine-readable form; the prose table in tests/fixtures/bakeoff-rich/README.md is illustrative. When they disagree, gold.json wins.

Record schema (voyage-eval-gold/1)

A corpus file is one JSON object:

{
  "schema": "voyage-eval-gold/1",
  "source": "<path to the prose/fixture this was distilled from>",
  "description": "<one line>",
  "expected_verdict": "BLOCK | WARN | ALLOW",   // review-coordinator Pass-4 outcome
  "findings": [
    {
      "file": "<repo-relative path in the reviewed diff>",
      "line": 0,                                  // integer >= 0; 0 = file-scoped
      "rule_key": "<member of lib/review/rule-catalogue.mjs RULE_CATALOGUE>",
      "severity": "BLOCKER | MAJOR | MINOR | SUGGESTION",
      "owner_reviewer": "conformance | correctness"
      // optional eval-extension fields are permitted, e.g.:
      // "dual_flaggable": "<a second rule_key the same issue could carry>"
    }
  ]
}

Hard constraints

• rule_key must be a member of RULE_CATALOGUE (lib/review/rule-catalogue.mjs). The catalogue is the contract; an invented rule_key is a corpus bug.
• severity must be one of SEVERITY_VALUES (BLOCKER, MAJOR, MINOR, SUGGESTION).
• expected_verdict is the deterministic Pass-4 outcome of lib/review/coordinator-contract.mjs::computeVerdict: BLOCKER ≥ 1 → BLOCK, else MAJOR ≥ 1 → WARN, else ALLOW.
• Finding-level fields (file, line, rule_key, severity) mirror FINDING_REQUIRED_FIELDS (lib/review/findings-schema.mjs); owner_reviewer and any extension fields are eval-specific additions (superset).

Adding a new frozen failure

1. Distil the misfire into a voyage-eval-gold/1 record (one .json file here, or a new entry in an existing corpus file).
2. Confirm every rule_key is in the catalogue and expected_verdict matches the Pass-4 computation.
3. Add (or extend) a loader test in the tests/lib/gold-corpus.test.mjs shape so the record's shape + catalogue membership are pinned under node --test.

Gold-scored output eval (SKAL-1·4b)

The offline scoring run that grades a recorded agent run against this corpus. Offline = committed reviewer payloads, no live agent spawn, no LLM, no network (the LLM-in-the-loop grading is the separate 4c tier).

• Committed runs live under tests/fixtures/bakeoff-rich/runs/. Each file is the JSON reviewer payloads (one object per reviewer, { reviewer, findings }) that a recorded run produced. run-perfect.json is the regression guard: fed through the coordinator contract it must reproduce every seeded gold finding.
• The contract lib/review/coordinator-contract.mjs::runContract(payloads) turns those payloads into the deterministic coordinator output (4a).
• The scorer lib/review/gold-scorer.mjs:
  • scoreFindings(runFindings, goldFindings) matches at (file, rule_key) granularity (line + severity ignored) → { tp, fp, fn, precision, recall, f1, matched, missed, spurious }. Vacuous-set conventions (empty run → recall 0; empty gold → precision 0; f1 collapses to 0) are documented in the module header.
  • scoreVerdict(runVerdict, goldVerdict) → exact verdict match.
• The scoring run tests/lib/gold-eval.test.mjs asserts run-perfect reproduces gold at precision/recall/f1 = 1.0 and verdict === expected_verdict.
• Third test-census category. lib/util/test-census.mjs now reports a goldEval bucket (matched by GOLD_EVAL_FILE_RE) separately from behavior and docPins — a scoring run is neither behavior coverage nor a prose pin, so the honest-count invariant is now a 3-way sum.

Future hardening (not in this tier)

• A prose↔JSON cross-assertion (parse the fixture README table, diff against gold.json) to mechanically bound the two-source-of-truth drift.
• Degraded-run fixtures (a run that misses or invents findings) to exercise the scorer's discriminating path end-to-end; today that path is covered by tests/lib/gold-scorer.test.mjs with inline synthetic findings.
• SKAL-1·4c: the LLM-in-the-loop eval that grades live agent runs (needs a filesystem + model judgement, deliberately excluded from this deterministic tier).