voyage/docs/eval-corpus

Eval corpus — frozen failures for Voyage's own agents

This directory is the home for the golden / frozen-failure corpus that grounds Voyage's self-evaluation (SKAL-1·4a, the eval-foundation tier). It follows the Anthropic "collect 20–50 real cases" practice: every time a Voyage review/coordinator agent misfires on a real task, the case is distilled into a machine-readable record and added here, so the failure can never silently regress.

This tier is the staging ground for the offline gold-scored output eval (SKAL-1·4b). It is deliberately not wired into CI yet — these are committed fixtures + a schema, not a scoring run.

Seed example

The first corpus entry is tests/fixtures/bakeoff-rich/gold.json — the 5 brief-traceable seeded findings of the bakeoff-rich JWT-auth fixture. gold.json is the canonical machine-readable form; the prose table in tests/fixtures/bakeoff-rich/README.md is illustrative. When they disagree, gold.json wins.

Record schema (voyage-eval-gold/1)

A corpus file is one JSON object:

{
  "schema": "voyage-eval-gold/1",
  "source": "<path to the prose/fixture this was distilled from>",
  "description": "<one line>",
  "expected_verdict": "BLOCK | WARN | ALLOW",   // review-coordinator Pass-4 outcome
  "findings": [
    {
      "file": "<repo-relative path in the reviewed diff>",
      "line": 0,                                  // integer >= 0; 0 = file-scoped
      "rule_key": "<member of lib/review/rule-catalogue.mjs RULE_CATALOGUE>",
      "severity": "BLOCKER | MAJOR | MINOR | SUGGESTION",
      "owner_reviewer": "conformance | correctness"
      // optional eval-extension fields are permitted, e.g.:
      // "dual_flaggable": "<a second rule_key the same issue could carry>"
    }
  ]
}

Hard constraints

• rule_key must be a member of RULE_CATALOGUE (lib/review/rule-catalogue.mjs). The catalogue is the contract; an invented rule_key is a corpus bug.
• severity must be one of SEVERITY_VALUES (BLOCKER, MAJOR, MINOR, SUGGESTION).
• expected_verdict is the deterministic Pass-4 outcome of lib/review/coordinator-contract.mjs::computeVerdict: BLOCKER ≥ 1 → BLOCK, else MAJOR ≥ 1 → WARN, else ALLOW.
• Finding-level fields (file, line, rule_key, severity) mirror FINDING_REQUIRED_FIELDS (lib/review/findings-schema.mjs); owner_reviewer and any extension fields are eval-specific additions (superset).

Adding a new frozen failure

1. Distil the misfire into a voyage-eval-gold/1 record (one .json file here, or a new entry in an existing corpus file).
2. Confirm every rule_key is in the catalogue and expected_verdict matches the Pass-4 computation.
3. Add (or extend) a loader test in the tests/lib/gold-corpus.test.mjs shape so the record's shape + catalogue membership are pinned under node --test.

Future hardening (not in this tier)

• SKAL-1·4b: an offline gold-scored output eval that scores recorded agent runs against these records at (file, rule_key) granularity, with committed runs and a third test-census category.
• A prose↔JSON cross-assertion (parse the fixture README table, diff against gold.json) to mechanically bound the two-source-of-truth drift.