# Eval corpus — frozen failures for Voyage's own agents

This directory is the home for the **golden / frozen-failure corpus** that
grounds Voyage's self-evaluation (SKAL-1·4a, the eval-foundation tier). It
follows the Anthropic "collect 20–50 real cases" practice: every time a Voyage
review/coordinator agent **misfires** on a real task, the case is distilled into
a machine-readable record and added here, so the failure can never silently
regress.

This tier is the **staging ground** for the offline gold-scored output eval
(SKAL-1·4b). It is deliberately **not wired into CI** yet — these are committed
fixtures + a schema, not a scoring run.

## Seed example

The first corpus entry is
[`tests/fixtures/bakeoff-rich/gold.json`](../../tests/fixtures/bakeoff-rich/gold.json)
— the 5 brief-traceable seeded findings of the bakeoff-rich JWT-auth fixture.
`gold.json` is the **canonical machine-readable form**; the prose table in
`tests/fixtures/bakeoff-rich/README.md` is illustrative. When they disagree,
`gold.json` wins.

## Record schema (`voyage-eval-gold/1`)

A corpus file is one JSON object:

```jsonc
{
  "schema": "voyage-eval-gold/1",
  "source": "<path to the prose/fixture this was distilled from>",
  "description": "<one line>",
  "expected_verdict": "BLOCK | WARN | ALLOW",   // review-coordinator Pass-4 outcome
  "findings": [
    {
      "file": "<repo-relative path in the reviewed diff>",
      "line": 0,                                  // integer >= 0; 0 = file-scoped
      "rule_key": "<member of lib/review/rule-catalogue.mjs RULE_CATALOGUE>",
      "severity": "BLOCKER | MAJOR | MINOR | SUGGESTION",
      "owner_reviewer": "conformance | correctness"
      // optional eval-extension fields are permitted, e.g.:
      // "dual_flaggable": "<a second rule_key the same issue could carry>"
    }
  ]
}
```

### Hard constraints

- **`rule_key` must be a member of `RULE_CATALOGUE`** (`lib/review/rule-catalogue.mjs`).
  The catalogue is the contract; an invented rule_key is a corpus bug.
- **`severity` must be one of `SEVERITY_VALUES`** (`BLOCKER`, `MAJOR`, `MINOR`, `SUGGESTION`).
- **`expected_verdict`** is the deterministic Pass-4 outcome of
  `lib/review/coordinator-contract.mjs::computeVerdict`: `BLOCKER ≥ 1 → BLOCK`,
  else `MAJOR ≥ 1 → WARN`, else `ALLOW`.
- Finding-level fields (`file`, `line`, `rule_key`, `severity`) mirror
  `FINDING_REQUIRED_FIELDS` (`lib/review/findings-schema.mjs`); `owner_reviewer`
  and any extension fields are eval-specific additions (superset).

## Adding a new frozen failure

1. Distil the misfire into a `voyage-eval-gold/1` record (one `.json` file here,
   or a new entry in an existing corpus file).
2. Confirm every `rule_key` is in the catalogue and `expected_verdict` matches
   the Pass-4 computation.
3. Add (or extend) a loader test in the `tests/lib/gold-corpus.test.mjs` shape so
   the record's shape + catalogue membership are pinned under `node --test`.

## Future hardening (not in this tier)

- SKAL-1·4b: an offline gold-scored output eval that scores recorded agent runs
  against these records at `(file, rule_key)` granularity, with committed runs
  and a third test-census category.
- A prose↔JSON cross-assertion (parse the fixture README table, diff against
  `gold.json`) to mechanically bound the two-source-of-truth drift.