74 lines
3.3 KiB
Markdown
74 lines
3.3 KiB
Markdown
# Eval corpus — frozen failures for Voyage's own agents
|
||
|
||
This directory is the home for the **golden / frozen-failure corpus** that
|
||
grounds Voyage's self-evaluation (SKAL-1·4a, the eval-foundation tier). It
|
||
follows the Anthropic "collect 20–50 real cases" practice: every time a Voyage
|
||
review/coordinator agent **misfires** on a real task, the case is distilled into
|
||
a machine-readable record and added here, so the failure can never silently
|
||
regress.
|
||
|
||
This tier is the **staging ground** for the offline gold-scored output eval
|
||
(SKAL-1·4b). It is deliberately **not wired into CI** yet — these are committed
|
||
fixtures + a schema, not a scoring run.
|
||
|
||
## Seed example
|
||
|
||
The first corpus entry is
|
||
[`tests/fixtures/bakeoff-rich/gold.json`](../../tests/fixtures/bakeoff-rich/gold.json)
|
||
— the 5 brief-traceable seeded findings of the bakeoff-rich JWT-auth fixture.
|
||
`gold.json` is the **canonical machine-readable form**; the prose table in
|
||
`tests/fixtures/bakeoff-rich/README.md` is illustrative. When they disagree,
|
||
`gold.json` wins.
|
||
|
||
## Record schema (`voyage-eval-gold/1`)
|
||
|
||
A corpus file is one JSON object:
|
||
|
||
```jsonc
|
||
{
|
||
"schema": "voyage-eval-gold/1",
|
||
"source": "<path to the prose/fixture this was distilled from>",
|
||
"description": "<one line>",
|
||
"expected_verdict": "BLOCK | WARN | ALLOW", // review-coordinator Pass-4 outcome
|
||
"findings": [
|
||
{
|
||
"file": "<repo-relative path in the reviewed diff>",
|
||
"line": 0, // integer >= 0; 0 = file-scoped
|
||
"rule_key": "<member of lib/review/rule-catalogue.mjs RULE_CATALOGUE>",
|
||
"severity": "BLOCKER | MAJOR | MINOR | SUGGESTION",
|
||
"owner_reviewer": "conformance | correctness"
|
||
// optional eval-extension fields are permitted, e.g.:
|
||
// "dual_flaggable": "<a second rule_key the same issue could carry>"
|
||
}
|
||
]
|
||
}
|
||
```
|
||
|
||
### Hard constraints
|
||
|
||
- **`rule_key` must be a member of `RULE_CATALOGUE`** (`lib/review/rule-catalogue.mjs`).
|
||
The catalogue is the contract; an invented rule_key is a corpus bug.
|
||
- **`severity` must be one of `SEVERITY_VALUES`** (`BLOCKER`, `MAJOR`, `MINOR`, `SUGGESTION`).
|
||
- **`expected_verdict`** is the deterministic Pass-4 outcome of
|
||
`lib/review/coordinator-contract.mjs::computeVerdict`: `BLOCKER ≥ 1 → BLOCK`,
|
||
else `MAJOR ≥ 1 → WARN`, else `ALLOW`.
|
||
- Finding-level fields (`file`, `line`, `rule_key`, `severity`) mirror
|
||
`FINDING_REQUIRED_FIELDS` (`lib/review/findings-schema.mjs`); `owner_reviewer`
|
||
and any extension fields are eval-specific additions (superset).
|
||
|
||
## Adding a new frozen failure
|
||
|
||
1. Distil the misfire into a `voyage-eval-gold/1` record (one `.json` file here,
|
||
or a new entry in an existing corpus file).
|
||
2. Confirm every `rule_key` is in the catalogue and `expected_verdict` matches
|
||
the Pass-4 computation.
|
||
3. Add (or extend) a loader test in the `tests/lib/gold-corpus.test.mjs` shape so
|
||
the record's shape + catalogue membership are pinned under `node --test`.
|
||
|
||
## Future hardening (not in this tier)
|
||
|
||
- SKAL-1·4b: an offline gold-scored output eval that scores recorded agent runs
|
||
against these records at `(file, rule_key)` granularity, with committed runs
|
||
and a third test-census category.
|
||
- A prose↔JSON cross-assertion (parse the fixture README table, diff against
|
||
`gold.json`) to mechanically bound the two-source-of-truth drift.
|