From 5144e53129ff045be8ea511dd5845b2f3d746764 Mon Sep 17 00:00:00 2001
From: Kjell Tore Guttormsen <hello@fromaitochitta.com>
Date: Fri, 26 Jun 2026 11:57:39 +0200
Subject: [PATCH] docs(eval): establish eval-corpus frozen-failure home

---
 docs/eval-corpus/README.md | 74 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 74 insertions(+)
 create mode 100644 docs/eval-corpus/README.md

diff --git a/docs/eval-corpus/README.md b/docs/eval-corpus/README.md
new file mode 100644
index 0000000..8f9d675
--- /dev/null
+++ b/docs/eval-corpus/README.md
@@ -0,0 +1,74 @@
+# Eval corpus — frozen failures for Voyage's own agents
+
+This directory is the home for the **golden / frozen-failure corpus** that
+grounds Voyage's self-evaluation (SKAL-1·4a, the eval-foundation tier). It
+follows the Anthropic "collect 20–50 real cases" practice: every time a Voyage
+review/coordinator agent **misfires** on a real task, the case is distilled into
+a machine-readable record and added here, so the failure can never silently
+regress.
+
+This tier is the **staging ground** for the offline gold-scored output eval
+(SKAL-1·4b). It is deliberately **not wired into CI** yet — these are committed
+fixtures + a schema, not a scoring run.
+
+## Seed example
+
+The first corpus entry is
+[`tests/fixtures/bakeoff-rich/gold.json`](../../tests/fixtures/bakeoff-rich/gold.json)
+— the 5 brief-traceable seeded findings of the bakeoff-rich JWT-auth fixture.
+`gold.json` is the **canonical machine-readable form**; the prose table in
+`tests/fixtures/bakeoff-rich/README.md` is illustrative. When they disagree,
+`gold.json` wins.
+
+## Record schema (`voyage-eval-gold/1`)
+
+A corpus file is one JSON object:
+
+```jsonc
+{
+  "schema": "voyage-eval-gold/1",
+  "source": "<path to the prose/fixture this was distilled from>",
+  "description": "<one line>",
+  "expected_verdict": "BLOCK | WARN | ALLOW",   // review-coordinator Pass-4 outcome
+  "findings": [
+    {
+      "file": "<repo-relative path in the reviewed diff>",
+      "line": 0,                                  // integer >= 0; 0 = file-scoped
+      "rule_key": "<member of lib/review/rule-catalogue.mjs RULE_CATALOGUE>",
+      "severity": "BLOCKER | MAJOR | MINOR | SUGGESTION",
+      "owner_reviewer": "conformance | correctness"
+      // optional eval-extension fields are permitted, e.g.:
+      // "dual_flaggable": "<a second rule_key the same issue could carry>"
+    }
+  ]
+}
+```
+
+### Hard constraints
+
+- **`rule_key` must be a member of `RULE_CATALOGUE`** (`lib/review/rule-catalogue.mjs`).
+  The catalogue is the contract; an invented rule_key is a corpus bug.
+- **`severity` must be one of `SEVERITY_VALUES`** (`BLOCKER`, `MAJOR`, `MINOR`, `SUGGESTION`).
+- **`expected_verdict`** is the deterministic Pass-4 outcome of
+  `lib/review/coordinator-contract.mjs::computeVerdict`: `BLOCKER ≥ 1 → BLOCK`,
+  else `MAJOR ≥ 1 → WARN`, else `ALLOW`.
+- Finding-level fields (`file`, `line`, `rule_key`, `severity`) mirror
+  `FINDING_REQUIRED_FIELDS` (`lib/review/findings-schema.mjs`); `owner_reviewer`
+  and any extension fields are eval-specific additions (superset).
+
+## Adding a new frozen failure
+
+1. Distil the misfire into a `voyage-eval-gold/1` record (one `.json` file here,
+   or a new entry in an existing corpus file).
+2. Confirm every `rule_key` is in the catalogue and `expected_verdict` matches
+   the Pass-4 computation.
+3. Add (or extend) a loader test in the `tests/lib/gold-corpus.test.mjs` shape so
+   the record's shape + catalogue membership are pinned under `node --test`.
+
+## Future hardening (not in this tier)
+
+- SKAL-1·4b: an offline gold-scored output eval that scores recorded agent runs
+  against these records at `(file, rule_key)` granularity, with committed runs
+  and a third test-census category.
+- A prose↔JSON cross-assertion (parse the fixture README table, diff against
+  `gold.json`) to mechanically bound the two-source-of-truth drift.