92 lines
4.1 KiB
Markdown
92 lines
4.1 KiB
Markdown
# R11 flag format — the judge-pass → fix-work-list contract
|
||
|
||
**Spec for the flag record the R7–R10 judge pass emits and R11 (the fix step)
|
||
consumes. A flag is a judged claim the pass could NOT ground against its cited
|
||
Microsoft Learn source. The pass never fixes — it stamps (born-verified) or
|
||
flags; R11 does the human-confirmed fix.**
|
||
|
||
Status: design spec (R6 Step 5). Builds on the frozen v3.1 claim judge
|
||
(`scripts/kb-eval/judge-claim-prompt-v3.1.md`) and the judge-pass ledger
|
||
(`scripts/kb-eval/judge-pass-manifest.mjs`, R6 Step 4). Consumed by R11.
|
||
|
||
---
|
||
|
||
## Where a flag comes from
|
||
|
||
The v3.1 judge returns, per claim, a strict-JSON result (no fence):
|
||
|
||
```
|
||
{"file":"<FILE>","results":[
|
||
{"id":"<claim id>","judge_verdict":"grounded|not_grounded|source_silent",
|
||
"rule":"<R1-R8 or empty>","evidence_url":"<url actually used>",
|
||
"evidence_quote":"<verbatim quote or empty>","reason":"<one sentence>"}
|
||
]}
|
||
```
|
||
|
||
The pass augments each **non-grounded** result (`judge_verdict !== 'grounded'`)
|
||
into a flag record. This is a **minimal augmentation, not a pass-through** — the
|
||
judge output is a subset of the flag record; the pass adds four fields from the
|
||
fan-out context and the claim manifest:
|
||
|
||
| Added field | Source |
|
||
|-------------|--------|
|
||
| `file` | the fan-out context (one subagent per file) — also echoed by the judge output's top-level `file` |
|
||
| `line` | the claim manifest (`scripts/kb-eval/extract-judge-claims.mjs` output) |
|
||
| `claim` | the claim manifest (the verbatim claim text that was judged) |
|
||
| `disposition` | mapped from `judge_verdict` (see the vocabulary mapping below) |
|
||
|
||
## The flag record
|
||
|
||
A flag record is the union of the judge result and the four augmented fields:
|
||
|
||
```
|
||
{
|
||
"id": "<claim id>",
|
||
"judge_verdict": "not_grounded" | "source_silent", // 'grounded' is never flagged
|
||
"rule": "R1".."R8" | "",
|
||
"evidence_url": "<url the judge actually used>",
|
||
"evidence_quote":"<verbatim quote or empty>",
|
||
"reason": "<one sentence: what the source said vs the claim>",
|
||
"file": "skills/.../x.md",
|
||
"line": <number>,
|
||
"claim": "<verbatim claim text>",
|
||
"disposition": "outdated" | "wrong" | "unsourced" // R11 fix disposition
|
||
}
|
||
```
|
||
|
||
In the ledger (R6 Step 4), a flagged file is a manifest record with
|
||
`per_file_verdict: "flagged"` and one `flags[]` entry per non-grounded claim —
|
||
the structural mirror of `spor0-fix-manifest.json`'s "fix-record with
|
||
`applied:false`". A file is `flagged` if **any** judgeable claim is non-grounded
|
||
(the born-verified rule is AND: `pass` requires EVERY claim grounded).
|
||
|
||
## The two vocabularies + the mapping
|
||
|
||
There are two distinct enumerations, and conflating them is the trap this spec
|
||
exists to prevent:
|
||
|
||
1. **`judge_verdict`** — the judge's per-claim output (v3.1 prompt):
|
||
- `grounded` — the source confirms the claim → never flagged.
|
||
- `not_grounded` — the source contradicts / does not support the claim.
|
||
- `source_silent` — the source does not address the claim at all.
|
||
|
||
2. **`disposition`** — the R11 fix disposition, the correctness vocabulary from
|
||
`scripts/kb-eval/lib/base-rate.mjs` (`VERDICTS`): `correct`, `outdated`,
|
||
`wrong`, `unsourced`. Its `ERROR_VERDICTS` subset — `outdated` and `wrong` —
|
||
are the only ones R11 treats as **fix targets**. `unsourced` is recorded but
|
||
is not, on its own, a content error (no MS source confirms it either way).
|
||
|
||
**Mapping** (`judge_verdict` → `disposition`):
|
||
|
||
| `judge_verdict` | `disposition` | R11 fix target? |
|
||
|-----------------|---------------|-----------------|
|
||
| `grounded` | `correct` | no (never flagged) |
|
||
| `not_grounded` | `outdated` or `wrong` | **yes** — the human assigns which at R11 |
|
||
| `source_silent` | `unsourced` | no (flagged, but not a fix target) |
|
||
|
||
`not_grounded → {outdated, wrong}` is deliberately a one-to-two mapping: the
|
||
judge establishes the claim is not grounded; whether it is stale-but-once-true
|
||
(`outdated`) or never-true (`wrong`) is a human call at R11, re-verified against
|
||
the live source (verification duty). The pass never auto-fixes — every flag is
|
||
human-confirmed in R11, so a judge false-positive becomes a human review, never
|
||
silent corruption of a public file.
|