docs(retrieval-gate): the round the gate had to resist the work it judges
README's row table and the `--real` paragraph, CLAUDE.md's gate entry, CHANGELOG under 0.10.1 (no new version, no tag), and the full report. The report carries the eight attacks before and after, the row table before and after, row 8's three-set measurement with its two granularities apart, the mutation table as properties rather than as code lines, and § 6 -- the three things this round does NOT close: row 3 can still be made green by an honest change that costs elsewhere (the gate is the unit, not the row), the judge's parser is still `consume`'s and is bounded by measurement rather than removed, and `SPECS_SHA256` digests the corpus's fields rather than the bundle's bytes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
149836c722
commit
23588e5ada
SSH key fingerprint:
SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
4 changed files with 335 additions and 16 deletions
199
docs/2026-09-19-gjenfinningsgaten-motstand.md
Normal file
199
docs/2026-09-19-gjenfinningsgaten-motstand.md
Normal file
|
|
@ -0,0 +1,199 @@
|
|||
# The retrieval gate had to resist the work it judges (2026-09-19)
|
||||
|
||||
PM's checkpoint on `2c8296b` ran eight cheating attacks at
|
||||
`tools/okf_retrieval_gate.py` and **four went through**: rows 3, 5, 7 and 8
|
||||
could each be taken GREEN without one label becoming true or one concept
|
||||
ranking better. An eval written before the capability has one job beyond being
|
||||
red today — it must not be closable by anything except the work — so the gate
|
||||
was repaired before anything is built against it. `src/` is untouched, and
|
||||
`git diff --stat` over it is empty for this whole round.
|
||||
|
||||
This report states what was closed, how each closure was measured, and the
|
||||
three things it does NOT close.
|
||||
|
||||
## 1. The eight attacks, before and after
|
||||
|
||||
Each attack is a test now, named for PM's label, and the four that were
|
||||
already refused stay as regression guards, so the table is 8 of 8 rather than
|
||||
4 of 4.
|
||||
|
||||
| attack | before | after | where |
|
||||
|---|---|---|---|
|
||||
| J1 two self-written files as a hold-out | `7 of 7 GREEN` | RED, three git checks fail | `test_j1_a_registration_this_session_wrote_is_not_a_hold_out` |
|
||||
| J2 one real set of three | `6 of 6 GREEN` | NOT RUN, the measured set still printed | `test_j2_row_eight_is_not_run_when_a_required_set_is_left_out` |
|
||||
| J2b the headline sums two granularities | `12 of 12` | questions answered, the two unit totals apart | `test_j2b_row_eight_never_sums_the_two_granularities_into_its_headline` |
|
||||
| J3 seven duplicate mutants appended | `18 of 20 GREEN` | NOT RUN, the roster is pinned | `test_j3_row_seven_refuses_a_mutant_list_that_is_not_the_pinned_roster` |
|
||||
| J10 `k = 32` | rows 1, 2, 3, 6 all GREEN | rows 2 RED and 3 not green, denominators held | `test_j10_a_wider_cut_does_not_make_rows_two_and_three_green` |
|
||||
| J8 the quota removed | `6 of 6 GREEN` | NOT RUN, the label was never printed | `test_j8_removing_the_quota_leaves_row_three_unable_to_say_anything` |
|
||||
| J4 the corpus tuned | caught by row 2's forced classes alone | also `SPECS_SHA256`, exit 2 | `test_a_corpus_that_is_not_the_pinned_corpus_is_refused` |
|
||||
| J5 a tampered set | exit 2 | unchanged | `test_a_tampered_set_makes_the_command_exit_two` |
|
||||
| J7/J11 deliver nothing | an error from `consume`, exit 2 | unchanged, now with a test | `test_j7_a_cut_that_delivers_nothing_is_exit_two_and_never_a_quiet_green_row` |
|
||||
| G10 the sha comparison removed | felled by the suite | unchanged | `test_the_pin_refuses_a_set_whose_bytes_moved` |
|
||||
|
||||
## 2. The row table, before and after
|
||||
|
||||
Nothing here became greener. Row 8 acquired a NUMBER, which is not the same
|
||||
thing.
|
||||
|
||||
| row | before (`2c8296b`) | after | status |
|
||||
|---|---|---|---|
|
||||
| 1 | 9 of 9 GREEN | 10 of 10 GREEN | one fixture added (`set-quota.json`), one hit |
|
||||
| 2 | 7 of 7 GREEN | 7 of 7 GREEN | unchanged |
|
||||
| 3 | 2 of 5 RED | 2 of 5 RED | unchanged; the denominator is now the set's |
|
||||
| 4 | 3 of 6 RED | 3 of 6 RED | unchanged |
|
||||
| 5 | 0 of 1 RED | 0 of 1 RED | unchanged; ten checks now, three of them git |
|
||||
| 6 | 9 of 9 GREEN | 10 of 10 GREEN | the same added fixture |
|
||||
| 7 | 11 of 13 RED | 12 of 14 RED | `M14` added and felled; bar 12 of 13 -> 13 of 14 |
|
||||
| 8 | 0 of 3 NOT RUN | 44 of 64 questions RED | measured against all three real sets |
|
||||
| 9 | 0 of 6 RED | 0 of 6 RED | unchanged; it takes an input now |
|
||||
|
||||
`GATE RED: rows 3, 4, 5, 7, 8, 9` before and after, exit 1, byte-identical
|
||||
over two runs.
|
||||
|
||||
## 3. Row 8 — the three sets, measured
|
||||
|
||||
Run once, on this machine, with all three sets read from their own
|
||||
repositories by path + sha256. Nothing from them is committed here.
|
||||
|
||||
```
|
||||
8 the real sets (wiki-20, r761-sk2, vegnormal-32) | 44 of 64 | RED
|
||||
wiki-20: 7 of 29 fasit entries (citation) | 6 of 20 questions | sha256 972d0f5715d1
|
||||
r761-sk2: 7 of 7 fasit entries (concept) | 7 of 7 questions | sha256 c834a478e488
|
||||
vegnormal-32: 31 of 43 fasit entries (concept) | 31 of 37 questions | sha256 c6d9305a0497
|
||||
NOT SUMMED INTO ONE NUMBER: 7 of 29 at citation granularity,
|
||||
38 of 50 at concept granularity
|
||||
```
|
||||
|
||||
34 misses, **33 of them class b** (below k with the quota off as well) and one
|
||||
class c.
|
||||
|
||||
**Two of the three reproduce PM's recorded figures exactly**: wiki at 6 of 20
|
||||
questions and r761 at 7 of 7. **The third does not, and the difference is
|
||||
stated rather than smoothed.** PM recorded `32 of 43 citations = 21 of 32
|
||||
questions`; this gate measures 31 of 43 and 31 of 37. The question counts are
|
||||
not comparable at all: the gate splits a question citing two standards into
|
||||
one Question per bundle, because a payload is built against one bundle, so 32
|
||||
questions become 37. The citation count differs by ONE and that is a real
|
||||
disagreement between two instruments over the same pinned bytes; it is not
|
||||
resolved here and it is not this round's subject.
|
||||
|
||||
**The headline is at QUESTION granularity**, which is the fix for J2b: the
|
||||
three sets share no unit — wiki names a citation, r761 a section title,
|
||||
vegnormal a requirement number — and a citation hit plus a concept hit is a
|
||||
number that is neither. A question is the one unit all three have. "Answered"
|
||||
means at least one of its fasit entries arrived, the reading row 1 already
|
||||
prints beside its own units.
|
||||
|
||||
## 4. What each closure rests on
|
||||
|
||||
**Row 8 requires all three sets by name** (`REQUIRED_REAL_SETS`) and is NOT
|
||||
RUN otherwise — with the numbers the run DID measure still printed, because a
|
||||
missing set must not cost the reader the set that was measured. This
|
||||
repository's own test asserted `(1, 1, GREEN)` for a single set; that
|
||||
assertion was the defect and it is corrected rather than worked around.
|
||||
|
||||
**Rows 2 and 3 take their denominator from the pinned set.** A forced fixture
|
||||
that stops producing its declared class is a BROKEN PREMISE, printed as one
|
||||
and counted against its row. At `k = 32` the three fixtures declaring class b
|
||||
are delivered; before, they left the denominator and the rows read 4 of 4 and
|
||||
2 of 2 GREEN.
|
||||
|
||||
**Row 3 carries a known-positive.** With `--source-quota` off every printed
|
||||
reason is true — that is not a lie, it is an empty measurement — so a set may
|
||||
declare `source_quota_in_force`, and the row is NOT RUN for it when the
|
||||
default cut and the quota-off cut deliver the same concepts everywhere.
|
||||
|
||||
**The control's own premise was measured before it was built on, and it was
|
||||
FALSE where it was first put.** Over the five existing sets the two cuts
|
||||
deliver the SAME concepts — the quota is topped back up — so 52 labels move
|
||||
`source_quota_exceeded` to `below_k` without one delivery changing.
|
||||
`set-quota.json` is the one set where the quota genuinely decides: measured,
|
||||
`oversikt-08` is delivered without the quota and withheld with it, and the
|
||||
fasit `svar/broennproeve` is delivered only with it. The control survives the
|
||||
honest fix, which changes labels and not the cut.
|
||||
|
||||
**Row 5 reads git for the half a registration cannot assert.** Three of its
|
||||
ten checks: the file is committed and unmodified; the commit that ADDED it is
|
||||
not itself a change to `src/llm_ingestion_okf/consume.py`; a change to that
|
||||
path landed AFTER it. The third is the one that cannot be self-attested. What
|
||||
git cannot prove is in the row's own docstring: history shows ORDER, never
|
||||
that nobody read the number before writing the threshold.
|
||||
|
||||
**Row 7's roster is pinned apart from the list it names.** The bar is a share,
|
||||
so a longer list is a lower bar per survivor. `MUTANT_ROSTER` and
|
||||
`MUTANT_COUNT` are separate constants, duplicates are refused outright, and
|
||||
the bar is taken from the roster's length.
|
||||
|
||||
**PM's G9 is decided: the term was observable and unobserved.** `hit =
|
||||
bool(hit_ids) and bool(confirmed)` is reached only by a delivery that still
|
||||
carries the citation and is no longer the concept file's bytes; every other
|
||||
mutation of the text empties `hit_ids` one step earlier. `M14` is that
|
||||
mutation, it is felled, and no production line changed.
|
||||
|
||||
**And that measures the judge's independence rather than asserting it.** PM
|
||||
found the judge reads the bundle through `consume.delivered_text`, the same
|
||||
function it judges. Measured both ways: with the index warmed BEFORE the patch
|
||||
(which is what the gate does, and what row 7's docstring already relied on)
|
||||
every unit comes back a miss with `confirmed False`; with the index built
|
||||
UNDER the patch every unit comes back a hit. The gate never builds one under a
|
||||
mutation. This is in `LIMITS` with the measurement, rather than closed by
|
||||
re-implementing a normalisation rule this repository already owns once.
|
||||
|
||||
## 5. The mutation run
|
||||
|
||||
Twelve mutants of the gate plus one control that must survive, run in a
|
||||
scratch copy of HEAD (`git archive` into `/tmp`, its own
|
||||
`uv sync --frozen --extra extract --group dev`), each judged by
|
||||
`tests/test_retrieval_gate.py`. The harness itself stays in the scratch copy:
|
||||
it rewrites the gate file and restores it, which is not a thing to keep in a
|
||||
tracked tree. What is recorded here is each mutant as a PROPERTY rather than
|
||||
as a line of code, so the table does not go stale the first time the gate is
|
||||
refactored.
|
||||
|
||||
| # | the guard it removes | felled by |
|
||||
|---|---|---|
|
||||
| X1 | row 8 counts whatever it was given, three sets or one | J2 |
|
||||
| X2 | row 8's headline is the two granularities summed | J2b |
|
||||
| X3 | row 2's denominator is the run's misses again | J10 |
|
||||
| X4 | row 3 judges only what this run withheld | J10 |
|
||||
| X5 | row 3 judges a run with no quota in force | J8 |
|
||||
| X6 | row 5 believes the registration about its own history | J1 |
|
||||
| X7 | row 5 accepts a registration no ranking change came after | the three-arm row-5 test |
|
||||
| X8 | row 7 takes the mutant list as given | J3 |
|
||||
| X9 | a delivery is a hit without checking the bundle's bytes (PM's G9) | the G9 test |
|
||||
| X10 | the synthetic corpus is not pinned | the `SPECS_SHA256` refusal |
|
||||
| X11 | row 9 ignores the set it was handed | the row-9 green test |
|
||||
| X12 | a K2 set of any size wears K2's name | the size refusal |
|
||||
| control | a detail line nobody asserts on is reworded | nothing — it MUST survive |
|
||||
|
||||
**First run: killed 11 of 12, and X2 SURVIVED.** Put row 8's headline back to
|
||||
the sum and all 64 tests stayed green. The cause was the fixture, not the row:
|
||||
three sets of one fasit entry each make "questions answered" and "units hit"
|
||||
the same number, so the test agreed with the mutant by construction — the same
|
||||
shape as round 158's X5. One set now carries two fasit entries, the only shape
|
||||
where the two readings differ.
|
||||
|
||||
**Second run, on the commit that fixed it: killed 12 of 12, control survived.**
|
||||
|
||||
## 6. What this round does NOT close
|
||||
|
||||
- **Row 3 can still be made green by an honest change that costs elsewhere.**
|
||||
Removing the quota makes every printed reason true; the price lands on row 1
|
||||
(8 of 9) and on `M05`, which is felled. The gate is the unit, not the row.
|
||||
- **The judge's parser is `consume`'s.** Bounded by measurement (§ 4) rather
|
||||
than removed.
|
||||
- **`SPECS_SHA256` is a digest of the corpus's fields, not of the bundle's
|
||||
bytes.** `build_bundle` is deterministic, so the two move together; that is
|
||||
an argument, not a measurement.
|
||||
- **Rows 8 and 9 still cannot be green without inputs this repository must
|
||||
not hold** — the three sets are on one machine and the K2 gold set exists
|
||||
nowhere. Both are operator questions and both are open.
|
||||
|
||||
## 7. Denominators, re-measured against the sources
|
||||
|
||||
| source | measured here 2026-09-19 |
|
||||
|---|---|
|
||||
| `claude-code-llm-wiki/eval/fase-sporsmaal.json` | sha `972d0f57…`, 20 questions, 29 fasit entries |
|
||||
| `vegnormal-okf/docs/2026-09-10-sk2-r761-sporsmal.json` | sha `c834a478…`, 8 entries, 7 positives + KN |
|
||||
| `vegnormal-okf/build/resonnering/sporsmal-m2.json` | sha `c6d9305a…`, 32 questions, 43 `must_cite` |
|
||||
| K2 | 6 questions recorded; no gold set anywhere |
|
||||
Loading…
Add table
Add a link
Reference in a new issue