docs(retrieval-gate): the round the gate had to resist the work it judges
README's row table and the `--real` paragraph, CLAUDE.md's gate entry, CHANGELOG under 0.10.1 (no new version, no tag), and the full report. The report carries the eight attacks before and after, the row table before and after, row 8's three-set measurement with its two granularities apart, the mutation table as properties rather than as code lines, and § 6 -- the three things this round does NOT close: row 3 can still be made green by an honest change that costs elsewhere (the gate is the unit, not the row), the judge's parser is still `consume`'s and is bounded by measurement rather than removed, and `SPECS_SHA256` digests the corpus's fields rather than the bundle's bytes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
149836c722
commit
23588e5ada
SSH key fingerprint:
SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
4 changed files with 335 additions and 16 deletions
65
CHANGELOG.md
65
CHANGELOG.md
|
|
@ -282,6 +282,71 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||
|
||||
### Fixed
|
||||
|
||||
- **The retrieval gate had to resist the work it judges: four of eight
|
||||
cheating attacks went through it, and they are closed (2026-09-19).** PM's
|
||||
checkpoint on `2c8296b` took rows 3, 5, 7 and 8 GREEN without one label
|
||||
becoming true or one concept ranking better. An eval written before the
|
||||
capability has one job beyond being red today, so the gate was repaired
|
||||
before anything is built against it. `src/` is untouched.
|
||||
- **Row 8 requires all three named sets** (`wiki-20`, `r761-sk2`,
|
||||
`vegnormal-32`) and is NOT RUN otherwise. It counted whatever `--real`
|
||||
gave it, so one set of three read `6 of 6 GREEN` — and this repository's
|
||||
own test asserted `(1, 1, GREEN)` for a single set. The numbers the run
|
||||
DID measure are still printed: a missing set must not cost the reader the
|
||||
set that was measured.
|
||||
- **Its headline is at QUESTION granularity**, and was `quoted + concept`
|
||||
over `quoted_units + concept_units` on the line above the detail saying
|
||||
the two are not summed. The three sets share no unit — a citation, a
|
||||
section title and a requirement number — so their sum is a number that is
|
||||
none of them.
|
||||
- **Rows 2 and 3 take their denominator from the pinned set, not the run.**
|
||||
At `k = 32` the fixtures declaring class b are delivered, and they used to
|
||||
leave the denominator: rows 1, 2, 3 and 6 all read green at once. A forced
|
||||
fixture that stops producing its declared class is a BROKEN PREMISE now,
|
||||
printed as one and counted against its row.
|
||||
- **Row 3 carries a known-positive.** With `--source-quota` off every
|
||||
printed reason is true — not a lie, an empty measurement — so a set may
|
||||
declare `source_quota_in_force` and the row is NOT RUN for it when the
|
||||
default and quota-off cuts deliver the same concepts. **The control's own
|
||||
premise was measured first and was false where it was first put:** over
|
||||
the five existing sets the two cuts deliver the SAME concepts (the quota
|
||||
is topped back up), 52 labels moving `source_quota_exceeded` → `below_k`
|
||||
with 0 deliveries changing. `set-quota.json` is the one set where the
|
||||
quota genuinely decides.
|
||||
- **Row 5 reads git for the half a registration cannot assert about
|
||||
itself.** Two files PM wrote in the moment came back `7 of 7 GREEN`. Three
|
||||
of its ten checks now read history: committed and unmodified, its commit
|
||||
is not itself a ranking change, and a ranking change landed AFTER it —
|
||||
the last being the one that cannot be self-attested. What git cannot prove
|
||||
(that nobody read the number first) is stated in the row.
|
||||
- **Row 7's roster is pinned apart from the list it names.** The bar is a
|
||||
share, so seven duplicate `k = 1` mutants read `18 of 20 GREEN` with the
|
||||
same two survivors. `MUTANT_ROSTER` and `MUTANT_COUNT` are separate
|
||||
constants, duplicates are refused, and the bar is the roster's length.
|
||||
- **The synthetic corpus is pinned like the sets** (`SPECS_SHA256`). A tuned
|
||||
corpus was caught by row 2's forced classes and not by a pin.
|
||||
- **`M14` closes PM's G9**: `hit = bool(hit_ids) and bool(confirmed)` is
|
||||
reached only by a delivery that still carries the citation and is no
|
||||
longer the concept file's bytes. It is felled and no production line
|
||||
changed — the term was observable and unobserved. The judge's
|
||||
independence is measured with it: index warmed BEFORE the patch, every
|
||||
unit a miss; index built UNDER it, every unit a hit. The gate never builds
|
||||
one under a mutation, and that is in `LIMITS`.
|
||||
- **Row 9 takes `--k2 SET SHA256 BUNDLE`** in this gate's own set shape, and
|
||||
a set of another size is refused as another set wearing K2's name. Without
|
||||
one it stays RED rather than NOT RUN: its denominator is known.
|
||||
- **Row 8 ran, against all three real sets**: **44 of 64 questions**, 7 of
|
||||
29 at citation granularity and 38 of 50 at concept granularity, 33 of 34
|
||||
misses class b. wiki (6 of 20) and r761 (7 of 7) reproduce PM's recorded
|
||||
figures exactly; vegnormal measures 31 of 43 citations where PM recorded
|
||||
32, a one-citation disagreement between two instruments over the same
|
||||
pinned bytes, stated and not resolved here.
|
||||
- Rows 1 and 6 go 9 of 9 to 10 of 10 (one added fixture, one added hit).
|
||||
Every other row is unchanged and the verdict is unchanged:
|
||||
`GATE RED: rows 3, 4, 5, 7, 8, 9`, exit 1, byte-identical over two runs.
|
||||
`mypy --strict` on the gate goes 8 errors to 0. Report:
|
||||
[`docs/2026-09-19-gjenfinningsgaten-motstand.md`](docs/2026-09-19-gjenfinningsgaten-motstand.md).
|
||||
|
||||
- **The conversion claim the content-accounting gate believes now comes from
|
||||
the RUN, not from the bundle's prose (0.10.1).** The previous round bound
|
||||
the claim to a pointer block, which closed the two forgeries PM had
|
||||
|
|
|
|||
49
CLAUDE.md
49
CLAUDE.md
|
|
@ -1216,13 +1216,54 @@ and fixtures, never code.
|
|||
what a bundle looks like, this asks whether the payload for a question
|
||||
carries the fasit. **Written RED, no capability**: nine rows, exit 0 only
|
||||
when all are green, 1 otherwise, 2 on wrong input, `--json`. Rows 1-4, 6, 7
|
||||
run against a synthetic corpus this repo GENERATES and four sets it ships
|
||||
run against a synthetic corpus this repo GENERATES and six sets it ships
|
||||
pinned by sha256 -- no network, no private corpus, no clock; a set whose
|
||||
bytes moved is exit 2, never a quiet continuation, because **a gold set is
|
||||
an INPUT and never a constant here** (`okf_consume_measure.py`'s rule, and
|
||||
this repo is public). Today: **1 (9/9) - 2 (7/7) - 3 (2/5) - 4 (3/6) -
|
||||
5 (0/1) - 6 (9/9) - 7 (11/13) - 8 not run - 9 (0/6)**, `GATE RED: rows 3,
|
||||
4, 5, 7, 8, 9`. **Row 3 is the finding to read first**: in a bundle built
|
||||
this repo is public). **THE CORPUS IS PINNED THE SAME WAY** since
|
||||
2026-09-19 (`SPECS_SHA256`). Today: **1 (10/10) - 2 (7/7) - 3 (2/5) -
|
||||
4 (3/6) - 5 (0/1) - 6 (10/10) - 7 (12/14) - 8 (44/64 questions, all three
|
||||
real sets measured 2026-09-19) - 9 (0/6)**, `GATE RED: rows 3, 4, 5, 7, 8,
|
||||
9`.
|
||||
**FOUR OF EIGHT CHEATING ATTACKS WENT THROUGH IT AND ARE CLOSED
|
||||
(2026-09-19).** PM's checkpoint on `2c8296b` found rows 3, 5, 7 and 8 could
|
||||
go GREEN without one label becoming true or one concept ranking better, and
|
||||
an eval written before the capability has exactly that one job beyond being
|
||||
red today. **Row 8** counted whatever `--real` gave it, so one set of three
|
||||
read `6 of 6 GREEN` -- and this repo's OWN test asserted `(1, 1, GREEN)` for
|
||||
a single set; it now requires all three by name and is NOT RUN otherwise,
|
||||
the measured sets' numbers still printed, and its headline is at QUESTION
|
||||
granularity because the three sets share no unit. **Rows 2 and 3** took
|
||||
their denominator from the run, so `k = 32` shrank it to the cases that were
|
||||
already honest (rows 1, 2, 3 and 6 all green at once); the denominator is
|
||||
now the PINNED set's, and a forced fixture that stops producing its declared
|
||||
class is a BROKEN PREMISE counted against its row. **Row 3 also carries a
|
||||
known-positive**: with the quota off every printed reason is true, which is
|
||||
not a lie but an empty measurement, so a set may declare
|
||||
`source_quota_in_force` and the row is NOT RUN when the default cut and the
|
||||
quota-off cut deliver the same concepts. **The control's own premise was
|
||||
measured first and was FALSE where it was first put** -- over the five
|
||||
existing sets the two cuts deliver the SAME concepts (the quota is topped
|
||||
back up) and 52 labels move `source_quota_exceeded` -> `below_k` with 0
|
||||
deliveries changing -- so `set-quota.json` is the one set where the quota
|
||||
genuinely decides. **Row 5** was seven checks the registration made about
|
||||
itself; three now read GIT (committed and unmodified, its commit is not
|
||||
itself a ranking change, a ranking change landed AFTER it), the last being
|
||||
the one that cannot be self-attested. What git cannot prove -- that nobody
|
||||
read the number first -- is stated, not implied. **Row 7's bar** was 90 % of
|
||||
a list in the file a capability session edits, so seven duplicate `k = 1`
|
||||
mutants read `18 of 20 GREEN`; `MUTANT_ROSTER` and `MUTANT_COUNT` are pinned
|
||||
apart from `MUTANTS`, duplicates are refused, and the bar is the roster's.
|
||||
**PM's G9 is decided**: `hit = bool(hit_ids) and bool(confirmed)` is
|
||||
observable and was merely unobserved -- `M14` (a delivery that carries the
|
||||
citation and is not the concept file's bytes) is the shape that reaches it,
|
||||
it is FELLED, and no production line changed. The judge's independence is
|
||||
measured rather than asserted: the index is warmed BEFORE the first
|
||||
mutation, so patching `delivered_text` moves the payload and not the judge
|
||||
(index built UNDER the patch: every unit a hit; warmed first: every unit a
|
||||
miss). Row 9 takes `--k2 SET SHA BUNDLE` and stays RED without one, because
|
||||
ITS denominator is known -- a deliberate divergence from the order, stated.
|
||||
Report: `docs/2026-09-19-gjenfinningsgaten-motstand.md`. **Row 3 is the finding to read first**: in a bundle built
|
||||
from ONE source document every concept past the first two carries that
|
||||
document's `source_file`, so a concept the RANK had already lost is withheld
|
||||
as `source_quota_exceeded` -- the truth is decided by the SAME cut run
|
||||
|
|
|
|||
38
README.md
38
README.md
|
|
@ -1011,24 +1011,26 @@ python3 tools/okf_retrieval_gate.py --json # the same rows as JSON
|
|||
|
||||
Nine rows, exit 0 only when every one is green, 1 otherwise, 2 for wrong
|
||||
input. Rows 1–4, 6 and 7 run against a synthetic corpus this repository
|
||||
generates and four question sets it ships, pinned by sha256: no network, no
|
||||
generates and six question sets it ships, pinned by sha256: no network, no
|
||||
private corpus, no clock. A question set is always an input — `sha256` is
|
||||
checked before a byte is measured and a mismatch is exit 2 — because a gold
|
||||
set names a consumer's documents and this repository is public.
|
||||
set names a consumer's documents and this repository is public. **The corpus
|
||||
is pinned the same way** (`SPECS_SHA256`): every row counts against those
|
||||
documents, so moving them without moving the pin is exit 2.
|
||||
|
||||
**It is RED today, on rows 3, 4, 5, 7, 8 and 9**, and each of those is a
|
||||
finding rather than a defect in the gate:
|
||||
|
||||
| row | what it asks | today |
|
||||
|---|---|---|
|
||||
| 1 | hit@payload, one fasit entry = one unit | 9 of 9 |
|
||||
| 1 | hit@payload, one fasit entry = one unit | 10 of 10 |
|
||||
| 2 | every miss carries exactly one class, each forced by its own fixture | 7 of 7 |
|
||||
| 3 | the `rule` the payload prints for a withheld fasit is the true one | 2 of 5 |
|
||||
| 4 | an uncovered question comes back marked, a covered one does not | 3 of 6 |
|
||||
| 5 | a hold-out set, frozen and with its threshold written first | 0 of 1 |
|
||||
| 6 | every delivery confirmed against the bundle's own bytes | 9 of 9 |
|
||||
| 7 | mechanical mutants of the ranking and the cut, felled | 11 of 13 |
|
||||
| 8 | the three real sets, from path + sha256 | not run |
|
||||
| 6 | every delivery confirmed against the bundle's own bytes | 10 of 10 |
|
||||
| 7 | mechanical mutants of the ranking and the cut, felled | 12 of 14 |
|
||||
| 8 | the three real sets, from path + sha256 | 44 of 64 questions |
|
||||
| 9 | K2 | 0 of 6, no gold set exists |
|
||||
|
||||
Row 3 is the one to read first: in a bundle built from ONE source document,
|
||||
|
|
@ -1047,13 +1049,25 @@ killing the document prior and flattening the fusion (`RRF_K`) each moved
|
|||
a question that names its document reaches it through the title-and-id signal
|
||||
as well, and `1/(K+r)` is strictly decreasing in `r` for every `K`.
|
||||
|
||||
Rows 8 and 9 are never green by leaving something out. The three real sets
|
||||
live in other repositories and are read, never written:
|
||||
Rows 8 and 9 are never green by leaving something out, and since 2026-09-19
|
||||
that is enforced rather than stated: row 8 requires **all three** named sets
|
||||
(`wiki-20`, `r761-sk2`, `vegnormal-32`) and is NOT RUN until it has them,
|
||||
whatever the ones that ran scored — one set of three used to read `6 of 6
|
||||
GREEN`. The sets live in other repositories and are read, never written:
|
||||
`--real wiki <set.json> <sha256> <bundle>` runs one, and
|
||||
`--real vegnormal <set.json> <sha256> "N100:2023=<bundle>,N200:2024=<bundle>"`
|
||||
runs one that spans bundles. Granularity is stated on every line and the two
|
||||
forms are never summed: a set naming a citation is measured at citation
|
||||
granularity, a set naming only a section is measured at concept granularity.
|
||||
`--real vegnormal <set.json> <sha256> "N100=<bundle>,N200=<bundle>"` runs one
|
||||
that spans bundles. Row 9 takes `--k2 <set.json> <sha256> <bundle>` in this
|
||||
gate's own set shape; without one it stays RED against its recorded
|
||||
denominator of six.
|
||||
|
||||
Granularity is stated on every line and the two forms are never summed: a set
|
||||
naming a citation is measured at citation granularity, a set naming only a
|
||||
section is measured at concept granularity. **Row 8's own headline is
|
||||
therefore at QUESTION granularity** — the one unit all three sets share —
|
||||
with the two unit totals printed below it, each with its own denominator.
|
||||
Measured 2026-09-19 against the three real sets on one machine: **44 of 64
|
||||
questions**, and below it *7 of 29 at citation granularity, 38 of 50 at
|
||||
concept granularity*.
|
||||
|
||||
## Consume in Claude Code
|
||||
|
||||
|
|
|
|||
199
docs/2026-09-19-gjenfinningsgaten-motstand.md
Normal file
199
docs/2026-09-19-gjenfinningsgaten-motstand.md
Normal file
|
|
@ -0,0 +1,199 @@
|
|||
# The retrieval gate had to resist the work it judges (2026-09-19)
|
||||
|
||||
PM's checkpoint on `2c8296b` ran eight cheating attacks at
|
||||
`tools/okf_retrieval_gate.py` and **four went through**: rows 3, 5, 7 and 8
|
||||
could each be taken GREEN without one label becoming true or one concept
|
||||
ranking better. An eval written before the capability has one job beyond being
|
||||
red today — it must not be closable by anything except the work — so the gate
|
||||
was repaired before anything is built against it. `src/` is untouched, and
|
||||
`git diff --stat` over it is empty for this whole round.
|
||||
|
||||
This report states what was closed, how each closure was measured, and the
|
||||
three things it does NOT close.
|
||||
|
||||
## 1. The eight attacks, before and after
|
||||
|
||||
Each attack is a test now, named for PM's label, and the four that were
|
||||
already refused stay as regression guards, so the table is 8 of 8 rather than
|
||||
4 of 4.
|
||||
|
||||
| attack | before | after | where |
|
||||
|---|---|---|---|
|
||||
| J1 two self-written files as a hold-out | `7 of 7 GREEN` | RED, three git checks fail | `test_j1_a_registration_this_session_wrote_is_not_a_hold_out` |
|
||||
| J2 one real set of three | `6 of 6 GREEN` | NOT RUN, the measured set still printed | `test_j2_row_eight_is_not_run_when_a_required_set_is_left_out` |
|
||||
| J2b the headline sums two granularities | `12 of 12` | questions answered, the two unit totals apart | `test_j2b_row_eight_never_sums_the_two_granularities_into_its_headline` |
|
||||
| J3 seven duplicate mutants appended | `18 of 20 GREEN` | NOT RUN, the roster is pinned | `test_j3_row_seven_refuses_a_mutant_list_that_is_not_the_pinned_roster` |
|
||||
| J10 `k = 32` | rows 1, 2, 3, 6 all GREEN | rows 2 RED and 3 not green, denominators held | `test_j10_a_wider_cut_does_not_make_rows_two_and_three_green` |
|
||||
| J8 the quota removed | `6 of 6 GREEN` | NOT RUN, the label was never printed | `test_j8_removing_the_quota_leaves_row_three_unable_to_say_anything` |
|
||||
| J4 the corpus tuned | caught by row 2's forced classes alone | also `SPECS_SHA256`, exit 2 | `test_a_corpus_that_is_not_the_pinned_corpus_is_refused` |
|
||||
| J5 a tampered set | exit 2 | unchanged | `test_a_tampered_set_makes_the_command_exit_two` |
|
||||
| J7/J11 deliver nothing | an error from `consume`, exit 2 | unchanged, now with a test | `test_j7_a_cut_that_delivers_nothing_is_exit_two_and_never_a_quiet_green_row` |
|
||||
| G10 the sha comparison removed | felled by the suite | unchanged | `test_the_pin_refuses_a_set_whose_bytes_moved` |
|
||||
|
||||
## 2. The row table, before and after
|
||||
|
||||
Nothing here became greener. Row 8 acquired a NUMBER, which is not the same
|
||||
thing.
|
||||
|
||||
| row | before (`2c8296b`) | after | status |
|
||||
|---|---|---|---|
|
||||
| 1 | 9 of 9 GREEN | 10 of 10 GREEN | one fixture added (`set-quota.json`), one hit |
|
||||
| 2 | 7 of 7 GREEN | 7 of 7 GREEN | unchanged |
|
||||
| 3 | 2 of 5 RED | 2 of 5 RED | unchanged; the denominator is now the set's |
|
||||
| 4 | 3 of 6 RED | 3 of 6 RED | unchanged |
|
||||
| 5 | 0 of 1 RED | 0 of 1 RED | unchanged; ten checks now, three of them git |
|
||||
| 6 | 9 of 9 GREEN | 10 of 10 GREEN | the same added fixture |
|
||||
| 7 | 11 of 13 RED | 12 of 14 RED | `M14` added and felled; bar 12 of 13 -> 13 of 14 |
|
||||
| 8 | 0 of 3 NOT RUN | 44 of 64 questions RED | measured against all three real sets |
|
||||
| 9 | 0 of 6 RED | 0 of 6 RED | unchanged; it takes an input now |
|
||||
|
||||
`GATE RED: rows 3, 4, 5, 7, 8, 9` before and after, exit 1, byte-identical
|
||||
over two runs.
|
||||
|
||||
## 3. Row 8 — the three sets, measured
|
||||
|
||||
Run once, on this machine, with all three sets read from their own
|
||||
repositories by path + sha256. Nothing from them is committed here.
|
||||
|
||||
```
|
||||
8 the real sets (wiki-20, r761-sk2, vegnormal-32) | 44 of 64 | RED
|
||||
wiki-20: 7 of 29 fasit entries (citation) | 6 of 20 questions | sha256 972d0f5715d1
|
||||
r761-sk2: 7 of 7 fasit entries (concept) | 7 of 7 questions | sha256 c834a478e488
|
||||
vegnormal-32: 31 of 43 fasit entries (concept) | 31 of 37 questions | sha256 c6d9305a0497
|
||||
NOT SUMMED INTO ONE NUMBER: 7 of 29 at citation granularity,
|
||||
38 of 50 at concept granularity
|
||||
```
|
||||
|
||||
34 misses, **33 of them class b** (below k with the quota off as well) and one
|
||||
class c.
|
||||
|
||||
**Two of the three reproduce PM's recorded figures exactly**: wiki at 6 of 20
|
||||
questions and r761 at 7 of 7. **The third does not, and the difference is
|
||||
stated rather than smoothed.** PM recorded `32 of 43 citations = 21 of 32
|
||||
questions`; this gate measures 31 of 43 and 31 of 37. The question counts are
|
||||
not comparable at all: the gate splits a question citing two standards into
|
||||
one Question per bundle, because a payload is built against one bundle, so 32
|
||||
questions become 37. The citation count differs by ONE and that is a real
|
||||
disagreement between two instruments over the same pinned bytes; it is not
|
||||
resolved here and it is not this round's subject.
|
||||
|
||||
**The headline is at QUESTION granularity**, which is the fix for J2b: the
|
||||
three sets share no unit — wiki names a citation, r761 a section title,
|
||||
vegnormal a requirement number — and a citation hit plus a concept hit is a
|
||||
number that is neither. A question is the one unit all three have. "Answered"
|
||||
means at least one of its fasit entries arrived, the reading row 1 already
|
||||
prints beside its own units.
|
||||
|
||||
## 4. What each closure rests on
|
||||
|
||||
**Row 8 requires all three sets by name** (`REQUIRED_REAL_SETS`) and is NOT
|
||||
RUN otherwise — with the numbers the run DID measure still printed, because a
|
||||
missing set must not cost the reader the set that was measured. This
|
||||
repository's own test asserted `(1, 1, GREEN)` for a single set; that
|
||||
assertion was the defect and it is corrected rather than worked around.
|
||||
|
||||
**Rows 2 and 3 take their denominator from the pinned set.** A forced fixture
|
||||
that stops producing its declared class is a BROKEN PREMISE, printed as one
|
||||
and counted against its row. At `k = 32` the three fixtures declaring class b
|
||||
are delivered; before, they left the denominator and the rows read 4 of 4 and
|
||||
2 of 2 GREEN.
|
||||
|
||||
**Row 3 carries a known-positive.** With `--source-quota` off every printed
|
||||
reason is true — that is not a lie, it is an empty measurement — so a set may
|
||||
declare `source_quota_in_force`, and the row is NOT RUN for it when the
|
||||
default cut and the quota-off cut deliver the same concepts everywhere.
|
||||
|
||||
**The control's own premise was measured before it was built on, and it was
|
||||
FALSE where it was first put.** Over the five existing sets the two cuts
|
||||
deliver the SAME concepts — the quota is topped back up — so 52 labels move
|
||||
`source_quota_exceeded` to `below_k` without one delivery changing.
|
||||
`set-quota.json` is the one set where the quota genuinely decides: measured,
|
||||
`oversikt-08` is delivered without the quota and withheld with it, and the
|
||||
fasit `svar/broennproeve` is delivered only with it. The control survives the
|
||||
honest fix, which changes labels and not the cut.
|
||||
|
||||
**Row 5 reads git for the half a registration cannot assert.** Three of its
|
||||
ten checks: the file is committed and unmodified; the commit that ADDED it is
|
||||
not itself a change to `src/llm_ingestion_okf/consume.py`; a change to that
|
||||
path landed AFTER it. The third is the one that cannot be self-attested. What
|
||||
git cannot prove is in the row's own docstring: history shows ORDER, never
|
||||
that nobody read the number before writing the threshold.
|
||||
|
||||
**Row 7's roster is pinned apart from the list it names.** The bar is a share,
|
||||
so a longer list is a lower bar per survivor. `MUTANT_ROSTER` and
|
||||
`MUTANT_COUNT` are separate constants, duplicates are refused outright, and
|
||||
the bar is taken from the roster's length.
|
||||
|
||||
**PM's G9 is decided: the term was observable and unobserved.** `hit =
|
||||
bool(hit_ids) and bool(confirmed)` is reached only by a delivery that still
|
||||
carries the citation and is no longer the concept file's bytes; every other
|
||||
mutation of the text empties `hit_ids` one step earlier. `M14` is that
|
||||
mutation, it is felled, and no production line changed.
|
||||
|
||||
**And that measures the judge's independence rather than asserting it.** PM
|
||||
found the judge reads the bundle through `consume.delivered_text`, the same
|
||||
function it judges. Measured both ways: with the index warmed BEFORE the patch
|
||||
(which is what the gate does, and what row 7's docstring already relied on)
|
||||
every unit comes back a miss with `confirmed False`; with the index built
|
||||
UNDER the patch every unit comes back a hit. The gate never builds one under a
|
||||
mutation. This is in `LIMITS` with the measurement, rather than closed by
|
||||
re-implementing a normalisation rule this repository already owns once.
|
||||
|
||||
## 5. The mutation run
|
||||
|
||||
Twelve mutants of the gate plus one control that must survive, run in a
|
||||
scratch copy of HEAD (`git archive` into `/tmp`, its own
|
||||
`uv sync --frozen --extra extract --group dev`), each judged by
|
||||
`tests/test_retrieval_gate.py`. The harness itself stays in the scratch copy:
|
||||
it rewrites the gate file and restores it, which is not a thing to keep in a
|
||||
tracked tree. What is recorded here is each mutant as a PROPERTY rather than
|
||||
as a line of code, so the table does not go stale the first time the gate is
|
||||
refactored.
|
||||
|
||||
| # | the guard it removes | felled by |
|
||||
|---|---|---|
|
||||
| X1 | row 8 counts whatever it was given, three sets or one | J2 |
|
||||
| X2 | row 8's headline is the two granularities summed | J2b |
|
||||
| X3 | row 2's denominator is the run's misses again | J10 |
|
||||
| X4 | row 3 judges only what this run withheld | J10 |
|
||||
| X5 | row 3 judges a run with no quota in force | J8 |
|
||||
| X6 | row 5 believes the registration about its own history | J1 |
|
||||
| X7 | row 5 accepts a registration no ranking change came after | the three-arm row-5 test |
|
||||
| X8 | row 7 takes the mutant list as given | J3 |
|
||||
| X9 | a delivery is a hit without checking the bundle's bytes (PM's G9) | the G9 test |
|
||||
| X10 | the synthetic corpus is not pinned | the `SPECS_SHA256` refusal |
|
||||
| X11 | row 9 ignores the set it was handed | the row-9 green test |
|
||||
| X12 | a K2 set of any size wears K2's name | the size refusal |
|
||||
| control | a detail line nobody asserts on is reworded | nothing — it MUST survive |
|
||||
|
||||
**First run: killed 11 of 12, and X2 SURVIVED.** Put row 8's headline back to
|
||||
the sum and all 64 tests stayed green. The cause was the fixture, not the row:
|
||||
three sets of one fasit entry each make "questions answered" and "units hit"
|
||||
the same number, so the test agreed with the mutant by construction — the same
|
||||
shape as round 158's X5. One set now carries two fasit entries, the only shape
|
||||
where the two readings differ.
|
||||
|
||||
**Second run, on the commit that fixed it: killed 12 of 12, control survived.**
|
||||
|
||||
## 6. What this round does NOT close
|
||||
|
||||
- **Row 3 can still be made green by an honest change that costs elsewhere.**
|
||||
Removing the quota makes every printed reason true; the price lands on row 1
|
||||
(8 of 9) and on `M05`, which is felled. The gate is the unit, not the row.
|
||||
- **The judge's parser is `consume`'s.** Bounded by measurement (§ 4) rather
|
||||
than removed.
|
||||
- **`SPECS_SHA256` is a digest of the corpus's fields, not of the bundle's
|
||||
bytes.** `build_bundle` is deterministic, so the two move together; that is
|
||||
an argument, not a measurement.
|
||||
- **Rows 8 and 9 still cannot be green without inputs this repository must
|
||||
not hold** — the three sets are on one machine and the K2 gold set exists
|
||||
nowhere. Both are operator questions and both are open.
|
||||
|
||||
## 7. Denominators, re-measured against the sources
|
||||
|
||||
| source | measured here 2026-09-19 |
|
||||
|---|---|
|
||||
| `claude-code-llm-wiki/eval/fase-sporsmaal.json` | sha `972d0f57…`, 20 questions, 29 fasit entries |
|
||||
| `vegnormal-okf/docs/2026-09-10-sk2-r761-sporsmal.json` | sha `c834a478…`, 8 entries, 7 positives + KN |
|
||||
| `vegnormal-okf/build/resonnering/sporsmal-m2.json` | sha `c6d9305a…`, 32 questions, 43 `must_cite` |
|
||||
| K2 | 6 questions recorded; no gold set anywhere |
|
||||
Loading…
Add table
Add a link
Reference in a new issue