fix(v1-gate): row 2 binds a round to the run's own artefact family, and says out loud what it still cannot prove
The 18.09 re-measurement took row 2 to 3 of 3 GREEN on a tree this product had never
run in: four handwritten outcome.json, four handwritten <run_id>-coverage.json in an
outbox the forger named in those same files, and os.utime for the ordering. "Round 0
must be a named real run" was implemented as "a file with that name exists" — which
touch satisfies. The attack is committed as a red test in b769537.
Three bindings, chosen because each removes one of the forger's three moves:
1. The outbox is DERIVED, never declared. It is <rounds-dir>/<n>/outbox/, and an
outcome.json that names one is refused by name. A path a round file chooses is a
path it can point at a directory the same hand just wrote.
2. "The run exists" now means the run's own artefacts agree WITH EACH OTHER on content
the gate recomputes (verify_run). Every evaluated approach has the proposal/outcome
pair write_outbox actually persists; outcome_type IS the coverage status, reason IS
its detail, a validated row's figure IS the proposal's own claimed_saving_nok, the
provenance stamp follows the same branch, and verdict_id is RE-MINTED here from the
proposal's own IR with the product's one minting rule (A5) rather than read. A
not_evaluated approach wrote neither file, and an artefact naming an approach the
coverage omits belongs to another run. verdicts._features_from_ir is made public for
this: a second private copy in the gate could drift from the rule the run stamped
with, which would turn the binding into a coincidence.
3. mtime decides nothing. The run's time is the round's declared ran_at (ISO-8601 with
zone, required). An mtime is not evidence — it is a filesystem attribute one utime
call sets.
What this does NOT do is prove a run happened. Nothing in a directory can: the outbox
writers are byte-deterministic and carry no clock by contract. So row 2 states its own
limit on every run instead of leaving GREEN to imply it (Row.attests / RUN_ATTESTATION):
that a run was actually made, and when, is the operator's to confirm. The cost of a
forgery moves from touch to reproducing the product's own artefact set, minting rule
included.
Measured, in a scratch copy, never in the work tree — four new mutants in this class,
each one line, each felled by the whole suite:
M7 if "outbox" in data: -> if False: 1 failed (declares its own outbox)
M8 if verdict_id != minted: -> if False: 1 failed (key is not the IR's)
M9 if strays: -> if False: 1 failed (artefact of another run)
M10 ran_at -> coverage mtime 18 failed (incl. the utime test)
Control, same scratch copy, unmutated: 1993 passed, 10 skipped, 5 xfailed.
Work tree, re-run after git add: uv run pytest -q -> 1998 passed, 5 skipped, 5 xfailed.
Gate: uv run python -m portfolio_optimiser.evals.v1_gate -> exit 1, row 2 RED (0 of 3).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
b769537830
commit
a362504108
5 changed files with 389 additions and 42 deletions
|
|
@ -250,7 +250,7 @@ def test_semretrieval_imports_no_network_modules() -> None:
|
|||
#
|
||||
# Every feature set carries ``description == measure_type``, which is exactly what both live
|
||||
# minting paths emit (``run._features_of`` sets ``description=proposal.measure``;
|
||||
# ``verdicts._features_from_ir`` sets ``description=ir["measure"]``).
|
||||
# ``verdicts.features_from_ir`` sets ``description=ir["measure"]``).
|
||||
|
||||
_MEASURE = "asfalt"
|
||||
_QUERY_CODES = frozenset({"05.1", "05.2"})
|
||||
|
|
@ -373,7 +373,7 @@ def test_minted_and_authored_verdicts_embed_identically_on_a_structural_tie() ->
|
|||
affected_codes=_CORRECT_CODES,
|
||||
measure_type=_MEASURE,
|
||||
claimed_saving_nok=200_000.0,
|
||||
description=_MEASURE, # the shape run._features_of / _features_from_ir emit
|
||||
description=_MEASURE, # the shape run._features_of / features_from_ir emit
|
||||
)
|
||||
minted = capture_verdict(features, "approved", "framework-captured ruling")
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue