The pin is doing its job: one new tracked file, and the arm that counts the git manifest
independently went red rather than letting the denominator drift. Row 3's fifteen guards
still measure zero hits, now over 516 files.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two rows and the matching CHANGELOG entries. The first records the measurement rather than
the decision: `--outbox-dir <d>/inni --run-id ../../ESCAPE` answered 0 and wrote two levels
above the directory the caller named, on a door whose directory argument was already
guarded. It also records the check that did NOT survive -- the containment half was
unreachable after the string rule and is written down as dead code removed, not as a second
layer of defence.
The second row is the smaller and older failure: a rule called load-bearing in prose, with
no arm that would notice if it were deleted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GREEN. `_checked_run_id` rejects a run id that carries a path -- a separator (`/`, `\`,
`os.sep`, `os.altsep`), a bare `.` or `..`, an absolute prefix, an embedded NUL, or the
empty string -- before any door touches the filesystem. A refusal, exit 3 with the reason
named, and no directory left behind: the call parsed, so 2 would be the wrong code.
Applied as a class. The composition is identical in all seven writers, so fixing the door
the escape was measured on would have left six the same shape.
One check, not two, and that is a measurement. The first draft also required the composed
path to resolve inside the resolved output directory; mutating that check away left all 36
arms green, because after the string rule there is no composition that can leave the
directory. An unreachable check is not defence in depth, it is dead code that reads like
defence, so it is gone -- and the property it claimed is asserted where it IS reachable,
in the probe's accepting arm.
Mutants, each run against the door probes with the tree restored from scratch in between:
guard dropped from one door -> 4 arms fall; whole guard disabled -> all 28 escape arms
fall; `--stop-reason` required -> default="" -> the new stop-reason arm falls, where it
previously survived the entire suite. The one that survived (the containment half) became
the finding above rather than a green tick.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
RED. Seven outbox doors compose `<out_dir>/<run_id>-<artefact>.json` and never look at the
run id, so `--outbox-dir <d>/inni --run-id ../../ESCAPE` writes two levels ABOVE the
directory the caller named and answers 0. The guard stood on one half of the path and was
missing on the other. Reproduced on write-prepass before this file existed.
The class is measured, not one case: all seven doors against four path-carrying forms --
relative-up, separator, bare `..`, absolute. Each arm asserts a non-zero exit AND that the
file tree under tmp_path is byte-unchanged afterwards; the first alone would be satisfied
by a door that refused after writing. An rc-0 control per door runs the same argv with a
clean run id, so the refusals measure the run id rather than a malformed call.
The file also carries the guard the `--stop-reason` rule never had. `required=True` is
called load-bearing in the ledger -- "the run finished" and "we never found out" must not
be the same value -- but swapping it for `default=""` passed the whole suite. The arm
fells that mutant: no flag is a usage error, and the empty string stays a value the caller
can supply on purpose.
Measured at this commit: 28 failed, 8 passed (the seven controls and the stop-reason arm).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The version moves 1.1.0 -> 1.2.0 in every place it is written: pyproject, the package
__init__, the smoke test that pins it, the README's wheel-install command (gate-pinned
against the build) and uv.lock -- five sites, plus the new CHANGELOG section. A new,
backward-compatible surface, no removals: fourteen run-path steps that had only ever been
reachable through a path that builds a chat client now each have a command of their own.
The README gains a status block at the top, and half of it is what v1 is NOT: no domain
expert has validated an output, three planned rounds of expert feedback were set aside
rather than passed, five of the eight kinds of feedback the design names have no typed way
back into a run, and the multi-agent debate is exercised in the suite only against scripted
clients. That half is the point -- a reader who installs this should learn the limits from
the first screen, not from a gate they have to run themselves.
Measured on this tree at this commit: suite 2238 passed / 0 failed / 5 skipped / 5 xfailed
(769 s), ruff check and mypy src clean, and both gates re-run after staging and unchanged
(v1 gate exit 1: 0/3 - 0/3 - 3/8 - no report - 3/8 - NOT MEASURED - 1/20; B gate exit 1:
15/17 - 0/2 - 15/15 - 0/3 - 4/5 - NOT MEASURED).
No tag is created here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The row the seam needs: why the outcome is derived rather than declared (the writer
branches on the TYPE, so a door taking it as an argument would let anyone author an
outbox of claims), why the outbox directory is always the caller's to name, why the
shape guard is a 3 and not a 2, and where the ground truth for each probe comes from.
It also writes down what did NOT move and why, so the next session does not rediscover
it: `rundebinding` and `rapport` share `round_builder`'s `main` with nothing in argv to
tell them apart, and `build_report` is a sub-step of `build_round` — a `report`
subcommand would be new capability, not a door over an existing step.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`_EXTERNAL_TODAY` and `_WITH_A_DOOR_TODAY` are the arm's own count, not the gate's, so
the seven new doors have to be written down here before the suite agrees they exist. The
two assertions that pinned `kjørekonfig`/`utboks`/`prepass-artefakt`/`parse-feil` as
door-LESS are inverted rather than deleted: they were the arm's discriminator, and a
door that silently went away must still turn this red.
The manifest count moves 514 -> 515 for the one new probe file.
Also a probe for the shape guard: the writers take `Mapping`/`Sequence` and would
serialize whatever they were handed, so a list where an object belongs would become a
valid file with the wrong shape and the run that reads it later would be the one to
fail. The door refuses it as a 3 (the call parsed, so 2 is the wrong code) and leaves no
directory behind. Carries its rc-0 control.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every one of the run path's seven `outbox.write_*` steps was reachable only through
`run.main`, and every path through that builds a chat client. The steps need no model:
they take already-rendered data and put it on disk in a byte-deterministic form.
Seven subcommands, seven thin adapters. The outbox directory is always the caller's to
name -- never a default, never the repository's own, because a step that wrote into a
folder the framework also reads as an inbox would bypass the Step-8 promotion gate.
`write-outbox` is the one that is not purely mechanical: `outbox.write_outbox` branches
on the outcome TYPE, so a door that took the outcome as an argument would let anyone
author an outbox of claims and hand it to Step 8 as results. The door DERIVES it through
`validate_proposal` -- the run path's own composition -- and a blocked proposal exits 3
with the artefacts still written, since that is where the rejection is recorded.
`verdict_id` stays an argument: `verdict-key` already owns that minting.
`--stop-reason` is required rather than defaulted to the empty string, inheriting the
core writer's measured reason: "the run finished" and "we never found out" must not be
the same value.
Eight probes, each a subprocess with the subcommand in argv, each asserting on the FILE
the command wrote. The ground truth is composed in the test -- the payload it wrote and
counted itself, and the byte form the contract requires -- never `outbox._dump`, which
would have measured the module against itself. The refusal arm carries its rc-0 control.
Measured, own run of the B gate: row 1 8 of 17 -> 15 of 17, exit 1 unchanged, no other
row moved. 0 chat-client names reachable from the toolbox (known-positive control: 24 in
run.py).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The ledger row for toolbox step 3: what was built, the row-1 movement (5 -> 8 of 17), the
int/float measurement that decided how the door reads a proposal, the exit-3 carrier for a
blocked proposal, the fasit each probe uses outside the door, and the stage the door does NOT
expose (P7 input grounding) with the reason.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The entries for validering / verdict-id / dom were console-script pointers at run.py::main, which
the gate itself refused with a named reason: the way from that main() requires a chat client.
They now point at the toolbox subcommand that reaches each symbol, with the probe that drives it.
Measured, same denominator both ways, no other row moved:
row 1 5 av 17 -> 8 av 17 (RED both ways)
row 2 0/2 · row 3 15/15 GREEN over 514 published files · row 4 0/3 · row 5 4/5 · row 6 NOT
MEASURED -- unchanged, exit 1 unchanged.
_EXTERNAL_TODAY follows the measurement, and _WITH_A_DOOR_TODAY names WHICH eight, so a door that
moves without being built shows up as a red test rather than as a different number. The arm that
asserted "validering has no door" had to pick a step that still does not: kjorekonfig.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
B's premise applied to the three steps that DECIDE a proposal: a proposal authored outside po --
by a human, or by an agent that is not po -- now meets the blocking deterministic gate, mints the
learning key, and is captured as a Verdict, all without a chat client on the way.
Three thin adapters, no second implementation. The reason is the one the first four doors were
built on, but it bites harder here: the refusal SENTENCE is fed back verbatim into the next
attempt by step 5, so a door that reworded it would break the repair loop while still looking
correct. The probes assert the sentence, not a substring two stages share.
One measurement decided a design detail. The IR writes whole magnitudes as JSON integers
(30000), the run path carries the pydantic float, and verdicts._mint_id hashes the raw value --
so minting from the undeclared JSON would hand out a DIFFERENT verdict id than the debate does
for the same proposal. The door therefore reads the proposal through SavingsProposal and feeds
model_dump() to the public features_from_ir; the probe pins both forms and asserts they differ,
so the shortcut cannot come back silently.
A blocked proposal exits 3, carrying the verdict rather than an exception envelope. "You asked
right and the answer is no" is the same fact whether a file was missing or a claim was
infeasible, and a caller that only reads the exit code must not see a blocked proposal as a
cleared one.
Fasit outside the door in every arm: the base's own checked-in golden suite (written before the
toolbox existed, so it cannot have been fitted to it), a cost baseline authored in the test, a
method cap computed by hand from the fixture, and the public minting rule. Each refusal arm has
an rc-0 control on an argv that would otherwise be accepted.
STATED LIMIT: the input-grounding stage (P7, stage 0b) has no flag here. It falsifies a proposal
against the rendered prompt the model received, and an outside caller has no such prompt.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The constant-sync gate is fail-closed on a name it cannot find, and it searches the package's top
level; `RUNBOOK_MIN_BODY` lives in `evals/b_gate.py`. Citing it as `NAME = value` therefore read as
a claim about a constant that does not exist. Same shape as the note at the top of this file about
the finding-99 row, and the same repair: the name in the span, the value in prose.
Caught by the suite, not by reading.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One row for the whole delivery: why `portfolio-optimiser-toolbox` is the third console script (it
is the one thing the other two cannot be used for), what each subcommand is bound to, why every
handler is a thin adapter and every dispatch an explicit branch, and how the probes assert on what
the door wrote rather than on the function it calls.
Plus the third repair of B-gate's binding with its measurements: the judge's seven forms at 0 of 1,
the naming rule turned into a discriminator, named-prose reasons, row 3 at 15 of 15 over 514 files,
and the runbook heading that is no longer a section. The row states its own limit (the gate reads
that the probe drives the door; it does not re-run the probe with the door broken) and records that
N7 stays open in the gate and closed in the suite.
The mutation run is in the row because two of its findings could not have come from reading: the
dead-code pruning survived two mutants until the arms that reach it were written, and three
mutants are named as non-measurements (two renamed a label, one was equivalent).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The measured escape: a runbook that was the five contract-named headings plus "x" (125 characters
in all), with a correctly checksummed attestation, read `2 av 2 GRØNN`. Only a file whose whole
content was "x" had been refused -- the rule asked whether the section NAME appeared in the text,
never whether anything stood under it. A table of contents is not a runbook anyone can follow.
Section names are now bound to a HEADING line, and what counts is the body beneath it
(`section_bodies`, 80 non-whitespace characters as a floor). The row says so itself, and says what
the floor is not: a length is never a measure of whether the runbook is true. That stays the
operator's, which is why row 6 is still `IKKE MÅLT`.
Two rc-0 controls in the suite carried section bodies of "noe" and "steg 1: naviger pakken" -- both
would now fail on the new rule rather than on the rule they were written for, so both got a real
body. A control that falls on the wrong rule has stopped controlling.
Also: the one very long line in the ledger (the arm name mid-paragraph) rewrapped. Cosmetic, named
in the 2026-09-20 checkpoint's leftovers.
NOT taken from that same list: freezing the outbox shape so the round-builder's denominator arm
stops skipping outside a tree with `scratchpad/`. It needs a checked-in fixture of the measured
form, which is not the "only if cheap" the order allowed, and the checkpoint proposed no round for
it either.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
M1 (`if False:` counted as live) and M2 (code after `return` counted as live) both SURVIVED the
whole suite. The pruning was real and nothing reached it: every dishonest form the arms already
carried fell on the data-dependence rule first, so removing the pruning changed no verdict.
The escape the pruning actually closes needs a probe that is GREEN in pytest: run a real process
that is NOT the door, bind the name the assert reads, and put the door call in dead code under
the same name. The taint then arrives from a line that can never run, and the probe "proves" a
step it never touched. Both forms are planted here, each with the honest rc-0 control beside it.
[mutation-ok] -- no guard is switched off. The `if False:` in this diff is INSIDE a Python string
that the arm writes to a tmp_path file; the gate only PARSES that file, never runs it, and the
arm asserts the form gives 0 of 1. The commit is the arm that makes dead-code pruning fall when
it is removed.
Found by mutating, not by reading. A rule no mutant can kill is a rule the suite has not measured.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`_PUBLISHED_TODAY` is the row-3 denominator, pinned in the suite against `git ls-files`. It read
512 and the arm was green in the worktree only because `toolbox.py` and `test_toolbox_doors.py`
were still untracked -- the clone measured 514 the moment they were committed. The pin is what it
is for: a surface that grows without anyone noticing is a surface the guard has not measured.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
B-gate row 1's premise, made callable. Every path through the framework CLI constructs a chat
client, so an outside caller -- a human at a terminal, or an agent that is NOT po -- could not
reach a single run-path step without paying for a model. These four steps need no model at all.
One CLI, four subcommands, one core call each:
navigate-bundle --bundle-dir -> okf.navigate_bundle
cost-baseline --bundle-dir --project-id -> okf.derive_cost_baseline
retrieve-chunks --query --docs-dir [--top-k] -> datasource.retrieve_chunks
prepass-admit --payload --bundle-dir [--dimension] -> prepass.admit_payload
Each handler is a thin adapter: strings in, the SAME function the run path calls, JSON on stdout,
and an exit code that says what happened (0 ran, 2 malformed call, 3 the step refused, named).
A handler that computed anything of its own would be a second implementation of a run-path step,
and the outside caller would stop getting what the debate gets.
Dispatch is an explicit branch per command, not argparse's `set_defaults(handler=...)`: the table
hides the one thing a reader wants to see, and B-gate row 1 asks the same question of the source
(it walks the call graph from `main` down to the step's symbol), where a callable in a Namespace
is a hop neither can follow.
Probes (`tests/test_toolbox_doors.py`, 10 arms): each starts the door as a SUBPROCESS with the
subcommand in argv and asserts on what it wrote -- never by importing the core function, which is
the whole difference the gate exists to measure. The yardstick is outside the door in every arm:
the filesystem (navigate-bundle, including the one deliberate outside-bundle link), a table
transcribed from the priced fixture (cost-baseline), the in-process seam it must equal byte for
byte (retrieve-chunks), and the producer's own checked-in payload (prepass-admit). Every refusal
arm has an rc-0 control beside it.
`portfolio-optimiser-toolbox` is the THIRD console script, and the pin test now says why: it is
the door the other two cannot be used for. README and CLAUDE.md updated with the command and the
reason it exists; every documented invocation was run.
Row 1: 1 -> 5 of 17 (four subcommands + `gate`, which the class fix in e47be68 stopped rejecting
on a name technicality). No other row moved; exit 1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Third repair of the same failure, and the first that does not read a NAME. The judge measured
row 1 to 17 of 17 with 17 one-line probes and a door module without a single import: "the probe
touches the door" was still a text check, one notch further in.
What is measured now, in the probe's own source: an EXECUTION SITE (a process starter carrying
the door's name in its arguments, or the entry imported from the door's module and called), and
an assert that is DATA-DEPENDENT on that call. Dead code is pruned first -- a call in a branch
that never runs, or after a return, is no execution. The honest forms the suite already uses
keep counting: the command built in a variable first, and the subprocess started in a helper
that returns it.
All seven forms the judge measured (a1, a2, b1, b2, c1, c2 + the one-liner recipe) now give
0 of 1, each with an rc-0 control in the same setup.
[mutation-ok] -- no guard is switched off. The two `if False:` lines in this diff are the
judge's b1 probe form planted as test DATA (a Python string written into a tmp_path file the
gate only PARSES, never runs) and the docstring naming why dead code is pruned. The commit adds
the rule that makes `if False:` around a door call stop counting.
Naming the step becomes a DISCRIMINATOR, as a class: it is required only when more than one step
stands behind the same door. Driving a door with a sole step IS driving that step -- which is why
`gate` (a real end-to-end door probe against v1_gate) was rejected on a name technicality. Row 1:
0 -> 1 of 17. For a subcommand door the command name must stand in what was actually executed,
and the CLI itself must be registered or `-m`-runnable: an add_parser in a module nobody can
start is no door.
held_out: a reason must be named prose. "-", "todo", "x", "." were all accepted as a reason by
`.strip()`; measured against the contract's own 28 reasons (shortest: 30 chars, three words).
Row 3: the five forms the checkpoint measured as open AND cheap to close are closed -- importlib
for both SDK names, deno/npm exec/yarn dlx, and the official TypeScript SDK. 12 of 12 -> 15 of 15,
still 0 hits over 512 published files. The limit sentence stops enumerating what remains ("EXACTLY
TWO", then "THREE" -- both falsified by the first new measurement) and says what the guard IS.
N7 stays open in the gate and is closed in the suite: a deleted off-path step still shrinks the
denominator silently there, so the four are written out in an arm that goes red when one goes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measured, not asserted: PM's P14 reproduced in BOTH directions (survived 43 of 43 pre-round arms,
felled by three arms after), 7 of 7 own mutants felled on AssertionError about behaviour with 0
IndexError, and the class the order named -- "an assert built from the binder's own table" --
counted at 4 of 12 rb.<table> assert sites and closed, with the independent side named for the
other eight.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Third writing of the same sentence, third time untrue. Measured in this working tree, by the arm
committed red before this one: FOUR of the five types outside the measured seven do not sit in an
outbox that also holds a coverage -- and for `plan-review` that says nothing at all, because the
type has 0 artefacts in the repo. `multibase` DOES sit there, in four outboxes
(`p17b-multibase/`, `p20-stress/`, `p21-stress/`, `p22-stress/`, all `lindaas`), and each of those
four holds TWO coverage files. That is why they fall outside the "exactly one coverage" rule the
union of seven is counted over -- and the same fact is the proof that seven is a FLOOR and not a
ceiling, which is the argument the false sentence was trying to make.
The comment above `_RUN_LEVEL_TYPES` carried the same claim in English and is corrected with it.
The arm now counts the outboxes and holds the ledger's row to the count, so a fourth writing is a
failing test rather than a review finding.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
RED FIRST, and only one of these can be red in the ordinary way: the ledger's own sentence about
the five artefact types outside the measured seven is UNTRUE, so the new arm that counts the
outboxes fails on ASSERT (1 of 44 arms red, 0 on ImportError/AttributeError/collection). The
other changes fix ARMS, not code -- src/ is untouched -- so their discrimination is shown by the
mutant campaign in the next commit's message: surviving before, felled after.
- `_STAGE_SENTENCES`: stage -> (the clause the refusal heading opens with, a word only that
stage's explanation carries), written out here the way `_STATUS_WORDS` is. The arm that names
where an approach fell built its expected heading from `rb.STAGE_PROSE` and then agreed with
it: PM's mutant P14 (the sentences for `stage4-p90` and `stage0b-grounding` swapped) passed all
43 arms, and the report then said an approach fell on the grounding when it fell on the
uncertainty calculation. Both halves are compared against this file's table, on the ONE heading
line located by prefix, and the whole table is covered -- not only the two stages the fixture
exercises, because a fixture is not a denominator.
- That each mark belongs to exactly ONE stage is COUNTED in the test, so swapping explanations
alone is a failure too.
- `_section(report, heading)`: every `split("## ...")[1]` in the file goes through it. Renaming a
heading failed four arms on 19.09, three of them on IndexError -- a traceback that says a list
was too short, not that the report lost the section the expert reads first.
- New arm: `multibase` DOES lie in outboxes that also hold coverage (four of them, two coverage
files each). The arm counts the outboxes and requires the ledger's sentence to say what the
count says; the false clause is pinned verbatim so its fourth writing is a failing test.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Row updated, not appended to: the B-gate invariant now carries both repairs of
19.09 and the numbers that moved in each. Two precision errors the checkpoint
named are corrected in place -- "no row became greener" is true of colour and not
of numbers (row 5 went 1 of 2 -> 4 of 5 when the unit was split), and ENTRY_KINDS
has three arts, not four.
What the new paragraph records: the probe binding and the two traps measured while
building it (substring, and a local module alias), the price in row 1 (3 of 17 ->
0 of 17, each of the three named with its reason), the limit statement that claimed
two remaining ways where six were measured -- four closed, three named and left
open with their reason -- the held_out door out of the denominator, the five small
holes closed and the two limits stated rather than closed, and the thirteenth
mutant: it survived twice, and the first arm written to fell it proved something
already true.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
First attempt at M12 still passed 86 of 86 -- the arm planted a docstring that
MENTIONED the module in prose, and the gate compares whole strings, so prose never
reached the check either way. The arm proved something already true and left the
mutant alive.
The cheapest cheat is the bare name: a string that IS the dotted module, standing
alone. That is what the arm plants now, and the same for the step's symbol.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Mutant M12 against the delivered tree: let _meaningful return the whole body, so a
string standing alone counts. 85 of 85 GREEN. That is a finding, not a pass: the
repair's own rule -- "the gate never reads a string that stands alone" -- had no arm
watching it, and without one the fix would have drifted back to "mentioning the door
is touching it", which is the same defect one floor in.
The arm carries its rc-0 control first (a real probe WITH a docstring still counts),
then breaks each half on its own: a probe whose docstring names the real door while
its call drives another module, and one whose docstring names the step while its call
passes another word. Both must be refused, each with its own reason.
12 of 12 mutants now felled, control 86 of 86.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Row 1 goes 3 of 17 -> 0 of 17. Nothing was removed from the product and no row
changed colour; the three that counted stopped counting because the contract they
satisfied was satisfiable without capability.
BEARING 1 -- the probe is bound to the step, and the binding is MEASURED in the
probe's own source. Chosen: read the probe (ast) rather than demand it live in a
contract-named file, because a file name is a convention a stub meets as easily as a
real probe. Three traits, each measured: it drives the DOOR (the dotted module or the
registered command name appears as a string it uses -- anywhere but a docstring,
because the honest form assembles argv in a variable first), it names the STEP (the
symbol, id or subcommand as a whole WORD in what it passes INTO a call or calls), and
it asserts at all. A probe claimed by two steps proves at most one and the gate cannot
tell which -- so neither.
Two traps found while measuring, both closed:
- substring: "gate" is not named by portfolio_optimiser.evals.v1_gate
- local alias: the first cut accepted step `gate` because the probe file imports the
module AS `gate`. Names are therefore read only where they are sent or called.
What that costs, measured against the contract that stands:
rundebinding drives the door, names no step (was green)
rapport never goes through the door at all (was green)
gate drives the door, names no step (was green)
BEARING 2 -- the limit statement said exactly TWO ways remained; the checkpoint
planted 21 call forms and measured SIX. Four are closed with a guard each (the
official Python SDK in both spellings, the node and uv runners, a dynamic import);
three remain and are now named: a runtime-composed name, a name from an environment
variable, a base64-decoded name. Left open deliberately -- the encodings are not
enumerable and our own contract stores base64 by design. Row 3: 9 of 9 -> 12 of 12,
still GREEN, 0 hits over 512 files. One of the three caught a command written in this
round's own test docstring; it was rewritten, not exempted.
BEARING 3 -- held_out accepted an EMPTY reason and shrank the denominator, while the
summary said "held out with a reason" either way. A blank reason is no reason: the
symbol stays in the denominator as a call without a door, the summary counts reasons,
and the four steps declared OUTSIDE the run path are now named one by one as having no
derived source instead of being counted in silence.
Five small rests, closed: a pruned manifest (451 of 512 was still GREEN) is now NOT
MEASURED, one sentinel per area the old handlist missed; a non-UTF-8 file is read as
byte text instead of counted and skipped; a symlink out of the tree is named and
fails the row; a runbook whose whole content is "x" no longer passes, the contract
names its sections; and the row states that its ratio is not a coverage measure.
Two stated, not closed, each with its reason in the row's own attestation: a po call
moved one floor down into a helper leaves the denominator (following helpers would
pull private ones in and make the denominator the curated list this row exists to
avoid), and row 3's k/n can still be padded by a guard with no measured escape behind
it. The ledger's two precision errors are corrected: "no row became greener" is true
of colour, not of numbers, and ENTRY_KINDS has three arts, not four.
Suite: 2172 passed, 5 skipped, 5 xfailed in 645 s. ruff and mypy clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
RED FIRST, second repair. PM's checkpoint on c1bdb37 measured N2b: row 1 goes from
3 of 17 to 17 of 17 GREEN with fourteen stub doors and ONE unrelated green test used
as the behaviour probe for all of them. The gate only looked up whether a nodeid was
`passed`; nothing bound the probe to its step. EXTERNAL_DOOR promised the probe
"calls the door and reads the artefact it writes" and there was no code for it --
the same class of defect as entry["kind"] one round ago.
20 new arms, 19 RED, every one on an assert about behaviour (43 assert lines, 0
ImportError, 0 AttributeError, 0 collection errors). The twentieth is green by
construction: every held_out entry in the contract does carry a reason today, and
the arm exists so that stops being true loudly.
Each refusal arm carries its own rc-0 control FIRST -- the same probe, the same
surface, accepted -- then breaks exactly one trait. The constructed probe is now
WRITTEN (`_write_probe`), not merely named: the repaired gate reads the probe's own
source, so a nodeid that does not exist cannot be a behaviour.
Red on the three bearing rests:
probe not bound to its step door untouched, step unnamed, nothing asserted,
test missing, one probe shared by two steps, and
N2b against the contract that stands (17 -> k=0)
limit statement untrue the six escapes PM measured as unseen, planted one
at a time, and the "TWO ways remain" sentence
held_out accepts empty a blank reason shrinks the denominator; the summary
counts entries, not reasons; the four steps outside
the run path are counted without being named
Red on five of the nine small rests: pruned manifest (451 of 512 still GREEN), a
non-UTF-8 file counted but never scanned, a symlink out of the tree, a runbook whose
whole content is "x", and the row's ratio read as a coverage measure.
One test-hygiene change, not a product fix: the git-dependent arm now skips itself
without .git instead of failing, so "red in a clean extract" stops growing with each
new arm of that class.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The round binder's row gets its two corrected denominators, and the first screen gets a row of
its own: what survived, why none of it was a code defect, and why that makes "red first"
impossible to satisfy with a commit rather than with a mutant run.
The 7-type sentence said "en ekte kjøring" and measured four archived runs. Re-counted: `run.py`
calls TEN of outbox.py's ten writers, so five more types exist; and the union over every outbox
in this repo with exactly one coverage (15 directories, of 25 coverage files in 20 directories)
is also 7. Seven is the floor the measurement gives, not a ceiling — the binder copies on a glob,
not on that list.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
RED FIRST, HONESTLY: 0 of the 3 new arms can be red at HEAD, because neither finding is a code
defect. The binder already printed every one of these things correctly; what was missing was an
arm that looked. There is no fix to be red before. The proof that the arms discriminate is the
mutant run, not this commit — and that run is reported here in full.
REPRODUCED AS SURVIVORS FIRST, against the 40 arms as they stood: the checkpoint's N3
(`_status_word`, "rejected" -> "validert") and N2 (the heading's amount truncated to whole
kroner) both passed 40 of 40. Eight more of my own against the same first screen: the verdict
word dropped entirely, the label paired with the NEXT row's word, `not_evaluated` read as
"avvist", `unsupported` read as "validert", the commissioned count off by one, and the heading's
"kroner" changed to "kr" — 8 of 10 survived. Two were caught (the sum off by one øre, the
overview sorted).
AFTER: 12 of 12 felled, every one on an AssertionError about behaviour, 0 on
ImportError/AttributeError/ModuleNotFoundError. Control unmutated: 43 of 43. Mutants lived only
in a scratch clone of `git archive HEAD`; the work tree was never mutated.
THE CAUSE WAS MEASURED, NOT GUESSED. `"60 000,01" in text` also matched the "Berørte
kostnadslinjer" line, which is built from `unit_cost` and which the mutant never touched. Counted
over the fixture's own report, 4 of this file's 11 positive substring-assert sites have their
needle on MORE THAN ONE line, and 3 of those 4 are asserts about one particular line. Those three
now compare the WHOLE line: `- **{label}** — {word}`, `### {label} — {amount} kroner`,
`### {label} — falt på {stage}`. The fourth claims only presence, and presence is what it
measures, so it still says `in`. The docstring's "cannot pass by accident" now says which half it
covers.
The word table is written out HERE rather than imported from `_status_word`: an arm that asks the
binder which word it chose and then agrees with it cannot tell "avvist" from "validert". It is
pinned to the MANDATE's own status list (`ApproachOutcome.status`, four values) the way the stage
vocabulary is pinned to the validator's — `_status_word` falls back on `str(row["status"])`, so a
status added there without a word here reaches the expert as a bare English identifier.
"validert" is a PREFIX of "validert, men uten erklært krav", which is exactly why no substring
assert can separate those two.
Also, two prose claims that promised more than the measurement:
- "Every artefact TYPE a real run leaves in an outbox" -> the types the runs MEASURED left.
`run.py` calls TEN of outbox.py's ten writers; `exploration`, `prepass`, `multibase`,
`plan-review` and `proposal-reviews` are five more types. Re-counted over every outbox in this
repo holding exactly one coverage — 15 directories, out of 25 coverage files in 20 directories
— the union is still 7, so 7 is a measured floor and not a ceiling. The arm is renamed to say
so.
- "Four of the seven are RUN-level" -> FIVE. `_RUN_LEVEL_TYPES` has five members.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The row claimed three things the checkpoint measured as false: that every denominator comes from
the source (row 1's M=13 was a curated list in the gate's own b_gate.json), that the surface is
what is published (it was a hand list of eleven roots seeing 433 of 512 files), and that the gate
counts an MCP-registered door as callable (there was no code for it — "kind" appeared 0 times).
The 435-file figure was the working tree, two gitignored .local.md files included, so it was not
reproducible from the commit either.
Rewritten to what is true after the repair, with the old numbers kept as history so the row reads
as a correction and not as a clean slate: 3 of 17 · 0 of 2 · 9 of 9 · 0 of 3 · 4 of 5 · IKKE MAALT.
The markdown-fence rule's own measurement is refreshed too: on the bigger surface it now carries
TWO prose lines, not one — the second is a plan document quoting a command the operator ran in his
own session.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three of the new row-3 arms pinned 512 files and the literal source label, which only a tree WITH
git metadata can answer; a clean extract (git archive | tar -x) would have turned all three red for
the same reason the three already-named extraction artefacts go red. The property they test holds
in both worlds, so they now assert it there: main.py, examples/, spikes/ and contexts/ are inside
the guard whichever manifest was read, and the row prints whatever denominator it measured.
The exact number and the label belong to ONE arm — the one whose subject IS the git manifest. In an
extract it is an extraction artefact of the same class as the others, and says so in its docstring.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Row 1's M was a curated list of 13 in the gate's OWN b_gate.json. It is now DERIVED: run_path_calls
reads what run.py::run_project actually calls (41 po-functions, re-counted here against PM's 39 —
the two that differ are generate_via_llm and fresh_workflow, held out because they need a chat
client). A call that is neither a declared step nor named-with-a-reason counts in the denominator
WITHOUT a door, so the number cannot shrink in silence: removing a step from the contract now
leaves N unchanged and names the orphan. The four outbox writers the run path uses and nobody had
declared (write_prepass, write_parse_failures, write_proposal_reviews, write_debate_tools) are
steps now; 28 calls are held out, each with its reason printed under the row.
A door must be REGISTERED and PROVEN. entry["kind"] is read (it was read 0 times before) and must
be one of three kinds the gate has code to verify: console-script in pyproject, module-main with
its own __main__ guard, subcommand registered in the module's argparse. On top of that every step
needs a named probe that calls the door and reads the artefact. The MCP sentence is struck from
EXTERNAL_DOOR: it claimed a capability with no code behind it.
Row 3 now scans the repo manifest (git ls-files, or the tree itself in an extract), not a hand
list of 11 roots: 512 published files instead of 433, so main.py, examples/, spikes/, contexts/,
CLAUDE.md and llms.txt are inside the guard for the first time. Three new patterns catch the
indirect invocations that walked past the old six — absolute path, list in a variable, constant,
shell string — 5 of 5 of PM's variants are refused now, with 0 false positives measured over the
whole surface. An empty surface is IKKE MAALT, not GREEN: the row demands a sentinel file and
prints the file count and the manifest it read.
Rows 4, 5 and 6 get sources for their denominators. Row 4 counts only checks whose named source
symbol resolves in the code. Row 5 counts each structural requirement on its own (2 profile
members + factory + seam + probe = 5) instead of collapsing three into one unit. Row 6's N comes
from the artefacts the contract names, and the attestation is VALIDATED: it must name the
contract's runbook, carry its sha256, say who ran it, and bear a real ISO date that is not in the
future — v1_gate's own date rule, reused, BOM tolerated as there.
Measured in a scratch clone (/tmp/claude-po/bgate-mut): 12 of 12 mutants felled, control 65 of 65.
All six of PM's broken attacks reproduced as refused, with the rc-0 control green.
No row got greener: 3 of 17 (was 3 of 13), 0 of 2, GREEN, 0 of 3, 4 of 5 (was 1 of 2), IKKE MAALT.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The PM checkpoint on 207337c judged the gate DELVIS: row 1's M=13 is a curated list in the
gate's OWN b_gate.json (the run path has 41 po-calls, 7 of 10 outbox writers), rows 4, 5 and 6
have denominators with no source at all, 6 of 10 cheat-attacks got through, and the
never-Claude guard sees 433 of 512 published files.
This commit is the red half. Every arm fails on an ASSERT about behaviour, never at collection:
the four names that do not exist yet (ENTRY_KINDS, run_path_calls, registered_entry,
published_files) are stubbed here with DELIBERATELY wrong values — everything is a door, the
run path calls nothing, the surface is empty — so each arm measures the defect rather than the
absence of a symbol.
30 av 31 red on assert. The one that is green is the rc-0 control
(test_a_valid_attestation_is_the_only_thing_that_turns_row6_green): a valid attestation must
turn row 6 green both before and after, or the row refuses everything, which proves as little
as refusing nothing. All 34 pre-existing arms stay green — measured, not assumed.
The planted claude-invocations are base64 in the test file for the same reason the contract's
patterns are: tests/ is itself part of the surface row 3 scans, and a cleartext variant here
would register as its own finding.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The round builder's row claimed "et sitat bærer ANTALLET siterte steder. Load-bearing
MÅLT" for a half that was not measured: the mutant dropping the count survived the whole
suite. The claim is true from today, and the row now says from WHEN — a ledger that
back-dates a measurement is worth less than one that admits the gap.
The new row records what the checkpoint found and what it cost to close: seven mutants
survived, none of them because the code was wrong, all of them because no arm looked.
Three were unreachable rather than merely unmeasured — the fixture wrote three of seven
artefact types, and every proposal carried the same citation stamp, so "1 av 1 siterte
steder" could not tell a dropped count from a kept one.
The denominator is written with its method, not as a number to be remembered: over the
four archived runs, -06/-07/-08 hold all seven types and -04 holds six (parse-failures is
written only when something failed to parse, so its absence is the signal). Union = 7,
from two commands.
And the citation cause is recorded as a measurement rather than a diagnosis, because
"the builder reads the wrong field" and "the outbox says the same thing five times"
want opposite fixes: one sha256 across all five proposals, in all four runs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A green mutation is a finding. Of the twelve planted against this file, eleven fell and
one survived: dropping `row["validated"]` from validated_ore's filter changed nothing.
It is equivalent — but only for as long as the OTHER guard holds. derive_outcome already
refuses to put a refused row's figure into (d), so on every outcome this builder writes,
validated_nok is None wherever validated is False and the two guards are
indistinguishable from the outside. Deleting either one alone is free today; deleting
both is the leak, and nothing witnessed that.
validated_ore is public and takes any outcome mapping, so the arm that separates them
hands it the outcome a future coverage writer could produce: a refused row whose amount
already sits in validated_nok. 137 000 002 øre, counted from the table, against the
227 000 002 a leak would give.
12 of 12 mutants now fall; control in the clone is 40 of 40.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The four arms that were red on assert, in the order the reader meets them.
A DANGLING SYMLINK as the round directory now refuses instead of tracebacking.
exists() FOLLOWS a link, so a dangling one answers False and slipped straight past
"a round is never overwritten"; the build then died on the filesystem's own
FileExistsError. is_symlink() is checked FIRST, the message says what a link would cost
(the round's content would sit somewhere the gate does not measure), and the link is
left exactly as it was found — nothing is written, exit code 1 like every other refusal.
ONE CITATION LIST SHARED BY EVERY PROPOSAL is now stated once. The cause was measured
before anything was written, because "the builder reads the wrong field" and "the outbox
says the same thing five times" want opposite fixes: in all four archived runs every
proposal carries a byte-identical 270-citation list — the run's whole retrieved context,
stamped once per proposal. No report can make that quote say something about the
individual measure. So when every proposal carries the same list, the report says so
once, says what the list actually is ("hva kjøringen leste, ikke hva det enkelte tiltaket
bygger på"), and drops the five copies. When the lists differ, nothing changes: the quote
and its COUNT stay under each proposal, which is where they mean something.
THE SAME COST LINE ON BOTH SIDES OF THE VERDICT is named where it happens. The 19.09
report refused TUN-LYS-01 under one label and validated the same line under another and
said nothing, so a reader met two figures for one budget line with no way to see they
collided. Both sides now carry the sentence, in the run's own row order.
A REMOVED APPROACH is shown by the label the expert saw, with the id in parentheses. It
is the one row whose human name is not in this run's coverage, so the label is read from
the coverage inside the PREVIOUS round's own outbox — derived from what the round already
carries, not a new column in outcome.json. An unreadable coverage falls back on the bare
id; a missing label is not a reason to refuse a round.
39 of 39 arms green. Nothing here touches what the gate reads: outcome.json keeps its
four columns and the builder still never writes the attestation.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The 19.09 checkpoint planted 20 mutants against this file and SEVEN survived the whole
suite: the report's round number, an artefact type outside proposal/outcome/coverage
dropped in silence, a never-evaluated approach left off its own list, the source quote
removed entirely, the quote's COUNT removed, the affected cost lines removed, and a
refused row's amount counted into the validated total. None of them survived because the
builder was wrong — it does all seven things correctly. They survived because no arm in
709 lines ever looked (grep for citation/cost-line words gave 2 hits, both in the
fixture, none in an assert). An unwitnessed seam is a seam the next edit may delete for
free.
Two fixture defects made three of those blind spots unreachable rather than merely
unmeasured:
* THE DENOMINATOR. A real run leaves SEVEN artefact types; the fixture wrote three.
Counted here, not remembered: over the four archived runs the checkpoint read
(tunnel-hauglia-2027-04/-06/-07/-08), each <run_id>-<rest>.json is typed proposal /
outcome when <rest> ends there and <rest> itself otherwise. -06, -07 and -08 hold all
seven (15 files each); -04 holds six (12 files, no parse-failures — that file is
written only when something failed to parse, so its absence is the signal). Union =
7, written by two commands: run.py writes six, stress.py writes -verdict.json. Six
are now written through the product's own writers; -verdict.json is bytes, because
the builder never reads it and that difference is stated rather than hidden.
* THE CITATIONS. Every proposal carried an identical stamp, so "1 av 1 siterte steder"
could not tell a dropped COUNT from a kept one. Counts are now distinct per approach,
and a shared-list mode reproduces what the real runs actually do.
The refused row now carries an amount, and no archived run does: measured on the same
four outboxes, every rejected row has saving_nok = None. The guard that only a validated
row's figure becomes validated_nok therefore aims at a coverage writer that does not
exist yet — and a fixture that cannot produce the number cannot witness the guard at all.
4 of the 11 new arms are RED on ASSERT, all four on behaviour that does not exist yet:
a dangling symlink as the round directory (exists() follows the link, so the build dies
on the filesystem's own FileExistsError instead of a sentence), the identical citation
list stated once instead of five times, the same cost line named on both sides of the
verdict, and a removed approach shown by the label the expert saw instead of a raw id.
The other 7 pin behaviour that is already correct; they are not red here, and the proof
that they discriminate is the mutant run, not this commit.
The cause of the identical quote was MEASURED before anything was written, because
"the builder reads the wrong field" and "the outbox says the same thing five times" want
opposite fixes: all five proposals in tunnel-hauglia-2027-08 carry byte-identical
270-citation lists (one sha256 across all five), and so do -04, -06 and -07. It is the
run's whole retrieved context stamped once per proposal. The report cannot make that
quote informative; it can stop repeating it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Repo convention: every measured decision gets a row in the ledger with the load-bearing test that
turns red when the decision is undone. The row records the three choices the gate states in its
own output (CLI door over MCP, no budget ceiling in toolbox mode, row 6 never green without the
operator), the base64 reason, and the eight mutants that fell in the scratch clone — including
M-6, which shows row 3's green is a measurement and not a vacuous zero: without the
fenced-block rule one prose line in a research doc turns the row red.
v1 gate re-measured after the work: 0/3 · 0/3 · 3/8 · ingen rapport · 3/8 · IKKE MÅLT · 1/20 —
unchanged. PLAN.md § Ferdig-kriteriet untouched, as the order required.
Suite after git add: 2106 passed / 5 skipped / 5 xfailed, rc=0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
python -m portfolio_optimiser.evals.b_gate — one command, offline, no model call, exit 1 today:
1 steg i kjørestien kallbare utenfra 3 av 13 RØD
2 roller som kan leveres utenfra 0 av 2 RØD
3 vakter mot en vei fra po til Claude 6 av 6 GRØNN (435 published files)
4 løpet drevet uten et eneste modellkall 0 av 3 RØD
5 Foundry-veien urørt og samme artefaktfamilie 1 av 2 RØD
6 kjøreboka finnes og er kjørt 0 av 2 IKKE MÅLT
Every denominator is read off the source, never off a list in the gate. Row 1 counts the steps
of the run path that resolve to a symbol AND have a call site; a step is externally callable only
when a CLI (or MCP-registered) entry reaches it without any chat-client construct on the way —
which is why the ten run.py steps are red and round_builder's two plus the v1 gate are green. Row
2 reads the roles off workflow._MAKER_CHECKER_ROLES. Row 3's patterns each carry a known-positive
AND a known-negative fixture, so a guard that cannot hit is not counted as a zero.
Three decisions the operator cannot answer without reading code, made here and stated in the
gate's own output:
* the external door is a CLI subcommand, not MCP — po already has five main() and two console
commands, and MCP would need a server the run path does not have. The gate still counts an
MCP-registered door, so the choice does not bind the next order.
* the budget guard in B is NOT po's: BudgetMiddleware is fail-closed on missing usage and is
never constructed without a chat client, so keeping it here would turn fail-closed into
fail-open. The ceiling in B is the Claude Code session's own spend, which po neither sees nor
steers. The Foundry path keeps its ceiling unchanged.
* row 6 is IKKE MÅLT, never green, until the operator attests that the runbook actually drove an
analysis — a file the gate never writes, the same rule as the v1 gate's attestation.
Row 3's pattern text is base64 in the config so the contract cannot register as its own finding;
that is what lets the row run without an exclusion list, and a row without exclusions is a row
nobody can switch off by adding a filename.
Suite after: 2106 passed / 5 skipped / 5 xfailed (was 2072/5/5; +34 new, none changed).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Operator decision 19.09.2026: in development and test Claude Code LEADS and portfolio-optimiser
is the toolbox. po never calls Claude; production stays on Foundry. This commit writes the
measurement RED — the contract first, the capability later.
Six rows, each with a denominator taken from the SOURCE and counted independently here:
1. toolbox complete — every deterministic step of the run path, anchored to (module, symbol) and
to the scope that calls it; k = steps reachable from a CLI entry WITHOUT a chat client.
2. what the model delivered can be delivered from outside — denominator read off
workflow._MAKER_CHECKER_ROLES, k = roles with passing named probes.
3. po has no path to Claude — six patterns, each carrying its own known-positive AND
known-negative fixture. Pattern text is base64 in the config so the gate's own contract cannot
register as a hit against the surface it scans.
4. no model calls in toolbox mode — three named probes.
5. the Foundry path untouched — profiles, factory and injection seam read from source, plus a
schema-comparison probe.
6. the runbook exists — NOT MEASURED until the operator attests, and the gate never writes that
attestation itself.
Red now for the reason that matters: the tests fail to import a module that does not exist yet.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Found while killing mutants: an ordering made to depend on hash() survived
test_two_builds_of_the_same_outbox_are_byte_identical untouched. One interpreter has one hash
seed, so the two builds agreed with each other and with nothing else -- the arm could not tell a
deterministic report from a report that is merely consistent within a process. The ordering arm
caught that mutant, so the seam was witnessed; this one was not witnessing it.
The second build now runs as a subprocess with PYTHONHASHSEED=1. Re-measured with the same
mutant: both arms red, where before only one was.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
python -m portfolio_optimiser.evals.round_builder --outbox <dir> --round <n> --ran-at <ISO>
writes <rounds-dir>/<n>/ with the run's artefacts COPIED in, outcome.json derived from that
copy, and report.md -- the one artefact in a round a domain expert reads and corrects. Round 0
of the v1 criterion can now be made; it counted 0 of 3 because it could not be, which is a
different failure from a round nobody had held.
What it derives it derives with the gate's own functions rather than a second copy: verify_run
decides whether the run stands up to itself (an artefact contradicting its coverage row, a
half-missing family and a stray artefact are all refused AT THE SOURCE, before a byte is
written), stage_of gives column (c), row_changed gives the report's "changed since the previous
round", parse_time refuses a stamp without a zone, safe_rounds_dir refuses a round directory the
repo would commit. The validated total is ledger.to_ore per amount, summed as integers.
Two things it never does, and both are the point. It never writes the operator's attestation --
the gate stops at FORM OK without one, and that is correct, because no arrangement of files can
witness that a run happened. And it never invents: --ran-at is required because no outbox
artefact carries a clock, and feedback_ids stays empty because no run records which feedback
item produced which row. The report says "ingen tilbakemelding forklarer dette" on every changed
row rather than hiding that model noise and an answered objection look alike.
Chosen and why: --ran-at as a required argument rather than the coverage file's mtime, because
an mtime is a filesystem attribute one call sets and reading it as evidence made row 2 green on
a tree nothing had run in (18.09). The report carries no raw stage identifier -- every stage
sentence is "<short name>: <explanation>" so the one-line diff of what changed has words a
reader can act on. A citation shows its COUNT, because a run that cited 446 places and one that
cited one must not look the same.
[skip-docs]: the ledger row is in docs/invarianter.md, which is where this repo's rules live.
README is the product's front door and this is an operator tool behind `python -m`, the same
class as costsim/hitl/preflight, which README deliberately does not carry; v1-rounds/ is
gitignored internal machinery and the gate itself is not in README either. CLAUDE.md was emptied
of exactly this kind of row in session 130 and is not the place to put one back.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Round 0 of the v1 criterion cannot be made today: nothing binds a finished run's outbox to
<rounds-dir>/<n>/, and nothing in src writes markdown a domain expert could read. These tests
state what a builder has to do before one exists, and every number they assert is counted a
second time from the fixture's own table rather than read back from the builder.
Red on assertions, not on import: round_builder.py lands as a contract -- dataclass, signatures,
neutral returns -- so each test fails in its own body.
Two gate helpers become public rather than being copied: row_changed (the report's "changed since
the previous round" section must not disagree with the gate about what changed) and
safe_rounds_dir (the builder CREATES the directory the gate only reads, and the writer is where a
leak of the expert's feedback has to be stopped).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
K5 survived a full run (a mutant ignoring an extra or removed file left the suite green) and K4
(sort removed) was caught only by corpus data. Now pinned with no corpus: the add arm, the remove
arm, and a shuffled directory listing digesting the same. The gate says "pin-drift" for a drifted
copy instead of "artefakter mangler"; the unread "store" key is gone from frozen_bundles.json; the
forbidden-path scan covers md/yaml/yml/toml and proves a known positive per suffix.
Five mutants killed in a scratch clone (drop-last-entry 8 failed, no-sort, no-names, drift-as-missing,
scan-suffixes-narrowed 1 failed each).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Chosen: a key written twice is RED (PM recommendation: two answers to one question are not an answer). A time without an offset is RED as ambiguous. A hard link is NOT refused but declared a limit in the output (content binds round, run and date either way). A BOM is tolerated. The clock is a parameter (now=), read by ONE check.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The PM checkpoint of 18.09 paid M-6's price three times and row 2 still read 3 of 3 GREEN:
a whole artefact family written by hand and made internally consistent (verdict_id minted
with the product's own rule, about sixty lines of script), four REAL runs' artefacts under
a handwritten feedback file, and `<n>/outbox` symlinked out of the round to a real run
elsewhere. `RUN_ATTESTATION` stood in the output the whole time, and the row was green
anyway — which is the finding: a row that PRINTS its limit and reports GREEN has not
stated a limit, it has decorated one.
One of the three is a hole and is closed; two are not holes at all.
1. The outbox must lie in the round ITSELF (`outbox_escape`). Derived is only derived if
the filesystem cannot redirect it: `<n>/outbox` as a symlink, or any symlinked artefact
inside it, is refused by name.
2. Rows 1 and 2 can no longer be green from files. What the computation gives is
`FORM_OK` ("FORM OK, IKKE BEVIST"): every check the gate knows how to make passed, the
round counts 0 against the criterion, and the exit code still fails — it is un-green in
every way RED is. What separates the two is what the reader must do next, not what the
gate concludes.
3. Green is `<n>/attestering.txt`, per round: the operator's own statement, naming the
round, the run and the date. Plain text with three keyed lines, deliberately not JSON —
it is the one artefact in this contract a PERSON types, and a missing brace must never
be able to read as "not confirmed". Nothing in the product writes one, and a test in
`src/` says so: a product that can produce a witness to its own run has produced
exactly the thing these rows cannot.
A MISSING attestation is FORM OK: nobody has confirmed anything yet, and the gate is
not entitled to call that a lie. A PRESENT one that names another round or run, carries
no date, an unparsable date, or a date before the run, is RED: it is a statement about
something this is not. Row 2 needs round 0 attested as well, since round 1 is measured
against it — an unattested baseline is an unattested comparison.
The attestation is read raw off `outcome.json` (`_declared_run`), not through
`read_outcome`, so row 1 — which reads no run — does not inherit row 2's artefact checks
through the back door.
Nothing moved on the criterion: 0/3 · 0/3 · 3/8 · no report · 3/8 · IKKE MÅLT · 1/20,
exit 1. What changed is that rows 1-2 now name what they are waiting for, and it is not
another file.
Suite after `git add` on a clean tree: `uv run pytest -q` = 2014 passed, 5 skipped,
5 xfailed (508 s). The attack of b769537/ab00016 is the red half of this commit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
M-6 moved the price of a forgery from `touch` to reproducing the product's own artefact
set. The PM checkpoint of 18.09 then paid that price three ways, and row 2 read 3 of 3
GREEN each time:
F1 the whole artefact family written by hand and made internally consistent — coverage,
one proposal/outcome pair per approach, `verdict_id` minted with the product's own
rule. About sixty lines of script. `RUN_ATTESTATION` stood in the output; the row was
green anyway.
F2 four REAL runs' artefacts under a handwritten feedback file.
F4 `<n>/outbox` made a symlink to a real run's directory somewhere else, so a DERIVED
path was redirected by the filesystem.
F4 is a hole. F1 and F2 are not: no arrangement of files can be told apart from a round
that happened, because what is missing is not a check but a WITNESS. These tests say so.
Red here, 16 of them:
- rows 1 and 2 must stop at `gate.FORM_OK` — 0 against the criterion, exit still 1 — on a
tree where every computed check passes but nobody has attested the rounds (F1, F2);
- F2 is built through `outbox.write_outbox` and `write_coverage`, the exact bytes a real
run leaves behind, so it doubles as the control that `verify_run` reads the PRODUCT's
output and not merely the shape this test file writes;
- an outbox that is a symlink out of the round, and an artefact symlinked into one, are
refused by name (F4);
- the step from FORM OK to GREEN is `<n>/attestering.txt`, per round, round 0 included
because row 2 measures round 1 against it: present and matching -> GREEN (the rc-0
control, asserted first in every arm), missing -> FORM OK, present but naming another
round, another run, no date, an unparsable date or a date before the run -> RED;
- nothing in `src/` may write that file: a product that can produce a witness to its own
run has produced exactly the thing rows 1-2 cannot.
One more arm, green on arrival, and that is the finding: the guard at `v1_gate.py:415`
(`keyed != {(run_id, aid)}`), pinned to a constant-false branch, left the suite at 68
passed, identical to baseline. The guard bit, nothing read it — so
`_mut_the_family_is_labelled_for_another_run` gives the pair the right FILENAME and
another run's labels inside.
The fixture now writes the operator's attestation on every round it builds, and the
file's name is pinned in the test file as well (`_ATTEST_FILE`), the way M-5 pins the
rest of the contract: renaming it in the gate alone must fail a test.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The 18.09 re-measurement took row 2 to 3 of 3 GREEN on a tree this product had never
run in: four handwritten outcome.json, four handwritten <run_id>-coverage.json in an
outbox the forger named in those same files, and os.utime for the ordering. "Round 0
must be a named real run" was implemented as "a file with that name exists" — which
touch satisfies. The attack is committed as a red test in b769537.
Three bindings, chosen because each removes one of the forger's three moves:
1. The outbox is DERIVED, never declared. It is <rounds-dir>/<n>/outbox/, and an
outcome.json that names one is refused by name. A path a round file chooses is a
path it can point at a directory the same hand just wrote.
2. "The run exists" now means the run's own artefacts agree WITH EACH OTHER on content
the gate recomputes (verify_run). Every evaluated approach has the proposal/outcome
pair write_outbox actually persists; outcome_type IS the coverage status, reason IS
its detail, a validated row's figure IS the proposal's own claimed_saving_nok, the
provenance stamp follows the same branch, and verdict_id is RE-MINTED here from the
proposal's own IR with the product's one minting rule (A5) rather than read. A
not_evaluated approach wrote neither file, and an artefact naming an approach the
coverage omits belongs to another run. verdicts._features_from_ir is made public for
this: a second private copy in the gate could drift from the rule the run stamped
with, which would turn the binding into a coincidence.
3. mtime decides nothing. The run's time is the round's declared ran_at (ISO-8601 with
zone, required). An mtime is not evidence — it is a filesystem attribute one utime
call sets.
What this does NOT do is prove a run happened. Nothing in a directory can: the outbox
writers are byte-deterministic and carry no clock by contract. So row 2 states its own
limit on every run instead of leaving GREEN to imply it (Row.attests / RUN_ATTESTATION):
that a run was actually made, and when, is the operator's to confirm. The cost of a
forgery moves from touch to reproducing the product's own artefact set, minting rule
included.
Measured, in a scratch copy, never in the work tree — four new mutants in this class,
each one line, each felled by the whole suite:
M7 if "outbox" in data: -> if False: 1 failed (declares its own outbox)
M8 if verdict_id != minted: -> if False: 1 failed (key is not the IR's)
M9 if strays: -> if False: 1 failed (artefact of another run)
M10 ran_at -> coverage mtime 18 failed (incl. the utime test)
Control, same scratch copy, unmutated: 1993 passed, 10 skipped, 5 xfailed.
Work tree, re-run after git add: uv run pytest -q -> 1998 passed, 5 skipped, 5 xfailed.
Gate: uv run python -m portfolio_optimiser.evals.v1_gate -> exit 1, row 2 RED (0 of 3).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The 18.09 re-measurement of 9825b26 made row 2 read 3 of 3 GREEN without a run existing
anywhere: four handwritten outcome.json, four handwritten <run_id>-coverage.json in an outbox
named by those same files, and os.utime for the ordering. read_outcome does cross-check the
outcome against the coverage — but `outbox` is a free path from the round file and the coverage
file is written by the same hand, so "round 0 must be a named real run" is implemented as "a file
with that name exists", which touch satisfies.
The attack is rebuilt in the test rather than driven through the _outcome fixture, so the
fixtures can be raised to a whole run family without the attack drifting with them. Row 1 is
asserted GREEN on the same tree: the feedback IS well formed there, which is what makes this an
attack on row 2 and not a broken fixture.
RED as committed (3, 'GRØNN') != (0, 'RØD') — the fix is the next commit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
