M1 (`if False:` counted as live) and M2 (code after `return` counted as live) both SURVIVED the
whole suite. The pruning was real and nothing reached it: every dishonest form the arms already
carried fell on the data-dependence rule first, so removing the pruning changed no verdict.
The escape the pruning actually closes needs a probe that is GREEN in pytest: run a real process
that is NOT the door, bind the name the assert reads, and put the door call in dead code under
the same name. The taint then arrives from a line that can never run, and the probe "proves" a
step it never touched. Both forms are planted here, each with the honest rc-0 control beside it.
[mutation-ok] -- no guard is switched off. The `if False:` in this diff is INSIDE a Python string
that the arm writes to a tmp_path file; the gate only PARSES that file, never runs it, and the
arm asserts the form gives 0 of 1. The commit is the arm that makes dead-code pruning fall when
it is removed.
Found by mutating, not by reading. A rule no mutant can kill is a rule the suite has not measured.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`_PUBLISHED_TODAY` is the row-3 denominator, pinned in the suite against `git ls-files`. It read
512 and the arm was green in the worktree only because `toolbox.py` and `test_toolbox_doors.py`
were still untracked -- the clone measured 514 the moment they were committed. The pin is what it
is for: a surface that grows without anyone noticing is a surface the guard has not measured.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
B-gate row 1's premise, made callable. Every path through the framework CLI constructs a chat
client, so an outside caller -- a human at a terminal, or an agent that is NOT po -- could not
reach a single run-path step without paying for a model. These four steps need no model at all.
One CLI, four subcommands, one core call each:
navigate-bundle --bundle-dir -> okf.navigate_bundle
cost-baseline --bundle-dir --project-id -> okf.derive_cost_baseline
retrieve-chunks --query --docs-dir [--top-k] -> datasource.retrieve_chunks
prepass-admit --payload --bundle-dir [--dimension] -> prepass.admit_payload
Each handler is a thin adapter: strings in, the SAME function the run path calls, JSON on stdout,
and an exit code that says what happened (0 ran, 2 malformed call, 3 the step refused, named).
A handler that computed anything of its own would be a second implementation of a run-path step,
and the outside caller would stop getting what the debate gets.
Dispatch is an explicit branch per command, not argparse's `set_defaults(handler=...)`: the table
hides the one thing a reader wants to see, and B-gate row 1 asks the same question of the source
(it walks the call graph from `main` down to the step's symbol), where a callable in a Namespace
is a hop neither can follow.
Probes (`tests/test_toolbox_doors.py`, 10 arms): each starts the door as a SUBPROCESS with the
subcommand in argv and asserts on what it wrote -- never by importing the core function, which is
the whole difference the gate exists to measure. The yardstick is outside the door in every arm:
the filesystem (navigate-bundle, including the one deliberate outside-bundle link), a table
transcribed from the priced fixture (cost-baseline), the in-process seam it must equal byte for
byte (retrieve-chunks), and the producer's own checked-in payload (prepass-admit). Every refusal
arm has an rc-0 control beside it.
`portfolio-optimiser-toolbox` is the THIRD console script, and the pin test now says why: it is
the door the other two cannot be used for. README and CLAUDE.md updated with the command and the
reason it exists; every documented invocation was run.
Row 1: 1 -> 5 of 17 (four subcommands + `gate`, which the class fix in e47be68 stopped rejecting
on a name technicality). No other row moved; exit 1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Third repair of the same failure, and the first that does not read a NAME. The judge measured
row 1 to 17 of 17 with 17 one-line probes and a door module without a single import: "the probe
touches the door" was still a text check, one notch further in.
What is measured now, in the probe's own source: an EXECUTION SITE (a process starter carrying
the door's name in its arguments, or the entry imported from the door's module and called), and
an assert that is DATA-DEPENDENT on that call. Dead code is pruned first -- a call in a branch
that never runs, or after a return, is no execution. The honest forms the suite already uses
keep counting: the command built in a variable first, and the subprocess started in a helper
that returns it.
All seven forms the judge measured (a1, a2, b1, b2, c1, c2 + the one-liner recipe) now give
0 of 1, each with an rc-0 control in the same setup.
[mutation-ok] -- no guard is switched off. The two `if False:` lines in this diff are the
judge's b1 probe form planted as test DATA (a Python string written into a tmp_path file the
gate only PARSES, never runs) and the docstring naming why dead code is pruned. The commit adds
the rule that makes `if False:` around a door call stop counting.
Naming the step becomes a DISCRIMINATOR, as a class: it is required only when more than one step
stands behind the same door. Driving a door with a sole step IS driving that step -- which is why
`gate` (a real end-to-end door probe against v1_gate) was rejected on a name technicality. Row 1:
0 -> 1 of 17. For a subcommand door the command name must stand in what was actually executed,
and the CLI itself must be registered or `-m`-runnable: an add_parser in a module nobody can
start is no door.
held_out: a reason must be named prose. "-", "todo", "x", "." were all accepted as a reason by
`.strip()`; measured against the contract's own 28 reasons (shortest: 30 chars, three words).
Row 3: the five forms the checkpoint measured as open AND cheap to close are closed -- importlib
for both SDK names, deno/npm exec/yarn dlx, and the official TypeScript SDK. 12 of 12 -> 15 of 15,
still 0 hits over 512 published files. The limit sentence stops enumerating what remains ("EXACTLY
TWO", then "THREE" -- both falsified by the first new measurement) and says what the guard IS.
N7 stays open in the gate and is closed in the suite: a deleted off-path step still shrinks the
denominator silently there, so the four are written out in an arm that goes red when one goes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
First attempt at M12 still passed 86 of 86 -- the arm planted a docstring that
MENTIONED the module in prose, and the gate compares whole strings, so prose never
reached the check either way. The arm proved something already true and left the
mutant alive.
The cheapest cheat is the bare name: a string that IS the dotted module, standing
alone. That is what the arm plants now, and the same for the step's symbol.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Mutant M12 against the delivered tree: let _meaningful return the whole body, so a
string standing alone counts. 85 of 85 GREEN. That is a finding, not a pass: the
repair's own rule -- "the gate never reads a string that stands alone" -- had no arm
watching it, and without one the fix would have drifted back to "mentioning the door
is touching it", which is the same defect one floor in.
The arm carries its rc-0 control first (a real probe WITH a docstring still counts),
then breaks each half on its own: a probe whose docstring names the real door while
its call drives another module, and one whose docstring names the step while its call
passes another word. Both must be refused, each with its own reason.
12 of 12 mutants now felled, control 86 of 86.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Row 1 goes 3 of 17 -> 0 of 17. Nothing was removed from the product and no row
changed colour; the three that counted stopped counting because the contract they
satisfied was satisfiable without capability.
BEARING 1 -- the probe is bound to the step, and the binding is MEASURED in the
probe's own source. Chosen: read the probe (ast) rather than demand it live in a
contract-named file, because a file name is a convention a stub meets as easily as a
real probe. Three traits, each measured: it drives the DOOR (the dotted module or the
registered command name appears as a string it uses -- anywhere but a docstring,
because the honest form assembles argv in a variable first), it names the STEP (the
symbol, id or subcommand as a whole WORD in what it passes INTO a call or calls), and
it asserts at all. A probe claimed by two steps proves at most one and the gate cannot
tell which -- so neither.
Two traps found while measuring, both closed:
- substring: "gate" is not named by portfolio_optimiser.evals.v1_gate
- local alias: the first cut accepted step `gate` because the probe file imports the
module AS `gate`. Names are therefore read only where they are sent or called.
What that costs, measured against the contract that stands:
rundebinding drives the door, names no step (was green)
rapport never goes through the door at all (was green)
gate drives the door, names no step (was green)
BEARING 2 -- the limit statement said exactly TWO ways remained; the checkpoint
planted 21 call forms and measured SIX. Four are closed with a guard each (the
official Python SDK in both spellings, the node and uv runners, a dynamic import);
three remain and are now named: a runtime-composed name, a name from an environment
variable, a base64-decoded name. Left open deliberately -- the encodings are not
enumerable and our own contract stores base64 by design. Row 3: 9 of 9 -> 12 of 12,
still GREEN, 0 hits over 512 files. One of the three caught a command written in this
round's own test docstring; it was rewritten, not exempted.
BEARING 3 -- held_out accepted an EMPTY reason and shrank the denominator, while the
summary said "held out with a reason" either way. A blank reason is no reason: the
symbol stays in the denominator as a call without a door, the summary counts reasons,
and the four steps declared OUTSIDE the run path are now named one by one as having no
derived source instead of being counted in silence.
Five small rests, closed: a pruned manifest (451 of 512 was still GREEN) is now NOT
MEASURED, one sentinel per area the old handlist missed; a non-UTF-8 file is read as
byte text instead of counted and skipped; a symlink out of the tree is named and
fails the row; a runbook whose whole content is "x" no longer passes, the contract
names its sections; and the row states that its ratio is not a coverage measure.
Two stated, not closed, each with its reason in the row's own attestation: a po call
moved one floor down into a helper leaves the denominator (following helpers would
pull private ones in and make the denominator the curated list this row exists to
avoid), and row 3's k/n can still be padded by a guard with no measured escape behind
it. The ledger's two precision errors are corrected: "no row became greener" is true
of colour, not of numbers, and ENTRY_KINDS has three arts, not four.
Suite: 2172 passed, 5 skipped, 5 xfailed in 645 s. ruff and mypy clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
RED FIRST, second repair. PM's checkpoint on c1bdb37 measured N2b: row 1 goes from
3 of 17 to 17 of 17 GREEN with fourteen stub doors and ONE unrelated green test used
as the behaviour probe for all of them. The gate only looked up whether a nodeid was
`passed`; nothing bound the probe to its step. EXTERNAL_DOOR promised the probe
"calls the door and reads the artefact it writes" and there was no code for it --
the same class of defect as entry["kind"] one round ago.
20 new arms, 19 RED, every one on an assert about behaviour (43 assert lines, 0
ImportError, 0 AttributeError, 0 collection errors). The twentieth is green by
construction: every held_out entry in the contract does carry a reason today, and
the arm exists so that stops being true loudly.
Each refusal arm carries its own rc-0 control FIRST -- the same probe, the same
surface, accepted -- then breaks exactly one trait. The constructed probe is now
WRITTEN (`_write_probe`), not merely named: the repaired gate reads the probe's own
source, so a nodeid that does not exist cannot be a behaviour.
Red on the three bearing rests:
probe not bound to its step door untouched, step unnamed, nothing asserted,
test missing, one probe shared by two steps, and
N2b against the contract that stands (17 -> k=0)
limit statement untrue the six escapes PM measured as unseen, planted one
at a time, and the "TWO ways remain" sentence
held_out accepts empty a blank reason shrinks the denominator; the summary
counts entries, not reasons; the four steps outside
the run path are counted without being named
Red on five of the nine small rests: pruned manifest (451 of 512 still GREEN), a
non-UTF-8 file counted but never scanned, a symlink out of the tree, a runbook whose
whole content is "x", and the row's ratio read as a coverage measure.
One test-hygiene change, not a product fix: the git-dependent arm now skips itself
without .git instead of failing, so "red in a clean extract" stops growing with each
new arm of that class.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three of the new row-3 arms pinned 512 files and the literal source label, which only a tree WITH
git metadata can answer; a clean extract (git archive | tar -x) would have turned all three red for
the same reason the three already-named extraction artefacts go red. The property they test holds
in both worlds, so they now assert it there: main.py, examples/, spikes/ and contexts/ are inside
the guard whichever manifest was read, and the row prints whatever denominator it measured.
The exact number and the label belong to ONE arm — the one whose subject IS the git manifest. In an
extract it is an extraction artefact of the same class as the others, and says so in its docstring.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Row 1's M was a curated list of 13 in the gate's OWN b_gate.json. It is now DERIVED: run_path_calls
reads what run.py::run_project actually calls (41 po-functions, re-counted here against PM's 39 —
the two that differ are generate_via_llm and fresh_workflow, held out because they need a chat
client). A call that is neither a declared step nor named-with-a-reason counts in the denominator
WITHOUT a door, so the number cannot shrink in silence: removing a step from the contract now
leaves N unchanged and names the orphan. The four outbox writers the run path uses and nobody had
declared (write_prepass, write_parse_failures, write_proposal_reviews, write_debate_tools) are
steps now; 28 calls are held out, each with its reason printed under the row.
A door must be REGISTERED and PROVEN. entry["kind"] is read (it was read 0 times before) and must
be one of three kinds the gate has code to verify: console-script in pyproject, module-main with
its own __main__ guard, subcommand registered in the module's argparse. On top of that every step
needs a named probe that calls the door and reads the artefact. The MCP sentence is struck from
EXTERNAL_DOOR: it claimed a capability with no code behind it.
Row 3 now scans the repo manifest (git ls-files, or the tree itself in an extract), not a hand
list of 11 roots: 512 published files instead of 433, so main.py, examples/, spikes/, contexts/,
CLAUDE.md and llms.txt are inside the guard for the first time. Three new patterns catch the
indirect invocations that walked past the old six — absolute path, list in a variable, constant,
shell string — 5 of 5 of PM's variants are refused now, with 0 false positives measured over the
whole surface. An empty surface is IKKE MAALT, not GREEN: the row demands a sentinel file and
prints the file count and the manifest it read.
Rows 4, 5 and 6 get sources for their denominators. Row 4 counts only checks whose named source
symbol resolves in the code. Row 5 counts each structural requirement on its own (2 profile
members + factory + seam + probe = 5) instead of collapsing three into one unit. Row 6's N comes
from the artefacts the contract names, and the attestation is VALIDATED: it must name the
contract's runbook, carry its sha256, say who ran it, and bear a real ISO date that is not in the
future — v1_gate's own date rule, reused, BOM tolerated as there.
Measured in a scratch clone (/tmp/claude-po/bgate-mut): 12 of 12 mutants felled, control 65 of 65.
All six of PM's broken attacks reproduced as refused, with the rc-0 control green.
No row got greener: 3 of 17 (was 3 of 13), 0 of 2, GREEN, 0 of 3, 4 of 5 (was 1 of 2), IKKE MAALT.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The PM checkpoint on 207337c judged the gate DELVIS: row 1's M=13 is a curated list in the
gate's OWN b_gate.json (the run path has 41 po-calls, 7 of 10 outbox writers), rows 4, 5 and 6
have denominators with no source at all, 6 of 10 cheat-attacks got through, and the
never-Claude guard sees 433 of 512 published files.
This commit is the red half. Every arm fails on an ASSERT about behaviour, never at collection:
the four names that do not exist yet (ENTRY_KINDS, run_path_calls, registered_entry,
published_files) are stubbed here with DELIBERATELY wrong values — everything is a door, the
run path calls nothing, the surface is empty — so each arm measures the defect rather than the
absence of a symbol.
30 av 31 red on assert. The one that is green is the rc-0 control
(test_a_valid_attestation_is_the_only_thing_that_turns_row6_green): a valid attestation must
turn row 6 green both before and after, or the row refuses everything, which proves as little
as refusing nothing. All 34 pre-existing arms stay green — measured, not assumed.
The planted claude-invocations are base64 in the test file for the same reason the contract's
patterns are: tests/ is itself part of the surface row 3 scans, and a cleartext variant here
would register as its own finding.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
python -m portfolio_optimiser.evals.b_gate — one command, offline, no model call, exit 1 today:
1 steg i kjørestien kallbare utenfra 3 av 13 RØD
2 roller som kan leveres utenfra 0 av 2 RØD
3 vakter mot en vei fra po til Claude 6 av 6 GRØNN (435 published files)
4 løpet drevet uten et eneste modellkall 0 av 3 RØD
5 Foundry-veien urørt og samme artefaktfamilie 1 av 2 RØD
6 kjøreboka finnes og er kjørt 0 av 2 IKKE MÅLT
Every denominator is read off the source, never off a list in the gate. Row 1 counts the steps
of the run path that resolve to a symbol AND have a call site; a step is externally callable only
when a CLI (or MCP-registered) entry reaches it without any chat-client construct on the way —
which is why the ten run.py steps are red and round_builder's two plus the v1 gate are green. Row
2 reads the roles off workflow._MAKER_CHECKER_ROLES. Row 3's patterns each carry a known-positive
AND a known-negative fixture, so a guard that cannot hit is not counted as a zero.
Three decisions the operator cannot answer without reading code, made here and stated in the
gate's own output:
* the external door is a CLI subcommand, not MCP — po already has five main() and two console
commands, and MCP would need a server the run path does not have. The gate still counts an
MCP-registered door, so the choice does not bind the next order.
* the budget guard in B is NOT po's: BudgetMiddleware is fail-closed on missing usage and is
never constructed without a chat client, so keeping it here would turn fail-closed into
fail-open. The ceiling in B is the Claude Code session's own spend, which po neither sees nor
steers. The Foundry path keeps its ceiling unchanged.
* row 6 is IKKE MÅLT, never green, until the operator attests that the runbook actually drove an
analysis — a file the gate never writes, the same rule as the v1 gate's attestation.
Row 3's pattern text is base64 in the config so the contract cannot register as its own finding;
that is what lets the row run without an exclusion list, and a row without exclusions is a row
nobody can switch off by adding a filename.
Suite after: 2106 passed / 5 skipped / 5 xfailed (was 2072/5/5; +34 new, none changed).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Operator decision 19.09.2026: in development and test Claude Code LEADS and portfolio-optimiser
is the toolbox. po never calls Claude; production stays on Foundry. This commit writes the
measurement RED — the contract first, the capability later.
Six rows, each with a denominator taken from the SOURCE and counted independently here:
1. toolbox complete — every deterministic step of the run path, anchored to (module, symbol) and
to the scope that calls it; k = steps reachable from a CLI entry WITHOUT a chat client.
2. what the model delivered can be delivered from outside — denominator read off
workflow._MAKER_CHECKER_ROLES, k = roles with passing named probes.
3. po has no path to Claude — six patterns, each carrying its own known-positive AND
known-negative fixture. Pattern text is base64 in the config so the gate's own contract cannot
register as a hit against the surface it scans.
4. no model calls in toolbox mode — three named probes.
5. the Foundry path untouched — profiles, factory and injection seam read from source, plus a
schema-comparison probe.
6. the runbook exists — NOT MEASURED until the operator attests, and the gate never writes that
attestation itself.
Red now for the reason that matters: the tests fail to import a module that does not exist yet.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
