portfolio-optimiser/src/portfolio_optimiser/evals
Kjell Tore Guttormsen 59f35fde22
fix(b-gate): every denominator answers to a source outside the gate, and a name is no longer a door [skip-docs]
Row 1's M was a curated list of 13 in the gate's OWN b_gate.json. It is now DERIVED: run_path_calls
reads what run.py::run_project actually calls (41 po-functions, re-counted here against PM's 39 —
the two that differ are generate_via_llm and fresh_workflow, held out because they need a chat
client). A call that is neither a declared step nor named-with-a-reason counts in the denominator
WITHOUT a door, so the number cannot shrink in silence: removing a step from the contract now
leaves N unchanged and names the orphan. The four outbox writers the run path uses and nobody had
declared (write_prepass, write_parse_failures, write_proposal_reviews, write_debate_tools) are
steps now; 28 calls are held out, each with its reason printed under the row.

A door must be REGISTERED and PROVEN. entry["kind"] is read (it was read 0 times before) and must
be one of three kinds the gate has code to verify: console-script in pyproject, module-main with
its own __main__ guard, subcommand registered in the module's argparse. On top of that every step
needs a named probe that calls the door and reads the artefact. The MCP sentence is struck from
EXTERNAL_DOOR: it claimed a capability with no code behind it.

Row 3 now scans the repo manifest (git ls-files, or the tree itself in an extract), not a hand
list of 11 roots: 512 published files instead of 433, so main.py, examples/, spikes/, contexts/,
CLAUDE.md and llms.txt are inside the guard for the first time. Three new patterns catch the
indirect invocations that walked past the old six — absolute path, list in a variable, constant,
shell string — 5 of 5 of PM's variants are refused now, with 0 false positives measured over the
whole surface. An empty surface is IKKE MAALT, not GREEN: the row demands a sentinel file and
prints the file count and the manifest it read.

Rows 4, 5 and 6 get sources for their denominators. Row 4 counts only checks whose named source
symbol resolves in the code. Row 5 counts each structural requirement on its own (2 profile
members + factory + seam + probe = 5) instead of collapsing three into one unit. Row 6's N comes
from the artefacts the contract names, and the attestation is VALIDATED: it must name the
contract's runbook, carry its sha256, say who ran it, and bear a real ISO date that is not in the
future — v1_gate's own date rule, reused, BOM tolerated as there.

Measured in a scratch clone (/tmp/claude-po/bgate-mut): 12 of 12 mutants felled, control 65 of 65.
All six of PM's broken attacks reproduced as refused, with the rc-0 control green.
No row got greener: 3 of 17 (was 3 of 13), 0 of 2, GREEN, 0 of 3, 4 of 5 (was 1 of 2), IKKE MAALT.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-19 19:40:42 +02:00
..
__init__.py test(v1-gate): the v1 gate, written RED 2026-09-17 15:31:48 +02:00
b_gate.json fix(b-gate): every denominator answers to a source outside the gate, and a name is no longer a door [skip-docs] 2026-09-19 19:40:42 +02:00
b_gate.py fix(b-gate): every denominator answers to a source outside the gate, and a name is no longer a door [skip-docs] 2026-09-19 19:40:42 +02:00
round_builder.py fix(round-builder): a link is refused with a reason, an identical quote is said once, a shared cost line is named on both sides [skip-docs] 2026-09-19 10:20:30 +02:00
v1_gate.json test(v1-gate): harden the gate against a handwritten green 2026-09-17 17:53:40 +02:00
v1_gate.py feat(round-builder): one command turns a run's outbox into a round the gate can read [skip-docs] 2026-09-19 06:09:12 +02:00