open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 4 Activity

fix(b-gate): every denominator answers to a source outside the gate, and a name is no longer a door [skip-docs]

Browse source 
Row 1's M was a curated list of 13 in the gate's OWN b_gate.json. It is now DERIVED: run_path_calls
reads what run.py::run_project actually calls (41 po-functions, re-counted here against PM's 39 —
the two that differ are generate_via_llm and fresh_workflow, held out because they need a chat
client). A call that is neither a declared step nor named-with-a-reason counts in the denominator
WITHOUT a door, so the number cannot shrink in silence: removing a step from the contract now
leaves N unchanged and names the orphan. The four outbox writers the run path uses and nobody had
declared (write_prepass, write_parse_failures, write_proposal_reviews, write_debate_tools) are
steps now; 28 calls are held out, each with its reason printed under the row.

A door must be REGISTERED and PROVEN. entry["kind"] is read (it was read 0 times before) and must
be one of three kinds the gate has code to verify: console-script in pyproject, module-main with
its own __main__ guard, subcommand registered in the module's argparse. On top of that every step
needs a named probe that calls the door and reads the artefact. The MCP sentence is struck from
EXTERNAL_DOOR: it claimed a capability with no code behind it.

Row 3 now scans the repo manifest (git ls-files, or the tree itself in an extract), not a hand
list of 11 roots: 512 published files instead of 433, so main.py, examples/, spikes/, contexts/,
CLAUDE.md and llms.txt are inside the guard for the first time. Three new patterns catch the
indirect invocations that walked past the old six — absolute path, list in a variable, constant,
shell string — 5 of 5 of PM's variants are refused now, with 0 false positives measured over the
whole surface. An empty surface is IKKE MAALT, not GREEN: the row demands a sentinel file and
prints the file count and the manifest it read.

Rows 4, 5 and 6 get sources for their denominators. Row 4 counts only checks whose named source
symbol resolves in the code. Row 5 counts each structural requirement on its own (2 profile
members + factory + seam + probe = 5) instead of collapsing three into one unit. Row 6's N comes
from the artefacts the contract names, and the attestation is VALIDATED: it must name the
contract's runbook, carry its sha256, say who ran it, and bear a real ISO date that is not in the
future — v1_gate's own date rule, reused, BOM tolerated as there.

Measured in a scratch clone (/tmp/claude-po/bgate-mut): 12 of 12 mutants felled, control 65 of 65.
All six of PM's broken attacks reproduced as refused, with the rc-0 control green.
No row got greener: 3 of 17 (was 3 of 13), 0 of 2, GREEN, 0 of 3, 4 of 5 (was 1 of 2), IKKE MAALT.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-19 19:40:42 +02:00
commit 59f35fde22
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
3 changed files with 734 additions and 221 deletions
Download patch file Download diff file Expand all files Collapse all files

343
src/portfolio_optimiser/evals/b_gate.json
View file

@ -1,5 +1,124 @@
{
"outcome": "En Claude Code-økt driver en hel analyse med po som verktøykasse, uten at po gjør ett eneste modellkall.",
"run_path": {
"module": "run.py",
"scope": "run_project",
"package": "portfolio_optimiser",
"held_out": [
{
"symbol": "admits",
"reason": "dimensjonsfilteret inne i mandat-vurderingen; skriver ingen egen artefakt"
},
{
"symbol": "baseline_from_project",
"reason": "utleder grunnlaget fra prosjektobjektet inne i «kostnadsgrunnlag»"
},
{
"symbol": "build_mcp_tools",
"reason": "bygger verktøylista debatten kaller; hører til modellveien"
},
{
"symbol": "bundle_citations",
"reason": "siteringshjelper inne i «kontekst»"
},
{
"symbol": "bundle_excerpt_citations",
"reason": "siteringshjelper inne i «kontekst»"
},
{
"symbol": "chunk_dict_to_citation",
"reason": "formkonvertering inne i «kontekst»"
},
{
"symbol": "classify_codes",
"reason": "klassifiserer avvisningskodene inne i «validering»"
},
{
"symbol": "criteria_block",
"reason": "renderer mandatkriteriene inn i prompten; modellveien"
},
{
"symbol": "fresh_workflow",
"reason": "krever chatklient: bygger MAF-debatten"
},
{
"symbol": "generate_via_llm",
"reason": "krever chatklient: ER modellkallet"
},
{
"symbol": "grounding_offer",
"reason": "bygger grunnlagstilbudet til prompten; modellveien"
},
{
"symbol": "load_contracts",
"reason": "leser kontraktsfilene før kjøringen; egen dør hører i preflight"
},
{
"symbol": "load_verdicts_from_dir",
"reason": "leser tidligere dommer inn i ExpeL-konteksten; inne i «dom»"
},
{
"symbol": "make_retrieval_tool",
"reason": "pakker «kontekst» som FunctionTool for agentene; modellveien"
},
{
"symbol": "navigator_tools",
"reason": "explore-modusens verktøy; rad 2 avgrenser B til proposer og checker"
},
{
"symbol": "optional_bundle_candidate_features",
"reason": "henter valgfrie trekk inne i «verdict-id»"
},
{
"symbol": "proposal_reviews_payload",
"reason": "bygger nyttelasten «forslagsvurderinger» skriver"
},
{
"symbol": "requirement_payload",
"reason": "explore-modus; utenfor B slik rad 2 avgrenser den"
},
{
"symbol": "resolve_model",
"reason": "slår opp deployment-navnet for chatklienten; modellveien"
},
{
"symbol": "tool_call_payload",
"reason": "bygger nyttelasten «debatt-verktøy» skriver"
},
{
"symbol": "tool_server_index",
"reason": "indekserer MCP-serverne for debatten; modellveien"
},
{
"symbol": "assert_declared_ids_agree",
"reason": "konsistenssjekk inne i «pakke»"
},
{
"symbol": "declared_reference_numbers",
"reason": "leser de erklærte referansetallene inne i «pakke»"
},
{
"symbol": "load_optional_cost_baseline",
"reason": "valgfri overstyring inne i «kostnadsgrunnlag»"
},
{
"symbol": "reconcile_bundle_id",
"reason": "binder bundle-id-en inne i «pakke»"
},
{
"symbol": "declaration_of",
"reason": "leser prepass-erklæringen inne i «prepass»"
},
{
"symbol": "declaration_payload",
"reason": "bygger erklæringens nyttelast inne i «prepass»"
},
{
"symbol": "render_context",
"reason": "renderer prepass-konteksten inn i prompten; modellveien"
}
]
},
"steps": [
{
"id": "pakke",
@ -11,10 +130,11 @@
"scope": "run_project"
},
"entry": {
"kind": "cli",
"kind": "console-script",
"module": "run.py",
"scope": "main"
}
},
"probe": []
},
{
"id": "kostnadsgrunnlag",
@ -26,10 +146,11 @@
"scope": "run_project"
},
"entry": {
"kind": "cli",
"kind": "console-script",
"module": "run.py",
"scope": "main"
}
},
"probe": []
},
{
"id": "kontekst",
@ -41,10 +162,11 @@
"scope": "run_project"
},
"entry": {
"kind": "cli",
"kind": "console-script",
"module": "run.py",
"scope": "main"
}
},
"probe": []
},
{
"id": "prepass",
@ -56,10 +178,11 @@
"scope": "run_project"
},
"entry": {
"kind": "cli",
"kind": "console-script",
"module": "run.py",
"scope": "main"
}
},
"probe": []
},
{
"id": "validering",
@ -71,10 +194,11 @@
"scope": "evaluate_mandate_candidates"
},
"entry": {
"kind": "cli",
"kind": "console-script",
"module": "run.py",
"scope": "main"
}
},
"probe": []
},
{
"id": "verdict-id",
@ -86,10 +210,11 @@
"scope": "run_project"
},
"entry": {
"kind": "cli",
"kind": "console-script",
"module": "run.py",
"scope": "main"
}
},
"probe": []
},
{
"id": "dom",
@ -101,10 +226,11 @@
"scope": "run_project"
},
"entry": {
"kind": "cli",
"kind": "console-script",
"module": "run.py",
"scope": "main"
}
},
"probe": []
},
{
"id": "kjørekonfig",
@ -116,10 +242,11 @@
"scope": "run_project"
},
"entry": {
"kind": "cli",
"kind": "console-script",
"module": "run.py",
"scope": "main"
}
},
"probe": []
},
{
"id": "coverage",
@ -131,10 +258,11 @@
"scope": "run_project"
},
"entry": {
"kind": "cli",
"kind": "console-script",
"module": "run.py",
"scope": "main"
}
},
"probe": []
},
{
"id": "utboks",
@ -146,10 +274,11 @@
"scope": "run_project"
},
"entry": {
"kind": "cli",
"kind": "console-script",
"module": "run.py",
"scope": "main"
}
},
"probe": []
},
{
"id": "rundebinding",
@ -161,10 +290,13 @@
"scope": "main"
},
"entry": {
"kind": "cli",
"kind": "module-main",
"module": "evals/round_builder.py",
"scope": "main"
}
},
"probe": [
"tests/test_round_builder_loadbearing.py::test_the_command_line_builds_a_round_and_refuses_one_it_cannot"
]
},
{
"id": "rapport",
@ -176,10 +308,13 @@
"scope": "build_round"
},
"entry": {
"kind": "cli",
"kind": "module-main",
"module": "evals/round_builder.py",
"scope": "main"
}
},
"probe": [
"tests/test_round_builder_loadbearing.py::test_the_report_names_every_approach_that_was_considered"
]
},
{
"id": "gate",
@ -191,10 +326,77 @@
"scope": "main"
},
"entry": {
"kind": "cli",
"kind": "module-main",
"module": "evals/v1_gate.py",
"scope": "main"
}
},
"probe": [
"tests/test_v1_gate.py::test_the_command_is_red_today_with_every_row_in_its_output"
]
},
{
"id": "prepass-artefakt",
"label": "skriv prepassets egen artefakt",
"module": "outbox.py",
"symbol": "write_prepass",
"driver": {
"module": "run.py",
"scope": "run_project"
},
"entry": {
"kind": "console-script",
"module": "run.py",
"scope": "main"
},
"probe": []
},
{
"id": "parse-feil",
"label": "skriv det som ikke lot seg lese",
"module": "outbox.py",
"symbol": "write_parse_failures",
"driver": {
"module": "run.py",
"scope": "run_project"
},
"entry": {
"kind": "console-script",
"module": "run.py",
"scope": "main"
},
"probe": []
},
{
"id": "forslagsvurderinger",
"label": "skriv vurderingene av hvert forslag",
"module": "outbox.py",
"symbol": "write_proposal_reviews",
"driver": {
"module": "run.py",
"scope": "run_project"
},
"entry": {
"kind": "console-script",
"module": "run.py",
"scope": "main"
},
"probe": []
},
{
"id": "debatt-verktøy",
"label": "skriv verktøyene debatten kalte",
"module": "outbox.py",
"symbol": "write_debate_tools",
"driver": {
"module": "run.py",
"scope": "run_project"
},
"entry": {
"kind": "console-script",
"module": "run.py",
"scope": "main"
},
"probe": []
}
],
"roles": {
@ -214,19 +416,20 @@
}
},
"no_claude_path": {
"roots": [
"src",
"scripts",
"shared",
"docs",
"tests",
"pyproject.toml",
"README.md",
"CHANGELOG.md",
"DEPLOY.md",
"CONTRIBUTING.md",
"SECURITY.md"
],
"manifest": {
"sentinel": "src/portfolio_optimiser/run.py",
"skip_dirs": [
".git",
"__pycache__",
".venv",
".pytest_cache",
".ruff_cache",
".mypy_cache",
"node_modules",
".idea",
".vscode"
]
},
"patterns": [
{
"id": "prosess",
@ -269,20 +472,59 @@
"pattern_b64": "KEFOVEhST1BJQ19CQVNFX1VSTHxBTlRIUk9QSUNfQVBJX0tFWXxhcGlcLmFudGhyb3BpY1wuY29tKQ==",
"known_positive_b64": "b3MuZW52aXJvblsiQU5USFJPUElDX0JBU0VfVVJMIl0gPSBlbmRwb2ludA==",
"known_negative_b64": "b3MuZW52aXJvblsiQVpVUkVfT1BFTkFJX0VORFBPSU5UIl0gPSBlbmRwb2ludA=="
},
{
"id": "argv-literal",
"label": "claude som programnavn i en argumentliste",
"pattern_b64": "W1xbKCxdXHMqWyInXSg/OlteIidcc10qLyk/Y2xhdWRlWyInXQ==",
"known_positive_b64": "Y21kID0gWyJjbGF1ZGUiLCAiLXAiLCAiYW5hbHlzZXIgcHJvc2pla3RldCJd",
"known_negative_b64": "bW9kZWxsZXIgPSBbImNsYXVkZS1vcHVzLTUiLCAiZ3B0LTUiXQ=="
},
{
"id": "kommando-streng",
"label": "claude med flagg inne i en streng",
"pattern_b64": "WyInXSg/OlteIidcc10qLyk/Y2xhdWRlXHMrLXsxLDJ9W0EtWmEtel0=",
"known_positive_b64": "c3VicHJvY2Vzcy5ydW4oc2hsZXguc3BsaXQoImNsYXVkZSAtcCBhbmFseXNlciIpKQ==",
"known_negative_b64": "c3VicHJvY2Vzcy5ydW4oc2hsZXguc3BsaXQoInB5dGhvbiAtbSBwb3J0Zm9saW9fb3B0aW1pc2VyLnJ1biBQMSIpKQ=="
},
{
"id": "binaersti",
"label": "en konstant som navngir claude-binaeren",
"pattern_b64": "PVxzKlsiJ10oPzpbXiInXHNdKi8pP2NsYXVkZVsiJ10=",
"known_positive_b64": "Q0xBVURFX0JJTiA9ICIvdXNyL2xvY2FsL2Jpbi9jbGF1ZGUi",
"known_negative_b64": "TU9ERUxMID0gImNsYXVkZS1vcHVzLTUi"
}
]
},
"no_model_calls": {
"checks": {
"fullfører": [
"tests/test_b_toolbox_probes.py::test_the_whole_run_completes_with_the_chat_client_raising"
],
"rundekatalog": [
"tests/test_b_toolbox_probes.py::test_the_toolbox_outbox_binds_to_a_scratch_round_directory"
],
"gate-form-ok": [
"tests/test_b_toolbox_probes.py::test_v1_gate_reads_that_round_as_form_ok_not_proven"
]
"fullfører": {
"source": {
"module": "run.py",
"symbol": "run_project"
},
"probe": [
"tests/test_b_toolbox_probes.py::test_the_whole_run_completes_with_the_chat_client_raising"
]
},
"rundekatalog": {
"source": {
"module": "evals/round_builder.py",
"symbol": "build_round"
},
"probe": [
"tests/test_b_toolbox_probes.py::test_the_toolbox_outbox_binds_to_a_scratch_round_directory"
]
},
"gate-form-ok": {
"source": {
"module": "evals/v1_gate.py",
"symbol": "FORM_OK"
},
"probe": [
"tests/test_b_toolbox_probes.py::test_v1_gate_reads_that_round_as_form_ok_not_proven"
]
}
}
},
"foundry": {
@ -310,10 +552,15 @@
}
},
"runbook": {
"artefacts": [
"kjørebok",
"attestering"
],
"path": "docs/kjoerebok-verktoeykassen.md",
"attestation": "v1-rounds/kjoerebok-attestering.txt",
"keys": [
"kjørebok",
"sjekksum",
"kjørt",
"dato"
]