open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 3 Activity

fix(b-gate): every denominator answers to a source outside the gate, and a name is no longer a door [skip-docs]

Browse source 
Row 1's M was a curated list of 13 in the gate's OWN b_gate.json. It is now DERIVED: run_path_calls
reads what run.py::run_project actually calls (41 po-functions, re-counted here against PM's 39 —
the two that differ are generate_via_llm and fresh_workflow, held out because they need a chat
client). A call that is neither a declared step nor named-with-a-reason counts in the denominator
WITHOUT a door, so the number cannot shrink in silence: removing a step from the contract now
leaves N unchanged and names the orphan. The four outbox writers the run path uses and nobody had
declared (write_prepass, write_parse_failures, write_proposal_reviews, write_debate_tools) are
steps now; 28 calls are held out, each with its reason printed under the row.

A door must be REGISTERED and PROVEN. entry["kind"] is read (it was read 0 times before) and must
be one of three kinds the gate has code to verify: console-script in pyproject, module-main with
its own __main__ guard, subcommand registered in the module's argparse. On top of that every step
needs a named probe that calls the door and reads the artefact. The MCP sentence is struck from
EXTERNAL_DOOR: it claimed a capability with no code behind it.

Row 3 now scans the repo manifest (git ls-files, or the tree itself in an extract), not a hand
list of 11 roots: 512 published files instead of 433, so main.py, examples/, spikes/, contexts/,
CLAUDE.md and llms.txt are inside the guard for the first time. Three new patterns catch the
indirect invocations that walked past the old six — absolute path, list in a variable, constant,
shell string — 5 of 5 of PM's variants are refused now, with 0 false positives measured over the
whole surface. An empty surface is IKKE MAALT, not GREEN: the row demands a sentinel file and
prints the file count and the manifest it read.

Rows 4, 5 and 6 get sources for their denominators. Row 4 counts only checks whose named source
symbol resolves in the code. Row 5 counts each structural requirement on its own (2 profile
members + factory + seam + probe = 5) instead of collapsing three into one unit. Row 6's N comes
from the artefacts the contract names, and the attestation is VALIDATED: it must name the
contract's runbook, carry its sha256, say who ran it, and bear a real ISO date that is not in the
future — v1_gate's own date rule, reused, BOM tolerated as there.

Measured in a scratch clone (/tmp/claude-po/bgate-mut): 12 of 12 mutants felled, control 65 of 65.
All six of PM's broken attacks reproduced as refused, with the rc-0 control green.
No row got greener: 3 of 17 (was 3 of 13), 0 of 2, GREEN, 0 of 3, 4 of 5 (was 1 of 2), IKKE MAALT.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-19 19:40:42 +02:00
commit 59f35fde22
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
3 changed files with 734 additions and 221 deletions
Download patch file Download diff file Expand all files Collapse all files

335
src/portfolio_optimiser/evals/b_gate.json
View file

@ -1,5 +1,124 @@
{ {
"outcome": "En Claude Code-økt driver en hel analyse med po som verktøykasse, uten at po gjør ett eneste modellkall.", "outcome": "En Claude Code-økt driver en hel analyse med po som verktøykasse, uten at po gjør ett eneste modellkall.",
"run_path": {
"module": "run.py",
"scope": "run_project",
"package": "portfolio_optimiser",
"held_out": [
{
"symbol": "admits",
"reason": "dimensjonsfilteret inne i mandat-vurderingen; skriver ingen egen artefakt"
},
{
"symbol": "baseline_from_project",
"reason": "utleder grunnlaget fra prosjektobjektet inne i «kostnadsgrunnlag»"
},
{
"symbol": "build_mcp_tools",
"reason": "bygger verktøylista debatten kaller; hører til modellveien"
},
{
"symbol": "bundle_citations",
"reason": "siteringshjelper inne i «kontekst»"
},
{
"symbol": "bundle_excerpt_citations",
"reason": "siteringshjelper inne i «kontekst»"
},
{
"symbol": "chunk_dict_to_citation",
"reason": "formkonvertering inne i «kontekst»"
},
{
"symbol": "classify_codes",
"reason": "klassifiserer avvisningskodene inne i «validering»"
},
{
"symbol": "criteria_block",
"reason": "renderer mandatkriteriene inn i prompten; modellveien"
},
{
"symbol": "fresh_workflow",
"reason": "krever chatklient: bygger MAF-debatten"
},
{
"symbol": "generate_via_llm",
"reason": "krever chatklient: ER modellkallet"
},
{
"symbol": "grounding_offer",
"reason": "bygger grunnlagstilbudet til prompten; modellveien"
},
{
"symbol": "load_contracts",
"reason": "leser kontraktsfilene før kjøringen; egen dør hører i preflight"
},
{
"symbol": "load_verdicts_from_dir",
"reason": "leser tidligere dommer inn i ExpeL-konteksten; inne i «dom»"
},
{
"symbol": "make_retrieval_tool",
"reason": "pakker «kontekst» som FunctionTool for agentene; modellveien"
},
{
"symbol": "navigator_tools",
"reason": "explore-modusens verktøy; rad 2 avgrenser B til proposer og checker"
},
{
"symbol": "optional_bundle_candidate_features",
"reason": "henter valgfrie trekk inne i «verdict-id»"
},
{
"symbol": "proposal_reviews_payload",
"reason": "bygger nyttelasten «forslagsvurderinger» skriver"
},
{
"symbol": "requirement_payload",
"reason": "explore-modus; utenfor B slik rad 2 avgrenser den"
},
{
"symbol": "resolve_model",
"reason": "slår opp deployment-navnet for chatklienten; modellveien"
},
{
"symbol": "tool_call_payload",
"reason": "bygger nyttelasten «debatt-verktøy» skriver"
},
{
"symbol": "tool_server_index",
"reason": "indekserer MCP-serverne for debatten; modellveien"
},
{
"symbol": "assert_declared_ids_agree",
"reason": "konsistenssjekk inne i «pakke»"
},
{
"symbol": "declared_reference_numbers",
"reason": "leser de erklærte referansetallene inne i «pakke»"
},
{
"symbol": "load_optional_cost_baseline",
"reason": "valgfri overstyring inne i «kostnadsgrunnlag»"
},
{
"symbol": "reconcile_bundle_id",
"reason": "binder bundle-id-en inne i «pakke»"
},
{
"symbol": "declaration_of",
"reason": "leser prepass-erklæringen inne i «prepass»"
},
{
"symbol": "declaration_payload",
"reason": "bygger erklæringens nyttelast inne i «prepass»"
},
{
"symbol": "render_context",
"reason": "renderer prepass-konteksten inn i prompten; modellveien"
}
]
},
"steps": [ "steps": [
{ {
"id": "pakke", "id": "pakke",
@ -11,10 +130,11 @@
"scope": "run_project" "scope": "run_project"
}, },
"entry": { "entry": {
"kind": "cli", "kind": "console-script",
"module": "run.py", "module": "run.py",
"scope": "main" "scope": "main"
} },
"probe": []
}, },
{ {
"id": "kostnadsgrunnlag", "id": "kostnadsgrunnlag",
@ -26,10 +146,11 @@
"scope": "run_project" "scope": "run_project"
}, },
"entry": { "entry": {
"kind": "cli", "kind": "console-script",
"module": "run.py", "module": "run.py",
"scope": "main" "scope": "main"
} },
"probe": []
}, },
{ {
"id": "kontekst", "id": "kontekst",
@ -41,10 +162,11 @@
"scope": "run_project" "scope": "run_project"
}, },
"entry": { "entry": {
"kind": "cli", "kind": "console-script",
"module": "run.py", "module": "run.py",
"scope": "main" "scope": "main"
} },
"probe": []
}, },
{ {
"id": "prepass", "id": "prepass",
@ -56,10 +178,11 @@
"scope": "run_project" "scope": "run_project"
}, },
"entry": { "entry": {
"kind": "cli", "kind": "console-script",
"module": "run.py", "module": "run.py",
"scope": "main" "scope": "main"
} },
"probe": []
}, },
{ {
"id": "validering", "id": "validering",
@ -71,10 +194,11 @@
"scope": "evaluate_mandate_candidates" "scope": "evaluate_mandate_candidates"
}, },
"entry": { "entry": {
"kind": "cli", "kind": "console-script",
"module": "run.py", "module": "run.py",
"scope": "main" "scope": "main"
} },
"probe": []
}, },
{ {
"id": "verdict-id", "id": "verdict-id",
@ -86,10 +210,11 @@
"scope": "run_project" "scope": "run_project"
}, },
"entry": { "entry": {
"kind": "cli", "kind": "console-script",
"module": "run.py", "module": "run.py",
"scope": "main" "scope": "main"
} },
"probe": []
}, },
{ {
"id": "dom", "id": "dom",
@ -101,10 +226,11 @@
"scope": "run_project" "scope": "run_project"
}, },
"entry": { "entry": {
"kind": "cli", "kind": "console-script",
"module": "run.py", "module": "run.py",
"scope": "main" "scope": "main"
} },
"probe": []
}, },
{ {
"id": "kjørekonfig", "id": "kjørekonfig",
@ -116,10 +242,11 @@
"scope": "run_project" "scope": "run_project"
}, },
"entry": { "entry": {
"kind": "cli", "kind": "console-script",
"module": "run.py", "module": "run.py",
"scope": "main" "scope": "main"
} },
"probe": []
}, },
{ {
"id": "coverage", "id": "coverage",
@ -131,10 +258,11 @@
"scope": "run_project" "scope": "run_project"
}, },
"entry": { "entry": {
"kind": "cli", "kind": "console-script",
"module": "run.py", "module": "run.py",
"scope": "main" "scope": "main"
} },
"probe": []
}, },
{ {
"id": "utboks", "id": "utboks",
@ -146,10 +274,11 @@
"scope": "run_project" "scope": "run_project"
}, },
"entry": { "entry": {
"kind": "cli", "kind": "console-script",
"module": "run.py", "module": "run.py",
"scope": "main" "scope": "main"
} },
"probe": []
}, },
{ {
"id": "rundebinding", "id": "rundebinding",
@ -161,10 +290,13 @@
"scope": "main" "scope": "main"
}, },
"entry": { "entry": {
"kind": "cli", "kind": "module-main",
"module": "evals/round_builder.py", "module": "evals/round_builder.py",
"scope": "main" "scope": "main"
} },
"probe": [
"tests/test_round_builder_loadbearing.py::test_the_command_line_builds_a_round_and_refuses_one_it_cannot"
]
}, },
{ {
"id": "rapport", "id": "rapport",
@ -176,10 +308,13 @@
"scope": "build_round" "scope": "build_round"
}, },
"entry": { "entry": {
"kind": "cli", "kind": "module-main",
"module": "evals/round_builder.py", "module": "evals/round_builder.py",
"scope": "main" "scope": "main"
} },
"probe": [
"tests/test_round_builder_loadbearing.py::test_the_report_names_every_approach_that_was_considered"
]
}, },
{ {
"id": "gate", "id": "gate",
@ -191,10 +326,77 @@
"scope": "main" "scope": "main"
}, },
"entry": { "entry": {
"kind": "cli", "kind": "module-main",
"module": "evals/v1_gate.py", "module": "evals/v1_gate.py",
"scope": "main" "scope": "main"
} },
"probe": [
"tests/test_v1_gate.py::test_the_command_is_red_today_with_every_row_in_its_output"
]
},
{
"id": "prepass-artefakt",
"label": "skriv prepassets egen artefakt",
"module": "outbox.py",
"symbol": "write_prepass",
"driver": {
"module": "run.py",
"scope": "run_project"
},
"entry": {
"kind": "console-script",
"module": "run.py",
"scope": "main"
},
"probe": []
},
{
"id": "parse-feil",
"label": "skriv det som ikke lot seg lese",
"module": "outbox.py",
"symbol": "write_parse_failures",
"driver": {
"module": "run.py",
"scope": "run_project"
},
"entry": {
"kind": "console-script",
"module": "run.py",
"scope": "main"
},
"probe": []
},
{
"id": "forslagsvurderinger",
"label": "skriv vurderingene av hvert forslag",
"module": "outbox.py",
"symbol": "write_proposal_reviews",
"driver": {
"module": "run.py",
"scope": "run_project"
},
"entry": {
"kind": "console-script",
"module": "run.py",
"scope": "main"
},
"probe": []
},
{
"id": "debatt-verktøy",
"label": "skriv verktøyene debatten kalte",
"module": "outbox.py",
"symbol": "write_debate_tools",
"driver": {
"module": "run.py",
"scope": "run_project"
},
"entry": {
"kind": "console-script",
"module": "run.py",
"scope": "main"
},
"probe": []
} }
], ],
"roles": { "roles": {
@ -214,19 +416,20 @@
} }
}, },
"no_claude_path": { "no_claude_path": {
"roots": [ "manifest": {
"src", "sentinel": "src/portfolio_optimiser/run.py",
"scripts", "skip_dirs": [
"shared", ".git",
"docs", "__pycache__",
"tests", ".venv",
"pyproject.toml", ".pytest_cache",
"README.md", ".ruff_cache",
"CHANGELOG.md", ".mypy_cache",
"DEPLOY.md", "node_modules",
"CONTRIBUTING.md", ".idea",
"SECURITY.md" ".vscode"
], ]
},
"patterns": [ "patterns": [
{ {
"id": "prosess", "id": "prosess",
@ -269,21 +472,60 @@
"pattern_b64": "KEFOVEhST1BJQ19CQVNFX1VSTHxBTlRIUk9QSUNfQVBJX0tFWXxhcGlcLmFudGhyb3BpY1wuY29tKQ==", "pattern_b64": "KEFOVEhST1BJQ19CQVNFX1VSTHxBTlRIUk9QSUNfQVBJX0tFWXxhcGlcLmFudGhyb3BpY1wuY29tKQ==",
"known_positive_b64": "b3MuZW52aXJvblsiQU5USFJPUElDX0JBU0VfVVJMIl0gPSBlbmRwb2ludA==", "known_positive_b64": "b3MuZW52aXJvblsiQU5USFJPUElDX0JBU0VfVVJMIl0gPSBlbmRwb2ludA==",
"known_negative_b64": "b3MuZW52aXJvblsiQVpVUkVfT1BFTkFJX0VORFBPSU5UIl0gPSBlbmRwb2ludA==" "known_negative_b64": "b3MuZW52aXJvblsiQVpVUkVfT1BFTkFJX0VORFBPSU5UIl0gPSBlbmRwb2ludA=="
},
{
"id": "argv-literal",
"label": "claude som programnavn i en argumentliste",
"pattern_b64": "W1xbKCxdXHMqWyInXSg/OlteIidcc10qLyk/Y2xhdWRlWyInXQ==",
"known_positive_b64": "Y21kID0gWyJjbGF1ZGUiLCAiLXAiLCAiYW5hbHlzZXIgcHJvc2pla3RldCJd",
"known_negative_b64": "bW9kZWxsZXIgPSBbImNsYXVkZS1vcHVzLTUiLCAiZ3B0LTUiXQ=="
},
{
"id": "kommando-streng",
"label": "claude med flagg inne i en streng",
"pattern_b64": "WyInXSg/OlteIidcc10qLyk/Y2xhdWRlXHMrLXsxLDJ9W0EtWmEtel0=",
"known_positive_b64": "c3VicHJvY2Vzcy5ydW4oc2hsZXguc3BsaXQoImNsYXVkZSAtcCBhbmFseXNlciIpKQ==",
"known_negative_b64": "c3VicHJvY2Vzcy5ydW4oc2hsZXguc3BsaXQoInB5dGhvbiAtbSBwb3J0Zm9saW9fb3B0aW1pc2VyLnJ1biBQMSIpKQ=="
},
{
"id": "binaersti",
"label": "en konstant som navngir claude-binaeren",
"pattern_b64": "PVxzKlsiJ10oPzpbXiInXHNdKi8pP2NsYXVkZVsiJ10=",
"known_positive_b64": "Q0xBVURFX0JJTiA9ICIvdXNyL2xvY2FsL2Jpbi9jbGF1ZGUi",
"known_negative_b64": "TU9ERUxMID0gImNsYXVkZS1vcHVzLTUi"
} }
] ]
}, },
"no_model_calls": { "no_model_calls": {
"checks": { "checks": {
"fullfører": [ "fullfører": {
"source": {
"module": "run.py",
"symbol": "run_project"
},
"probe": [
"tests/test_b_toolbox_probes.py::test_the_whole_run_completes_with_the_chat_client_raising" "tests/test_b_toolbox_probes.py::test_the_whole_run_completes_with_the_chat_client_raising"
], ]
"rundekatalog": [ },
"rundekatalog": {
"source": {
"module": "evals/round_builder.py",
"symbol": "build_round"
},
"probe": [
"tests/test_b_toolbox_probes.py::test_the_toolbox_outbox_binds_to_a_scratch_round_directory" "tests/test_b_toolbox_probes.py::test_the_toolbox_outbox_binds_to_a_scratch_round_directory"
], ]
"gate-form-ok": [ },
"gate-form-ok": {
"source": {
"module": "evals/v1_gate.py",
"symbol": "FORM_OK"
},
"probe": [
"tests/test_b_toolbox_probes.py::test_v1_gate_reads_that_round_as_form_ok_not_proven" "tests/test_b_toolbox_probes.py::test_v1_gate_reads_that_round_as_form_ok_not_proven"
] ]
} }
}
}, },
"foundry": { "foundry": {
"profile": { "profile": {
@ -310,10 +552,15 @@
} }
}, },
"runbook": { "runbook": {
"artefacts": [
"kjørebok",
"attestering"
],
"path": "docs/kjoerebok-verktoeykassen.md", "path": "docs/kjoerebok-verktoeykassen.md",
"attestation": "v1-rounds/kjoerebok-attestering.txt", "attestation": "v1-rounds/kjoerebok-attestering.txt",
"keys": [ "keys": [
"kjørebok", "kjørebok",
"sjekksum",
"kjørt", "kjørt",
"dato" "dato"
] ]

479
src/portfolio_optimiser/evals/b_gate.py
View file

@ -27,9 +27,12 @@ import argparse
import ast import ast
import base64 import base64
import binascii import binascii
import hashlib
import json import json
import re import re
import subprocess
from collections.abc import Callable, Mapping, Sequence from collections.abc import Callable, Mapping, Sequence
from datetime import datetime, timezone
from dataclasses import asdict, dataclass from dataclasses import asdict, dataclass
from pathlib import Path from pathlib import Path
from typing import Any from typing import Any
@ -39,6 +42,7 @@ from portfolio_optimiser.evals.v1_gate import (
NOT_MEASURED, NOT_MEASURED,
RED, RED,
Row, Row,
_parse_given,
exit_code, exit_code,
run_probes, run_probes,
) )
@ -75,13 +79,18 @@ CHAT_CLIENT_NAMES = frozenset(
#: Rad 1: hvilken dør som teller, og hvorfor den ble valgt framfor den andre. #: Rad 1: hvilken dør som teller, og hvorfor den ble valgt framfor den andre.
EXTERNAL_DOOR = ( EXTERNAL_DOOR = (
"valgt CLI-underkommando som dør UT av verktøykassen, fordi po allerede har fem main() og to " "valgt kommandolinja som dør UT av verktøykassen, fordi operatørens ord er «som alle andre "
"konsoll-kommandoer, og operatørens ord er «som alle andre tools Claude Code kan bruke» — en " "tools Claude Code kan bruke» — en kommando er nettopp det. Gaten teller tre arter den kan "
"kommando er nettopp det. Grunnen til å velge bort MCP her er at MCP ville krevd en server po " "VERIFISERE mot kilden: console-script (registrert i pyproject), module-main (kjørbar med "
"ikke har i kjørestien i dag; gaten teller likevel en MCP-registrert dør som kallbar, så " "python -m, med sin egen __main__-vakt) og subcommand (et navn registrert i modulens egen "
"valget binder ikke neste ordre" "argparse). En art gaten ikke kan verifisere er ikke en dør her — og et navn er uansett ikke "
"nok: hvert steg må ha en navngitt probe som kaller døren og leser artefaktet den skriver"
) )
#: Inngangsartene rad 1 kan VERIFISERE mot kilden. En art utenfor lista teller aldri som en dør:
#: gaten skal ikke kunne bli grønn av en påstand den ikke har kode for å etterprøve.
ENTRY_KINDS: tuple[str, ...] = ("console-script", "module-main", "subcommand")
#: Rad 2: det ene kravet som skiller en ekte felles vei fra to veier som ser like ut. #: Rad 2: det ene kravet som skiller en ekte felles vei fra to veier som ser like ut.
BYTE_IDENTICAL_RULE = ( BYTE_IDENTICAL_RULE = (
"et forslag gitt UTENFRA skal gjennom NØYAKTIG samme validator, samme avvisningsstadier og " "et forslag gitt UTENFRA skal gjennom NØYAKTIG samme validator, samme avvisningsstadier og "
@ -123,37 +132,6 @@ finnes er RØD, aldri grønn og aldri hoppet over.
""" """
# ---------------------------------------------------------------------------------------------
# STUBBER (rød-først). Navnene finnes her for at armene under skal felle på en ASSERT om ATFERD
# og ikke ved innsamling; verdiene er MED VILJE gale og erstattes i reparasjons-commiten.
# ---------------------------------------------------------------------------------------------
#: Inngangsartene gaten kan VERIFISERE. Tom her med vilje: stubben kjenner ingen.
ENTRY_KINDS: tuple[str, ...] = ()
def run_path_calls(src: Path, spec: Mapping[str, Any]) -> dict[str, str]:
"""STUB: skal lese kjørestiens faktiske kall av KILDEN. Returnerer ingenting ennå."""
return {}
def console_scripts(repo_root: Path) -> dict[str, str]:
"""STUB: skal lese [project.scripts] fra pyproject.toml."""
return {}
def registered_entry(
src: Path, repo_root: Path, package: str, entry: Mapping[str, Any]
) -> tuple[bool, str]:
"""STUB: godtar ALT som dør — nøyaktig feilen reparasjonen skal fjerne."""
return True, ""
def published_files(root: Path, manifest: Mapping[str, Any]) -> tuple[list[Path], str]:
"""STUB: skal utlede den publiserte flaten av repo-manifestet."""
return [], "stub"
def load_config(path: Path = _DATA) -> dict[str, Any]: def load_config(path: Path = _DATA) -> dict[str, Any]:
data: dict[str, Any] = json.loads(path.read_text(encoding="utf-8")) data: dict[str, Any] = json.loads(path.read_text(encoding="utf-8"))
return data return data
@ -285,7 +263,148 @@ def entry_reaches(src: Path, entry: Mapping[str, Any], symbol: str) -> tuple[boo
return True, "" return True, ""
def measure_steps(steps: Sequence[Mapping[str, Any]], src: Path) -> list[Step]: def console_scripts(repo_root: Path) -> dict[str, str]:
"""``[project.scripts]`` i pyproject.toml, lest som tekst.
Lest som tekst og ikke med ``tomllib``, fordi pakken støtter Python 3.10, der ``tomllib``
ikke finnes — og en gate som krever en nyere tolk enn produktet er en gate som ikke kjører
der produktet gjør."""
path = repo_root / "pyproject.toml"
if not path.is_file():
return {}
try:
text = path.read_text(encoding="utf-8")
except (OSError, UnicodeDecodeError): # pragma: no cover - et ulesbart manifest er ingen dør
return {}
scripts: dict[str, str] = {}
inside = False
for line in text.splitlines():
stripped = line.strip()
if stripped.startswith("["):
inside = stripped == "[project.scripts]"
continue
if not inside or "=" not in stripped or stripped.startswith("#"):
continue
name, _, target = stripped.partition("=")
scripts[name.strip().strip("\"'")] = target.strip().strip("\"'")
return scripts
def _dotted(package: str, module: str) -> str:
return f"{package}." + str(module).removesuffix(".py").replace("/", ".")
def _has_dunder_main(tree: ast.AST, scope: str) -> bool:
"""``if __name__ == "__main__":`` som faktisk kaller ``scope`` — en modul kjørbar med -m."""
for node in ast.walk(tree):
if not isinstance(node, ast.If):
continue
test = node.test
if not isinstance(test, ast.Compare) or not isinstance(test.left, ast.Name):
continue
if test.left.id != "__name__":
continue
if not any(isinstance(c, ast.Constant) and c.value == "__main__" for c in test.comparators):
continue
if any(scope in _call_names(child) for child in node.body):
return True
return False
def _registers_subcommand(tree: ast.AST, command: str) -> bool:
for node in ast.walk(tree):
if not isinstance(node, ast.Call) or not isinstance(node.func, ast.Attribute):
continue
if node.func.attr != "add_parser":
continue
if any(isinstance(a, ast.Constant) and a.value == command for a in node.args):
return True
return False
def registered_entry(
src: Path, repo_root: Path, package: str, entry: Mapping[str, Any]
) -> tuple[bool, str]:
"""(inngangen er REGISTRERT, grunnen når den ikke er det).
Målt 19.09.2026: en modul med ``def main(): return <symbol>()`` og ingenting annet tok rad 1
fra 3 til 4 av 13. Et navn er ikke en dør. Døren må være registrert et sted utenfor modulen
selv — i pakkemanifestet, i en ``__main__``-vakt, eller i en argparse-underkommando — og
arten må være en gaten har kode for å etterprøve."""
kind = str(entry.get("kind", ""))
if kind not in ENTRY_KINDS:
return False, f"inngangsarten {kind!r} er ikke en gaten kan verifisere"
module = str(entry["module"])
scope = str(entry.get("scope", "main"))
tree = _tree(src, module)
if tree is None:
return False, f"inngangsmodulen {module} finnes ikke"
if kind == "console-script":
target = f"{_dotted(package, module)}:{scope}"
if target not in console_scripts(repo_root).values():
return False, f"{target} er ikke registrert i pyproject [project.scripts]"
return True, ""
if kind == "module-main":
if not _has_dunder_main(tree, scope):
return False, f"{module} har ingen __main__-vakt som kaller {scope}()"
return True, ""
command = str(entry.get("command", ""))
if not command:
return False, f"{module} navngir ingen underkommando"
if not _registers_subcommand(tree, command):
return False, f"{module} registrerer ingen underkommando {command!r}"
return True, ""
def run_path_calls(src: Path, spec: Mapping[str, Any]) -> dict[str, str]:
"""``symbol -> modulen det kom fra``: po-funksjonene kjørestien FAKTISK kaller.
Dette er rad 1s nevner, og den leses av KILDEN. Regelen: et navn importert fra en
undermodul av pakken og kalt i scopet, eller et attributt på en modul importert som
``from <pakke> import X``. Klasser og unntak (stor forbokstav) er ikke steg, og private
hjelpere i driveren heller ikke. Et kall som verken er erklært som steg eller navngitt som
holdt utenfor, teller med i nevneren UTEN en dør — nevneren kan derfor ikke krympe stille,
som en kuratert liste i gatens egen datafil kunne."""
tree = _tree(src, str(spec["module"]))
if tree is None:
return {}
package = str(spec.get("package", "portfolio_optimiser"))
symbols: dict[str, str] = {}
aliases: set[str] = set()
for node in ast.walk(tree):
if not isinstance(node, ast.ImportFrom):
continue
module = node.module or ""
if module == package:
aliases |= {a.asname or a.name for a in node.names}
elif module.startswith(f"{package}."):
for alias in node.names:
symbols[alias.asname or alias.name] = module.split(".")[-1] + ".py"
found: dict[str, str] = {}
for scope in _scopes(tree, str(spec["scope"])):
for call in ast.walk(scope):
if not isinstance(call, ast.Call):
continue
func = call.func
if isinstance(func, ast.Name) and func.id in symbols and func.id[:1].islower():
found[func.id] = symbols[func.id]
elif (
isinstance(func, ast.Attribute)
and isinstance(func.value, ast.Name)
and func.value.id in aliases
and func.attr[:1].islower()
):
found[func.attr] = f"{func.value.id}.py"
return found
def measure_steps(
steps: Sequence[Mapping[str, Any]],
src: Path,
outcomes: Mapping[str, str] | None = None,
repo_root: Path = _REPO_ROOT,
package: str = "portfolio_optimiser",
) -> list[Step]:
measured: list[Step] = [] measured: list[Step] = []
for step in steps: for step in steps:
tree = _tree(src, str(step["module"])) tree = _tree(src, str(step["module"]))
@ -299,7 +418,16 @@ def measure_steps(steps: Sequence[Mapping[str, Any]], src: Path) -> list[Step]:
why = f"{step['driver']['module']}::{step['driver']['scope']} kaller ikke {step['symbol']}" why = f"{step['driver']['module']}::{step['driver']['scope']} kaller ikke {step['symbol']}"
else: else:
ok, why = entry_reaches(src, step["entry"], str(step["symbol"])) ok, why = entry_reaches(src, step["entry"], str(step["symbol"]))
why = "" if ok else why if ok:
ok, why = registered_entry(src, repo_root, package, step["entry"])
if ok:
probes = list(step.get("probe", ()))
verdict = _probe_verdict(probes, outcomes or {})
why = (
"ingen atferdsprobe registrert — et navn er ikke en atferd"
if not probes
else (f"atferdsprobe: {verdict}" if verdict else "")
)
external = resolved and called and not why external = resolved and called and not why
measured.append(Step(str(step["id"]), str(step["label"]), resolved, called, external, why)) measured.append(Step(str(step["id"]), str(step["label"]), resolved, called, external, why))
return measured return measured
@ -312,18 +440,28 @@ def score_toolbox(
outcomes: Mapping[str, str] | None = None, outcomes: Mapping[str, str] | None = None,
repo_root: Path = _REPO_ROOT, repo_root: Path = _REPO_ROOT,
) -> Row: ) -> Row:
measured = measure_steps(steps, src) package = str((run_path or {}).get("package", "portfolio_optimiser"))
measured = measure_steps(steps, src, outcomes, repo_root, package)
in_path = [s for s in measured if s.resolved and s.called] in_path = [s for s in measured if s.resolved and s.called]
undeclared = sorted( declared = {str(s["symbol"]) for s in steps}
name held = {
for name in (run_path_calls(src, run_path) if run_path else {}) str(h["symbol"]): str(h.get("reason", "")) for h in (run_path or {}).get("held_out", ())
if name not in {str(s["symbol"]) for s in steps} }
and name not in {str(h["symbol"]) for h in (run_path or {}).get("held_out", ())} calls = run_path_calls(src, run_path) if run_path else {}
) undeclared = sorted(name for name in calls if name not in declared and name not in held)
n = len(in_path) + len(undeclared) n = len(in_path) + len(undeclared)
k = sum(1 for s in in_path if s.external) k = sum(1 for s in in_path if s.external)
exceptions = tuple(f"{s.id}: {s.why}" for s in measured if not s.external) + tuple( exceptions = tuple(f"{s.id}: {s.why}" for s in measured if not s.external) + tuple(
f"udeklarert: {name} kalles i kjørestien uten å være et steg" for name in undeclared f"udeklarert: {name} kalles i {calls[name]} fra kjørestien uten å være et steg"
for name in undeclared
)
diagnostics = tuple(
f"holdt utenfor: {symbol} — {reason or 'UTEN GRUNN'}"
for symbol, reason in sorted(held.items())
) + tuple(
f"holdt utenfor: {symbol} kalles ikke lenger i kjørestien (foreldet unntak)"
for symbol in sorted(held)
if calls and symbol not in calls
) )
return Row( return Row(
"verktøykasse", "verktøykasse",
@ -331,8 +469,10 @@ def score_toolbox(
k, k,
n, n,
GREEN if n and k == n else RED, GREEN if n and k == n else RED,
f"{k} av {n} steg har en dør ut; {len(measured) - n} erklærte steg er ikke i kjørestien", f"{k} av {n} steg har en dør ut; {len(held)} kall holdt utenfor med grunn, "
f"{len(measured) - len(in_path)} erklærte steg er ikke i kjørestien",
exceptions=exceptions, exceptions=exceptions,
diagnostics=diagnostics,
attests=(EXTERNAL_DOOR,), attests=(EXTERNAL_DOOR,),
) )
@ -397,30 +537,55 @@ def score_roles(roles: Mapping[str, Any], outcomes: Mapping[str, str], src: Path
# Rad 3 — po har ingen vei til Claude # Rad 3 — po har ingen vei til Claude
# --------------------------------------------------------------------------------------------- # ---------------------------------------------------------------------------------------------
#: Flatene som publiseres, lest som tekst. Alt annet er binært og bærer ingen kjørbar linje.
TEXT_SUFFIXES = frozenset(
{".py", ".sh", ".toml", ".json", ".md", ".txt", ".cfg", ".ini", ".yaml", ".yml", ".template"}
)
_FENCE = re.compile(r"^\s*(```|~~~)") _FENCE = re.compile(r"^\s*(```|~~~)")
#: Rad 3: hva vakten IKKE rekker, sagt av raden selv. Fire av PMs fem indirekte kall-varianter
#: slapp forbi før 19.09; disse to gjør det fortsatt, og de er linje-lokale grenser i en
#: tekstvakt, ikke noe en regex kan lukke.
NO_CLAUDE_LIMIT = (
"rad 3 er en TEKSTVAKT over hele det publiserte manifestet, ikke en dataflyt-analyse: den "
"feller et programnavn skrevet ut, en absolutt sti, en liste lagt i en variabel, en "
"konstant og en shell-streng. To veier står igjen og kan ikke felles linje for linje — et "
"kommandonavn satt sammen av deler ved kjøretid, og et navn lest ut av en miljøvariabel. "
"Raden påstår derfor ikke at ingen vei finnes; den påstår at ingen SKREVET vei finnes, over "
"en nevner den oppgir i filer"
)
def surface_files(root: Path, roots: Sequence[str]) -> list[Path]:
found: list[Path] = [] def published_files(root: Path, manifest: Mapping[str, Any]) -> tuple[list[Path], str]:
for rel in roots: """(filene som PUBLISERES, hvor lista kom fra).
base = root / rel
if base.is_file(): Kilden er repo-manifestet, ikke en håndliste: ``git ls-files`` når det finnes en ``.git``,
found.append(base) og ellers filtreet selv — et rent uttrekk (``git archive | tar -x``) ER nøyaktig det
continue publiserte, og har ingen git-metadata å spørre. Håndlista den erstattet utelot ``main.py``,
if not base.is_dir(): ``examples/``, ``spikes/`` og ``contexts/``: 433 av 512 filer ble skannet, og alle seks
continue vaktenes kjent-positive kunne plantes i ``main.py`` uten at raden merket det.
for path in sorted(base.rglob("*")):
if not path.is_file() or path.is_symlink(): Ingen suffiks-allowlist heller: en fil som lar seg lese som tekst kan bære en kommandolinje
continue uansett hva den heter. Det som ikke lar seg avkode, telles og oppgis."""
if "__pycache__" in path.parts or ".git" in path.parts: skip = set(manifest.get("skip_dirs", (".git", "__pycache__")))
continue if (root / ".git").exists():
if path.suffix in TEXT_SUFFIXES: try:
found.append(path) listing = subprocess.run(
return found ["git", "ls-files", "-z"],
cwd=root,
capture_output=True,
text=True,
check=True,
).stdout
except (OSError, subprocess.CalledProcessError): # pragma: no cover - git kan mangle
listing = None
if listing is not None:
found = [root / rel for rel in listing.split("\0") if rel]
return [p for p in found if p.is_file() and not p.is_symlink()], "git-manifestet"
walked = [
path
for path in sorted(root.rglob("*"))
if path.is_file()
and not path.is_symlink()
and not (skip & set(path.relative_to(root).parts))
]
return walked, "filtreet (uttrekk uten git-metadata)"
def scannable_lines(path: Path, text: str) -> list[tuple[int, str]]: def scannable_lines(path: Path, text: str) -> list[tuple[int, str]]:
@ -451,17 +616,25 @@ class Pattern:
hits: tuple[str, ...] hits: tuple[str, ...]
def measure_patterns(config: Mapping[str, Any], root: Path) -> list[Pattern]: @dataclass(frozen=True)
class Surface:
files: int
unreadable: int
source: str
sentinel: bool
def measure_patterns(config: Mapping[str, Any], root: Path) -> tuple[list[Pattern], Surface]:
manifest = config.get("manifest", {})
files, source = published_files(root, manifest)
sentinel = str(manifest.get("sentinel", ""))
scanned: list[tuple[Path, list[tuple[int, str]]]] = [] scanned: list[tuple[Path, list[tuple[int, str]]]] = []
surface = ( unreadable = 0
surface_files(root, list(config["roots"])) for path in files:
if "roots" in config
else published_files(root, config.get("manifest", {}))[0]
)
for path in surface:
try: try:
text = path.read_text(encoding="utf-8") text = path.read_text(encoding="utf-8")
except (UnicodeDecodeError, OSError): except (UnicodeDecodeError, OSError):
unreadable += 1
continue continue
scanned.append((path, scannable_lines(path, text))) scanned.append((path, scannable_lines(path, text)))
measured: list[Pattern] = [] measured: list[Pattern] = []
@ -482,18 +655,14 @@ def measure_patterns(config: Mapping[str, Any], root: Path) -> list[Pattern]:
if rx.search(line) if rx.search(line)
) )
measured.append(Pattern(str(spec["id"]), str(spec["label"]), not why, why, hits)) measured.append(Pattern(str(spec["id"]), str(spec["label"]), not why, why, hits))
return measured present = bool(sentinel) and any(str(p.relative_to(root)) == sentinel for p in files)
return measured, Surface(len(files), unreadable, source, present)
def score_no_claude_path(config: Mapping[str, Any], root: Path) -> Row: def score_no_claude_path(config: Mapping[str, Any], root: Path) -> Row:
measured = measure_patterns(config, root) measured, surface = measure_patterns(config, root)
n = len(measured) n = len(measured)
k = sum(1 for p in measured if p.valid and not p.hits) k = sum(1 for p in measured if p.valid and not p.hits)
files = (
len(surface_files(root, list(config["roots"])))
if "roots" in config
else len(published_files(root, config.get("manifest", {}))[0])
)
exceptions: list[str] = [] exceptions: list[str] = []
for pattern in measured: for pattern in measured:
if not pattern.valid: if not pattern.valid:
@ -501,14 +670,23 @@ def score_no_claude_path(config: Mapping[str, Any], root: Path) -> Row:
elif pattern.hits: elif pattern.hits:
shown = ", ".join(pattern.hits[:3]) shown = ", ".join(pattern.hits[:3])
exceptions.append(f"{pattern.id}: {len(pattern.hits)} treff — {shown}") exceptions.append(f"{pattern.id}: {len(pattern.hits)} treff — {shown}")
sentinel = str(config.get("manifest", {}).get("sentinel", ""))
if not surface.sentinel:
exceptions.append(
f"flaten mangler sin sentinel ({sentinel}) — {surface.files} filer sett, "
"et fravær uten nevner er ikke et bevis"
)
status = NOT_MEASURED if not surface.sentinel else GREEN if n and k == n else RED
return Row( return Row(
"ingen-claude-vei", "ingen-claude-vei",
"3 vakter mot en vei fra po til Claude", "3 vakter mot en vei fra po til Claude",
k, k,
n, n,
GREEN if n and k == n else RED, status,
f"{k} av {n} vakter måler og gir null treff over {files} publiserte filer", f"{k} av {n} vakter måler og gir null treff over {surface.files} publiserte filer "
f"({surface.source}; {surface.unreadable} ulesbare)",
exceptions=tuple(exceptions), exceptions=tuple(exceptions),
attests=(NO_CLAUDE_LIMIT,),
) )
@ -520,18 +698,35 @@ def score_no_claude_path(config: Mapping[str, Any], root: Path) -> Row:
def score_no_model_calls( def score_no_model_calls(
config: Mapping[str, Any], outcomes: Mapping[str, str], src: Path | None = None config: Mapping[str, Any], outcomes: Mapping[str, str], src: Path | None = None
) -> Row: ) -> Row:
checks = config["checks"] """Rad 4s nevner er sjekkene hvis NAVNGITTE kilde faktisk finnes i koden.
verdicts = {name: _probe_verdict(list(ids), outcomes) for name, ids in checks.items()}
k = sum(1 for why in verdicts.values() if not why) M = 3 var en konstant i gatens egen konfig. Nå bærer hver sjekk et symbol i kilden den
n = len(verdicts) handler om — kjøringen selv, rundebinderen, og v1-gatens ``FORM_OK``. Forsvinner symbolet,
faller sjekken ut av nevneren OG navngis, i stedet for å bli en grønn rad over ingenting."""
source_root = src if src is not None else _PACKAGE_SRC
counted: dict[str, str] = {}
exceptions: list[str] = []
for name, check in config["checks"].items():
source = check["source"]
tree = _tree(source_root, str(source["module"]))
if tree is None or not _defined(tree, str(source["symbol"])):
exceptions.append(
f"{name}: kilden {source['module']}::{source['symbol']} finnes ikke — "
"sjekken teller ikke med i nevneren"
)
continue
counted[name] = _probe_verdict(list(check.get("probe", ())), outcomes)
k = sum(1 for why in counted.values() if not why)
n = len(counted)
exceptions += [f"{name}: {why}" for name, why in counted.items() if why]
return Row( return Row(
"ingen-modellkall", "ingen-modellkall",
"4 løpet drevet uten et eneste modellkall", "4 løpet drevet uten et eneste modellkall",
k, k,
n, n,
GREEN if n and k == n else RED, GREEN if n and k == n else RED,
f"{k} av {n} sjekker har en bestått probe", f"{k} av {n} sjekker har en bestått probe (nevneren er sjekker med en kilde i koden)",
exceptions=tuple(f"{name}: {why}" for name, why in verdicts.items() if why), exceptions=tuple(exceptions),
attests=(BUDGET_IN_B,), attests=(BUDGET_IN_B,),
) )
@ -582,12 +777,18 @@ def foundry_intact(config: Mapping[str, Any], src: Path) -> tuple[bool, tuple[st
def score_foundry(config: Mapping[str, Any], outcomes: Mapping[str, str], src: Path) -> Row: def score_foundry(config: Mapping[str, Any], outcomes: Mapping[str, str], src: Path) -> Row:
intact, missing = foundry_intact(config, src) """Hvert strukturelle krav telles for seg, og hvert av dem er et symbol i kilden.
M = 2 blandet profil-medlemmene, fabrikken og injeksjonssømmen til ÉN tellende enhet: tre
krav kunne falle uten at tallet endret seg med mer enn én. Nevneren er nå medlemmene
kontrakten navngir, pluss fabrikken, pluss sømmen, pluss probene."""
_intact, missing = foundry_intact(config, src)
structural = len(config["profile"]["members"]) + 2
verdicts = { verdicts = {
name: _probe_verdict(list(ids), outcomes) for name, ids in config["evidence"].items() name: _probe_verdict(list(ids), outcomes) for name, ids in config["evidence"].items()
} }
k = int(intact) + sum(1 for why in verdicts.values() if not why) k = structural - len(missing) + sum(1 for why in verdicts.values() if not why)
n = 1 + len(verdicts) n = structural + len(verdicts)
exceptions = [f"urørt: {m}" for m in missing] exceptions = [f"urørt: {m}" for m in missing]
exceptions += [f"{name}: {why}" for name, why in verdicts.items() if why] exceptions += [f"{name}: {why}" for name, why in verdicts.items() if why]
return Row( return Row(
@ -596,7 +797,8 @@ def score_foundry(config: Mapping[str, Any], outcomes: Mapping[str, str], src: P
k, k,
n, n,
GREEN if k == n else RED, GREEN if k == n else RED,
"profil, fabrikk og injeksjonssøm står" if intact else "Foundry-veien er rørt", f"{structural - len(missing)} av {structural} strukturkrav står"
+ ("" if not missing else " — Foundry-veien er rørt"),
exceptions=tuple(exceptions), exceptions=tuple(exceptions),
) )
@ -613,7 +815,23 @@ class RunbookAttestation:
why: str why: str
def read_runbook_attestation(path: Path, keys: Sequence[str]) -> RunbookAttestation: def read_runbook_attestation(
path: Path,
keys: Sequence[str],
runbook: Path,
runbook_name: str,
now: datetime | None = None,
) -> RunbookAttestation:
"""Operatørens bekreftelse på kjøreboka — VALIDERT, ikke bare til stede.
Målt 19.09.2026: den gamle parseren godtok ``kjørebok: x``, datoen ``x``, en framtidsdato og
en peker til en helt annen fil — alle tre ga 2 av 2 GRØNN. Fire ting må stemme nå: fila
navngir kontraktens kjørebok, sjekksummen er kjørebokas egen (en attestering av en annen
versjon er ikke en attestering av denne), noen står som den som kjørte den, og datoen er en
ekte ISO-dato som ikke ligger i framtiden. Datoregelen er v1-gatens egen (``_parse_given``),
gjenbrukt og ikke skrevet om: en dato med klokkeslett må ha tidssone.
BOM tåles (``utf-8-sig``), som på v1-gaten: en editor som skriver den er ikke personens feil."""
if path.is_symlink(): if path.is_symlink():
return RunbookAttestation(False, False, f"{path.name} er en symlenke, ikke en fil") return RunbookAttestation(False, False, f"{path.name} er en symlenke, ikke en fil")
if path.is_dir(): if path.is_dir():
@ -621,7 +839,7 @@ def read_runbook_attestation(path: Path, keys: Sequence[str]) -> RunbookAttestat
if not path.is_file(): if not path.is_file():
return RunbookAttestation(False, False, f"{path} finnes ikke") return RunbookAttestation(False, False, f"{path} finnes ikke")
try: try:
text = path.read_text(encoding="utf-8") text = path.read_text(encoding="utf-8-sig")
except (UnicodeDecodeError, OSError): except (UnicodeDecodeError, OSError):
return RunbookAttestation(True, False, f"{path.name} er ikke UTF-8") return RunbookAttestation(True, False, f"{path.name} er ikke UTF-8")
seen: dict[str, str] = {} seen: dict[str, str] = {}
@ -638,6 +856,37 @@ def read_runbook_attestation(path: Path, keys: Sequence[str]) -> RunbookAttestat
missing = [k for k in keys if not seen.get(k)] missing = [k for k in keys if not seen.get(k)]
if missing: if missing:
return RunbookAttestation(True, False, f"mangler {', '.join(missing)}") return RunbookAttestation(True, False, f"mangler {', '.join(missing)}")
if seen["kjørebok"] != runbook_name:
return RunbookAttestation(
True,
False,
f"attesterer kjørebok {seen['kjørebok']!r}, ikke kontraktens {runbook_name!r}",
)
if not runbook.is_file():
return RunbookAttestation(True, False, f"kjørebok {runbook_name} finnes ikke å sjekksumme")
digest = hashlib.sha256(runbook.read_bytes()).hexdigest()
if seen["sjekksum"].casefold() != digest:
return RunbookAttestation(
True,
False,
f"sjekksum {seen['sjekksum']!r} er ikke kjørebokas ({digest[:12]}…) — attesteringen "
"gjelder en annen versjon",
)
given, why = _parse_given(seen["dato"])
if given is None:
return RunbookAttestation(True, False, why)
clock = now if now is not None else datetime.now(tz=timezone.utc)
future = (
given > clock
if isinstance(given, datetime)
else given > clock.astimezone(timezone.utc).date()
)
if future:
return RunbookAttestation(
True,
False,
f"attestert {given.isoformat()}, i framtiden (nå: {clock.isoformat(timespec='seconds')})",
)
return RunbookAttestation(True, True, "") return RunbookAttestation(True, True, "")
@ -645,13 +894,16 @@ def score_runbook(
config: Mapping[str, Any], config: Mapping[str, Any],
repo_root: Path, repo_root: Path,
attest: Path | None = None, attest: Path | None = None,
now: Any | None = None, now: datetime | None = None,
) -> Row: ) -> Row:
runbook = repo_root / str(config["path"]) runbook = repo_root / str(config["path"])
has_runbook = runbook.is_file() and bool(runbook.read_text(encoding="utf-8").strip()) has_runbook = runbook.is_file() and bool(runbook.read_text(encoding="utf-8").strip())
attest_path = attest if attest is not None else repo_root / str(config["attestation"]) attest_path = attest if attest is not None else repo_root / str(config["attestation"])
attestation = read_runbook_attestation(attest_path, list(config["keys"])) attestation = read_runbook_attestation(
attest_path, list(config["keys"]), runbook, str(config["path"]), now
)
k = int(has_runbook) + int(attestation.ok) k = int(has_runbook) + int(attestation.ok)
n = len(config["artefacts"])
exceptions: list[str] = [] exceptions: list[str] = []
if not has_runbook: if not has_runbook:
exceptions.append(f"kjørebok: {config['path']} finnes ikke") exceptions.append(f"kjørebok: {config['path']} finnes ikke")
@ -661,9 +913,9 @@ def score_runbook(
"kjørebok", "kjørebok",
"6 kjøreboka finnes og er kjørt", "6 kjøreboka finnes og er kjørt",
k, k,
2, n,
GREEN if k == 2 else NOT_MEASURED, GREEN if k == n else NOT_MEASURED,
"kjøreboka og operatørens bekreftelse står" if k == 2 else "venter på operatøren", "kjøreboka og operatørens bekreftelse står" if k == n else "venter på operatøren",
exceptions=tuple(exceptions), exceptions=tuple(exceptions),
attests=(RUNBOOK_RULE,), attests=(RUNBOOK_RULE,),
) )
@ -675,13 +927,12 @@ def score_runbook(
def probe_nodeids(config: Mapping[str, Any]) -> list[str]: def probe_nodeids(config: Mapping[str, Any]) -> list[str]:
ids: list[str] = [] ids: list[str] = [n for step in config["steps"] for n in step.get("probe", ())]
for group in ( ids += [n for nodeids in config["roles"]["evidence"].values() for n in nodeids]
config["roles"]["evidence"], ids += [
config["no_model_calls"]["checks"], n for check in config["no_model_calls"]["checks"].values() for n in check.get("probe", ())
config["foundry"]["evidence"], ]
): ids += [n for nodeids in config["foundry"]["evidence"].values() for n in nodeids]
ids += [n for nodeids in group.values() for n in nodeids]
return ids return ids
@ -692,16 +943,17 @@ def evaluate(
src: Path = _PACKAGE_SRC, src: Path = _PACKAGE_SRC,
probe_runner: ProbeRunner | None = None, probe_runner: ProbeRunner | None = None,
attest: Path | None = None, attest: Path | None = None,
now: datetime | None = None,
) -> list[Row]: ) -> list[Row]:
runner = probe_runner or (lambda ids: run_probes(ids, repo_root)) runner = probe_runner or (lambda ids: run_probes(ids, repo_root))
outcomes = runner(probe_nodeids(config)) outcomes = runner(probe_nodeids(config))
return [ return [
score_toolbox(config["steps"], src), score_toolbox(config["steps"], src, config.get("run_path"), outcomes, repo_root),
score_roles(config["roles"], outcomes, src), score_roles(config["roles"], outcomes, src),
score_no_claude_path(config["no_claude_path"], repo_root), score_no_claude_path(config["no_claude_path"], repo_root),
score_no_model_calls(config["no_model_calls"], outcomes), score_no_model_calls(config["no_model_calls"], outcomes, src),
score_foundry(config["foundry"], outcomes, src), score_foundry(config["foundry"], outcomes, src),
score_runbook(config["runbook"], repo_root, attest), score_runbook(config["runbook"], repo_root, attest, now),
] ]
@ -713,6 +965,11 @@ def render(rows: Sequence[Row]) -> str:
for attest in row.attests: for attest in row.attests:
out.append(f" [{row.title.split()[0]}] {attest}.") out.append(f" [{row.title.split()[0]}] {attest}.")
out.append("") out.append("")
for row in rows:
for note in row.diagnostics:
out.append(f" [{row.title.split()[0]}] {note}")
if any(r.diagnostics for r in rows):
out.append("")
out.append("Unntak fra 100 %:") out.append("Unntak fra 100 %:")
for row in rows: for row in rows:
for exception in row.exceptions: for exception in row.exceptions:

133
tests/test_b_gate.py
View file

@ -41,10 +41,10 @@ _NOW = datetime(2026, 9, 19, 12, 0, tzinfo=timezone.utc)
#: Repoets egne tall ved skriving (19.09.2026), talt av armene under mot KILDEN. De står her for #: Repoets egne tall ved skriving (19.09.2026), talt av armene under mot KILDEN. De står her for
#: at en stille endring i kjørestien skal vise seg som en rød test og ikke som et nytt tall. #: at en stille endring i kjørestien skal vise seg som en rød test og ikke som et nytt tall.
_STEPS_TODAY = 13 _STEPS_TODAY = 17
_EXTERNAL_TODAY = 3 _EXTERNAL_TODAY = 3
_ROLES_TODAY = ("proposer", "checker") _ROLES_TODAY = ("proposer", "checker")
_PATTERNS_TODAY = 6 _PATTERNS_TODAY = 9
def _write(path: Path, text: str) -> None: def _write(path: Path, text: str) -> None:
@ -100,16 +100,18 @@ def _independently_counted_steps() -> tuple[int, int]:
def test_the_step_denominator_equals_an_independent_count_of_the_source() -> None: def test_the_step_denominator_equals_an_independent_count_of_the_source() -> None:
resolved, called = _independently_counted_steps() resolved, called = _independently_counted_steps()
assert resolved == called == _STEPS_TODAY, "kjørestien har endret seg — oppdater nevneren" assert resolved == called == _STEPS_TODAY, "kjørestien har endret seg — oppdater nevneren"
row = gate.score_toolbox(_CONFIG["steps"], _SRC) row = gate.score_toolbox(_CONFIG["steps"], _SRC, _CONFIG["run_path"], {}, _REPO)
assert row.n == _STEPS_TODAY assert row.n == _STEPS_TODAY
def test_row1_today_is_red_with_three_of_thirteen_steps_callable_from_outside() -> None: def test_row1_today_is_red_with_three_of_seventeen_steps_callable_from_outside() -> None:
row = gate.score_toolbox(_CONFIG["steps"], _SRC) passing = {n: "passed" for step in _CONFIG["steps"] for n in step.get("probe", ())}
row = gate.score_toolbox(_CONFIG["steps"], _SRC, _CONFIG["run_path"], passing, _REPO)
assert (row.k, row.n, row.status) == (_EXTERNAL_TODAY, _STEPS_TODAY, gate.RED) assert (row.k, row.n, row.status) == (_EXTERNAL_TODAY, _STEPS_TODAY, gate.RED)
named = {x.split(":")[0] for x in row.exceptions} named = {x.split(":")[0] for x in row.exceptions}
assert "rundebinding" not in named and "gate" not in named assert "rundebinding" not in named and "gate" not in named
assert "validering" in named and "utboks" in named assert "validering" in named and "utboks" in named
assert "prepass-artefakt" in named and "parse-feil" in named
def _fake_src(root: Path, *, chat_client: bool, symbol: str = "gjoer_noe") -> Path: def _fake_src(root: Path, *, chat_client: bool, symbol: str = "gjoer_noe") -> Path:
@ -126,11 +128,17 @@ def _fake_src(root: Path, *, chat_client: bool, symbol: str = "gjoer_noe") -> Pa
"def _hjelper():\n" + body + "\n\n" "def _hjelper():\n" + body + "\n\n"
"def kjor():\n" "def kjor():\n"
f" _hjelper()\n return {symbol}()\n\n\n" f" _hjelper()\n return {symbol}()\n\n\n"
"def main(argv=None):\n return kjor()\n", "def main(argv=None):\n return kjor()\n\n\n"
'if __name__ == "__main__":\n raise SystemExit(main())\n',
) )
return src return src
#: En dør er registrert OG bevist med atferd. De konstruerte kildene under bærer begge deler.
_FAKE_PROBE = "tests/test_konstruert.py::test_doeren_skriver_artefaktet"
_FAKE_PASSED = {_FAKE_PROBE: "passed"}
def _one_step(symbol: str = "gjoer_noe") -> list[dict[str, Any]]: def _one_step(symbol: str = "gjoer_noe") -> list[dict[str, Any]]:
return [ return [
{ {
@ -139,21 +147,22 @@ def _one_step(symbol: str = "gjoer_noe") -> list[dict[str, Any]]:
"module": "steg.py", "module": "steg.py",
"symbol": symbol, "symbol": symbol,
"driver": {"module": "driver.py", "scope": "kjor"}, "driver": {"module": "driver.py", "scope": "kjor"},
"entry": {"kind": "cli", "module": "driver.py", "scope": "main"}, "entry": {"kind": "module-main", "module": "driver.py", "scope": "main"},
"probe": [_FAKE_PROBE],
} }
] ]
def test_a_step_reachable_from_a_cli_entry_without_a_chat_client_counts(tmp_path: Path) -> None: def test_a_step_reachable_from_a_cli_entry_without_a_chat_client_counts(tmp_path: Path) -> None:
src = _fake_src(tmp_path, chat_client=False) src = _fake_src(tmp_path, chat_client=False)
row = gate.score_toolbox(_one_step(), src) row = gate.score_toolbox(_one_step(), src, None, _FAKE_PASSED, tmp_path)
assert (row.k, row.n, row.status) == (1, 1, gate.GREEN) assert (row.k, row.n, row.status) == (1, 1, gate.GREEN)
def test_m2_a_step_that_needs_a_chat_client_is_not_callable_from_outside(tmp_path: Path) -> None: def test_m2_a_step_that_needs_a_chat_client_is_not_callable_from_outside(tmp_path: Path) -> None:
"""M-2: fjernes chatklient-sjekken, hopper k fra 0 til 1 og denne armen feller mutanten.""" """M-2: fjernes chatklient-sjekken, hopper k fra 0 til 1 og denne armen feller mutanten."""
src = _fake_src(tmp_path, chat_client=True) src = _fake_src(tmp_path, chat_client=True)
row = gate.score_toolbox(_one_step(), src) row = gate.score_toolbox(_one_step(), src, None, _FAKE_PASSED, tmp_path)
assert (row.k, row.n, row.status) == (0, 1, gate.RED) assert (row.k, row.n, row.status) == (0, 1, gate.RED)
assert any("chatklient" in x for x in row.exceptions) assert any("chatklient" in x for x in row.exceptions)
@ -170,10 +179,10 @@ def test_m1_the_denominator_follows_the_source_and_is_not_the_config_length(
"module": "steg.py", "module": "steg.py",
"symbol": "finnes_ikke", "symbol": "finnes_ikke",
"driver": {"module": "driver.py", "scope": "kjor"}, "driver": {"module": "driver.py", "scope": "kjor"},
"entry": {"kind": "cli", "module": "driver.py", "scope": "main"}, "entry": {"kind": "module-main", "module": "driver.py", "scope": "main"},
} }
] ]
row = gate.score_toolbox(steps, src) row = gate.score_toolbox(steps, src, None, _FAKE_PASSED, tmp_path)
assert row.n == 1, "et steg uten symbol i kilden er ikke et steg" assert row.n == 1, "et steg uten symbol i kilden er ikke et steg"
assert any("borte" in x for x in row.exceptions) assert any("borte" in x for x in row.exceptions)
@ -188,10 +197,10 @@ def test_a_step_the_driver_no_longer_calls_leaves_the_denominator(tmp_path: Path
"module": "steg.py", "module": "steg.py",
"symbol": "ubrukt", "symbol": "ubrukt",
"driver": {"module": "driver.py", "scope": "kjor"}, "driver": {"module": "driver.py", "scope": "kjor"},
"entry": {"kind": "cli", "module": "driver.py", "scope": "main"}, "entry": {"kind": "module-main", "module": "driver.py", "scope": "main"},
} }
] ]
row = gate.score_toolbox(steps, src) row = gate.score_toolbox(steps, src, None, _FAKE_PASSED, tmp_path)
assert row.n == 1 assert row.n == 1
@ -199,7 +208,7 @@ def test_an_entry_without_a_main_is_no_entry(tmp_path: Path) -> None:
src = _fake_src(tmp_path, chat_client=False) src = _fake_src(tmp_path, chat_client=False)
text = (src / "driver.py").read_text(encoding="utf-8").replace("def main(", "def _main(") text = (src / "driver.py").read_text(encoding="utf-8").replace("def main(", "def _main(")
_write(src / "driver.py", text) _write(src / "driver.py", text)
row = gate.score_toolbox(_one_step(), src) row = gate.score_toolbox(_one_step(), src, None, _FAKE_PASSED, tmp_path)
assert (row.k, row.n) == (0, 1) assert (row.k, row.n) == (0, 1)
assert any("main" in x for x in row.exceptions) assert any("main" in x for x in row.exceptions)
@ -263,12 +272,18 @@ def test_every_pattern_hits_its_own_known_positive_and_misses_its_known_negative
assert not rx.search(gate.decode(spec["known_negative_b64"])), spec["id"] assert not rx.search(gate.decode(spec["known_negative_b64"])), spec["id"]
def _clean_surface(tmp_path: Path) -> dict[str, Any]:
"""En ren, konstruert flate med sin egen sentinel — nevneren er aldri 0 her."""
cfg = json.loads(json.dumps(_CONFIG["no_claude_path"]))
cfg["manifest"] = {"sentinel": "src/rent.py", "skip_dirs": [".git", "__pycache__"]}
_write(tmp_path / "src" / "rent.py", "print('ingenting her')\n")
return cfg
def test_m3_the_denominator_is_the_number_of_patterns_that_can_measure(tmp_path: Path) -> None: def test_m3_the_denominator_is_the_number_of_patterns_that_can_measure(tmp_path: Path) -> None:
"""M-3: fjernes et mønster, faller nevneren — og et mønster som ikke treffer sin egen """M-3: fjernes et mønster, faller nevneren — og et mønster som ikke treffer sin egen
kjent-positive telles ikke som en vakt.""" kjent-positive telles ikke som en vakt."""
cfg = json.loads(json.dumps(_CONFIG["no_claude_path"])) cfg = _clean_surface(tmp_path)
cfg["roots"] = ["src"]
_write(tmp_path / "src" / "rent.py", "print('ingenting her')\n")
row = gate.score_no_claude_path(cfg, tmp_path) row = gate.score_no_claude_path(cfg, tmp_path)
assert (row.k, row.n, row.status) == (_PATTERNS_TODAY, _PATTERNS_TODAY, gate.GREEN) assert (row.k, row.n, row.status) == (_PATTERNS_TODAY, _PATTERNS_TODAY, gate.GREEN)
@ -283,8 +298,8 @@ def test_m3_the_denominator_is_the_number_of_patterns_that_can_measure(tmp_path:
def test_a_planted_path_to_claude_turns_the_row_red_for_that_pattern(tmp_path: Path) -> None: def test_a_planted_path_to_claude_turns_the_row_red_for_that_pattern(tmp_path: Path) -> None:
cfg = json.loads(json.dumps(_CONFIG["no_claude_path"])) cfg = _clean_surface(tmp_path)
cfg["roots"] = ["src"] assert gate.score_no_claude_path(cfg, tmp_path).status == gate.GREEN, "rc-0-kontrollen først"
for spec in cfg["patterns"]: for spec in cfg["patterns"]:
_write( _write(
tmp_path / "src" / f"{spec['id']}.py", gate.decode(spec["known_positive_b64"]) + "\n" tmp_path / "src" / f"{spec['id']}.py", gate.decode(spec["known_positive_b64"]) + "\n"
@ -297,8 +312,7 @@ def test_a_planted_path_to_claude_turns_the_row_red_for_that_pattern(tmp_path: P
def test_markdown_counts_only_inside_fenced_blocks(tmp_path: Path) -> None: def test_markdown_counts_only_inside_fenced_blocks(tmp_path: Path) -> None:
"""«docs med kjørbare kommandoer»: en pakkenavn-omtale i brødtekst er ikke en vei til Claude; """«docs med kjørbare kommandoer»: en pakkenavn-omtale i brødtekst er ikke en vei til Claude;
den samme linja i en kodeblokk er en kommando noen kan kjøre.""" den samme linja i en kodeblokk er en kommando noen kan kjøre."""
cfg = json.loads(json.dumps(_CONFIG["no_claude_path"])) cfg = _clean_surface(tmp_path)
cfg["roots"] = ["docs"]
spec = next(s for s in cfg["patterns"] if s["id"] == "maf-anthropic") spec = next(s for s in cfg["patterns"] if s["id"] == "maf-anthropic")
line = gate.decode(spec["known_positive_b64"]) line = gate.decode(spec["known_positive_b64"])
_write(tmp_path / "docs" / "prosa.md", f"- vurdert og forkastet: `{line}` (aldri tatt inn)\n") _write(tmp_path / "docs" / "prosa.md", f"- vurdert og forkastet: `{line}` (aldri tatt inn)\n")
@ -317,7 +331,7 @@ def test_row3_on_the_real_published_surface_is_green_today() -> None:
def test_the_surface_names_its_denominator_in_files_not_in_prose() -> None: def test_the_surface_names_its_denominator_in_files_not_in_prose() -> None:
files = gate.surface_files(_REPO, _CONFIG["no_claude_path"]["roots"]) files, _ = gate.published_files(_REPO, _manifest())
assert len(files) > 100, "en tom flate ville gitt 0 treff uten å måle noe" assert len(files) > 100, "en tom flate ville gitt 0 treff uten å måle noe"
assert any(f.name == "run.py" for f in files) assert any(f.name == "run.py" for f in files)
@ -328,22 +342,23 @@ def test_the_surface_names_its_denominator_in_files_not_in_prose() -> None:
def test_row4_is_red_today_and_names_all_three_checks() -> None: def test_row4_is_red_today_and_names_all_three_checks() -> None:
row = gate.score_no_model_calls(_CONFIG["no_model_calls"], {}) row = gate.score_no_model_calls(_CONFIG["no_model_calls"], {}, _SRC)
assert (row.k, row.n, row.status) == (0, 3, gate.RED) assert (row.k, row.n, row.status) == (0, 3, gate.RED)
assert len(row.exceptions) == 3 assert len(row.exceptions) == 3
def test_row4_counts_only_checks_whose_probes_pass() -> None: def test_row4_counts_only_checks_whose_probes_pass() -> None:
checks = _CONFIG["no_model_calls"]["checks"] checks = _CONFIG["no_model_calls"]["checks"]
outcomes = {n: "passed" for ns in checks.values() for n in ns} outcomes = {n: "passed" for c in checks.values() for n in c["probe"]}
assert gate.score_no_model_calls(_CONFIG["no_model_calls"], outcomes).status == gate.GREEN row = gate.score_no_model_calls(_CONFIG["no_model_calls"], outcomes, _SRC)
outcomes[checks["fullfører"][0]] = "skipped" assert row.status == gate.GREEN
row = gate.score_no_model_calls(_CONFIG["no_model_calls"], outcomes) outcomes[checks["fullfører"]["probe"][0]] = "skipped"
row = gate.score_no_model_calls(_CONFIG["no_model_calls"], outcomes, _SRC)
assert (row.k, row.status) == (2, gate.RED) assert (row.k, row.status) == (2, gate.RED)
def test_row4_states_what_the_budget_guard_is_in_b_instead_of_pretending() -> None: def test_row4_states_what_the_budget_guard_is_in_b_instead_of_pretending() -> None:
row = gate.score_no_model_calls(_CONFIG["no_model_calls"], {}) row = gate.score_no_model_calls(_CONFIG["no_model_calls"], {}, _SRC)
assert any("budsjett" in a.casefold() for a in row.attests) assert any("budsjett" in a.casefold() for a in row.attests)
assert "valgt" in gate.BUDGET_IN_B.casefold() assert "valgt" in gate.BUDGET_IN_B.casefold()
@ -368,11 +383,12 @@ def test_the_foundry_check_fails_when_the_azure_profile_is_gone(tmp_path: Path)
def test_row5_is_red_until_the_schema_comparison_probe_passes() -> None: def test_row5_is_red_until_the_schema_comparison_probe_passes() -> None:
structural = len(_CONFIG["foundry"]["profile"]["members"]) + 2
row = gate.score_foundry(_CONFIG["foundry"], {}, _SRC) row = gate.score_foundry(_CONFIG["foundry"], {}, _SRC)
assert (row.k, row.n, row.status) == (1, 2, gate.RED) assert (row.k, row.n, row.status) == (structural, structural + 1, gate.RED)
nodeid = _CONFIG["foundry"]["evidence"]["samme-skjema"][0] nodeid = _CONFIG["foundry"]["evidence"]["samme-skjema"][0]
row = gate.score_foundry(_CONFIG["foundry"], {nodeid: "passed"}, _SRC) row = gate.score_foundry(_CONFIG["foundry"], {nodeid: "passed"}, _SRC)
assert (row.k, row.n, row.status) == (2, 2, gate.GREEN) assert (row.k, row.n, row.status) == (structural + 1, structural + 1, gate.GREEN)
# --------------------------------------------------------------------------------------------- # ---------------------------------------------------------------------------------------------
@ -394,20 +410,26 @@ def test_row6_is_still_not_measured_without_the_operators_attestation(tmp_path:
def test_row6_goes_green_only_with_every_required_key(tmp_path: Path) -> None: def test_row6_goes_green_only_with_every_required_key(tmp_path: Path) -> None:
cfg = _CONFIG["runbook"] cfg = _CONFIG["runbook"]
_write(tmp_path / cfg["path"], "# kjørebok\n") digest = _runbook(tmp_path)
attest = tmp_path / cfg["attestation"] attest = tmp_path / cfg["attestation"]
_write(attest, "kjørebok: docs/kjoerebok-verktoeykassen.md\nkjørt: operatøren\n") _write(attest, f"kjørebok: {cfg['path']}\nsjekksum: {digest}\nkjørt: operatøren\n")
assert gate.score_runbook(cfg, tmp_path).status == gate.NOT_MEASURED row = gate.score_runbook(cfg, tmp_path, now=_NOW)
_write(attest, "kjørebok: x\nkjørt: operatøren\ndato: 2026-09-20\n") assert row.status == gate.NOT_MEASURED
row = gate.score_runbook(cfg, tmp_path) assert any("dato" in x for x in row.exceptions)
_attestation_lines(tmp_path, digest)
row = gate.score_runbook(cfg, tmp_path, now=_NOW)
assert (row.k, row.n, row.status) == (2, 2, gate.GREEN) assert (row.k, row.n, row.status) == (2, 2, gate.GREEN)
def test_a_duplicated_key_in_the_attestation_is_red(tmp_path: Path) -> None: def test_a_duplicated_key_in_the_attestation_is_red(tmp_path: Path) -> None:
cfg = _CONFIG["runbook"] cfg = _CONFIG["runbook"]
_write(tmp_path / cfg["path"], "# kjørebok\n") digest = _runbook(tmp_path)
_write(tmp_path / cfg["attestation"], "kjørebok: x\nkjørt: a\nkjørt: b\ndato: 2026-09-20\n") _attestation_lines(tmp_path, digest)
assert gate.score_runbook(cfg, tmp_path).status == gate.NOT_MEASURED path = tmp_path / cfg["attestation"]
path.write_text(path.read_text(encoding="utf-8") + "kjørt: en annen\n", encoding="utf-8")
row = gate.score_runbook(cfg, tmp_path, now=_NOW)
assert row.status == gate.NOT_MEASURED
assert any("to ganger" in x for x in row.exceptions)
def test_m5_the_gate_never_writes_the_attestation_itself(tmp_path: Path) -> None: def test_m5_the_gate_never_writes_the_attestation_itself(tmp_path: Path) -> None:
@ -686,13 +708,11 @@ def test_a_console_script_is_a_door_only_when_the_manifest_registers_it(tmp_path
def test_a_module_main_is_a_door_only_with_the_dunder_guard(tmp_path: Path) -> None: def test_a_module_main_is_a_door_only_with_the_dunder_guard(tmp_path: Path) -> None:
src = _fake_src(tmp_path, chat_client=False) src = _fake_src(tmp_path, chat_client=False)
entry = {"kind": "module-main", "module": "driver.py", "scope": "main"} entry = {"kind": "module-main", "module": "driver.py", "scope": "main"}
assert gate.registered_entry(src, tmp_path, "fakesrc", entry)[0] is False
_write(
src / "driver.py",
(src / "driver.py").read_text(encoding="utf-8")
+ '\n\nif __name__ == "__main__":\n raise SystemExit(main())\n',
)
assert gate.registered_entry(src, tmp_path, "fakesrc", entry) == (True, "") assert gate.registered_entry(src, tmp_path, "fakesrc", entry) == (True, "")
stripped = (src / "driver.py").read_text(encoding="utf-8")
_write(src / "driver.py", stripped[: stripped.index("if __name__")])
ok, why = gate.registered_entry(src, tmp_path, "fakesrc", entry)
assert ok is False and "__main__" in why
def test_a_subcommand_is_a_door_only_when_the_parser_registers_that_name(tmp_path: Path) -> None: def test_a_subcommand_is_a_door_only_when_the_parser_registers_that_name(tmp_path: Path) -> None:
@ -719,7 +739,7 @@ def test_a_stub_main_is_not_a_door_without_a_registered_entry(tmp_path: Path) ->
_fake_pyproject(tmp_path, {}) _fake_pyproject(tmp_path, {})
steps = _one_step() steps = _one_step()
steps[0]["entry"] = {"kind": "console-script", "module": "driver.py", "scope": "main"} steps[0]["entry"] = {"kind": "console-script", "module": "driver.py", "scope": "main"}
row = gate.score_toolbox(steps, src, None, {}, tmp_path) row = gate.score_toolbox(steps, src, None, _FAKE_PASSED, tmp_path)
assert (row.k, row.n) == (0, 1) assert (row.k, row.n) == (0, 1)
assert any("pyproject" in x for x in row.exceptions), row.exceptions assert any("pyproject" in x for x in row.exceptions), row.exceptions
@ -729,32 +749,21 @@ def test_a_registered_door_without_a_passing_behaviour_probe_does_not_count(
) -> None: ) -> None:
"""«Kallet med en fixture gir det dokumenterte artefaktet» — et navn er ikke en atferd.""" """«Kallet med en fixture gir det dokumenterte artefaktet» — et navn er ikke en atferd."""
src = _fake_src(tmp_path, chat_client=False) src = _fake_src(tmp_path, chat_client=False)
_write(
src / "driver.py",
(src / "driver.py").read_text(encoding="utf-8")
+ '\n\nif __name__ == "__main__":\n raise SystemExit(main())\n',
)
steps = _one_step() steps = _one_step()
steps[0]["entry"] = {"kind": "module-main", "module": "driver.py", "scope": "main"}
steps[0]["probe"] = ["tests/test_noe.py::test_doeren_skriver_artefaktet"]
row = gate.score_toolbox(steps, src, None, {}, tmp_path) row = gate.score_toolbox(steps, src, None, {}, tmp_path)
assert (row.k, row.n) == (0, 1) assert (row.k, row.n) == (0, 1)
assert any("probe" in x for x in row.exceptions), row.exceptions assert any("atferdsprobe" in x for x in row.exceptions), row.exceptions
passed = {"tests/test_noe.py::test_doeren_skriver_artefaktet": "passed"} row = gate.score_toolbox(steps, src, None, {_FAKE_PROBE: "skipped"}, tmp_path)
row = gate.score_toolbox(steps, src, None, passed, tmp_path) assert row.k == 0, "hoppet over er ikke bestått"
row = gate.score_toolbox(steps, src, None, _FAKE_PASSED, tmp_path)
assert (row.k, row.n, row.status) == (1, 1, gate.GREEN) assert (row.k, row.n, row.status) == (1, 1, gate.GREEN)
def test_a_step_without_any_probe_is_never_a_door(tmp_path: Path) -> None: def test_a_step_without_any_probe_is_never_a_door(tmp_path: Path) -> None:
src = _fake_src(tmp_path, chat_client=False) src = _fake_src(tmp_path, chat_client=False)
_write(
src / "driver.py",
(src / "driver.py").read_text(encoding="utf-8")
+ '\n\nif __name__ == "__main__":\n raise SystemExit(main())\n',
)
steps = _one_step() steps = _one_step()
steps[0]["entry"] = {"kind": "module-main", "module": "driver.py", "scope": "main"} steps[0].pop("probe")
row = gate.score_toolbox(steps, src, None, {}, tmp_path) row = gate.score_toolbox(steps, src, None, _FAKE_PASSED, tmp_path)
assert (row.k, row.n) == (0, 1) assert (row.k, row.n) == (0, 1)
assert any("ingen atferdsprobe" in x for x in row.exceptions), row.exceptions assert any("ingen atferdsprobe" in x for x in row.exceptions), row.exceptions