open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 3 Activity

fix(b-gate): a probe is bound to its step, a reason cannot be empty, and the row's limit is what was measured [skip-docs]

Browse source 
Row 1 goes 3 of 17 -> 0 of 17. Nothing was removed from the product and no row
changed colour; the three that counted stopped counting because the contract they
satisfied was satisfiable without capability.

BEARING 1 -- the probe is bound to the step, and the binding is MEASURED in the
probe's own source. Chosen: read the probe (ast) rather than demand it live in a
contract-named file, because a file name is a convention a stub meets as easily as a
real probe. Three traits, each measured: it drives the DOOR (the dotted module or the
registered command name appears as a string it uses -- anywhere but a docstring,
because the honest form assembles argv in a variable first), it names the STEP (the
symbol, id or subcommand as a whole WORD in what it passes INTO a call or calls), and
it asserts at all. A probe claimed by two steps proves at most one and the gate cannot
tell which -- so neither.

Two traps found while measuring, both closed:
  - substring: "gate" is not named by portfolio_optimiser.evals.v1_gate
  - local alias: the first cut accepted step `gate` because the probe file imports the
    module AS `gate`. Names are therefore read only where they are sent or called.

What that costs, measured against the contract that stands:
  rundebinding  drives the door, names no step         (was green)
  rapport       never goes through the door at all     (was green)
  gate          drives the door, names no step         (was green)

BEARING 2 -- the limit statement said exactly TWO ways remained; the checkpoint
planted 21 call forms and measured SIX. Four are closed with a guard each (the
official Python SDK in both spellings, the node and uv runners, a dynamic import);
three remain and are now named: a runtime-composed name, a name from an environment
variable, a base64-decoded name. Left open deliberately -- the encodings are not
enumerable and our own contract stores base64 by design. Row 3: 9 of 9 -> 12 of 12,
still GREEN, 0 hits over 512 files. One of the three caught a command written in this
round's own test docstring; it was rewritten, not exempted.

BEARING 3 -- held_out accepted an EMPTY reason and shrank the denominator, while the
summary said "held out with a reason" either way. A blank reason is no reason: the
symbol stays in the denominator as a call without a door, the summary counts reasons,
and the four steps declared OUTSIDE the run path are now named one by one as having no
derived source instead of being counted in silence.

Five small rests, closed: a pruned manifest (451 of 512 was still GREEN) is now NOT
MEASURED, one sentinel per area the old handlist missed; a non-UTF-8 file is read as
byte text instead of counted and skipped; a symlink out of the tree is named and
fails the row; a runbook whose whole content is "x" no longer passes, the contract
names its sections; and the row states that its ratio is not a coverage measure.

Two stated, not closed, each with its reason in the row's own attestation: a po call
moved one floor down into a helper leaves the denominator (following helpers would
pull private ones in and make the denominator the curated list this row exists to
avoid), and row 3's k/n can still be padded by a guard with no measured escape behind
it. The ledger's two precision errors are corrected: "no row became greener" is true
of colour, not of numbers, and ENTRY_KINDS has three arts, not four.

Suite: 2172 passed, 5 skipped, 5 xfailed in 645 s. ruff and mypy clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-19 21:48:44 +02:00
commit 9eec31b7da
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
4 changed files with 441 additions and 73 deletions
Download patch file Download diff file Expand all files Collapse all files

44
tests/test_b_gate.py
View file

@ -44,9 +44,9 @@ _NOW = datetime(2026, 9, 19, 12, 0, tzinfo=timezone.utc)
#: Repoets egne tall ved skriving (19.09.2026), talt av armene under mot KILDEN. De står her for
#: at en stille endring i kjørestien skal vise seg som en rød test og ikke som et nytt tall.
_STEPS_TODAY = 17
_EXTERNAL_TODAY = 3
_EXTERNAL_TODAY = 0
_ROLES_TODAY = ("proposer", "checker")
_PATTERNS_TODAY = 9
_PATTERNS_TODAY = 12
def _write(path: Path, text: str) -> None:
@ -106,14 +106,18 @@ def test_the_step_denominator_equals_an_independent_count_of_the_source() -> Non
assert row.n == _STEPS_TODAY
def test_row1_today_is_red_with_three_of_seventeen_steps_callable_from_outside() -> None:
def test_row1_today_is_red_with_no_step_callable_from_outside() -> None:
"""Var «3 av 17» til 19.09. De tre falt da proben ble BUNDET til steget: to av dem driver
riktig dør uten å navngi hvilket steg de beviser, den tredje går ikke gjennom døren i det
hele tatt. Ingen av dem var en kodefeil — det var kontrakten som godtok dem."""
passing = {n: "passed" for step in _CONFIG["steps"] for n in step.get("probe", ())}
row = gate.score_toolbox(_CONFIG["steps"], _SRC, _CONFIG["run_path"], passing, _REPO)
assert (row.k, row.n, row.status) == (_EXTERNAL_TODAY, _STEPS_TODAY, gate.RED)
named = {x.split(":")[0] for x in row.exceptions}
assert "rundebinding" not in named and "gate" not in named
assert "validering" in named and "utboks" in named
assert "prepass-artefakt" in named and "parse-feil" in named
bound = {x.split(":")[0] for x in row.exceptions if "atferdsprobe" in x}
assert bound == {"rundebinding", "rapport", "gate"}, row.exceptions
def _fake_src(root: Path, *, chat_client: bool, symbol: str = "gjoer_noe") -> Path:
@ -441,7 +445,7 @@ def test_row6_is_not_measured_while_the_runbook_is_missing(tmp_path: Path) -> No
def test_row6_is_still_not_measured_without_the_operators_attestation(tmp_path: Path) -> None:
_write(tmp_path / _CONFIG["runbook"]["path"], "# kjørebok\n\nsteg 1 …\n")
_runbook(tmp_path)
row = gate.score_runbook(_CONFIG["runbook"], tmp_path)
assert (row.k, row.n, row.status) == (1, 2, gate.NOT_MEASURED)
@ -840,9 +844,7 @@ def test_the_surface_count_matches_an_independent_count_of_the_manifest() -> Non
repository») finnes ikke manifestet å telle mot, og armen HOPPER OVER seg selv i stedet for å
felle: ellers vokser «røde i uttrekket» med hver ny git-avhengig arm, og tallet slutter å
være et funn. Nevneren pinnes HER, ikke i armen over."""
listing = subprocess.run(
["git", "ls-files", "-z"], cwd=_REPO, capture_output=True, text=True
)
listing = subprocess.run(["git", "ls-files", "-z"], cwd=_REPO, capture_output=True, text=True)
if listing.returncode != 0:
pytest.skip("uttrekk uten git-metadata: manifestet finnes ikke å telle mot")
mine = {t for t in listing.stdout.split("\0") if t}
@ -870,7 +872,7 @@ def test_the_row_names_which_manifest_it_read() -> None:
files, source = gate.published_files(_REPO, _manifest())
assert f"over {len(files)} publiserte filer" in row.reason
assert source in row.reason
assert f"{_UNDECODABLE_TODAY} ulesbare" in row.reason
assert f"{_UNDECODABLE_TODAY} ikke-UTF-8, lest som byte-tekst" in row.reason
def test_the_surface_falls_back_to_the_tree_when_there_is_no_git_metadata(tmp_path: Path) -> None:
@ -982,7 +984,11 @@ def test_row6_denominator_comes_from_the_artefacts_the_contract_names(tmp_path:
def _runbook(root: Path) -> str:
text = "# kjørebok\n\nsteg 1: naviger pakken\n"
"""En kjørebok som bærer seksjonene kontrakten navngir — ellers er «x» en kjørebok (N3)."""
sections = "\n".join(
f"## {s}\n\nsteg 1: naviger pakken\n" for s in _CONFIG["runbook"]["sections"]
)
text = f"# kjørebok\n\n{sections}"
_write(root / _CONFIG["runbook"]["path"], text)
return hashlib.sha256(text.encode("utf-8")).hexdigest()
@ -1171,9 +1177,7 @@ def test_one_probe_cannot_prove_two_steps(tmp_path: Path) -> None:
a = _write_probe(tmp_path, needle="gjoer_noe", nodeid="tests/test_a.py::test_a")
b = _write_probe(tmp_path, needle="gjoer_noe_annet", nodeid="tests/test_b.py::test_b")
outcomes = {a: "passed", b: "passed"}
row = gate.score_toolbox(
_two_steps(a, b), src, _fake_run_path(), outcomes, tmp_path
)
row = gate.score_toolbox(_two_steps(a, b), src, _fake_run_path(), outcomes, tmp_path)
assert (row.k, row.n, row.status) == (2, 2, gate.GREEN), row.exceptions
shared = _write_probe(
@ -1188,7 +1192,9 @@ def test_one_probe_cannot_prove_two_steps(tmp_path: Path) -> None:
assert all("delt" in x for x in row.exceptions), row.exceptions
def test_the_same_unrelated_green_test_cannot_turn_the_real_row1_into_a_door(tmp_path: Path) -> None:
def test_the_same_unrelated_green_test_cannot_turn_the_real_row1_into_a_door(
tmp_path: Path,
) -> None:
"""N2b mot KONTRAKTEN som står: hver av de 17 stegenes probe byttet ut med én urelatert
bestått test. Målt 19.09 ga det «17 av 17 GRØNN» med fjorten stubber; her er nevneren
repoets egen kontrakt, og ingen av stegene skal telle."""
@ -1297,9 +1303,13 @@ _MEASURED_ESCAPES = (
def test_every_way_the_checkpoint_measured_as_open_and_cheap_to_close_is_caught(
tmp_path: Path,
) -> None:
"""De seks formene PM plantet og målte som USETTE 19.09: den offisielle ``anthropic``-SDK-en
(to skriveformer), ``npx @anthropic-ai/claude-code`` (to former), ``uvx claude-code`` og et
dynamisk import-kall. Hver plantes én om gangen, med rc-0-kontroll først og etter."""
"""De seks formene PM plantet og målte som USETTE 19.09: den offisielle Python-SDK-en (to
skriveformer), node-kjøreren (to former), uv-kjøreren og et dynamisk import-kall. Hver
plantes én om gangen, med rc-0-kontroll først og etter.
Formene STÅR IKKE skrevet her, bare base64-kodet — denne fila er selv en del av flaten raden
skanner, og en klartekst-variant ville vært sitt eget funn. Målt: den var det, i første
utkast av denne docstringen, og vakten fant den."""
cfg = json.loads(json.dumps(_CONFIG["no_claude_path"]))
cfg["manifest"] = {"sentinel": "src/run.py", "sentinels": ["src/run.py"]}
_write(tmp_path / "src" / "run.py", "x = 1\n")