open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 4 Activity

feat(p21): a declaration that must have LOOKED, and a refusal that names the neighbours

Browse source 
C1. Round 4 produced 13 declarations over six runs and NOT ONE named a fasit concept.
The distinct documents opened before each were 1,1,1,1,1,1,1,2,5,5,6,13,13: seven
declared the base's FIRST requirement after opening exactly ONE document.

The order offered two rules and asked which discriminates. Replayed against the real
listings: "the declared document must have come back from a read_dir filtered on a word
from the approach's label" refuses 13 of 13 -- including Soraasen's 12.11, the closest
any run came -- because ZERO of the 13 were reached through a filtered listing at all.
A gate that refuses every measured case, right and wrong alike, cannot discriminate.
"fewer than k distinct documents opened" at k=3 refuses 8 of 13 and keeps the five that
navigated. k=3, 4 and 5 refuse the SAME eight -- the distribution has a gap between 2
and 5 -- so the threshold is not on a cliff, and 3 is the lowest of that plateau.
DISTINCT paths, not calls, and capped by the base's own size so a small base stays
declarable.

C2. Over the same traces 18 of 143 path-bearing calls named a path the base does not
hold, ELEVEN of them one run walking R761/4-3, 4.3, 4-2, 4-1, 4-0, 4-5, 4-6 while the
real names are R761/4, R761/41, R761/42. The refusal already named the nearest listable
ancestor; now it also names up to five of that rung's own subdirectories, ranked by
longest common prefix with the segment that failed. ONE copy shared by both refusal
sites, built from context_files through in_dimension, so every name handed back resolves
and the verdict layer can never be advertised in an apology. MEASURED after: 16 of 18.

Load-bearing MEASURED, four mutations all red against the WHOLE suite, green control
1863/5, golden byte-unchanged: C3(i) the declaration gate detached (2 red) . C3(ii) the
neighbour list empty (6) . C3(iii) built from files (1, the verdict arm alone) . C3(iv)
count CALLS instead of distinct documents (1, the repetition arm alone).

Three existing arms REWRITTEN, not weakened: all three read one document and declared,
which is the measured failure class exactly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-15 11:38:57 +02:00
commit ac0bfdba27
8 changed files with 534 additions and 28 deletions
Download patch file Download diff file Expand all files Collapse all files

7
tests/test_across_bundles_cli_loadbearing.py
View file

@ -366,8 +366,11 @@ def test_the_requirement_gate_reads_the_second_bases_own_opened_list(tmp_path: P
assert refusal["refusal"] == "RequirementNotRead", refusal
assert not reqs_2, "base 2 recorded a requirement it never read"
tools_2["read_file"].func(bundle_2, doc_2)
opened_2.append(explore.ToolCall(name="read_file", bundle_id=bundle_2, path=doc_2))
# P21/C1: a declaration also needs the run to have LOOKED — at least three distinct documents
# of the base, capped by its size — so the correction here is the same one a real run makes.
for name in [f.name for f in okf.navigate_bundle(second).context_files][:3]:
tools_2["read_file"].func(bundle_2, name)
opened_2.append(explore.ToolCall(name="read_file", bundle_id=bundle_2, path=name))
accepted = tools_2["declare_requirement"].func(bundle_2, doc_2, "Krav 1")
assert accepted.get("declared") is True, accepted
assert [r.path for r in reqs_2] == [doc_2]

15
tests/test_binding_requirement_loadbearing.py
View file

@ -110,13 +110,24 @@ def test_a_requirement_the_run_never_opened_is_refused_and_recorded_nowhere() ->
def test_the_correction_is_to_read_it_and_then_it_is_accepted() -> None:
"""(b) The refusal is a turn the model can correct — the half that makes (a) a gate."""
"""(b) The refusal is a turn the model can correct — the half that makes (a) a gate.
P21/C1 added a SECOND precondition — the run must have opened at least three distinct
documents (capped by the base's own size) — so the correction is now "read it, and read enough
of the base to have looked". The arm keeps its meaning: a declaration the run's own trace
supports is ACCEPTED and RECORDED.
"""
tools, opened, declared = _wired()
path = _a_concept()
from portfolio_optimiser import okf
files = [f.name for f in okf.navigate_bundle(str(_TUNNEL)).context_files]
path = files[0]
body = tools["read_file"].func(bundle_id="tunnel-hauglia", path=path)
assert not body.startswith("REFUSED"), body[:120]
# The recorder is middleware in a real run; here the trace is appended directly, which is the
# SAME list the tool reads.
for other in files[1:3]:
opened.append(ToolCall(name="read_file", bundle_id="tunnel-hauglia", path=other))
opened.append(ToolCall(name="read_file", bundle_id="tunnel-hauglia", path=path))
answer = tools["declare_requirement"].func(
bundle_id="tunnel-hauglia", path=path, ref="Krav 12.1"

282
tests/test_declaration_and_neighbours_loadbearing.py Normal file
View file

@ -0,0 +1,282 @@
"""P21 DEL C — a declaration that must have LOOKED, and a refusal that names the neighbours.
**C1, and the order asked for a measurement rather than a preference.** P19 DEL A made a direction
name the requirement that binds it, and the rung works — round 4 produced 13 declarations over six
runs. Not one named a fasit concept, and the distinct documents opened before each were
``1,1,1,1,1,1,1,2,5,5,6,13,13``: seven declared the base's FIRST requirement after opening exactly
ONE document.
The order offered two rules and asked which discriminates. Replayed against the real listings:
* **"the declared document must have been returned by a ``read_dir`` filtered on a word from the
approach's label"** refuses **13 of 13** — including Søråsen's ``12.11``, which the order names
as the closest any run came. Measured, ZERO of the 13 declarations were reached through a
filtered listing at all. A gate that refuses every measured case, right and wrong alike, cannot
discriminate; it is the vacuous gate's mirror image.
* **"fewer than k distinct documents opened"** at k=3 refuses **8 of 13** and keeps the five that
navigated, ``12.11`` among them. k=3, 4 and 5 refuse the SAME eight — the distribution has a gap
between 2 and 5 — so the threshold is not on a cliff, and 3 is the lowest of that plateau.
The second is built. It is capped by the base's own document count, so a two-document base stays
declarable rather than becoming a level nobody can declare on.
**C2.** Over the same six traces, 18 of 143 path-bearing tool calls named a path the base does not
hold, and ELEVEN of them are one run walking ``R761/4-3``, ``4.3``, ``4-2``, ``4-1``, ``4-0``,
``4-5``, ``4-6`` — guessing a chapter-number spelling the corpus does not use, while the real names
are ``R761/4``, ``R761/41``, ``R761/42``. The refusal already named the nearest LISTABLE ancestor,
which is the right rung; what it could not say is which of that rung's names was meant. Measured
after: **16 of 18** refusals now name at least one neighbour, and the two that do not are paths
whose ancestor holds documents and no subdirectories — an empty list is omitted rather than
rendered as a sentence with nothing in it.
"""
from __future__ import annotations
from pathlib import Path
from typing import Any
import pytest
from portfolio_optimiser import okf
from portfolio_optimiser.explore import ToolCall, navigator_tools
_EXAMPLES = Path(__file__).resolve().parents[1] / "shared" / "examples"
_TUNNEL = _EXAMPLES / "tunnel-hauglia"
def _wired(bundle_dir: Path = _TUNNEL) -> tuple[dict[str, Any], list[ToolCall], list[Any]]:
opened: list[ToolCall] = []
declared: list[Any] = []
tools = navigator_tools((str(bundle_dir),), opened=opened, requirements=declared)
return {t.name: t for t in tools}, opened, declared
def _concepts(bundle_dir: Path = _TUNNEL) -> list[str]:
return [f.name for f in okf.navigate_bundle(str(bundle_dir)).context_files]
# --- C1 -------------------------------------------------------------------------------------
def test_c1_a_declaration_after_one_document_is_refused_with_the_denominator() -> None:
"""The KNOWN-POSITIVE: exactly the measured failure — open one, declare it.
Both numbers are in the refusal, because the pair is the diagnosis: "1 of 5" says something
"1" alone does not, and a refusal that stated neither would leave the model guessing how much
more is wanted (``BudgetExceeded``'s kø-(y) rule, one rung down).
"""
tools, opened, declared = _wired()
path = _concepts()[0]
opened.append(ToolCall(name="read_file", bundle_id="tunnel-hauglia", path=path))
answer = tools["declare_requirement"].func(
bundle_id="tunnel-hauglia", path=path, ref="Krav 1.1—1"
)
assert answer["refusal"] == "RequirementNotRead"
assert "1 distinct document(s)" in answer["refused"]
assert "of the 5" in answer["refused"], answer["refused"]
assert "at least 3" in answer["refused"]
assert declared == [], "a refused declaration must leave no record"
def test_c1_the_same_declaration_after_three_documents_is_accepted() -> None:
"""The KNOWN-NEGATIVE, and the half that keeps C1 from being a gate that only refuses.
Same base, same declared document, same ``ref`` — the ONLY difference is how much of the base
the run had opened. Without this arm the refusal above would be indistinguishable from one that
fires on everything, which is exactly what the rule the measurement rejected does.
"""
tools, opened, declared = _wired()
files = _concepts()
for name in files[:3]:
opened.append(ToolCall(name="read_file", bundle_id="tunnel-hauglia", path=name))
answer = tools["declare_requirement"].func(
bundle_id="tunnel-hauglia", path=files[0], ref="Krav 1.1—1"
)
assert answer["declared"] is True
assert len(declared) == 1
def test_c1_the_same_path_read_three_times_is_still_one_document() -> None:
"""DISTINCT paths, not calls. Re-reading one document is not navigation, and a rule counting
calls would be satisfied by the cheapest possible loop."""
tools, opened, declared = _wired()
path = _concepts()[0]
for _ in range(3):
opened.append(ToolCall(name="read_file", bundle_id="tunnel-hauglia", path=path))
answer = tools["declare_requirement"].func(
bundle_id="tunnel-hauglia", path=path, ref="Krav 1.1—1"
)
assert answer["refusal"] == "RequirementNotRead"
assert "1 distinct document(s)" in answer["refused"]
assert declared == []
def test_c1_a_small_base_stays_declarable(tmp_path: Path) -> None:
"""Capped by the base's OWN size: two documents can be read whole in two.
A fixed floor above a base's size would make declaration impossible there — a gate that can
only refuse, on exactly the small fixtures this repo is built on.
"""
base = tmp_path / "mini"
base.mkdir()
(base / "index.md").write_text(
"---\nbundle_id: mini\n---\n\n- [A](a.md) — one.\n- [B](b.md) — two.\n", encoding="utf-8"
)
(base / "a.md").write_text("---\ntype: Krav\ntitle: A\n---\n\nbody A\n", encoding="utf-8")
(base / "b.md").write_text("---\ntype: Krav\ntitle: B\n---\n\nbody B\n", encoding="utf-8")
tools, opened, declared = _wired(base)
for name in ("a.md", "b.md"):
opened.append(ToolCall(name="read_file", bundle_id="mini", path=name))
answer = tools["declare_requirement"].func(bundle_id="mini", path="a.md", ref="1")
assert answer["declared"] is True
assert len(declared) == 1
def test_c1_the_never_opened_refusal_still_fires_first() -> None:
"""The P19 gate is UNCHANGED and still the first thing checked: "you never read this" and "you
have barely read anything" are different facts, and the first is the one the caller can fix
with one call."""
tools, opened, declared = _wired()
files = _concepts()
for name in files[:3]:
opened.append(ToolCall(name="read_file", bundle_id="tunnel-hauglia", path=name))
answer = tools["declare_requirement"].func(
bundle_id="tunnel-hauglia", path=files[4], ref="Krav 1.1—1"
)
assert answer["refusal"] == "RequirementNotRead"
assert "is not one of them" in answer["refused"]
assert declared == []
# --- C2 -------------------------------------------------------------------------------------
def _r761_shaped(root: Path) -> Path:
"""A base shaped like the delivered R761: numbered chapter directories under one top level.
Crafted rather than mounted, for MAJOR-3's reason — R761 lives outside this repository and
cannot be a test dependency — and shaped from the MEASURED names (``R761/4``, ``R761/41``,
``R761/42``) so the arm tests the ranking the real corpus provoked.
"""
base = root / "proc"
(base / "R761" / "4").mkdir(parents=True)
(base / "R761" / "41").mkdir()
(base / "R761" / "42").mkdir()
links = "".join(f"- [{d}](R761/{d}/k.md) — chapter {d}.\n" for d in ("4", "41", "42"))
(base / "index.md").write_text(f"---\nbundle_id: proc\n---\n\n{links}", encoding="utf-8")
for d in ("4", "41", "42"):
(base / "R761" / d / "k.md").write_text(
f"---\ntype: Krav\ntitle: Chapter {d}\nprosessnr: '{d}'\n---\n\nbody {d}\n",
encoding="utf-8",
)
return base
def test_c2_a_guessed_chapter_spelling_is_answered_with_the_real_names(tmp_path: Path) -> None:
"""The order's known-positive: ``R761/4-3`` must come back with ``R761/4…`` candidates.
``R761/4`` is FIRST, because the ranking is longest-common-prefix with the segment that failed
— which is the one thing that distinguishes "the names at this level" from "the name you were
reaching for".
"""
base = _r761_shaped(tmp_path)
bundle = okf.navigate_bundle(str(base))
neighbours = okf.nearest_subdirectories(bundle, "R761/4-3")
assert neighbours[0] == "R761/4"
assert set(neighbours) == {"R761/4", "R761/41", "R761/42"}
def test_c2_the_read_file_refusal_carries_them(tmp_path: Path) -> None:
"""The seam: the message the MODEL sees. A helper nothing calls would leave every arm above
green and the measured defect untouched."""
base = _r761_shaped(tmp_path)
tools, _opened, _declared = _wired(base)
answer = tools["read_file"].func(bundle_id="proc", path="R761/4-3/k.md")
assert answer.startswith("REFUSED (BundlePathNotFound)")
assert "'R761'" in answer
assert "subdirectories include R761/4, R761/41, R761/42" in answer
def test_c2_the_read_dir_refusal_carries_them_too(tmp_path: Path) -> None:
"""The OTHER rung, and the reason both were changed: a caller that guesses a directory gets the
same help as one that guesses a document. One question about one base, one answer."""
base = _r761_shaped(tmp_path)
tools, _opened, _declared = _wired(base)
answer = tools["read_dir"].func(bundle_id="proc", path="R761/4-3")
assert answer["refusal"] == "BundlePathNotFound"
assert "R761/4, R761/41, R761/42" in answer["refused"]
def test_c2_an_ancestor_with_no_subdirectories_names_none(tmp_path: Path) -> None:
"""Omission, never an empty clause. MEASURED: 2 of the 18 guessed paths land here — their
ancestor holds documents and no directories — and a trailing "its subdirectories include "
with nothing after it is a sentence that says nothing."""
base = _r761_shaped(tmp_path)
tools, _opened, _declared = _wired(base)
answer = tools["read_file"].func(bundle_id="proc", path="R761/4/missing.md")
assert answer.startswith("REFUSED (BundlePathNotFound)")
assert "subdirectories include" not in answer
def test_c2_every_name_it_hands_back_resolves(tmp_path: Path) -> None:
"""The property that makes the help worth having (``_index_excerpt``'s rule): a path that never
was is worse than no path. Each suggestion is fed straight back to ``read_dir``."""
base = _r761_shaped(tmp_path)
tools, _opened, _declared = _wired(base)
bundle = okf.navigate_bundle(str(base))
for name in okf.nearest_subdirectories(bundle, "R761/4-3"):
listing = tools["read_dir"].func(bundle_id="proc", path=name)
assert "refused" not in listing, f"{name} does not resolve: {listing}"
def test_c2_the_verdict_layer_is_never_advertised_by_name(tmp_path: Path) -> None:
"""Built from ``context_files``, never ``files``: the one layer no listing mentions must not
appear in a REFUSAL either, which would be the same leak wearing an apology."""
base = tmp_path / "withverdict"
(base / "judged").mkdir(parents=True)
(base / "open").mkdir()
(base / "index.md").write_text(
"---\nbundle_id: wv\n---\n\n- [A](open/a.md) — one.\n- [V](judged/v.md) — two.\n",
encoding="utf-8",
)
(base / "open" / "a.md").write_text(
"---\ntype: Krav\ntitle: A\n---\n\nbody\n", encoding="utf-8"
)
(base / "judged" / "v.md").write_text(
"---\ntype: verdict\ntitle: V\n---\n\nprior judgement\n", encoding="utf-8"
)
bundle = okf.navigate_bundle(str(base))
neighbours = okf.nearest_subdirectories(bundle, "nope")
assert neighbours == ("open",), neighbours
@pytest.mark.parametrize("path", ["", "a"])
def test_c2_a_top_level_guess_names_the_top_level_directories(tmp_path: Path, path: str) -> None:
"""The fallback: no ancestor at all means the base's own top level, which ``directory_listing``
always answers. ``krav`` and ``r761`` were both measured as top-level guesses on R761."""
base = _r761_shaped(tmp_path)
bundle = okf.navigate_bundle(str(base))
assert okf.nearest_subdirectories(bundle, path) == ("R761",)

24
tests/test_right_requirement_loadbearing.py
View file

@ -38,6 +38,7 @@ What each arm pins:
from __future__ import annotations
from collections.abc import Sequence
from pathlib import Path
from typing import Any
@ -60,8 +61,22 @@ def _wired(bundle_dir: Path) -> tuple[dict[str, Any], list[ToolCall], list[Decla
return {t.name: t for t in tools}, opened, declared
def _declare(tools: dict[str, Any], opened: list[ToolCall], base: str, path: str) -> dict[str, Any]:
"""Read it the way a run does, then declare it — the (b) path of P19 DEL A."""
def _declare(
tools: dict[str, Any],
opened: list[ToolCall],
base: str,
path: str,
*,
also: Sequence[str] = (),
) -> dict[str, Any]:
"""Read it the way a run does, then declare it — the (b) path of P19 DEL A.
``also`` carries the OTHER documents the run opened. P21/C1 made "it opened enough of the base
to have looked" part of the declaration's precondition, capped by the base's own size, so a
crafted two-document base needs no extras while a five-document example does.
"""
for other in also:
opened.append(ToolCall(name="read_file", bundle_id=base, path=other))
opened.append(ToolCall(name="read_file", bundle_id=base, path=path))
return tools["declare_requirement"].func(bundle_id=base, path=path, ref="Krav 4.1.2-1")
@ -130,8 +145,9 @@ def test_a_verdict_document_is_never_named_back_by_title(tmp_path: Path) -> None
def test_the_tunnel_example_answers_its_own_title_and_no_number() -> None:
"""(a) control on a REAL example base: title present, reference number honestly absent."""
tools, opened, _declared = _wired(_TUNNEL)
path = okf.navigate_bundle(str(_TUNNEL)).context_files[0].name
answer = _declare(tools, opened, "tunnel-hauglia", path)
files = [f.name for f in okf.navigate_bundle(str(_TUNNEL)).context_files]
path = files[0]
answer = _declare(tools, opened, "tunnel-hauglia", path, also=files[1:3])
assert answer["title"].startswith("Kilder: tunnelbelysning")
assert answer["req_number"] == "", "this base declares none, and the reply must not invent one"