open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 4 Activity

refactor(examples): replace sector-specific example material with generic, fictitious examples

Browse source 
The context sets, the packaged knowledge bases and the example bundles are
replaced by one fictitious example set about IT operations in an invented
organisation: three context sets (serverrom-2027, driftsavtale-2027 and the
two-base drift-og-avtale-2027), two synthetic knowledge bases under
src/portfolio_optimiser/data/kunnskapsbaser and two example bundles under
src/portfolio_optimiser/data/bundles. Numbers, codes and structural values in
tests and fixtures are kept; names, ids and wording change. Dated measurement
documents that only recorded runs on the replaced material are deleted.

Gate figures measured on the new set are not comparable with earlier ones.
The exclusion gate from the previous commit is green: 0 tracked files hit
outside the shared/ subtree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-23 15:04:21 +02:00
commit 37547fe292
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
1147 changed files with 24138 additions and 9503 deletions
Download patch file Download diff file Expand all files Collapse all files

99
tests/test_declaration_and_neighbours_loadbearing.py
View file

@ -9,10 +9,10 @@ ONE document.
The order offered two rules and asked which discriminates. Replayed against the real listings:
* **"the declared document must have been returned by a ``read_dir`` filtered on a word from the
approach's label"** refuses **13 of 13** — including Søråsen's ``12.11``, which the order names
as the closest any run came. Measured, ZERO of the 13 declarations were reached through a
filtered listing at all. A gate that refuses every measured case, right and wrong alike, cannot
discriminate; it is the vacuous gate's mirror image.
approach's label"** refuses **13 of 13** — including the process-catalogue run's ``12.11``,
which the order names as the closest any run came. Measured, ZERO of the 13 declarations were
reached through a filtered listing at all. A gate that refuses every measured case, right and
wrong alike, cannot discriminate; it is the vacuous gate's mirror image.
* **"fewer than k distinct documents opened"** at k=3 refuses **8 of 13** and keeps the five that
navigated, ``12.11`` among them. k=3, 4 and 5 refuse the SAME eight — the distribution has a gap
between 2 and 5 — so the threshold is not on a cliff, and 3 is the lowest of that plateau.
@ -21,9 +21,10 @@ The second is built. It is capped by the base's own document count, so a two-doc
declarable rather than becoming a level nobody can declare on.
**C2.** Over the same six traces, 18 of 143 path-bearing tool calls named a path the base does not
hold, and ELEVEN of them are one run walking ``R761/4-3``, ``4.3``, ``4-2``, ``4-1``, ``4-0``,
hold, and ELEVEN of them are one run walking ``P900/4-3``, ``4.3``, ``4-2``, ``4-1``, ``4-0``,
``4-5``, ``4-6`` — guessing a chapter-number spelling the corpus does not use, while the real names
are ``R761/4``, ``R761/41``, ``R761/42``. The refusal already named the nearest LISTABLE ancestor,
are ``P900/4``, ``P900/41``, ``P900/42`` (the process catalogue the traces were taken on, shown here
under the example set's ``P900`` name). The refusal already named the nearest LISTABLE ancestor,
which is the right rung; what it could not say is which of that rung's names was meant. Measured
after: **16 of 18** refusals now name at least one neighbour, and the two that do not are paths
whose ancestor holds documents and no subdirectories — an empty list is omitted rather than
@ -40,18 +41,18 @@ import pytest
from portfolio_optimiser import okf
from portfolio_optimiser.explore import ToolCall, navigator_tools
_EXAMPLES = Path(__file__).resolve().parents[1] / "shared" / "examples"
_TUNNEL = _EXAMPLES / "tunnel-hauglia"
_BUNDLES = Path(__file__).resolve().parents[1] / "src" / "portfolio_optimiser" / "data" / "bundles"
_KJOLING = _BUNDLES / "driftssenter-kjoling"
def _wired(bundle_dir: Path = _TUNNEL) -> tuple[dict[str, Any], list[ToolCall], list[Any]]:
def _wired(bundle_dir: Path = _KJOLING) -> tuple[dict[str, Any], list[ToolCall], list[Any]]:
opened: list[ToolCall] = []
declared: list[Any] = []
tools = navigator_tools((str(bundle_dir),), opened=opened, requirements=declared)
return {t.name: t for t in tools}, opened, declared
def _concepts(bundle_dir: Path = _TUNNEL) -> list[str]:
def _concepts(bundle_dir: Path = _KJOLING) -> list[str]:
return [f.name for f in okf.navigate_bundle(str(bundle_dir)).context_files]
@ -67,10 +68,10 @@ def test_c1_a_declaration_after_one_document_is_refused_with_the_denominator() -
"""
tools, opened, declared = _wired()
path = _concepts()[0]
opened.append(ToolCall(name="read_file", bundle_id="tunnel-hauglia", path=path))
opened.append(ToolCall(name="read_file", bundle_id="driftssenter-kjoling", path=path))
answer = tools["declare_requirement"].func(
approach_id="a1", bundle_id="tunnel-hauglia", path=path, ref="Krav 1.1—1"
approach_id="a1", bundle_id="driftssenter-kjoling", path=path, ref="Krav 1.1—1"
)
assert answer["refusal"] == "RequirementNotRead"
@ -90,10 +91,10 @@ def test_c1_the_same_declaration_after_three_documents_is_accepted() -> None:
tools, opened, declared = _wired()
files = _concepts()
for name in files[:3]:
opened.append(ToolCall(name="read_file", bundle_id="tunnel-hauglia", path=name))
opened.append(ToolCall(name="read_file", bundle_id="driftssenter-kjoling", path=name))
answer = tools["declare_requirement"].func(
approach_id="a1", bundle_id="tunnel-hauglia", path=files[0], ref="Krav 1.1—1"
approach_id="a1", bundle_id="driftssenter-kjoling", path=files[0], ref="Krav 1.1—1"
)
assert answer["declared"] is True
@ -106,10 +107,10 @@ def test_c1_the_same_path_read_three_times_is_still_one_document() -> None:
tools, opened, declared = _wired()
path = _concepts()[0]
for _ in range(3):
opened.append(ToolCall(name="read_file", bundle_id="tunnel-hauglia", path=path))
opened.append(ToolCall(name="read_file", bundle_id="driftssenter-kjoling", path=path))
answer = tools["declare_requirement"].func(
approach_id="a1", bundle_id="tunnel-hauglia", path=path, ref="Krav 1.1—1"
approach_id="a1", bundle_id="driftssenter-kjoling", path=path, ref="Krav 1.1—1"
)
assert answer["refusal"] == "RequirementNotRead"
@ -150,10 +151,10 @@ def test_c1_the_never_opened_refusal_still_fires_first() -> None:
tools, opened, declared = _wired()
files = _concepts()
for name in files[:3]:
opened.append(ToolCall(name="read_file", bundle_id="tunnel-hauglia", path=name))
opened.append(ToolCall(name="read_file", bundle_id="driftssenter-kjoling", path=name))
answer = tools["declare_requirement"].func(
approach_id="a1", bundle_id="tunnel-hauglia", path=files[4], ref="Krav 1.1—1"
approach_id="a1", bundle_id="driftssenter-kjoling", path=files[4], ref="Krav 1.1—1"
)
assert answer["refusal"] == "RequirementNotRead"
@ -164,21 +165,22 @@ def test_c1_the_never_opened_refusal_still_fires_first() -> None:
# --- C2 -------------------------------------------------------------------------------------
def _r761_shaped(root: Path) -> Path:
"""A base shaped like the delivered R761: numbered chapter directories under one top level.
def _p900_shaped(root: Path) -> Path:
"""A base shaped like a process catalogue: numbered chapter directories under one top level.
Crafted rather than mounted, for MAJOR-3's reason — R761 lives outside this repository and
cannot be a test dependency — and shaped from the MEASURED names (``R761/4``, ``R761/41``,
``R761/42``) so the arm tests the ranking the real corpus provoked.
Crafted rather than mounted, for MAJOR-3's reason — the corpus the traces were taken on lived
outside this repository and could not be a test dependency — and shaped from the MEASURED
names (``P900/4``, ``P900/41``, ``P900/42``) so the arm tests the ranking the real corpus
provoked.
"""
base = root / "proc"
(base / "R761" / "4").mkdir(parents=True)
(base / "R761" / "41").mkdir()
(base / "R761" / "42").mkdir()
links = "".join(f"- [{d}](R761/{d}/k.md) — chapter {d}.\n" for d in ("4", "41", "42"))
(base / "P900" / "4").mkdir(parents=True)
(base / "P900" / "41").mkdir()
(base / "P900" / "42").mkdir()
links = "".join(f"- [{d}](P900/{d}/k.md) — chapter {d}.\n" for d in ("4", "41", "42"))
(base / "index.md").write_text(f"---\nbundle_id: proc\n---\n\n{links}", encoding="utf-8")
for d in ("4", "41", "42"):
(base / "R761" / d / "k.md").write_text(
(base / "P900" / d / "k.md").write_text(
f"---\ntype: Krav\ntitle: Chapter {d}\nprosessnr: '{d}'\n---\n\nbody {d}\n",
encoding="utf-8",
)
@ -186,54 +188,54 @@ def _r761_shaped(root: Path) -> Path:
def test_c2_a_guessed_chapter_spelling_is_answered_with_the_real_names(tmp_path: Path) -> None:
"""The order's known-positive: ``R761/4-3`` must come back with ``R761/4…`` candidates.
"""The order's known-positive: ``P900/4-3`` must come back with ``P900/4…`` candidates.
``R761/4`` is FIRST, because the ranking is longest-common-prefix with the segment that failed
``P900/4`` is FIRST, because the ranking is longest-common-prefix with the segment that failed
— which is the one thing that distinguishes "the names at this level" from "the name you were
reaching for".
"""
base = _r761_shaped(tmp_path)
base = _p900_shaped(tmp_path)
bundle = okf.navigate_bundle(str(base))
neighbours = okf.nearest_subdirectories(bundle, "R761/4-3")
neighbours = okf.nearest_subdirectories(bundle, "P900/4-3")
assert neighbours[0] == "R761/4"
assert set(neighbours) == {"R761/4", "R761/41", "R761/42"}
assert neighbours[0] == "P900/4"
assert set(neighbours) == {"P900/4", "P900/41", "P900/42"}
def test_c2_the_read_file_refusal_carries_them(tmp_path: Path) -> None:
"""The seam: the message the MODEL sees. A helper nothing calls would leave every arm above
green and the measured defect untouched."""
base = _r761_shaped(tmp_path)
base = _p900_shaped(tmp_path)
tools, _opened, _declared = _wired(base)
answer = tools["read_file"].func(bundle_id="proc", path="R761/4-3/k.md")
answer = tools["read_file"].func(bundle_id="proc", path="P900/4-3/k.md")
assert answer.startswith("REFUSED (BundlePathNotFound)")
assert "'R761'" in answer
assert "subdirectories include R761/4, R761/41, R761/42" in answer
assert "'P900'" in answer
assert "subdirectories include P900/4, P900/41, P900/42" in answer
def test_c2_the_read_dir_refusal_carries_them_too(tmp_path: Path) -> None:
"""The OTHER rung, and the reason both were changed: a caller that guesses a directory gets the
same help as one that guesses a document. One question about one base, one answer."""
base = _r761_shaped(tmp_path)
base = _p900_shaped(tmp_path)
tools, _opened, _declared = _wired(base)
answer = tools["read_dir"].func(bundle_id="proc", path="R761/4-3")
answer = tools["read_dir"].func(bundle_id="proc", path="P900/4-3")
assert answer["refusal"] == "BundlePathNotFound"
assert "R761/4, R761/41, R761/42" in answer["refused"]
assert "P900/4, P900/41, P900/42" in answer["refused"]
def test_c2_an_ancestor_with_no_subdirectories_names_none(tmp_path: Path) -> None:
"""Omission, never an empty clause. MEASURED: 2 of the 18 guessed paths land here — their
ancestor holds documents and no directories — and a trailing "its subdirectories include "
with nothing after it is a sentence that says nothing."""
base = _r761_shaped(tmp_path)
base = _p900_shaped(tmp_path)
tools, _opened, _declared = _wired(base)
answer = tools["read_file"].func(bundle_id="proc", path="R761/4/missing.md")
answer = tools["read_file"].func(bundle_id="proc", path="P900/4/missing.md")
assert answer.startswith("REFUSED (BundlePathNotFound)")
assert "subdirectories include" not in answer
@ -242,11 +244,11 @@ def test_c2_an_ancestor_with_no_subdirectories_names_none(tmp_path: Path) -> Non
def test_c2_every_name_it_hands_back_resolves(tmp_path: Path) -> None:
"""The property that makes the help worth having (``_index_excerpt``'s rule): a path that never
was is worse than no path. Each suggestion is fed straight back to ``read_dir``."""
base = _r761_shaped(tmp_path)
base = _p900_shaped(tmp_path)
tools, _opened, _declared = _wired(base)
bundle = okf.navigate_bundle(str(base))
for name in okf.nearest_subdirectories(bundle, "R761/4-3"):
for name in okf.nearest_subdirectories(bundle, "P900/4-3"):
listing = tools["read_dir"].func(bundle_id="proc", path=name)
assert "refused" not in listing, f"{name} does not resolve: {listing}"
@ -277,8 +279,9 @@ def test_c2_the_verdict_layer_is_never_advertised_by_name(tmp_path: Path) -> Non
@pytest.mark.parametrize("path", ["", "a"])
def test_c2_a_top_level_guess_names_the_top_level_directories(tmp_path: Path, path: str) -> None:
"""The fallback: no ancestor at all means the base's own top level, which ``directory_listing``
always answers. ``krav`` and ``r761`` were both measured as top-level guesses on R761."""
base = _r761_shaped(tmp_path)
always answers. ``krav`` and the catalogue's own lower-cased name were both measured as
top-level guesses on the process catalogue."""
base = _p900_shaped(tmp_path)
bundle = okf.navigate_bundle(str(base))
assert okf.nearest_subdirectories(bundle, path) == ("R761",)
assert okf.nearest_subdirectories(bundle, path) == ("P900",)