open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 4 Activity

refactor(examples): replace sector-specific example material with generic, fictitious examples

Browse source 
The context sets, the packaged knowledge bases and the example bundles are
replaced by one fictitious example set about IT operations in an invented
organisation: three context sets (serverrom-2027, driftsavtale-2027 and the
two-base drift-og-avtale-2027), two synthetic knowledge bases under
src/portfolio_optimiser/data/kunnskapsbaser and two example bundles under
src/portfolio_optimiser/data/bundles. Numbers, codes and structural values in
tests and fixtures are kept; names, ids and wording change. Dated measurement
documents that only recorded runs on the replaced material are deleted.

Gate figures measured on the new set are not comparable with earlier ones.
The exclusion gate from the previous commit is green: 0 tracked files hit
outside the shared/ subtree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-23 15:04:21 +02:00
commit 37547fe292
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
1147 changed files with 24138 additions and 9503 deletions
Download patch file Download diff file Expand all files Collapse all files

132
tests/test_navigation_window_loadbearing.py
View file

@ -1,12 +1,13 @@
"""P18/A — a listing is a WINDOW, and a path the caller invented is refused by name.
P16 (``docs/2026-09-14-p16-stressrunde-1.md``) measured the ladder S7a-3 built against a DELIVERED
corpus for the first time, and found two things the fixture bases could not show:
P16 (the first stress round; the ledger is ``docs/invarianter.md``) measured the ladder S7a-3
built against a DELIVERED corpus for the first time, and found two things the fixture bases could
not show:
* **one level is not bounded by being one level.** ``okf.directory_listing`` on ``krav/N100`` was
69 250 characters over 445 documents, ``krav/N200`` 169 974 over 1 132, and R761's own root
110 912 over 2 728 SUBDIRECTORIES — 27-113x the 1 500-character ceiling S7a-3 set, riding in
every later prompt. Three of seven paid runs died on the token cap.
* **one level is not bounded by being one level.** ``okf.directory_listing`` on one requirements
level was 69 250 characters over 445 documents, on another 169 974 over 1 132, and a process
catalogue's own root 110 912 over 2 728 SUBDIRECTORIES — 27-113x the 1 500-character ceiling
S7a-3 set, riding in every later prompt. Three of seven paid runs died on the token cap.
* **0 of 26 fasit concepts were opened** in 24 ``read_file`` calls, and 10 of those calls named a
path the base does not hold. Each reached the model as MAF's opaque ``"Error: Function failed."``
(``agent_framework/_tools.py:1410-1432``) while counting toward the three consecutive tool errors
@ -24,11 +25,11 @@ Three seams, each with its own arms below:
nearest directory that actually holds documents. Narrowness is arm (h) and lives in
``test_read_file_directory_refusal_loadbearing.py`` alongside the tripwire it replaced.
The bases this measures are the delivered vegnormal corpora (``PORTFOLIO_VEGNORMAL_ROOT``): arms
that need them SKIP with the root named when it is not mounted, exactly as MAJOR-3's ceiling arm
does — a hard failure would break ``uv run pytest`` inside the handover package. The arms that do
NOT need them (the window's own algebra, the filter's negative, the refusal) are UNCONDITIONAL and
run over a synthetic base, so this file can never be silently absent in full.
The bases this measures are the package's two example bases (``data/kunnskapsbaser/``, pinned
in ``frozen_bundles.json``), so the arms that need them run wherever the package does; they SKIP,
with the store named, only when a user's own store (``PORTFOLIO_FROZEN_BUNDLES``) lacks them. The
arms that do NOT need them (the window's own algebra, the filter's negative, the refusal) are
UNCONDITIONAL and run over a synthetic base, so this file can never be silently absent in full.
"""
from __future__ import annotations
@ -41,6 +42,8 @@ import pytest
from portfolio_optimiser import frozen_bundles, okf
from portfolio_optimiser.explore import navigator_tools
from portfolio_optimiser.mandate import load_mandate
from portfolio_optimiser.stress import read_bundle_declarations
#: S7a-3's ceiling, restated here rather than imported: a gate that imported the implementation's
#: own budget would move with it, and raising the budget is exactly the regression it guards.
@ -50,8 +53,8 @@ _CEILING_CHARS = 1_500
def _delivered(name: str) -> Path:
"""The FROZEN copy this repository pins, resolved at call time.
Absence SKIPS (MAJOR-3's ceiling: no corpus is mounted in the handover archive), drift is
allowed to propagate and FAIL — a measurement of the wrong corpus is not a missing one.
Absence SKIPS (a user's own store, named by ``PORTFOLIO_FROZEN_BUNDLES``, may not hold it),
drift is allowed to propagate and FAIL — a measurement of the wrong corpus is not a missing one.
"""
try:
return frozen_bundles.bundle_dir(name)
@ -105,36 +108,35 @@ def _synthetic(tmp_path: Path, *, dirs: int, per_dir: int) -> Path:
@pytest.mark.parametrize(
("name", "level", "before"),
[
("n100-2023", "krav/N100", 69_250),
("n200-2024", "krav/N200", 169_974),
("n500-2024", "krav/N500", 39_853),
("r761-2025", "R761", 110_912),
("driftskrav-2027", "krav/D200", 29_453),
("prosesskatalog-2027", "P900", 12_345),
],
)
def test_a_default_listing_of_a_delivered_level_is_bounded(
name: str, level: str, before: int
) -> None:
"""(a) The order binds this by name on ``krav/N100`` (445 blades); the other three are the
other levels P16 measured, including R761's root, whose 110 912 characters were DIRECTORIES —
which is why the window covers both kinds and not only documents.
"""(a) The two example levels that are far wider than one window: the requirements level
``krav/D200`` (220 documents) and the process catalogue's root ``P900``, whose 300 entries are
DIRECTORIES — which is why the window covers both kinds and not only documents.
``before`` is P16's measured cost of the SAME call, carried so the arm cannot pass by the level
``before`` is what reading the WHOLE level costs through the window (every page at the maximum
limit, summed), measured on the pinned base and carried so the arm cannot pass by the level
having shrunk. The control below proves the fixture is the thing that did not fit."""
base = _delivered(name)
listing = _read_dir(base, level)
assert _chars(listing) <= _CEILING_CHARS or name == "n200-2024", (
assert _chars(listing) <= _CEILING_CHARS, (
f"{name}/{level} default listing is {_chars(listing)} chars, over {_CEILING_CHARS}"
)
# n200 carries the longest titles measured (entries up to 209 chars), so ten of them land 2.5 %
# over. Stated rather than tuned away: the ceiling is a CHARACTER budget and the window is a
# COUNT, so the two can only agree up to the spread of one entry.
assert _chars(listing) <= 1_600
assert _chars(listing) * 20 < before, "the bound must be a fall, not a rounding"
assert int(listing["total"]) > 10 * int(listing["limit"]), (
"the CONTROL is inert: this level must hold far more than one window, or the bound above "
"is measuring a small level rather than the window"
)
whole = 0
for offset in range(0, int(listing["total"]), okf._DIRECTORY_PAGE_MAX):
whole += _chars(_read_dir(base, level, offset=offset, limit=okf._DIRECTORY_PAGE_MAX))
assert whole == before, f"{name}/{level}: the whole level now costs {whole}, pinned {before}"
def test_the_window_pages_the_whole_level_exactly_once(tmp_path: Path) -> None:
@ -184,19 +186,19 @@ def test_an_offset_past_the_end_is_an_empty_window_over_an_honest_total(tmp_path
def test_the_known_positive_filter_finds_the_fasit_concepts_and_not_the_level() -> None:
"""(e) The order's own known positive: ``filter="rundkjoring"`` on n100 must answer with the
two concepts gate-nordvik/a1 must cite, and NOT with 445 rows."""
base = _delivered("n100-2023")
fasit = json.loads(Path("contexts/gate-nordvik-2027/fasit.json").read_text(encoding="utf-8"))
"""(e) The known positive: ``filter="autonomitid"`` on ``krav/D200`` must answer with the
two concepts serverrom/a2 must cite, and NOT with 220 rows."""
base = _delivered("driftskrav-2027")
fasit = json.loads(Path("contexts/serverrom-2027/fasit.json").read_text(encoding="utf-8"))
wanted = {
c["path"]
for m in fasit["must_cite"]
if m["approach_id"] == "a1-rundkjoring-forenklet"
if m["approach_id"] == "a2-ups-autonomi"
for c in m["concepts"]
}
assert wanted, "the fixture must name concepts, or this arm proves nothing"
hits = _read_dir(base, "krav/N100", filter="rundkjøring", limit=50)
hits = _read_dir(base, "krav/D200", filter="autonomitid", limit=50)
assert wanted <= {str(d["name"]) for d in hits["documents"]}
assert int(hits["total_matches"]) < int(hits["total"]) / 50
@ -216,8 +218,9 @@ def test_a_filter_that_matches_nothing_is_an_answer_and_not_a_refusal(tmp_path:
def test_the_filter_reads_the_reference_number_and_not_only_the_title(tmp_path: Path) -> None:
"""(g) The second field is load-bearing: on R761 the thing a navigator knows is the process
number, which is not in the title. Written over a synthetic base so it is unconditional."""
"""(g) The second field is load-bearing: on a process catalogue the thing a navigator knows
is the process number, which is not in the title. Written over a synthetic base so it is
unconditional."""
base = _synthetic(tmp_path, dirs=2, per_dir=3)
by_ref = _read_dir(base, "seksjon-01", filter="krav 1.2")
@ -226,8 +229,9 @@ def test_the_filter_reads_the_reference_number_and_not_only_the_title(tmp_path:
def test_a_filter_narrows_directories_too(tmp_path: Path) -> None:
"""(h) R761's root is 2 728 DIRECTORIES: a filter that only narrowed documents would leave the
biggest measured level unnarrowable."""
"""(h) A process catalogue's root is all DIRECTORIES (P900's: 300; the largest ever measured
here: 2 728): a filter that only narrowed documents would leave the biggest level
unnarrowable."""
base = _synthetic(tmp_path, dirs=12, per_dir=2)
narrowed = _read_dir(base, "", filter="seksjon-0")
@ -242,35 +246,47 @@ def test_an_invented_path_is_refused_by_name_over_a_delivered_base() -> None:
"""(i) The measured live shape: a one-character slip in a UUID. Before, this reached the model
as ``"Error: Function failed."``; the refusal now names the path AND the level that holds
documents, which is the only thing the caller can act on."""
base = _delivered("n100-2023")
slip = "krav/N100/id-d2ebe771-5216-4d7f-92d2-95a31f2b2702.md"
base = _delivered("driftskrav-2027")
real = "krav/D200/id-b1e2ba25-9825-57b0-a358-569be457d2c8.md"
slip = _slip(real)
assert (base / real).is_file() and not (base / slip).exists(), "control: one real, one invented"
answer = _read_file(base, slip)
assert answer.startswith(f"REFUSED ({okf.BundlePathNotFound.__name__})")
assert slip in answer and "krav/N100" in answer and "read_dir" in answer
assert slip in answer and "krav/D200" in answer and "read_dir" in answer
# The named level resolves, and it is the one the caller was already in.
assert int(_read_dir(base, "krav/N100")["total"]) > 0
assert int(_read_dir(base, "krav/D200")["total"]) > 0
def test_all_ten_of_p16s_unresolvable_calls_now_answer_instead_of_failing() -> None:
"""(j) The nevner arm (ansikt 4). P16's four ``-debate.json`` artefacts ARE the population, and
the denominator is re-measured here rather than quoted: 32 ``read_file`` calls (the order says
24 — measured 14.09, that number is the four runs' DISTINCT paths, not their calls), of which
10 named a path the base does not hold.
def _slip(path: str) -> str:
"""The measured live shape of an invented path: ONE character of the document id changed."""
stem, suffix = path[:-3], path[-3:]
return stem[:-1] + ("b" if stem[-1] == "a" else "a") + suffix
Every one is replayed. Each of the 10 must now come back as a NAMED refusal, and the other 22
must still return the document — a gate that refused everything would pass the first half on
its own, which is the failure this file's own A2 negative arm is written against."""
artefacts = sorted(Path("scratchpad/p14-stress").glob("*/*-debate.json"))
if not artefacts:
pytest.skip("P16's stress artefacts are not present in this checkout")
calls: list[tuple[str, str]] = [
(call["bundle_id"].removeprefix("vegnormal-"), call["path"])
for artefact in artefacts
for call in json.loads(artefact.read_text(encoding="utf-8"))["tool_calls"]
if call["name"] == "read_file"
]
def test_every_invented_path_answers_by_name_and_every_real_one_is_served() -> None:
"""(j) The nevner arm (ansikt 4). P16 replayed its own population — 32 ``read_file`` calls from
four paid runs, 10 of them naming a path the base does not hold — but those runs read corpora
this repository no longer carries, so the population here is CONSTRUCTED from the three context
sets' own fasit: every cited path (routed at the base its approach names), and the same path
with one id character changed. The denominator is re-measured from the fasit, not quoted.
Every invented path must come back as a NAMED refusal, and every real one must still return
the document — a gate that refused everything would pass the first half on its own, which is
the failure this file's own A2 negative arm is written against."""
calls: list[tuple[str, str]] = []
for fasit_path in sorted(Path("contexts").glob("*/fasit.json")):
set_dir = fasit_path.parent
names = {
b["bundle_id"]: b["name"] for b in read_bundle_declarations(set_dir / "bundle.txt")
}
routed = {
a.id: names[a.bundle_id] for a in load_mandate(set_dir / "mandate.json").approaches
}
for row in json.loads(fasit_path.read_text(encoding="utf-8"))["must_cite"]:
for concept in row["concepts"]:
calls.append((routed[row["approach_id"]], concept["path"]))
calls.append((routed[row["approach_id"]], _slip(concept["path"])))
refused = served = 0
cache: dict[str, Any] = {}
for name, path in calls:
@ -285,4 +301,4 @@ def test_all_ten_of_p16s_unresolvable_calls_now_answer_instead_of_failing() -> N
else:
served += 1
assert (refused, served) == (10, 22), f"population moved: {refused} refused, {served} served"
assert (refused, served) == (18, 18), f"population moved: {refused} refused, {served} served"