portfolio-optimiser/tests/test_hierarchical_navigation_loadbearing.py

"""S7a-3 pkt. 2 - the ladder gains its missing rung: ``read_bundle`` -> ``read_dir`` -> ``read_file``.

**The measurement** (``docs/2026-09-03-syretest-s7a2-k2.md``). MAJOR-3 (session 77) rebuilt
``read_bundle`` from "the whole navigated base" into "one entry per concept document", and on the
three example bases that was a 77-91 % cut. Then the first real corpus arrived: K2 navigates to
**629 concept documents behind 478 nested indexes**, and a listing of 629 documents costs
**42 761 o200k tokens** and rides in 7 of 12 prompts - **90 % of every prompt token in the run**.
The bound held asymptotically and still priced the whole corpus, because the rung was priced by the
size of the base rather than by what the navigator had asked to see.

Worse, the 478 nested indexes were BUILT by the navigation, CONSUMED by it, and then **flattened
away**: the agent saw 629 sibling documents and was never told the corpus had a shape.

**A premise of the order, corrected by measurement.** The order says ``context_files`` must stop
discarding the hierarchy. It never held it back: every ``BundleFile.name`` is already the full
bundle-relative posix path (``del-ii-bilag-7-prisskjema/prissammenstilling.md``), so the
tree was always recoverable from the names alone. (That concept was named
``prissammenstilling-sheet-1.md`` in the bundle measured here and in every bundle built before okf
``6ff18fd``, which stopped emitting the converter's ``{#...}`` anchor that the id was derived from;
the path SHAPE the point rests on is unchanged either way.) What flattened it was the RENDERING. That matters
because it is why ``okf.bundle_context`` and both nav-goldens are byte-identical after this change -
the order requires exactly that, and it comes for free rather than by care.

So: ``okf.directory_listing`` is ONE renderer, and BOTH tools are it at a different path.
``read_bundle`` is the root level (subdirectories with a document count, plus the documents that sit
at the top), ``read_dir(id, path)`` is one level down, ``read_file(id, path)`` is unchanged. Every
byte is still exactly one call away; what changed is that a navigator pays for the level it asked
for.

**Directories are derived from PATHS, never from ``index.md`` files.** A nested index is navigation,
not content (``context_files`` drops it at every level, and must keep doing so or the verdict layer
comes with it). Deriving the tree from names means a directory reached by a link that skipped its
index is still visible - and it means the two renderings of one bundle cannot disagree about what
the concepts are.

**DEVIATION from the order, stated and measured.** The order words the ceiling as "``read_bundle``
over the K2 bundle < 1 500 chars". K2 lives outside the repository (``~/corpora/``) and cannot be a
test dependency - the MAJOR-3 gate had the same constraint and solved it the same way, by bounding
the largest base that IS shipped. And the number does not hold: K2's root level carries **39
identifiable entries** (28 directories + 11 top-level documents) and measures **3 954 chars ~ 1 366
o200k tokens**. Fitting 1 500 would mean dropping titles or dropping entries, which is the
vacuous-gate class this repo has been bitten by twelve times. The measured drop is from 42 761 to
1 366 tokens (-97 %); the K2 numbers live in the report, and the gate here bounds what it can
actually see, plus the PROPERTY that produced the drop (arm (b)).

Arms: (a) a FLAT base lists exactly what it listed before * (b) cost tracks entries at ONE LEVEL,
not documents in the base - with the flat control that proves the hierarchy did it * (c) ``read_dir``
over the biggest directory is bounded too * (d) anti-vacuity: every document is still reachable, and
the counts add up * (e) the verdict layer stays out of both halves * (f) the ladder is intact *
(g) an unknown path is refused BY NAME, never rendered as an empty directory * (h) the navigator's
instruction and the tool descriptions describe the ladder they now have.
"""

from __future__ import annotations

import json
from pathlib import Path
from typing import Any


from portfolio_optimiser import explore, okf
from portfolio_optimiser.explore import navigator_tools

_EXAMPLES = Path(__file__).resolve().parents[1] / "shared" / "examples"
#: The largest FLAT base shipped - the one the MAJOR-3 gate bounds.
_TUNNEL = _EXAMPLES / "tunnel-hauglia"
#: The only NESTED base shipped, and it is tiny: three levels, six concepts. It proves the shape is
#: read correctly; it cannot prove the shape pays for itself, which is what arm (b) is for.
_NESTED = _EXAMPLES / "nav-golden-hierarchy" / "bundle"

#: Characters ONE listing may cost - test-owned, for the catalogue gate's reason: a test that
#: imported the implementation's budget would move with it, and raising the budget is precisely the
#: regression this file exists to catch. Same number as the MAJOR-3 gate, deliberately: this rung's
#: price must not creep just because it now has a rung below it.
_CEILING_CHARS = 1_500


def _tools(bundle_dir: Path) -> dict[str, Any]:
    return {t.name: t for t in navigator_tools((str(bundle_dir),))}


def _read_bundle(bundle_dir: Path) -> dict[str, Any]:
    return _tools(bundle_dir)["read_bundle"].func(bundle_id=bundle_dir.name)


def _read_dir(bundle_dir: Path, path: str, **window: Any) -> dict[str, Any]:
    return _tools(bundle_dir)["read_dir"].func(bundle_id=bundle_dir.name, path=path, **window)


def _blob(payload: object) -> str:
    return json.dumps(payload, ensure_ascii=False)


def _entry(f: okf.BundleFile, name: str) -> dict[str, Any]:
    """The four-field document entry MAJOR-3 introduced, written out here rather than imported: an
    assert against the implementation's own helper could not tell the two shapes apart."""
    return {
        "name": name,
        "type": f.type or "document",
        "title": okf.unquote_scalar(f.frontmatter.get("title", f.name)),
        "chars": len(f.body),
    }


def _write_tree(root: Path, name: str, *, dirs: int, per_dir: int, top: int = 2) -> Path:
    """A base shaped like a real ingested corpus: a root index linking to per-directory indexes,
    each linking to its own documents. Bodies are small on purpose - this fixture is about COUNT."""
    base = root / name
    base.mkdir(parents=True)
    root_lines = [f"# {name}", ""]
    for t in range(top):
        root_lines.append(f"- [Toppdokument {t}](topp-{t}.md)")
        (base / f"topp-{t}.md").write_text(
            f"---\ntype: concept\ntitle: Toppdokument {t}\n---\n\nkort tekst.\n", encoding="utf-8"
        )
    for d in range(dirs):
        sub = base / f"kategori-{d:02d}"
        sub.mkdir()
        root_lines.append(f"- [kategori-{d:02d} (index)](kategori-{d:02d}/index.md)")
        sub_lines = [f"# kategori-{d:02d}", ""]
        for i in range(per_dir):
            sub_lines.append(f"- [Dokument {i}](dok-{i:02d}.md)")
            (sub / f"dok-{i:02d}.md").write_text(
                f"---\ntype: concept\ntitle: Dokument {d:02d}-{i:02d}\n---\n\nkort tekst.\n",
                encoding="utf-8",
            )
        (sub / "index.md").write_text(
            "---\ntype: index\n---\n\n" + "\n".join(sub_lines) + "\n", encoding="utf-8"
        )
    (base / "index.md").write_text(
        "---\ntype: index\n---\n\n" + "\n".join(root_lines) + "\n", encoding="utf-8"
    )
    return base


# --- (a) a flat base is untouched -----------------------------------------------------------------


def test_a_flat_base_lists_exactly_what_it_listed_before() -> None:
    """(a) The order names this one explicitly. Every base shipped in this repo except one is flat,
    and the whole MAJOR-3 measurement was taken over them: if the new rung changed what a flat base
    answers, this change would be a rewrite of that result rather than a level above it."""
    bundle = okf.navigate_bundle(str(_TUNNEL))
    listing = _read_bundle(_TUNNEL)

    assert listing["documents"] == [_entry(f, f.name) for f in bundle.context_files]
    assert listing["directories"] == [], "a flat base has no subdirectories to report"
    assert len(_blob(listing)) <= _CEILING_CHARS


# --- (b)/(c) the price is the LEVEL, not the base -------------------------------------------------


def test_the_root_listing_costs_the_level_not_the_base(tmp_path: Path) -> None:
    """(b) The headline, and the control that proves the hierarchy is what bounded it.

    240 documents behind 20 directories: the root level is 22 entries and fits, while the FLAT
    listing of the same base - what ``read_bundle`` returned until today - is many times the
    ceiling. Without the flat control, a green bound here could just as well mean the fixture was
    small (Verifiseringsloven face 4: a gate that can only pass proves nothing).
    """
    base = _write_tree(tmp_path, "stort-korpus", dirs=20, per_dir=12)
    bundle = okf.navigate_bundle(str(base))

    listing = _read_bundle(base)
    flat = [_entry(f, f.name) for f in bundle.context_files]

    assert len(bundle.context_files) == 242
    assert len(_blob(listing)) <= _CEILING_CHARS, (
        f"the root level costs {len(_blob(listing))} chars over the ceiling {_CEILING_CHARS}"
    )
    assert len(_blob(flat)) > 5 * _CEILING_CHARS, (
        "the CONTROL is inert: the flat listing must be the thing that does not fit, or the bound "
        "above is measuring a small fixture rather than the hierarchy"
    )


def test_read_dir_over_the_biggest_directory_is_bounded_too(tmp_path: Path) -> None:
    """(c) A rung that is only cheap at the top is not a ladder. The order asks for this one by
    name: ``read_dir`` over the largest directory must be bounded as well."""
    base = _write_tree(tmp_path, "stort-korpus", dirs=20, per_dir=12)

    listing = _read_dir(base, "kategori-00")

    # P18/A1: the answer is now a WINDOW over the level, so what the level HOLDS is read off
    # ``total`` and what this call was given is the page. The bound below is the same bound.
    assert listing["total"] == 12 and listing["directories"] == []
    assert len(listing["documents"]) == listing["limit"] <= listing["total"]
    assert len(_blob(listing)) <= _CEILING_CHARS


# --- (d)/(e) bounded is not the same as empty -----------------------------------------------------


def test_every_document_is_still_reachable_and_the_counts_add_up(tmp_path: Path) -> None:
    """(d) The anti-vacuity arm. Returning ``{"directories": [], "documents": []}`` passes every
    bound above perfectly and hands the navigator nothing. Two things are asserted: each directory
    entry PRICES its subtree (so a navigator can choose against a budget), and the whole base is
    accounted for - every concept document is in exactly one level."""
    base = _write_tree(tmp_path, "stort-korpus", dirs=20, per_dir=12)
    bundle = okf.navigate_bundle(str(base))

    root = _read_bundle(base)
    # P18/A1 made this arm STRONGER rather than weaker: the accounting now has to PAGE, so the same
    # assertion also proves the window is complete and non-overlapping. A pagination that dropped or
    # repeated an entry fails here, and an implementation that answered every page with the first
    # one fails on the directory set below.
    directories: dict[str, int] = {}
    documents: list[dict[str, object]] = []
    offset = 0
    while offset < int(root["total"]):
        page = _read_dir(base, "", offset=offset)
        assert page["total"] == root["total"], "the denominator must not move between pages"
        directories.update({str(d["path"]): int(d["documents"]) for d in page["directories"]})
        documents.extend(page["documents"])
        offset += int(page["limit"])

    counted = len(documents) + sum(directories.values())
    assert counted == len(bundle.context_files) == 242
    assert len(directories) + len(documents) == int(root["total"])
    assert set(directories) == {f"kategori-{d:02d}" for d in range(20)}
    assert all(count == 12 for count in directories.values())
    assert len({str(d["name"]) for d in documents}) == len(documents), "a page repeated an entry"
    for entry in documents:
        assert set(entry) == {"name", "type", "title", "chars"}
        assert str(entry["title"]).strip() and int(entry["chars"]) > 0


def test_the_verdict_layer_is_excluded_from_both_halves(tmp_path: Path) -> None:
    """(e) MAJOR-3's rule, extended to the new half: a listing built from ``bundle.files`` would
    route prior verdicts in front of the navigator AROUND the gated ExpeL fold (maalbilde §4) - and
    a DIRECTORY COUNT built from ``files`` would tell the navigator a subtree holds documents it is
    not allowed to be shown."""
    base = _write_tree(tmp_path, "med-dommer", dirs=2, per_dir=2, top=0)
    sub = base / "kategori-00"
    (sub / "dom-0.md").write_text(
        "---\ntype: verdict\ntitle: Dom\n---\n\nEksperten godkjente.\n", encoding="utf-8"
    )
    index = sub / "index.md"
    index.write_text(index.read_text(encoding="utf-8") + "- [dom](dom-0.md)\n", encoding="utf-8")

    root = _read_bundle(base)
    inner = _read_dir(base, "kategori-00")

    assert len(okf.navigate_bundle(str(base)).verdicts) == 1, "the fixture must HAVE one to leak"
    assert [d["documents"] for d in root["directories"] if d["path"] == "kategori-00"] == [2]
    assert not [e for e in inner["documents"] if e["type"] == "verdict"]
    assert len(inner["documents"]) == 2


# --- (f)/(g) the ladder, and its fail-closed edge -------------------------------------------------


def test_the_whole_document_is_still_one_call_away() -> None:
    """(f) The bound is a disclosure LEVEL, not data loss - MAJOR-3's arm (e), re-asserted over the
    new shape because a listing that no longer names documents the way ``read_file`` takes them
    would have broken the ladder while every cost arm stayed green."""
    listing = _read_bundle(_TUNNEL)
    biggest = max(listing["documents"], key=lambda e: int(e["chars"]))

    whole = _tools(_TUNNEL)["read_file"].func(bundle_id=_TUNNEL.name, path=str(biggest["name"]))

    assert len(whole) > _CEILING_CHARS
    assert int(biggest["chars"]) <= len(whole)


def test_a_nested_document_is_reachable_through_the_rung_below() -> None:
    """(f, nested) The three rungs composed on the only nested base shipped: the root names a
    directory, ``read_dir`` names a document in it, ``read_file`` returns that document whole."""
    root = _read_bundle(_NESTED)
    first_dir = str(root["directories"][0]["path"])

    inner = _read_dir(_NESTED, first_dir)
    name = str(inner["documents"][0]["name"])
    whole = _tools(_NESTED)["read_file"].func(bundle_id=_NESTED.name, path=name)

    assert root["documents"] and root["directories"], "the fixture must have BOTH halves"
    assert name.startswith(f"{first_dir}/"), (
        "every path a listing hands out must be usable AS IT IS: a level-relative name would have "
        "to be composed by a model, and a path that never existed is worse than no path"
    )
    assert whole.strip(), "the path the listing gave must resolve to the document itself"


def test_an_unknown_directory_is_refused_by_name(tmp_path: Path) -> None:
    """(g) Fail-closed, and it is the vacuity trap in its own right: an unknown path rendered as an
    empty listing is indistinguishable from a directory that exists and holds nothing. Validation,
    never invention (``write_concept_file``'s rule).

    Pinned to ``BundlePathNotFound`` by NAME since F3 added its sibling: both refusals quote the
    caller's path, so a match on the path alone could no longer tell "this is nothing" from "this
    is a document" - the substring two branches share.

    REWRITTEN, not weakened (F99-D3): the tool RETURNS the refusal instead of raising it, so the
    name it is pinned to is the ``refusal`` kind. Both halves stand - the path is quoted, and the
    kind still separates it from ``DocumentPathRefused``."""
    base = _write_tree(tmp_path, "korpus", dirs=2, per_dir=2)

    refusal = _read_dir(base, "kategori-99")

    assert refusal["refusal"] == okf.BundlePathNotFound.__name__
    assert refusal["refusal"] != okf.DocumentPathRefused.__name__
    assert "kategori-99" in refusal["refused"]


# --- (h) the description must not lie about the body ----------------------------------------------


def test_the_navigator_is_told_about_the_rung_it_now_has() -> None:
    """(h) The Fase-3 class applied to a tool: a description that lies about the body IS the
    model's instruction. MAJOR-3 had to move this same prose when ``read_bundle`` stopped returning
    the navigated context; the ladder is three rungs now, and all three must be named."""
    instruction = explore._INSTRUCTIONS[explore.NAVIGATOR_ROLE]
    tools = {t.name: t for t in navigator_tools((str(_NESTED),))}

    assert set(tools) == {"list_bundles", "read_bundle", "read_dir", "read_file"}
    for rung in ("list_bundles", "read_bundle", "read_dir", "read_file"):
        assert rung in instruction, f"the navigator is never told that {rung} exists"
    assert "read_dir" in tools["read_bundle"].description, (
        "read_bundle now answers with directories; a description that does not name the tool that "
        "opens one leaves the navigator holding a path it cannot use"
    )