open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 3 Activity

feat(explore): read_bundle costs O(documents), never O(bytes of the base)

Browse source 
[skip-docs]

S2c / MAJOR-3, order 20260902T151931Z-250257273. The measurement landed first
in ce7f687; this commit is the one seam it authorised, plus the after-table.

read_bundle returned okf.bundle_context -- the WHOLE navigated base. Because
the exploration's participants share one conversation history, that single
function_result rode in FIVE later prompts at full price without anyone asking
for it again: 54-59 percent of every prompt-token in a CLI --explore run.

It now returns the catalogue form one rung down the ladder -- one entry per
concept document (name, type, title, chars) -- with read_file as the next rung.
Tunnel base: 12 595 -> 259 o200k tokens, exploration prompt-tokens -91 percent.

The listing is built from Bundle.context_files and never from files: that is
the property which drops the type: verdict layer at every level, and a listing
built from files would route prior verdicts in front of the navigator around
the gated ExpeL fold while every cost arm stayed green.

A premise was felled before anything was built on it: the tunnel base's root
index body is 4 763 chars alone, nearly the whole ceiling, for a field the
catalogue already excerpts and read_file still returns whole. So read_bundle
carries the listing and not the index.

The tool description and the navigator's instruction both claimed "read its
navigated context" and were updated in the same move -- a description that lies
about the body IS the model's instruction. Two pre-existing asserts would have
gone vacuously true against a list and were strengthened rather than left.

Ceiling lives in the test, not in explore.py. Deviation stated there and in the
docs: it bounds CHARACTERS, not tokens, because tiktoken is not a project
dependency and a gate that skips when an optional package is missing is a gate
that can be silently absent; the conversion was measured (2.89 chars/token) and
the order's own token criterion verified once by the instrument.

Load-bearing measured: seven mutations, all red against the WHOLE suite; green
control 1195 passed / 5 skipped (from 1189/5, strict superset); golden
demo-transcript.stdout byte-unchanged; and the debate's three bundle_context
copies are byte-identical before and after, which proves run.py and the
nav-goldens were not touched rather than asserting it.
This commit is contained in:
Kjell Tore Guttormsen 2026-09-03 00:36:01 +02:00
commit b799cc527f
6 changed files with 356 additions and 10 deletions
Download patch file Download diff file Expand all files Collapse all files

205
tests/test_read_bundle_cost_loadbearing.py Normal file
View file

@ -0,0 +1,205 @@
"""``read_bundle`` costs O(DOCUMENTS in one base), never O(bytes of that base).
MAJOR-3 (``docs/2026-09-02-misjonsreview-v2.md`` § 7), measured before anything was changed in
``docs/2026-09-02-read-bundle-kontekstkostnad.md``:
read_bundle payload bygg 3 861 / veglys 10 406 / tunnel 12 595 o200k_base tokens
prompts one result rides in FIVE (navigator 1, manager 3, hypothesiser 1)
share of every prompt-token 54 % / 59 % / 59 % of one CLI ``--explore`` run
``read_bundle`` returned ``okf.bundle_context`` — the WHOLE navigated base. One call turned the
base into a ``function_result``, and because the exploration's participants share one conversation
history that result rides in **every later prompt**, at full price, without anyone asking for it
again. That is the same shape ``list_bundles`` was measured in and rebuilt out of in session 65
(``tests/test_catalogue_cost_loadbearing.py``), one rung down the ladder: the catalogue answers
*which bases exist*, ``read_bundle`` answers *what is in this one*, and ``read_file`` answers *what
does this document say*. Only the last of the three should cost what a document costs.
So ``read_bundle`` now returns the CATALOGUE FORM: one entry per concept document — ``name``,
``type``, ``title``, ``chars`` — and ``read_file(id, name)`` is the next rung. This is a disclosure
level, not data loss: every byte is still exactly one call away, and a navigator now pays for the
documents it chose to open instead of for the ones it did not.
**A premise felled before it was built on** (see the measurement doc § 2): "the index body is the
base's own navigation prose, so it belongs here". The tunnel base's root index is **4 763 chars
alone ≈ 1 400 tokens** — nearly the entire ceiling, for a field ``list_bundles`` already excerpts
and ``read_file(id, "index.md")`` still returns whole.
**The ceiling lives in THIS FILE, not in ``explore.py``** — the catalogue gate's rule, for the
catalogue gate's reason: a test that imported the implementation's budget would move with it, and
raising the budget is precisely the regression this file exists to catch.
**DEVIATION from the order, stated:** the order words the gate as "< 1 500 o200k-tokens". This
file bounds CHARACTERS instead. ``tiktoken`` is not a project dependency (and adding one for a
gate would be a bigger decision than the gate), and a gate that skips when an optional package is
missing is a gate that can be silently absent. The character ceiling is a proxy whose conversion
was MEASURED rather than assumed: the new payload over the real tunnel base is **748 chars / 259
o200k tokens** (2.89 chars/token for this Norwegian markdown), so 1 500 characters is ≈ 520 tokens
— comfortably inside the order's criterion, and twice the measured payload, so ordinary field
growth does not force a rewrite. The order's own criterion is verified directly, once, by the
instrument in the measurement doc.
What the arms pin, and what each one refuses:
(a) the bound itself, over the REAL base the order names — refuses the unbounded form. A synthetic
fixture here would measure the fixture writer, not the base;
(b) cost tracks DOCUMENT COUNT, not document SIZE — ten times the prose, the same price. That is
the property stated directly rather than inferred from (a);
(c) the anti-vacuity arm: every concept document is still IDENTIFIED, with all four fields, and the
entry count equals the navigated context files. Without this, "return an empty list" passes (a)
perfectly and hands the navigator nothing to choose between — the repo's vacuous-gate class,
which has now bitten twelve times;
(d) the ``type: verdict`` layer stays EXCLUDED. ``read_bundle`` built from ``bundle.files`` instead
of ``bundle.context_files`` would route prior verdicts into a hypothesis prompt around the
gated ExpeL fold (målbilde §4), and nothing else in the suite would notice;
(e) the ladder is intact — ``read_file`` still returns the COMPLETE document, so the bound is a
disclosure level, not data loss;
(f) the CONTROL the order names: ONE concept document alone blows the ceiling for the whole
listing. A green (a) then means the bound fired, not that the base was small
(Verifiseringsloven face 4: a gate that can only pass proves nothing).
"""
from __future__ import annotations
import json
from pathlib import Path
from typing import Any
from portfolio_optimiser import okf
from portfolio_optimiser.explore import navigator_tools
_EXAMPLES = Path(__file__).resolve().parents[1] / "shared" / "examples"
#: The base the order names. Measured, not chosen: the largest of the three example bases.
_TUNNEL = _EXAMPLES / "tunnel-hauglia"
#: Characters one ``read_bundle`` listing may cost. Test-owned on purpose — see the docstring, and
#: the measured chars/token conversion that ties it to the order's token criterion.
_CEILING_CHARS = 1_500
def _tools(bundle_dir: Path) -> dict[str, Any]:
return {t.name: t for t in navigator_tools((str(bundle_dir),))}
def _listing(bundle_dir: Path) -> list[dict[str, Any]]:
return _tools(bundle_dir)["read_bundle"].func(bundle_id=bundle_dir.name)
def _blob(payload: object) -> str:
return json.dumps(payload, ensure_ascii=False)
def _write_base(
root: Path, name: str, *, body_chars: int, concepts: int = 3, verdicts: int = 0
) -> Path:
base = root / name
base.mkdir(parents=True)
lines = [f"# {name}", ""]
for i in range(concepts):
lines.append(f"- [konsept-{i}](konsept-{i}.md)")
(base / f"konsept-{i}.md").write_text(
f"---\ntype: concept\ntitle: Konsept {i}\n---\n\n" + ("innhold. " * (body_chars // 9)),
encoding="utf-8",
)
for i in range(verdicts):
lines.append(f"- [dom-{i}](dom-{i}.md)")
(base / f"dom-{i}.md").write_text(
f"---\ntype: verdict\ntitle: Dom {i}\n---\n\nEksperten godkjente tiltaket.\n",
encoding="utf-8",
)
(base / "index.md").write_text(
"---\ntype: index\n---\n\n" + "\n".join(lines) + "\n", encoding="utf-8"
)
return base
def test_read_bundle_over_the_real_tunnel_base_is_bounded() -> None:
"""(a) The headline, over the base the order names — not a fixture of my own making."""
blob = _blob(_listing(_TUNNEL))
assert len(blob) <= _CEILING_CHARS, (
f"read_bundle over {_TUNNEL.name} costs {len(blob)} chars, over the ceiling "
f"{_CEILING_CHARS}; it used to be 39 583 (12 595 o200k tokens), riding in five prompts"
)
def test_read_bundle_cost_does_not_track_document_size(tmp_path: Path) -> None:
"""(b) Ten times the prose, the same price. O(documents), not O(bytes)."""
small = _write_base(tmp_path, "small", body_chars=500)
large = _write_base(tmp_path, "large", body_chars=5_000)
small_entries, large_entries = _listing(small), _listing(large)
# Same document COUNT, same number of entries — and the only field that grew is the honest,
# logarithmic ``chars`` digit, so the payloads differ by a handful of characters at most.
assert len(small_entries) == len(large_entries) == 3
assert abs(len(_blob(large_entries)) - len(_blob(small_entries))) < 20, (
"ten times the body must not cost ten times the listing"
)
assert len(_blob(large_entries)) <= _CEILING_CHARS
def test_the_listing_still_identifies_every_document(tmp_path: Path) -> None:
"""(c) The anti-vacuity arm: bounded is not the same as empty.
Bounded-and-useless passes (a) perfectly. What a navigator needs in order to choose a document
is what it IS (``type``), what it is CALLED (``title``), what to ask for (``name``) and what it
will cost (``chars``) — so all four are asserted, and the entry count is tied to the navigated
context files rather than to a number written here.
"""
bundle = okf.navigate_bundle(str(_TUNNEL))
entries = _listing(_TUNNEL)
assert len(entries) == len(bundle.context_files) > 0, (
"a listing that omits documents is a base the navigator cannot fully see"
)
by_name = {str(e["name"]): e for e in entries}
for f in bundle.context_files:
entry = by_name[f.name]
assert entry["type"] == (f.type or "document")
assert entry["title"] == okf.unquote_scalar(f.frontmatter.get("title", f.name))
assert entry["chars"] == len(f.body)
assert str(entry["title"]).strip(), "an untitled entry cannot be chosen between"
def test_the_verdict_layer_is_still_excluded(tmp_path: Path) -> None:
"""(d) Prior verdicts reach a hypothesis ONLY through the gated ExpeL fold (målbilde §4).
The old body returned ``okf.bundle_context``, which excludes ``type: verdict`` by construction.
A listing built from ``bundle.files`` instead of ``bundle.context_files`` would put them back
in front of the navigator — around the gate — and every other arm here would stay green.
"""
base = _write_base(tmp_path, "med-dommer", body_chars=200, concepts=2, verdicts=2)
entries = _listing(base)
assert len(entries) == 2, f"the verdict layer must not be listed as context: {entries!r}"
assert not [e for e in entries if e["type"] == "verdict"]
assert not [e for e in entries if str(e["name"]).startswith("dom-")]
# The control that proves the fixture actually HAS verdicts to leak — without it this arm is
# green against a bundle that simply carries none.
assert len(okf.navigate_bundle(str(base)).verdicts) == 2
def test_the_whole_document_is_still_one_call_away() -> None:
"""(e) The bound is a disclosure LEVEL, not data loss."""
entries = _listing(_TUNNEL)
biggest = max(entries, key=lambda e: int(e["chars"]))
whole = _tools(_TUNNEL)["read_file"].func(bundle_id=_TUNNEL.name, path=str(biggest["name"]))
assert len(whole) > _CEILING_CHARS, "read_file must still return the document, not a summary"
assert str(biggest["chars"]) != "0" and int(biggest["chars"]) <= len(whole)
def test_control_one_document_alone_would_blow_the_ceiling() -> None:
"""(f) The ceiling discriminates — proved, not assumed. The order names this control."""
bundle = okf.navigate_bundle(str(_TUNNEL))
biggest = max(len(f.body) for f in bundle.context_files)
assert biggest > _CEILING_CHARS, (
f"the largest document in {_TUNNEL.name} is {biggest} chars; a ceiling it does not exceed "
"would be a ceiling this base could pass while carrying everything"
)