open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 3 Activity

feat(explore): read_bundle costs O(documents), never O(bytes of the base)

Browse source 
[skip-docs]

S2c / MAJOR-3, order 20260902T151931Z-250257273. The measurement landed first
in ce7f687; this commit is the one seam it authorised, plus the after-table.

read_bundle returned okf.bundle_context -- the WHOLE navigated base. Because
the exploration's participants share one conversation history, that single
function_result rode in FIVE later prompts at full price without anyone asking
for it again: 54-59 percent of every prompt-token in a CLI --explore run.

It now returns the catalogue form one rung down the ladder -- one entry per
concept document (name, type, title, chars) -- with read_file as the next rung.
Tunnel base: 12 595 -> 259 o200k tokens, exploration prompt-tokens -91 percent.

The listing is built from Bundle.context_files and never from files: that is
the property which drops the type: verdict layer at every level, and a listing
built from files would route prior verdicts in front of the navigator around
the gated ExpeL fold while every cost arm stayed green.

A premise was felled before anything was built on it: the tunnel base's root
index body is 4 763 chars alone, nearly the whole ceiling, for a field the
catalogue already excerpts and read_file still returns whole. So read_bundle
carries the listing and not the index.

The tool description and the navigator's instruction both claimed "read its
navigated context" and were updated in the same move -- a description that lies
about the body IS the model's instruction. Two pre-existing asserts would have
gone vacuously true against a list and were strengthened rather than left.

Ceiling lives in the test, not in explore.py. Deviation stated there and in the
docs: it bounds CHARACTERS, not tokens, because tiktoken is not a project
dependency and a gate that skips when an optional package is missing is a gate
that can be silently absent; the conversion was measured (2.89 chars/token) and
the order's own token criterion verified once by the instrument.

Load-bearing measured: seven mutations, all red against the WHOLE suite; green
control 1195 passed / 5 skipped (from 1189/5, strict superset); golden
demo-transcript.stdout byte-unchanged; and the debate's three bundle_context
copies are byte-identical before and after, which proves run.py and the
nav-goldens were not touched rather than asserting it.
This commit is contained in:
Kjell Tore Guttormsen 2026-09-03 00:36:01 +02:00
commit b799cc527f
6 changed files with 356 additions and 10 deletions
Download patch file Download diff file Expand all files Collapse all files

52
src/portfolio_optimiser/explore.py
View file

@ -193,9 +193,10 @@ HYPOTHESIS_MARKER: Final = "HYPOTHESIS:"
_INSTRUCTIONS: Final = {
NAVIGATOR_ROLE: (
"You read the project's knowledge bases. Use list_bundles to see what exists, then "
"read_bundle to open ONE at a time and read_file to follow a specific document. Quote "
"what you found; never guess at content you have not read."
"You read the project's knowledge bases. Use list_bundles to see what exists, "
"read_bundle to list the documents inside ONE of them, and read_file to read a document "
"you picked from that list. read_bundle returns a LISTING, never the documents — quote "
"only what read_file gave you, and never guess at content you have not read."
),
HYPOTHESISER_ROLE: (
"You shape ONE candidate cost-saving direction at a time from what the navigator found. "
@ -817,8 +818,20 @@ def navigator_tools(bundle_dirs: Sequence[str]) -> list[FunctionTool]:
(session 51's "a skip is tolerated but no longer silent") while its per-link detail stays where
it is actionable, on ``RunResult.skipped_links`` / ``DryRunReport.skipped_links``.
``read_bundle`` returns ``okf.bundle_context``, which EXCLUDES the ``type: verdict`` layer by
construction — prior verdicts reach a hypothesis only through the gated ExpeL fold inside
**``read_bundle`` is the SAME rung one level down, and for the same measured reason.** It used
to return ``okf.bundle_context`` — the whole navigated base. Measured before the change
(``docs/2026-09-02-read-bundle-kontekstkostnad.md``): 3 861 / 10 406 / 12 595 o200k_base tokens
for the three example bases, and because the exploration's participants share one conversation
history, that single ``function_result`` rides in FIVE later prompts — 54-59 % of every
prompt-token in one CLI ``--explore`` run, none of it asked for twice. It now returns the
catalogue form: one entry per concept document (``name``, ``type``, ``title``, ``chars``), with
``read_file`` as the next rung, so a navigator pays for the documents it opened rather than for
the ones it did not. The root index body is deliberately NOT carried — the tunnel base's alone
is 4 763 characters, and ``list_bundles`` already excerpts it while ``read_file(id, "index.md")``
still returns it whole. Ceiling in ``tests/test_read_bundle_cost_loadbearing.py``, never here.
The listing is built from ``Bundle.context_files``, which EXCLUDES the ``type: verdict`` layer
by construction — prior verdicts reach a hypothesis only through the gated ExpeL fold inside
``run_project``, never by being read as context here.
"""
index = _bundle_index(bundle_dirs)
@ -856,15 +869,38 @@ def navigator_tools(bundle_dirs: Sequence[str]) -> list[FunctionTool]:
@tool(
name="read_bundle",
description="Open ONE knowledge base by id and read its navigated context.",
description=(
"Open ONE knowledge base by id and list what it holds: one entry per concept "
"document with its name, declared type, title and size in characters. This is a "
"LISTING, not the documents themselves — read_file(id, name) returns one whole."
),
)
def read_bundle(bundle_id: str) -> str:
def read_bundle(bundle_id: str) -> list[dict[str, Any]]:
bundle_dir = _resolve_bundle(index, bundle_id)
# The base is OPENED here, so this is where it is reconciled against its mount (Step 10).
# ``_bundle_index`` above stays PURE — it does no file I/O, and must not: two of its own
# arms configure directories that do not exist and expect an id error, not an I/O one.
okf.reconcile_bundle_id(bundle_dir)
return okf.bundle_context(okf.navigate_bundle(bundle_dir))
bundle = okf.navigate_bundle(bundle_dir)
# ``context_files``, NEVER ``files``: it is the property that drops the ``type: verdict``
# layer AND nested ``index.md`` at every level. Building from ``files`` would put prior
# verdicts in front of the navigator around the gated ExpeL fold (målbilde §4), and the
# cost arms of the gate would all stay green while it happened.
return [
{
"name": f.name,
# ``or "document"`` mirrors ``okf.bundle_context``'s own fallback for a file with
# no declared type, so the two renderings of one bundle cannot disagree about it.
"type": f.type or "document",
# ``okf`` owns the unquoting rule; a hand-rolled strip here would be a second copy
# of it, and a second copy is the one that drifts (kø-(p)).
"title": okf.unquote_scalar(f.frontmatter.get("title", f.name)),
# What the next rung COSTS, in the unit the ceiling is measured in. A navigator
# that cannot see the price cannot choose against a budget.
"chars": len(f.body),
}
for f in bundle.context_files
]
@tool(
name="read_file",