feat(explore): read_bundle costs O(documents), never O(bytes of the base)
[skip-docs]
S2c / MAJOR-3, order 20260902T151931Z-250257273. The measurement landed first
in ce7f687; this commit is the one seam it authorised, plus the after-table.
read_bundle returned okf.bundle_context -- the WHOLE navigated base. Because
the exploration's participants share one conversation history, that single
function_result rode in FIVE later prompts at full price without anyone asking
for it again: 54-59 percent of every prompt-token in a CLI --explore run.
It now returns the catalogue form one rung down the ladder -- one entry per
concept document (name, type, title, chars) -- with read_file as the next rung.
Tunnel base: 12 595 -> 259 o200k tokens, exploration prompt-tokens -91 percent.
The listing is built from Bundle.context_files and never from files: that is
the property which drops the type: verdict layer at every level, and a listing
built from files would route prior verdicts in front of the navigator around
the gated ExpeL fold while every cost arm stayed green.
A premise was felled before anything was built on it: the tunnel base's root
index body is 4 763 chars alone, nearly the whole ceiling, for a field the
catalogue already excerpts and read_file still returns whole. So read_bundle
carries the listing and not the index.
The tool description and the navigator's instruction both claimed "read its
navigated context" and were updated in the same move -- a description that lies
about the body IS the model's instruction. Two pre-existing asserts would have
gone vacuously true against a list and were strengthened rather than left.
Ceiling lives in the test, not in explore.py. Deviation stated there and in the
docs: it bounds CHARACTERS, not tokens, because tiktoken is not a project
dependency and a gate that skips when an optional package is missing is a gate
that can be silently absent; the conversion was measured (2.89 chars/token) and
the order's own token criterion verified once by the instrument.
Load-bearing measured: seven mutations, all red against the WHOLE suite; green
control 1195 passed / 5 skipped (from 1189/5, strict superset); golden
demo-transcript.stdout byte-unchanged; and the debate's three bundle_context
copies are byte-identical before and after, which proves run.py and the
nav-goldens were not touched rather than asserting it.
This commit is contained in:
parent
ce7f687717
commit
b799cc527f
6 changed files with 356 additions and 10 deletions
46
CLAUDE.md
46
CLAUDE.md
|
|
@ -1073,6 +1073,52 @@ Python ≥3.10. MAF (`agent-framework-core` 1.16.0, `-orchestrations` 1.1.1 —
|
||||||
nye verktøyet, så at en manager velger BEDRE med et utdrag enn med hele indeksen er ikke bevist
|
nye verktøyet, så at en manager velger BEDRE med et utdrag enn med hele indeksen er ikke bevist
|
||||||
(structured-output-grensens klasse); og ordrens nevner for N100:2023 var 34 mens disken viser 40 —
|
(structured-output-grensens klasse); og ordrens nevner for N100:2023 var 34 mens disken viser 40 —
|
||||||
tallene bruker den målte nevneren. Måling: `docs/2026-08-26-katalogkostnaden.md`.
|
tallene bruker den målte nevneren. Måling: `docs/2026-08-26-katalogkostnaden.md`.
|
||||||
|
- **`read_bundle` koster O(DOKUMENTER i én base), aldri O(bytes av den — S2c/MAJOR-3, økt 77):**
|
||||||
|
verktøyet returnerte `okf.bundle_context`, altså HELE den navigerte basen, og fordi utforskningens
|
||||||
|
deltakere deler ÉN samtalehistorikk red det ene `function_result`-et med i **hver senere prompt**
|
||||||
|
til full pris uten at noen ba om det igjen. **MÅLT FØRST, med nevner, og committet som docs før én
|
||||||
|
linje kode ble rørt** (`docs/2026-09-02-read-bundle-kontekstkostnad.md`): 3 861 / 10 406 / **12 595**
|
||||||
|
o200k-tokens for de tre eksempelbasene, **5 kopier** per CLI-`--explore`-kjøring (navigatør ×1,
|
||||||
|
manager ×3, hypotesiser ×1) = **54 % / 59 % / 59 %** av ALLE prompt-tokens. Instrumentet ble
|
||||||
|
validert mot en KJENT POSITIV før bruk (det reproduserte commons' egne publiserte
|
||||||
|
`bundle_context`-fasittall eksakt), og prompten måles som tekst + `function_call` +
|
||||||
|
`function_result` — `.text` alene måler en kontekstbærende prompt til noen få tegn.
|
||||||
|
**Formen er katalogens, ett trinn ned på stigen** (`list_bundles` = hvilke baser finnes,
|
||||||
|
`read_bundle` = hva er i DENNE, `read_file` = hva sier dokumentet): én oppføring per konseptfil med
|
||||||
|
`name`, `type`, `title`, `chars`. Etter: **259 tokens** på tunnelbasen, utforskningens
|
||||||
|
prompt-tokens −77 / −90 / **−91 %**. **Listen bygges av `Bundle.context_files`, ALDRI `files`** —
|
||||||
|
det er dén property som dropper `type: verdict`-laget (og nestede `index.md`) på hvert nivå, og en
|
||||||
|
liste bygget av `files` ville rutet tidligere dommer foran navigatøren UTENOM den gatede
|
||||||
|
ExpeL-folden mens hver kostnads-arm forble grønn. **Et premiss ble felt før noe ble bygget på det:**
|
||||||
|
«indeksbodyen er basens egen navigasjonsprosa, så den hører hjemme her» — tunnelbasens rot-indeks er
|
||||||
|
**alene 4 763 tegn ≈ 1 400 tokens**, altså nesten hele taket, for et felt katalogen alt gir et
|
||||||
|
bundet utdrag av og `read_file(id, "index.md")` fortsatt gir helt. **Taket bor i TESTEN**
|
||||||
|
(`_CEILING_CHARS = 1 500`), katalogtakets regel av katalogtakets grunn. **AVVIK fra ordren, uttalt:**
|
||||||
|
ordren ordlegger gaten i o200k-TOKENS; gaten bounder TEGN, fordi `tiktoken` ikke er en
|
||||||
|
prosjektavhengighet og en gate som skipper når en valgfri pakke mangler er en gate som kan være
|
||||||
|
stille fraværende — konverteringen er MÅLT (748 tegn / 259 tokens = 2,89 tegn/token), og ordrens
|
||||||
|
eget kriterium er verifisert direkte én gang av instrumentet. **«Ikke utløs» er bevist som en
|
||||||
|
MÅLING, ikke som en forsikring:** debattens tre `okf.bundle_context`-kopier er **byte-identiske før
|
||||||
|
og etter** i alle tre baser (12 047 / 32 567 / 39 104 prompt-tokens, hver prompt uendret) — sterkere
|
||||||
|
enn å påstå at `run.py` ikke ble rørt. **Verktøybeskrivelsen og navigatørens instruksjon er oppdatert
|
||||||
|
i samme trekk:** begge påsto «read its navigated context», og en beskrivelse som lyver om kroppen er
|
||||||
|
modellens instruks (Fase 3-klassen, påstander flaten gjør om SEG SELV). **To eksisterende asserts
|
||||||
|
ville blitt VAKUØSE i stillhet** (`read_bundle(...) != ""` er sant for enhver liste,
|
||||||
|
`test_explore_loadbearing.py` + `test_bundle_id_reconciliation_loadbearing.py`) og er styrket, ikke
|
||||||
|
latt stå. Load-bearing MÅLT (`tests/test_read_bundle_cost_loadbearing.py`, 6 armer), **sju mutasjoner
|
||||||
|
alle røde mot HELE suiten** + grønn kontroll **1 195/5** (fra 1 189/5, supersett, 0 fjernet) og golden
|
||||||
|
`demo-transcript.stdout` BYTE-UENDRET (`ea8c534773acdbe41ae68f2c55724d69aaf8be4f`): reverter sømmen
|
||||||
|
(7 røde) · bygg fra `files` (3) · bundet men VAKUØS tom liste (6) · dropp `chars` (2) · bær
|
||||||
|
indeksbodyen likevel (4) · `read_file` trunkerer (2 — inkludert katalog-gaten, et uavhengig vitne) ·
|
||||||
|
detach `reconcile_bundle_id` (1). **Ærlighets-grenser, uttalt:** dette er IKKE «−59 % kostnad» — en
|
||||||
|
navigatør som åpner *k* dokumenter betaler *k* `read_file`-resultater, og gevinsten er at den betaler
|
||||||
|
for det den VALGTE og at hvert resultat rir fra SITT kall og framover; at en LEVENDE modell velger
|
||||||
|
BEDRE med en liste enn med hele konteksten er IKKE bevist (structured-output-grensens klasse);
|
||||||
|
multiplikatoren 5 gjelder dette manuset (`misjonsreview-v2` § 4 målte 7× med et annet); debattens 3×
|
||||||
|
er PM-ens egen beslutning og er urørt; prefiks-caching (≈ 60 % debatt / ≈ 40 % utforskning cachebart
|
||||||
|
som koden står) er NOTERT, ikke bygget. **Et fravær som var instrumentfeil, ikke faktum:** første
|
||||||
|
ETTER-kjøring rapporterte 0 kopier på to baser — sonden var en midtskive som traff norske tegn
|
||||||
|
prompten serialiserer escaped; byttet til et ASCII-konseptfilnavn ga 5, som før.
|
||||||
- **Ekspertdommen kan ikke oppstå av STILLHET, og fraværet er en FØRSTEKLASSES tilstand (F2,
|
- **Ekspertdommen kan ikke oppstå av STILLHET, og fraværet er en FØRSTEKLASSES tilstand (F2,
|
||||||
non-goal 3, økt 66):** `run_project` KREVDE `verdict_input` og kjørte `capture_verdict`
|
non-goal 3, økt 66):** `run_project` KREVDE `verdict_input` og kjørte `capture_verdict`
|
||||||
ubetinget, CLI-en defaultet det til `{"approved", "reviewed by expert"}`, og hosting listet det
|
ubetinget, CLI-en defaultet det til `{"approved", "reviewed by expert"}`, og hosting listet det
|
||||||
|
|
|
||||||
|
|
@ -95,3 +95,57 @@ denne ordren gjør bare det siste.
|
||||||
| 3 | Prompt-størrelse inkluderer verktøyresultatet | wrapper på `_inner_get_response` serialiserer `contents` (`function_call`/`function_result`), ikke `.text` |
|
| 3 | Prompt-størrelse inkluderer verktøyresultatet | wrapper på `_inner_get_response` serialiserer `contents` (`function_call`/`function_result`), ikke `.text` |
|
||||||
| 4 | Nevner | 15 / 19 / 19 prompts per kjøring; 8 bærer konteksten i alle tre |
|
| 4 | Nevner | 15 / 19 / 19 prompts per kjøring; 8 bærer konteksten i alle tre |
|
||||||
| 5 | Indeksbodyen alene sprenger nesten hele taket | `len(navigate_bundle(tunnel).index_summary)` → 4 763 tegn ≈ 1 400 tok |
|
| 5 | Indeksbodyen alene sprenger nesten hele taket | `len(navigate_bundle(tunnel).index_summary)` → 4 763 tegn ≈ 1 400 tok |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. ETTER — samme instrument, samme kommandoer, samme baser
|
||||||
|
|
||||||
|
`read_bundle` returnerer nå katalogformen: én oppføring per konseptdokument (`name`, `type`,
|
||||||
|
`title`, `chars`), med `read_file` som neste trinn.
|
||||||
|
|
||||||
|
| Base | Nyttelast før → etter | Kopier i utforsknings-prompts | Nyttelast × kopier | Utforsknings-prompts totalt | Hele kjøringen |
|
||||||
|
|---|---|---:|---|---:|---|
|
||||||
|
| `bygg-energi-mikro` | 3 861 → **163 tok** | 5 → 5 | 19 305 → **815** | 23 726 → **5 434** (−77 %) | 35 773 → 17 481 (−51 %) |
|
||||||
|
| `veglys-fv-soer` | 10 406 → **237 tok** | 5 → 5 | 52 030 → **1 185** | 56 314 → **5 742** (−90 %) | 88 881 → 38 309 (−57 %) |
|
||||||
|
| `tunnel-hauglia` | 12 595 → **259 tok** | 5 → 5 | 62 975 → **1 295** | 67 415 → **5 973** (−91 %) | 106 519 → 45 077 (−58 %) |
|
||||||
|
|
||||||
|
**Ordrens eget kriterium, verifisert direkte:** `read_bundle` over tunnelbasen er **748 tegn / 259
|
||||||
|
o200k-tokens** — under taket på 1 500. Målt forhold 2,89 tegn/token for denne norske markdownen;
|
||||||
|
det er dét som lar gaten bounde TEGN uten å gjette (se testens docstring, som uttaler avviket).
|
||||||
|
|
||||||
|
**«Ikke utløs»-beviset er en MÅLING, ikke en forsikring.** Ordren forbyr å røre `okf.bundle_context`,
|
||||||
|
nav-goldenene og `run.py`. Debattens tre kontekstkopier kommer fra nettopp den sømmen, og de er
|
||||||
|
**byte-identiske før og etter** i alle tre baser — 12 047 / 32 567 / 39 104 prompt-tokens, hver
|
||||||
|
enkelt prompt uendret. Det er et sterkere utsagn enn «jeg rørte den ikke». I tillegg: hele suiten
|
||||||
|
grønn (**1 195 passed / 5 skipped**, mot 1 189/5 før — supersett, 0 fjernet), tre byte-fasiter OK,
|
||||||
|
og golden `demo-transcript.stdout` UENDRET (`ea8c534773acdbe41ae68f2c55724d69aaf8be4f`).
|
||||||
|
|
||||||
|
**Et fravær som var et instrumentfeil, ikke et faktum (Verifiseringsloven ansikt 4).** Den første
|
||||||
|
ETTER-kjøringen rapporterte **0 kopier** på veglys og tunnel. Det var ikke sant: sonden var en
|
||||||
|
160-tegns skive fra MIDTEN av nyttelasten, og den nye nyttelasten er kort nok til at midten treffer
|
||||||
|
norske tegn, som prompten serialiserer escaped (`å`) mens sonden holdt dem rå. Sonden ble byttet
|
||||||
|
til et konseptfilnavn (ASCII, ordrett i begge), og svaret ble 5 — samme tall som før endringen.
|
||||||
|
Et negativt resultat fra en ødelagt spørring er ikke en måling.
|
||||||
|
|
||||||
|
## 6. Ærlighetsgrenser, uttalt
|
||||||
|
|
||||||
|
1. **Dette er ikke «−59 % kostnad».** Det målte utsagnet er at `read_bundle`s EGET bidrag faller fra
|
||||||
|
5 × 12 595 til 5 × 259 tokens på tunnelbasen. En navigatør som deretter åpner *k* dokumenter
|
||||||
|
betaler *k* `read_file`-resultater, og en som åpner ALT betaler omtrent de samme bytene — bare
|
||||||
|
per kall. Gevinsten er at den betaler for det den valgte, og at hvert resultat rir fra SITT eget
|
||||||
|
kall og framover i stedet for at alt rir fra det første.
|
||||||
|
2. **At en LEVENDE modell velger BEDRE med en liste enn med hele konteksten er IKKE bevist.** Samme
|
||||||
|
klasse som structured-output-grensen: det som er bevist er formen og prisen, ikke dømmekraften.
|
||||||
|
3. **Multiplikatoren 5 gjelder dette manuset.** Se § 1.
|
||||||
|
4. **Debattens 3× står urørt** og er PM-ens beslutning, ikke denne ordrens.
|
||||||
|
5. **Prefiks-caching er notert (§ 3), ikke bygget** — ordren ba om notatet.
|
||||||
|
|
||||||
|
## 7. Verifiseringslogg (ETTER-halvdelen)
|
||||||
|
|
||||||
|
| # | Påstand | Kommando → resultat |
|
||||||
|
|---|---|---|
|
||||||
|
| 6 | Gaten er rød uten sømmen | 7 mutasjoner mot HELE suiten, alle røde: se CLAUDE.md-raden |
|
||||||
|
| 7 | Grønn kontroll | `uv run pytest -q` → 1 195 passed / 5 skipped (før: 1 189/5) |
|
||||||
|
| 8 | Golden uendret | `PYTHONIOENCODING=utf-8 uv run python -m portfolio_optimiser.simulation \| shasum` → `ea8c5347…` |
|
||||||
|
| 9 | Debatten urørt | ETTER-kjøringens `proposer`/`checker`-prompts identiske med FØR-kjøringens, per prompt |
|
||||||
|
| 10 | Lint + typer | `uv run ruff check .` → All checks passed; `uv run mypy src` → no issues, 35 files |
|
||||||
|
|
|
||||||
|
|
@ -193,9 +193,10 @@ HYPOTHESIS_MARKER: Final = "HYPOTHESIS:"
|
||||||
|
|
||||||
_INSTRUCTIONS: Final = {
|
_INSTRUCTIONS: Final = {
|
||||||
NAVIGATOR_ROLE: (
|
NAVIGATOR_ROLE: (
|
||||||
"You read the project's knowledge bases. Use list_bundles to see what exists, then "
|
"You read the project's knowledge bases. Use list_bundles to see what exists, "
|
||||||
"read_bundle to open ONE at a time and read_file to follow a specific document. Quote "
|
"read_bundle to list the documents inside ONE of them, and read_file to read a document "
|
||||||
"what you found; never guess at content you have not read."
|
"you picked from that list. read_bundle returns a LISTING, never the documents — quote "
|
||||||
|
"only what read_file gave you, and never guess at content you have not read."
|
||||||
),
|
),
|
||||||
HYPOTHESISER_ROLE: (
|
HYPOTHESISER_ROLE: (
|
||||||
"You shape ONE candidate cost-saving direction at a time from what the navigator found. "
|
"You shape ONE candidate cost-saving direction at a time from what the navigator found. "
|
||||||
|
|
@ -817,8 +818,20 @@ def navigator_tools(bundle_dirs: Sequence[str]) -> list[FunctionTool]:
|
||||||
(session 51's "a skip is tolerated but no longer silent") while its per-link detail stays where
|
(session 51's "a skip is tolerated but no longer silent") while its per-link detail stays where
|
||||||
it is actionable, on ``RunResult.skipped_links`` / ``DryRunReport.skipped_links``.
|
it is actionable, on ``RunResult.skipped_links`` / ``DryRunReport.skipped_links``.
|
||||||
|
|
||||||
``read_bundle`` returns ``okf.bundle_context``, which EXCLUDES the ``type: verdict`` layer by
|
**``read_bundle`` is the SAME rung one level down, and for the same measured reason.** It used
|
||||||
construction — prior verdicts reach a hypothesis only through the gated ExpeL fold inside
|
to return ``okf.bundle_context`` — the whole navigated base. Measured before the change
|
||||||
|
(``docs/2026-09-02-read-bundle-kontekstkostnad.md``): 3 861 / 10 406 / 12 595 o200k_base tokens
|
||||||
|
for the three example bases, and because the exploration's participants share one conversation
|
||||||
|
history, that single ``function_result`` rides in FIVE later prompts — 54-59 % of every
|
||||||
|
prompt-token in one CLI ``--explore`` run, none of it asked for twice. It now returns the
|
||||||
|
catalogue form: one entry per concept document (``name``, ``type``, ``title``, ``chars``), with
|
||||||
|
``read_file`` as the next rung, so a navigator pays for the documents it opened rather than for
|
||||||
|
the ones it did not. The root index body is deliberately NOT carried — the tunnel base's alone
|
||||||
|
is 4 763 characters, and ``list_bundles`` already excerpts it while ``read_file(id, "index.md")``
|
||||||
|
still returns it whole. Ceiling in ``tests/test_read_bundle_cost_loadbearing.py``, never here.
|
||||||
|
|
||||||
|
The listing is built from ``Bundle.context_files``, which EXCLUDES the ``type: verdict`` layer
|
||||||
|
by construction — prior verdicts reach a hypothesis only through the gated ExpeL fold inside
|
||||||
``run_project``, never by being read as context here.
|
``run_project``, never by being read as context here.
|
||||||
"""
|
"""
|
||||||
index = _bundle_index(bundle_dirs)
|
index = _bundle_index(bundle_dirs)
|
||||||
|
|
@ -856,15 +869,38 @@ def navigator_tools(bundle_dirs: Sequence[str]) -> list[FunctionTool]:
|
||||||
|
|
||||||
@tool(
|
@tool(
|
||||||
name="read_bundle",
|
name="read_bundle",
|
||||||
description="Open ONE knowledge base by id and read its navigated context.",
|
description=(
|
||||||
|
"Open ONE knowledge base by id and list what it holds: one entry per concept "
|
||||||
|
"document with its name, declared type, title and size in characters. This is a "
|
||||||
|
"LISTING, not the documents themselves — read_file(id, name) returns one whole."
|
||||||
|
),
|
||||||
)
|
)
|
||||||
def read_bundle(bundle_id: str) -> str:
|
def read_bundle(bundle_id: str) -> list[dict[str, Any]]:
|
||||||
bundle_dir = _resolve_bundle(index, bundle_id)
|
bundle_dir = _resolve_bundle(index, bundle_id)
|
||||||
# The base is OPENED here, so this is where it is reconciled against its mount (Step 10).
|
# The base is OPENED here, so this is where it is reconciled against its mount (Step 10).
|
||||||
# ``_bundle_index`` above stays PURE — it does no file I/O, and must not: two of its own
|
# ``_bundle_index`` above stays PURE — it does no file I/O, and must not: two of its own
|
||||||
# arms configure directories that do not exist and expect an id error, not an I/O one.
|
# arms configure directories that do not exist and expect an id error, not an I/O one.
|
||||||
okf.reconcile_bundle_id(bundle_dir)
|
okf.reconcile_bundle_id(bundle_dir)
|
||||||
return okf.bundle_context(okf.navigate_bundle(bundle_dir))
|
bundle = okf.navigate_bundle(bundle_dir)
|
||||||
|
# ``context_files``, NEVER ``files``: it is the property that drops the ``type: verdict``
|
||||||
|
# layer AND nested ``index.md`` at every level. Building from ``files`` would put prior
|
||||||
|
# verdicts in front of the navigator around the gated ExpeL fold (målbilde §4), and the
|
||||||
|
# cost arms of the gate would all stay green while it happened.
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
"name": f.name,
|
||||||
|
# ``or "document"`` mirrors ``okf.bundle_context``'s own fallback for a file with
|
||||||
|
# no declared type, so the two renderings of one bundle cannot disagree about it.
|
||||||
|
"type": f.type or "document",
|
||||||
|
# ``okf`` owns the unquoting rule; a hand-rolled strip here would be a second copy
|
||||||
|
# of it, and a second copy is the one that drifts (kø-(p)).
|
||||||
|
"title": okf.unquote_scalar(f.frontmatter.get("title", f.name)),
|
||||||
|
# What the next rung COSTS, in the unit the ceiling is measured in. A navigator
|
||||||
|
# that cannot see the price cannot choose against a budget.
|
||||||
|
"chars": len(f.body),
|
||||||
|
}
|
||||||
|
for f in bundle.context_files
|
||||||
|
]
|
||||||
|
|
||||||
@tool(
|
@tool(
|
||||||
name="read_file",
|
name="read_file",
|
||||||
|
|
|
||||||
|
|
@ -220,7 +220,10 @@ def test_explore_read_bundle_reconciles_the_base_it_opens(_spy: list[str]) -> No
|
||||||
"""(h) The tool is driven DIRECTLY: measured in okt 56, a scripted client returns TEXT and never
|
"""(h) The tool is driven DIRECTLY: measured in okt 56, a scripted client returns TEXT and never
|
||||||
emits a tool call, so a gate that only drove ``explore()`` would never enter this body."""
|
emits a tool call, so a gate that only drove ``explore()`` would never enter this body."""
|
||||||
tools = {t.name: t for t in explore.navigator_tools((str(_BYGG),))}
|
tools = {t.name: t for t in explore.navigator_tools((str(_BYGG),))}
|
||||||
assert tools["read_bundle"].func(bundle_id=_BYGG.name) != ""
|
# See test_read_bundle_cost_loadbearing: the tool returns a LISTING since S2c, and ``!= ""``
|
||||||
|
# holds for every list ever built — this arm is about the reconciliation, so the call must
|
||||||
|
# still be a call that DID something.
|
||||||
|
assert [e["name"] for e in tools["read_bundle"].func(bundle_id=_BYGG.name)]
|
||||||
assert str(_BYGG) in _spy
|
assert str(_BYGG) in _spy
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -683,7 +683,9 @@ async def test_an_exploration_leaves_the_knowledge_base_byte_identical(tmp_path:
|
||||||
# Every read tool, called on the same base, with model-shaped arguments.
|
# Every read tool, called on the same base, with model-shaped arguments.
|
||||||
tools = {t.name: t for t in explore.navigator_tools((str(base),))}
|
tools = {t.name: t for t in explore.navigator_tools((str(base),))}
|
||||||
assert tools["list_bundles"].func()[0]["id"] == base.name
|
assert tools["list_bundles"].func()[0]["id"] == base.name
|
||||||
assert tools["read_bundle"].func(bundle_id=base.name) != ""
|
# A LISTING since S2c, so ``!= ""`` would be vacuously true for any list at all — assert the
|
||||||
|
# shape the tool now promises instead of the shape it used to.
|
||||||
|
assert [e["name"] for e in tools["read_bundle"].func(bundle_id=base.name)]
|
||||||
assert tools["read_file"].func(bundle_id=base.name, path="index.md") != ""
|
assert tools["read_file"].func(bundle_id=base.name, path="index.md") != ""
|
||||||
explore.quick_validate_tool((str(base),)).func(
|
explore.quick_validate_tool((str(base),)).func(
|
||||||
bundle_id=base.name, proposal_json=json.dumps(_micro_projection())
|
bundle_id=base.name, proposal_json=json.dumps(_micro_projection())
|
||||||
|
|
|
||||||
205
tests/test_read_bundle_cost_loadbearing.py
Normal file
205
tests/test_read_bundle_cost_loadbearing.py
Normal file
|
|
@ -0,0 +1,205 @@
|
||||||
|
"""``read_bundle`` costs O(DOCUMENTS in one base), never O(bytes of that base).
|
||||||
|
|
||||||
|
MAJOR-3 (``docs/2026-09-02-misjonsreview-v2.md`` § 7), measured before anything was changed in
|
||||||
|
``docs/2026-09-02-read-bundle-kontekstkostnad.md``:
|
||||||
|
|
||||||
|
read_bundle payload bygg 3 861 / veglys 10 406 / tunnel 12 595 o200k_base tokens
|
||||||
|
prompts one result rides in FIVE (navigator 1, manager 3, hypothesiser 1)
|
||||||
|
share of every prompt-token 54 % / 59 % / 59 % of one CLI ``--explore`` run
|
||||||
|
|
||||||
|
``read_bundle`` returned ``okf.bundle_context`` — the WHOLE navigated base. One call turned the
|
||||||
|
base into a ``function_result``, and because the exploration's participants share one conversation
|
||||||
|
history that result rides in **every later prompt**, at full price, without anyone asking for it
|
||||||
|
again. That is the same shape ``list_bundles`` was measured in and rebuilt out of in session 65
|
||||||
|
(``tests/test_catalogue_cost_loadbearing.py``), one rung down the ladder: the catalogue answers
|
||||||
|
*which bases exist*, ``read_bundle`` answers *what is in this one*, and ``read_file`` answers *what
|
||||||
|
does this document say*. Only the last of the three should cost what a document costs.
|
||||||
|
|
||||||
|
So ``read_bundle`` now returns the CATALOGUE FORM: one entry per concept document — ``name``,
|
||||||
|
``type``, ``title``, ``chars`` — and ``read_file(id, name)`` is the next rung. This is a disclosure
|
||||||
|
level, not data loss: every byte is still exactly one call away, and a navigator now pays for the
|
||||||
|
documents it chose to open instead of for the ones it did not.
|
||||||
|
|
||||||
|
**A premise felled before it was built on** (see the measurement doc § 2): "the index body is the
|
||||||
|
base's own navigation prose, so it belongs here". The tunnel base's root index is **4 763 chars
|
||||||
|
alone ≈ 1 400 tokens** — nearly the entire ceiling, for a field ``list_bundles`` already excerpts
|
||||||
|
and ``read_file(id, "index.md")`` still returns whole.
|
||||||
|
|
||||||
|
**The ceiling lives in THIS FILE, not in ``explore.py``** — the catalogue gate's rule, for the
|
||||||
|
catalogue gate's reason: a test that imported the implementation's budget would move with it, and
|
||||||
|
raising the budget is precisely the regression this file exists to catch.
|
||||||
|
|
||||||
|
**DEVIATION from the order, stated:** the order words the gate as "< 1 500 o200k-tokens". This
|
||||||
|
file bounds CHARACTERS instead. ``tiktoken`` is not a project dependency (and adding one for a
|
||||||
|
gate would be a bigger decision than the gate), and a gate that skips when an optional package is
|
||||||
|
missing is a gate that can be silently absent. The character ceiling is a proxy whose conversion
|
||||||
|
was MEASURED rather than assumed: the new payload over the real tunnel base is **748 chars / 259
|
||||||
|
o200k tokens** (2.89 chars/token for this Norwegian markdown), so 1 500 characters is ≈ 520 tokens
|
||||||
|
— comfortably inside the order's criterion, and twice the measured payload, so ordinary field
|
||||||
|
growth does not force a rewrite. The order's own criterion is verified directly, once, by the
|
||||||
|
instrument in the measurement doc.
|
||||||
|
|
||||||
|
What the arms pin, and what each one refuses:
|
||||||
|
|
||||||
|
(a) the bound itself, over the REAL base the order names — refuses the unbounded form. A synthetic
|
||||||
|
fixture here would measure the fixture writer, not the base;
|
||||||
|
(b) cost tracks DOCUMENT COUNT, not document SIZE — ten times the prose, the same price. That is
|
||||||
|
the property stated directly rather than inferred from (a);
|
||||||
|
(c) the anti-vacuity arm: every concept document is still IDENTIFIED, with all four fields, and the
|
||||||
|
entry count equals the navigated context files. Without this, "return an empty list" passes (a)
|
||||||
|
perfectly and hands the navigator nothing to choose between — the repo's vacuous-gate class,
|
||||||
|
which has now bitten twelve times;
|
||||||
|
(d) the ``type: verdict`` layer stays EXCLUDED. ``read_bundle`` built from ``bundle.files`` instead
|
||||||
|
of ``bundle.context_files`` would route prior verdicts into a hypothesis prompt around the
|
||||||
|
gated ExpeL fold (målbilde §4), and nothing else in the suite would notice;
|
||||||
|
(e) the ladder is intact — ``read_file`` still returns the COMPLETE document, so the bound is a
|
||||||
|
disclosure level, not data loss;
|
||||||
|
(f) the CONTROL the order names: ONE concept document alone blows the ceiling for the whole
|
||||||
|
listing. A green (a) then means the bound fired, not that the base was small
|
||||||
|
(Verifiseringsloven face 4: a gate that can only pass proves nothing).
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from portfolio_optimiser import okf
|
||||||
|
from portfolio_optimiser.explore import navigator_tools
|
||||||
|
|
||||||
|
_EXAMPLES = Path(__file__).resolve().parents[1] / "shared" / "examples"
|
||||||
|
|
||||||
|
#: The base the order names. Measured, not chosen: the largest of the three example bases.
|
||||||
|
_TUNNEL = _EXAMPLES / "tunnel-hauglia"
|
||||||
|
|
||||||
|
#: Characters one ``read_bundle`` listing may cost. Test-owned on purpose — see the docstring, and
|
||||||
|
#: the measured chars/token conversion that ties it to the order's token criterion.
|
||||||
|
_CEILING_CHARS = 1_500
|
||||||
|
|
||||||
|
|
||||||
|
def _tools(bundle_dir: Path) -> dict[str, Any]:
|
||||||
|
return {t.name: t for t in navigator_tools((str(bundle_dir),))}
|
||||||
|
|
||||||
|
|
||||||
|
def _listing(bundle_dir: Path) -> list[dict[str, Any]]:
|
||||||
|
return _tools(bundle_dir)["read_bundle"].func(bundle_id=bundle_dir.name)
|
||||||
|
|
||||||
|
|
||||||
|
def _blob(payload: object) -> str:
|
||||||
|
return json.dumps(payload, ensure_ascii=False)
|
||||||
|
|
||||||
|
|
||||||
|
def _write_base(
|
||||||
|
root: Path, name: str, *, body_chars: int, concepts: int = 3, verdicts: int = 0
|
||||||
|
) -> Path:
|
||||||
|
base = root / name
|
||||||
|
base.mkdir(parents=True)
|
||||||
|
lines = [f"# {name}", ""]
|
||||||
|
for i in range(concepts):
|
||||||
|
lines.append(f"- [konsept-{i}](konsept-{i}.md)")
|
||||||
|
(base / f"konsept-{i}.md").write_text(
|
||||||
|
f"---\ntype: concept\ntitle: Konsept {i}\n---\n\n" + ("innhold. " * (body_chars // 9)),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
for i in range(verdicts):
|
||||||
|
lines.append(f"- [dom-{i}](dom-{i}.md)")
|
||||||
|
(base / f"dom-{i}.md").write_text(
|
||||||
|
f"---\ntype: verdict\ntitle: Dom {i}\n---\n\nEksperten godkjente tiltaket.\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
(base / "index.md").write_text(
|
||||||
|
"---\ntype: index\n---\n\n" + "\n".join(lines) + "\n", encoding="utf-8"
|
||||||
|
)
|
||||||
|
return base
|
||||||
|
|
||||||
|
|
||||||
|
def test_read_bundle_over_the_real_tunnel_base_is_bounded() -> None:
|
||||||
|
"""(a) The headline, over the base the order names — not a fixture of my own making."""
|
||||||
|
blob = _blob(_listing(_TUNNEL))
|
||||||
|
|
||||||
|
assert len(blob) <= _CEILING_CHARS, (
|
||||||
|
f"read_bundle over {_TUNNEL.name} costs {len(blob)} chars, over the ceiling "
|
||||||
|
f"{_CEILING_CHARS}; it used to be 39 583 (12 595 o200k tokens), riding in five prompts"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_read_bundle_cost_does_not_track_document_size(tmp_path: Path) -> None:
|
||||||
|
"""(b) Ten times the prose, the same price. O(documents), not O(bytes)."""
|
||||||
|
small = _write_base(tmp_path, "small", body_chars=500)
|
||||||
|
large = _write_base(tmp_path, "large", body_chars=5_000)
|
||||||
|
|
||||||
|
small_entries, large_entries = _listing(small), _listing(large)
|
||||||
|
|
||||||
|
# Same document COUNT, same number of entries — and the only field that grew is the honest,
|
||||||
|
# logarithmic ``chars`` digit, so the payloads differ by a handful of characters at most.
|
||||||
|
assert len(small_entries) == len(large_entries) == 3
|
||||||
|
assert abs(len(_blob(large_entries)) - len(_blob(small_entries))) < 20, (
|
||||||
|
"ten times the body must not cost ten times the listing"
|
||||||
|
)
|
||||||
|
assert len(_blob(large_entries)) <= _CEILING_CHARS
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_listing_still_identifies_every_document(tmp_path: Path) -> None:
|
||||||
|
"""(c) The anti-vacuity arm: bounded is not the same as empty.
|
||||||
|
|
||||||
|
Bounded-and-useless passes (a) perfectly. What a navigator needs in order to choose a document
|
||||||
|
is what it IS (``type``), what it is CALLED (``title``), what to ask for (``name``) and what it
|
||||||
|
will cost (``chars``) — so all four are asserted, and the entry count is tied to the navigated
|
||||||
|
context files rather than to a number written here.
|
||||||
|
"""
|
||||||
|
bundle = okf.navigate_bundle(str(_TUNNEL))
|
||||||
|
entries = _listing(_TUNNEL)
|
||||||
|
|
||||||
|
assert len(entries) == len(bundle.context_files) > 0, (
|
||||||
|
"a listing that omits documents is a base the navigator cannot fully see"
|
||||||
|
)
|
||||||
|
by_name = {str(e["name"]): e for e in entries}
|
||||||
|
for f in bundle.context_files:
|
||||||
|
entry = by_name[f.name]
|
||||||
|
assert entry["type"] == (f.type or "document")
|
||||||
|
assert entry["title"] == okf.unquote_scalar(f.frontmatter.get("title", f.name))
|
||||||
|
assert entry["chars"] == len(f.body)
|
||||||
|
assert str(entry["title"]).strip(), "an untitled entry cannot be chosen between"
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_verdict_layer_is_still_excluded(tmp_path: Path) -> None:
|
||||||
|
"""(d) Prior verdicts reach a hypothesis ONLY through the gated ExpeL fold (målbilde §4).
|
||||||
|
|
||||||
|
The old body returned ``okf.bundle_context``, which excludes ``type: verdict`` by construction.
|
||||||
|
A listing built from ``bundle.files`` instead of ``bundle.context_files`` would put them back
|
||||||
|
in front of the navigator — around the gate — and every other arm here would stay green.
|
||||||
|
"""
|
||||||
|
base = _write_base(tmp_path, "med-dommer", body_chars=200, concepts=2, verdicts=2)
|
||||||
|
|
||||||
|
entries = _listing(base)
|
||||||
|
|
||||||
|
assert len(entries) == 2, f"the verdict layer must not be listed as context: {entries!r}"
|
||||||
|
assert not [e for e in entries if e["type"] == "verdict"]
|
||||||
|
assert not [e for e in entries if str(e["name"]).startswith("dom-")]
|
||||||
|
# The control that proves the fixture actually HAS verdicts to leak — without it this arm is
|
||||||
|
# green against a bundle that simply carries none.
|
||||||
|
assert len(okf.navigate_bundle(str(base)).verdicts) == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_whole_document_is_still_one_call_away() -> None:
|
||||||
|
"""(e) The bound is a disclosure LEVEL, not data loss."""
|
||||||
|
entries = _listing(_TUNNEL)
|
||||||
|
biggest = max(entries, key=lambda e: int(e["chars"]))
|
||||||
|
|
||||||
|
whole = _tools(_TUNNEL)["read_file"].func(bundle_id=_TUNNEL.name, path=str(biggest["name"]))
|
||||||
|
|
||||||
|
assert len(whole) > _CEILING_CHARS, "read_file must still return the document, not a summary"
|
||||||
|
assert str(biggest["chars"]) != "0" and int(biggest["chars"]) <= len(whole)
|
||||||
|
|
||||||
|
|
||||||
|
def test_control_one_document_alone_would_blow_the_ceiling() -> None:
|
||||||
|
"""(f) The ceiling discriminates — proved, not assumed. The order names this control."""
|
||||||
|
bundle = okf.navigate_bundle(str(_TUNNEL))
|
||||||
|
|
||||||
|
biggest = max(len(f.body) for f in bundle.context_files)
|
||||||
|
|
||||||
|
assert biggest > _CEILING_CHARS, (
|
||||||
|
f"the largest document in {_TUNNEL.name} is {biggest} chars; a ceiling it does not exceed "
|
||||||
|
"would be a ceiling this base could pass while carrying everything"
|
||||||
|
)
|
||||||
Loading…
Add table
Add a link
Reference in a new issue