feat(s2c): debatten navigerer basen i stedet for aa faa den utlevert [skip-docs]
MAJOR-3/S7a-3 gjorde utforskningen billig og lot pipelinen staa. Maalt paa K2
(630 konsepter, S7bs eget instrument, kjent-positiv-kontrollen reprodusert
eksakt FOER bruk): okf.bundle_context er 648 962 o200k-tokens og rir i TRE
kopier = 1 947 342 = 99,1 % av en kjoerings prompt-tokens.
Et premiss i maaledokumentet ble presisert foerst: de tre kopiene er tre
DEBATT-turer (proposer x2, checker x1), mens genererings-prompten er 156
tokens, fordi gen_context = debate_output or context. Det avgjorde formen -
generering trengte ingen egen soem, for aa binde `context` binder
siste-utvei-fallbacken ved konstruksjon.
run_project sender naa en PEKER (fast tekst + erklaert bundle_id + antall
konseptdokumenter i scope + stigen, O(1) i korpuset) og gir debatten de SAMME
fire verktoeyene utforskningen bruker - explore.navigator_tools gjenbrukt,
aldri en andre kopi av policyen.
Etter: 753 tokens like-for-like (samme manus, samme fire prompter, -99,96 %)
og 8 942 med en debatt som faktisk gaar stigen (-99,5 %), mot operatoerens
terskel 195 000 = 4,6 % av taket. Validert besparelse og validatorens dom er
UENDRET (850 000 NOK av 3 852 500, 2 av 5 felt paa stage 4 og 5, samme
dom-noekkel), og utforskningens 18 355 er uendret til tokenet.
§4.1a maatte flytte, ikke forsvinne: dimensjonsfilteret bodde i renderingen og
bor naa i VERKTOEYENE, paa begge trinn - en listing som skjuler et fremmed
dokument mens read_file serverer det paa sti er et filter i navnet alene.
okf.in_dimension er eneste predikat.
Sporet er kaller-eid (ExplorationToolRecorder -> RunResult.debate_tool_calls ->
{run_id}-debate.json fra en finally) og skrives ogsaa TOMT: en debatt som
navigerer ingenting ER S2c-regresjonen, saa den maa kunne leses.
Load-bearing MAALT: aatte mutasjoner roede mot HELE suiten, groenn kontroll
1306/5 (fra 1295/5), golden demo-transcript.stdout BYTE-UENDRET
(shasum -a 1 av innholdet = ea8c534773acdbe41ae68f2c55724d69aaf8be4f).
M7 falsifiserte seg selv, ikke gaten - staar som maalt.
Maaling: docs/2026-09-04-s2c-debatt-k2.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
5d8844fef5
commit
da5f10f140
13 changed files with 1044 additions and 98 deletions
63
CLAUDE.md
63
CLAUDE.md
|
|
@ -1735,6 +1735,69 @@ Python ≥3.10. MAF (`agent-framework-core` 1.16.0, `-orchestrations` 1.1.1 —
|
||||||
dens `unkeyed_verdicts` når kun en bibliotek-kaller; stage 0 kan strukturelt ikke felle en
|
dens `unkeyed_verdicts` når kun en bibliotek-kaller; stage 0 kan strukturelt ikke felle en
|
||||||
mandat-avledet kandidat (bygget FRA baselinen — økt 82s M7-grense); prisene i syretesten er
|
mandat-avledet kandidat (bygget FRA baselinen — økt 82s M7-grense); prisene i syretesten er
|
||||||
SYNTETISKE og ingen levende modell er kalt. Måling: `docs/2026-09-04-syretest-s7b-k2.md`.
|
SYNTETISKE og ingen levende modell er kalt. Måling: `docs/2026-09-04-syretest-s7b-k2.md`.
|
||||||
|
- **Debatten NAVIGERER basen; den får den aldri utlevert — og §4.1a-filteret flyttet MED
|
||||||
|
(S2c, økt 85):** MAJOR-3/S7a-3 gjorde utforskningen billig og lot pipelinen stå. MÅLT på K2 (630
|
||||||
|
konsepter, S7bs eget instrument, kjent-positiv-kontrollen reprodusert eksakt FØR bruk):
|
||||||
|
`okf.bundle_context` er **648 962 o200k-tokens** og rir i TRE kopier = **1 947 342 = 99,1 %** av
|
||||||
|
en kjørings prompt-tokens, mot utforskningens 18 355. **Et premiss i måledokumentet ble presisert
|
||||||
|
først:** S7b skrev «2 debatt-turer + genererings-prompten», men de tre kopiene er TRE
|
||||||
|
DEBATT-TURER (proposer ×2, checker ×1) mens genererings-prompten er **156** tokens — fordi
|
||||||
|
`gen_context = debate_output or context` tar debattens output når den finnes. Dét avgjorde formen:
|
||||||
|
**generering trengte ingen egen søm**, for å binde `context` binder siste-utvei-fallbacken ved
|
||||||
|
KONSTRUKSJON, og en verktøysløyfe inne i `generate_via_llm` ville vært en andre mekanisme for et
|
||||||
|
problem den første alt løste. `run_project`s bundle-arm sender nå en **PEKER** (`_bundle_pointer`:
|
||||||
|
fast tekst + erklært `bundle_id` + antall konseptdokumenter i scope + stigen, O(1) i korpuset) og
|
||||||
|
gir debatten de SAMME fire verktøyene utforskningen bruker — `explore.navigator_tools` GJENBRUKT,
|
||||||
|
aldri en andre kopi av policyen. Etter: **753** tokens like-for-like (samme manus, samme fire
|
||||||
|
prompter, −99,96 %) og **8 942** med en debatt som faktisk går stigen (11 prompter, −99,5 %),
|
||||||
|
mot operatørens terskel 195 000 = **4,6 % av taket**. Prompt-ANTALLET stiger, og det er handelen
|
||||||
|
MAJOR-3-raden alt beskriver: man betaler per kall, men hvert resultat rir kun fra SITT kall og
|
||||||
|
framover. **§4.1a måtte flytte, ikke forsvinne:** løftet «agentene leser KUN dimensjons-matchet
|
||||||
|
kunnskap» ble holdt av `bundle_context`s filter, og holdes nå av VERKTØYENE på **begge trinn** —
|
||||||
|
en listing som skjuler et fremmed dokument mens `read_file` serverer det på sti er et filter i
|
||||||
|
navnet alene; `okf.in_dimension` er ENESTE predikat (kø-(p)), og `DimensionScopeRefused` er en
|
||||||
|
`ValueError` (`BundlePathNotFound`-presedensen: kalleren er en modell som velger en sti, så
|
||||||
|
nekten hører på CLI-ens nekt-tuppel, aldri krasj-kanalen). **Sporet er KALLER-EID**
|
||||||
|
(`ExplorationToolRecorder` på debattens middleware → `RunResult.debate_tool_calls` →
|
||||||
|
`{run_id}-debate.json` fra en `finally`): en returverdi ville vært tapt på nøyaktig den kjøringen
|
||||||
|
som trenger beviset, for et budsjettstopp midt i debatten reiser ut av `debate.run` og
|
||||||
|
konstruerer aldri et `RunResult`. **Artefaktet skrives også TOMT** — motsatt av
|
||||||
|
`write_parse_failures`, hvis TILSTEDEVÆRELSE er signalet: her ER det tomme tilfellet S2c-
|
||||||
|
regresjonen (en debatt som navigerer ingenting ser billig ut av feil grunn), så det må kunne
|
||||||
|
LESES, ikke utledes av en fil som ikke er der; MÅLT på kontroll-kjøringen med det uendrede
|
||||||
|
manuset, som etterlot `{"tool_calls": []}`. `explore.tool_call_payload` er ENESTE renderer for
|
||||||
|
begge artefakter. **`--scripted-replies` tar nå steg-lister for debattens roller:** nekten som
|
||||||
|
forsvant begrunnet seg med at «proposeren svarer `generate`s eget kall, ikke en agent-løkke som
|
||||||
|
kan kalle et verktøy mellom turer» — målbart usant siden proposer og checker ER agenter med de
|
||||||
|
fire verktøyene, og å beholde den ville holdt måleprotokollens GRATIS trinn borte fra nøyaktig
|
||||||
|
den sømmen S2c bygger (MAJOR-1s vakuitet, ett lag over). **Validert besparelse og validatorens
|
||||||
|
dom er UENDRET** — 850 000 NOK av 3 852 500, 2 av 5 felt på stage 4 og stage 5, samme dom-nøkkel
|
||||||
|
`be8535e204cdc4c6` — og utforskningens 18 355 er uendret til tokenet (`dimension` defaulter til
|
||||||
|
`None`, som er dét som gjør uendretheten til en måling). Load-bearing MÅLT
|
||||||
|
(`tests/test_debate_navigation_cost_loadbearing.py`, 11 armer), **åtte mutasjoner røde mot HELE
|
||||||
|
suiten** + grønn kontroll **1306/5** (fra 1295/5) og golden `demo-transcript.stdout`
|
||||||
|
BYTE-UENDRET (`shasum -a 1` av INNHOLDET = `ea8c534773acdbe41ae68f2c55724d69aaf8be4f`):
|
||||||
|
M1 reverter sømmen (3) · M2 verktøyene ut av `debate_tools` (7, spredt over fire filer hvorav tre
|
||||||
|
eldre gates) · M3 detach recorderen (2) · M4 skriv aldri artefaktet (5) · M5 `directory_listing`
|
||||||
|
ignorerer dimensjonen (2) · M6 detach `read_file`-gaten (2) · M7b pekeren navngir ingen base (1) ·
|
||||||
|
M8 sporet dropper `path` (3 — hvorav TO i `test_tool_call_path_loadbearing`, altså uavhengige
|
||||||
|
vitner på at rendereren er delt). **ÉN MUTASJON FALSIFISERTE SEG SELV, ikke gaten:** M7 (fjern
|
||||||
|
basens id fra pekerens overskriftslinje) var GRØNN, fordi pekeren navngir basen FIRE ganger og å
|
||||||
|
fjerne én omtale ikke fjerner egenskapen; M7b er mutasjonen som treffer den. Det står som målt.
|
||||||
|
**Tre eksisterende armer er SKREVET OM, ikke slettet:** dimensjonens to §4.1a-armer flyttet fra
|
||||||
|
prompt-teksten til listingen (den positive ville ellers blitt vakuøs — sentinelen kan aldri nå en
|
||||||
|
prompt igjen), MCP-kontrollens `captured_tools[0] == []` ble til «ingen EKSTERN tool + nøyaktig
|
||||||
|
navigatør-settet», og `test_a_step_list_is_refused_for_a_debate_role` ble sin egen positive (den
|
||||||
|
ENE omdøpte node-ID-en; suiten er ellers et strengt supersett). **Ærlighets-grenser, uttalt:**
|
||||||
|
ingen levende modell har navigert (structured-output-grensens klasse); multiplikatoren gjelder
|
||||||
|
dette manuset; et proposer-manus konsumeres PÅ NYTT av `generate_via_llm`s ferske klient, så et
|
||||||
|
manus som åpner med et verktøykall brenner genererings-forsøk (uttalt i loaderens docstring, ikke
|
||||||
|
reparert — å gjette hvilke steg som var «ment for» hvilket kallsted er reparasjon); den hostede
|
||||||
|
flaten er urørt; og **`read_file` er fortsatt sti-adresserbar til `type: verdict`-laget** — ingen
|
||||||
|
listing navngir det, men en gjettet sti når en dom utenom den gatede ExpeL-folden. Egenskapen er
|
||||||
|
ARVET fra S7a-3 og er nå nåbar fra debatten også: MÅLT, RAPPORTERT, IKKE FIKSET (én regel ett
|
||||||
|
sted, som også endrer utforskningen — egen beslutning). Måling:
|
||||||
|
`docs/2026-09-04-s2c-debatt-k2.md`.
|
||||||
- **STATE.md er local-only** (gitignored). Voyage session-state er efemert; STATE.md er kanonisk kontinuitet.
|
- **STATE.md er local-only** (gitignored). Voyage session-state er efemert; STATE.md er kanonisk kontinuitet.
|
||||||
- Prosess: Voyage-plugin (`/trekbrief → /trekplan → /trekexecute → /trekreview`) per større fase.
|
- Prosess: Voyage-plugin (`/trekbrief → /trekplan → /trekexecute → /trekreview`) per større fase.
|
||||||
|
|
||||||
|
|
|
||||||
233
docs/2026-09-04-s2c-debatt-k2.md
Normal file
233
docs/2026-09-04-s2c-debatt-k2.md
Normal file
|
|
@ -0,0 +1,233 @@
|
||||||
|
# S2c — debatten får navigatør-drevet lesing, målt på K2
|
||||||
|
|
||||||
|
**Dato:** 2026-09-04 · **Ordre:** `20260904T135452Z-328127358-from-.claude`
|
||||||
|
**Base:** `~/corpora/okf-telling-20260829/K2-bundle-20260903/` (erklært id `k2-trinn1-20260903`),
|
||||||
|
kopiert til `scratchpad/s7b-syretest/K2-priset-SYNTETISK` med MAJOR-4s syntetisk prisede prisskjema
|
||||||
|
montert — nøyaktig samme oppsett som S7b § 3.1.
|
||||||
|
|
||||||
|
> **⚠️ PRISENE ER SYNTETISKE, og ingen levende modell er kalt.** Hvert tall under er
|
||||||
|
> prompt-tokens i en SKRIPTET kjøring. Det som måles er hva rammeverket sender, ikke hva en modell
|
||||||
|
> ville svart.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 0. Sammendrag
|
||||||
|
|
||||||
|
| ordrens spørsmål | svar |
|
||||||
|
|---|---|
|
||||||
|
| Ble premisset verifisert først? | Ja — § 1, reprodusert eksakt |
|
||||||
|
| Debatt + generering på K2, FØR | **1 947 342** o200k-tokens |
|
||||||
|
| … ETTER, samme manus | **753** (−99,96 %) |
|
||||||
|
| … ETTER, med en debatt som faktisk NAVIGERER | **8 942** (−99,5 %) |
|
||||||
|
| Ordrens terskel (< 195 000) | Holdt med margin: 4,6 % av terskelen |
|
||||||
|
| Validert besparelse / validatorens dom | **UENDRET** — § 4 |
|
||||||
|
| Golden `demo-transcript.stdout` | **BYTE-UENDRET** (`shasum -a 1` = `ea8c534773acdbe41ae68f2c55724d69aaf8be4f`, INNHOLD, ikke git-blob) |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Premisset, verifisert før noe ble bygget
|
||||||
|
|
||||||
|
Ordren ba om det, og instrumentet er S7bs eget (`scratchpad/s7b-syretest/profile_pipeline.py`,
|
||||||
|
`o200k_base`, prompt lest som tekst + `function_call` + `function_result`). Den **kjent-positive
|
||||||
|
kontrollen** kjøres først: rotnivå-listingen på LEVERT K2 måles til `3954 tegn / 1495 o200k-tokens
|
||||||
|
over 629 konseptfiler`, som reproduserer S7a-3s publiserte tall eksakt. Deretter:
|
||||||
|
|
||||||
|
```
|
||||||
|
=== FASE 2: debatt + generering (skriptet, null modellkall) ===
|
||||||
|
checker 1 prompt(er) 649062 o200k-tokens
|
||||||
|
proposer 3 prompt(er) 1298280 o200k-tokens
|
||||||
|
SUM 4 prompt(er) 1947342 o200k-tokens
|
||||||
|
```
|
||||||
|
|
||||||
|
Identisk med S7b § 3.4. **Én presisering mot måledokumentet:** S7b skriver «2 debatt-turer +
|
||||||
|
genererings-prompten». Målt er fordelingen en annen — de tre kopiene er TRE DEBATT-TURER
|
||||||
|
(proposer ×2, checker ×1 à 649 062), mens genererings-prompten er 156 tokens, fordi
|
||||||
|
`gen_context = debate_output or context` tar debattens output når den finnes. Summen er den
|
||||||
|
samme, og konklusjonen er den samme; det som flytter seg er hvor generering står i regnskapet, og
|
||||||
|
det avgjorde formen på arbeidet (§ 2).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Grepet — én søm, ikke to policyer
|
||||||
|
|
||||||
|
`run_project`s bundle-arm sluttet å rendre basen inn i oppgavemeldingen. Den sender nå en
|
||||||
|
**PEKER** og gir debatten de **SAMME fire navigatør-verktøyene** utforskningen bruker
|
||||||
|
(`explore.navigator_tools` — gjenbrukt, ikke kopiert; ordrens eget krav).
|
||||||
|
|
||||||
|
* **Generering trengte ingen egen søm, og det er en MÅLING.** `gen_context = debate_output or
|
||||||
|
context`: den ene grenen er debattens output (allerede liten), den andre er `context`. Ved å
|
||||||
|
binde `context` blir siste-utvei-fallbacken bundet **ved konstruksjon** i stedet for å være hele
|
||||||
|
korpuset. En verktøysløyfe inne i `generate_via_llm` ville vært en andre mekanisme for et
|
||||||
|
problem som allerede var løst av den første.
|
||||||
|
* **Pekeren er O(1) i korpuset:** fast tekst + basens id + antall konseptdokumenter (i scope) +
|
||||||
|
stigen. Taket som vokter den bor i TESTEN (`_CEILING_CHARS = 1 500`), av `read_bundle`-takets
|
||||||
|
grunn: et tak importert fra implementasjonen flytter seg med den.
|
||||||
|
* **§4.1a-dimensjonen måtte FLYTTE, ikke forsvinne.** Løftet var at agentene leser KUN
|
||||||
|
dimensjons-matchet kunnskap. Det ble holdt av `bundle_context`s filter; med navigasjon må det
|
||||||
|
holdes av VERKTØYENE — og på **begge trinn**: en listing som skjuler et fremmed dokument mens
|
||||||
|
`read_file` fortsatt serverer det på sti er et filter i navnet alene. Én predikat
|
||||||
|
(`okf.in_dimension`) tjener begge renderere, så to kopier av «hva er i scope» ikke kan bli uenige
|
||||||
|
om samme base.
|
||||||
|
* **Sporet er KALLER-EID** (`ExplorationToolRecorder` på debattens middleware,
|
||||||
|
`RunResult.debate_tool_calls`, `{run_id}-debate.json` skrevet fra en `finally`). En returverdi
|
||||||
|
ville vært tapt på nøyaktig den kjøringen som trenger beviset: et budsjettstopp midt i debatten
|
||||||
|
reiser ut av `debate.run` og konstruerer aldri et `RunResult`.
|
||||||
|
**Artefaktet skrives også når det er TOMT** — ulikt `write_parse_failures`, hvis TILSTEDEVÆRELSE
|
||||||
|
er signalet. Her er det tomme tilfellet selve S2c-regresjonen (en debatt som navigerer ingenting
|
||||||
|
ser billig ut av feil grunn), så det må kunne LESES, ikke utledes av en fil som ikke er der.
|
||||||
|
* **Én renderer for sporet** (`explore.tool_call_payload`), delt av utforskningens og debattens
|
||||||
|
artefakt — målt, ikke påstått: M8 under gjør TO eksisterende tester i
|
||||||
|
`test_tool_call_path_loadbearing` røde.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Tokens per fase, FØR og ETTER
|
||||||
|
|
||||||
|
Samme instrument, samme base, samme syntetiske prisfikstur.
|
||||||
|
|
||||||
|
### 3.1 Like-for-like — identisk manus, identiske fire prompter
|
||||||
|
|
||||||
|
| fase | prompter | FØR | ETTER | endring |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| 1 — utforskning (navigatør 5, manager 6, hypotesiser 1) | 12 | 18 355 | **18 355** | uendret |
|
||||||
|
| 2 — debatt + generering (proposer 3, checker 1) | 4 | 1 947 342 | **753** | **−99,96 %** |
|
||||||
|
| 3 — deterministisk dom | 0 | 0 | 0 | — |
|
||||||
|
| **sum** | **16** | **1 965 697** | **19 108** | **−99,0 %** |
|
||||||
|
|
||||||
|
Utforskningens 18 355 er uendret til tokenet, med samme fire verktøykall i samme rekkefølge —
|
||||||
|
`navigator_tools` fikk en `dimension`-parameter som defaulter til `None`, og det er dét som gjør
|
||||||
|
den uendretheten til en måling og ikke en forhåpning.
|
||||||
|
|
||||||
|
### 3.2 Med en debatt som faktisk LESER — det ærlige tallet
|
||||||
|
|
||||||
|
753 tokens er billig delvis fordi det skriptede manuset fra S7b aldri kaller et verktøy. En
|
||||||
|
gjenbrukt konstant-svar-rehearsal beviser at debatten KJØRER, aldri at den ÅPNER basen — nøyaktig
|
||||||
|
vakuiteten MAJOR-1 lukket ett lag over. Manuset ble derfor utvidet slik at debatten går den samme
|
||||||
|
stigen navigatøren gikk i S7b § 3.5:
|
||||||
|
|
||||||
|
```
|
||||||
|
verktoykall debatten gjorde (RunResult.debate_tool_calls):
|
||||||
|
read_bundle bundle_id=k2-trinn1-20260903 path=-
|
||||||
|
read_dir bundle_id=k2-trinn1-20260903 path=del-ii-bilag-7-prisskjema
|
||||||
|
read_file bundle_id=k2-trinn1-20260903 path=del-ii-bilag-7-prisskjema/prisskjema-SYNTETISK.md
|
||||||
|
read_file bundle_id=k2-trinn1-20260903 path=del-ii-bilag-7-prisskjema/prisskjema-SYNTETISK.md
|
||||||
|
checker 2 prompt(er) 633 o200k-tokens
|
||||||
|
proposer 9 prompt(er) 8309 o200k-tokens
|
||||||
|
SUM 11 prompt(er) 8942 o200k-tokens
|
||||||
|
```
|
||||||
|
|
||||||
|
**8 942 tokens, −99,5 %**, mot ordrens terskel på 195 000: **4,6 % av taket**. Antall prompter
|
||||||
|
STIGER (4 → 11), og det er handelen MAJOR-3-raden allerede beskriver: man betaler per kall, men
|
||||||
|
hvert resultat rir kun fra SITT kall og framover, i stedet for at hele basen rir i hver prompt.
|
||||||
|
|
||||||
|
Tallet er dessuten et **øvre estimat**: proposer-manuset konsumeres på nytt av den ferske klienten
|
||||||
|
`generate_via_llm` bygger, så tre genererings-forsøk brenner på et verktøykall før JSON-en
|
||||||
|
kommer (§ 6, uttalt ærlighets-grense). En operatør som skriver manuset riktig betaler mindre.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Validert besparelse og validatorens dom — UENDRET
|
||||||
|
|
||||||
|
Ordren sier at en endring her er et funn, ikke støy. Målt med SAMME kommando og SAMME
|
||||||
|
`scripted-replies.json` som S7b § 3.2:
|
||||||
|
|
||||||
|
```
|
||||||
|
K2: ValidatedProposal (no expert verdict given; verdict key=be8535e204cdc4c6)
|
||||||
|
hypothesis-1 VALIDATED 200000 NOK
|
||||||
|
own-proposal VALIDATED 200000 NOK
|
||||||
|
Validated: 2 of 2 approaches — best 200000 NOK
|
||||||
|
```
|
||||||
|
|
||||||
|
Identisk med S7b, ned til dom-nøkkelen. Den deterministiske døra (§ 3.3 der) er også uendret:
|
||||||
|
|
||||||
|
| id | krav | utfall | stage |
|
||||||
|
|---|---|---|---|
|
||||||
|
| a1 | 200 000 | **VALIDERT** | — |
|
||||||
|
| a2 | 900 000 | FELT | stage 4 — P90 feasible 612 000 |
|
||||||
|
| a3 | 150 000 | **VALIDERT** | — |
|
||||||
|
| a4 | 500 000 | **VALIDERT** | — |
|
||||||
|
| a5 | 150 000 | FELT | stage 5 — `energy_efficiency`-cap 112 500 |
|
||||||
|
|
||||||
|
**850 000 NOK validert av 3 852 500. 2 av 5 felt, på to ulike stages.** Uendret.
|
||||||
|
|
||||||
|
Kontroll-kjøringen med det UENDREDE manuset etterlot `{"tool_calls": []}` — den skriptede
|
||||||
|
konstanten åpnet ingenting, og artefaktet sier det i stedet for å mangle. Det er nettopp derfor
|
||||||
|
det tomme tilfellet skrives.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Mutasjonsmålingen
|
||||||
|
|
||||||
|
Grønn kontroll **1306 passed / 5 skipped** (fra 1295/5), golden `demo-transcript.stdout`
|
||||||
|
BYTE-UENDRET. Hver mutasjon kjørt mot HELE suiten, restaurert fra scratchpad og verifisert med
|
||||||
|
`shasum -c`.
|
||||||
|
|
||||||
|
| # | detach | røde |
|
||||||
|
|---|---|---|
|
||||||
|
| M1 | `context` tilbake til `okf.bundle_context` | 3 |
|
||||||
|
| M2 | navigatør-verktøyene ut av `debate_tools` | 7 |
|
||||||
|
| M3 | `ExplorationToolRecorder` av debattens middleware | 2 |
|
||||||
|
| M4 | `{run_id}-debate.json` skrives ikke | 5 |
|
||||||
|
| M5 | `directory_listing` ignorerer dimensjonen | 2 |
|
||||||
|
| M6 | `read_file`s dimensjonsgate detached | 2 |
|
||||||
|
| M7 | pekeren dropper basens id i overskriftslinja | **0 — se under** |
|
||||||
|
| M7b | pekeren navngir ingen base overhodet | 1 |
|
||||||
|
| M8 | sporet dropper `path` | 3 |
|
||||||
|
|
||||||
|
**M7 FALSIFISERTE MUTASJONEN, ikke gaten.** Pekeren navngir basen FIRE ganger (overskriften pluss
|
||||||
|
de tre stige-kallene), så å fjerne én omtale fjerner ikke egenskapen gaten påstår («en debatt som
|
||||||
|
kan kalle verktøyene sine»). M7b er mutasjonen som faktisk løsner den, og den er rød alene. Dette
|
||||||
|
står som målt, ikke som spådd — en mutasjon som ikke treffer egenskapen er en opplysning om
|
||||||
|
mutasjonen.
|
||||||
|
|
||||||
|
**M8 er den sterkeste av dem:** to av de tre røde bor i `test_tool_call_path_loadbearing`, altså i
|
||||||
|
tester som fantes FØR dette arbeidet. Det er det som gjør «én renderer, ikke to kopier» til en
|
||||||
|
måling.
|
||||||
|
|
||||||
|
**M2s syv røde** spenner fire filer, hvorav tre er eldre gates — dimensjonens to armer,
|
||||||
|
MCP-kontrollen og den skriptede døra. Sømmen er altså vitnet fra flere uavhengige hold.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Ærlighets-grenser, uttalt
|
||||||
|
|
||||||
|
* **Ingen levende modell.** At en modell NAVIGERER godt med en peker i stedet for hele basen er
|
||||||
|
ikke bevist — samme klasse som structured-output-grensen. Det som er bevist er hva rammeverket
|
||||||
|
sender.
|
||||||
|
* **Dette er ikke «−99,5 % for enhver kjøring».** En debatt som åpner *k* dokumenter betaler *k*
|
||||||
|
verktøyresultater; gevinsten er at den betaler for det den VALGTE, og at hvert resultat rir fra
|
||||||
|
sitt eget kall og framover. Multiplikatoren gjelder dette manuset.
|
||||||
|
* **Det skriptede proposer-manuset konsumeres to ganger.** Manus er per KLIENT, og
|
||||||
|
`generate_via_llm` bygger en fersk proposer-klient som starter manuset på nytt. Et manus som
|
||||||
|
åpner med et verktøykall svarer derfor også genererings-kallet med et `function_call`, som ikke
|
||||||
|
parser. Det er uttalt i `_load_scripted_replies`' docstring og ikke reparert: å gjette hvilke
|
||||||
|
steg som var «ment for» hvilket kallsted er reparasjon, og denne loaderen validerer.
|
||||||
|
* **`read_file` er fortsatt sti-adresserbar til `type: verdict`-laget.** Ingen listing navngir det
|
||||||
|
(`context_files` dropper det ved konstruksjon), men en modell som gjetter en sti kan lese en
|
||||||
|
dom utenom den gatede ExpeL-folden. Egenskapen er ARVET fra utforskningen (S7a-3) og er derfor
|
||||||
|
ikke ny her, men den er nå nåbar fra debatten også. **MÅLT, RAPPORTERT, IKKE FIKSET** — å lukke
|
||||||
|
den er én regel ett sted (`read_file` nekter det ingen listing ville vist), og den endrer
|
||||||
|
utforskningens oppførsel også, altså en egen beslutning.
|
||||||
|
* **`--scripted-replies` tar nå steg-lister for debattens roller.** Nekten som forsvant hadde en
|
||||||
|
begrunnelse som nå er målbart usann (proposer og checker ER agenter med verktøy). Arm
|
||||||
|
`test_a_step_list_is_refused_for_a_debate_role` er SKREVET OM til sin positive, ikke slettet —
|
||||||
|
node-ID-en er derfor omdøpt, og suiten er et supersett bortsett fra det ene navnet.
|
||||||
|
* **Den hostede flaten er urørt.** Ingen nye felt i whitelisten; `{run_id}-debate.json` når kun
|
||||||
|
CLI-ens/bibliotekets outbox, som `{run_id}-exploration.json` alt gjør.
|
||||||
|
* **Prefiks-caching er fortsatt NOTERT, ikke bygget.** Med korpuset ute av prompten er den også
|
||||||
|
langt mindre verdt enn den var.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Kommandoer
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# premiss + ETTER, like-for-like (samme manus)
|
||||||
|
uv run --with tiktoken==0.12.0 python scratchpad/s7b-syretest/profile_pipeline.py
|
||||||
|
# ETTER med en navigerende debatt
|
||||||
|
uv run --with tiktoken==0.12.0 python scratchpad/s2c-debatt/profile_pipeline_after.py
|
||||||
|
# fase 1 + 3 (uendret)
|
||||||
|
uv run --with tiktoken==0.12.0 python scratchpad/s7b-syretest/profile_phases.py
|
||||||
|
# mutasjon nr. N mot hele suiten
|
||||||
|
uv run python scratchpad/s2c-debatt/mut.py M1
|
||||||
|
```
|
||||||
|
|
@ -214,6 +214,17 @@ class ExplorationError(RuntimeError):
|
||||||
"""The exploration cannot be honoured as configured, or produced something unreadable."""
|
"""The exploration cannot be honoured as configured, or produced something unreadable."""
|
||||||
|
|
||||||
|
|
||||||
|
class DimensionScopeRefused(ValueError):
|
||||||
|
"""A navigator asked for a document belonging to ANOTHER dimension than the run is scoped to.
|
||||||
|
|
||||||
|
A ``ValueError``, the ``BundlePathNotFound``/``BundleIdMismatch`` precedent: the caller is a
|
||||||
|
model choosing a path, so the refusal must land on the CLI's refusal tuple and hosting's 400
|
||||||
|
arm rather than the crash channel. It is deliberately NOT an ``ExplorationError``
|
||||||
|
(a ``RuntimeError``): this is a refused read inside a run that is otherwise fine, not an
|
||||||
|
exploration that cannot be honoured as configured.
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
class LedgerEntry:
|
class LedgerEntry:
|
||||||
"""One progress-ledger round, reduced to the five fields the manager steers on (C.2).
|
"""One progress-ledger round, reduced to the five fields the manager steers on (C.2).
|
||||||
|
|
@ -375,6 +386,20 @@ class ExplorationTrace:
|
||||||
tokens_spent: int = 0
|
tokens_spent: int = 0
|
||||||
|
|
||||||
|
|
||||||
|
def tool_call_payload(calls: Sequence[ToolCall]) -> list[dict[str, Any]]:
|
||||||
|
"""The ONE rendering of a tool trace into plain data, in CALL ORDER.
|
||||||
|
|
||||||
|
Two surfaces now write one: ``trace_payload`` for ``{run_id}-exploration.json`` and
|
||||||
|
``run_project`` for ``{run_id}-debate.json`` (S2c). Two copies of "what a recorded call looks
|
||||||
|
like" would drift into two answers about the same fact, which is the kø-(p) defect — and here
|
||||||
|
the drift would land in the artefacts an operator reads to find out what a paid run opened.
|
||||||
|
|
||||||
|
Plain mappings only, so the RAW output layer stays MAF-free (``outbox.py`` may not import
|
||||||
|
this module).
|
||||||
|
"""
|
||||||
|
return [{"name": call.name, "bundle_id": call.bundle_id, "path": call.path} for call in calls]
|
||||||
|
|
||||||
|
|
||||||
def trace_payload(
|
def trace_payload(
|
||||||
trace: ExplorationTrace, *, stop: str | None, completed: bool, mandate: Mandate | None
|
trace: ExplorationTrace, *, stop: str | None, completed: bool, mandate: Mandate | None
|
||||||
) -> dict[str, Any]:
|
) -> dict[str, Any]:
|
||||||
|
|
@ -442,10 +467,7 @@ def trace_payload(
|
||||||
}
|
}
|
||||||
for call in trace.quick_validations
|
for call in trace.quick_validations
|
||||||
],
|
],
|
||||||
"tool_calls": [
|
"tool_calls": tool_call_payload(trace.tool_calls),
|
||||||
{"name": call.name, "bundle_id": call.bundle_id, "path": call.path}
|
|
||||||
for call in trace.tool_calls
|
|
||||||
],
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -859,7 +881,9 @@ def _index_excerpt(body: str) -> tuple[str, bool]:
|
||||||
return (body[:cut] if cut > 0 else body[:_CATALOGUE_EXCERPT_CHARS]), True
|
return (body[:cut] if cut > 0 else body[:_CATALOGUE_EXCERPT_CHARS]), True
|
||||||
|
|
||||||
|
|
||||||
def navigator_tools(bundle_dirs: Sequence[str]) -> list[FunctionTool]:
|
def navigator_tools(
|
||||||
|
bundle_dirs: Sequence[str], *, dimension: str | None = None
|
||||||
|
) -> list[FunctionTool]:
|
||||||
"""The navigator's three tools: survey the catalogue, open one base, read one document.
|
"""The navigator's three tools: survey the catalogue, open one base, read one document.
|
||||||
|
|
||||||
Progressive disclosure, not stuffing (målbilde §2/§4): ``list_bundles`` never returns content,
|
Progressive disclosure, not stuffing (målbilde §2/§4): ``list_bundles`` never returns content,
|
||||||
|
|
@ -898,6 +922,13 @@ def navigator_tools(bundle_dirs: Sequence[str]) -> list[FunctionTool]:
|
||||||
The listing is built from ``Bundle.context_files``, which EXCLUDES the ``type: verdict`` layer
|
The listing is built from ``Bundle.context_files``, which EXCLUDES the ``type: verdict`` layer
|
||||||
by construction — prior verdicts reach a hypothesis only through the gated ExpeL fold inside
|
by construction — prior verdicts reach a hypothesis only through the gated ExpeL fold inside
|
||||||
``run_project``, never by being read as context here.
|
``run_project``, never by being read as context here.
|
||||||
|
|
||||||
|
**``dimension`` scopes BOTH rungs, and both halves are the promise** (§4.1a, carried over in
|
||||||
|
S2c when the DEBATE started navigating instead of being handed ``bundle_context``). The filter
|
||||||
|
used to live in the rendering; with navigation it has to live in the tools, and a listing that
|
||||||
|
hides a foreign-dimension document while ``read_file`` still serves it by path is a filter in
|
||||||
|
name only — a model-chosen path is untrusted input, so the gate belongs where the bytes leave.
|
||||||
|
``None`` (the exploration's own call) admits everything, byte-identical to before.
|
||||||
"""
|
"""
|
||||||
index = _bundle_index(bundle_dirs)
|
index = _bundle_index(bundle_dirs)
|
||||||
|
|
||||||
|
|
@ -921,7 +952,9 @@ def navigator_tools(bundle_dirs: Sequence[str]) -> list[FunctionTool]:
|
||||||
"id": bundle_id,
|
"id": bundle_id,
|
||||||
"index_excerpt": excerpt,
|
"index_excerpt": excerpt,
|
||||||
"index_truncated": truncated,
|
"index_truncated": truncated,
|
||||||
"documents": len(bundle.context_files),
|
"documents": sum(
|
||||||
|
1 for f in bundle.context_files if okf.in_dimension(f, dimension)
|
||||||
|
),
|
||||||
"verdict_count": len(bundle.verdicts),
|
"verdict_count": len(bundle.verdicts),
|
||||||
# Tolerant on CONTENT, fail-fast on the PATH: an operator's bad directory is
|
# Tolerant on CONTENT, fail-fast on the PATH: an operator's bad directory is
|
||||||
# refused by navigate_bundle above, while a navigable base that simply has no
|
# refused by navigate_bundle above, while a navigable base that simply has no
|
||||||
|
|
@ -953,7 +986,7 @@ def navigator_tools(bundle_dirs: Sequence[str]) -> list[FunctionTool]:
|
||||||
# ONE renderer for both rungs (kø-(p)): this tool and ``read_dir`` differ only in WHICH
|
# ONE renderer for both rungs (kø-(p)): this tool and ``read_dir`` differ only in WHICH
|
||||||
# level they ask for, and two copies of a listing rule would drift into two answers about
|
# level they ask for, and two copies of a listing rule would drift into two answers about
|
||||||
# one bundle. ``okf`` owns it, so the context seam stays framework-neutral.
|
# one bundle. ``okf`` owns it, so the context seam stays framework-neutral.
|
||||||
return okf.directory_listing(bundle)
|
return okf.directory_listing(bundle, dimension=dimension)
|
||||||
|
|
||||||
@tool(
|
@tool(
|
||||||
name="read_dir",
|
name="read_dir",
|
||||||
|
|
@ -968,7 +1001,7 @@ def navigator_tools(bundle_dirs: Sequence[str]) -> list[FunctionTool]:
|
||||||
bundle_dir = _resolve_bundle(index, bundle_id)
|
bundle_dir = _resolve_bundle(index, bundle_id)
|
||||||
bundle = okf.navigate_bundle(bundle_dir)
|
bundle = okf.navigate_bundle(bundle_dir)
|
||||||
okf.assert_declared_ids_agree(bundle)
|
okf.assert_declared_ids_agree(bundle)
|
||||||
return okf.directory_listing(bundle, path)
|
return okf.directory_listing(bundle, path, dimension=dimension)
|
||||||
|
|
||||||
@tool(
|
@tool(
|
||||||
name="read_file",
|
name="read_file",
|
||||||
|
|
@ -979,7 +1012,30 @@ def navigator_tools(bundle_dirs: Sequence[str]) -> list[FunctionTool]:
|
||||||
# safe_resolve is the ONE in-/out-of-bundle test in this repo, and it is fail-closed. A
|
# safe_resolve is the ONE in-/out-of-bundle test in this repo, and it is fail-closed. A
|
||||||
# model-chosen path is untrusted input by definition, so it goes through the same gate the
|
# model-chosen path is untrusted input by definition, so it goes through the same gate the
|
||||||
# navigation walk uses rather than a second, laxer check.
|
# navigation walk uses rather than a second, laxer check.
|
||||||
return Path(safe_resolve(bundle_dir, path)).read_text(encoding="utf-8")
|
resolved = Path(safe_resolve(bundle_dir, path))
|
||||||
|
if dimension is not None:
|
||||||
|
# The SECOND half of the scope (§4.1a). A listing that hides a document while this rung
|
||||||
|
# still serves it by path is a filter in name only, and the caller here is a model that
|
||||||
|
# can name a path no listing gave it. Only a NAVIGATED concept file is judged: the walk
|
||||||
|
# is what knows a file's declared dimension, and a path outside it is already refused —
|
||||||
|
# or, for ``index.md``, is navigation rather than scoped knowledge.
|
||||||
|
bundle = okf.navigate_bundle(bundle_dir)
|
||||||
|
foreign = next(
|
||||||
|
(
|
||||||
|
f
|
||||||
|
for f in bundle.context_files
|
||||||
|
if Path(safe_resolve(bundle_dir, f.name)) == resolved
|
||||||
|
and not okf.in_dimension(f, dimension)
|
||||||
|
),
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
if foreign is not None:
|
||||||
|
raise DimensionScopeRefused(
|
||||||
|
f"document {path!r} in knowledge base {bundle_id!r} declares dimension "
|
||||||
|
f"{foreign.frontmatter.get('dimension')!r}; this run is scoped to "
|
||||||
|
f"{dimension!r} and reads only knowledge in scope"
|
||||||
|
)
|
||||||
|
return resolved.read_text(encoding="utf-8")
|
||||||
|
|
||||||
return [list_bundles, read_bundle, read_dir, read_file]
|
return [list_bundles, read_bundle, read_dir, read_file]
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1015,6 +1015,25 @@ def assert_declared_ids_agree(bundle: Bundle) -> None:
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def in_dimension(file: BundleFile, dimension: str | None) -> bool:
|
||||||
|
"""Whether a concept file belongs to a run scoped to ``dimension`` (§4.1a).
|
||||||
|
|
||||||
|
ONE copy of the rule (kø-(p)), because two renderings now answer it: ``bundle_context`` renders
|
||||||
|
the matched bodies, and ``directory_listing`` lists the matched documents for a navigator that
|
||||||
|
reads them one at a time. Two copies of "which knowledge is in scope" would be free to disagree
|
||||||
|
about the same base, and the disagreement would show up as an agent being shown a document it
|
||||||
|
is then refused.
|
||||||
|
|
||||||
|
``dimension=None`` admits everything (byte-identical to the unscoped rendering), and a file
|
||||||
|
carrying NO ``dimension`` is never dropped: un-scoped knowledge belongs to every scope, which
|
||||||
|
is what makes a method or a cost reference usable across dimensions.
|
||||||
|
"""
|
||||||
|
if dimension is None:
|
||||||
|
return True
|
||||||
|
declared = file.frontmatter.get("dimension")
|
||||||
|
return declared is None or declared == dimension
|
||||||
|
|
||||||
|
|
||||||
class BundlePathNotFound(ValueError):
|
class BundlePathNotFound(ValueError):
|
||||||
"""A listing was asked for a directory the navigated bundle does not have.
|
"""A listing was asked for a directory the navigated bundle does not have.
|
||||||
|
|
||||||
|
|
@ -1025,7 +1044,9 @@ class BundlePathNotFound(ValueError):
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
|
||||||
def directory_listing(bundle: Bundle, path: str = "") -> dict[str, Any]:
|
def directory_listing(
|
||||||
|
bundle: Bundle, path: str = "", *, dimension: str | None = None
|
||||||
|
) -> dict[str, Any]:
|
||||||
"""One LEVEL of a navigated bundle: the subdirectories under ``path`` with what each holds, and
|
"""One LEVEL of a navigated bundle: the subdirectories under ``path`` with what each holds, and
|
||||||
the concept documents that sit directly in it.
|
the concept documents that sit directly in it.
|
||||||
|
|
||||||
|
|
@ -1049,6 +1070,12 @@ def directory_listing(bundle: Bundle, path: str = "") -> dict[str, Any]:
|
||||||
relative name would have to be composed by the caller, and the caller is a model: a path that
|
relative name would have to be composed by the caller, and the caller is a model: a path that
|
||||||
never existed is worse than no path (``_index_excerpt``'s rule, one rung up).
|
never existed is worse than no path (``_index_excerpt``'s rule, one rung up).
|
||||||
|
|
||||||
|
``dimension`` scopes the listing exactly as it scopes ``bundle_context`` — ONE predicate
|
||||||
|
(``in_dimension``) serves both, so a navigator is never shown a document the run would then
|
||||||
|
refuse to open. Under a scope a directory holding only foreign-dimension documents raises
|
||||||
|
``BundlePathNotFound`` like any other unknown path: within this run it holds nothing, and
|
||||||
|
answering with an empty listing is the very confusion the refusal exists to prevent.
|
||||||
|
|
||||||
``documents`` on a directory entry is the count of concept documents in its whole SUBTREE — what
|
``documents`` on a directory entry is the count of concept documents in its whole SUBTREE — what
|
||||||
the subtree holds, not what one ``read_dir`` on it returns. It is the price signal a navigator
|
the subtree holds, not what one ``read_dir`` on it returns. It is the price signal a navigator
|
||||||
chooses against, and the tool description says which of the two it is rather than leaving the
|
chooses against, and the tool description says which of the two it is rather than leaving the
|
||||||
|
|
@ -1060,7 +1087,7 @@ def directory_listing(bundle: Bundle, path: str = "") -> dict[str, Any]:
|
||||||
directories: dict[str, int] = {}
|
directories: dict[str, int] = {}
|
||||||
documents: list[dict[str, Any]] = []
|
documents: list[dict[str, Any]] = []
|
||||||
for f in bundle.context_files:
|
for f in bundle.context_files:
|
||||||
if not f.name.startswith(prefix):
|
if not f.name.startswith(prefix) or not in_dimension(f, dimension):
|
||||||
continue
|
continue
|
||||||
rest = f.name[len(prefix) :]
|
rest = f.name[len(prefix) :]
|
||||||
head, sep, _ = rest.partition("/")
|
head, sep, _ = rest.partition("/")
|
||||||
|
|
@ -1113,10 +1140,8 @@ def bundle_context(bundle: Bundle, *, dimension: str | None = None) -> str:
|
||||||
against — see ``tests/test_okf.py`` nav-golden gates."""
|
against — see ``tests/test_okf.py`` nav-golden gates."""
|
||||||
sections = [bundle.index_summary.strip("\n")]
|
sections = [bundle.index_summary.strip("\n")]
|
||||||
for f in bundle.context_files:
|
for f in bundle.context_files:
|
||||||
if dimension is not None:
|
if not in_dimension(f, dimension):
|
||||||
file_dim = f.frontmatter.get("dimension")
|
continue
|
||||||
if file_dim is not None and file_dim != dimension:
|
|
||||||
continue
|
|
||||||
title = unquote_scalar(f.frontmatter.get("title", f.name))
|
title = unquote_scalar(f.frontmatter.get("title", f.name))
|
||||||
body = f.body.strip("\n")
|
body = f.body.strip("\n")
|
||||||
sections.append(f"## {f.type or 'document'}: {title}\n\n{body}")
|
sections.append(f"## {f.type or 'document'}: {title}\n\n{body}")
|
||||||
|
|
|
||||||
|
|
@ -189,6 +189,36 @@ def write_exploration(
|
||||||
return path
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
def write_debate_tools(
|
||||||
|
outbox_dir: str,
|
||||||
|
run_id: str,
|
||||||
|
*,
|
||||||
|
tool_calls: Sequence[Mapping[str, Any]],
|
||||||
|
) -> Path:
|
||||||
|
"""Write ``{run_id}-debate.json`` — WHICH documents the debate opened, in call order (S2c).
|
||||||
|
|
||||||
|
The sibling of ``write_exploration`` one phase over. Since S2c the debate navigates the
|
||||||
|
knowledge base instead of being handed it whole, so "what did this run actually read" is a
|
||||||
|
question about the debate too, and over a 630-concept corpus it is not answerable from a
|
||||||
|
prompt log: the whole point is that the prompts no longer carry the base.
|
||||||
|
|
||||||
|
Written on EVERY run that has an outbox, including one whose agents opened nothing — unlike
|
||||||
|
``write_parse_failures``, whose presence IS its signal. Here the empty case is the S2c
|
||||||
|
regression itself (a debate that navigates nothing looks exactly like a cheap one), so it must
|
||||||
|
be readable off the artefact rather than inferred from a file that is not there.
|
||||||
|
|
||||||
|
Takes already-rendered plain mappings (``explore.tool_call_payload``) so the RAW output layer
|
||||||
|
stays MAF-free — ``write_exploration``'s own rule, same reason."""
|
||||||
|
directory = Path(outbox_dir)
|
||||||
|
directory.mkdir(parents=True, exist_ok=True)
|
||||||
|
path = directory / f"{run_id}-debate.json"
|
||||||
|
path.write_text(
|
||||||
|
_dump({"run_id": run_id, "tool_calls": [dict(call) for call in tool_calls]}),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
def write_plan_review(
|
def write_plan_review(
|
||||||
outbox_dir: str,
|
outbox_dir: str,
|
||||||
run_id: str,
|
run_id: str,
|
||||||
|
|
|
||||||
|
|
@ -60,18 +60,22 @@ from portfolio_optimiser.explore import (
|
||||||
NAVIGATOR_ROLE,
|
NAVIGATOR_ROLE,
|
||||||
ExplorationContract,
|
ExplorationContract,
|
||||||
ExplorationResult,
|
ExplorationResult,
|
||||||
|
ExplorationToolRecorder,
|
||||||
ExplorationTrace,
|
ExplorationTrace,
|
||||||
ParkedStateError,
|
ParkedStateError,
|
||||||
PlanReviewDecision,
|
PlanReviewDecision,
|
||||||
PlanReviewParked,
|
PlanReviewParked,
|
||||||
|
ToolCall,
|
||||||
explore,
|
explore,
|
||||||
exploration_notice,
|
exploration_notice,
|
||||||
load_exploration_contract,
|
load_exploration_contract,
|
||||||
load_parked,
|
load_parked,
|
||||||
parked_notice,
|
parked_notice,
|
||||||
parked_payload,
|
parked_payload,
|
||||||
|
navigator_tools,
|
||||||
resume_exploration,
|
resume_exploration,
|
||||||
terminal_plan_reviewer,
|
terminal_plan_reviewer,
|
||||||
|
tool_call_payload,
|
||||||
trace_payload,
|
trace_payload,
|
||||||
)
|
)
|
||||||
from portfolio_optimiser.generate import ParseFailure, generate_via_llm
|
from portfolio_optimiser.generate import ParseFailure, generate_via_llm
|
||||||
|
|
@ -199,6 +203,16 @@ class RunResult:
|
||||||
#: ever report zero — unlike ``cost_baseline_anchored`` and ``skipped_links``, both resolved
|
#: ever report zero — unlike ``cost_baseline_anchored`` and ``skipped_links``, both resolved
|
||||||
#: above that cut.
|
#: above that cut.
|
||||||
unkeyed_verdicts: int = 0
|
unkeyed_verdicts: int = 0
|
||||||
|
#: Which documents the DEBATE opened, in call order (S2c). Since the debate navigates the base
|
||||||
|
#: instead of being handed ``bundle_context``, "what did this run read" is no longer answerable
|
||||||
|
#: from the prompts — that is the whole saving — so the run carries the trace itself.
|
||||||
|
#:
|
||||||
|
#: EMPTY is an honest POSITIVE statement ("the debate opened nothing"), which is why it
|
||||||
|
#: defaults, exactly as ``skipped_links`` does; and it is also the S2c regression signal, which
|
||||||
|
#: is why the outbox artefact is written even when it is empty rather than only on activity.
|
||||||
|
#: The RESULT of each call is deliberately absent — that is the base's content, i.e. the very
|
||||||
|
#: thing too big to ride along (``ToolCall``'s own rule, MAJOR-1).
|
||||||
|
debate_tool_calls: tuple[ToolCall, ...] = ()
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def verdict_key(self) -> str:
|
def verdict_key(self) -> str:
|
||||||
|
|
@ -500,6 +514,38 @@ def _authored_texts(result: Any, name: str) -> list[str]:
|
||||||
return texts
|
return texts
|
||||||
|
|
||||||
|
|
||||||
|
#: The pointer's shape is fixed text plus the base id, its document count and (when scoped) the
|
||||||
|
#: dimension — O(1) in the corpus by construction. The ceiling that guards it lives in the TEST
|
||||||
|
#: (``test_debate_navigation_cost_loadbearing``), for ``_CATALOGUE_EXCERPT_CHARS``' reason: a bound
|
||||||
|
#: imported from the implementation moves with it, and widening this is the regression the gate
|
||||||
|
#: exists to catch.
|
||||||
|
def _bundle_pointer(bundle: okf.Bundle, bundle_id: str, *, dimension: str | None = None) -> str:
|
||||||
|
"""What the debate is told about the knowledge base INSTEAD of being given it (S2c).
|
||||||
|
|
||||||
|
It must do exactly two things: NAME the base by the id the tools take — a bounded prompt that
|
||||||
|
omits it is a debate that cannot make a single call, which is this seam's vacuous form — and
|
||||||
|
say the ladder exists. It carries no content: the whole point is that the corpus is read on
|
||||||
|
demand and each result rides only from the call that asked for it.
|
||||||
|
|
||||||
|
The document COUNT is the price signal a proposer chooses against (``directory_listing``'s own
|
||||||
|
``chars`` rule, one rung up), and it is the count IN SCOPE: under a dimension, advertising
|
||||||
|
documents the tools will then refuse would be a number that describes a different run.
|
||||||
|
"""
|
||||||
|
documents = sum(1 for f in bundle.context_files if okf.in_dimension(f, dimension))
|
||||||
|
scope = (
|
||||||
|
f" Scope: dimension {dimension!r} — only knowledge in that scope is readable."
|
||||||
|
if dimension
|
||||||
|
else ""
|
||||||
|
)
|
||||||
|
return (
|
||||||
|
f"Knowledge base: {bundle_id} ({documents} concept documents)."
|
||||||
|
f"{scope}\n"
|
||||||
|
"It is NOT included here — read it with your tools: list_bundles() for the bases, "
|
||||||
|
f"read_bundle({bundle_id!r}) for its top level, read_dir({bundle_id!r}, path) for one "
|
||||||
|
f"directory, read_file({bundle_id!r}, path) for one document. Open what you need."
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _debate_text(result: Any) -> str:
|
def _debate_text(result: Any) -> str:
|
||||||
"""The PROPOSER's converged output (fed into generation, F1). With ``output_from=agents`` both
|
"""The PROPOSER's converged output (fed into generation, F1). With ``output_from=agents`` both
|
||||||
participants surface, so we select proposer-authored outputs specifically — taking the last of
|
participants surface, so we select proposer-authored outputs specifically — taking the last of
|
||||||
|
|
@ -844,8 +890,10 @@ async def run_project(
|
||||||
# 2-3. Project + agent read-context + first-class citations. A bundle run derives ALL THREE from
|
# 2-3. Project + agent read-context + first-class citations. A bundle run derives ALL THREE from
|
||||||
# the navigated OKF bundle via progressive disclosure (verdict layer EXCLUDED — målbilde §2/§4),
|
# the navigated OKF bundle via progressive disclosure (verdict layer EXCLUDED — målbilde §2/§4),
|
||||||
# NOT keyword chunk-stuffing; the road path keeps the chunk-retrieval data source. ``debate_tools``
|
# NOT keyword chunk-stuffing; the road path keeps the chunk-retrieval data source. ``debate_tools``
|
||||||
# is the query-time retrieval surface — empty on the bundle path (navigation already placed the
|
# is the query-time retrieval surface: since S2c the bundle path carries the four NAVIGATOR
|
||||||
# curated context in the prompt, and a docs_dir==bundle_dir tool would re-leak the verdict layer).
|
# tools there (progressive disclosure taken to its conclusion — the agents open what they need
|
||||||
|
# instead of being handed the base), and a ``docs_dir==bundle_dir`` chunk tool is still refused,
|
||||||
|
# because that one would re-leak the verdict layer the navigation excludes by construction.
|
||||||
# S4.0 (F3): the run path SETS the validator's cost baseline, so the deterministic gate is
|
# S4.0 (F3): the run path SETS the validator's cost baseline, so the deterministic gate is
|
||||||
# anchored to the project's real cost lines instead of the ones the proposal asserts.
|
# anchored to the project's real cost lines instead of the ones the proposal asserts.
|
||||||
# * road path: the reference project's own ``cost_items`` ARE the baseline -> always anchored.
|
# * road path: the reference project's own ``cost_items`` ARE the baseline -> always anchored.
|
||||||
|
|
@ -858,7 +906,8 @@ async def run_project(
|
||||||
# the mount is carried alongside, so a base delivered under a directory name of its own is
|
# the mount is carried alongside, so a base delivered under a directory name of its own is
|
||||||
# opened rather than refused. What is still refused, before a single model call: a base
|
# opened rather than refused. What is still refused, before a single model call: a base
|
||||||
# whose concepts declare two different corpora.
|
# whose concepts declare two different corpora.
|
||||||
resolved_bundle_id: okf.ResolvedBundleId | None = okf.reconcile_bundle_id(bundle_dir)
|
resolved = okf.reconcile_bundle_id(bundle_dir)
|
||||||
|
resolved_bundle_id: okf.ResolvedBundleId | None = resolved
|
||||||
okf.assert_declared_ids_agree(bundle)
|
okf.assert_declared_ids_agree(bundle)
|
||||||
project = _project_from_bundle(bundle_dir, project_id, bundle=bundle)
|
project = _project_from_bundle(bundle_dir, project_id, bundle=bundle)
|
||||||
# The THIRD projection into ``CostBaseline`` (MAJOR-4), behind an EXPLICIT commission and
|
# The THIRD projection into ``CostBaseline`` (MAJOR-4), behind an EXPLICIT commission and
|
||||||
|
|
@ -872,14 +921,22 @@ async def run_project(
|
||||||
if derive_cost_baseline
|
if derive_cost_baseline
|
||||||
else okf.load_optional_cost_baseline(bundle_dir)
|
else okf.load_optional_cost_baseline(bundle_dir)
|
||||||
)
|
)
|
||||||
# §4.1a context-scope: agents read ONLY dimension-scoped bundle knowledge (Step-3 filter);
|
# S2c: the debate NAVIGATES the base; it is never handed the whole of it. Measured on K2
|
||||||
# dimension=None keeps the full context, byte-identical to before.
|
# (630 concepts, docs/2026-09-04-syretest-s7b-k2.md § 3.4) the rendered context was 648 962
|
||||||
context = okf.bundle_context(bundle, dimension=dimension.id if dimension else None)
|
# o200k tokens riding in THREE prompts — 99,1 % of a run's prompt cost, none of it asked
|
||||||
|
# for twice. The task message now carries a POINTER, and the agents get the SAME four
|
||||||
|
# tools the exploration uses; ``gen_context = debate_output or context`` below means the
|
||||||
|
# generation fallback is bounded by the same change rather than by a second policy.
|
||||||
|
dimension_id = dimension.id if dimension else None
|
||||||
|
context = _bundle_pointer(bundle, resolved.id, dimension=dimension_id)
|
||||||
citations = bundle_citations(bundle)
|
citations = bundle_citations(bundle)
|
||||||
# What the navigation could NOT reach, taken from the run's ONE walk. The road path below
|
# What the navigation could NOT reach, taken from the run's ONE walk. The road path below
|
||||||
# navigates no bundle at all, so its empty tuple is literally true rather than a stand-in.
|
# navigates no bundle at all, so its empty tuple is literally true rather than a stand-in.
|
||||||
skipped_links: tuple[okf.SkippedLink, ...] = bundle.skipped
|
skipped_links: tuple[okf.SkippedLink, ...] = bundle.skipped
|
||||||
debate_tools: list[Any] = []
|
# §4.1a context-scope, carried over: the agents read ONLY dimension-matched knowledge. The
|
||||||
|
# filter used to live in the rendering; with navigation it lives in the TOOLS, on both
|
||||||
|
# rungs (``navigator_tools``' own gate), because that is now where the bytes leave.
|
||||||
|
debate_tools: list[Any] = list(navigator_tools([bundle_dir], dimension=dimension_id))
|
||||||
else:
|
else:
|
||||||
project = _project_by_id(project_id)
|
project = _project_by_id(project_id)
|
||||||
baseline = baseline_from_project(project)
|
baseline = baseline_from_project(project)
|
||||||
|
|
@ -915,12 +972,23 @@ async def run_project(
|
||||||
# called. Attached only when servers are configured — with none there is nothing to attribute a
|
# called. Attached only when servers are configured — with none there is nothing to attribute a
|
||||||
# call to, and the middleware list stays exactly what it was before Trekk B.
|
# call to, and the middleware list stays exactly what it was before Trekk B.
|
||||||
call_recorder = ToolCallRecorder(tool_server_index(mcp_servers)) if mcp_servers else None
|
call_recorder = ToolCallRecorder(tool_server_index(mcp_servers)) if mcp_servers else None
|
||||||
|
# S2c: a CALLER-OWNED sink for what the debate opens (the ``parse_failures``/``ExplorationTrace``
|
||||||
|
# shape). A returned value would be lost on exactly the run that most needs the evidence — a
|
||||||
|
# budget stop mid-debate raises out of ``debate.run`` and constructs no ``RunResult`` at all.
|
||||||
|
# ``ExplorationToolRecorder`` is REUSED rather than re-implemented: it is already the recorder
|
||||||
|
# for in-process navigator calls, ordered and un-deduplicated, which is exactly the question
|
||||||
|
# here too ("did this run open anything, and in what sequence"). Its sibling
|
||||||
|
# ``mcp_tools.ToolCallRecorder`` stays what it is — a sorted, de-duplicated EGRESS claim.
|
||||||
|
debate_tool_calls: list[ToolCall] = []
|
||||||
|
debate_middleware: list[Any] = [budget_mw, ExplorationToolRecorder(debate_tool_calls)]
|
||||||
|
if call_recorder is not None:
|
||||||
|
debate_middleware.append(call_recorder)
|
||||||
debate = fresh_workflow(
|
debate = fresh_workflow(
|
||||||
factory,
|
factory,
|
||||||
max_rounds=max_rounds,
|
max_rounds=max_rounds,
|
||||||
enable_layer1_hitl=enable_layer1_hitl,
|
enable_layer1_hitl=enable_layer1_hitl,
|
||||||
tools=debate_tools,
|
tools=debate_tools,
|
||||||
middleware=[budget_mw] if call_recorder is None else [budget_mw, call_recorder],
|
middleware=debate_middleware,
|
||||||
)
|
)
|
||||||
# S4.2 cut (comparison protocol §4 pkt 2/3): everything above is offline — contracts, budget, and
|
# S4.2 cut (comparison protocol §4 pkt 2/3): everything above is offline — contracts, budget, and
|
||||||
# the EAGER client build (fresh_workflow constructs the proposer+checker clients, workflow.py:64).
|
# the EAGER client build (fresh_workflow constructs the proposer+checker clients, workflow.py:64).
|
||||||
|
|
@ -957,12 +1025,23 @@ async def run_project(
|
||||||
# opens a connection — its promise to stop before the first call covers egress too. Constructed
|
# opens a connection — its promise to stop before the first call covers egress too. Constructed
|
||||||
# tools that are never entered expose nothing, and ones never exited leave the process hanging,
|
# tools that are never entered expose nothing, and ones never exited leave the process hanging,
|
||||||
# so the stack owns both halves.
|
# so the stack owns both halves.
|
||||||
async with AsyncExitStack() as mcp_stack:
|
try:
|
||||||
for live_tool in live_mcp_tools:
|
async with AsyncExitStack() as mcp_stack:
|
||||||
await mcp_stack.enter_async_context(live_tool)
|
for live_tool in live_mcp_tools:
|
||||||
result = await debate.run(
|
await mcp_stack.enter_async_context(live_tool)
|
||||||
f"Find a cost-saving measure for {project.id}.\nContext:\n{context}"
|
result = await debate.run(
|
||||||
)
|
f"Find a cost-saving measure for {project.id}.\nContext:\n{context}"
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
# ``finally``, the ``write_parse_failures`` precedent: any exception leaving the debate —
|
||||||
|
# a budget stop is today's known one — destroys the same evidence, and a list of exception
|
||||||
|
# types is a list that goes stale. Written even when EMPTY: "the debate opened nothing" is
|
||||||
|
# the S2c regression itself, so it must be readable rather than inferred from an absence.
|
||||||
|
if outbox_dir is not None:
|
||||||
|
assert run_id is not None # narrowed by the step-0 guard (no wall-clock default)
|
||||||
|
outbox.write_debate_tools(
|
||||||
|
outbox_dir, run_id, tool_calls=tool_call_payload(debate_tool_calls)
|
||||||
|
)
|
||||||
# F1: the candidate must derive from the DEBATE. Feed the proposer's converged output into
|
# F1: the candidate must derive from the DEBATE. Feed the proposer's converged output into
|
||||||
# generation (retrieval context is the last-resort fallback only). The checker's verdict
|
# generation (retrieval context is the last-resort fallback only). The checker's verdict
|
||||||
# (Step 3/4) is parsed from the SAME debate result and gates the outcome below.
|
# (Step 3/4) is parsed from the SAME debate result and gates the outcome below.
|
||||||
|
|
@ -1219,6 +1298,7 @@ async def run_project(
|
||||||
refinements=tuple(refinements),
|
refinements=tuple(refinements),
|
||||||
skipped_links=skipped_links,
|
skipped_links=skipped_links,
|
||||||
unkeyed_verdicts=unkeyed_verdicts,
|
unkeyed_verdicts=unkeyed_verdicts,
|
||||||
|
debate_tool_calls=tuple(debate_tool_calls),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -1940,12 +2020,25 @@ def _load_scripted_replies(
|
||||||
``scripted_factory``'s lookup, mid-run, long after the run appeared to start cleanly (MAJOR-2:
|
``scripted_factory``'s lookup, mid-run, long after the run appeared to start cleanly (MAJOR-2:
|
||||||
measured for the three ``explore()`` adds on top of the debate's own two).
|
measured for the three ``explore()`` adds on top of the debate's own two).
|
||||||
|
|
||||||
**An EXPLORATION role may also be given a step LIST** (MAJOR-1 b), because a single constant
|
**ANY role may be given a step LIST** (MAJOR-1 b), because a single constant string can never
|
||||||
string can never emit a ``function_call`` — measured: 0 tool calls / 0 approaches / 1 round on
|
emit a ``function_call`` — measured: 0 tool calls / 0 approaches / 1 round on 4/4 bases, an
|
||||||
4/4 bases, an offline rehearsal that was vacuous by construction. The list form is refused for
|
offline rehearsal that was vacuous by construction.
|
||||||
the debate's roles BY NAME rather than accepted and ignored: the proposer answers
|
|
||||||
``generate``'s own call, not an agent loop that would invoke a tool between turns, so a script
|
Until S2c the list form was REFUSED for the debate's two roles, on the stated ground that "the
|
||||||
of calls there describes a rehearsal that cannot happen."""
|
proposer answers ``generate``'s own call, not an agent loop that would invoke a tool between
|
||||||
|
turns, so a script of calls there describes a rehearsal that cannot happen". That ground is now
|
||||||
|
measurably false: the debate's proposer and checker are ``Agent``s in a GroupChat and they hold
|
||||||
|
the four navigator tools, so a constant-string rehearsal proves the debate RUNS while proving
|
||||||
|
nothing about whether it OPENS the base — which is the identical vacuity MAJOR-1 closed one
|
||||||
|
surface over. Keeping the refusal would have made the free half of the measurement ladder
|
||||||
|
unable to reach the very seam S2c builds.
|
||||||
|
|
||||||
|
**Honesty limit, stated rather than encoded.** Scripts are per-CLIENT and each client gets its
|
||||||
|
own copy, so a ``proposer`` script is consumed once by the DEBATE client and again, from the
|
||||||
|
start, by the fresh client ``generate_via_llm`` builds. A script whose first step is a tool
|
||||||
|
call therefore answers the generation call with a ``function_call`` too, which will not parse.
|
||||||
|
That is the operator's to write correctly: guessing which steps were "meant for" which call
|
||||||
|
site would be repair, and this loader validates."""
|
||||||
try:
|
try:
|
||||||
raw = json.loads(Path(path).read_text(encoding="utf-8"))
|
raw = json.loads(Path(path).read_text(encoding="utf-8"))
|
||||||
except FileNotFoundError as exc:
|
except FileNotFoundError as exc:
|
||||||
|
|
@ -1963,12 +2056,6 @@ def _load_scripted_replies(
|
||||||
for role in required_roles:
|
for role in required_roles:
|
||||||
if isinstance(raw[role], str):
|
if isinstance(raw[role], str):
|
||||||
continue
|
continue
|
||||||
if role not in _EXPLORATION_SCRIPTED_ROLES:
|
|
||||||
raise ValueError(
|
|
||||||
f"--scripted-replies[{role!r}] is a step list, but that form is the exploration's: "
|
|
||||||
f"only {', '.join(_EXPLORATION_SCRIPTED_ROLES)} run inside an agent loop that can "
|
|
||||||
f"invoke a tool between turns ({path})"
|
|
||||||
)
|
|
||||||
_validate_script(role, raw[role], path)
|
_validate_script(role, raw[role], path)
|
||||||
return {role: raw[role] for role in required_roles}
|
return {role: raw[role] for role in required_roles}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -175,7 +175,8 @@ def scripted_proposer(
|
||||||
|
|
||||||
**Why the project id and not the cost code or the measure name** (measured, not assumed — the
|
**Why the project id and not the cost code or the measure name** (measured, not assumed — the
|
||||||
demo-week plan §6 flagged this as unverified): two prompt shapes reach this selector. The debate
|
demo-week plan §6 flagged this as unverified): two prompt shapes reach this selector. The debate
|
||||||
prompt (``run.py``) carries the whole bundle context; the generation prompt
|
prompt (``run.py``) carries the task line plus a POINTER to the knowledge base — since S2c it no
|
||||||
|
longer carries the base itself, which only sharpens the argument; the generation prompt
|
||||||
(``generate._build_messages``) carries ``Project: {id} - {name}`` plus, as its context, the
|
(``generate._build_messages``) carries ``Project: {id} - {name}`` plus, as its context, the
|
||||||
*debate output* — which is this selector's own earlier reply. So the cost code and measure name
|
*debate output* — which is this selector's own earlier reply. So the cost code and measure name
|
||||||
are present in the generation prompt only because the script put them there; keying on them
|
are present in the generation prompt only because the script put them there; keying on them
|
||||||
|
|
@ -186,8 +187,8 @@ def scripted_proposer(
|
||||||
|
|
||||||
Anything other than exactly one match raises ``ScriptedCandidateError``. Validation, never
|
Anything other than exactly one match raises ``ScriptedCandidateError``. Validation, never
|
||||||
repair: a default reply would let an unregistered project be answered with another project's
|
repair: a default reply would let an unregistered project be answered with another project's
|
||||||
numbers, which on screen is indistinguishable from a correct run. An ambiguous blob (a bundle
|
numbers, which on screen is indistinguishable from a correct run. An ambiguous blob (a tool
|
||||||
context that names a sibling project) is a DATA problem, and it must surface at the rehearsal
|
result that names a sibling project) is a DATA problem, and it must surface at the rehearsal
|
||||||
rather than be silently decided by registry order.
|
rather than be silently decided by registry order.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
|
@ -806,9 +807,12 @@ async def simulate_exploration(
|
||||||
shutil.copytree(bundle_dir, copy)
|
shutil.copytree(bundle_dir, copy)
|
||||||
copy_s = str(copy)
|
copy_s = str(copy)
|
||||||
|
|
||||||
# The vacuity guard. A label the base already states would reach the hypothesis prompt as
|
# The vacuity guard. A label the base already states is a label the walkthrough cannot
|
||||||
# navigated context whether or not the exploration ran, so the scenario's own assertion would
|
# attribute to the exploration: since S2c the debate reads the base through its own tools, so
|
||||||
# hold against an implementation that never wired the mandate at all.
|
# such a label could reach the hypothesis prompt as an ordinary tool result whether or not the
|
||||||
|
# exploration ran, and the scenario's assertion would hold against an implementation that never
|
||||||
|
# wired the mandate at all. ``bundle_context`` is used here as "everything the base states",
|
||||||
|
# which is what it still renders — not as a claim about what any prompt carries.
|
||||||
context = okf.bundle_context(okf.navigate_bundle(copy_s))
|
context = okf.bundle_context(okf.navigate_bundle(copy_s))
|
||||||
if label in context:
|
if label in context:
|
||||||
raise ValueError(
|
raise ValueError(
|
||||||
|
|
|
||||||
|
|
@ -27,7 +27,8 @@ Detach points, each RED on its own:
|
||||||
* stamp every per-approach artefact with the run's single verdict id -> judging one clears all.
|
* stamp every per-approach artefact with the run's single verdict id -> judging one clears all.
|
||||||
|
|
||||||
The control (``test_without_a_mandate_the_outbox_is_byte_unchanged``) proves the addition is inert
|
The control (``test_without_a_mandate_the_outbox_is_byte_unchanged``) proves the addition is inert
|
||||||
on the no-mandate path: the same two filenames as before, carrying no ``approach_id`` key at all.
|
on the no-mandate path: the same un-suffixed filenames as before, carrying no ``approach_id`` key at
|
||||||
|
all.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
@ -257,8 +258,12 @@ async def test_without_a_mandate_the_outbox_is_byte_unchanged(tmp_path: Path) ->
|
||||||
outbox = tmp_path / "outbox"
|
outbox = tmp_path / "outbox"
|
||||||
await _run(None, outbox)
|
await _run(None, outbox)
|
||||||
|
|
||||||
|
# ``-debate.json`` is S2c's tool trace, written on every bundle-path run with an outbox and
|
||||||
|
# unrelated to the per-approach keying this control is about; it is listed so the assertion
|
||||||
|
# stays an EXACT set rather than degrading into "at least these".
|
||||||
written = sorted(p.name for p in outbox.glob("*.json"))
|
written = sorted(p.name for p in outbox.glob("*.json"))
|
||||||
assert written == [
|
assert written == [
|
||||||
|
f"{_RUN_ID}-debate.json",
|
||||||
f"{_RUN_ID}-outcome.json",
|
f"{_RUN_ID}-outcome.json",
|
||||||
f"{_RUN_ID}-proposal.json",
|
f"{_RUN_ID}-proposal.json",
|
||||||
f"{_RUN_ID}-runconfig.json",
|
f"{_RUN_ID}-runconfig.json",
|
||||||
|
|
|
||||||
384
tests/test_debate_navigation_cost_loadbearing.py
Normal file
384
tests/test_debate_navigation_cost_loadbearing.py
Normal file
|
|
@ -0,0 +1,384 @@
|
||||||
|
"""S2c LOAD-BEARING: the DEBATE navigates the knowledge base; it is never handed the whole of it.
|
||||||
|
|
||||||
|
MAJOR-3 and S7a-3 made the EXPLORATION cheap — 18 355 o200k tokens over the whole of K2 — and
|
||||||
|
left the pipeline's own two phases stuffing. Measured on K2 (630 concepts,
|
||||||
|
``docs/2026-09-04-syretest-s7b-k2.md`` § 3.4, re-measured with the same instrument before this
|
||||||
|
change): ``okf.bundle_context`` is **648 962 o200k tokens** and rides in **three** prompts (two
|
||||||
|
proposer turns + the checker's), so debate + generation is **1 947 342 tokens = 99,1 %** of a
|
||||||
|
run's prompt cost. Ninety-nine percent of what a run pays for is context nobody asked for twice.
|
||||||
|
|
||||||
|
The move is S7a-3's own, one seam over: ``run_project``'s bundle path stops rendering the base
|
||||||
|
into the task message and instead hands the debate the SAME four navigator tools the exploration
|
||||||
|
uses (``navigator_tools``) plus a POINTER naming the base. Generation is covered by the same
|
||||||
|
change: ``gen_context = debate_output or context``, so the last-resort fallback is bounded by
|
||||||
|
construction rather than being the whole corpus.
|
||||||
|
|
||||||
|
Arms, each with a named detach point:
|
||||||
|
|
||||||
|
* **(a) no prompt carries the base.** A sentinel that lives only in a concept body is ABSENT from
|
||||||
|
every prompt the debate and the generation call see. CONTROL: the same sentinel IS in
|
||||||
|
``bundle_context``, so its absence is caused by the seam and not by an empty fixture.
|
||||||
|
* **(b) bounded, and not vacuously so.** Every prompt is under a ceiling that lives HERE, never in
|
||||||
|
``run.py`` (the ``read_bundle``/catalogue rule: raising the constant is the regression the gate
|
||||||
|
exists to catch), while ``bundle_context`` for the same base is over FIVE times it — without the
|
||||||
|
flat control a green bound could just mean the fixture is small. And the pointer must NAME the
|
||||||
|
base, because a bounded prompt that omits the id is a debate that cannot call the tools at all.
|
||||||
|
* **(c) the tools are actually there.** The four navigator tools reach the built workflow.
|
||||||
|
* **(d) the tool trace is a first-class artefact.** ``RunResult.debate_tool_calls`` and
|
||||||
|
``{run_id}-debate.json`` carry name + bundle_id + path in CALL ORDER — S7a-3 pkt. 3's rule, on
|
||||||
|
the second surface that now opens a base: over 629 concepts a trace reading ``read_file`` twice
|
||||||
|
answers nothing about which two.
|
||||||
|
* **(e) the dimension scope survives the move.** §4.1a promised the agents read ONLY
|
||||||
|
dimension-matched bundle knowledge. That promise used to be kept by ``bundle_context``'s filter;
|
||||||
|
with navigation it has to be kept by the TOOLS, on BOTH rungs — a listing that hides a foreign
|
||||||
|
document while ``read_file`` still serves it is a filter in name only.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import shutil
|
||||||
|
from collections.abc import Callable
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from agent_framework import BaseChatClient
|
||||||
|
|
||||||
|
from portfolio_optimiser import okf
|
||||||
|
from portfolio_optimiser.dimension import Dimension
|
||||||
|
from portfolio_optimiser.explore import DimensionScopeRefused, navigator_tools
|
||||||
|
from portfolio_optimiser.run import run_project
|
||||||
|
from portfolio_optimiser.simulation import ScriptedChatClient, scripted_factory
|
||||||
|
|
||||||
|
BUNDLE_DIR = Path(__file__).resolve().parents[1] / "shared" / "examples" / "bygg-energi-mikro"
|
||||||
|
_PID = "BYGG-KONTOR-NORD"
|
||||||
|
_VERDICT_INPUT = {"decision": "approved", "rationale": "expert reviewed (sim)"}
|
||||||
|
|
||||||
|
#: The prompt budget one debate/generation call may spend on bundle context, in CHARACTERS. It
|
||||||
|
#: lives in the TEST for ``_CATALOGUE_EXCERPT_CHARS``' reason: a ceiling imported from the
|
||||||
|
#: implementation moves with it, and raising it is precisely the regression this gate catches.
|
||||||
|
#: Characters rather than o200k tokens because ``tiktoken`` is not a project dependency, and a gate
|
||||||
|
#: that skips when an optional package is missing is a gate that can be silently absent
|
||||||
|
#: (MAJOR-3's own stated deviation, same reason).
|
||||||
|
_CEILING_CHARS = 1_500
|
||||||
|
|
||||||
|
_VALID_REPLY = (
|
||||||
|
'{"measure":"LED-retrofit","affected_items":'
|
||||||
|
'[{"code":"ENERGI-TOTAL-EL","quantity":300000,"unit_cost":1.0}],'
|
||||||
|
'"claimed_saving_nok":30000}'
|
||||||
|
)
|
||||||
|
_CHECKER_REPLY = "Reasoning holds.\nVERDICT: APPROVE"
|
||||||
|
|
||||||
|
|
||||||
|
def _prompt_blob(messages: Any) -> str:
|
||||||
|
"""Everything a prompt actually carries: text PLUS ``function_call``/``function_result``
|
||||||
|
contents. ``Message.text`` alone measures a context-bearing prompt at a few characters — the
|
||||||
|
corrected S7a-2 instrument, and the reason S2c could be measured at all."""
|
||||||
|
parts: list[str] = []
|
||||||
|
for message in messages:
|
||||||
|
text = getattr(message, "text", "") or ""
|
||||||
|
if text:
|
||||||
|
parts.append(text)
|
||||||
|
for content in getattr(message, "contents", ()) or ():
|
||||||
|
for attr in ("result", "arguments"):
|
||||||
|
value = getattr(content, attr, None)
|
||||||
|
if value:
|
||||||
|
parts.append(str(value))
|
||||||
|
return "\n".join(parts)
|
||||||
|
|
||||||
|
|
||||||
|
def _recording_factory(
|
||||||
|
sink: list[str], *, script: dict[str, Any] | None = None
|
||||||
|
) -> Callable[[str], BaseChatClient]:
|
||||||
|
"""A scripted client factory whose every prompt lands in ``sink`` as the FULL blob."""
|
||||||
|
|
||||||
|
def factory(role: str) -> BaseChatClient:
|
||||||
|
if script is not None and role in script:
|
||||||
|
client: BaseChatClient = scripted_factory(script, [])(role)
|
||||||
|
else:
|
||||||
|
client = ScriptedChatClient(
|
||||||
|
_CHECKER_REPLY if role == "checker" else _VALID_REPLY, role=role
|
||||||
|
)
|
||||||
|
original = client._inner_get_response # type: ignore[attr-defined]
|
||||||
|
|
||||||
|
def recording(*, messages, options, stream=False, **kwargs): # type: ignore[no-untyped-def]
|
||||||
|
sink.append(_prompt_blob(messages))
|
||||||
|
return original(messages=messages, options=options, stream=stream, **kwargs)
|
||||||
|
|
||||||
|
client._inner_get_response = recording # type: ignore[attr-defined,method-assign]
|
||||||
|
return client
|
||||||
|
|
||||||
|
return factory
|
||||||
|
|
||||||
|
|
||||||
|
async def _run(**kwargs: Any) -> tuple[Any, list[str]]:
|
||||||
|
sink: list[str] = []
|
||||||
|
script = kwargs.pop("script", None)
|
||||||
|
result = await run_project(
|
||||||
|
_PID,
|
||||||
|
"local",
|
||||||
|
docs_dir=str(BUNDLE_DIR),
|
||||||
|
bundle_dir=str(BUNDLE_DIR),
|
||||||
|
verdict_input=_VERDICT_INPUT,
|
||||||
|
client_factory=_recording_factory(sink, script=script),
|
||||||
|
**kwargs,
|
||||||
|
)
|
||||||
|
return result, sink
|
||||||
|
|
||||||
|
|
||||||
|
def _sentinel_from_the_base() -> str:
|
||||||
|
"""A string that exists ONLY inside a concept body of the fixture base — the leak probe."""
|
||||||
|
bundle = okf.navigate_bundle(str(BUNDLE_DIR))
|
||||||
|
reference = next(f for f in bundle.context_files if f.name == "kilder-realiseringsgap.md")
|
||||||
|
line = next(row for row in reference.body.splitlines() if len(row.strip()) > 60)
|
||||||
|
return line.strip()
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- (a) nothing carries the base
|
||||||
|
|
||||||
|
|
||||||
|
async def test_no_debate_or_generation_prompt_carries_the_whole_base() -> None:
|
||||||
|
"""LOAD-BEARING (a): a sentinel living only in a concept body reaches NO prompt.
|
||||||
|
|
||||||
|
Detach point: restore ``context = okf.bundle_context(bundle, ...)`` in ``run_project``'s
|
||||||
|
bundle arm → RED (the sentinel is back in all three prompts)."""
|
||||||
|
sentinel = _sentinel_from_the_base()
|
||||||
|
_, sink = await _run()
|
||||||
|
|
||||||
|
assert sink, "the debate never ran — no prompt was captured"
|
||||||
|
leaking = [i for i, prompt in enumerate(sink) if sentinel in prompt]
|
||||||
|
assert not leaking, (
|
||||||
|
f"the whole knowledge base is still being stuffed into prompt(s) {leaking}: "
|
||||||
|
"the debate is handed the corpus instead of navigating it"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_sentinel_is_really_in_the_base() -> None:
|
||||||
|
"""CAUSALITY CONTROL for (a): the sentinel IS what ``bundle_context`` renders, so its absence
|
||||||
|
above is caused by the seam rather than by a fixture that never held it."""
|
||||||
|
sentinel = _sentinel_from_the_base()
|
||||||
|
assert sentinel in okf.bundle_context(okf.navigate_bundle(str(BUNDLE_DIR)))
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------- (b) bounded, not vacuous
|
||||||
|
|
||||||
|
|
||||||
|
async def test_every_prompt_stays_under_the_ceiling() -> None:
|
||||||
|
"""LOAD-BEARING (b): the debate + generation prompts are bounded by the POINTER, so their cost
|
||||||
|
follows the number of bases configured (which the operator chose) and not the size of the
|
||||||
|
corpus (which they did not)."""
|
||||||
|
_, sink = await _run()
|
||||||
|
|
||||||
|
worst = max(len(prompt) for prompt in sink)
|
||||||
|
assert worst <= _CEILING_CHARS, (
|
||||||
|
f"the widest debate/generation prompt is {worst} characters, over the {_CEILING_CHARS} "
|
||||||
|
"ceiling — the corpus is riding along again"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_flat_form_is_far_over_the_ceiling() -> None:
|
||||||
|
"""FLAT CONTROL for (b): ``bundle_context`` on the SAME base is over five times the ceiling, so
|
||||||
|
a green bound above cannot just mean the fixture is small."""
|
||||||
|
whole = okf.bundle_context(okf.navigate_bundle(str(BUNDLE_DIR)))
|
||||||
|
assert len(whole) > 5 * _CEILING_CHARS, (
|
||||||
|
"the fixture base is too small for the bound to prove anything"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
async def test_the_pointer_names_the_base_the_tools_take() -> None:
|
||||||
|
"""LOAD-BEARING (b, second half): a bounded prompt that does not NAME the base is a debate
|
||||||
|
that cannot call a single tool — bounded and useless is the vacuous form of this gate.
|
||||||
|
|
||||||
|
Detach point: drop the bundle id from the pointer → RED."""
|
||||||
|
_, sink = await _run()
|
||||||
|
|
||||||
|
bundle_id = okf.reconcile_bundle_id(str(BUNDLE_DIR)).id
|
||||||
|
assert any(bundle_id in prompt for prompt in sink), (
|
||||||
|
f"no prompt names the knowledge base {bundle_id!r}; the navigator tools take that id, so "
|
||||||
|
"the debate has been given a bounded prompt it cannot act on"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------------- (c) the tools exist
|
||||||
|
|
||||||
|
|
||||||
|
async def test_the_debate_is_given_the_navigator_tools(monkeypatch) -> None:
|
||||||
|
"""LOAD-BEARING (c): the bundle path hands the debate the SAME four tools the exploration uses.
|
||||||
|
|
||||||
|
Detach point: drop ``navigator_tools(...)`` from ``debate_tools`` → RED."""
|
||||||
|
import portfolio_optimiser.run as run_module
|
||||||
|
|
||||||
|
captured: list[list[Any]] = []
|
||||||
|
original = run_module.fresh_workflow
|
||||||
|
|
||||||
|
def spy(*args: Any, **kwargs: Any) -> Any:
|
||||||
|
captured.append(list(kwargs.get("tools") or []))
|
||||||
|
return original(*args, **kwargs)
|
||||||
|
|
||||||
|
monkeypatch.setattr(run_module, "fresh_workflow", spy)
|
||||||
|
await _run()
|
||||||
|
|
||||||
|
assert captured, "the debate was never built"
|
||||||
|
names = {getattr(t, "name", "") for t in captured[0]}
|
||||||
|
assert {"list_bundles", "read_bundle", "read_dir", "read_file"} <= names, (
|
||||||
|
f"the debate's tool list is {sorted(names)} — it cannot navigate the base it was pointed at"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------------- (d) the tool trace
|
||||||
|
|
||||||
|
|
||||||
|
async def test_the_debate_tool_trace_reaches_the_result_and_the_outbox(tmp_path) -> None:
|
||||||
|
"""LOAD-BEARING (d): what the debate OPENED is recorded, in call order, with the path.
|
||||||
|
|
||||||
|
The proposer is driven by a step MANUSCRIPT (MAJOR-1 b) — the only offline form that can make a
|
||||||
|
scripted role emit a ``function_call``, and therefore the only way a free rehearsal can prove
|
||||||
|
the debate opens anything at all.
|
||||||
|
|
||||||
|
Detach points, each RED on its own: drop the ``ExplorationToolRecorder`` from the debate's
|
||||||
|
middleware; stop writing ``{run_id}-debate.json``; drop ``path`` from the payload."""
|
||||||
|
bundle_id = okf.reconcile_bundle_id(str(BUNDLE_DIR)).id
|
||||||
|
outbox_dir = tmp_path / "outbox"
|
||||||
|
result, _ = await _run(
|
||||||
|
outbox_dir=str(outbox_dir),
|
||||||
|
run_id="s2c",
|
||||||
|
script={
|
||||||
|
"proposer": [
|
||||||
|
{"call": "read_bundle", "args": {"bundle_id": bundle_id}},
|
||||||
|
{
|
||||||
|
"call": "read_file",
|
||||||
|
"args": {"bundle_id": bundle_id, "path": "metode-ipmvp-a.md"},
|
||||||
|
},
|
||||||
|
_VALID_REPLY,
|
||||||
|
],
|
||||||
|
"checker": _CHECKER_REPLY,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
observed = [(c.name, c.bundle_id, c.path) for c in result.debate_tool_calls]
|
||||||
|
assert observed[:2] == [
|
||||||
|
("read_bundle", bundle_id, ""),
|
||||||
|
("read_file", bundle_id, "metode-ipmvp-a.md"),
|
||||||
|
], f"the debate's tool trace is {observed} — the call sequence is not recorded as it happened"
|
||||||
|
|
||||||
|
payload = json.loads((outbox_dir / "s2c-debate.json").read_text(encoding="utf-8"))
|
||||||
|
assert payload["tool_calls"][:2] == [
|
||||||
|
{"name": "read_bundle", "bundle_id": bundle_id, "path": ""},
|
||||||
|
{"name": "read_file", "bundle_id": bundle_id, "path": "metode-ipmvp-a.md"},
|
||||||
|
], f"the artefact does not carry the call sequence: {payload['tool_calls']}"
|
||||||
|
|
||||||
|
|
||||||
|
async def test_a_debate_that_opened_nothing_says_so(tmp_path) -> None:
|
||||||
|
"""CONTROL for (d): a run whose agents called no tool leaves an EMPTY trace rather than no
|
||||||
|
artefact — "the debate never opened the base" is the S2c regression signal itself, so it must
|
||||||
|
be readable off the leaving, not inferred from a missing file."""
|
||||||
|
outbox_dir = tmp_path / "outbox"
|
||||||
|
result, _ = await _run(outbox_dir=str(outbox_dir), run_id="s2c-quiet")
|
||||||
|
|
||||||
|
assert result.debate_tool_calls == ()
|
||||||
|
payload = json.loads((outbox_dir / "s2c-quiet-debate.json").read_text(encoding="utf-8"))
|
||||||
|
assert payload["tool_calls"] == []
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ (e) the dimension survives
|
||||||
|
|
||||||
|
_ENERGY_DIM = Dimension(
|
||||||
|
id="energi", label="Energi", allowed_measure_types=frozenset({"energy_efficiency"})
|
||||||
|
)
|
||||||
|
_ASFALT_SENTINEL = "ASFALT-LEAK-SENTINEL-x7y8z9"
|
||||||
|
_ASFALT_FILE = "asfalt-dekke.md"
|
||||||
|
|
||||||
|
|
||||||
|
def _bundle_with_a_foreign_dimension(tmp_path: Path) -> str:
|
||||||
|
"""A copy of the fixture base plus ONE concept file marked ``dimension: asfalt``, linked from
|
||||||
|
the index so navigation reaches it."""
|
||||||
|
copy = tmp_path / "bundle"
|
||||||
|
shutil.copytree(BUNDLE_DIR, copy)
|
||||||
|
(copy / _ASFALT_FILE).write_text(
|
||||||
|
f"---\ntype: reference\ntitle: Asfaltdekke\ndimension: asfalt\n---\n\n{_ASFALT_SENTINEL}\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
index = copy / "index.md"
|
||||||
|
index.write_text(
|
||||||
|
index.read_text(encoding="utf-8") + f"\n- [Asfaltdekke]({_ASFALT_FILE})\n", encoding="utf-8"
|
||||||
|
)
|
||||||
|
return str(copy)
|
||||||
|
|
||||||
|
|
||||||
|
def _tools(bundle_dir: str, dimension: str | None) -> dict[str, Any]:
|
||||||
|
return {t.name: t for t in navigator_tools([bundle_dir], dimension=dimension)}
|
||||||
|
|
||||||
|
|
||||||
|
async def _invoke(tool: Any, **arguments: Any) -> str:
|
||||||
|
"""A tool's answer as text. ``FunctionTool.invoke`` returns ``[Content]``, so a test that
|
||||||
|
stringified the list would compare object reprs and pass against anything."""
|
||||||
|
return "".join(getattr(c, "text", "") or "" for c in await tool.invoke(arguments=arguments))
|
||||||
|
|
||||||
|
|
||||||
|
async def test_a_foreign_dimension_document_is_neither_listed_nor_readable(tmp_path) -> None:
|
||||||
|
"""LOAD-BEARING (e): under a dimension the navigator can neither SEE nor READ a document from
|
||||||
|
another one — both rungs, because a listing filter alone is a filter in name only.
|
||||||
|
|
||||||
|
The tools are called DIRECTLY: a ``ScriptedChatClient`` returns text and never emits a tool
|
||||||
|
call, so a test that only drove ``run_project`` would leave the whole tool surface outside the
|
||||||
|
gate (``test_explore_loadbearing``'s own measured correction).
|
||||||
|
|
||||||
|
Detach points: drop ``dimension`` from ``directory_listing``; drop the gate in ``read_file``."""
|
||||||
|
bundle_dir = _bundle_with_a_foreign_dimension(tmp_path)
|
||||||
|
bundle_id = okf.reconcile_bundle_id(bundle_dir).id
|
||||||
|
tools = _tools(bundle_dir, "energi")
|
||||||
|
|
||||||
|
listing = await _invoke(tools["read_bundle"], bundle_id=bundle_id)
|
||||||
|
assert _ASFALT_FILE not in listing, (
|
||||||
|
"a document from another dimension is still listed to the agents"
|
||||||
|
)
|
||||||
|
with pytest.raises(DimensionScopeRefused):
|
||||||
|
await _invoke(tools["read_file"], bundle_id=bundle_id, path=_ASFALT_FILE)
|
||||||
|
|
||||||
|
|
||||||
|
async def test_without_a_dimension_the_same_document_is_listed_and_readable(tmp_path) -> None:
|
||||||
|
"""CAUSALITY CONTROL for (e): with ``dimension=None`` the SAME file is both listed and read, so
|
||||||
|
its refusal above is caused by the scope and not by the file being unreachable."""
|
||||||
|
bundle_dir = _bundle_with_a_foreign_dimension(tmp_path)
|
||||||
|
bundle_id = okf.reconcile_bundle_id(bundle_dir).id
|
||||||
|
tools = _tools(bundle_dir, None)
|
||||||
|
|
||||||
|
listing = await _invoke(tools["read_bundle"], bundle_id=bundle_id)
|
||||||
|
assert _ASFALT_FILE in listing
|
||||||
|
assert _ASFALT_SENTINEL in await _invoke(
|
||||||
|
tools["read_file"], bundle_id=bundle_id, path=_ASFALT_FILE
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
async def test_a_dimension_scoped_run_gives_the_debate_scoped_tools(tmp_path, monkeypatch) -> None:
|
||||||
|
"""LOAD-BEARING (e, wiring): ``run_project``'s ``dimension`` reaches the TOOLS, not just the
|
||||||
|
(now absent) rendered context — otherwise §4.1a's promise is kept by nothing at all.
|
||||||
|
|
||||||
|
Detach point: build the debate's navigator tools without ``dimension=`` → RED."""
|
||||||
|
import portfolio_optimiser.run as run_module
|
||||||
|
|
||||||
|
bundle_dir = _bundle_with_a_foreign_dimension(tmp_path)
|
||||||
|
bundle_id = okf.reconcile_bundle_id(bundle_dir).id
|
||||||
|
captured: list[list[Any]] = []
|
||||||
|
original = run_module.fresh_workflow
|
||||||
|
|
||||||
|
def spy(*args: Any, **kwargs: Any) -> Any:
|
||||||
|
captured.append(list(kwargs.get("tools") or []))
|
||||||
|
return original(*args, **kwargs)
|
||||||
|
|
||||||
|
monkeypatch.setattr(run_module, "fresh_workflow", spy)
|
||||||
|
sink: list[str] = []
|
||||||
|
await run_project(
|
||||||
|
_PID,
|
||||||
|
"local",
|
||||||
|
docs_dir=bundle_dir,
|
||||||
|
bundle_dir=bundle_dir,
|
||||||
|
dimension=_ENERGY_DIM,
|
||||||
|
verdict_input=_VERDICT_INPUT,
|
||||||
|
client_factory=_recording_factory(sink),
|
||||||
|
)
|
||||||
|
|
||||||
|
read_file = next(t for t in captured[0] if getattr(t, "name", "") == "read_file")
|
||||||
|
with pytest.raises(DimensionScopeRefused):
|
||||||
|
await _invoke(read_file, bundle_id=bundle_id, path=_ASFALT_FILE)
|
||||||
|
|
@ -6,10 +6,17 @@ things, each with a named detach point:
|
||||||
The proposal validates on the numbers, so the ONLY possible rejecter is the ``admits`` scope
|
The proposal validates on the numbers, so the ONLY possible rejecter is the ``admits`` scope
|
||||||
gate (closes the green-but-dead trap). RED if ``admits`` is removed ⇒ the foreign candidate
|
gate (closes the green-but-dead trap). RED if ``admits`` is removed ⇒ the foreign candidate
|
||||||
slips through. Control: an in-dimension candidate passes.
|
slips through. Control: an in-dimension candidate passes.
|
||||||
- **Context scope (§4.1a):** a dimension-scoped ``run_project`` feeds ONLY dimension-matched bundle
|
- **Context scope (§4.1a), as it stands after S2c:** the promise is unchanged — a dimension-scoped
|
||||||
text into the agent prompt — a sentinel from ANOTHER dimension's concept file is ABSENT from the
|
run lets the agents read ONLY dimension-matched bundle knowledge — but the debate no longer
|
||||||
captured prompt. RED if the ``dimension=`` arg to ``bundle_context`` is dropped ⇒ the foreign-
|
RECEIVES a rendered context, it NAVIGATES the base, so the filter had to move from
|
||||||
dimension context leaks in. Control: ``dimension=None`` ⇒ the sentinel is present.
|
``bundle_context`` to the tools the debate is handed. These two arms gate the FIRST rung: the
|
||||||
|
listing the debate can see never names a foreign-dimension document. RED if the ``dimension=``
|
||||||
|
arg is dropped anywhere along ``run_project`` → ``navigator_tools`` → ``directory_listing``.
|
||||||
|
Control: ``dimension=None`` ⇒ the same document IS listed. The SECOND rung (``read_file``
|
||||||
|
refusing a foreign document by path) is gated in
|
||||||
|
``tests/test_debate_navigation_cost_loadbearing.py`` — a listing filter with an ungated reader
|
||||||
|
behind it is a filter in name only, so the two halves get their own arms and their own
|
||||||
|
mutations.
|
||||||
|
|
||||||
Patterns: ``test_checker_gate_loadbearing.py:59/87`` (gate + causality control),
|
Patterns: ``test_checker_gate_loadbearing.py:59/87`` (gate + causality control),
|
||||||
``conftest.py:184`` (recording client), ``test_step8_promotion_loadbearing.py:51`` (bundle copy).
|
``conftest.py:184`` (recording client), ``test_step8_promotion_loadbearing.py:51`` (bundle copy).
|
||||||
|
|
@ -21,9 +28,11 @@ import shutil
|
||||||
from collections.abc import Callable
|
from collections.abc import Callable
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
from agent_framework import BaseChatClient
|
from agent_framework import BaseChatClient
|
||||||
from conftest import SyntheticUsageChatClient
|
from conftest import SyntheticUsageChatClient
|
||||||
|
|
||||||
|
from portfolio_optimiser import okf
|
||||||
from portfolio_optimiser.dimension import Dimension
|
from portfolio_optimiser.dimension import Dimension
|
||||||
from portfolio_optimiser.run import run_project
|
from portfolio_optimiser.run import run_project
|
||||||
from portfolio_optimiser.validator import Rejection, ValidatedProposal
|
from portfolio_optimiser.validator import Rejection, ValidatedProposal
|
||||||
|
|
@ -128,54 +137,72 @@ def _bundle_with_asfalt_file(tmp_path: Path) -> str:
|
||||||
return str(dst)
|
return str(dst)
|
||||||
|
|
||||||
|
|
||||||
async def _run_and_capture(bundle_dir: str, dimension: Dimension | None) -> str:
|
async def _listing_the_debate_can_see(bundle_dir: str, dimension: Dimension | None) -> str:
|
||||||
"""Run the bundle path with a prompt-recording client and return the concatenated prompt text
|
"""Run the bundle path, capture the tools the debate was built with, and return what its
|
||||||
that reached the agents."""
|
``read_bundle`` rung answers — the S2c successor to reading the prompt text.
|
||||||
sink: list[str] = []
|
|
||||||
|
The tool is invoked DIRECTLY afterwards rather than through an agent turn, because a
|
||||||
|
``ScriptedChatClient`` returns text and never emits a ``function_call``: a test that only drove
|
||||||
|
the run would leave the entire tool surface outside the gate (``test_explore_loadbearing``'s
|
||||||
|
own measured correction)."""
|
||||||
|
import portfolio_optimiser.run as run_module
|
||||||
|
|
||||||
|
captured: list[list[object]] = []
|
||||||
|
original = run_module.fresh_workflow
|
||||||
|
|
||||||
|
def spy(*args: object, **kwargs: object) -> object:
|
||||||
|
captured.append(list(kwargs.get("tools") or [])) # type: ignore[arg-type]
|
||||||
|
return original(*args, **kwargs) # type: ignore[arg-type]
|
||||||
|
|
||||||
def factory(role: str) -> BaseChatClient:
|
def factory(role: str) -> BaseChatClient:
|
||||||
client = SyntheticUsageChatClient(
|
return SyntheticUsageChatClient(
|
||||||
default_reply=_valid_reply("energy_efficiency", "ENERGI-TOTAL-EL")
|
default_reply=_valid_reply("energy_efficiency", "ENERGI-TOTAL-EL")
|
||||||
)
|
)
|
||||||
_orig = client._inner_get_response
|
|
||||||
|
|
||||||
def _recording(*, messages, stream, options, **kwargs): # type: ignore[no-untyped-def]
|
monkeypatch = pytest.MonkeyPatch()
|
||||||
sink.append(" ".join(getattr(m, "text", "") or "" for m in messages))
|
try:
|
||||||
return _orig(messages=messages, stream=stream, options=options, **kwargs)
|
monkeypatch.setattr(run_module, "fresh_workflow", spy)
|
||||||
|
await run_project(
|
||||||
|
"BYGG-KONTOR-NORD",
|
||||||
|
"local",
|
||||||
|
docs_dir=bundle_dir,
|
||||||
|
bundle_dir=bundle_dir,
|
||||||
|
verdict_input=_VERDICT_INPUT,
|
||||||
|
dimension=dimension,
|
||||||
|
client_factory=factory,
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
monkeypatch.undo()
|
||||||
|
|
||||||
client._inner_get_response = _recording # type: ignore[method-assign]
|
assert captured, "the debate was never built"
|
||||||
return client
|
read_bundle = next(t for t in captured[0] if getattr(t, "name", "") == "read_bundle")
|
||||||
|
bundle_id = okf.reconcile_bundle_id(bundle_dir).id
|
||||||
await run_project(
|
answer = await read_bundle.invoke(arguments={"bundle_id": bundle_id})
|
||||||
"BYGG-KONTOR-NORD",
|
return "".join(getattr(c, "text", "") or "" for c in answer)
|
||||||
"local",
|
|
||||||
docs_dir=bundle_dir,
|
|
||||||
bundle_dir=bundle_dir,
|
|
||||||
verdict_input=_VERDICT_INPUT,
|
|
||||||
dimension=dimension,
|
|
||||||
client_factory=factory,
|
|
||||||
)
|
|
||||||
return " ".join(sink)
|
|
||||||
|
|
||||||
|
|
||||||
async def test_dimension_scopes_the_agent_context(tmp_path) -> None:
|
async def test_dimension_scopes_the_agent_context(tmp_path) -> None:
|
||||||
"""LOAD-BEARING (§4.1a): a dimension-scoped run feeds ONLY dimension-matched bundle text into
|
"""LOAD-BEARING (§4.1a, first rung): under a dimension the listing the debate can see does NOT
|
||||||
the prompt — the asfalt sentinel is ABSENT. RED if the ``dimension=`` arg to ``bundle_context``
|
name the foreign-dimension document. RED if ``dimension=`` is dropped anywhere between
|
||||||
is dropped (the foreign-dimension context leaks into the prompt)."""
|
``run_project`` and ``directory_listing``.
|
||||||
|
|
||||||
|
Asserts on the FILE NAME rather than the body sentinel, because a listing carries names and
|
||||||
|
sizes, never bodies — an assert on the sentinel would be green against every implementation
|
||||||
|
and would prove nothing (the vacuous form this repo keeps measuring)."""
|
||||||
bundle_dir = _bundle_with_asfalt_file(tmp_path)
|
bundle_dir = _bundle_with_asfalt_file(tmp_path)
|
||||||
|
|
||||||
scoped_prompt = await _run_and_capture(bundle_dir, _ENERGY_DIM)
|
scoped = await _listing_the_debate_can_see(bundle_dir, _ENERGY_DIM)
|
||||||
assert _ASFALT_SENTINEL not in scoped_prompt, (
|
assert "asfalt-note.md" not in scoped, (
|
||||||
"another dimension's context leaked into the prompt — bundle_context is not dimension-scoped"
|
"another dimension's document is listed to the agents — the tools are not dimension-scoped"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
async def test_no_dimension_leaves_context_unscoped(tmp_path) -> None:
|
async def test_no_dimension_leaves_context_unscoped(tmp_path) -> None:
|
||||||
"""CAUSALITY CONTROL: with ``dimension=None`` the asfalt sentinel IS present — proving its
|
"""CAUSALITY CONTROL: with ``dimension=None`` the asfalt document IS listed — proving its
|
||||||
absence above is caused by the dimension scope, not by the file being unreachable."""
|
absence above is caused by the dimension scope, not by the file being unreachable."""
|
||||||
bundle_dir = _bundle_with_asfalt_file(tmp_path)
|
bundle_dir = _bundle_with_asfalt_file(tmp_path)
|
||||||
|
|
||||||
full_prompt = await _run_and_capture(bundle_dir, None)
|
unscoped = await _listing_the_debate_can_see(bundle_dir, None)
|
||||||
assert _ASFALT_SENTINEL in full_prompt, (
|
assert "asfalt-note.md" in unscoped, (
|
||||||
"the asfalt file is unreachable even without a filter — the control does not prove causality"
|
"the asfalt file is unreachable even without a filter — the control does not prove causality"
|
||||||
)
|
)
|
||||||
|
|
|
||||||
|
|
@ -4,8 +4,8 @@ opened and closed around the debate, and nothing is contacted that was not annou
|
||||||
Krav 3 is only met if the external service is reachable *while the run works*. Three detach points,
|
Krav 3 is only met if the external service is reachable *while the run works*. Three detach points,
|
||||||
each RED on its own:
|
each RED on its own:
|
||||||
|
|
||||||
* drop the MCP tools from ``debate_tools`` -> the agents never get the tool (and on the bundle path
|
* drop the MCP tools from ``debate_tools`` -> the agents never get the tool (on the bundle path
|
||||||
they get NO tools at all, which is what that path had before);
|
they are then left with the four navigator tools alone — before S2c that path had none);
|
||||||
* skip the ``AsyncExitStack`` entry -> the tools are constructed but never connected, so they are
|
* skip the ``AsyncExitStack`` entry -> the tools are constructed but never connected, so they are
|
||||||
present and useless — the failure mode that looks like success;
|
present and useless — the failure mode that looks like success;
|
||||||
* let a dry run enter them -> ``--live-dry-run`` would contact a third party while claiming to stop
|
* let a dry run enter them -> ``--live-dry-run`` would contact a third party while claiming to stop
|
||||||
|
|
@ -115,7 +115,8 @@ async def _run(**kwargs: Any):
|
||||||
|
|
||||||
|
|
||||||
async def test_configured_server_becomes_a_tool_the_agents_have(fake_tools, captured_tools) -> None:
|
async def test_configured_server_becomes_a_tool_the_agents_have(fake_tools, captured_tools) -> None:
|
||||||
"""On the bundle path the agents had NO tools at all; a configured server is the first one."""
|
"""On the bundle path a configured server is the first EXTERNAL tool; since S2c the four
|
||||||
|
in-process navigator tools sit alongside it."""
|
||||||
await _run(mcp_servers=(_SERVER,))
|
await _run(mcp_servers=(_SERVER,))
|
||||||
assert captured_tools, "the debate was never built"
|
assert captured_tools, "the debate was never built"
|
||||||
assert any(isinstance(t, _FakeMcpTool) for t in captured_tools[0])
|
assert any(isinstance(t, _FakeMcpTool) for t in captured_tools[0])
|
||||||
|
|
@ -138,10 +139,21 @@ async def test_dry_run_never_contacts_a_configured_server(fake_tools) -> None:
|
||||||
|
|
||||||
|
|
||||||
async def test_run_without_mcp_servers_keeps_the_tool_list_unchanged(captured_tools) -> None:
|
async def test_run_without_mcp_servers_keeps_the_tool_list_unchanged(captured_tools) -> None:
|
||||||
"""CONTROL: with nothing configured the bundle path still hands the agents no tools — the
|
"""CONTROL: with nothing configured the bundle path hands the agents no EXTERNAL tool, so every
|
||||||
pre-Trekk-B behaviour, so every assertion above rests on the configuration and not on the run."""
|
assertion above rests on the configuration and not on something the run does anyway.
|
||||||
|
|
||||||
|
Before S2c this read ``captured_tools[0] == []`` — the bundle path had no tools at all. It now
|
||||||
|
navigates its knowledge base, so the control asserts what it always meant: nothing here reaches
|
||||||
|
outside the process. Asserting the exact navigator set as well keeps it from degrading into
|
||||||
|
"some tools, whatever they are"."""
|
||||||
await _run()
|
await _run()
|
||||||
assert captured_tools[0] == []
|
assert not any(isinstance(t, _FakeMcpTool) for t in captured_tools[0])
|
||||||
|
assert {getattr(t, "name", "") for t in captured_tools[0]} == {
|
||||||
|
"list_bundles",
|
||||||
|
"read_bundle",
|
||||||
|
"read_dir",
|
||||||
|
"read_file",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
async def test_portfolio_mode_gives_every_project_the_configured_tools(
|
async def test_portfolio_mode_gives_every_project_the_configured_tools(
|
||||||
|
|
|
||||||
|
|
@ -237,8 +237,13 @@ async def test_a_run_whose_replies_parse_writes_no_artefact(tmp_path: Path) -> N
|
||||||
|
|
||||||
assert not _artefact(outbox_dir, run_id).exists()
|
assert not _artefact(outbox_dir, run_id).exists()
|
||||||
# The pre-existing artefacts are untouched — the addition is inert on the clean path.
|
# The pre-existing artefacts are untouched — the addition is inert on the clean path.
|
||||||
|
# ``-debate.json`` (S2c) is written on EVERY bundle-path run that has an outbox,
|
||||||
|
# unlike this artefact whose PRESENCE is its signal: a debate that opened nothing is
|
||||||
|
# the S2c regression itself, so it must be readable rather than inferred from a
|
||||||
|
# file that is not there.
|
||||||
written = sorted(p.name for p in outbox_dir.iterdir())
|
written = sorted(p.name for p in outbox_dir.iterdir())
|
||||||
assert written == [
|
assert written == [
|
||||||
|
f"{run_id}-debate.json",
|
||||||
f"{run_id}-outcome.json",
|
f"{run_id}-outcome.json",
|
||||||
f"{run_id}-proposal.json",
|
f"{run_id}-proposal.json",
|
||||||
f"{run_id}-runconfig.json",
|
f"{run_id}-runconfig.json",
|
||||||
|
|
|
||||||
|
|
@ -442,16 +442,25 @@ def test_a_step_naming_an_unknown_key_is_refused_by_name(tmp_path, capsys) -> No
|
||||||
assert "result" in err, err
|
assert "result" in err, err
|
||||||
|
|
||||||
|
|
||||||
def test_a_step_list_is_refused_for_a_debate_role(tmp_path, capsys) -> None:
|
def test_a_step_list_is_accepted_for_a_debate_role(tmp_path) -> None:
|
||||||
"""The list form is the EXPLORATION's, and is refused elsewhere by name rather than accepted
|
"""REWRITTEN at S2c, not deleted: this arm used to pin the refusal of a step list for the
|
||||||
and quietly ignored. The debate's proposer is driven by ``generate``'s own call, not by an
|
debate's roles, on the ground that the proposer answers ``generate``'s own call rather than an
|
||||||
agent loop that would invoke a tool between turns, so a script of calls there would describe a
|
agent loop that could invoke a tool between turns. Since S2c the proposer and the checker ARE
|
||||||
rehearsal that cannot happen.
|
agents holding the four navigator tools, so that ground is measurably false and the refusal
|
||||||
|
would have kept the free half of the measurement ladder away from the very seam S2c builds —
|
||||||
|
the identical vacuity MAJOR-1 closed for the navigator.
|
||||||
|
|
||||||
|
What replaces it is the positive: a scripted proposer opens the base, and the debate's own
|
||||||
|
artefact names WHICH document. RED if the list form is refused for a debate role again, and RED
|
||||||
|
if the debate's ``ExplorationToolRecorder`` is detached (the run succeeds, the trace is empty).
|
||||||
"""
|
"""
|
||||||
replies = _replies_file(
|
replies = _replies_file(
|
||||||
tmp_path,
|
tmp_path,
|
||||||
{
|
{
|
||||||
"proposer": [{"call": "retrieve_cost_docs", "args": {"query": "x"}}],
|
"proposer": [
|
||||||
|
{"call": "read_bundle", "args": {"bundle_id": "bygg-energi-mikro"}},
|
||||||
|
_PROPOSER_REPLY,
|
||||||
|
],
|
||||||
"checker": "VERDICT: APPROVE",
|
"checker": "VERDICT: APPROVE",
|
||||||
"manager": list(_MANAGER_STAGES),
|
"manager": list(_MANAGER_STAGES),
|
||||||
"navigator": "NAVIGATOR: read the index.",
|
"navigator": "NAVIGATOR: read the index.",
|
||||||
|
|
@ -461,6 +470,12 @@ def test_a_step_list_is_refused_for_a_debate_role(tmp_path, capsys) -> None:
|
||||||
|
|
||||||
rc = run.main(_explore_argv(tmp_path, replies, "scripted-debate-list"))
|
rc = run.main(_explore_argv(tmp_path, replies, "scripted-debate-list"))
|
||||||
|
|
||||||
assert rc == 1
|
assert rc == 0
|
||||||
err = capsys.readouterr().err
|
payload = json.loads(
|
||||||
assert "run refused" in err and "proposer" in err, err
|
(tmp_path / "outbox" / "scripted-debate-list-debate.json").read_text(encoding="utf-8")
|
||||||
|
)
|
||||||
|
assert payload["tool_calls"][0] == {
|
||||||
|
"name": "read_bundle",
|
||||||
|
"bundle_id": "bygg-energi-mikro",
|
||||||
|
"path": "",
|
||||||
|
}, payload["tool_calls"]
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue