Compare commits
52 commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 3bdc5c1b7c | |||
| 56f4f6d084 | |||
| a1f8522bdf | |||
| c08ae91809 | |||
| 6020f4453d | |||
| 84e8de8679 | |||
| 444fea7e94 | |||
| 932ece345b | |||
| 98cbb80a45 | |||
| d01a157986 | |||
| 785261f229 | |||
| 18af86e422 | |||
| 3cbea91a72 | |||
| 2d44c5e224 | |||
| 118eabf9db | |||
| 234c8d138a | |||
| 8ba824c96f | |||
| ddc33eed8b | |||
| f0c54cc8dc | |||
| 4a19d39e63 | |||
| 9f0843ab6d | |||
| 2eb4622f44 | |||
| 2e33905ffe | |||
| ed696d2650 | |||
| 56c48f6f65 | |||
| b6397229ca | |||
| 3340fa19da | |||
| 143c5865b9 | |||
| 270ec87345 | |||
| 156312c32e | |||
| 0bc8dae5c0 | |||
| 9d149b3503 | |||
| 4cf8c4f6ba | |||
| 6d2837fdca | |||
| 241b50d6c4 | |||
| 77076b922c | |||
| f4c98101eb | |||
| c2d150495c | |||
| bb4807aff7 | |||
| 2d1264088e | |||
| 986fc19350 | |||
| 605c02aa62 | |||
| 642ce8ae9a | |||
| d371475ec9 | |||
| e3718908d0 | |||
| 5bd8e1caa1 | |||
| 88c223276c | |||
| a3300ab0f6 | |||
| 817bcf130f | |||
| 46b3e0e97d | |||
| 4da00cefb0 | |||
| 9148df0113 |
80 changed files with 17749 additions and 331 deletions
641
CLAUDE.md
641
CLAUDE.md
|
|
@ -24,7 +24,7 @@ Python ≥3.10. MAF (`agent-framework-core` 1.9.0). Pakkehåndtering: `uv`. To b
|
|||
- Modell-valg som konfig (modell-map rolle→Foundry-deployment), ikke spredt i kode.
|
||||
- Metode kodifiseres som **Agent Skill** (`agentskills.io`: `SKILL.md` + `scripts/` + `references/`).
|
||||
- Datatilgang: in-process `FunctionTool` er default-sømmen i kjørestien. **MCP er wiret som opt-in i kjørestien** (`mcp_tools.py` + `--mcp-config`, Trekk B 2026-08-05): konkrete eksterne servere blir verktøy agentene kan kalle UNDER debatten. Uten konfig gjøres null nettverkskall og verktøylista er uendret. Tre regler er load-bearing: **allowlist er påkrevd** (tom liste ville latt motparten bestemme hva agentene får kalle), **hver server og hvert tillatte verktøy navngis i kunngjøringen før første kall** (også uten `--mandate` — ingen udeklarert egress), og `--live-dry-run` åpner **ingenting**. Egen søm fra `ingest_mcp.py` (kildedokumenter FØR kjøring, null-argument-tools) — samme protokoll, ulik jobb. `build_mcp_server` (`datasource.py`) er fortsatt kun demo. Data-source-konfig JSON-Schema-validert, fail-fast.
|
||||
- `shared/` er en **git subtree** av [`portfolio-optimiser-commons`](https://git.fromaitochitta.com/open/portfolio-optimiser-commons) (source of truth, R1 realisert 2026-07-03; publisert i `open/` 2026-08-04 — `commons`-remoten peker fortsatt på `ktg/` og virker uendret). Synk er **pull-only**: endringer committes i commons og hentes med `git subtree pull --prefix=shared commons main --squash`. ALDRI `git subtree push` fra konsument — re-split lekker hele konsument-historikken inn i commons (observert + opprydd 2026-07-03). Se `shared/README.md`. Wheelen bærer treet som pakkede data siden Fase 4a — se invarianten under.
|
||||
- `shared/` er en **git subtree** av [`portfolio-optimiser-commons`](https://git.fromaitochitta.com/open/portfolio-optimiser-commons) (source of truth, R1 realisert 2026-07-03; publisert i `open/` 2026-08-04 — `commons`-remoten peker fortsatt på den private namespacen og virker uendret). Synk er **pull-only**: endringer committes i commons og hentes med `git subtree pull --prefix=shared commons main --squash`. ALDRI `git subtree push` fra konsument — re-split lekker hele konsument-historikken inn i commons (observert + opprydd 2026-07-03). Se `shared/README.md`. Wheelen bærer treet som pakkede data siden Fase 4a — se invarianten under.
|
||||
|
||||
## Kommandoer
|
||||
- Sync: `uv sync` — installerer to konsoll-kommandoer: `portfolio-optimiser` (CLI, `run:main`) og
|
||||
|
|
@ -63,12 +63,47 @@ Python ≥3.10. MAF (`agent-framework-core` 1.9.0). Pakkehåndtering: `uv`. To b
|
|||
commons-eide nav-goldens (`shared/examples/nav-golden-*/expected-read-context.md`, byte-nivå
|
||||
fasit): `test_nav_golden_hierarchy_*` (positiv) + `test_nav_golden_escape_*` (negativ — en gate
|
||||
som bare kan bli grønn beviser ingenting).
|
||||
**Et hopp er TOLERERT, men ikke lenger TAUST (21.08):** `_walk` registrerer hver lenke den ikke
|
||||
fulgte på `Bundle.skipped` — hvilken fil lenken sto i, lenketeksten ORDRETT (operatøren redigerer
|
||||
den teksten, ikke den resolverte stien), og hvilken av de TO grunnene som gjaldt: `outside-bundle`
|
||||
(escape — ofte bevisst, en lenke til nabobasen) eller `missing` (inne i basen, ingen lesbar fil —
|
||||
nesten alltid en skrivefeil). Den tredje grenen, `canonical in seen`, er DEDUP og registreres
|
||||
ALDRI — den er korrekt navigasjon og dét som terminerer sykler; en implementasjon som logget hvert
|
||||
`continue` ville rapportert en frisk base som halvlest. Toleransen er URØRT (§4 krever at det ikke
|
||||
kastes) — dette er synlighet, ikke en ny nekt. **Feltet DEFAULTER til tom tuppel, og det er
|
||||
MOTSATT av `cost_baseline_anchored`s «påkrevd uten default»:** en tom trace er et ærlig POSITIVT
|
||||
utsagn («hver lenke ble fulgt», `external_calls`-presedensen), mens en manglende bool måtte påstå
|
||||
noe om en hendelse og begge påstandene ville iblant vært usanne. Sporet forlater kjøringen på
|
||||
`RunResult.skipped_links` (RUN-nivå — navigasjonen skjer ÉN gang per kjøring, før noe forslag
|
||||
finnes) og `DryRunReport.skipped_links`, aldri på `ProvenanceStamp`, som beskriver gaten som dømte
|
||||
ÉN kandidat. `run.skipped_links_notice` er ENESTE renderer, tar den alt oppløste tuppelen og
|
||||
returnerer `None` når ingenting ble hoppet over (omisjon, aldri tom rad — `announce`-regelen);
|
||||
reason-TOKENET printes rått, så det finnes ingen andre display-vokabular å drifte fra feltet.
|
||||
Ingenting av dette når `bundle_context` (som bygges av `index_summary` + `context_files` alene) —
|
||||
dét er hva som holder nav-goldenene byte-uendret, og `Bundle(` har fortsatt ÉN konstruksjons-sted
|
||||
(`okf.py`, i `navigate_bundle`). Load-bearing MÅLT
|
||||
(`tests/test_navigation_visibility_loadbearing.py`), åtte mutasjoner alle røde mot HELE suiten +
|
||||
grønn kontroll 897/5: detach `missing`-registreringen (6 røde) · detach `outside-bundle` (2 røde) ·
|
||||
kollaps de to grunnene til én (2 røde) · registrer dedup-grenen (1 rød) · renderer returnerer alltid
|
||||
linja (3 røde — inkl. kontrollene, altså er omisjonen selv gatet) · detach dry-run-printen (1 rød) ·
|
||||
detach full-run-printen (1 rød) · konstant tom trace ut av `run_project` (4 røde).
|
||||
- **Kuraterte skrivere kan ikke forfalske ingest-stempelet** (`ingest-spec` §3): `write_concept_file`
|
||||
er repoets ene authoring-primitiv som materialiserer en konseptfil fra CALLER-oppgitt frontmatter,
|
||||
og avviser derfor det KOMPLETTE eierskaps-stempelet (`generated: true` + `ingest_manifest`) med
|
||||
`IngestStampError` — mens hver halvdel alene er lovlig (kuratert innhold kan bære ett
|
||||
provenance-felt). Validering, ALDRI reparasjon: ingenting skrives. Uten dette kunne en kuratert fil
|
||||
bli stille slettet av en senere re-materialisering, som fjerner nøyaktig det som bærer stempelet.
|
||||
**`generated`-verdien er FAIL-CLOSED på YAML-1.1-sannhetsformer, ikke bare literalen `"true"`**
|
||||
(funn 21.08, økt 52): `_YAML_TRUE_LITERALS` (`{"true", "yes", "on"}`, case-insensitivt) er
|
||||
ENESTE vokabular, målt mot PyYAML sin `safe_load`-resolver — bare `1`/bare `y`/`n` er BEVISST
|
||||
UTELATT (resolves til int/streng, aldri bool, så en YAML-leser ville uansett ikke lest dem som
|
||||
stempelet). Uten dette var sjekken inert kun i kraft av at pinnet `llm-ingestion-okf v0.3.2`
|
||||
skriver strengen `"true"` — en fremtidig `uv sync` mot en skrivemåte som `yes`/`on` ville latt
|
||||
vakten slutte å vokte uten én lokal diff. Load-bearing MÅLT
|
||||
(`tests/test_ingest_stamp_fail_closed_loadbearing.py`), fire mutasjoner alle røde mot HELE
|
||||
suiten: revert til literalen `"true"` (2 røde — de nye sannhetsformene alene) · over-widen til å
|
||||
inkludere `1`/`y` (1 rød) · `and`→`or` (4 røde, halv-stempel-lovligheten brutt) · detach gaten
|
||||
helt (4 røde).
|
||||
- **`IngestError` må overleve anyio-task-gruppene (kø-(x), 2026-08-03):** `stdio_client` og
|
||||
`ClientSession` er hver sin task group, og anyio pakker ALT som forlater en av dem i en
|
||||
`BaseExceptionGroup`. Derfor nådde `stdio_call_tool`s egne feil (`mcp_tool_error`,
|
||||
|
|
@ -188,18 +223,16 @@ Python ≥3.10. MAF (`agent-framework-core` 1.9.0). Pakkehåndtering: `uv`. To b
|
|||
ukjente felt NEKTES ved navn (400), aldri stille droppet (valg-doc §0-fella anvendt på vår egen
|
||||
flate); `profile` defaulter til `azure` KUN her (containeren har ingen lokal endpoint;
|
||||
`run_project`s egen default forblir LOCAL). Feilmapping ærlig: `ValueError` (pydantic-kontrakter
|
||||
subklasser den) → 400, alt annet → 500 `{error_type, error}` (speiler `RunFailure`), og en
|
||||
subklasser den) → 400, `BudgetExceeded` → 429 (EGEN rad under, 14.08), alt annet → 500
|
||||
`{error_type, error}` (speiler `RunFailure`), og en
|
||||
`Rejection` er en VELLYKKET kjøring → 200 — det negative utfallet tilhører payloaden, aldri
|
||||
transporten. `outbox.outcome_payload` er den ENE kopien av validated/rejected-forgreningen
|
||||
(delt av fil-skriveren og HTTP-responsen — to kopier drifter, kø-(p)-regelen). `azure.yaml`
|
||||
validert GRØNN mot begge autoritative skjemaer; ingen `env:` (redeklarer aldri
|
||||
`FOUNDRY_PROJECT_ENDPOINT`), ingen `startupCommand` (imagets `CMD` er den ene kopien av
|
||||
startkommandoen). `git archive <tree> | docker build --platform linux/amd64 -` grønn på
|
||||
indeks-treet. Load-bearing MÅLT (`tests/test_hosting_loadbearing.py`), seks mutasjoner alle
|
||||
(delt av fil-skriveren og HTTP-responsen — to kopier drifter, kø-(p)-regelen). **Container-innpakningen
|
||||
(`Dockerfile`/`azure.yaml`) ER FJERNET 14.08** — se python-only-invarianten under; resten av
|
||||
denne raden står, for `main.py` startes nå direkte (`python main.py`). Load-bearing MÅLT (`tests/test_hosting_loadbearing.py`), seks mutasjoner alle
|
||||
røde mot hele suiten på riktig test: detach felt-mappingen · dropp ukjente felt stille · flipp
|
||||
400/500 · detach azure-defaulten · detach SIGTERM-handleren · detach main.py-shimen (de to
|
||||
siste fanges KUN av subprosess-testen — P4-presedensen). Deploy er IKKE utført (azd-steget er
|
||||
operatørens); chunked request-bodies støttes ikke, og under CPU-bundne strekk (CBC-solven)
|
||||
siste fanges KUN av subprosess-testen — P4-presedensen). Chunked request-bodies støttes ikke, og under CPU-bundne strekk (CBC-solven)
|
||||
står readiness — uttalt, ikke skjult.
|
||||
- **Whitelisten må komponere med den EKTE `run_project`, og artefaktene gates som RÅ TEKST
|
||||
(Fase 4e):** alle 4d-testene ga `invoke` en stand-in som sluker `**kwargs`, så whitelisten kunne
|
||||
|
|
@ -213,16 +246,61 @@ Python ≥3.10. MAF (`agent-framework-core` 1.9.0). Pakkehåndtering: `uv`. To b
|
|||
et Foundry-deployment-navn i modell-mappet FØR noen klient bygges (`run.py` stempler provenance
|
||||
med det), så den kan ikke fullføre offline — **containeren trenger altså `PORTFOLIO_MODEL_MAP`
|
||||
eller et utfylt `data/model_map.json`, ikke bare et endepunkt** (målt her, ikke antatt).
|
||||
`Dockerfile`/`azure.yaml` KJØRES av ingen test (`docker build`/`azd deploy` er operatør-gatet), så
|
||||
rå-tekst er eneste tilgjengelige gate: `--platform linux/amd64` (målt påkrevd, spike §1.4 — uten
|
||||
det arver imaget byggerens arkitektur og bygger grønt lokalt mens det ikke kan starte i skyen) +
|
||||
ÉN kopi av startkommandoen (imagets `CMD` navngir `main.py`, `azure.yaml` har ingen
|
||||
`startupCommand`). **Nøkkel-sjekkene er LINJEFORANKRET, ikke delstreng:** `azure.yaml`s egen
|
||||
kommentar NAVNGIR `startupCommand` og `env` for å begrunne fraværet, så en substring-gate ville
|
||||
vært rød på prosaen den beskytter. Load-bearing MÅLT (`tests/test_hosting_loadbearing.py`), fem
|
||||
mutasjoner alle røde på riktig test og på INGEN annen (836 øvrige grønne hver gang): send
|
||||
`project_id` to ganger · whitelist et felt `run_project` ikke tar · fjern `bundle_dir` fra
|
||||
whitelisten · fjern `--platform linux/amd64` · gi `azure.yaml` en `startupCommand`-nøkkel.
|
||||
**Artefakt-halvdelen av denne raden er PENSJONERT 14.08** sammen med `Dockerfile`/`azure.yaml`
|
||||
(rå-tekst-gaten pinnet `--platform linux/amd64` + ÉN kopi av startkommandoen; to av radens fem
|
||||
mutasjoner traff nettopp den). Whitelist-halvdelen står URØRT. Load-bearing MÅLT
|
||||
(`tests/test_hosting_loadbearing.py`), de tre gjenværende mutasjonene alle røde på riktig test og
|
||||
på INGEN annen: send `project_id` to ganger · whitelist et felt `run_project` ikke tar · fjern
|
||||
`bundle_dir` fra whitelisten.
|
||||
- **Et tak som fyrer er IKKE en krasj — `BudgetExceeded` får sin EGEN kanal (429), og trippelen
|
||||
bæres som STRUKTUR (1b-køen, 14.08):** prosjektets første levende kjøring døde på
|
||||
`rounds limit=12 observed=13`, og den hostede flaten svarte `500 {error_type, error}` — altså
|
||||
nøyaktig det samme den sier når modell-endepunktet faller. **Beslutningen er S3.4-invarianten
|
||||
anvendt på transporten:** `budget_stop` ble holdt UTENFOR `stop_reason` fordi de to stoppene
|
||||
betyr motsatte ting, og å svare ressurs-utmattelse på krasj-kanalen gjør «det gikk ikke»
|
||||
uleselig på nøyaktig samme måte. **IKKE 200, og det er dét som skiller den fra `Rejection`:**
|
||||
en `Rejection` er en kjøring som KONKLUDERTE (og hører derfor i payloaden), mens et uttømt
|
||||
budsjett produserte ingen `proposal` i det hele tatt — en 2xx ville latt en automatisk kaller
|
||||
bokføre «analysert» for en kjøring som analyserte ingenting. **429 fordi betingelsen oppstår av
|
||||
en TILDELING** (`max_rounds`/`max_tokens` er whitelistede request-felt, og å heve dem er
|
||||
kallerens egen botemiddel), aldri av en serverfeil — derfor 4xx, ikke 5xx.
|
||||
`kind`/`limit`/`observed` legges ut som felt, ALDRI `str(exc)` (kø-(y): de beskriver ÉN ledger,
|
||||
og «hvilket tak bandt, og hvor langt forbi» er hele det operative spørsmålet); `error_type`
|
||||
holdes UTE — den nøkkelen tilhører feilkanalen, og en kaller som switcher på dens
|
||||
tilstedeværelse skal ikke finne den her. `budget_exhausted` er IKKE foldet inn i `outcome_type`,
|
||||
og kunne ikke vært det: `outbox.outcome_payload` er den ENE kopien av den forgreningen og tar
|
||||
`ValidatedProposal | Rejection`, som en uttømt kjøring ikke har noen av. **Ærlighets-grense,
|
||||
uttalt:** ingen `Retry-After` — å vente endrer ingenting, botemiddelet er et større tak eller å
|
||||
akseptere stoppet, og en header som lover tid ville vært en løgn. Load-bearing MÅLT
|
||||
(`tests/test_hosting_loadbearing.py`), fem mutasjoner alle røde mot HELE suiten, hver med sin
|
||||
egen signatur + grønn kontroll 867/4: detach armen (2 røde) · flat streng i stedet for struktur
|
||||
(1 rød — struktur-testen ALENE, altså rir den ikke på status-asserten) · ekko `limit` som
|
||||
`observed` (1 rød) · utvid armen til `Exception` (6 røde, inkl. 400-armen) · stemple
|
||||
`error_type` på budsjett-kroppen (1 rød). **500-armens vitne ble byttet, ikke slettet:** den
|
||||
eksisterende testen brukte `BudgetExceeded` som sin 500-prøve, så å bare legge til en ny arm
|
||||
ville etterlatt krasj-kanalen uten vitne — den bærer nå en ekte ikke-budsjett-`RuntimeError`,
|
||||
og er dét som holder den nye armen SMAL.
|
||||
- **Påstander flaten gjør om SEG SELV gates som rå tekst, linjeforankret (Fase 3, A5):** to påstander
|
||||
bodde i prosa der ingen test kunne se dem, og begge drev. (1) `env.template` sa at credential
|
||||
resolves via `DefaultAzureCredential` — den har ALDRI gjort det; gaten leser de klassene
|
||||
`backends.py` faktisk konstruerer **fra selve tilordningslinja**, ikke fra modulen, fordi
|
||||
kommentarene NAVNGIR `DefaultAzureCredential` fire ganger for å begrunne hvorfor den ikke brukes —
|
||||
en fil-bred substring-gate ville vært rød på nøyaktig den prosaen den beskytter (repoets
|
||||
08-09-klasse, fjerde gang). (2) README-ens wheel-filnavn bærer versjonen bygget stempler på fila,
|
||||
så en versjonsbump ville stille etterlatt en publisert install-kommando som peker på en fil som
|
||||
ikke finnes. **Hver positiv assert er paret med en KONTROLL** på at det søkes etter noe som finnes
|
||||
— en ekstraktor som stille finner null lager en gate som bare kan bli grønn. Load-bearing MÅLT
|
||||
(`tests/test_public_surface_claims_loadbearing.py`) mot HELE suiten, begge røde på riktig test og
|
||||
på INGEN annen: gjeninnfør credential-påstanden (2 røde, 844 grønne) · la wheel-filnavnet drifte
|
||||
(1 rød, 845 grønne). Bumpen selv var den tredje målingen — `pyproject` 1.0.0 → 1.1.0 gjorde
|
||||
README-gaten rød alene, FØR README ble rettet. **`repo-standard`-gaten kan IKKE verifisere denne
|
||||
fasen:** den var OK/20 sjekker før arbeidet startet, og `RELEASE-STALE` er strukturelt blind for
|
||||
repo med null utgivelser (org-ops hovedbok #18). Bevisene er Forgejo-APIet, filinnholdet og
|
||||
ren-klon-kjøringen.
|
||||
- **GOVERNANCE er en LENKE, aldri en kopi (org-ops D11):** én kanonisk `GOVERNANCE.md` bor i
|
||||
`repo-standard` og hvert repo lenker den fra README. Å skrive vår egen ville gjort oss til kopi
|
||||
nr. 12 av en fil D11-bølgen holder på å rydde vekk. Bus-faktor 1 står uttalt i den kanoniske
|
||||
teksten, ikke i vår.
|
||||
- **Stoppkriterier + budsjett-tak påkrevd ved oppstart** (fail-fast, aldri ubegrenset loop).
|
||||
- **Group Chat maker-checker** som debatt-default (IKKE Magentic, som er eksperimentell).
|
||||
- **To falsifiserere, samme kandidat (Steg 3/4, målbilde §2/§6):** den deterministiske validatoren
|
||||
|
|
@ -331,6 +409,26 @@ Python ≥3.10. MAF (`agent-framework-core` 1.9.0). Pakkehåndtering: `uv`. To b
|
|||
S3.2); **D7-speiling ÅPEN.** Load-bearing MÅLT (`tests/test_s40_cost_baseline_loadbearing.py`), seks
|
||||
mutasjoner alle røde: detach avstemmings-stagen · detach magnitude-toleransen · detach road-wiringen ·
|
||||
detach bundle-wiringen · ignorer det injiserte cap-registeret · gjør den valgfrie loaderen tolerant.
|
||||
**En UFORANKRET kjøring sier det nå — og BEGGE utsagn stammer fra kjøringens ENE oppslag, aldri
|
||||
en andre lesing av bundelen (21.08):** `ProvenanceStamp.cost_baseline_anchored` er PÅKREVD uten
|
||||
default (begge defaults lyver: `True` lar en glemsom konstruktør påstå en ankring som ikke skjedde,
|
||||
`False` underrapporterer en ekte — en binær kjensgjerning om en falsifiserer har ingen ærlig
|
||||
default), og `DryRunReport` bærer det samme fordi en dry-run stopper før noe stempel finnes.
|
||||
`run.cost_baseline_notice(anchored)` er ENESTE renderer, tar den alt oppløste BOOLEANEN, og
|
||||
returnerer `None` når kjøringen ER forankret — omisjon, aldri en tom rad (`announce`-regelen).
|
||||
**IKKE foldet inn i `mandate.announce`, og det er en MÅLING:** den fyrer kun med `--mandate`, så
|
||||
nettopp de bare bundle-dry-runsene defekten ble målt på ville fortsatt sagt ingenting — og den
|
||||
renderes FØR `run_project`, altså før noen har oppløst baselinen. Utboksen trengte ingen endring
|
||||
(`write_proposal` dumper hele stempelet). Ankeringen forblir VALGFRI: dette er synlighet, ikke en
|
||||
ny nekt, og golden-transkriptet er byte-uendret fordi demoen kjører en base som HAR fila.
|
||||
Portefølje-armen er DEFENSIV og uttalt (ingen referanse-prosjekt setter `bundle_dir`, så den er
|
||||
unåbar i dag — `budget_stop`-presedensen; testen driver en crafted `PortfolioResult`). Load-bearing
|
||||
MÅLT (`tests/test_baseline_visibility_loadbearing.py`), seks mutasjoner alle røde mot HELE suiten +
|
||||
grønn kontroll 885/5: konstant stamp-wiring (3 røde) · konstant dry-run-wiring (1 rød) · detach
|
||||
dry-run-printen (1 rød) · renderer returnerer alltid linja (2 røde — inkl. den forankrede
|
||||
kontrollen, altså er omisjonen selv gatet) · detach full-run-printen (1 rød) · detach
|
||||
portefølje-printen (1 rød). Det PÅKREVDE feltet tvang fem eksisterende test-konstruktører til å ta
|
||||
stilling — det er egenskapen, ikke friksjonen.
|
||||
- **Globalt token-tak håndheves FØR kall, aldri bare etterpå (S3.4, F10):** `PortfolioBudget` +
|
||||
`PortfolioMeter` er ÉN ledger over hele porteføljepasset (og — seedet av `read_spend` — på tvers
|
||||
av pass), mens per-run `Budget`/`TokenMeter` er uendret. Taket har tre tenner, med hver sin jobb:
|
||||
|
|
@ -386,6 +484,35 @@ Python ≥3.10. MAF (`agent-framework-core` 1.9.0). Pakkehåndtering: `uv`. To b
|
|||
`run.py` · endre avrundingsmodus · la `realize` gå utenom `to_ore`. **Ærlighets-grense:**
|
||||
`sum_claimed_saving_nok` (`run.py:_aggregate`) er BEVISST urørt — et float-NOK-rapportfelt som
|
||||
aldri kvantiseres og aldri sammenlignes mot ledgeren, altså utenfor ordens-defekten.
|
||||
- **En BETALT test får sin EGEN opt-in, og instrumentet bevises GRATIS (Fase 1b, siste trinn):**
|
||||
`tests/test_full_run_live.py` kjører hele `run_project`-stien mot et ekte Foundry-deployment, og
|
||||
gates på **fire** ting — de to Foundry-variablene, `PORTFOLIO_MODEL_MAP`, og et TREDJE, distinkt
|
||||
`PORTFOLIO_LIVE_FULL_RUN` lest på **truthiness** (4b-invarianten). **Den tredje variabelen er
|
||||
load-bearing, ikke pynt:** `test_foundry_profile_live.py` (klient-probe) og `test_portfolio_live.py`
|
||||
(fan-out) gatet på nøyaktig SAMME to variabler, så å gjenbruke det paret ville betydd at en
|
||||
operatør som eksporterer dem for den BILLIGE ett-ords-proben også fyrer den dyre fullkjøringen —
|
||||
altså at måleprotokollens stige («bevis så mye som mulig før det dyre trinnet, så en feil er
|
||||
attribuerbar») kollapser til ett trinn. **MÅLT:** med begge Foundry-variablene satt SKIPPET den
|
||||
dyre, og den billige var grønn. **Regelen gjelder HVER betalt arm, ellers er den ingen regel:**
|
||||
`test_portfolio_live.py` passerer ingen `client_factory` og er derfor selv en betalt kjøring — den
|
||||
fyrte på to-variabel-paret fra et bart `uv run pytest`, og ble gatet på den TREDJE variabelen i
|
||||
samme slengen. Å la den stå ville gjort denne raden halvt usann den dagen den ble skrevet; en
|
||||
invariant som beskriver én av to armer er en påstand flaten gjør om seg selv uten dekning, som er
|
||||
nøyaktig Fase 3-klassen. Den billige klient-proben beholder to-variabel-gaten med vilje — den ER
|
||||
det billige trinnet. `PORTFOLIO_MODEL_MAP` er med av en annen grunn — attribusjon: uten
|
||||
den feiler kjøringen av en KONFIGURASJONS-årsak som ser ut som en modell-feil.
|
||||
**Asserten bor i ÉN kopi** (`conftest.assert_full_run_contract`, kø-(p)) og er smal med vilje:
|
||||
fraværet av `{run_id}-parse-failures.json` (økt 35-invarianten «filens tilstedeværelse er
|
||||
signalet») + at `validator_decision` avgjorde. **En `rejected` BESTÅR** — påstanden som felles er
|
||||
at det strukturerte skjemaet ER akseptert av det levende endepunktet, ikke at modellen resonnerer
|
||||
godt; å kreve `validated` ville vært en modell-dømmekraft-påstand ingen enkelt kjøring kan bære.
|
||||
**Iron Law uten å betale to ganger:** et betalt kall kan ikke kjøres rødt-så-grønt, så
|
||||
diskrimineringen bevises OFFLINE av `tests/test_live_full_run_contract.py` — to armer over samme
|
||||
helper (én parse-feil → kontrakten MÅ feile; alle parser → MÅ passere). Load-bearing MÅLT, to
|
||||
mutasjoner med hver sin distinkte signatur: detach artefakt-sjekken (T1 rød ALENE — kontrakten
|
||||
degraderer da til `test_portfolio_live.py`s `len(runs)==1`-klasse) · raise ubetinget (T2 rød
|
||||
ALENE — den motsatte vakuiteten, en live-test som bare kan bli rød). **Det betalte kallet er
|
||||
MÅLINGEN, aldri beviset på at måleinstrumentet virker.**
|
||||
- **Kostnadsdisiplin:** utvikle primært på lokal profil (gratis); Foundry/Azure (privat tenant finnes) kun til målrettet, minimal verifisering; billigste modeller + små syntetiske data + harde token-tak. Ingen tunge test-kjøringer.
|
||||
- **Offline simulering = primært metode-bevis (kostnadsdrevet, erstatter §11.8):** operatøren kjører
|
||||
IKKE MAF mot ekte modell (verken Azure/Foundry eller Ollama — API for begge repoene er for kostbart
|
||||
|
|
@ -510,6 +637,482 @@ Python ≥3.10. MAF (`agent-framework-core` 1.9.0). Pakkehåndtering: `uv`. To b
|
|||
import), eksempelet er gyldig pipeline-input inkl. `FeedbackContract` (RØD på skjema-/kontrakt-drift,
|
||||
på en throwaway-kopi — aldri den git-tracked fixturen), og sim-ens markør følger artefakt-fila (RØD i
|
||||
det øyeblikk personaen re-inlines).
|
||||
- **Den råe svarteksten fanges i en KALLER-EID SINK, ikke i en returverdi (Fase 1b, funn 1):**
|
||||
`generate._fetch_parsed` kastet hvert uparsebart modellsvar i `except: continue`, så prosjektets
|
||||
første levende kjøring brant tolv runder på formatfeil og etterlot **null tegn** av det modellen
|
||||
faktisk sa — enhver videre betalt kjøring ville vært gjetning. **HVOR teksten overflates er avgjort
|
||||
av en MÅLING, ikke av symmetri med Steg 5:** `meter.tick_round()` raiser `BudgetExceeded` INNE i
|
||||
`_fetch_parsed`, og uten mandat fanger ingen den (`run.py`s ene `except BudgetExceeded` er
|
||||
mandat-armen) — så på nøyaktig den stien fangsten finnes for, RETURNERER `generate_via_llm`
|
||||
ingenting. Et felt på `GenerationResult` (Steg 5-formen) er derfor blindt for den, og et
|
||||
outbox-artefakt skrevet ETTER kjøringen likeså. Sinken speiler i stedet `meter`: en kaller-eid
|
||||
akkumulator løkka muterer, hvis innhold kalleren holder uansett hvordan løkka endte. Steg 5s
|
||||
«returverdi, ikke out-parameter» gjelder en verdi som NÅR kalleren; her gjør den ikke det, og å
|
||||
kopiere regelen blindt ville gjenoppbygd defekten ett lag opp. Artefaktet
|
||||
`{run_id}-parse-failures.json` skrives fra en **`finally`**, ikke `except BudgetExceeded` — enhver
|
||||
exception ut av genereringen ødelegger samme bevis, og en liste over exception-typer er en liste
|
||||
som blir foreldet. **Teksten er VERBATIM** (en forkortelse gjør beviset om til en parafrase), og
|
||||
fila skrives KUN når noe faktisk feilet, så dens tilstedeværelse ER signalet. Byte-determinisme
|
||||
påstås IKKE for dette ene artefaktet — innholdet er en levende modells prosa. Load-bearing MÅLT
|
||||
(`tests/test_parse_failure_capture_loadbearing.py`), seks mutasjoner alle røde mot HELE suiten +
|
||||
grønn kontroll 859/4: detach fangsten (3 røde) · flytt skrivingen ut av `finally` (1 rød, KUN
|
||||
budsjett-testen) · detach run-wiringen (2 røde, generate-testen grønn) · skriv artefaktet alltid
|
||||
(kontrollen + den eksisterende `a5`-inerthetstesten) · trunker teksten til 40 tegn (3 røde) ·
|
||||
trunker til 100 tegn slik at sentinelen OVERLEVER (1 rød — verbatim-asserten alene, den skarpe
|
||||
diskriminatoren). Ærlighets-grense: `_charge_usage` kan raise FØR parse, og et svar tapt der er
|
||||
ikke en parse-feil og fanges ikke.
|
||||
- **Proposeren får en GRAMMATIKK, og skjemaet er DERIVERT + fail-closed (Fase 1b, funn 1b):**
|
||||
`generate_via_llm` sender `options={"response_format": proposal_response_format()}` på hvert
|
||||
genererings-kall. **Formen er MÅLT, ikke valgt:** `ChatOptions.response_format` tar
|
||||
`type[BaseModel] | Mapping`, og BEGGE profiler ærer den — LOCAL
|
||||
(`OpenAIChatCompletionClient`) sender en Mapping ordrett til Chat Completions, AZURE
|
||||
(`FoundryChatClient` → `RawFoundryChatClient` → `RawOpenAIChatClient`) konverterer SAMME
|
||||
envelope til Responses-APIets `text.format`. **Klassen er AVVIST på bevis:** gitt en klasse
|
||||
konverterer klienten med `type_to_response_format_param`, som (målt) emitterer `minimum` /
|
||||
`exclusiveMinimum` / `minItems` / `prefixItems` og et `assumptions`-node hvis
|
||||
`additionalProperties` er et SKJEMA — fire ting Azures publiserte subset utelukker
|
||||
(Learn: «Unsupported type-specific keywords» + `additionalProperties: false` i hvert objekt).
|
||||
Vår egen mapping er eneste måte å styre hva som når tråden. **Å stripe beskrankningene koster
|
||||
ingenting:** skjemaets jobb er FORM, validatorens jobb er VERDIER — `minItems`/`gt=0` gjenreises
|
||||
av pydantic i `_parse_ir` og av `validate_proposal`. **Skjemaet DERIVERES fra `SavingsProposal`**
|
||||
(`strict_json_schema`), aldri håndskrevet: en andre kopi av en form som alt bor i `ir.py` drifter
|
||||
stille, og modellen ville fortsatt blitt bestilt for den gamle. **`assumptions` KAN IKKE bare
|
||||
droppes, og det er en MÅLING:** feltet er det ene uttrykksløse (fri-form map av 2-tupler), men
|
||||
`validator._monte_carlo` faller tilbake på `item.unit_cost` for hver kode uten bånd — uten bånd
|
||||
i det hele tatt er alle 512 samples IDENTISKE og P10 == P50 == P90. Den stokastiske
|
||||
falsifisereren ville gått inert mens den fortsatt rapporterte persentiler: repoets kardinalklasse
|
||||
(en gate som bare kan bli grønn). Derfor bærer WIRE-en et array av navngitte entries og
|
||||
`_parse_ir` folder det tilbake til IR-ens map — **additivt, aldri erstatning** (map-formen
|
||||
parser uendret; alle scriptede svar i suiten og golden-transkriptet bruker den). Sanitiseren er
|
||||
**fail-closed** (`StructuredOutputUnsupported`) på `prefixItems`/`oneOf`/`allOf`/fri-form map
|
||||
uten deklarert override — validering, ALDRI reparasjon (speiler `write_concept_file`).
|
||||
Prompt-linja «Respond with ONLY a JSON object» + parse-retry + funn-1-fangsten står URØRT: en
|
||||
leverandør som ignorerer `response_format` må fortsatt få beskjed, og backstoppen er poenget.
|
||||
Load-bearing MÅLT (`tests/test_structured_output_loadbearing.py`), seks mutasjoner alle røde +
|
||||
grønn kontroll 864/4: detach wiringen (1 rød) · detach sanitiseren (3 røde) · dropp
|
||||
`assumptions` fra skjemaet (1 rød) · fail-closed → stille reparasjon (1 rød) · detach
|
||||
normaliseringen (3 røde) · erstatning i stedet for tillegg (2 røde — T5 PLUSS
|
||||
golden-transkriptet, et uavhengig vitne). **T3 ble skrevet VAKUØS først** (repoets 08-09-klasse,
|
||||
sjette gang): den påsto å bli rød når `assumptions` forsvant fra skjemaet, men den scriptede
|
||||
klienten ignorerer skjemaet — påstanden ble bevist usann av M3 og testen fikk en DIREKTE
|
||||
assert på skjemaet. **ÆRLIGHETS-GRENSE, UTTALT:** ingen betalt kjøring er gjort, så at det
|
||||
emitterte skjemaet ER akseptert av det levende endepunktet er IKKE verifisert — testen beviser
|
||||
konformitet med det DOKUMENTERTE subsettet, ikke aksept. Ollamas oppførsel på
|
||||
`response_format` er likeledes uverifisert.
|
||||
- **Overleverings-pakka ER `git archive HEAD`, aldri en kuratert kopi (Fase 5):**
|
||||
`scripts/make-handover-package.sh` bygger én zip en ekstern organisasjon deployer uten å klone
|
||||
repoet. **Tracked files only er hele eksponerings-kontrollen** — `STATE.md`, `*.local.md` og
|
||||
`.env` er gitignorert, så de KAN ikke komme inn; et filter vedlikeholdt i skriptet ville vært den
|
||||
andre kopien av den regelen, og den andre kopien er den som drifter (kø-(p)). Mottakeren får altså
|
||||
HEAD selv. Versjonen LESES fra `pyproject.toml` — et hardkodet tall her ville råtnet ved neste bump
|
||||
nøyaktig som README-ens wheel-filnavn gjorde (Fase 3). `DEPLOY.md` ligger i treet og blir dermed
|
||||
med i arkivet av seg selv; den bærer mottakerens tre første spørsmål — hvem gjør hva
|
||||
(plattform-operatør / bestiller / fagperson), prosessen ende-til-ende, og **hvorfor det ikke
|
||||
finnes et chat-grensesnitt** (flaten er `POST /invocations`, og `as_agent()` er bevisst vraket
|
||||
fordi validator, baseline-forankring, checker-gate og ledger ligger UTENFOR grafen — et chat-lag
|
||||
ville rutet forespørsler rundt nøyaktig det som gjør svaret etterprøvbart). Den navngir også det
|
||||
4e målte deploy-kravet som ingen rad hadde skrevet ned: pakket `model_map.json` bærer
|
||||
`REPLACE-WITH-*`, så uten `PORTFOLIO_MODEL_MAP` starter tjenesten, svarer på `/readiness` og
|
||||
feiler HVER invocation. Gaten er `tests/test_handover_package_loadbearing.py`, og
|
||||
DEPLOY.md-asserten er LINJEFORANKRET: `PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT` INNEHOLDER
|
||||
`FOUNDRY_PROJECT_ENDPOINT`, så en delstreng-assert på det injiserte navnet ville vært oppfylt av
|
||||
vårt eget (repoets 08-09-klasse, femte gang).
|
||||
- **Overleveringen er KUN kjørbar Python, og fraværet er FJERNING — ikke filtrering (14.08,
|
||||
operatørdirektiv etter ekstern test):** `Dockerfile` og `azure.yaml` er slettet fra TREET.
|
||||
**Sømmen er valgt av den eksisterende invarianten, ikke av smak:** pakka ER `git archive HEAD`, så
|
||||
å ekskludere filene fra arkivet ville krevd en kurerings-mekanisme (skript-filter eller
|
||||
`export-ignore`) — den andre kopien av «hva mottakeren får», fri til å drifte fra HEAD, altså
|
||||
nøyaktig kø-(p)-regelen raden over finnes for. Å beholde dem som «opt-in» ville ikke oppfylt
|
||||
direktivet i det hele tatt. Fjerning holder arkivet ukurert OG gjør fraværet til en egenskap ved
|
||||
HEAD, som er det eneste en gate kan måle. **De to gatene som pinnet flaten er håndtert BEVISST,
|
||||
aldri stille svekket:** 4e-rå-tekst-gaten (`--platform linux/amd64` + ÉN kopi av startkommandoen)
|
||||
er SLETTET med et notat der den sto — en gate som pinner en fjernet flate kan bare bli grønn — og
|
||||
handover-gatens `_REQUIRED_MEMBERS` er ikke bare fratatt de to navnene, men erstattet av en
|
||||
POSITIV fraværs-assert; å kun slutte å KREVE dem ville gitt en gate som ikke kan skille «fjernet»
|
||||
fra «shippes fortsatt». Matchingen skjer på arkiv-MEDLEMSNAVN, ikke på prosa (dokumentene må
|
||||
kunne forklare at ingen image shippes — repoets 08-09-klasse, sjette gang), og dokument-gaten
|
||||
forbyr kommando-FRAGMENTER (`docker build`, `azd deploy`), ikke ordet. **Startkommandoen har nå
|
||||
ÉN kopi igjen — DEPLOY.md-ens `python main.py`** — og den navngir inngangen subprosess-testen
|
||||
faktisk kjører. **Ærlighets-grense, uttalt:** azd/hosted-agent-stien finnes ikke lenger i pakka;
|
||||
hvordan prosessen driftes er mottakerens valg. `git archive HEAD` leser HEAD, ikke arbeidstreet,
|
||||
så gaten er ekte men forsinket med én commit (funn 35). Load-bearing MÅLT
|
||||
(`tests/test_handover_package_loadbearing.py`).
|
||||
- **Sporing er OPT-IN, og «av» betyr at MAF ALDRI kalles (U14, økt 55):** `PORTFOLIO_OTEL` leses på
|
||||
**truthiness** (4b-regelen) og er ENESTE bryter; uten den kalles `configure_otel_providers` ikke i
|
||||
det hele tatt — spans LAGES fortsatt (`ENABLE_INSTRUMENTATION` defaulter `True`,
|
||||
`observability.py:697`) og kastes, så ingenting KAN forlate prosessen. Et kall med tom
|
||||
exporter-liste ville derimot installert providere og lest hver `OTEL_EXPORTER_OTLP_*` i det
|
||||
omkringliggende miljøet — «av» må være fravær av kall, ikke kall uten innhold. **To regler er
|
||||
MÅLT, ikke valgt** (`observability.py:849` bygger exporter-lista i fast rekkefølge: (1) env-avledede
|
||||
OTLP-exportere UBETINGET, (2) de innsendte, (3) `ConsoleSpanExporter()` — default-sink **stdout** —
|
||||
når `enable_console_exporters` er sann fra argument ELLER `ENABLE_CONSOLE_EXPORTERS`): (a)
|
||||
`enable_console_exporters=False` sendes EKSPLISITT i BEGGE moduser, ellers gir en operatør med den
|
||||
variabelen eksportert et span-dump på stdout — nøyaktig det S6 målte som ødeleggende for
|
||||
golden-transkriptet; (b) `console` NEKTER når en OTLP-endepunkt-variabel finnes, fordi steg (1)
|
||||
ville lagt til en nettverks-exporter ordet «console» lover ikke er der. **Validering, ALDRI
|
||||
reparasjon** — vi fjerner ikke operatørens miljøvariabel bak ryggen på dem
|
||||
(`write_concept_file`-regelen); nekten NAVNGIR variabelen. `otlp` uten deklarert endepunkt nektes
|
||||
også: providere med ingenting å eksportere til er en kjøring som SER sporet ut og ikke er det. En
|
||||
ukjent verdi nektes ved navn, aldri stille fallback til av. **`tracing_notice` er ENESTE renderer**,
|
||||
tar den alt oppløste `TracingSetup` og returnerer `None` når sporing er av — omisjon, aldri tom rad
|
||||
(`announce`-regelen), og her bærende utover stil: demoens pinnede stderr er FIRE linjer. **Tre
|
||||
kallsteder, ikke ett** (`run.main`, `simulation.main`, `hosting.main`): demoen er et skriptet bevis,
|
||||
ikke produktet, og en søm bare demoen når ville latt de to inngangene en virksomhet faktisk kjører
|
||||
være usporbare. **OTLP-exporter-PAKKENE er BEVISST ikke deklarert** (egress + grpc/protobuf-vekt i
|
||||
et publisert wheel; MAF raiser selv en `ImportError` som navngir pakka) — uttalt ærlighets-grense.
|
||||
**`PLAN_CREATED`/`REPLANNED`/`PROGRESS_LEDGER_UPDATED`-eventene planen navngir er IKKE bygget:**
|
||||
de hører til utforskningssløyfa (U4) som ikke finnes ennå, og en emitter skrevet før kallstedet er
|
||||
en form gjettet i stedet for målt. Load-bearing MÅLT
|
||||
(`tests/test_tracing_loadbearing.py`), ni mutasjoner alle røde mot HELE suiten + grønn kontroll
|
||||
943/5: exporteren tar sin stdout-default (4 røde) · `enable_console_exporters` overlatt til miljøet
|
||||
(3 røde — inkludert den ATFERDSMESSIGE, som kjører demoen med variabelen eksportert; uten den ville
|
||||
raden bare vært en keyword-assert) · detach console-nekten (4 røde) · detach otlp-endepunkt-nekten
|
||||
(1 rød) · ukjent modus faller stille til av (2 røde) · «av» kaller MAF likevel + renderer returnerer
|
||||
alltid en linje (9 røde, hvorav TRE i tester som fantes fra før — `test_golden_transcript` sin
|
||||
fire-linjers stderr og `test_portfolio_cli_offline`s stille-pass — altså er omisjonen gatet av
|
||||
uavhengige vitner) · detach demo-wiringen + detach CLI-wiringen (5 røde) · detach hosting-wiringen
|
||||
(1 rød, KUN subprosess-testen — P4-presedensen).
|
||||
- **Utforskningssløyfa er en MANDAT-FORMER, og de tre garantinivåene er strukturelle (U4+U13
|
||||
synkron, økt 56):** `explore.py` legger en Magentic-manager OVER den normative sløyfa — `prompt +
|
||||
kunnskapsbaser → Mandate → run_project(mandate=…)` UENDRET, Steg 3s maker-checker urørt (commons-eid
|
||||
og normativ). Manageren velger VEI; det som forlater friheten er `mandate.Mandate`, aldri et forslag.
|
||||
**Nivå 1** = `quick_validate`-verktøyet (SAMME `validate_proposal`, SAMME baseline, men rådgivende —
|
||||
når ALDRI provenance); **nivå 2** = pipelinen som stempler; **nivå 3** = skriverettigheter, som kun
|
||||
pipelinen har. `explore()` skriver INGENTING. **Mandatet bygges fra hypotesiserens MERKEDE turer**
|
||||
(`HYPOTHESIS: {"label","rationale"}`), aldri fra sluttsvaret — sluttsvaret er RÅTT per design, og en
|
||||
parser på det ville gjort det til et forslag. Markøren er dét som gjør fail-closed mulig: en umerket
|
||||
tur er ikke en påstand (ingen stillhet å lukke), mens en MERKET-men-uleselig linje raiser
|
||||
(`write_concept_file`-regelen). **Frø-approaches bevares ALLTID og FØRST** — også når sløyfa fant
|
||||
ingenting og også ved stopp (§ C.6 dør 1 er en bevaringsregel, ikke en belønning for å bli ferdig).
|
||||
**Tre kanaler, aldri én:** tokens OG runder raiser `BudgetExceeded` (rundene som
|
||||
`kind="exploration_rounds"`, oversatt av VÅRT lag fordi orkestreringen MÅLT ikke raiser ved sitt eget
|
||||
rundetak — den returnerer en kanonisk assistent-melding som ved transporten er uskillbar fra suksess),
|
||||
mens alt semantisk er en VERDI i `stop` (S3.4-splitten: utmattelse og utfall er ikke samme sak).
|
||||
Diskriminatoren mellom «nådde taket» og «ble kappet av taket» er SISTE ledgers
|
||||
`is_request_satisfied` — samme felt orkestratoren selv forgrener på (`:1106`) — aldri
|
||||
termineringsmeldingen, som er en inline f-string (`:1253`) uten konstant å pinne mot og som en modells
|
||||
eget sluttsvar kan inneholde. **`speaker_known` sjekkes FØRST og slår alt annet:** en `next_speaker`
|
||||
uten treff gir stille sluttsvar med NULL deltakerarbeid (`:1128-1131`), altså et plausibelt svar
|
||||
ingen jobbet for (E2-klassen) — sløyfas egne funn holdes da tilbake, frøene ikke.
|
||||
**Kontrakten nekter tre ting ved konstruksjon:** hvert av seks felt er PÅKREVD uten default (MAF
|
||||
defaulter `max_round_count`/`max_reset_count` til ubegrenset, så et utelatt felt faller ikke tilbake
|
||||
til noe forsiktig, men til dét `method-spec` §8 forbyr); `max_reset_count=0` nektes — **MÅLT**, ikke
|
||||
resonnert: `reset_count >= max_reset_count` mot en teller som starter på 0 gjør at kjøringen
|
||||
terminerer FØR første runde med kun `facts`+`plan`, null ledger-events og «maximum reset count», altså
|
||||
en utforskning som utforsket ingenting, forkledd som en stall som aldri skjedde (`max_stall_count=0`
|
||||
er derimot LOVLIG — strengt `>` gjør 0 til «reset ved første stallede runde»); og
|
||||
`max_plan_revisions>0` med `enable_plan_review=False` nektes (en cap på en hendelse som ikke kan skje).
|
||||
`max_plan_revisions` finnes fordi A3 MÅLTE at en `revise` koster 2 manager-kall, **null** ledger-kall
|
||||
og **null** runder og så spør PÅ NYTT — under rundetaket alene er en alltid-reviderende ekspert
|
||||
ubundet forbruk under vakter som alle ser tilfredse ut. Ved cap: typet stopp, ALDRI en påtvunget
|
||||
approve (repair av et menneskes beslutning er den verste sorten). **U14s tre utsatte events er
|
||||
landet** (`plan_created`/`replanned`/`progress_ledger_updated` som span-events på ÉN
|
||||
`exploration`-span) — emisjon er UBETINGET og «av» betyr at OTel kaster dem, samme form MAFs egen
|
||||
instrumentering alt har; en flagget emitter ville vært en andre oppløsning av regelen `tracing.py`
|
||||
eier. Uttalt ærlighets-grense: eventene registreres når event-strømmen foldes, så REKKEFØLGEN er
|
||||
tro og tidsstemplene er ikke øyeblikkene manageren handlet.
|
||||
**Load-bearing MÅLT** (`tests/test_explore_loadbearing.py`, 32 tester), tolv mutasjoner alle røde mot
|
||||
HELE suiten + grønn kontroll 975/5: detach rundetak-oversettelsen (1 rød) · test rundetaket FØR
|
||||
tilfredsstillelse (1 rød — den motsatte feilen, som gjør en fullført utforskning til en budsjettfeil) ·
|
||||
detach ukjent-taler-sjekken (1) · `BudgetMiddleware` av manageren, deltakerne beholder den (1) ·
|
||||
detach revisjons-capen (1) · dropp frø-bevaringen (3) · la en ukjent-taler-kjøring levere funnene
|
||||
videre (1) · detach alle tre span-events (1) · tillat `max_reset_count=0` (1) · gjør en merket-men-
|
||||
uleselig hypotese tolerant (1) · la `read_bundle` skrive i basen den leser (1) · send spans til OTels
|
||||
default-sink (2). **TO av dem FALSIFISERTE testen først, og begge er repoets vakuøs-gate-klasse:**
|
||||
(i) skrivefrihets-testen drev kun `explore()`, men en `ScriptedChatClient` returnerer TEKST og
|
||||
emitterer aldri et verktøykall — så ingen scriptet kjøring når en verktøykropp, og hele lesesømmen
|
||||
(eneste sted en skriving realistisk kan komme fra) lå utenfor gaten; testen kaller nå hvert verktøy
|
||||
DIREKTE. (ii) stdout-testen brukte `capsys`, men `ConsoleSpanExporter`s `out`-default bindes når
|
||||
`opentelemetry.sdk.trace.export` FØRST importeres — under pytest er det stdout ved COLLECTION, som
|
||||
`capsys` aldri ser; spans lå faktisk på stdout mens asserten var grønn. Dét er ikke en test-quirk å
|
||||
omgå, det er nøyaktig faktumet U14 finnes for, og arven er P4-presedensen: **subprosessen er
|
||||
målingen**. Begge armene kjøres nå i et barn (av: null trace-data noe sted; `PORTFOLIO_OTEL=console`:
|
||||
spanet + `progress_ledger_updated` på **stderr** og stdout tomt), med `EXPLORATION-OK` på stderr som
|
||||
kontroll — uten den ville «stdout var tomt» vært like sant om et barn som krasjet ved import.
|
||||
**Ærlighets-grenser, uttalt:** multi-base-dispatch (`Approach.bundle_id`, § C.7) venter til
|
||||
`run_project` tar mer enn én `bundle_dir` — å shippe feltet før konsumenten er en form gjettet i
|
||||
stedet for målt; `quick_validate`-dommene hypotesiseren så bor ikke i `ExplorationResult`, de er nivå
|
||||
1 og hører hjemme i `{run_id}-exploration.json` som CLI-wiringen skriver; utforskningsrollene løses
|
||||
via `resolve_model`s `default`-fallback til en operatør mapper dem eksplisitt; at en LEVENDE modell
|
||||
kaller verktøyene er ikke bevist offline (samme klasse som structured-output-grensen).
|
||||
- **Utforskningens KALLSTEDER: sporet er kaller-eid, og whitelisten ble en TREDELING (økt 57):**
|
||||
`--explore "<prompt>" --explore-config FILE` i `run.py`, `explore_prompt` + `explore_contract` på
|
||||
den hostede flaten, og `simulate_exploration` som et TREDJE sim-scenario — alle opt-in, alle over
|
||||
den uendrede sløyfa. **`ExplorationTrace` er en KALLER-EID akkumulator (funn-1-sinken, ett lag
|
||||
opp), og formen er tvunget av en måling, ikke valgt:** `explore()` raiser `BudgetExceeded` på
|
||||
rundetaket og tokentaket fyrer fra middleware midt i løpet — på BEGGE stier konstrueres aldri et
|
||||
`ExplorationResult`, mens § C.2 krever at artefaktet er lesbart «uansett hvilken vakt som fyrte».
|
||||
Steg-5-regelen («returverdi, ALDRI en out-parameter») styrer en verdi som NÅR kalleren; her gjør
|
||||
den ikke det, og å kopiere regelen blindt ville gjenoppbygd defekten den ble skrevet mot.
|
||||
`ExplorationResult.ledger_log`/`.plan_reviews` BYGGES FRA akkumulatoren (`tuple(trace.ledger)`),
|
||||
aldri ved siden av — to beholdere om ett faktum er kø-(p). `{run_id}-exploration.json` skrives fra
|
||||
en **`finally`** (`write_parse_failures`-presedensen) via `explore.trace_payload` → `outbox`s
|
||||
plain-mapping-skriver (RAW-laget forblir MAF-fritt); **`completed` er et EGET påkrevd felt**, fordi
|
||||
en `stop: null` som betyr BÅDE «avsluttet normalt» og «vi fikk aldri vite» er stillheten
|
||||
`cost_baseline_anchored` ble påkrevd for å lukke. **Åtte CLI-nekter, alle ved navn**, hvorav to
|
||||
bærer en beslutning: (i) `--explore` + `--mandate` er TO KILDER TIL ETT MANDAT og NEKTES, aldri
|
||||
slås sammen — `explore()` tar objective fra prompten og hardkoder `allow_own_proposals=True`, så
|
||||
komposisjon ville stille overskrevet tre felt operatøren skrev selv; nekten NAVNGIR
|
||||
biblioteksdøra (`seed_approaches`), fordi § C.6 dør 1 er et ekte behov flaten ikke betjener.
|
||||
(ii) `enable_plan_review=true` nektes på BEGGE flater FØR `explore()` kalles, og det er en
|
||||
TYPE-måling: `ExplorationError` er en `RuntimeError` og ligger utenfor `main()`s
|
||||
`(ValueError, FileNotFoundError, ValidationError)`-tuppel og utenfor hostings 400-arm, så å
|
||||
overlate den til sløyfa ville gitt traceback på CLI-en og 500 — krasj-kanalen — på HTTP.
|
||||
(`TracingConfigError` er derimot en `ValueError`; 400-armen dekket den alt.) Etter nektene er hver
|
||||
konfig-formet `ExplorationError` UNÅBAR fra begge inngangene ved konstruksjon; det som fortsatt kan
|
||||
slippe ut (uleselig merket hypotese, uttømt budsjett) er RUN-en som feiler, ikke kalleren som tar
|
||||
feil. **Hostings whitelist er nå `_REQUIRED` / `_OPTIONAL` / `_CONSUMED`:** utforskningsfeltene er
|
||||
IKKE `run_project`-parametre, så Fase 4e-beviset fikk en NEGATIV halvdel — hvert videresendt felt
|
||||
MÅ finnes i `inspect.signature(run_project)`, hvert konsumert felt MÅ ikke; uten den ville et felt
|
||||
som glir fra konsumert til videresendt vært nøyaktig driften 4e finnes for. **Demo-scenarioet er
|
||||
nåbart ved NAVN og bare der** (`main()` kaller det ikke, og at golden-transkriptet er byte-uendret
|
||||
etter at det ble lagt til ER målingen av det), med en **vakuitets-vakt**: en label kunnskapsbasen
|
||||
ALLEREDE oppgir refuseres, fordi den ville nådd hypotese-prompten som ordinær kontekst enten
|
||||
utforskningen kjørte eller ei — `simulate_learning_loop`s to-markør-vakt i demo-form. Manager-
|
||||
manuset nøkles på PROMPT-STADIET, ikke prosjekt-ID-en, og det er ikke et unntak fra
|
||||
`scripted_proposer`-regelen: manageren får FEM ulike spørsmål og prosjekt-ID-en er konstant over
|
||||
alle fem. **`--outbox-dir` uten `--run-id` NEKTES i utforskningsblokka, og det er en HOIST — ikke
|
||||
en andre kopi av regelen:** `run_project` eier outbox-kontrakten og nekter på sin FØRSTE setning,
|
||||
tidlig nok for enhver sti som fantes før U4, men utforskningen kjører FORAN det kallet — uten
|
||||
hoisten brukes hele utforskningsbudsjettet på modellkall før nekten, og artefakt-skrivingen hoppes
|
||||
over, så ikke engang regnskapet over hva som ble brukt overlever. Funnet i review FØR commit;
|
||||
testen asserterer at INGEN modellkall skjedde, ikke bare at rc er 1 — ved exit-koden ser en nekt
|
||||
etter forbruket identisk ut. **Scenarioet har BEVISST intet `label_in_bundle`-felt:** vakten
|
||||
raiser før et resultat finnes, så feltet kunne kun vært `False`, og en assert på det ville vært
|
||||
grønn mot enhver implementasjon — vakten ER kontrollen, og en alltid-sann gjentakelse av den ville
|
||||
bare gjort den ekte lettere å avfeie. Load-bearing MÅLT
|
||||
(`tests/test_explore_callsites_loadbearing.py`, 24 tester), sytten mutasjoner alle røde mot HELE
|
||||
suiten, hver mot kontrollen som gjaldt da (990/5 for CLI-en + sporet, 996/5 for hosting, 998/5 for
|
||||
sim-scenarioet, 999/5 for de to siste): detach sink-appenden (1) · andre liste
|
||||
for rundene (4) · detach `--mandate`-nekten (1) · detach `--explore-config`-nekten (1) · skriv
|
||||
artefaktet kun ved fullført kjøring (1) · detach CLI-ens `mandate=` (1) · slipp
|
||||
`enable_plan_review` gjennom, CLI (1) · detach `--bundle-dir`-kravet, CLI (1) · detach
|
||||
`--live-dry-run`-nekten (1) · fjern `--explore` fra portefølje-partisjonen (1) · videresend de
|
||||
konsumerte feltene (2) · detach hostings `mandate=` (1) · detach `enable_plan_review`-nekten,
|
||||
hosting (1) · detach `bundle_dir`-kravet, hosting (1) · detach sim-scenarioets `mandate=` (1) ·
|
||||
detach vakuitets-vakten (1) · detach outbox/run-id-hoisten (1). **ÉN MUTASJON FALSIFISERTE TESTEN FØRST (repoets vakuøs-gate-klasse,
|
||||
syvende gang):** portefølje-testen asserterte kun at meldingen nevnte `--explore`, og sto GRØNN
|
||||
uten partisjonen — kjøringen falt da gjennom til «`--explore` requires `--bundle-dir`», som nevner
|
||||
`--explore` også. To nekter som deler en delstreng er «assert aldri på ordlyd to grener deler»,
|
||||
fanget av sin egen mutasjon; testen navngir nå `--portfolio`. **Ærlighets-grenser, uttalt:**
|
||||
multi-base (`Approach.bundle_id`, § C.7) er FORTSATT ikke bygget — `run_project` tar én
|
||||
`bundle_dir`; utforskningens egne modellkall er UANNONSERTE (annonseringens kontrakt er at en
|
||||
KOMMISJON erklæres før arbeidet den bestiller, og før `explore()` returnerer finnes ingen —
|
||||
`exploration_notice` dekker gapet i det sløyfa er ferdig); `BudgetExceeded` ut av `--explore`
|
||||
tracebacker som den gjør for debatten i dag; og **den HOSTEDE flaten gir ingen innsyn i hva som
|
||||
formet mandatet** — det er ingen outbox der og intet utforskningsfelt i `_response_payload`, så
|
||||
ledgeren og de rådgivende dommene når kun CLI-ens artefakt. En bevisst scope-grense, men uttalt,
|
||||
fordi flatens hele argument er at svaret er etterprøvbart.
|
||||
- **Multi-base er en PARTISJON, aldri en videre `run_project`-signatur (U4+U13 del 3, § C.7, økt 58):**
|
||||
planens § C.7 og økt 56s egen ærlighets-grense leste som om leveransen var «`run_project` tar mer
|
||||
enn én `bundle_dir`». **Den kan ikke det, og nekten er STRUKTURELL:** på bundle-stien avleder
|
||||
`run_project` FIRE enkeltverdier fra DEN basen — prosjektet (`_project_from_bundle`, som
|
||||
fail-faster når basens egen `validator-input.json` ikke navngir det forespurte prosjektet),
|
||||
validatorens stage-0-baseline (S4.0s hele poeng er at gaten er forankret i DETTE prosjektets
|
||||
kostlinjer), agentenes lesekontekst og ExpeL-nøkkelen — og returnerer ETT stemplet `RunResult`.
|
||||
En andre katalog på den signaturen ville tvunget et stille velg-en for alle fire, som er den
|
||||
gjettede-form-klassen repoet nekter. **Planens egen setning sier det samme lest nært:**
|
||||
«pipelinen kjøres per bundle som i dag (`run_portfolio`-formen)» = N kall, ikke ETT kall med N.
|
||||
Premisset ble felt FØR bygging; ordren ba selv om nettopp den sjekken. **Konsekvensen er at INGEN
|
||||
eksisterende kaller endrer signatur** — CLI, hosting og simulation sender fortsatt én base hver,
|
||||
og kan fortsatt gjøre det. Tre sømmer: (1) `mandate.Approach.bundle_id`, default `""`, så hvert
|
||||
mandat skrevet før i dag er fortsatt gyldig OG dispatchbart uendret; (2) `mandate.route_by_bundle`
|
||||
— ren partisjon i `bundle_ids`-rekkefølge (aldri i approach-rekkefølge: spend-ordenen er en
|
||||
egenskap ved hvordan kjøringen ble konfigurert, ikke ved hvordan en modell tilfeldigvis sekvenserte
|
||||
hypotesene), **fail-fast på et mandat som ikke kan utføres som skrevet** (`load_mandate`-regelen —
|
||||
en kjøring skal aldri gå videre på en stille degradert bestilling); (3)
|
||||
`run.run_mandate_across_bundles` — dispatchen. **Den tar INGEN `project_id`-parameter, og det er
|
||||
designet:** hver bases prosjekt leses fra DEN basens egen IR-projeksjon, altså nøyaktig verdien
|
||||
`_project_from_bundle` allerede fail-faster mot, så en kaller-oppgitt konstant kunne uansett bare
|
||||
vært riktig for én base av N — den eksisterende fail-fasten blir rutingsnøkkelen, og gjetningen
|
||||
forsvinner. Ett `VerdictStore` trådes på tvers (kryss-base-læring, `run_portfolio`-formen), og
|
||||
**delt INSTANS er påstanden — ikke lik verdi** (se vakuitets-funnet under). En base ingen approach
|
||||
navngir kjøres IKKE (en kjøring koster penger, og bestillingen ba om ingenting der); med NØYAKTIG
|
||||
én base absorberer den alt uten navn, som ikke er en gjetning men det eneste mulige svaret — og
|
||||
det er dét som holder hvert pre-multi-base-mandat dispatchbart. `explore()` stempler `bundle_id`
|
||||
på hver MYNTET approach, men **skriver ALDRI om et frø** (§ C.6 dør 1 er en bevaringsregel — å
|
||||
fylle inn feltet på ekspertens vegne ville satt deres navn på en rutingsbeslutning de ikke tok);
|
||||
frøene VALIDERES i stedet, **FØR første modellkall** (økt-57-hoisten: ved unntaket alene ser en
|
||||
nekt etter forbruket identisk ut med en før). En umerket markør med flere baser NEKTES
|
||||
(`HypothesisParseError`), med én base resolveres den. **Budsjett: de to S3.4-tennene som HAR
|
||||
mening her** — oppstartsnekt (`BudgetRefused`) og aldri-startet + `budget_stop` +
|
||||
`not_evaluated`-rader i `MultiBaseResult.unreached`; bølge-reservasjonen har ingen motpart, for
|
||||
dispatchen er SEKVENSIELL. **Ærlighets-grenser, uttalt:** en base som RAISER propagerer
|
||||
(collect-and-continue tilhører `run_portfolio`, der kalleren sendte inn en batch uavhengige
|
||||
prosjekter); uten `portfolio_meter` er taket antall rutede baser × `max_tokens`, hver kjøring
|
||||
bundet for seg; outboxen er IKKE wiret (N kjøringer trenger N `run_id`-er, og å mynte dem her
|
||||
ville defaultet en nøkkel repoet krever at en kaller oppgir); og **CLI-en er BEVISST urørt** —
|
||||
§ C.8 ber om ETT nytt kallsted i `run.py` (`--explore`, levert i 57), og et repeterbart
|
||||
`--bundle-dir` er en NY operatørflate, altså en egen beslutning. Load-bearing MÅLT
|
||||
(`tests/test_multibase_loadbearing.py`, 22 tester), tolv mutasjoner alle røde mot HELE suiten +
|
||||
grønn kontroll 1020/5 og golden `demo-transcript.stdout` BYTE-UENDRET
|
||||
(`ea8c534773acdbe41ae68f2c55724d69aaf8be4f`): detach myntet `bundle_id` (2 røde) · stille
|
||||
gjennomfall ved >1 base (1) · ukjent id resolvert etter rekkefølge (1) · frø-sjekk etter forbruket
|
||||
(2 — asserten er på at NULL modellkall skjedde, ikke på unntaket) · ruteren gjetter første base (1)
|
||||
· uroutbar approach droppet (1) · dispatchen kollapser til én base (5) · `project_id` fra første
|
||||
base (2) · detach aldri-startet-tannen (1) · `unreached` urapportert (1) · fersk store per base (1)
|
||||
· detach oppstartsnekten (2). **ÉN MUTASJON FALSIFISERTE TESTEN FØRST (repoets vakuøs-gate-klasse,
|
||||
åttende gang):** store-testen sammenlignet med `==`, og `VerdictStore` er en pydantic-modell med
|
||||
VERDI-likhet — tre ulike TOMME stores er alle like, så «fersk store per base» lot HELE suiten stå
|
||||
grønn. Delt instans er påstanden, så testen asserterer nå på `is`.
|
||||
- **«Be om svar, BRUKE svarene» er nåbar fra CLI-en, og gaten er den ANDRE halvdelen (F4, økt 63):**
|
||||
før dette nektet BEGGE operatørflatene `enable_plan_review` (`run.py`, `hosting.py`) og eneste dør
|
||||
var `explore(..., plan_reviewer=...)` — MÅLT mot kilden, ikke lest ut av reviewens prosa.
|
||||
`--plan-review` bygger en `terminal_plan_reviewer()` og gir den til den UENDREDE sløyfa: operatøren
|
||||
vises planen og svarer `approve` eller `revise <hva>`; en revisjon går tilbake til manageren, som
|
||||
replanlegger og spør IGJEN om den NYE planen. **Diskriminatoren er dét siste** — en dør som printer
|
||||
planen, leser linja og kaster den består «operatøren ble spurt» og feiler målbildet (repoets
|
||||
vakuøs-gate-klasse); T1 er derfor bygget som `test_explore_loadbearing`s T15 løftet til CLI-nivå og
|
||||
er RØD mot en alltid-godkjenn-reviewer. **Vitnet er `{run_id}-exploration.json`, ikke skrapet
|
||||
stdout:** `trace_payload` bærer alt tre (rekkefølge, beslutning, feedback verbatim) og skrives fra
|
||||
en `finally`, så den ene kjøringen som mest trenger beviset — den et tak eller en ubesvart review
|
||||
kappet — etterlater det. **Fail-closed på operatørens EGEN input:** alt utenfor det lukkede
|
||||
vokabularet spørres på nytt (aldri lest som en beslutning), og **EOF raiser `PlanReviewInputError`**
|
||||
— å lese stillhet som ja ville latt en autonom sløyfe kjøre på en plan ingen signerte, usynlig.
|
||||
Strømmene resolveres ved KALL-tid (`shared_root()`-idiomet), ellers svarer reviewer-en fra strømmen
|
||||
som fantes da den ble BYGGET. **Fire nekter, alle ved navn**, hvorav to lukker et stille dropp
|
||||
ingen test dekket: `report_forbidden` (report-modus returnerer FØR hver utforsknings-nekt) og
|
||||
portefølje-partisjonen. De to konfig-avhengige nektene DELER tokenet `enable_plan_review` og har
|
||||
derfor bevisst ULIK særtekst («no reviewer was offered» / «no review is ever requested») — den
|
||||
eksisterende testen asserterte på det delte tokenet og er rettet (økt-57-mutasjonen, niende gang).
|
||||
**Hosting NEKTER fortsatt, og det er en beslutning:** reviewen er synkron, så den ville blokkert
|
||||
HTTP-requesten på et menneske OG event-løkka som svarer `/readiness` — meldingen navngir nå
|
||||
CLI-døra i stedet for å påstå at biblioteket er den eneste (Fase 3-klassen). **Mid-løp-spørsmål er
|
||||
IKKE bygget, og fraværet er MÅLT:** `_magentic.py` har nøyaktig ETT `ctx.request_info` (`:1044`,
|
||||
plan review) i hele modulen, så stacken kan ikke levere et spørsmål midt i løpet uten en ny
|
||||
emitter. Reviewens «kun plan-review FØR løpet» er derimot upresist: samme forespørsel fyrer også
|
||||
ved re-plan etter en stall (`is_stalled=True`), så døra ER nåbar midt i en kjøring på den ene
|
||||
måten stacken støtter. Load-bearing MÅLT (`tests/test_plan_review_cli_door_loadbearing.py`, 12
|
||||
tester), elleve mutasjoner alle røde mot HELE suiten + grønn kontroll 1040/5 og golden
|
||||
`demo-transcript.stdout` BYTE-UENDRET (`ea8c534773acdbe41ae68f2c55724d69aaf8be4f`): detach
|
||||
`plan_reviewer`-wiringen (5 røde) · EOF blir en godkjenning (1) · alltid-godkjenn (3) · alt som
|
||||
ikke er en revisjon blir en signatur (1) · dropp `--plan-review` fra portefølje-partisjonen (1) ·
|
||||
dropp den fra `report_forbidden` (1) · detach `--plan-review requires --explore` (1) · detach
|
||||
review-uten-reviewer-nekten (2, hvorav én i en test som fantes fra før) · detach
|
||||
reviewer-ingen-spør-nekten (1) · hostet nekt beholder påstanden fra før F4 (1) · strømmene fanget
|
||||
ved bygge-tid (1).
|
||||
- **Plan-reviewen kan besvares over DAGER, og det eneste som krysser prosessgrensen er DISK (U12 +
|
||||
asynkron U13, planens § D.2 rad 3, økt 64):** F4 gjorde «be om svar, BRUKE svarene» nåbar, men
|
||||
bare SYNKRONT — `terminal_plan_reviewer` blokkerer løkka på et menneske ved en terminal, så
|
||||
svaret må komme mens prosessen lever. `--checkpoint-dir` PARKERER i stedet reviewen
|
||||
(`FileCheckpointStorage` + `{run_id}-plan-review.json`), og `--resume <run_id>` leser svaret fra
|
||||
`--review-inbox` i en prosess som ALDRI så kjøringen. **MÅLT FELLE (ansikt 4):**
|
||||
`list_checkpoints` (`_checkpoint.py:386-388`) svelger en blokkert deserialisering til en
|
||||
`logger.warning` og returnerer TOM liste — uten BEGGE
|
||||
`MagenticPlanReviewRequest`/`…Response` i `allowed_checkpoint_types` feiler en resume som et
|
||||
FRAVÆR, ikke som en feil, og en test som asserterte «listingen er tom, altså er det ingenting å
|
||||
gjenoppta» ville vært GRØNN mot nøyaktig den defekten. `_ALLOWED_CHECKPOINT_TYPES` har derfor ÉN
|
||||
kopi og `checkpoint_storage` er ENESTE konstruksjonssted (BEGGE prosesser må deklarere dem; en
|
||||
andre kopi er kø-(p)-driften). **Vi er LOUDERE enn rammeverket der det tier:** en tom listing ved
|
||||
park raiser `CheckpointUnreadable` i stedet for å skrive et spørsmål ingen kan besvare.
|
||||
**Diskriminatoren er den ANDRE halvdelen:** en dør som skriver en spørsmålsfil og en resume som
|
||||
leser en svarfil består begge «eksperten ble spurt» — så måltesten krever at et `revise` skrevet
|
||||
dag 1 får manageren til å REPLANLEGGE og stille et NYTT spørsmål (indeks 1, nytt `request_id`) i
|
||||
en fersk interpreter, med en approve-kontroll som beviser at døra også kan AVSLUTTE (en gate som
|
||||
bare kunne parke igjen er en hengning i løkkeklær). **Budsjettet og revisjons-capen spenner over
|
||||
suspensjonen:** `meter.charge(parked.tokens_spent)` (gjennom `charge`, ikke ved å sette `tokens`
|
||||
— ladingen re-tester taket) og `trace.ledger.extend(parked.ledger)`, ellers får hver park et helt
|
||||
budsjett på nytt: S3.4-klassen, ubundet forbruk under vakter som alle ser tilfredse ut. En
|
||||
`revise` koster to manager-kall, emitterer null ledger og bruker null runde (§ F, A3), så
|
||||
`max_plan_revisions` er det ENESTE båndet på den. **Fail-closed på ekspertens EGEN fil:**
|
||||
`request_id`-mismatch, ord utenfor vokabularet og `revise` uten innhold refuseres alle ved navn.
|
||||
`hitl.pending_plan_reviews` er registeret over hvem som fortsatt venter — tolerant på LESE-siden,
|
||||
fail-closed på BESLUTTE-siden, og joinen er på `request_id` i BEGGE ender (to distinkte sømmer,
|
||||
MÅLT: hver har sin egen mutasjon og sin egen røde test). Load-bearing MÅLT
|
||||
(`tests/test_async_plan_review_loadbearing.py`, 17 tester), tretten mutasjoner alle røde mot HELE
|
||||
suiten + grønn kontroll 1059/5 og golden `demo-transcript.stdout` BYTE-UENDRET
|
||||
(`ea8c534773acdbe41ae68f2c55724d69aaf8be4f`): tom `_ALLOWED_CHECKPOINT_TYPES` (11 røde) · park
|
||||
uten checkpoint (1) · resume alltid-approve (2) · tolerant `request_id` (1) · tolerant vokabular
|
||||
(1) · `revise` uten feedback (1) · detach `meter.charge` (1) · detach
|
||||
`trace.plan_reviews.extend` (1) · detach ledger/hypotese-carry-overen (1) ·
|
||||
`pending_plan_reviews` ignorerer `request_id` (1) · detach to-dører-nekten (1) · detach
|
||||
outbox/run-id-hoisten (1) · detach `--resume`-armen i `required_scripted_roles` (4 — MAJOR-2s
|
||||
`KeyError: 'navigator'` på den andre flaten som bygger en utforskning).
|
||||
**ÉN MUTASJON FALSIFISERTE SUITEN (repoets vakuøs-gate-klasse, TIENDE gang):** `trace.plan_reviews
|
||||
.extend(parked.plan_reviews)` kunne detaches med HELE suiten grønn (1058/5) — capen leser
|
||||
`parked.plan_reviews` DIREKTE, så den binder uansett, og de to første legene er identiske under
|
||||
begge implementasjoner. Gaten måtte derfor bli det TREDJE leget, der artefaktet ellers taper dag
|
||||
1s revisjon og to ULIKE planer deler indeks 1; den nye testen er rød mot mutasjonen og alene.
|
||||
**Ærlighets-grenser, uttalt:** den hostede flaten NEKTER fortsatt (en synkron review ville
|
||||
blokkert både requesten og event-løkka som svarer `/readiness`); en park MIDT i løpet (etter en
|
||||
stall) har ingen nåbar sti under det skriptede manuset, så carry-overen som betjener den drives
|
||||
gjennom en CRAFTED parkert tilstand (`budget_stop`-presedensen); og resume-legets
|
||||
`PlanReviewParked` er et NORMALT utfall, ikke en feil.
|
||||
- **Katalogkallet koster O(BASER), aldri O(KORPUS) — og det er stigens billigste trinn, ikke dens
|
||||
dyreste (ordre `20260825T213645Z`, økt 65):** `list_bundles` returnerte hele rot-indeksens body
|
||||
for HVER konfigurert base samtidig, pluss ett JSON-objekt per ufulgt kryss-lenke. Begge vokser med
|
||||
korpuset, så prisen på å finne ut *hvilke baser som finnes* ble satt av hvor mye de *inneholder* —
|
||||
progressiv disclosure snudd på hodet (målbilde §2/§4). **MÅLT med `o200k_base`, instrumentet først
|
||||
validert mot commons' egne fasittall:** 112 116 tokens over tre flate Vegnormal-baser, og
|
||||
**124 942 over de 171 grenbasene** som erstattet dem — grenformen (`vegnormal-okf` `8145c23`)
|
||||
lukket bundle-siden (−82…92 % på `read_bundle`) og gjorde katalogsiden VERRE, nøyaktig som det
|
||||
repoet forutså. Etter: **362** og **21 448** (per base 37 372 → 121 og 731 → 125).
|
||||
**Et premiss ble felt FØR noe ble bygget på det:** «indeksbodyen forteller hva basen handler om»
|
||||
er USANT for maskin-importerte baser — grenbasenes `index.md` har verken frontmatter eller prosa,
|
||||
den er en ren lenkeliste (målt: 959 bytes, første tegn `-`), så feltet var dyrt OG innholdsløst
|
||||
der. **Fast vindu, aldri en andel av basen** (`_CATALOGUE_EXCERPT_CHARS = 200`): en andel skalerer
|
||||
med korpuset igjen, bare med mindre konstant. **Avkorting ANNONSERES som FELT**
|
||||
(`index_truncated` ved siden av utdraget, aldri en markør limt inn i det — `BudgetExceeded`s
|
||||
kø-(y)-regel), og en base som PASSER blir ikke merket avkortet og får hele bodyen: omisjon, aldri
|
||||
en løgn i noen av retningene. **En ufulgt lenke overlever som ANTALL** — økt 51s «et hopp er
|
||||
tolerert, men ikke lenger taust» står, mens per-lenke-detaljen blir liggende der den er
|
||||
handlingsbar (`RunResult.skipped_links` / `DryRunReport.skipped_links`) og ikke rir med i et kall
|
||||
hvis hele jobb er å være billig. Hele indeksen er fortsatt ETT `read_file(id, "index.md")` unna —
|
||||
et disclosure-nivå, ikke datatap. **Taket (500 tegn/base) bor i TESTEN, ikke i `explore.py`:** en
|
||||
test som importerte implementasjonens budsjett ville flyttet seg med det, og å heve budsjettet er
|
||||
nøyaktig regresjonen gaten finnes for. Load-bearing MÅLT
|
||||
(`tests/test_catalogue_cost_loadbearing.py`, 7 armer), **ni mutasjoner alle røde mot HELE suiten**
|
||||
+ grønn kontroll 1066/5 og golden `demo-transcript.stdout` BYTE-UENDRET
|
||||
(`ea8c534773acdbe41ae68f2c55724d69aaf8be4f`): ingen binding (4 røde) · bundet men vakuøst (2) ·
|
||||
stille kutt (1) · over-annonsert (1) · per-lenke-lista rir med igjen (1) · det ufulgte faktumet
|
||||
slettet (1) · suffiks i stedet for ordrett prefiks (2) · en andel i stedet for fast vindu (2) ·
|
||||
`documents` gjort konstant (1). **M9 ble kjørt fordi `documents` var et felt uten gate** — et felt
|
||||
ingen test kan se, råtner. **MAJOR-1 var IKKE nødvendig:** bindingen sitter i verktøykroppen bak
|
||||
en uendret CLI-flate. Ærlighets-grenser, uttalt: `navigate_bundle` kalles fortsatt per base per
|
||||
katalogkall (I/O og veggklokke, ikke tokens — ikke målt her); ingen LEVENDE modell har kalt det
|
||||
nye verktøyet, så at en manager velger BEDRE med et utdrag enn med hele indeksen er ikke bevist
|
||||
(structured-output-grensens klasse); og ordrens nevner for N100:2023 var 34 mens disken viser 40 —
|
||||
tallene bruker den målte nevneren. Måling: `docs/2026-08-26-katalogkostnaden.md`.
|
||||
- **Ekspertdommen kan ikke oppstå av STILLHET, og fraværet er en FØRSTEKLASSES tilstand (F2,
|
||||
non-goal 3, økt 66):** `run_project` KREVDE `verdict_input` og kjørte `capture_verdict`
|
||||
ubetinget, CLI-en defaultet det til `{"approved", "reviewed by expert"}`, og hosting listet det
|
||||
som PÅKREVD. Netto: hver flaggløs kjøring myntet en ekspertgodkjenning ingen ga, den gikk inn i
|
||||
den delte storen, og `run_portfolio` bar den inn i neste prosjekts hypotese-prompt som en
|
||||
*prior expert verdict* — på flaten som ble overlevert 14.08. **`RunResult.verdict` er nå
|
||||
`Verdict | None`**, og `None` er hva stillhet produserer: ingenting myntes, ingenting lagres,
|
||||
ingenting varsles. Prinsippet sto allerede skrevet i repoet — `RunFailure`s docstring: å fylle et
|
||||
felt med en dummy legger FABRIKKERT proveniens inn i aggregatet. **Traceability koster ingenting,
|
||||
fordi nøkkelen DERIVERES fra kandidaten:** `RunResult.verdict_key` (property, ikke lagret felt —
|
||||
en andre kopi av en nøklingsregel er kø-(p)) er `verdicts.verdict_key`s alt dokumenterte formål,
|
||||
identisk med `verdict.id` når en dom BLE gitt, og fortsatt meningsfull når ingen ble det; det er
|
||||
den outboxen og den hostede responsen stempler, så et artefakt fra en ukommentert kjøring er
|
||||
fortsatt dømbart og joiner tilbake via Steg-7-innboksen. **Halv dom NEKTES på begge dører**
|
||||
(`FeedbackContract` er ENESTE sted formen valideres, og CLI-en nekter ved navn FØR enhver
|
||||
mode-dispatch): den manglende halvdelen er ekspertens å skrive, aldri vår å defaulte — validering,
|
||||
ALDRI reparasjon (`write_concept_file`-presedensen). **Hosting er WIDENING, ikke bryting:**
|
||||
`verdict_input` flyttet `_REQUIRED_FIELDS` → `_OPTIONAL_FIELDS`, så hvert kall som finnes ute
|
||||
virker uendret; en kaller som utelot det fikk før 400 på et felt som ikke KUNNE fylles ærlig.
|
||||
**De to mode-partisjonene fikk `--decision`/`--rationale` inn — og det er en KONSEKVENS, ikke
|
||||
scope-krype:** kommentarene på begge stedene sa ordrett at en ærlig nekt var *uimplementerbar*
|
||||
fordi de non-None argparse-defaultene gjorde en eksplisitt verdi uskillbar fra defaulten. Med
|
||||
defaultene borte er den implementerbar, og «refused, never ignored» er partisjonens egen regel.
|
||||
`run.verdict_notice` er ENESTE renderer og leser dommen av kjøringens EGET stempel, ikke av argv.
|
||||
**Ærlighets-grense, uttalt:** referanse-fixturens SYNTETISKE `verdict_input`-rader står URØRT —
|
||||
de er merket SYNTETISK på fire steder og er reviewens F5 (måling av misjonspåstanden), ikke F2;
|
||||
`Project.verdict_input` er nå valgfri, så en rad UTEN dom er lovlig. Load-bearing MÅLT
|
||||
(`tests/test_ungiven_verdict_loadbearing.py`, 15 armer), **åtte mutasjoner alle røde mot HELE
|
||||
suiten** + grønn kontroll 1080/5 og golden `demo-transcript.stdout` BYTE-UENDRET
|
||||
(`ea8c534773acdbe41ae68f2c55724d69aaf8be4f`): detach fangst-gaten (5 røde) · gjeninnfør
|
||||
argparse-defaultene (22) · hosting krever fortsatt feltet (1) · CLI-en REPARERER en halv dom (2) ·
|
||||
kontrakten reparerer en halv dom (1) · `verdict_key` lest av dommen i stedet for derivert (2) ·
|
||||
begge partisjons-radene fjernet (2) · rendereren påstår en dom som aldri ble gitt (2). **ÉN
|
||||
MUTASJON FALSIFISERTE TESTEN FØRST (repoets vakuøs-gate-klasse, ELLEVTE gang):** `--report`-armen
|
||||
brukte et bart `--report`, som nekter med rc 1 uansett fordi `--ledger` mangler — testen sto
|
||||
GRØNN med partisjons-raden fjernet. Den kjører nå mot en argv report-modus ellers ville AKSEPTERT
|
||||
(gyldig `--ledger` + en kontroll som beviser rc 0 uten flaggene), så rc 1 er mutantens motsatte
|
||||
utfall. **Migreringsnote:** ingen ekstern kaller brekker — hosting utvider, CLI-ens gamle
|
||||
flaggform er uendret, og det som ENDRER seg er at en flaggløs kjøring nå SIER at ingen dømte i
|
||||
stedet for å påstå `decision=approved`.
|
||||
- **STATE.md er local-only** (gitignored). Voyage session-state er efemert; STATE.md er kanonisk kontinuitet.
|
||||
- Prosess: Voyage-plugin (`/trekbrief → /trekplan → /trekexecute → /trekreview`) per større fase.
|
||||
|
||||
|
|
|
|||
232
DEPLOY.md
Normal file
232
DEPLOY.md
Normal file
|
|
@ -0,0 +1,232 @@
|
|||
# Deploying and using this package
|
||||
|
||||
This archive is a complete, self-contained copy of `portfolio-optimiser`. You do not need to clone
|
||||
anything or have an account anywhere to deploy it. Unzip it and follow this document.
|
||||
|
||||
It is written in English because its readers are implementing developers and platform operators.
|
||||
The two documents that describe the *human* side of the loop are written in Norwegian and ship in
|
||||
this same archive: [`docs/bestille-en-kjoring.md`](docs/bestille-en-kjoring.md) (ordering a run) and
|
||||
[`docs/ekspert-svar.md`](docs/ekspert-svar.md) (giving the verdict).
|
||||
|
||||
---
|
||||
|
||||
## 1. What this is — and what it is not
|
||||
|
||||
It finds cost savings **inside** a single project: agents propose candidate measures, a
|
||||
**deterministic validator** decides whether the numbers hold, a second agent (the "checker") gates
|
||||
the *reasoning*, and a domain expert gives the final verdict. The system learns from those verdicts.
|
||||
|
||||
**There is no chat interface, and that is a design decision rather than a missing feature.**
|
||||
|
||||
The hosted surface is a task API — `POST /invocations` with a JSON body, `GET /readiness` — not a
|
||||
conversation. The system is deliberately **not** exposed as a Microsoft Foundry agent via
|
||||
`as_agent()`, because the validator, the cost-baseline anchoring, the checker gate and the savings
|
||||
ledger all live *outside* the agent graph. Wrapping it as a chat agent would route requests around
|
||||
precisely the components that make an answer trustworthy, and you would get a fluent assistant
|
||||
instead of a validated result.
|
||||
|
||||
The dialogue with humans happens in two places instead, both file-shaped and both auditable:
|
||||
|
||||
- **before** a run, through a mandate file (what should be considered), and
|
||||
- **after** a run, through a verdict file (what a qualified person decided).
|
||||
|
||||
If you want an interactive front end, build it on top of `POST /invocations` — but keep the verdict
|
||||
step human. An approval loop that a model can close by itself is not an approval loop.
|
||||
|
||||
---
|
||||
|
||||
## 2. Who does what
|
||||
|
||||
The package has three distinct audiences. Only the first one is technical.
|
||||
|
||||
| Role | What they do | Where it is documented |
|
||||
|---|---|---|
|
||||
| **Platform operator** | Installs the package, sets two environment variables, starts the service, verifies with the offline preflight. One-time work. | This document, sections 3–6. |
|
||||
| **Ordering domain expert** (*bestiller*) | Writes the mandate file: the objective, which approaches are worth trying, and why. The *why* is the part the system cannot read out of cost figures. | `docs/bestille-en-kjoring.md` |
|
||||
| **Reviewing domain expert** (*fagperson*) | Receives a validated proposal and answers `approved` or `rejected` with a written rationale. That verdict is what the system learns from. | `docs/ekspert-svar.md` |
|
||||
|
||||
**The ordering expert steers what is *considered*, never what is *approved*.** If you ask for
|
||||
something the numbers do not support, the deterministic validator rejects it and the rejection comes
|
||||
back with its reason. That property is the point of the whole design; do not build around it.
|
||||
|
||||
---
|
||||
|
||||
## 3. The process, end to end
|
||||
|
||||
1. **Prepare a knowledge base** for the project — cost documents, technical sources, prior
|
||||
experience — as a document bundle. A working five-document example ships in
|
||||
`shared/examples/bygg-energi-mikro`, and `docs/knowledge-base-recipe.md` describes how to build
|
||||
your own.
|
||||
2. **Order a run** with a mandate file (optional but recommended). See
|
||||
`docs/bestille-en-kjoring.md`.
|
||||
3. **Run it** — from the CLI, or by `POST /invocations` against the running service (section 6).
|
||||
The run navigates the knowledge base, debates candidate measures, validates the numbers against
|
||||
the project's actual cost baseline, and returns either a **validated proposal** or a
|
||||
**rejection with a reason**. A rejection is a successful run: the negative outcome belongs to the
|
||||
answer, not to the transport, and the API returns HTTP 200 for it.
|
||||
4. **A domain expert judges the outcome** and writes a verdict — either inline during the run, or as
|
||||
a JSON file dropped into an inbox directory afterwards, possibly days later. See
|
||||
`docs/ekspert-svar.md`.
|
||||
5. **Approved verdicts can be promoted into the knowledge base**, so the next run reads them as
|
||||
context and does not re-propose something the organisation has already rejected. Promotion is
|
||||
gated: only an approved verdict can be promoted, and raw agent output never enters the knowledge
|
||||
base on its own.
|
||||
|
||||
Steps 4 and 5 are what make this a learning loop rather than a report generator.
|
||||
|
||||
---
|
||||
|
||||
## 4. Prerequisites in your own tenant
|
||||
|
||||
You need, in Microsoft Foundry:
|
||||
|
||||
1. A **Foundry resource** with project management enabled and a **project** inside it. A Foundry
|
||||
project is required — an Azure OpenAI resource (`kind: OpenAI`) is a different client surface and
|
||||
will not work.
|
||||
2. At least one **model deployment** (any current chat model; a small one is enough).
|
||||
3. The **Foundry User** role (role definition id `53ca6127-db72-4b80-b1b0-d745d6d5456d`) for the
|
||||
identity that will run it, assigned at project scope. Azure `Owner` and `Contributor` grant
|
||||
management permissions only, **not** the data-plane permissions this needs.
|
||||
|
||||
---
|
||||
|
||||
## 5. Configuration — the two variables that matter, and one that is optional
|
||||
|
||||
**5.1 The project endpoint.** Resolved as the first non-empty of:
|
||||
|
||||
```
|
||||
PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT # yours; wins if set
|
||||
FOUNDRY_PROJECT_ENDPOINT # injected by the Foundry hosting platform
|
||||
```
|
||||
|
||||
Where something injects the second one for you — the Foundry hosting platform does — you need to
|
||||
set nothing. Everywhere else (a laptop, a pipeline, your own runtime) set the first. The value looks
|
||||
like `https://<resource>.services.ai.azure.com/api/projects/<project>`.
|
||||
|
||||
**5.2 The model map — this is the one that catches people.** The packaged
|
||||
`src/portfolio_optimiser/data/model_map.json` ships **placeholders**:
|
||||
|
||||
```json
|
||||
"azure": { "default": "REPLACE-WITH-FOUNDRY-DEPLOYMENT", ... }
|
||||
```
|
||||
|
||||
Deployment names are tenant-specific, so they are deliberately never committed. The code
|
||||
**fail-fasts** on a `REPLACE-WITH-` value rather than guessing. Point `PORTFOLIO_MODEL_MAP` at your
|
||||
own file:
|
||||
|
||||
```json
|
||||
{
|
||||
"local": { "default": "qwen3:4b", "proposer": "qwen3:4b", "checker": "qwen3:4b" },
|
||||
"azure": { "default": "my-deployment", "proposer": "my-deployment", "checker": "my-deployment" }
|
||||
}
|
||||
```
|
||||
|
||||
```bash
|
||||
export PORTFOLIO_MODEL_MAP=/path/to/model_map.json
|
||||
```
|
||||
|
||||
If you skip this, the service starts and answers `/readiness` — and fails every invocation. That
|
||||
is the single most likely first-deployment failure, which is why it has its own section.
|
||||
|
||||
**5.3 Tracing — optional, and off until you ask.** `PORTFOLIO_OTEL` is the only switch:
|
||||
|
||||
```bash
|
||||
export PORTFOLIO_OTEL=console # OpenTelemetry spans on STDERR — in a container, that is your log
|
||||
```
|
||||
|
||||
Unset, no OpenTelemetry provider is configured at all: the spans are still created and then
|
||||
discarded, so nothing can leave the process. `console` writes them to stderr and leaves stdout
|
||||
byte-for-byte as it was, so you can trace a run without changing what the run prints. `otlp` sends
|
||||
them to a collector, and only to an endpoint you name yourself:
|
||||
|
||||
```bash
|
||||
export PORTFOLIO_OTEL=otlp
|
||||
export OTEL_EXPORTER_OTLP_ENDPOINT=http://your-collector:4317
|
||||
pip install opentelemetry-exporter-otlp-proto-grpc # not a declared dependency — egress is yours to opt into
|
||||
```
|
||||
|
||||
Asking for `console` while an `OTEL_EXPORTER_OTLP_*_ENDPOINT` variable is set is **refused**, not
|
||||
quietly honoured: those exporters are constructed unconditionally by the framework underneath, so
|
||||
the run would have shipped its contents to the collector while calling itself console-only. The
|
||||
service names its tracing destination on stderr at startup, before the first span exists.
|
||||
|
||||
**5.4 Verify before you pay for a model call:**
|
||||
|
||||
```bash
|
||||
uv run python -m portfolio_optimiser.preflight --profile azure
|
||||
```
|
||||
|
||||
Exit code 0 means every offline-checkable precondition holds: endpoint shape, model-map structure,
|
||||
no placeholders left. It is **necessary but not sufficient** — RBAC (403), token or tenant problems
|
||||
(401) and a well-formed but non-existent deployment (404) can only surface at the live call.
|
||||
|
||||
---
|
||||
|
||||
## 6. Running it
|
||||
|
||||
**This package is Python and nothing else.** There is no image to build and no platform-specific
|
||||
deployment manifest — that was removed deliberately after an external trial, because it forced a
|
||||
container toolchain on receivers who only needed to run the code. How the process is supervised,
|
||||
packaged or placed in your environment is your decision, and this package does not pre-empt it.
|
||||
|
||||
Install the locked environment and start the service:
|
||||
|
||||
```bash
|
||||
uv sync --frozen --no-dev
|
||||
uv run python main.py
|
||||
```
|
||||
|
||||
`uv sync --frozen` installs the exact resolution every measurement in this repository ran against —
|
||||
never a fresh resolve, which picks other versions. Two dependencies are git-tag-pinned direct
|
||||
references, so **`git` must be on PATH** for the install to work.
|
||||
|
||||
`main.py` serves port 8088 (override with `PORT`), answers `GET /readiness`, accepts
|
||||
`POST /invocations`, and exits cleanly on SIGTERM. For a single run without the service, the CLI is
|
||||
the same code path:
|
||||
|
||||
```bash
|
||||
uv run portfolio-optimiser --help
|
||||
```
|
||||
|
||||
**The invocations payload** is whitelisted against the run signature — unknown fields are refused by
|
||||
name with HTTP 400, never silently ignored:
|
||||
|
||||
| Field | Required | Notes |
|
||||
|---|---|---|
|
||||
| `project_id` | yes | The project to analyse. |
|
||||
| `docs_dir` | yes | Source documents. |
|
||||
| `verdict_input` | no | An expert verdict for this run, `{"decision", "rationale"}`, **when a human actually gave one**. Omit it otherwise — no verdict is then recorded and the response's `verdict_id` is simply the key a later verdict on this candidate will arrive under. Supplying only one of the two keys is refused (400). |
|
||||
| `bundle_dir` | no | Knowledge base to navigate. |
|
||||
| `profile` | no | Defaults to `azure` on this HTTP surface. |
|
||||
| `max_rounds` | no | Debate round cap. |
|
||||
| `max_tokens` | no | **Per-invocation token ceiling.** Set it. The built-in default is 100 000 tokens per run, which is a ceiling nobody chose for your workload. |
|
||||
| `top_k` | no | Retrieval breadth. |
|
||||
|
||||
HTTP 400 means the request was malformed; 500 means the run itself failed, with `error_type` and
|
||||
`error` in the body. A rejected proposal is **200** — see section 3.
|
||||
|
||||
**429 is not a failure.** It means the run reached its own allowance — the round cap or the token
|
||||
cap — and stopped, which is the cap doing its job rather than anything breaking. The body carries
|
||||
the ledger's own numbers, so you can tell *which* cap bound and how far past it the run got:
|
||||
|
||||
```json
|
||||
{"budget_exhausted": {"kind": "rounds", "limit": 12, "observed": 13},
|
||||
"error": "budget exceeded: rounds limit=12 observed=13"}
|
||||
```
|
||||
|
||||
There is no `error_type` (nothing failed) and no `Retry-After` (waiting changes nothing). Retrying
|
||||
the same body hits the same cap; the remedies are raising `max_rounds`/`max_tokens` for that
|
||||
invocation, or accepting that this project did not resolve within the allowance you set.
|
||||
|
||||
---
|
||||
|
||||
## 7. Honest limits
|
||||
|
||||
- **This framework has never been run against a live model in our environment.** Everything above is
|
||||
measured against tests and an offline simulation. The first live run in *your*
|
||||
tenant is the first live run, full stop.
|
||||
- The system is a **technical framework**. Data protection assessments, lawful basis and risk
|
||||
ownership belong to whoever deploys it — see `README.md` and `SECURITY.md`.
|
||||
- Readiness does not respond during CPU-bound stretches of a run (the optimisation solve).
|
||||
- Chunked request bodies are not supported.
|
||||
- Verdict promotion writes to the knowledge base non-atomically; it assumes a single writing process.
|
||||
35
Dockerfile
35
Dockerfile
|
|
@ -1,35 +0,0 @@
|
|||
# Hosted-agent container (Fase 4d). Build context is `git archive HEAD` — what a stranger
|
||||
# actually downloads, never the working tree — and the hosting platform requires x86_64
|
||||
# (spike §1.4), so the measured build command is:
|
||||
#
|
||||
# git archive HEAD | docker build --platform linux/amd64 -t portfolio-optimiser -
|
||||
#
|
||||
# Base: the version pinned by .python-version (3.12), mid-band of the resolution matrix
|
||||
# (3.10–3.13 have numpy wheels, 3.14 does not — spike §4), and the same base Microsoft's
|
||||
# own MAF sample uses. Runs as root like that sample: no non-root requirement is
|
||||
# documented (spike, Ikke verifisert pkt. 7).
|
||||
FROM python:3.12-slim-bookworm
|
||||
|
||||
# git: the two security components are git-tag-pinned direct references, and
|
||||
# [tool.uv.sources] never reaches wheel metadata (spike §3/§7.2) — uv needs the git CLI
|
||||
# to fetch them, and -slim ships without it.
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends git \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Pinned to the uv the tree was measured with (13.08) — an unpinned installer would let
|
||||
# the container's resolver drift from the working tree's.
|
||||
RUN pip install --no-cache-dir uv==0.9.8
|
||||
|
||||
WORKDIR /app
|
||||
COPY . .
|
||||
|
||||
# The LOCKED, dev-free environment — the same resolution the working tree was measured
|
||||
# with (agent-framework-core 1.9.0). Never a fresh resolve from pyproject.toml: that
|
||||
# picks other versions (1.13.0) than every measurement in this repo ran against (§3).
|
||||
RUN uv sync --frozen --no-dev
|
||||
|
||||
# The hosted-agent runtime contract: port 8088, GET /readiness, POST /invocations,
|
||||
# SIGTERM shutdown (spike §1.1) — served by main.py (portfolio_optimiser.hosting).
|
||||
EXPOSE 8088
|
||||
CMD ["/app/.venv/bin/python", "main.py"]
|
||||
274
README.md
274
README.md
|
|
@ -20,6 +20,23 @@ decides the numbers; domain experts judge the outcomes (human-in-the-loop); and
|
|||
*AI-generated: all code produced by Claude Code through dialog-driven development, with human
|
||||
review, test and judgement before anything ships.*
|
||||
|
||||
A short visual introduction — 12 slides, in Norwegian, for a general audience — ships with the
|
||||
repo: open [docs/kort-presentasjon.html](docs/kort-presentasjon.html) in any browser.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Install](#install)
|
||||
- [Walk the whole chain offline](#walk-the-whole-chain-offline)
|
||||
- [Non-goals](#non-goals)
|
||||
- [Built on an LLM wiki: Karpathy's idea, Google's format](#built-on-an-llm-wiki-karpathys-idea-googles-format)
|
||||
- [AI-first, humans on top](#ai-first-humans-on-top)
|
||||
- [How it works](#how-it-works)
|
||||
- [How it is set up](#how-it-is-set-up)
|
||||
- [What this enables](#what-this-enables)
|
||||
- [The task API — runnable Python, no wrapper](#the-task-api--runnable-python-no-wrapper)
|
||||
- [Docs](#docs)
|
||||
- [Stack & develop](#stack--develop)
|
||||
|
||||
## Install
|
||||
|
||||
Python ≥3.10, with [`uv`](https://docs.astral.sh/uv/). The package is not published to a package
|
||||
|
|
@ -137,11 +154,28 @@ project's own numbers cannot support, no matter how confidently the proposer ass
|
|||
the part of the method that carries the weight — the agents propose, and something that cannot be
|
||||
argued with decides.
|
||||
|
||||
Read that summary line carefully: `Rejection (verdict id=…, decision=approved)` is not a
|
||||
contradiction. `Rejection` is the **validator's** outcome, while `decision=` echoes the
|
||||
**human's** recorded verdict — here the `--decision` default, since nobody reviewed this run.
|
||||
The two are deliberately separate: a machine gate that blocks, and a human judgement that
|
||||
approves, are different questions and are never collapsed into one field.
|
||||
Read that summary line carefully. Nobody reviewed this run, so it says exactly that:
|
||||
`Rejection (no expert verdict given; verdict key=…)`. `Rejection` is the **validator's** outcome;
|
||||
the second half is about the **human**, and there was no human here. The key it quotes is the id
|
||||
under which an expert verdict on this candidate will arrive later — your join back into the
|
||||
[expert-answer channel](docs/ekspert-svar.md).
|
||||
|
||||
Record one and the line changes:
|
||||
|
||||
```bash
|
||||
uv run python -m portfolio_optimiser.run BYGG-KONTOR-NORD \
|
||||
--docs-dir shared/examples/bygg-energi-mikro \
|
||||
--bundle-dir shared/examples/bygg-energi-mikro \
|
||||
--scripted-replies replies.json \
|
||||
--decision approved --rationale "the retrofit is within scope"
|
||||
```
|
||||
|
||||
Now it reads `Rejection (verdict id=…, decision=approved)`, which is not a contradiction: a machine
|
||||
gate that blocks and a human judgement that approves are different questions and are never
|
||||
collapsed into one field. The two flags go together or not at all — half a verdict is refused by
|
||||
name, because the missing half is the expert's to write and never ours to default. Until 1.1.0
|
||||
`--decision` defaulted to `approved`, so every flagless run recorded an approval nobody gave and
|
||||
carried it into the next project's hypothesis; that default is gone.
|
||||
|
||||
**5 — See what it would cost with a real model**, before spending anything:
|
||||
|
||||
|
|
@ -244,16 +278,24 @@ per amount, before anything is summed.
|
|||
- **Not a model benchmark.** The end-to-end proof runs offline against a scripted stand-in client:
|
||||
it shows that the loop closes, not how well a given LLM proposes or judges.
|
||||
|
||||
> **Status:** the full 8-step agentic loop is wired and proven with load-bearing tests, and the
|
||||
> end-to-end proof is an **offline simulation** with a scripted stand-in client — no live-model
|
||||
> run yet. The **ingest layer** (real data sources) is implemented — file/CSV and SQL on both
|
||||
> stacks with bit-identical golden extractions from the shared spec, plus HTTP as a MAF-only
|
||||
> demonstrated extension point against a local mock — but exercised only against committed
|
||||
> fixtures: no bundle has yet been materialized from a live source. Ingested content passes a
|
||||
> **content gate** (`materialize_gated`) that scans every generated concept with
|
||||
> **Status:** the full 8-step agentic loop is wired and proven with load-bearing tests. The
|
||||
> end-to-end proof is primarily an **offline simulation** with a scripted stand-in client, but one
|
||||
> **live run** against a real endpoint (`gpt-4.1-mini`, 2026-08-14) has also completed: it ended
|
||||
> in a correct `rejected` outcome — the deterministic validator caught a cost line the model had
|
||||
> invented outright (a code absent from the knowledge base), on the tolerance gate rather than the
|
||||
> stricter existence gate, because the bundle it ran against ships no cost baseline to anchor
|
||||
> against. No run has yet produced a **validated** proposal against a live model, and every
|
||||
> human-in-the-loop verdict currently seeded into the knowledge base is a synthetic, AI-authored
|
||||
> seed marked as such — no genuine expert verdict has entered the tree yet. The **ingest layer**
|
||||
> (real data sources) is implemented — file/CSV and SQL on both stacks with bit-identical golden
|
||||
> extractions from the shared spec, plus HTTP as a MAF-only demonstrated extension point against a
|
||||
> local mock — but exercised only against committed fixtures: no bundle has yet been materialized
|
||||
> from a live source. Ingested content passes a **content gate** (`materialize_gated`) that scans
|
||||
> every generated concept with
|
||||
> [`llm-ingestion-guard`](https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security)
|
||||
> before any of it reaches the bundle; a refused run writes nothing. A sibling implementation of
|
||||
> the same method on the **Claude Agents SDK** is built in parallel from the same shared spec.
|
||||
> the same method on the **Claude Agents SDK** exists in a separate repository but is currently
|
||||
> **parked**, not developed in parallel.
|
||||
|
||||
> **Disclaimer — technical framework only.** Deploying organizations own their processing
|
||||
> purposes and assessments (DPIA, risk/ROS, security review). The framework ships the technical
|
||||
|
|
@ -283,6 +325,13 @@ cross-links, with progressive disclosure — never by keyword retrieval or stuff
|
|||
bundle into a prompt. Query-time retrieval against the bundle is explicitly forbidden by the
|
||||
method spec: it would leak the verdict layer around the learning gate.
|
||||
|
||||
A cross-link that cannot be followed is **tolerated** (OKF SPEC §4 — navigation never raises) but
|
||||
no longer **silent**: each one is recorded on `Bundle.skipped` with the file it was written in, the
|
||||
link text verbatim, and which of the two reasons applied (`missing` — resolves inside the bundle
|
||||
with no readable file there; `outside-bundle` — resolves outside the bundle root). `--live-dry-run`
|
||||
and a full run both print the list, and print nothing when every link was followed — so a bundle
|
||||
that was only half read stops looking like a bundle that was simply smaller.
|
||||
|
||||
## AI-first, humans on top
|
||||
|
||||
A traditional wiki is built for *people* — optimized for humans finding and reading
|
||||
|
|
@ -353,6 +402,16 @@ when the seam is detached, so the loop cannot silently degrade into theater.
|
|||
|
||||
Deployment names are never committed: the role→model map ships `REPLACE-WITH-*` placeholders that
|
||||
fail fast, and `PORTFOLIO_MODEL_MAP` points at an out-of-tree map that wins over the bundled one.
|
||||
- **Tracing:** off unless `PORTFOLIO_OTEL` holds a value, and then it says where the spans go
|
||||
before it emits one. `console` writes them to **stderr**, so a traced run's stdout is
|
||||
byte-identical to an untraced one; `otlp` exports them over the network, and only to an endpoint
|
||||
you named in one of the standard `OTEL_EXPORTER_OTLP_*_ENDPOINT` variables. Asking for `console`
|
||||
while such a variable is set is refused rather than quietly honoured — those exporters are built
|
||||
unconditionally by the framework underneath, so "console" would have been a false statement about
|
||||
where the run's contents went. With the variable unset, no provider is configured at all: spans
|
||||
are still made and discarded, and nothing can leave the process. The OTLP exporter packages are
|
||||
not declared dependencies (they are egress, and heavy in a published wheel); install one yourself
|
||||
if you use that mode.
|
||||
- **Run:** the `run.py` CLI has **three modes** — a documented partition, since one invocation
|
||||
cannot exercise every flag:
|
||||
- **Single-project** — `PROJECT_ID --docs-dir <dir>`, plus optional `--bundle-dir`,
|
||||
|
|
@ -360,7 +419,9 @@ when the seam is detached, so the loop cannot silently degrade into theater.
|
|||
`--semantic-retrieval`, `--decision`/`--rationale`, `--live-dry-run`, and
|
||||
`--scripted-replies <file>` (the offline whole-loop door — see
|
||||
[Walk the whole chain offline](#walk-the-whole-chain-offline); mutually exclusive with
|
||||
`--live-dry-run`, which stops before the first model call rather than answering it).
|
||||
`--live-dry-run`, which stops before the first model call rather than answering it), and
|
||||
`--explore "<prompt>" --explore-config <file>` (opt-in: run an exploration first and let it
|
||||
shape the mandate this run evaluates — see below).
|
||||
- **Portfolio** — `--portfolio`, plus optional `--goals`, `--ledger`, `--dimension-config`,
|
||||
`--semantic-retrieval`; it stops early and prints a `goal reached: …` line when the
|
||||
accumulated ledger meets a goal.
|
||||
|
|
@ -380,6 +441,104 @@ when the seam is detached, so the loop cannot silently degrade into theater.
|
|||
uv run python -m portfolio_optimiser.run --report --ledger ledger.json
|
||||
```
|
||||
|
||||
`--explore` (U4) is **opt-in** and sits *over* the eight-step loop, never inside it. Given a
|
||||
prompt and a knowledge base, a Magentic manager decides what to read and which cost-saving
|
||||
directions are worth testing; what leaves that freedom is a **mandate**, and every number in it
|
||||
is still gated by the same blocking deterministic validator. The exploration writes nothing —
|
||||
no outbox artefact, no wiki entry, no verdict.
|
||||
|
||||
```bash
|
||||
uv run python -m portfolio_optimiser.run FV42-GSV-E1 --docs-dir <docs> --bundle-dir <bundle> \
|
||||
--explore "Find the cheapest saving worth testing here" --explore-config exploration.json
|
||||
```
|
||||
|
||||
`--explore-config` states the bounds, and **every field is required** — `max_rounds`,
|
||||
`max_tokens`, `max_stall_count`, `max_reset_count`, `max_plan_revisions`, `enable_plan_review`.
|
||||
None of them has a default, because an omitted cap falls back to an *unbounded* loop rather than
|
||||
a conservative one.
|
||||
|
||||
**Answering the plan review (`--plan-review`).** With `enable_plan_review` set, the exploration
|
||||
stops before the loop is allowed to run and asks you to sign the plan off. `--plan-review`
|
||||
answers it *at your terminal*: you are shown the plan, and you type `approve` or
|
||||
`revise <what to change>`. A revision goes back to the manager, which replans and asks you
|
||||
again about the **new** plan; `max_plan_revisions` bounds how many revisions are applied. Every
|
||||
round trip is recorded in `{run_id}-exploration.json` with your words verbatim. Input that ends
|
||||
without an answer is an error, never a sign-off — an autonomous loop must not run on a plan
|
||||
nobody approved. The two flags are refused apart: `enable_plan_review` without `--plan-review`
|
||||
would stop at a review nobody can answer, and `--plan-review` without it would build a reviewer
|
||||
nobody ever asks.
|
||||
|
||||
```bash
|
||||
uv run python -m portfolio_optimiser.run FV42-GSV-E1 --docs-dir <docs> --bundle-dir <bundle> \
|
||||
--explore "Find the cheapest saving worth testing here" --explore-config exploration.json \
|
||||
--plan-review --outbox-dir out --run-id r1
|
||||
```
|
||||
|
||||
The review is **synchronous**: the loop waits on you. That is why the hosted surface refuses it
|
||||
— blocking an HTTP request on a human would also block the event loop that answers
|
||||
`/readiness`.
|
||||
|
||||
**Answering it days later (`--checkpoint-dir` / `--resume`).** A domain expert is rarely at the
|
||||
terminal when the loop reaches the plan, so the same review can be *parked* to disk instead.
|
||||
`--checkpoint-dir` writes the suspended workflow there and the open question to
|
||||
`{run_id}-plan-review.json`, and the process exits. Whenever the expert gets to it — another
|
||||
day, in a process that never saw the run — they drop `{run_id}-plan-review-answer.json` into a
|
||||
review inbox, and `--resume` picks it up:
|
||||
|
||||
```bash
|
||||
# day 1 — park the review and exit
|
||||
uv run python -m portfolio_optimiser.run FV42-GSV-E1 --docs-dir <docs> --bundle-dir <bundle> \
|
||||
--explore "Find the cheapest saving worth testing here" --explore-config exploration.json \
|
||||
--checkpoint-dir checkpoints --outbox-dir out --run-id r1
|
||||
|
||||
# day N — a fresh process, resuming from what is on disk and nothing else
|
||||
uv run python -m portfolio_optimiser.run FV42-GSV-E1 --docs-dir <docs> --bundle-dir <bundle> \
|
||||
--outbox-dir out --checkpoint-dir checkpoints --review-inbox inbox --resume r1
|
||||
```
|
||||
|
||||
A `revise` answered this way does the same thing it does at the terminal: the manager replans
|
||||
and asks again about the **new** plan. The answer names the `request_id` it answers, and a
|
||||
mismatch is refused rather than applied — two reviews of one run share a file name, so an answer
|
||||
left over from the previous round would otherwise sign off a plan the expert never saw. The
|
||||
vocabulary is the same closed one, anything outside it is refused rather than read as approval,
|
||||
and `revise` with nothing to revise is refused too. `hitl.pending_plan_reviews(outbox, inbox)`
|
||||
lists every review still waiting on somebody.
|
||||
|
||||
The budget and the revision cap span the suspension — the resumed leg starts from what the
|
||||
parked one already spent, so a park never hands back a fresh budget. `--plan-review` and
|
||||
`--checkpoint-dir` are refused together (two doors onto one review), as are `--resume` and
|
||||
`--explore` (two sources of one exploration).
|
||||
|
||||
`--explore` is refused together with `--mandate` — they are two sources of one mandate, and
|
||||
merging would silently overwrite what you wrote. To seed an exploration with a domain expert's
|
||||
own hypotheses, use `explore(..., seed_approaches=[Approach(...)])`; seeds are always preserved
|
||||
and always come first, including when the loop stops early. With `--outbox-dir`/`--run-id` the
|
||||
run also writes `{run_id}-exploration.json`: the per-round ledger, the plan reviews and the
|
||||
in-loop advisory verdicts, written even when a cap cut the exploration short.
|
||||
|
||||
The hosted surface takes the same door as `explore_prompt` + `explore_contract` on
|
||||
`POST /invocations`.
|
||||
|
||||
**Several knowledge bases (library API).** An exploration may be given more than one base
|
||||
(`explore(..., bundle_dirs=[a, b])`). Each approach it shapes records which base it belongs to
|
||||
(`Approach.bundle_id`), and `run_mandate_across_bundles(mandate, bundle_dirs, ...)` then runs the
|
||||
pipeline **once per base** — the ordinary `run_project`, with that base's own sub-mandate, and
|
||||
with each run's project read from that base's own `validator-input.json`. `run_project` itself
|
||||
still takes one `bundle_dir`, deliberately: it derives the project, the validator's cost
|
||||
baseline, the agents' read context and the retrieval key from the base it is handed, so a second
|
||||
directory on that call would mean silently picking one of them. A hypothesis that names no base
|
||||
is refused when several are configured, rather than routed to a guess. The CLI's `--bundle-dir`
|
||||
stays single-valued; multi-base is a library door today.
|
||||
|
||||
Surveying those bases is deliberately cheap. `list_bundles` costs **O(bases), never O(corpus)**:
|
||||
each entry carries the base's id, a bounded verbatim opening of its index (with
|
||||
`index_truncated` beside it when the opening was cut), how many documents and prior expert
|
||||
verdicts it holds, whether it ships a cost baseline, and how many cross-links could not be
|
||||
followed — never the whole index. The full index stays one `read_file(id, "index.md")` away, so
|
||||
the bound is a disclosure level rather than data loss. Measured 2026-08-26 over a real corpus:
|
||||
112 116 → 362 tokens for three bases, 124 942 → 21 448 for 171
|
||||
([report](docs/2026-08-26-katalogkostnaden.md)).
|
||||
|
||||
`--semantic-retrieval` (S3.1) is an **opt-in** ranking change, **off by default**. Off, prior
|
||||
verdicts are ranked exactly as before: a structural score over the affected cost-code set,
|
||||
measure type and magnitude bucket, with surface text deliberately excluded. On, that score is
|
||||
|
|
@ -419,9 +578,19 @@ when the seam is detached, so the loop cannot silently degrade into theater.
|
|||
reference-domain (non-bundle) path the project's own cost items are the baseline, so those runs
|
||||
are always anchored.
|
||||
|
||||
**An un-anchored run says so.** Every run stamps `provenance.cost_baseline_anchored` (a required
|
||||
boolean, so no stamp can omit it), which reaches the outbox in `{run_id}-proposal.json`; and when
|
||||
a run is un-anchored the CLI prints one line naming the skipped stage — on `--live-dry-run`, on a
|
||||
full single run, and per project in portfolio mode. An anchored run prints no such line at all:
|
||||
a line for something the run does not have is omitted rather than rendered empty. Anchoring stays
|
||||
optional; this is visibility, not a new refusal.
|
||||
|
||||
The **prior-verdict fold — the learning step — happens only on the `--bundle-dir` path**; a
|
||||
plain `--docs-dir`-only run is single-shot (no fold). `--decision`/`--rationale` apply to the
|
||||
single-project path only and are inert in portfolio mode. **`--outbox-dir` must differ from
|
||||
single-project path only and are **refused** in portfolio and report mode rather than ignored —
|
||||
a portfolio pass takes each project's verdict from its own row, so a run-level verdict flag has
|
||||
nowhere to go, and silently dropping a judgement an expert actually typed is the failure this
|
||||
partition exists to prevent. **`--outbox-dir` must differ from
|
||||
`--verdict-dir`**: writing the raw outbox into a folder later read as an inbox would re-ingest
|
||||
raw agent output past the promotion gate (self-contamination) — documented here, deliberately
|
||||
not CLI-enforced. Stop criteria and budget caps are required at startup. Try the offline
|
||||
|
|
@ -456,37 +625,57 @@ only an expert has:
|
|||
- **Vendor-neutral knowledge** — the same bundles drive two different agent stacks; switching
|
||||
frameworks does not orphan the organization's curated knowledge.
|
||||
|
||||
## Hosted on Foundry Agent Service
|
||||
## The task API — runnable Python, no wrapper
|
||||
|
||||
The repo ships the three files a hosted-agent deployment needs. All three are measured —
|
||||
schema-validated and container-built — but an actual `azd` deploy has **not** been run; that
|
||||
step is the operator's.
|
||||
`main.py` is the ONE process entry. It serves the hosted-agent runtime contract (port 8088 /
|
||||
`PORT`, `GET /readiness`, `POST /invocations`, clean SIGTERM exit) from
|
||||
`portfolio_optimiser.hosting`, wrapping `run_project` — never `Workflow.as_agent()`, which
|
||||
would bypass the deterministic validator and the rest of the gate chain. No protocol library
|
||||
is used: the prerelease `agent-framework-foundry-hosting` requires
|
||||
`agent-framework-core>=1.13.0` while this tree locks 1.9.0, so the contract is a small
|
||||
stdlib-asyncio server instead (single event loop, no threads — the same concurrency model as
|
||||
the portfolio waves).
|
||||
|
||||
- `main.py` — the ONE process entry. It serves the hosted-agent runtime contract (port 8088 /
|
||||
`PORT`, `GET /readiness`, `POST /invocations`, clean SIGTERM exit) from
|
||||
`portfolio_optimiser.hosting`, wrapping `run_project` — never `Workflow.as_agent()`, which
|
||||
would bypass the deterministic validator and the rest of the gate chain. No protocol library
|
||||
is used: the prerelease `agent-framework-foundry-hosting` requires
|
||||
`agent-framework-core>=1.13.0` while this tree locks 1.9.0, so the contract is a small
|
||||
stdlib-asyncio server instead (single event loop, no threads — the same concurrency model as
|
||||
the portfolio waves).
|
||||
- `Dockerfile` — `python:3.12-slim-bookworm` + git (the two security components are
|
||||
git-tag-pinned direct references; wheel metadata alone cannot install them) +
|
||||
`uv sync --frozen --no-dev` (the exact locked resolution every measurement ran against).
|
||||
Build it the way a stranger would:
|
||||
`git archive HEAD | docker build --platform linux/amd64 -t portfolio-optimiser -`
|
||||
- `azure.yaml` — closed against the two authoritative JSON schemas (see
|
||||
`docs/2026-08-13-fase4-azure-yaml-valg.md`): `language: docker`, invocations protocol pinned
|
||||
to `2.0.0`, no `env:` block (the platform injects `FOUNDRY_PROJECT_ENDPOINT`; model choice is
|
||||
config, not environment), no `startupCommand` (the image `CMD` is the one copy).
|
||||
```bash
|
||||
uv sync --frozen --no-dev # the exact locked resolution every measurement ran against
|
||||
uv run python main.py # serves the task API
|
||||
```
|
||||
|
||||
An invocation is a JSON object whitelisted onto `run_project`'s signature — `project_id`,
|
||||
`docs_dir` and `verdict_input` required; `bundle_dir`, `profile`, `max_rounds`, `max_tokens`
|
||||
and `top_k` optional. Unknown fields are refused by name (400), never silently dropped.
|
||||
`git` must be on PATH for the install: two dependencies are git-tag-pinned direct references,
|
||||
and wheel metadata alone cannot fetch them.
|
||||
|
||||
A `Dockerfile` and an `azure.yaml` shipped here until **14 August 2026** and were removed on an
|
||||
operator directive after an external trial: what is delivered is runnable Python, and how the
|
||||
process is containerised, supervised or deployed belongs to whoever runs it. Git history keeps
|
||||
both files. The raw-text gate that pinned them (`--platform linux/amd64`, one copy of the start
|
||||
command) was **deleted with them** rather than weakened into a check that could only pass — the
|
||||
start command now has exactly one copy, in [`DEPLOY.md`](DEPLOY.md), and
|
||||
`tests/test_handover_package_loadbearing.py` is what keeps it there.
|
||||
|
||||
An invocation is a JSON object whitelisted onto `run_project`'s signature — `project_id` and
|
||||
`docs_dir` required; `verdict_input`, `bundle_dir`, `profile`, `max_rounds`, `max_tokens`
|
||||
and `top_k` optional. `verdict_input` was required until 1.1.0, which forced an external caller to
|
||||
invent an expert verdict just to get a run at all; omitting it now means nobody reviewed the run,
|
||||
and the response's `verdict_id` is the key one would arrive under. Unknown fields are refused by name (400), never silently dropped.
|
||||
`profile` defaults to `azure` on this surface: the AZURE profile reads its endpoint and
|
||||
credential from the environment at call time, so the same image runs hosted (managed identity)
|
||||
credential from the environment at call time, so the same process runs hosted (managed identity)
|
||||
and locally (`az login`) without rewiring.
|
||||
|
||||
### Handing it to someone else
|
||||
|
||||
`scripts/make-handover-package.sh` builds one archive a receiving organisation can deploy without
|
||||
cloning this repository or having an account here:
|
||||
|
||||
```bash
|
||||
scripts/make-handover-package.sh # → dist/portfolio-optimiser-foundry-<version>.zip
|
||||
```
|
||||
|
||||
The archive is `git archive HEAD` — tracked files only, which is why local-only files cannot enter
|
||||
it and why nothing curates what a receiver sees. [`DEPLOY.md`](DEPLOY.md) rides
|
||||
along inside it and answers the receiver's first questions: what the three roles do, what the
|
||||
process is end to end, why there is no chat interface, and the two environment variables that decide
|
||||
whether the first deployment works. Gated by `tests/test_handover_package_loadbearing.py`.
|
||||
|
||||
## Docs
|
||||
|
||||
- [Bestille en kjøring](docs/bestille-en-kjoring.md) *(norsk)* — for the domain expert who
|
||||
|
|
@ -501,6 +690,11 @@ and locally (`az login`) without rewiring.
|
|||
- [Building a knowledge base](docs/knowledge-base-recipe.md) — the team recipe (technical +
|
||||
domain expert) for curating a bundle, with the honest expectation that a good base takes 1–2
|
||||
weeks of dedicated work.
|
||||
- [Kunnskapsbase for én kjøring](docs/kunnskapsbase-for-en-kjoring.md) *(norsk)* — how to compose
|
||||
the base for ONE specific run: which categories of knowledge follow the project, the domain and
|
||||
the organisation; a content-type table (owner, delivery form, role in the loop, what happens
|
||||
when it is missing); and a worked road project from the commission to a base that passes the
|
||||
dry-run check. Every technical claim is marked verified or assumed.
|
||||
- [Target picture](docs/plan/2026-06-26-maalbilde-agentic-loop.md) — the agentic loop + OKF
|
||||
knowledge architecture (north star).
|
||||
- [Prior-art & platform research](docs/research/2026-06-23-prior-art-platform.md) (incl.
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ We take security seriously. If you discover a security vulnerability, please rep
|
|||
|
||||
### How to Report
|
||||
|
||||
Email: hello@fromaitochitta.com
|
||||
Email: security@fromaitochitta.com
|
||||
|
||||
Include:
|
||||
- Description of the vulnerability
|
||||
|
|
|
|||
28
azure.yaml
28
azure.yaml
|
|
@ -1,28 +0,0 @@
|
|||
# yaml-language-server: $schema=https://raw.githubusercontent.com/Azure/azure-dev/main/schemas/v1.0/azure.yaml.json
|
||||
#
|
||||
# Fase 4d. The four open choices were closed against the two authoritative JSON schemas
|
||||
# (docs/2026-08-13-fase4-azure-yaml-valg.md), not against Learn prose:
|
||||
#
|
||||
# * `language: docker`, NO `codeConfiguration` — codeConfiguration is ZIP-source mode,
|
||||
# and the git-pinned dependencies need our own Dockerfile (git in the base image).
|
||||
# * `protocols` pinned to invocations 2.0.0, stated outright: the header contract in
|
||||
# spike §1.5 (x-agent-user-id / x-agent-foundry-call-id, gateway allowlist) is a
|
||||
# 2.0.0 property, and 1.0.0 is deprecated with a blocking deadline.
|
||||
# * `project` is REQUIRED for host azure.ai.agent (the $ref'd azure.ai.agent.json);
|
||||
# `config:` is deprecated there and not used.
|
||||
# * NO `env:` block, deliberately: FOUNDRY_PROJECT_ENDPOINT must never be redeclared
|
||||
# (it would risk shadowing the platform's injected value — spike pkt. 2), and model
|
||||
# selection is config (the packaged data/model_map.json, override via
|
||||
# PORTFOLIO_MODEL_MAP), not environment variables.
|
||||
# * NO `startupCommand`, deliberately: the image's CMD (Dockerfile) is the ONE copy of
|
||||
# the start command — a second copy here could drift from it.
|
||||
name: portfolio-optimiser
|
||||
services:
|
||||
portfolio-optimiser:
|
||||
host: azure.ai.agent
|
||||
kind: hosted
|
||||
project: .
|
||||
language: docker
|
||||
protocols:
|
||||
- protocol: invocations
|
||||
version: "2.0.0"
|
||||
|
|
@ -1054,8 +1054,8 @@
|
|||
<h2>Tre forbehold</h2>
|
||||
<div class="cards">
|
||||
<div class="card">
|
||||
<span class="t">Svarene er skriptet</span>
|
||||
<p>Det var ingen levende språkmodell involvert i kjøringen. Den viser at maskineriet virker — ikke at en modell ville foreslått nettopp dette. Grunnen er nøktern: modellkjøringer koster penger, og dette er bygget uten budsjett.</p>
|
||||
<span class="t">Svarene i demoen er skriptet</span>
|
||||
<p>Kjøringen dere nettopp så, hadde ingen levende språkmodell involvert — svarene er forhåndsskrevne, så tallene blir de samme hver gang. Rammeverket har kjørt mot en levende modell én gang, 14. august 2026: modellen fant opp en kostnadslinje som ikke finnes i kunnskapsbasen, og regneporten avviste forslaget. At en levende modell foreslår noe som slipper gjennom porten, er fortsatt ikke vist.</p>
|
||||
</div>
|
||||
<div class="card">
|
||||
<span class="t">Kunnskapsbasen er laget for hånd</span>
|
||||
|
|
@ -1078,7 +1078,9 @@
|
|||
<div class="stat-row"><span><span class="pill ok">Bygget</span></span><span>Hele kjeden, fra kunnskapsgrunnlag til lagret fagvurdering</span></div>
|
||||
<div class="stat-row"><span><span class="pill ok">Bygget</span></span><span>Den blokkerende kontrollen — kan ikke slås av</span></div>
|
||||
<div class="stat-row"><span><span class="pill ok">810 tester</span></span><span>Grønne, målt på nøyaktig den versjonen tallene her kommer fra</span></div>
|
||||
<div class="stat-row"><span><span class="pill ok">Kjørt én gang</span></span><span>Mot en levende språkmodell, 14. august 2026 — modellen svarte i riktig form, og kjøringen konkluderte med en avvisning</span></div>
|
||||
<div class="stat-row"><span><span class="pill warn">Står for tur</span></span><span>Automatisk produksjon av kunnskapsbaser — mekanismen finnes, koblingen gjenstår</span></div>
|
||||
<div class="stat-row"><span><span class="pill bad">Ikke vist</span></span><span>Et forslag fra en levende modell som passerer regneporten</span></div>
|
||||
<div class="stat-row"><span><span class="pill bad">Ikke gjort</span></span><span>Pilot på ekte prosjektdata</span></div>
|
||||
</div>
|
||||
<p class="foot-note">Koden er åpen og fritt tilgjengelig. Ingen lisenskostnad i selve rammeverket, og ingen binding til én leverandør.</p>
|
||||
|
|
|
|||
|
|
@ -334,7 +334,7 @@
|
|||
<div class="prose"><p>Grunnregelen systemet er bygget på, er at det ikke får påstå mer enn det gjør. En demo som overselger, bryter med akkurat det den demonstrerer — så disse tre står like tydelig som resten.</p></div>
|
||||
<div class="forbehold">
|
||||
<div class="fb"><span class="n">1</span>
|
||||
<p><span class="t">Agentenes svar i demoen er skriptet — det er ingen levende språkmodell i rommet.</span>Det som demonstreres er at dataflyten virker, at den deterministiske ryggraden faktisk blokkerer, og at læringssløyfa lukkes. Det som <em>ikke</em> demonstreres, er at en levende modell ville produsert nettopp dette forslaget. Grunnen er nøktern: modellkjøringer koster penger, og dette er utviklet uten budsjett. Selve rammeverket er bygget for å kjøre mot ekte modeller, og gjør det — det er demoen som er skrudd av, ikke koden.</p>
|
||||
<p><span class="t">Agentenes svar i demoen er skriptet — det er ingen levende språkmodell i rommet.</span>Det som demonstreres er at dataflyten virker, at den deterministiske ryggraden faktisk blokkerer, og at læringssløyfa lukkes. Rammeverket selv har kjørt mot en levende modell én gang, 14. august 2026 — etter at dette underlaget ble skrevet: modellen svarte i den formen systemet bestiller, den fant opp en kostnadslinje som ikke finnes i kunnskapsbasen, og regneporten avviste forslaget. At et forslag fra en levende modell kommer <em>gjennom</em> porten, er fortsatt ikke vist.</p>
|
||||
</div>
|
||||
<div class="fb"><span class="n">2</span>
|
||||
<p><span class="t">Kunnskapsbasen i demoen er laget for hånd.</span>Et menneske har skrevet den. Det finnes en vei for å hente eksterne kilder inn i formatet — og den skanner nå innholdet for manipulert kildetekst før det skrives — men eksempelet her gikk ikke gjennom den, og den generiske «fabrikken» som skal produsere slike baser for vilkårlige fagområder, er bevisst ikke bygget ennå.</p>
|
||||
|
|
|
|||
247
docs/2026-08-14-fase1b-forste-levende-kjoring.md
Normal file
247
docs/2026-08-14-fase1b-forste-levende-kjoring.md
Normal file
|
|
@ -0,0 +1,247 @@
|
|||
# Fase 1b — den første levende kjøringen (14. august 2026)
|
||||
|
||||
Dette dokumentet er et **måleprotokoll**, ikke en konklusjon. Alt under er kjørt; ingenting er utledet.
|
||||
Der noe ikke er bevist, står det.
|
||||
|
||||
## 0. Hva som ble opprettet
|
||||
|
||||
| Ressurs | Verdi |
|
||||
|---|---|
|
||||
| Ressursgruppe | `<resource-group>` (eastus) |
|
||||
| Foundry-ressurs | `<resource>` — `kind: AIServices`, `--allow-project-management` |
|
||||
| Prosjekt | `<project>` |
|
||||
| Deployment | `gpt-4-1-mini` (modell `gpt-4.1-mini`, versjon `2025-04-14`, GlobalStandard) |
|
||||
| Prosjekt-endepunkt | `https://<resource>.services.ai.azure.com/api/projects/<project>` |
|
||||
| Rolle | `Foundry User` (`53ca6127-…`) på **prosjekt**-scope |
|
||||
|
||||
Ressurs-, prosjekt- og vertsnavnene er byttet mot plassholdere (`<resource-group>` / `<resource>` /
|
||||
`<project>`, samme form som `DEPLOY.md` og auth-oppskriften). Modellnavn, versjon og region er
|
||||
offentlig Azure-nomenklatur og står som målt.
|
||||
|
||||
**`gpt-4o-mini` ble avvist av plattformen** med `ServiceModelDeprecating` — modellnavnet måtte måles
|
||||
med `az cognitiveservices model list`, ikke hentes fra hukommelsen. Kommandoene er hentet fra
|
||||
Microsoft Learn (`az cognitiveservices account create` / `account project create`), ikke formulert
|
||||
fritt.
|
||||
|
||||
**Rotårsaken til at dette ikke fantes før** var prosedural, ikke teknisk: den påloggede identiteten
|
||||
har vært **Owner på abonnementet** hele tiden, mens repoets egen state-fil hadde ført opprettelsen
|
||||
opp som operatørens oppgave og samtidig sagt at operatøren aldri kjører kommandoer selv. Arbeidet
|
||||
tilhørte ingen, og hver økt målte lydig på nytt at det ikke fantes.
|
||||
|
||||
## 1. Stigen — billigste trinn først
|
||||
|
||||
Disiplinen fra fase 1a: bevis så mye som mulig før det dyre trinnet, så en feil er attribuerbar.
|
||||
|
||||
| Trinn | Kommando | Utfall |
|
||||
|---|---|---|
|
||||
| 1 | `az account get-access-token --resource https://ai.azure.com` | exit 0 |
|
||||
| 2 | `preflight --profile azure` (fra **utpakket overleveringspakke**) | `preflight OK (azure)` |
|
||||
| 3 | `pytest tests/test_foundry_profile_live.py` (gatet triviell probe) | **1 passed, 5,02 s** |
|
||||
| 4 | Full `run_project` mot levende modell | **RC=1 — `BudgetExceeded`** |
|
||||
|
||||
**Trinn 3 er prosjektets første levende modellkall noensinne.** Det beviser at auth, RBAC,
|
||||
endepunkt-form og deployment-navn komponerer — og at en rød trinn 4 derfor *ikke* kan skyldes noen
|
||||
av dem.
|
||||
|
||||
## 2. Trinn 4 — hva som faktisk skjedde
|
||||
|
||||
```
|
||||
uv run python -m portfolio_optimiser.run BYGG-KONTOR-NORD \
|
||||
--profile azure \
|
||||
--docs-dir shared/examples/bygg-energi-mikro \
|
||||
--bundle-dir shared/examples/bygg-energi-mikro \
|
||||
--outbox-dir <tmp>/outbox --run-id live-001
|
||||
```
|
||||
|
||||
stdout var **tom**. stderr bar to linjer som betyr noe:
|
||||
|
||||
```
|
||||
GroupChatOrchestrator reached max_rounds=3; forcing completion.
|
||||
portfolio_optimiser.budget.BudgetExceeded: budget exceeded: rounds limit=12 observed=13
|
||||
```
|
||||
|
||||
Den første er forventet — maker/checker-debatten kjører til taket også i den offline demoen. Den
|
||||
andre er funnet.
|
||||
|
||||
### Diagnosen, utledet av kode og ikke av flere betalte kjøringer
|
||||
|
||||
`generate._fetch_parsed` er en `while True` som kaller `meter.tick_round()` for **hvert** forsøk og
|
||||
`continue`-er ved parse-feil. Rund-budsjettet i genereringsfasen er `max(max_rounds * 4, 4)` = **12**
|
||||
(`run.py:546`). Hadde modellsvarene parset til IR-formen, ville tre validerings-forsøk kostet tre
|
||||
runder. At alle tolv gikk med betyr at **de fleste svarene fra `gpt-4.1-mini` ikke lot seg parse** —
|
||||
taket ble brent på formatfeil, ikke på validator-avslag.
|
||||
|
||||
**Dette er et FUNN, ikke en diagnose som er ferdig.** Det som mangler for å lukke den, er den råe
|
||||
svarteksten, og den finnes ikke i noen artefakt i dag (`--outbox-dir` skrev kun
|
||||
`live-001-runconfig.json`). Å skaffe den krever en endring i koden, altså Iron-Law-arbeid — ikke en
|
||||
rask omkjøring med høyere tak, som ville kostet penger og fortsatt ikke sagt hvorfor.
|
||||
|
||||
### En bieffekt som er verdt å uttale
|
||||
|
||||
`BudgetExceeded` forlot kjøringen som en **uhåndtert exception med traceback**, ikke som et
|
||||
strukturert utfall. På den hostede flaten ville nøyaktig dette blitt `HTTP 500` — altså ville
|
||||
ressurs-utmattelse (en normal, forventet tilstand) presentert seg for en ekstern kaller som en
|
||||
serverfeil. Det er ikke rettet her; det er notert.
|
||||
|
||||
## 3. Hva som ER bevist, og hva som IKKE er det
|
||||
|
||||
**Bevist, målt:**
|
||||
|
||||
- Rammeverket når en levende Foundry-modell: auth, RBAC på prosjekt-scope, endepunkt-form,
|
||||
deployment-oppslag og klientbygging virker.
|
||||
- Kunnskapsbase-navigasjon, debatt-orkestrering og genererings-løkka kjører mot ekte modellsvar —
|
||||
kjøringen døde *inne i* løkka, ikke før den.
|
||||
- Preflight fra den **utpakkede overleveringspakka** er grønn mot et ekte prosjekt, både med vårt
|
||||
eget endepunkt-variabelnavn og med plattformens injiserte.
|
||||
|
||||
**Ikke bevist:**
|
||||
|
||||
- At systemet produserer et **validert forslag** mot en levende modell. Det har det aldri gjort.
|
||||
Kjøringen nådde aldri fram til validatoren med en parsebar kandidat.
|
||||
- At `gpt-4.1-mini` er en egnet modell for proposer-rollen. Målingen peker mot at den ikke er det
|
||||
uten endret prompting eller structured output — men én kjøring er én kjøring.
|
||||
- Noe som helst om kostnad i drift. Denne kjøringen kostet noen få øre; det sier ingenting om en
|
||||
reell portefølje.
|
||||
|
||||
## 4. Neste steg, i rekkefølge
|
||||
|
||||
1. **Fang den råe svarteksten** ved parse-feil (i dag forsvinner den i `except: continue`). Test
|
||||
først — dette er en søm, ikke en logg-linje.
|
||||
2. Vurder **structured output** mot Foundry for proposer-rollen, framfor å prompte fram JSON.
|
||||
3. Vurder om `BudgetExceeded` skal bli et strukturert utfall i stedet for en traceback, særlig for
|
||||
den hostede flaten.
|
||||
4. Først når 1 og 2 er på plass: en ny bundet kjøring, og en gatet test som dekker **hele
|
||||
`run_project`-stien** (den eksisterende gatede testen dekker kun klient-nivået).
|
||||
|
||||
**Rettelse til punkt 4, målt 14.08 (økt 40):** parentesen er upresis. `tests/test_portfolio_live.py`
|
||||
fantes allerede og dekker `run_portfolio`-utfoldingen — altså hele `run_project`-stien — med samme
|
||||
env-gate. Men den asserterer `len(result.runs) == 1`, og fordi `runs` og `failures` *partisjonerer*,
|
||||
kan den kun skille «kjøringen fullførte» fra «kjøringen raiste». Den kan **ikke** skille et validert
|
||||
forslag fra et avvist, og heller ikke en kjøring der svarene parset fra en der de ikke gjorde det.
|
||||
Den bærer derfor ikke påstanden 1b skal felle. Punkt 4 står, men grunnen er en annen enn skrevet.
|
||||
|
||||
## 5. Pre-registrerte utfall (skrevet FØR kjøringen)
|
||||
|
||||
Denne kjøringen har ett formål: å felle den ene gjenstående ærlighets-grensen fra økt 37 — **at det
|
||||
emitterte `response_format`-skjemaet ER akseptert av det LEVENDE endepunktet er uverifisert**;
|
||||
testene beviser konformitet med Azures *dokumenterte* subset, ikke aksept.
|
||||
|
||||
**Instrumentet** er `conftest.assert_full_run_contract`, og diskriminatoren er et artefakt repoet
|
||||
allerede eier: `{run_id}-parse-failures.json` skrives hvis og bare hvis et svar ikke lot seg parse
|
||||
(økt 35). Artefaktets **fravær** ved siden av et `RunResult` beviser at hvert genererings-svar kom
|
||||
tilbake i den bestilte formen. Kontraktens evne til å skille er bevist **offline og gratis**
|
||||
(`tests/test_live_full_run_contract.py`, to mutasjoner med hver sin signatur: detach artefakt-sjekken
|
||||
→ T1 rød alene; raise ubetinget → T2 rød alene). Det betalte kallet er *målingen*, ikke beviset på at
|
||||
måleinstrumentet virker.
|
||||
|
||||
**Hva hvert utfall betyr — avgjort på forhånd:**
|
||||
|
||||
| Utfall | Betydning |
|
||||
|---|---|
|
||||
| Ingen parse-failure-artefakt + validatoren avgjorde (`validated` **eller** `rejected`) | **Ærlighets-grensen er felt.** Skjemaet ble akseptert av det levende endepunktet. En P90-avvisning er et *bestått* utfall — kjøringen KONKLUDERTE. |
|
||||
| Parse-failure-artefaktet finnes | Skjemaet ble **ikke** honorert. Et FUNN, ikke et bevis — og denne gangen finnes den råe teksten (økt 35), så neste steg kan begrunnes i stedet for gjettes. |
|
||||
| `BudgetExceeded` | Fortsatt ubevist, men artefaktet forklarer hvorfor. Samme form som 1a-kjøringen. |
|
||||
|
||||
**Taket heves IKKE.** `max_rounds`/`max_tokens` står på defaultene den første levende kjøringen døde
|
||||
på: fyrer ledgeren igjen, er dét informasjon, og å heve taket ville brukt mer penger på en sti som
|
||||
kanskje fortsatt er brukket.
|
||||
|
||||
**Gate-variabelen er en TREDJE, distinkt opt-in** (`PORTFOLIO_LIVE_FULL_RUN`, lest på *truthiness*).
|
||||
Begge de eksisterende live-testene gater på nøyaktig `PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT` +
|
||||
`PORTFOLIO_FOUNDRY_DEPLOYMENT`; å gjenbruke det paret ville betydd at en operatør som eksporterer de
|
||||
to for å kjøre den **billige** ett-ords-proben også fyrer den dyre fullkjøringen — altså at stigen i
|
||||
§1 kollapser til ett trinn. `PORTFOLIO_MODEL_MAP` er med i skip-betingelsen av en annen grunn:
|
||||
uten den feiler kjøringen av en *konfigurasjons*-årsak som ser ut som en modell-feil.
|
||||
|
||||
## 6. Den bundne fullkjøringen — målt 14.08 kl. 19:54
|
||||
|
||||
### Oppsettet, reproduserbart
|
||||
|
||||
Model-mappet ligger **out-of-tree** med vilje: `data/model_map.json` i treet bærer
|
||||
`REPLACE-WITH-FOUNDRY-DEPLOYMENT`-plassholdere, og tenant-spesifikke deployment-navn skal aldri
|
||||
committes (B12). `PORTFOLIO_MODEL_MAP` peker på en fil operatøren eier:
|
||||
|
||||
```json
|
||||
{
|
||||
"local": {"default": "qwen3:4b", "proposer": "qwen3:4b", "checker": "qwen3:4b"},
|
||||
"azure": {
|
||||
"default": "gpt-4-1-mini",
|
||||
"proposer": "gpt-4-1-mini",
|
||||
"checker": "gpt-4-1-mini"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```bash
|
||||
export PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT="https://<resource>.services.ai.azure.com/api/projects/<project>"
|
||||
export PORTFOLIO_FOUNDRY_DEPLOYMENT="gpt-4-1-mini"
|
||||
export PORTFOLIO_MODEL_MAP="/absolutt/sti/til/model_map.live.json"
|
||||
export PORTFOLIO_LIVE_FULL_RUN=1 # BETALER — utelat den for alt annet enn trinn 4
|
||||
```
|
||||
|
||||
Merk at `PORTFOLIO_LIVE_FULL_RUN` bevisst settes SIST og alene for det dyre trinnet: uten den er
|
||||
trinn 2 og 3 gratis-nok til å kjøres fritt, og det er hele stigens poeng.
|
||||
|
||||
### Stigen
|
||||
|
||||
Stigen på nytt, samme disiplin som §1. Alt under er kjørt.
|
||||
|
||||
| Trinn | Kommando | Utfall |
|
||||
|---|---|---|
|
||||
| 1a | `az account get-access-token --resource https://ai.azure.com` (uten rør) | exit 0, `expiresOn 21:07:40` |
|
||||
| 1b | `az ad signed-in-user show` (ekte Graph-kall, ikke lokal cache) | exit 0 — gmail-kontoen, ikke jobbkontoen |
|
||||
| 2 | `preflight --profile azure` | `preflight OK (azure)` |
|
||||
| 3 | `pytest tests/test_foundry_profile_live.py` | **1 passed, 5,48 s** |
|
||||
| 3b | samme kall, med `test_full_run_live.py` samlet | **SKIPPED** — det tredje flagget holder stigen |
|
||||
| 4 | `pytest tests/test_full_run_live.py` (`PORTFOLIO_LIVE_FULL_RUN=1`) | **1 passed, 22,87 s** |
|
||||
|
||||
Trinn 3b er verdt å uttale: den dyre testen hoppet over **selv med begge Foundry-variablene satt**.
|
||||
Det er den empiriske bekreftelsen på at det tredje opt-in-flagget gjør jobben sitt design lover.
|
||||
|
||||
### Ærlighets-grensen ER felt
|
||||
|
||||
Outboksen inneholder `-proposal.json`, `-outcome.json`, `-runconfig.json` — og **ingen
|
||||
`-parse-failures.json`**. Hvert eneste genererings-svar fra `gpt-4.1-mini` kom tilbake som et
|
||||
parsebart objekt i den bestilte formen. Økt 37s uttalte grense — *«at det emitterte skjemaet ER
|
||||
akseptert av det LEVENDE endepunktet er IKKE verifisert»* — er dermed **lukket ved måling**, ikke
|
||||
ved resonnement. Kontrasten til den første levende kjøringen er hele funnet: der brant tolv runder
|
||||
på formatfeil og etterlot null tegn; her feilet ingen.
|
||||
|
||||
`assumptions` kom tilbake som forventet (`{"EL-LIGHTING-OP-HR": [10.0, 13.0]}`), altså virker økt 37s
|
||||
additive wire-form → IR-map-normalisering ende-til-ende mot et levende endepunkt. Det var
|
||||
beslutningen som holdt den stokastiske falsifisereren fra å gå inert, og den er nå prøvd i felt.
|
||||
|
||||
### Utfallet: `rejected` — og det er et bestått utfall
|
||||
|
||||
```
|
||||
outcome_type: rejected
|
||||
reason: claimed saving 34500 exceeds P90 feasible 11488
|
||||
checker_verdict: approve
|
||||
validator_decision: rejected
|
||||
token_usage: 15 306
|
||||
```
|
||||
|
||||
Kjøringen KONKLUDERTE. Per §5s pre-registrering er dette det positive utfallet: validatoren ble nådd
|
||||
med en parsebar kandidat og avgjorde. **Og de to falsifisererne skilte lag for første gang mot en
|
||||
levende modell** — checkeren godkjente *resonnementet*, validatoren avviste *tallene*. Nøyaktig den
|
||||
uavhengigheten `checker_verdict` holdes atskilt fra `provenance.validator_decision` for.
|
||||
|
||||
### FUNN som er viktigere enn den grønne testen: modellen fant opp en kostkode
|
||||
|
||||
Forslaget bar `code: "EL-LIGHTING-OP-HR"` (3 000 × 11,5). **Den koden finnes ikke noe sted i
|
||||
kunnskapsbasen** (`grep` over hele bundelen: null treff). Kunnskapsbasen instruerer eksplisitt
|
||||
mappingen `ENERGI-TOTAL-EL`, 300 000 kWh × 1,00 NOK — modellen konstruerte i stedet sin egen
|
||||
kostlinje med en egen enhet.
|
||||
|
||||
Avvisningen var derfor **riktig, men skjedde på feil gate**: 30 %-cap-en fanget den på *magnitude*
|
||||
(34 500 > P90 11 488), ikke stage 0 på *eksistens*. Grunnen er en kjent og uttalt egenskap, ikke en
|
||||
defekt: S4.0-forankringen aktiveres på bundle-stien KUN når bundelen shipper `cost-baseline.json`,
|
||||
og `bygg-energi-mikro` gjør ikke det (målt) — «en pre-amendment-bundle er legitimt uforankret».
|
||||
|
||||
Dette er akkurat den hallusinasjons-klassen S4.0 ble bygget for, observert i felt for første gang.
|
||||
At den uforankrede gaten fanget den likevel er betryggende; at den fanget den på den dyre gaten
|
||||
i stedet for den billige er en kø-post, ikke noe som endres her.
|
||||
|
||||
**Fortsatt ikke bevist:** at systemet produserer et *validert* forslag mot en levende modell. Denne
|
||||
kjøringen avviste — korrekt, og med en begrunnelse som kan leses. Én kjøring er én kjøring, og
|
||||
`gpt-4.1-mini`s egnethet for proposer-rollen er ikke avgjort av den.
|
||||
199
docs/2026-08-18-vurdering-azure-omdoeping.md
Normal file
199
docs/2026-08-18-vurdering-azure-omdoeping.md
Normal file
|
|
@ -0,0 +1,199 @@
|
|||
# Vurdering: skal Azure-ressursene døpes om?
|
||||
|
||||
**Bestilt av** ordre `20260818T103716Z-251212929` (fra `.claude`). **Leveransen er en vurdering, ikke
|
||||
en omdøping** — ingenting i Azure er rørt. Beslutningen er operatørens.
|
||||
|
||||
Alt under er enten **målt** i dette repoet eller **verifisert mot Microsoft Learn**. Der noe ikke er
|
||||
verifisert, står det. Kildene er listet i §6.
|
||||
|
||||
## 1. Hva røper koordinatene faktisk?
|
||||
|
||||
Koordinatene som nådde en ekstern organisasjon 14.08 (i `dist/portfolio-optimiser-foundry-1.1.0.zip`,
|
||||
bygget 14:24):
|
||||
|
||||
| Verdi | Type |
|
||||
|---|---|
|
||||
| `<resource-group>` | ressursgruppe (ARM-planet) |
|
||||
| `<resource>` | Foundry-ressurs = **custom subdomain** (DNS) |
|
||||
| `<project>` | prosjekt (ARM + data-plan-sti) |
|
||||
| `eastus` | region |
|
||||
| `https://<resource>.services.ai.azure.com/api/projects/<project>` | prosjekt-endepunkt |
|
||||
|
||||
De literale navnene er byttet mot plassholdere her, samme form som `DEPLOY.md`, auth-oppskriften og
|
||||
måleprotokollen etter `241b50d`. Dokumentet handler om hva navnene *er*, ikke om hvilke de var — og
|
||||
innholds-gaten i `tests/test_handover_package_loadbearing.py` avviste førsteutkastet som bar dem.
|
||||
|
||||
**Utledbart uansett — ikke lekket av oss:**
|
||||
|
||||
- **At det er en AIServices-ressurs med prosjekter.** URL-formen `…/api/projects/<project>` ER den
|
||||
dokumenterte Foundry-prosjekt-endepunkt-formen. Enhver som ser en slik URL vet ressurstypen.
|
||||
- **Modell, versjon og deployment-navn** (`gpt-4.1-mini`, `2025-04-14`, GlobalStandard). Offentlig
|
||||
katalog-nomenklatur.
|
||||
- **Rolle-GUID-en `53ca6127-db72-4b80-b1b0-d745d6d5456d`.** Azures **offentlige** innebygde
|
||||
role definition id for `Foundry User`, identisk i hver tenant. Den står i Learn-dokumentasjonen.
|
||||
Den er **ikke** en koordinat, og skal ikke plassholdes.
|
||||
- **At vertsnavnet i det hele tatt er et globalt, gjettbart navnerom.** Custom subdomain ligger under
|
||||
ett felles DNS-navnerom, og navnet er unikt på tvers av alle kunder — så *eksistensen* av et gitt
|
||||
navn kan enhver bekrefte ved å slå det opp. Lekkasjen fjernet **gjettingen**, ikke muligheten.
|
||||
|
||||
**Kun kjent fordi navnene lekket:**
|
||||
|
||||
- **Ressursgruppenavnet.** Det finnes ikke i DNS og ikke i noen data-plan-URL. Det er rent
|
||||
ARM-plan-informasjon.
|
||||
- **Regionen** (`eastus`). Ikke utledbar fra vertsnavnet for `*.services.ai.azure.com`.
|
||||
- **Prosjektnavnet** (`<project>`) — det står riktignok i endepunkts-URL-en, men URL-en er selv en
|
||||
del av lekkasjen.
|
||||
- **Koblingen mellom dem.** Det operativt verdifulle er ikke ett navn, men at ressursgruppe,
|
||||
ressurs, prosjekt, region og rolle-scope kommer som ett ferdig sett.
|
||||
|
||||
**Kort:** dette er rekognoseringsinformasjon om ARM-planet. Det er ikke nøkler, og det er ikke
|
||||
tenant-id eller abonnements-id — ingen av de to sto i dokumentet (målt: `0` treff på
|
||||
`/subscriptions/<guid>` i hele treet og i den leverte pakka).
|
||||
|
||||
## 2. Hva skal til for å misbruke dem?
|
||||
|
||||
### Det Entra faktisk stopper
|
||||
|
||||
Et kall mot prosjekt-endepunktet krever **både** et gyldig Entra-token for scopet
|
||||
`https://ai.azure.com/.default` **og** en RBAC-tildeling på ressurs- eller prosjekt-scope. Microsofts
|
||||
egen feilkode-tabell skiller de to: **401** = manglende/utløpt token, **403** = manglende
|
||||
rolletildeling. Å kjenne adressen gir altså i seg selv **null** inferens-tilgang.
|
||||
|
||||
Token-basert auth krever dessuten et custom subdomain — regionale endepunkter støtter ikke Entra i
|
||||
det hele tatt. Vi bruker custom subdomain, altså er Entra-stien tilgjengelig.
|
||||
|
||||
### Det Entra ikke stopper
|
||||
|
||||
1. **Nøkkelbasert auth, hvis den er på.** Entra blir *eneste* autorisasjonsmetode først når
|
||||
`disableLocalAuth` er satt til `true` — det er en eksplisitt handling (Azure Policy på
|
||||
abonnement/ressursgruppe, `disableLocalAuth` i ARM/Bicep, eller `Set-AzCognitiveServicesAccount
|
||||
-DisableLocalAuth $true`). Er den ikke satt, finnes det nøkler som omgår Entra fullstendig.
|
||||
**Ikke verifisert for denne ressursen:** `az cognitiveservices account create`-kommandoen i
|
||||
måleprotokollen (`docs/2026-08-14-fase1b-forste-levende-kjoring.md` §0) ba ikke om det, og om
|
||||
abonnementet har policyen er ukjent herfra. **Dette er den ene sjekken som faktisk endrer
|
||||
risikobildet** — se §4.
|
||||
Merk også at avslåing ikke slår inn momentant: endringen skjer i kontrollplanet med én gang, men
|
||||
gatewayen kan godta tidligere gyldige nøkler til cachen oppdateres — typisk minutter, opptil
|
||||
flere timer. Og allerede utdelte nøkler må regenereres separat; å slå av lokal auth *tilbakekaller*
|
||||
dem ikke.
|
||||
2. **Målrettet phishing og consent-phishing.** Koordinatene gjør en henvendelse troverdig — avsender
|
||||
kan navngi ressursgruppe, ressurs og prosjekt riktig. Entra beskytter identiteten, ikke
|
||||
overtalelsen. Dette er den mest realistiske misbruksveien for denne typen lekkasje.
|
||||
3. **Kvote- og kostnadsmisbruk ved kompromittert identitet.** Kvote tildeles **per abonnement, per
|
||||
region, per modell og deployment-type**, i tokens-per-minutt, og deles av alle deployments av
|
||||
samme modell i samme region i abonnementet. En misbrukt identitet med `Foundry User` på dette
|
||||
prosjektet spiser altså av en pott som er felles, og kan strupe *andre* deployments av samme
|
||||
modell i samme abonnement — ikke bare denne. Forbruket faktureres.
|
||||
4. **Nettverksflaten.** `publicNetworkAccess` er en egenskap som må settes til `Disabled` for å stenge
|
||||
den offentlige inngangen. Er den ikke det, er endepunktet nåbart fra internett — det var sant før
|
||||
lekkasjen også. Forskjellen er at adressen nå er *kjent*, ikke at den ble *nåbar*.
|
||||
|
||||
**Ikke verifisert:** om et uautentisert kall skiller et eksisterende prosjektnavn fra et
|
||||
ikke-eksisterende (altså om `<project>` kan bekreftes uten token). Det ville krevd et faktisk kall
|
||||
mot en fremmed ressurs, og det er ikke gjort.
|
||||
|
||||
## 3. Hva koster omdøping?
|
||||
|
||||
### I repoet: null
|
||||
|
||||
Målt på `6d2837f` (= `open/main` = `origin/main`), nevner **327 sporede filer**:
|
||||
|
||||
| Sted | Treff på de tre literale navnene |
|
||||
|---|---|
|
||||
| Sporet tre (kode, tester, docs, `env.template`) | **0** |
|
||||
| Usporet deck `docs/presentasjon-portfolio-optimiser.html` | **0** |
|
||||
| `shared/` (subtree) | **0** |
|
||||
|
||||
De 18 gjenværende linjene med `services.ai.azure.com` er plassholderformen (`<resource>.`),
|
||||
wildcard-formen (`*.`) eller testdummies (`x.`, `platform.`, `wrong.`). `env.template` bærer
|
||||
variabelnavn, aldri verdier. **Omdøping krever altså ingen redigering i repoet** — `241b50d` gjorde
|
||||
allerede den jobben.
|
||||
|
||||
### I historikken og i den leverte pakka: kan ikke tilbakekalles
|
||||
|
||||
- **Git-historikken:** 3 linjer i 1 fil, i 3 commits (`5bd8e1c`, `bb4807a`, `241b50d`). Publisert på
|
||||
`open/main`. Ordren forbyr å skrive om historikken, og vei B ble alt avvist 18.08.
|
||||
- **Den leverte pakka:** 4 linjer, hos en tredjepart siden 14.08. En omdøping i Azure gjør ikke det
|
||||
usett.
|
||||
|
||||
### I Azure: en full riving og gjenoppbygging
|
||||
|
||||
**Et custom subdomain kan ikke endres.** Microsoft er eksplisitt: navnet kan ikke endres etter at det
|
||||
er opprettet og knyttet til ressursen, og for å gjenbruke et navn må den eksisterende ressursen
|
||||
slettes. «Omdøping» finnes derfor ikke som operasjon — det er:
|
||||
|
||||
1. opprett ny AIServices-ressurs med nytt subdomain (+ `--allow-project-management`)
|
||||
2. opprett nytt prosjekt
|
||||
3. redeploy `gpt-4-1-mini` (ny kvotetildeling i regionen)
|
||||
4. tildel `Foundry User` på nytt prosjekt-scope
|
||||
5. oppdater `PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT` og `PORTFOLIO_MODEL_MAP` lokalt
|
||||
6. slett — og purge — den gamle ressursen (purge krever `Contributor` på abonnements-scope)
|
||||
7. re-kjør stigen i måleprotokollen §1 for å bevise at auth/RBAC/endepunkt fortsatt komponerer
|
||||
|
||||
**Ikke verifisert:** om det gamle subdomain-navnet holdes reservert en periode etter sletting slik
|
||||
App Service og API Management gjør (anti-subdomain-takeover). Learn dokumenterer den mekanismen for
|
||||
de tjenestene, men ikke for Foundry-/Cognitive Services-subdomener. Ikke anta at navnet frigis — og
|
||||
ikke anta at det er låst.
|
||||
|
||||
## 4. Anbefaling — og hva hvert valg koster
|
||||
|
||||
**Anbefalingen er: ikke døp om. Fjern i stedet den ene veien som ikke går gjennom Entra, og gjør
|
||||
misbruk synlig.**
|
||||
|
||||
Begrunnelsen er at omdøping løser feil problem. Det lekkasjen ga en motpart er **målretting**, ikke
|
||||
**tilgang**. Og målrettingen kan ikke trekkes tilbake: navnene ligger i publisert git-historikk og i
|
||||
en zip hos en tredjepart. En omdøping ville altså kjøpt at *dagens* ressurs ikke er den som ble
|
||||
navngitt — ikke at navngivingen forsvinner. Det er en reell, men liten gevinst, og den betales med en
|
||||
full riving av det eneste levende Foundry-oppsettet prosjektet har.
|
||||
|
||||
**Valg A — behold navnene, herd oppsettet (anbefalt).**
|
||||
- Sjekk `DisableLocalAuth` på ressursen. Er den ikke `true`: regenerer begge nøklene *og* sett den.
|
||||
Dette er den eneste tiltaket som fjerner en autorisasjonsvei Entra ikke dekker.
|
||||
- Sjekk at rolletildelingen står på **prosjekt**-scope og ikke bredere, og vurder
|
||||
`Foundry Agent Consumer` framfor `Foundry User` dersom kjøringene bare gjør inferens.
|
||||
- Sett et kostnadsvarsel på abonnementet og hold TPM-tildelingen på deploymentet lav. Kvotemisbruk
|
||||
blir da både begrenset og synlig.
|
||||
- *Konsekvens:* måleprotokollen forblir reproduserbar, ingen ny kjøring må betales, og navnene
|
||||
fortsetter å stå i historikken — som de ville gjort uansett.
|
||||
|
||||
**Valg B — riv og bygg opp igjen med et intetsigende navn.**
|
||||
- *Kjøper:* at et navn en motpart eventuelt sitter og venter på, ikke lenger peker på noe levende.
|
||||
- *Koster:* de sju stegene i §3, ny betalt verifiseringskjøring, og at
|
||||
`docs/2026-08-14-fase1b-forste-levende-kjoring.md` beskriver et oppsett som ikke finnes lenger.
|
||||
- *Kjøper ikke:* at navnene forsvinner fra historikken eller fra den leverte pakka.
|
||||
- Velg denne hvis vurderingen er at ressursgruppe- og prosjektnavnet i seg selv er sensitivt i
|
||||
organisasjonssammenheng — det er en vurdering operatøren kan gjøre og ikke jeg.
|
||||
|
||||
**Valg C — gjør begge.** Herdingen i A er verdt å gjøre *uansett* hvilket av A og B som velges; B
|
||||
uten A etterlater den samme nøkkel-veien åpen på en ny ressurs.
|
||||
|
||||
## 5. Hva denne vurderingen ikke dekker
|
||||
|
||||
- Ingenting i Azure er inspisert. Alle utsagn om *denne* ressursens faktiske konfigurasjon
|
||||
(`disableLocalAuth`, `publicNetworkAccess`, rolle-scope) er markert som uverifiserte over.
|
||||
- **Innholds-gaten fanger vertsnavnet, ikke ressursgruppe- og prosjektnavnet.** Et vertsnavn har en
|
||||
struktur (`<label>.services.ai.azure.com`); en ressursgruppe heter hva som helst. De to andre
|
||||
navnene ble bare oppdaget fordi de sto i samme tabell som verten. En gate kan ikke lukke det
|
||||
gapet uten en navneliste, og en navneliste er den andre kopien av eksponeringsregelen.
|
||||
- Ordre-teksten sa at koordinatene fortsatt sto i repoet. Det er **ikke** tilfelle per `6d2837f` —
|
||||
målt, se §3. Premisset var riktig da ordren ble skrevet (10:37Z) og sluttet å være det 13:03.
|
||||
|
||||
## 6. Kilder
|
||||
|
||||
Alle verifisert 18.08.2026 mot Microsoft Learn:
|
||||
|
||||
- Custom subdomain kan ikke endres; må slette ressursen for å gjenbruke navnet —
|
||||
`learn.microsoft.com/azure/ai-services/cognitive-services-custom-subdomains`
|
||||
- Entra-auth krever custom subdomain; 401 vs 403; scope `https://ai.azure.com/.default` —
|
||||
`learn.microsoft.com/azure/foundry/concepts/authentication-authorization-foundry`
|
||||
- `disableLocalAuth` er en eksplisitt handling; propagering minutter til timer; nøkler må regenereres
|
||||
separat — `learn.microsoft.com/azure/ai-services/disable-local-auth`
|
||||
- Rolle-id `53ca6127-db72-4b80-b1b0-d745d6d5456d` = `Foundry User`, offentlig og lik i hver tenant;
|
||||
`Foundry Agent Consumer` som minste-privilegium for ren inferens —
|
||||
`learn.microsoft.com/azure/foundry/concepts/rbac-foundry`
|
||||
- Kvote per abonnement/region/modell/deployment-type i TPM; deles av deployments i samme region —
|
||||
`learn.microsoft.com/azure/foundry/openai/how-to/quota`
|
||||
- `publicNetworkAccess` / `disableLocalAuth` som ARM-egenskaper —
|
||||
`learn.microsoft.com/azure/templates/microsoft.cognitiveservices/accounts`
|
||||
- Anti-subdomain-takeover-reservasjon (dokumentert for App Service / API Management, **ikke** for
|
||||
Cognitive Services) — `learn.microsoft.com/azure/security/fundamentals/subdomain-takeover`
|
||||
327
docs/2026-08-25-fable-misjonsreview.md
Normal file
327
docs/2026-08-25-fable-misjonsreview.md
Normal file
|
|
@ -0,0 +1,327 @@
|
|||
# Misjons-review — nærmer portfolio-optimiser seg faktisk målet? (2026-08-25)
|
||||
|
||||
> **Reviewer:** Fable 5 (xhigh, uten advisor), ordre `20260825T104711Z-7149413623-from-.claude`.
|
||||
> **Mandat:** uavhengig, adversarial review av om det LEVERTE systemet gjør det README og operatøren
|
||||
> har sagt det skal gjøre — ikke en kodekorrekthets-review. Ingen kodeendringer.
|
||||
> **Metode:** hvert tall under kommer fra en kommando kjørt i denne økten (verifiseringslogg i § 6);
|
||||
> hvert U-punkt er grep-et mot `src/` og kallstedet lest; ingen tall er sitert fra `STATE.md`.
|
||||
> Målestokkene er sitert ordrett fra kildene ordren navngir. Eksterne MAF-påstander (§ 3) er
|
||||
> verifisert av en egen Opus-agent mot Microsoft Learn / PyPI / GitHub der det står «[ekstern]»;
|
||||
> alt annet er lokalt målt.
|
||||
|
||||
## 1. Sammendrag — er «nærmer seg mål» rettferdig?
|
||||
|
||||
**DELVIS.** Tre setninger, én per akse:
|
||||
|
||||
- **Mekanisme (README-løftet, steg 1–8): JA.** Alle åtte steg finnes i kode, er load-bearing-testet,
|
||||
og suiten er grønn målt i dag: **1021 passed / 5 skipped (166 s)**, `ruff` ren, `mypy` ren
|
||||
(34 filer), golden-transkriptet `ea8c534773acdbe41ae68f2c55724d69aaf8be4f` (målt `shasum`).
|
||||
- **Misjonsbevis (operatørens målbilde, pkt. 4): NEI.** Null validerte forslag mot en levende modell
|
||||
(den ene levende fullkjøringen 14.08 KONKLUDERTE med `rejected` — korrekt, men modellen fant opp
|
||||
en kostkode), null ekte ekspertdommer (hver dom i treet er merket `SYNTHETIC` / «AI-forfattet»),
|
||||
null kunnskapsbase materialisert fra en levende kilde, og null måling av at utforskningen finner
|
||||
noe debatten ikke finner (planens rad 4 er ugjort). «Stille spørsmål, be om svar, bruke svarene»
|
||||
er ikke nåbart fra noen operatørflate — kun fra bibliotek-APIet.
|
||||
- **MAF-ekspertise (§15.1, 19 punkter): LAV mot en ærlig telling.** **5 fullt brukt, 5 delvis,
|
||||
9 ikke** — og av de fem fullt brukte er to opt-in utenfor den normative stien (Magentic, MCP).
|
||||
Alt som er bærende — fan-out, blokkerende validator, HITL, læringsinjeksjon — er håndrullet
|
||||
rundt MAF, ikke bygget i MAF. Det er i stor grad bevisst og begrunnet (§ 3), men det gjør
|
||||
«eksperter på bruken av MAF» til en påstand om tre orkestrerings-konstrukter og én chat-klient
|
||||
— på en versjon (1.9.0, 18.06) som ligger **seks minor-releaser bak** dagens 1.15.0 (21.08), og
|
||||
som allerede shipper en primitiv (`MiddlewareTermination`) registeret ba om under U8 og repoet
|
||||
ikke bruker.
|
||||
|
||||
Kort: repoet er nær målet *«en komplett, ærlig, testet mekanisme»*. Det er ikke nær målet
|
||||
*«finner innsparinger mennesker ikke ser»* — for det målet finnes det i dag **ingen måling**, og
|
||||
den offentlige README-statusen er stale i begge retninger (§ 5, F1).
|
||||
|
||||
## 2. U1–U19 mot faktisk kildekode
|
||||
|
||||
Nevner: `grep -rn --include="*.py" -F <konstrukt> src` over 34 moduler / 11 650 linjer (`wc -l`),
|
||||
deretter kallstedet lest. `tests/` og `spikes/` teller IKKE som bruk. «Bevis» er der konstruktet
|
||||
faktisk kalles, ikke der det nevnes i prosa.
|
||||
|
||||
| ID | Kapabilitet (§15.1) | Brukt | Bevis (fil:linje) | Kommentar |
|
||||
|----|----|----|----|----|
|
||||
| U1 | `ConcurrentBuilder` fan-out | **nei** | `run.py:1274` `await asyncio.gather(` | Håndrullet bølgemodell. Begrunnet (`docs/plan/2026-08-23-…:368`: bølgen bærer budsjett-reservasjon S3.4). Ikke B-listet — et bevisst avvik fra USE, dokumentert. |
|
||||
| U2 | `WorkflowBuilder` graf | **nei** | 0 treff i `src` | P10 «Eksplisitt NULL» (`docs/plan/2026-08-09-egnethetsreview-plan.md:534`). Begrunnet. |
|
||||
| U3 | `GroupChatBuilder` maker-checker | **ja** | `workflow.py:24` import · `:104` `GroupChatBuilder(` · `:108` `termination_condition=` · `:113` `.with_max_rounds` | Eneste MAF-orkestrering på den normative stien. Debatten er en fast 2-agent round-robin på 3 runder; hver kjøring ender i «forcing completion» (`docs/2026-08-14-…:59`, målt live). |
|
||||
| U4 | Magentic (åpne delsteg) | **ja** (opt-in) | `explore.py:32-37` import · `:596-603` `MagenticBuilder(…).build()` | Landet økt 56–58 OVER den normative sløyfa. Ikke eksperimentell i installert `orchestrations 1.0.1` (0 `@experimental`-treff i `_magentic.py`) — §15 G8/A2 sin premiss er utdatert. |
|
||||
| U5 | Agent Skills / `SkillsProvider` | **nei** | 0 treff; `persona.py:27/:47-51` leser KUN `references/example-verdict.json` med egen loader | `SKILL.md`-prosaen når aldri en agent. Metode-skillen finnes ikke (`find shared -name SKILL.md` → kun `expert-reviewer`). `SkillsProvider` er `@experimental` i installert core (`_skills.py:76/:120`). Begrunnet (`…2026-08-23:370`), men CLAUDE.md-konvensjonen «Metode kodifiseres som Agent Skill» er uoppfylt. |
|
||||
| U6 | MCP-tools | **ja** (opt-in) | `mcp_tools.py:154` `MCPStdioTool(` · `:170` `MCPStreamableHTTPTool(` | Null nettverkskall uten `--mcp-config`. `as_mcp_server()`: 0 treff (ikke savnet). |
|
||||
| U7 | Solver/validator/MC som Function Tools | **delvis** | `datasource.py:65/:69` `@tool` retrieval · `explore.py:461/:487` `quick_validate` | Retrieval ER et Function Tool. Validatoren er det IKKE på den normative stien — den kalles etter generering i `run.py`; kun den RÅDGIVENDE `quick_validate` i utforskningen er et tool. Registerets formål er ikke realisert slik det ble skrevet; B1-valget forklarer hvorfor (§ 3). |
|
||||
| U8 | Middleware (intercept av tool-calls → blokkerende validator) | **delvis** | `budget.py:228` `class BudgetMiddleware(ChatMiddleware)` · `mcp_tools.py:197` `class ToolCallRecorder(FunctionMiddleware)` | Middleware brukes — til budsjett og til OBSERVASJON av tool-kall. Ingen middleware blokkerer noe; validatoren gater utenfor grafen. Konstruktet: ja. Formålet i U8: nei. |
|
||||
| U9 | Læringssløyfe-injeksjon via `ContextProvider` | **delvis** | `verdicts.py:319` `class ExpeLContextProvider(ContextProvider)` · `:353` `extend_instructions` — MEN `run.py:736-739` kaller `.format_fewshot()` direkte og string-konkatenerer inn i prompten; `:848-857` kjører `before_run` inn i en `SessionContext` som (egen kommentar) «is NOT what reaches the prompt» | MAF-hooken er **dekorativ**: konstruktet subklasses, men injeksjonen som bærer læringen går utenom MAF. Fungerer — men er ikke «bruk av `ContextProvider`» i MAF-forstand. |
|
||||
| U10 | Vektorlagre (MAF-integrasjoner) | **nei** | `semretrieval.py:1-70`: numpy brute-force, «MAF-free (D7-portable)» | Begrunnet (D-C / D7). Shippet embedder er «a semantics-free sha256 projection» (`run.py:1681`-hjelpetekst) — utvidelsespunkt, ikke semantisk gjenfinning. |
|
||||
| U11 | Citation-aware RAG (`TextSearchProvider`/agentic retrieval) | **delvis** | `provenance.py:18` `from agent_framework import Annotation, TextSpanRegion` (display only); egen `datasource.retrieve_chunks` | Sitatbæring finnes, med MAF-TYPER. MAF-providerne: 0 treff. Begrunnet («navigate, never stuff», `…2026-08-23:371`). |
|
||||
| U12 | Checkpointing / kræsj-overlevelse | **nei** | 0 kode-treff; `workflow.py:14` prosa «no checkpoint» | P10 sa NULL; gjenåpnet 23.08 som planens rad 3 = STATE-ens NESTE. Spike `spikes/e_magentic_resume.py` MÅLT grønn (fersk interpreter). Ikke bygget. |
|
||||
| U13 | HITL-gates (`request_info` / plan review / tool approval) | **delvis, uøvd** | `workflow.py:116` `with_request_info(agents=[agents[-1]])` bak `enable_layer1_hitl=False` (`run.py:536`, intet CLI-flagg); `explore.py:363` `PlanReviewer`, `:849` `plan_reviewer=`; `approval_mode`: 0 treff | Layer-1 er «green-but-dead»: ENESTE test er `tests/test_workflow.py:58` som bygger, aldri kjører gaten. Plan review er bibliotek-only: CLI nekter (`run.py:1975-1988`), hosting nekter (`hosting.py:167-172`). |
|
||||
| U14 | OpenTelemetry | **ja** (opt-in) | `tracing.py:160` `configure_otel_providers` · `:265` `get_tracer`; span-events i `explore.py` | `PORTFOLIO_OTEL` på truthiness; OTLP-pakker bevisst ikke deklarert. |
|
||||
| U15 | Evaluering (`LocalEvaluator`/`evaluate_workflow`) | **nei** | 0 treff | «AVVIST som flate» (`STATE.md:99`); begrunnelse `…2026-08-23:390` (`@experimental`, 20 dekoratorer målt i installert `_evaluation.py`; «feil form»). Egen sammenligning (rad 4) IKKE gjort → ingen måling finnes. |
|
||||
| U16 | `CompactionProvider` | **nei** | 0 treff | **Uadressert**: kun telt (`…2026-08-23:58` «Utenfor de 15»), ingen beslutning noe sted. Relevant for lange Magentic-løp der token-taket er eneste vern. |
|
||||
| U17 | Agent-som-verktøy `as_tool()` | **nei** | 0 treff | **Uadressert.** Lav relevans: A3 knyttet det til kryss-prosjekt, som er non-goal 2. |
|
||||
| U18 | `FoundryChatClient` + modell-map | **ja** | `backends.py:29` import · `:150` `FoundryChatClient(project_endpoint=…, model=…, credential=…)`; B12 `model_map` 24 treff | Bevist mot levende Foundry 14.08 (`docs/2026-08-14-…:42`). |
|
||||
| U19 | Agent Harness (`HarnessAgent`, `FileMemoryStore`, …) | **nei** | 0 treff | **Uadressert.** Installert pakke sier selv `[HARNESS] MemoryStore is experimental` (pytest-warning målt) — legitimt å hoppe over, men ingen har sagt det. |
|
||||
|
||||
**Telling:** ja 5 (U3, U4, U6, U14, U18) · delvis 5 (U7, U8, U9, U11, U13) · nei 9 (U1, U2, U5,
|
||||
U10, U12, U15, U16, U17, U19). Av de ni: 6 med skriftlig begrunnelse i repoet, 3 uten (U16, U17,
|
||||
U19). **Nevneren er 19, ikke 15:** «6 av 15» stammer fra `docs/plan/2026-08-23-magentic-utforskningssloeyfe.md:58`,
|
||||
som teller U1–U15 og setter U16–U19 «utenfor» — filen `docs/research/2026-06-23-prior-art-platform.md`
|
||||
har én commit (`ec9ac74`) og 19 rader (`awk`-lest i dag).
|
||||
|
||||
## 3. B-lista re-eksaminert — er «SKAL bygges selv» fortsatt riktig på 1.9.0?
|
||||
|
||||
Installert (`uv pip list`): `agent-framework-core 1.9.0`, `-orchestrations 1.0.1`, `-foundry 1.8.2`,
|
||||
`-openai 1.8.2`; `uv.lock` pinner de samme. Registeret er datert 23.06; MAF-status i dag er sjekket
|
||||
mot installert pakke (lokalt) og mot offisielle kilder ([ekstern] — se § 3.1).
|
||||
|
||||
| ID | BUILD-valget | Status i kode | Fortsatt riktig? |
|
||||
|----|----|----|----|
|
||||
| B1 | Blokkerende hybrid-validator | `validator.py:31` `import pulp` · `:106` `PULP_CBC_CMD` · `:127` seedet MC; kalles i `run.py` ETTER generering, utenfor grafen | **HALVVEIS.** Tall-halvdelen (solver + MC + baseline-avstemming + `Rejection` som VERDI med provenance) er fortsatt ikke en MAF-primitiv — riktig egenbygg. Men emisjons-halvdelen («svaret forlater ikke agenten uten sjekk») ER nå en MAF-primitiv: `MiddlewareTermination` finnes i installert 1.9.0 uten experimental-markør (§ 3.1). Registerets U8 beskrev nøyaktig den formen; repoet har den ikke (0 treff). `response_format` (brukt, `generate.py`) gir FORM, ikke VERDI-kontroll. |
|
||||
| B2 | VerdictStore + ExpeL | `verdicts.py` + `run.py:736-739` (fold) + `run.py:1331` `_merge_wave` (kryss-prosjekt) | **JA, men med et forbehold:** kravet i misjonen er «learns from their verdicts across runs» — MENNESKE-dommer om TIDLIGERE OUTPUT, hentet strukturelt. Ingen MAF-memory-provider gjør det (de lagrer samtalehistorikk/fakta, ikke dommer nøklet på kandidat-features). Forbeholdet er at koden i dag ikke bruker MAF-hooken for injeksjonen (U9) — så B2 er «bygg selv» i sterkere grad enn registeret sa. |
|
||||
| B3 | To-lags HITL | Layer-1 `with_request_info` (uøvd, § 2 U13); Layer-2 = fil-innboks (`verdicts.load_verdicts_from_dir`, Steg 7) + `verdict_input`-argumentet | **DELVIS RIKTIG.** Den asynkrone fil-innboksen er riktig egenbygg (MAF gir ingen fler-dagers HITL uten checkpoint + ekstern varsling). Men `verdict_input` (dom gitt FØR forslaget finnes) er ikke HITL — se F2. |
|
||||
| B4 | Termineringskontrakt + budsjett-breakers | `contracts.py` `TerminationContract` · `budget.py` `Budget`/`TokenMeter`/`PortfolioMeter` · `explore.py:50-87` `ExplorationContract` (alle felt påkrevd) | **JA.** Magentic defaulter fortsatt `max_round_count`/`max_reset_count` til `None` (målt, `explore.py:54-56`). |
|
||||
| B5 | Generisk data-source-konfig | `mcp_tools.McpServerConfig` (pydantic), `datasource.py`; `jsonschema`: 0 treff | **JA** (pydantic i stedet for JSON Schema — form, ikke prinsipp). |
|
||||
| B6 | Provenance | `provenance.py:18` MAF-typer for display; egen `ProvenanceStamp` | **JA.** Ingen MAF-native provenance. |
|
||||
| B7 | State-isolasjon i fan-out | `workflow.py` `fresh_workflow`; `explore.py` `manager_agent_factory` | **JA** (G2 målt i spikes). |
|
||||
| B8 | Sandboxing av skill-scripts | 0 treff (`sandbox`, `Subprocess`) | **IKKE AKTUELT** — ingen brukerleverte scripts kjøres, fordi U5 ikke er bygget. Blir aktuelt den dagen metode-skillen finnes. |
|
||||
| B9 | Onboarding-intervju | `preflight.py` (operatørverktøy) | **DELVIS.** Preflight sjekker konfig; det finnes intet intervju som tvinger fram termineringskontrakt + feedback-skjema FØR første kjøring — `--decision` defaulter til `approved` i stedet (F2). |
|
||||
| B10 | Feedback-skjema + rejection-taksonomi + konfliktregel | `contracts.FeedbackContract` (binær `approved`/`rejected`); taksonomi: 0 treff (`wrong_assumption`, `overestimated`); `verdicts.py:313-314` «deliberately deferred until real experts produce conflicting verdicts» | **UTSATT, uttalt.** Riktig å vente på ekte dommer — men det betyr at læringssignalet i dag er én bit + fri prosa. |
|
||||
| B11 | Ekstern varsling | `notify.py:117` webhook-notifier (fail-closed opt-in) | **JA — levert**, ikke lenger stub. |
|
||||
| B12 | Modell-map | `backends.py` `resolve_model`, `data/model_map.json` | **JA.** |
|
||||
|
||||
### 3.1 Ekstern verifisering (MAF-status per 25.08.2026) [ekstern]
|
||||
|
||||
Utført av en Opus-agent (xhigh) mot `learn.microsoft.com` (WebFetch — `microsoft-learn`-MCP-en var
|
||||
utilgjengelig i agentens økt), PyPI (to uavhengige hentinger, samsvar) og GitHub-releases (delvis
|
||||
summert av fetch-laget → merket LAVERE KONFIDENS). Hver rad under er deretter kryss-sjekket
|
||||
LOKALT mot installert pakke der det var mulig (`grep` i `.venv`).
|
||||
|
||||
| Tema | Ekstern kilde (verbatim-utdrag) | Lokalt målt i installert 1.9.0 | Konsekvens |
|
||||
|---|---|---|---|
|
||||
| **Versjoner** | `agent-framework-core` siste = **1.15.0 (2026-08-21)**; i vinduet 23.06→25.08: 1.10.0, 1.11.0, 1.12.0, 1.12.1, 1.13.0, 1.14.0, 1.15.0. Pinnet 1.9.0 er fra 18.06 — FØR registeret ble skrevet. `orchestrations` siste = 1.1.1, krever `core>=1.15.0`. (pypi.org/pypi/agent-framework-core/json) | `uv pip list`: core 1.9.0, orch 1.0.1 | Repoet er **seks minor-versjoner bak**, og `STATE.md:99` kaller pinnen «ENDELIG, operatørbekreftet 23.08» — ni dager etter at 1.14.0 kom. Se F15. Upgrade-guiden (`python-2026-significant-changes`) stopper angivelig ved 1.8.0 — **IKKE VERIFISERT** (summert fetch). |
|
||||
| **B2** — læring fra menneskedommer på tvers av kjøringer | «`ContextProvider` and `HistoryProvider` are the canonical Python base classes.» `FileMemoryProvider`: «the model should decide what to store and recall». Foundry Memory: «**Memory (preview)** … subject to terms applicable to "Previews"», typene er «User profile / Chat summary / Procedural memory». **NOT FOUND:** verdict-store, experience replay, feedback-API. | — | **B2 står som BUILD.** Ingen MAF-mekanisme lagrer eller henter menneskedommer om tidligere output. G5 (Foundry memory = preview) står. |
|
||||
| **B1** — blokkér output uten deterministisk sjekk | Side «Termination & Guardrails»: «middleware stops execution by setting `context.result` when needed and raising `MiddlewareTermination`, or by short-circuiting the chain without calling `call_next()`.» «**Post-termination middleware** — Middleware that terminates after agent execution — useful for validating responses». `response_format` «parses but does not reject». (learn.microsoft.com/…/agents/middleware/termination) | `class MiddlewareTermination` finnes i `agent_framework/_middleware.py`, **uten** `@experimental` (de to markørene i fila gjelder `PROGRESSIVE_TOOLS`) | **B1 er DELVIS flippet — i den installerte versjonen.** Halvdelen «nekt at svaret forlater agenten» er nå en MAF-primitiv; halvdelen «avgjør TALLENE mot ekstern baseline og stemple provenance» er fortsatt vår. Registerets U8 beskrev nøyaktig denne formen. Ikke brukt (0 treff `src`). Se F16. |
|
||||
| **Magentic** | Konseptsiden (oppdatert 2026-08-10) har **ingen** experimental-admonisjon for Python; eneste markør er i C#-eksempelet (`MAAIW001`). Python-API-referansen for `MagenticBuilder` er stale (2025-12-12, fluent form). Verbatim: «in Python, plan review is **off** by default (`enable_plan_review=False`)». | 0 `experimental`-treff i `agent_framework_orchestrations/_magentic.py` | §15 G8/A2 («Magentic er eksperimentell») er **utdatert** for Python. Økt 56-avgjørelsen om å legge Magentic OVER, ikke I, den normative stien hviler da på arkitektur (mandat-former), ikke lenger på modenhet. Riktig konklusjon, foreldet premiss. |
|
||||
| **U5 SkillsProvider** | «`SkillsProvider` (Python) is a context provider that exposes skills to an agent. It advertises the available skills in the system prompt and registers the tools the agent uses to load skills, read resources, and run scripts.» Fire stadier; «**SKILL.md is NOT auto-loaded**» — kroppen hentes kun via `load_skill`. Ingen preview-banner på siden; kun `MCPSkillsSource` er experimental. | `@experimental(feature_id=ExperimentalFeature.SKILLS)` `_skills.py:76/:120` | Docs (1.15) og installert kode (1.9) er uenige om status — enda et argument for at pinnen koster. Avvisningen i `…2026-08-23:370` var riktig FOR 1.9.0. |
|
||||
| **U15 Evals** | «In Python, the evaluation framework is part of the core `agent_framework` package … `evaluate_agent()` and `evaluate_workflow()`». `LocalEvaluator` «runs checks locally without API calls». Sjekker: `keyword_check`, `tool_called_check`, `tool_calls_present`, `tool_call_args_match`; custom via `@evaluator`. Foundry-siden: «agent evaluation remains in public preview». **«Foundry Adaptive Evals» — NOT FOUND** som produktnavn. | 20 `@experimental` i `_evaluation.py` | Registerets U15-tekst navngir et produkt som ikke finnes på Learn. `tool_call_args_match` er relevant for «kaller en levende modell faktisk `quick_validate`?» — nettopp grensen CLAUDE.md sier er ubevist. Avvisningen var riktig for 1.9.0; for 1.15 er den ikke re-vurdert. |
|
||||
| **U12 Checkpointing** | Tre providere (`InMemory`/`File`/`Cosmos`), samme protokoll, ingen preview-banner. Verbatim: «both providers use a **restricted unpickler** by default … Any unsupported type causes deserialization to fail with a `WorkflowCheckpointException`.» «pass them via the `allowed_checkpoint_types` parameter using `"module:qualname"` format». «Starting in Python version 1.13.0, workflows also create an entry checkpoint … when responses to request events are delivered … minor breaking changes». | `FileCheckpointStorage` + `allowed_checkpoint_types` finnes i `_workflows/_checkpoint.py` | STATE-ens «MÅLT FELLE» (`STATE.md:27-29`) er bekreftet av dokumentasjonen. Rad 3 (NESTE) bygger på en 1.9.0-form som 1.13.0 endret — en bump etter rad 3 vil kreve re-måling av resume-stien. |
|
||||
| **Sequential / Handoff** | Begge i `agent_framework.orchestrations`, ingen status-admonisjon (kun handoff «autonomous mode» er experimental). `SequentialBuilder(...).with_request_info(agents=[…])` «pause[s] after specific agents respond, allowing external input (such as human review)». | `class SequentialBuilder` / `class HandoffBuilder` finnes | P10-nullen står (debatten er 2 agenter; Sequential gir ingen ny kapabilitet), men `with_request_info` finnes ALLEREDE på GroupChat (`workflow.py:116`) og er uøvd (F7). |
|
||||
| **U16 Kompaksjon** | «**Important** — The compaction framework is currently experimental in Python.» Python har INGEN `CompactionProvider`-klasse (C#-navn); Python-formen er `Agent(..., compaction_strategy=..., tokenizer=...)` / `apply_compaction(...)` med `SlidingWindowStrategy`, `SummarizationStrategy`, `TokenBudgetComposedStrategy` m.fl. | `SlidingWindowStrategy`/`SummarizationStrategy`/`apply_compaction` finnes i `_compaction.py`, under `@experimental` | Registerets U16 navngir C#-konstruktet. Grep-en i § 2 fant 0 fordi Python-navnet er et annet — **re-grep på `compaction_strategy\|apply_compaction`: fortsatt 0.** Uadressert står. |
|
||||
| **U19 Harness** | «`create_harness_agent` is released. Background agents, file access, and looping remain experimental». `HarnessAgent` er **.NET-only**; `ToolApprovalAgent` — **NOT FOUND** som Python-navn (Python: `ToolApprovalMiddleware`). | `create_harness_agent`, `ToolApprovalMiddleware`, `AgentLoopMiddleware` finnes; 28 `@experimental`-markører på COMPACTION/HARNESS/LOOP | Registerets U19 lister to navn som ikke finnes i Python. F5 i §15.6 korrigerte versjonen, ikke navnene. |
|
||||
| **Nytt siden 23.06, rangert av agenten** | (1) `AgentLoopMiddleware` «re-invokes an agent until a completion condition is satisfied … `(continue, feedback)` to pass feedback to the next iteration» — **experimental**; ≈ Steg 5. (2) Evals (over). (3) `MiddlewareTermination` (over). (4) Structured outputs som førsteklasses sti (`response.value`). (5) Entry-checkpoints + Cosmos (1.13/1.15). (6) `SequentialBuilder.with_request_info`. (7) `SkillsProvider`. LAVERE KONFIDENS (GitHub): 1.14.0 «provider-based Foundry state stores for agent sessions, checkpoints, and function approvals», 1.15.0 «process-wide workflow checkpoint type registry». | `AgentLoopMiddleware` finnes alt i 1.9.0 (`_harness/_loop.py`, experimental) | Steg 5s håndrullede `max_attempts`-løkke har nå en MAF-form med samme bindingsdisiplin («Always bound autonomous loops»). Experimental → riktig å ikke bygge på den; men den finnes, og ingen har målt den. |
|
||||
|
||||
**Konklusjon § 3:** B2 står. B1 er halvveis flippet i den versjonen repoet allerede kjører. B3/B4/B6/B7/B12
|
||||
står. B8/B9/B10 er uttalt utsatt. Fire av registerets U-rader (U15, U16, U19 og G8/A2 om Magentic)
|
||||
bærer premisser som ikke stemmer for Python i dag — og repoet har ingen rutine som re-verifiserer
|
||||
registeret mot MAF-releasene (målt: siste re-lesing var 23.08-planen, mot 1.9.0).
|
||||
|
||||
## 4. Misjonstekst-samsvar
|
||||
|
||||
### 4.1 README-løftet (første skjerm, `README.md:9-13`), ordrett
|
||||
|
||||
> A generic, open framework — built on **Microsoft Agent Framework (MAF)** — that finds cost
|
||||
> savings *inside* each project of a portfolio of independent projects. A swarm of agents
|
||||
> generates candidate measures; a **mandatory deterministic validator** (solver + Monte Carlo)
|
||||
> decides the numbers; domain experts judge the outcomes (human-in-the-loop); and the system
|
||||
> **learns from their verdicts** across runs.
|
||||
|
||||
| Løfte | Kode | Samsvar |
|
||||
|---|---|---|
|
||||
| «built on MAF» | 12 moduler importerer `agent_framework`; 3 orkestrerings-konstrukter (GroupChat, Magentic, middleware) + 2 chat-klienter | **Delvis.** Bygget *rundt* MAF (`hosting.py:2` «a wrapper, never `Workflow.as_agent()`»). Riktig valg, men ordet «built on» lover mer MAF enn det er. |
|
||||
| «A swarm of agents» | Debatt: 2 agenter (`workflow.py:26`); utforskning: 3 (`explore.py`) | **Salgsspråk.** To-tre agenter er ikke en sverm. |
|
||||
| «mandatory deterministic validator (solver + Monte Carlo)» | `validator.py:106` CBC · `:127` seedet MC · `run.py` kaller den ubetinget | **JA.** Verifisert, blokkerende, forankret (S4.0) når baselinen finnes. |
|
||||
| «domain experts judge the outcomes» | Ekte kanal: Steg 7 innboks (`docs/ekspert-svar.md`). Parallell kanal: `verdict_input` PÅKREVD FØR kjøring (`run.py:525`; `hosting.py:80`; CLI-default `approved`, `run.py:1691-1692`) | **BRUTT på den parallelle kanalen** — dommen gis før utfallet finnes (F2). |
|
||||
| «learns from their verdicts across runs» | `run.py:736-739` fold; `test_portfolio_learning_loadbearing.py` | **JA mekanisk**, men hver dom som finnes i treet er syntetisk (`reference_projects.json:2`; `shared/examples/bygg-energi-mikro/verdict-led-fro.md` «frø — AI-forfattet»). Ingen ekte dom har noen gang gått gjennom sløyfa. |
|
||||
|
||||
### 4.2 Non-goals (`README.md:248-262`), fem punkter
|
||||
|
||||
1. **«Not a compliance product.»** — Samsvar. Ingen compliance-funksjoner; disclaimer står.
|
||||
2. **«Not a portfolio-level reallocator. … Moving budget between projects, ranking projects
|
||||
against one another and portfolio governance sit above the method and are out of scope.»** —
|
||||
Samsvar med ett forbehold: `--goals`/`--ledger` (`run.py:1669-1680`) stopper et porteføljepass
|
||||
når et PORTEFØLJE-mål er nådd. Det er ikke reallokering eller rangering (bølgerekkefølge =
|
||||
konfigrekkefølge), men det er portefølje-nivå-logikk i metoden. Uttalt grense, ikke brudd.
|
||||
3. **«Not autonomous decision-making. The deterministic validator can only block; approving a
|
||||
measure is a domain expert's call (human-in-the-loop), and the framework implements nothing on
|
||||
the agents' say-so.»** — **BRUTT i én søm:** `portfolio-optimiser <PROJECT>` uten flagg fanger
|
||||
en dom med `decision="approved"`, `rationale="reviewed by expert"` (`run.py:1691-1692` →
|
||||
`:2231`/`:2289` → `:861` `capture_verdict` → `store.add`). Ingen ekspert er involvert. I et
|
||||
porteføljepass når den dommen neste prosjekts hypotese-prompt (`run.py:1331` `_merge_wave`;
|
||||
bevist av `tests/test_portfolio_learning_loadbearing.py:108`). Validatoren blokkerer fortsatt
|
||||
— men «godkjent» er defaulten, ikke en ekspertbeslutning.
|
||||
4. **«Not a turnkey vertical solution.»** — Samsvar.
|
||||
5. **«Not a model benchmark. The end-to-end proof runs offline against a scripted stand-in
|
||||
client.»** — Samsvar. (Planens rad 4 — utforskning vs. debatt — ville vært en METODE-måling,
|
||||
ikke en modell-benchmark, og er ikke gjort.)
|
||||
|
||||
### 4.3 README-ens status-advarsel (`README.md:264-273`), ordrett
|
||||
|
||||
> **Status:** the full 8-step agentic loop is wired and proven with load-bearing tests, and the
|
||||
> end-to-end proof is an **offline simulation** with a scripted stand-in client — no live-model
|
||||
> run yet. The **ingest layer** (real data sources) is implemented — … — but exercised only against
|
||||
> committed fixtures: no bundle has yet been materialized from a live source. … A sibling
|
||||
> implementation of the same method on the **Claude Agents SDK** is built in parallel from the
|
||||
> same shared spec.
|
||||
|
||||
| Påstand | Målt | Samsvar |
|
||||
|---|---|---|
|
||||
| «no live-model run yet» | `docs/2026-08-14-fase1b-forste-levende-kjoring.md:42` «prosjektets første levende modellkall»; `:217` `outcome_type: rejected`, 15 306 tokens, `checker_verdict: approve`, `validator_decision: rejected` | **STALE.** Sist endret `d8ee8d3` 2026-07-04 (`git log -L`). En levende fullkjøring HAR konkludert. Det som fortsatt er sant: «ikke bevist at systemet produserer et *validert* forslag mot en levende modell» (`:245`). |
|
||||
| «no bundle has yet been materialized from a live source» | Ingen dokumentasjon på det motsatte funnet | **Står** (ikke falsifisert i dag; nevner: `docs/` grep). |
|
||||
| «sibling … is built in parallel» | `STATE.md:103` «po-claude PARKERT» | **STALE / ikke verifiserbar herfra.** |
|
||||
|
||||
### 4.4 Operatørens målbilde (23.08, gjengitt av .claude), ordrett
|
||||
|
||||
> «Vi skal bli EKSPERTER på bruken av Microsoft Agent Framework.» Målbildet: gitt EN prompt +
|
||||
> OKF-bundles skal løsningen stille nødvendige spørsmål, teste hypoteser, be om svar, bruke svarene
|
||||
> og utforske løsningsrommet — og finne innsparinger mennesker ikke ser, eller regne/utforske
|
||||
> hypoteser fagpersoner leverer, langt raskere.
|
||||
|
||||
| Delmål | Kode | Status |
|
||||
|---|---|---|
|
||||
| «gitt EN prompt + OKF-bundles» | `--explore "<prompt>" --explore-config F --bundle-dir D` (`run.py:1607-1627`) | **Delvis:** én base fra CLI/hosting; flere baser kun via `run.run_mandate_across_bundles` (bibliotek). Åpen operatørbeslutning (STATE). |
|
||||
| «stille nødvendige spørsmål» | Eneste spørsmål sløyfa kan stille et menneske er plan-review — én gang, før løpet (`explore.py:837-838` `_pending_plan_reviews`) | **Nei fra operatørflatene.** CLI nekter (`run.py:1975-1988`), hosting nekter (`hosting.py:167-172`). |
|
||||
| «teste hypoteser» | `quick_validate` (`explore.py:461-500`) — samme validator, rådgivende | **Ja** (offline; at en levende modell faktisk kaller verktøyet er ikke bevist — uttalt i CLAUDE.md). |
|
||||
| «be om svar, bruke svarene» | `plan_reviewer` bibliotek-only; ingen «spør eksperten»-tool blant de fire (`list_bundles`/`read_bundle`/`read_file`/`quick_validate`) | **Nei.** Planens rad 3 (U12 + asynkron U13) er nøyaktig dette, og er NESTE. |
|
||||
| «utforske løsningsrommet» | Magentic-manager + navigator (`explore.py:596-603`) | **Mekanisme ja**, virkning umålt. |
|
||||
| «finne innsparinger mennesker ikke ser» | — | **Ingen måling.** Null validerte levende forslag; den ene levende kjøringen fant opp `EL-LIGHTING-OP-HR` (`docs/2026-08-14-…` § 6). |
|
||||
| «regne/utforske hypoteser fagpersoner leverer» | `--mandate` (`run.py:1597`) + `seed_approaches` (`explore.py:848`, bibliotek) | **Ja** for regning (mandat-stien); utforskning av ekspertfrø kun via bibliotek. |
|
||||
| «EKSPERTER på MAF» | § 2: 5/19 fullt | **Nei.** Ekspertisen som finnes er *hvorfor MAF ikke passer* (as_agent, evals, Skills, graf, Concurrent — alle målt og avvist). Det er kompetanse, men ikke den operatøren beskrev. |
|
||||
|
||||
## 5. Funn, sortert etter alvorlighet
|
||||
|
||||
**BLOCKER** — ingen. Ingenting hindrer at systemet gjør det README lover *mekanisk*; problemene er
|
||||
påstander som ikke stemmer og misjonsmål uten måling.
|
||||
|
||||
### MAJOR
|
||||
|
||||
**F1 — README-status er stale, offentlig, i begge retninger.** `README.md:264-273`. «no
|
||||
live-model run yet» — falsk siden 14.08 (`docs/2026-08-14-fase1b-forste-levende-kjoring.md:217`).
|
||||
«sibling … built in parallel» — `STATE.md:103` sier parkert. Status-avsnittet sist endret
|
||||
2026-07-04 (`d8ee8d3`). `tests/test_public_surface_claims_loadbearing.py` gater to andre påstander
|
||||
(credential-klasse, wheel-navn), ikke denne. Operatøren lever av tillit; en README som
|
||||
underrapporterer er mindre farlig enn en som overrapporterer, men den er fortsatt usann.
|
||||
|
||||
**F2 — Ekspertdommen gis FØR forslaget finnes, og defaulter til `approved`.** `run.py:525`
|
||||
(`verdict_input: dict[str, str]` påkrevd), `:861` `capture_verdict(features, verdict_input["decision"], …)`,
|
||||
`:1691-1692` (`--decision` default `"approved"`, `--rationale` default `"reviewed by expert"`),
|
||||
`hosting.py:80` (`verdict_input` i `_REQUIRED_FIELDS`), `DEPLOY.md:198` («The expert verdict for
|
||||
this run»). `reference_domain.py:48` kaller det selv «SYNTHETIC Layer-2». Konsekvens: (a) hver
|
||||
CLI-kjøring uten flagg produserer et `RunResult.verdict` med en ekspertgodkjenning ingen ekspert
|
||||
ga; (b) i `run_portfolio` når den dommen neste prosjekts prompt som «prior expert verdict»
|
||||
(`run.py:1331`; `tests/test_portfolio_learning_loadbearing.py:108` beviser dataflyten); (c) på
|
||||
den hostede flaten MÅ en ekstern kaller dikte opp en dom for å få kjørt i det hele tatt — den
|
||||
når ikke svaret (`hosting.py:181-190` legger kun `verdict_id` ut), men den er et påkrevd felt som
|
||||
ikke kan fylles ærlig. Dette er non-goal 3 brutt i én søm, på flaten som ble overlevert 14.08.
|
||||
Den ærlige kanalen (Steg 7-innboksen, `docs/ekspert-svar.md`) finnes ved siden av — F2 handler om
|
||||
at den uærlige ikke er fjernet.
|
||||
|
||||
**F3 — `ExpeLContextProvider` er en MAF-hook som ikke bærer noe.** `run.py:736-739` folder
|
||||
læringen inn ved `.format_fewshot()` + string-konkatenering; `run.py:848-857` kjører `before_run`
|
||||
inn i en `SessionContext` kommentaren selv kaller «NOT what reaches the prompt». For U9 betyr det
|
||||
at MAF-konstruktet subklasses for å *se* brukt ut, mens injeksjonen skjer utenom. Funksjonelt
|
||||
riktig; som MAF-ekspertise er det motsatt av registerets intensjon.
|
||||
|
||||
**F4 — Målbildets «be om svar, bruke svarene» er ikke nåbart fra noen operatørflate.**
|
||||
`run.py:1975-1988`, `hosting.py:167-172`: begge nekter `enable_plan_review`. Eneste dør er
|
||||
`explore(…, plan_reviewer=…)` i bibliotek-APIet (`explore.py:849`). Ingen mid-løp-spørsmål
|
||||
finnes overhodet — kun plan-review før løpet. Planlagt (rad 3), ikke bygget.
|
||||
|
||||
**F5 — Ingen måling av misjonens kjernepåstand.** «finner innsparinger mennesker ikke ser» har
|
||||
null belegg: null validerte forslag mot levende modell (`docs/2026-08-14-…:245`), null ekte
|
||||
ekspertdommer i treet (`reference_projects.json:2`; alle `type: verdict`-filer merket frø/AI),
|
||||
planens rad 4 (utforskning vs. debatt på golden-bundelen) ugjort. Systemet kan i dag ikke skille
|
||||
«mekanismen virker» fra «metoden gir verdi».
|
||||
|
||||
**F15 — MAF-pinnen er erklært «ENDELIG» seks minor-releaser bak, uten re-verifisering.**
|
||||
`STATE.md:99` («orchestrations låst til `1.0.1` (core 1.9.0) — ENDELIG, operatørbekreftet 23.08»),
|
||||
`pyproject.toml` `agent-framework-core>=1.9.0,<2`. Målt [ekstern]: core 1.15.0 (21.08) og
|
||||
orchestrations 1.1.1 (krever core ≥1.15) var ute før bekreftelsen; 1.13.0 endret checkpoint-formen
|
||||
rad 3 skal bygges på. Pinnen er begrunnet (privat-API-premisser, `tests/test_maf_version_guard.py`),
|
||||
men den er ikke en MAF-ekspertise-posisjon — det er en frys. Ingen rutine i repoet re-leser §15 mot
|
||||
releasene; registeret har én commit (`ec9ac74`, 23.06). For målbildets «EKSPERTER på MAF» er
|
||||
dette det tyngste enkeltfunnet: ekspertisen er datert.
|
||||
|
||||
**F16 — B1-premisset er halvveis foreldet i den installerte versjonen.** Registeret (`§15.2 B1`):
|
||||
«Deterministisk verdikontroll er ikke en MAF-primitiv». Installert `agent_framework/_middleware.py`
|
||||
shipper `class MiddlewareTermination` uten `@experimental`; Learn (termination-siden): «Post-termination
|
||||
middleware — … useful for validating responses». Det er formen U8 ba om, og den ville gjort
|
||||
«forslaget forlater aldri agenten uvalidert» til en egenskap ved grafen i stedet for ved `run.py`.
|
||||
Tall-halvdelen (CBC/MC/baseline/provenance) står som egenbygg. 0 treff i `src`. Ikke et krav om å
|
||||
bygge om — et krav om at avvisningen av «graf-adopsjon» (P10) re-vurderes med dette premisset.
|
||||
|
||||
### MINOR
|
||||
|
||||
**F6 — U16/U17/U19 er uadresserte, ikke avviste.** Eneste spor er en telling
|
||||
(`docs/plan/2026-08-23-…:58`). U16 (`CompactionProvider`) er den relevante: Magentic-løp har kun
|
||||
token-taket som vern mot kontekstvekst.
|
||||
|
||||
**F7 — Layer-1 HITL er «green-but-dead».** `workflow.py:116` bak `enable_layer1_hitl=False`
|
||||
(`run.py:536`), intet CLI-flagg, eneste test `tests/test_workflow.py:58` bygger uten å kjøre.
|
||||
Repoets egen defektklasse (CLAUDE.md «grønn-men-død»).
|
||||
|
||||
**F8 — «Swarm» og «built on» er salgsspråk** (`README.md:9-10`). To agenter i debatt, tre i
|
||||
utforskning; MAF er en klient + én orkestrering på den normative stien.
|
||||
|
||||
**F9 — Metode-skillen finnes ikke.** CLAUDE.md-konvensjon «Metode kodifiseres som Agent Skill»;
|
||||
`find shared -name SKILL.md` → kun `expert-reviewer`, og dens prosa lastes aldri (`persona.py:27`
|
||||
leser kun JSON-eksempelet). U5 er dermed «nei» også som artefakt.
|
||||
|
||||
**F10 — Privat-API-avhengighet er pinnet, ikke fjernet.** `explore.py:73/:152` siterer
|
||||
`_magentic.py:1118/:1128-1131` linjenumre som premisser; `tests/test_maf_version_guard.py:1-14`
|
||||
pinner `_inner_get_response`/`_build_response_stream`; `orchestrations` låst til `1.0.1`. Riktig
|
||||
håndtert (tripwire + to-sidig pin), men det betyr at «MAF-ekspertise» her inkluderer avhengighet
|
||||
av interna som kan flytte seg ved neste minor.
|
||||
|
||||
**F11 — Debatten er en fast sekvens som alltid tvinges ferdig.** `workflow.py:99-113` round-robin,
|
||||
terminerings-nett `max_rounds*2+1` fyrer aldri; «forcing completion» i hver kjøring, også live
|
||||
(`docs/2026-08-14-…:59`). P9 i egnethetsplanen noterte det («Group Chat beholdes av byttekost») —
|
||||
det er ikke en debatt som konvergerer, det er tre tvungne turer.
|
||||
|
||||
**F12 — Feil nevner har sirkulert.** «6 av 15» kommer fra `docs/plan/2026-08-23-…:58`; registeret
|
||||
har 19 rader. Rett tall etter denne tellingen: 5 fullt / 5 delvis / 9 nei av 19.
|
||||
|
||||
**F17 — Fire register-rader bærer premisser som ikke stemmer for Python i dag** [ekstern +
|
||||
lokalt]: U15 navngir «Foundry Adaptive Evals» (ikke funnet på Learn); U16 navngir
|
||||
`CompactionProvider` (C#-navn — Python-formen er `compaction_strategy=`/`apply_compaction`, 0 treff
|
||||
i `src` også med det navnet); U19 navngir `HarnessAgent`/`ToolApprovalAgent` som Python 1.7.0
|
||||
(.NET-only; Python er `create_harness_agent`/`ToolApprovalMiddleware`); G8/A2 kaller Magentic
|
||||
eksperimentell (0 markører i installert `_magentic.py`, ingen Python-admonisjon på Learn).
|
||||
§15.6 «FIX» korrigerte åtte påstander i juni; ingen av disse fire er blant dem.
|
||||
|
||||
### NICE
|
||||
|
||||
**F13 — Non-goal 2-grensen bør uttales i README.** `--goals`/`--ledger` er portefølje-nivå-logikk
|
||||
(stopp, ikke reallokering). Én setning i Non-goals ville lukket tvetydigheten.
|
||||
|
||||
**F14 — B10 er utsatt til ekte dommer finnes** (`verdicts.py:313-314`) — riktig, men bør stå i
|
||||
README-status som en uttalt grense for hva «learns» betyr i dag (én bit + prosa).
|
||||
|
||||
## 6. Verifiseringslogg (kommandoer kjørt i denne økten, 2026-08-25)
|
||||
|
||||
| # | Påstand | Kommando → resultat |
|
||||
|---|---|---|
|
||||
| 1 | Suiten grønn | `PYTHONIOENCODING=utf-8 uv run pytest -q` → **1021 passed, 5 skipped, 2 warnings in 166.18s**, `EXIT=0` |
|
||||
| 2 | De 5 skip er env-gatede live-tester | `uv run pytest -q -rs tests/test_*_live.py` → 4 SKIPPED med grunn (Foundry / `PORTFOLIO_LIVE_FULL_RUN` / LOCAL endpoint); den femte er i samme klasse (full kjøring ga 5) |
|
||||
| 3 | Lint/typer rene | `uv run ruff check .` → «All checks passed!»; `ruff format --check` → 168 filer formatert; `uv run mypy src` → «Success: no issues found in 34 source files» |
|
||||
| 4 | Golden-hash | `shasum tests/golden/demo-transcript.stdout` → `ea8c534773acdbe41ae68f2c55724d69aaf8be4f` |
|
||||
| 5 | Testomfang | `ls tests/*.py \| wc -l` → 115 filer; `grep -rhE "^\s*(async )?def test_" tests \| wc -l` → 974 funksjoner |
|
||||
| 6 | src-omfang | `find src -name "*.py" \| xargs wc -l` → 34 filer, 11 650 linjer |
|
||||
| 7 | MAF-versjoner | `uv pip list` → core 1.9.0, orchestrations 1.0.1, foundry 1.8.2, openai 1.8.2; `uv.lock` samme |
|
||||
| 8 | §15 har 19 U-rader og én commit | `awk '/^## 15/,/^## 16/'` → U1–U19; `git log --follow -- docs/research/2026-06-23-prior-art-platform.md` → kun `ec9ac74` |
|
||||
| 9 | Hvert U-konstrukt mot src | `for pat in …; grep -rn --include="*.py" -F "$pat" src` — tabellen i § 2 |
|
||||
| 10 | README-status sist endret | `git log -L '/^> \*\*Status:\*\*/,+3:README.md'` → `d8ee8d3 2026-07-04` |
|
||||
| 11 | Levende kjøring konkluderte | `sed -n '156,260p' docs/2026-08-14-fase1b-forste-levende-kjoring.md` → `outcome_type: rejected`, `token_usage: 15 306` |
|
||||
| 12 | `verdict_input`-flyt | `grep -n verdict_input src/portfolio_optimiser/run.py` → `:525/:861/:2231/:2289`; `grep -nE '"--decision"\|"--rationale"'` → `:1691-1692`; `hosting.py:80` |
|
||||
| 13 | ExpeL-fold utenom hooken | `sed -n '700,760p;848,870p' src/portfolio_optimiser/run.py` |
|
||||
| 14 | Magentic ikke eksperimentell i installert pakke | `grep -n "experimental" .venv/…/agent_framework_orchestrations/_magentic.py` → 0 treff; `_skills.py` → `@experimental` `:76/:120`; `_evaluation.py` → 20 |
|
||||
| 15 | Multi-base-ordren arkivert (stoppbetingelse) | `find ~/.claude/coord/portfolio-optimiser -name "*080753Z*"` → `orders/archive/…` |
|
||||
| 16 | Ingen ekte ekspertdom i treet | `grep -rl "type: verdict" shared/examples src/…/data/bundles` → 12 filer; frontmatter `provenance: "frø — AI-forfattet …"`; `reference_projects.json:2` «SYNTHETIC» |
|
||||
| 17 | Nye MAF-konstrukter finnes i INSTALLERT 1.9.0 | `grep -rl -F "class MiddlewareTermination" .venv/…/agent_framework` → `_middleware.py` (kun `PROGRESSIVE_TOOLS`-markører i fila); `AgentLoopMiddleware` → `_harness/_loop.py`; `apply_compaction` → `_compaction.py`; `create_harness_agent` → `_harness/_agent.py`; `SequentialBuilder`/`HandoffBuilder` → orchestrations; `FileCheckpointStorage` + `allowed_checkpoint_types` → `_workflows/_checkpoint.py`; 28 `@experimental` på COMPACTION/HARNESS/LOOP |
|
||||
| 18 | Ingen av dem brukt | `grep -rn --include="*.py" -E "compaction_strategy\|apply_compaction\|MiddlewareTermination\|AgentLoopMiddleware\|create_harness_agent" src` → 0 |
|
||||
| 19 | MAF-versjoner på PyPI/Learn [ekstern] | Opus-agent, WebFetch mot `pypi.org/pypi/agent-framework-core/json` (×2, samsvar) + `learn.microsoft.com` — rapporten er gjengitt i § 3.1; GitHub-release-punkter merket LAVERE KONFIDENS der fetch-laget summerte |
|
||||
| 20 | Doc-gaten godtar det nye dokumentet (datert sti) | `uv run pytest -q tests/test_doc_constant_sync_loadbearing.py tests/test_public_surface_claims_loadbearing.py` → 20 passed |
|
||||
|
||||
**Ikke verifisert i denne økten:** at ingen bundle er materialisert fra en levende kilde (bare
|
||||
ikke funnet motbevis); Claude-SDK-søskenets faktiske tilstand (annet repo); at en levende modell
|
||||
faktisk kaller utforskningens verktøy (uttalt grense i CLAUDE.md, ikke målt her).
|
||||
283
docs/2026-08-25-syretest-vei-ab.md
Normal file
283
docs/2026-08-25-syretest-vei-ab.md
Normal file
|
|
@ -0,0 +1,283 @@
|
|||
# Syretesten vei A/B — de tre Vegnormal-basene gjennom portfolio-optimiser
|
||||
|
||||
**Ordre:** `20260825T111038Z-1174613178-from-.claude` (programplanens spor 3, gap G12).
|
||||
**Dato:** 2026-08-25 (økt 60). **Mandat: MÅL, IKKE BYGG.** Ingen fil under `src/` er endret;
|
||||
`uv run pytest -q` → **1021 passed / 5 skipped (166 s)**, identisk med tallet før økten.
|
||||
|
||||
**Eksponerings-grense (ordrens harde krav, holdt):** dette repoet pusher til `open/`. Rapporten
|
||||
bærer derfor kun **tall, stier, kommandoer og egne observasjoner**. Ingen bundle-fil er kopiert,
|
||||
og ikke én linje kravtekst fra et konsept er lest inn eller gjengitt — alle konsept-tall under er
|
||||
lengdemålinger, ikke innhold.
|
||||
|
||||
**Stopp-betingelsen var oppfylt:** multi-base-ordren `20260825T080753Z-103813595` lå i
|
||||
`orders/archive/` (commit `18af86e` + `785261f`) da økten startet. Multi-base-formen er lest slik
|
||||
den **faktisk landet** (`run.py`, `explore.py`, `mandate.py`, README), ikke slik planens § C.7
|
||||
omtalte den.
|
||||
|
||||
---
|
||||
|
||||
## 1. Sammendrag — én setning per målepunkt
|
||||
|
||||
| # | Punkt | Status | Kjernetall |
|
||||
|---|---|---|---|
|
||||
| 1 | Katalogen (`index_summary`, konsepter, cost-baseline, verdicts) | **MÅLT** | 446 / 1017 / 270 konsepter; **0 av 3** har `cost-baseline.json`; **0 av 3** har `validator-input.json`; **0** `type: verdict`-filer i alle tre |
|
||||
| 2 | Kontekstkostnad (`bundle_context(navigate_bundle(...))`) | **MÅLT** | 93 422 / 250 785 / 85 937 o200k_base-tokens — **sum 430 144** mot 3861/12595/10406 for de tre eksempelbundlene (instrumentet reproduserte commons' tall eksakt) |
|
||||
| 3 | Dry-run med tre baser i multi-base-formen | **MÅLT — og formen finnes ikke fra CLI-en** | Fire `--bundle-dir` gir **exit 0** og en kjøring mot ÉN base (siste vinner, stille); bibliotekdøra `run_mandate_across_bundles` **ruter korrekt** men feiler i dispatch: `FileNotFoundError @ okf.py:433` |
|
||||
| 4 | Offline-simuleringen med samme oppsett | **MÅLT** | Golden `demo-transcript.stdout` **BYTE-UENDRET** (`ea8c534773acdbe41ae68f2c55724d69aaf8be4f`), stderr 4 linjer; demoen **kan ikke** peke på en Vegnormal-base (samme `okf.py:433`) |
|
||||
| 5 | `--explore` med full seks-felts `--explore-config` | **MÅLT — delvis vakuøst, som ordren forutså** | Sløyfa **fullfører** offline mot tre baser og returnerer et **rutet** mandat (`bundle_id='B-n100-…'`, `stop=None`); men **0 verktøykall** og **0 `quick_validate`** — navigatoren åpnet aldri en base. Fra CLI-en er utforskningen **ikke kjørbar offline i det hele tatt** (`KeyError: 'navigator'`) |
|
||||
| 6 | G14: navigasjon inn i nestede `index.md` | **IKKE PRØVBAR HER — nevner oppgitt** | **0 nestede `index.md` av 1733 konsepter** i alle tre basene (0 underkataloger); kjent-positiv kontroll `nav-golden-hierarchy/bundle` finner 2 nestede index og 2 dypt-nådde kontekstfiler, så instrumentet **kan** se dem |
|
||||
|
||||
---
|
||||
|
||||
## 2. Tabellen (punkt 1 + 2)
|
||||
|
||||
Kommando bak hver rad: `okf.navigate_bundle(d)` → `okf.bundle_context(bundle)`, med
|
||||
`tiktoken.get_encoding("o200k_base")` (kjørt via `uv run --with tiktoken`; `tiktoken` er **ikke**
|
||||
lagt til som prosjekt-avhengighet).
|
||||
|
||||
| Base | Konsepter | Kontekstfiler | `index_summary` (tegn / tokens) | `bundle_context` tegn | bytes | **o200k_base-tokens** | verdicts | cost-baseline | validator-input | skipped links |
|
||||
|---|---:|---:|---|---:|---:|---:|---:|:--:|:--:|---:|
|
||||
| `B-n100-2023-uten-sources-importert` | 446 | 446 | 51 214 / 28 289 | 221 916 | 226 430 | **93 422** | 0 | nei | **nei** | 0 |
|
||||
| `B-n200-2024-uten-sources-importert` | 1017 | 1017 | 116 879 / 64 764 | 616 179 | 626 034 | **250 785** | 0 | nei | **nei** | 0 |
|
||||
| `B-n500-2024-uten-sources-importert` | 270 | 270 | 30 974 / 17 197 | 240 714 | 245 251 | **85 937** | 0 | nei | **nei** | 0 |
|
||||
| **Sum, tre baser** | **1733** | **1733** | 199 067 / 110 250 | 1 078 809 | 1 097 715 | **430 144** | 0 | — | — | 0 |
|
||||
| *kontroll:* `veglys-fv-soer` | 6 | 5 | 3 646 / — | 32 201 | 32 884 | **10 406** | 1 | ja | ja | 1 |
|
||||
| *kontroll:* `tunnel-hauglia` | 6 | 5 | 4 763 / — | 39 583 | 40 475 | **12 595** | 1 | ja | ja | 1 |
|
||||
| *kontroll:* `bygg-energi-mikro` | 5 | 4 | 1 884 / — | 12 005 | 12 270 | **3 861** | 1 | nei | ja | 1 |
|
||||
|
||||
**Instrumentet er validert mot kjent fasit** (Verifiseringsloven ansikt 4): de tre
|
||||
kontrollradene reproduserer commons' egne tall — 3861 / 12 595 / 10 406 — eksakt. Uten den
|
||||
kontrollen ville Vegnormal-tallene vært en måling ingen visste kunne treffe.
|
||||
|
||||
**Ordrens tall bekreftet mot ground truth** før noe ble bygget på dem: `447 / 1018 / 271` `.md`-filer
|
||||
på disk = `446 / 1017 / 270` konsepter + `index.md` i hver. Hver `index.md` har nøyaktig like mange
|
||||
lenker som det er konsepter (446 / 1017 / 270), alle unike, alle fulgt — `skipped = 0`.
|
||||
|
||||
**Det manageren faktisk ser.** Ett `list_bundles()`-kall (`explore.py:419-438`) returnerer hele
|
||||
`index_summary` for **alle** baser samtidig:
|
||||
|
||||
| | tegn | bytes | **o200k_base-tokens** |
|
||||
|---|---:|---:|---:|
|
||||
| `list_bundles()` over de tre basene | 201 196 | 201 196 | **112 116** |
|
||||
|
||||
Dette er ett verktøykall, og det er katalogverktøyets **eneste** form.
|
||||
|
||||
### Hva tallene sier om `.claude`s § 9.1-analyse
|
||||
|
||||
`.claude` sin `docs/okf-bundle-prosessen.md § 9.1` («en fil uten lenke finnes ikke; alt som lenkes
|
||||
leses helt») er **bekreftet mot et ekte korpus, og den er kostbar her**: alle 1733 konsepter er
|
||||
lenket fra rot-`index.md`, `skipped = 0`, og «leses helt» betyr 430 144 tokens for de tre basene.
|
||||
Progressiv disclosure gir ingen lettelse på denne bundle-formen, fordi importformen legger *alt*
|
||||
på ett nivå — se funn **MINOR-1**.
|
||||
|
||||
---
|
||||
|
||||
## 3. Funn
|
||||
|
||||
### BLOCKER-1 — kontekstkostnaden gjør en live utforskning mot disse basene ugjennomførbar som de står
|
||||
|
||||
`list_bundles()` = **112 116 tokens** i ett kall; `read_bundle("B-n200-…")` = **250 785 tokens**.
|
||||
`ExplorationContract.max_tokens` (`explore.py:70`) er ledgeren `BudgetMiddleware` håndhever, og et
|
||||
enkelt katalogkall bruker mer enn et normalt tak. En 128k-modell kan ikke ta N200 i det hele tatt.
|
||||
|
||||
**Dette er ikke en defekt i rammeverket** — det er korpusets form møtt av § 9.1-kontrakten. Men det
|
||||
er den harde grensen for vei A/B live, og den var ikke målt før i dag.
|
||||
**Fil:linje:** `src/portfolio_optimiser/explore.py:419-438` (`list_bundles`), `:444-445` (`read_bundle`).
|
||||
|
||||
> **Oppdatert 2026-08-26 (økt 65, ordre `20260825T213645Z-9019120455`):** katalog-halvdelen er
|
||||
> **lukket**. `list_bundles()` over de samme tre basene koster nå **362 tokens** (fra 112 116), og
|
||||
> over alle 171 grenbaser **21 448** (fra 124 942). `read_bundle`-halvdelen ble lukket på korpussiden
|
||||
> av `vegnormal-okf` `8145c23` (grener som egne baser). Måling og gate:
|
||||
> [docs/2026-08-26-katalogkostnaden.md](2026-08-26-katalogkostnaden.md). Setningen over står som
|
||||
> den ble målt 25.08 — den er historikk, ikke en gjeldende tilstand.
|
||||
|
||||
### MAJOR-1 — gjentatt `--bundle-dir` forkastes STILLE; kjøringen ser ut som multi-base og er det ikke
|
||||
|
||||
Ordrens pkt. 3 forutsatte at CLI-en tar tre baser. Målt:
|
||||
|
||||
```
|
||||
uv run python -m portfolio_optimiser.run VEGLYS-FV-SOER \
|
||||
--docs-dir shared/examples/veglys-fv-soer \
|
||||
--bundle-dir <N100> --bundle-dir <N200> --bundle-dir <N500> \
|
||||
--bundle-dir shared/examples/veglys-fv-soer --live-dry-run
|
||||
→ EXIT 0, "VEGLYS-FV-SOER: LIVE-DRY-RUN OK (…)"
|
||||
```
|
||||
|
||||
**Exit 0.** De tre Vegnormal-basene ble droppet uten ett ord. Bytter man rekkefølgen slik at en
|
||||
Vegnormal-base står sist, feiler samme kommando i stedet (`live-dry-run refused: IR projection not
|
||||
found in bundle: 'validator-input.json'`) — altså **siste `--bundle-dir` vinner**, som er argparse
|
||||
sin default når `action="append"` mangler.
|
||||
|
||||
Dette er repoets egen defektklasse, anvendt på operatørflaten: hosting-whitelisten nekter ukjente
|
||||
felt **ved navn** nettopp fordi stille dropping er uleselig utenfra, og multi-base-invarianten
|
||||
avviser en andre `bundle_dir` på `run_project`-signaturen fordi den ville tvunget «et stille
|
||||
velg-en». Her *er* det et stille velg-en — bare i argv i stedet for i signaturen.
|
||||
|
||||
**Fil:linje:** `src/portfolio_optimiser/run.py:1581-1583` (`add_argument("--bundle-dir", default=None…)`,
|
||||
ingen `action="append"`), konsumert `run.py:2070` (`bundle_dirs=(args.bundle_dir,)`) og `run.py:607-610`.
|
||||
**Minste ærlige rettelse (ikke bygget — ordren er MÅL, IKKE BYGG):** nekt et gjentatt `--bundle-dir`
|
||||
ved navn, på linje med de åtte eksisterende utforskningsnektene. STATE fører allerede
|
||||
«repeterbart `--bundle-dir`» som en **åpen operatørbeslutning** fra økt 58; denne målingen sier at
|
||||
inntil den er tatt, er *stillheten* selv problemet — ikke fraværet av funksjonen.
|
||||
|
||||
### MAJOR-2 — `--explore --scripted-replies` krasjer med rå traceback: `KeyError: 'navigator'`
|
||||
|
||||
Den ene offline-døra CLI-en har til `--explore` er ubrukelig. `_SCRIPTED_ROLES = ("proposer",
|
||||
"checker")` er debattens to roller; utforskningen trenger i tillegg `navigator`, `hypothesiser` og
|
||||
`manager`. `_load_scripted_replies` er eksplisitt fail-fast **for de to den kjenner** («a missing
|
||||
role would otherwise surface as a `KeyError` deep inside `scripted_factory`'s lookup, mid-run») —
|
||||
og så inntreffer nøyaktig det den advarer mot, for de tre den ikke kjenner:
|
||||
|
||||
```
|
||||
File ".../explore.py", line 576, in fresh_exploration_workflow
|
||||
client_factory(role),
|
||||
File ".../simulation.py", line 470, in factory
|
||||
reply = replies[role]
|
||||
KeyError: 'navigator'
|
||||
```
|
||||
|
||||
Ingen `run refused:`-linje, ingen rc-1 med forklaring — en traceback, som er den kanalen
|
||||
økt 57 betalte for å holde konfigurasjonsfeil UTE av.
|
||||
|
||||
**Positivt målt i samme kjøring:** `finally`-blokka holdt. `{run_id}-exploration.json` ble skrevet
|
||||
selv om kjøringen krasjet, med `"completed": false` og `"stop": null` — nøyaktig det
|
||||
`completed`-feltet finnes for.
|
||||
|
||||
**Fil:linje:** `src/portfolio_optimiser/run.py:1531` (`_SCRIPTED_ROLES`), `:1559-1564`
|
||||
(fail-fast-listen), krasjer i `src/portfolio_optimiser/simulation.py:470`.
|
||||
**Merk:** `simulation.scripted_exploration_factory` (`simulation.py:713-736`) dekker allerede alle
|
||||
fem rollene. Sømmen finnes; CLI-en når den bare ikke.
|
||||
|
||||
### MAJOR-3 — regelverksbaser kan ikke være baser i multi-base-dispatchen (arkitektonisk, ikke en bug)
|
||||
|
||||
Bibliotekdøra `run_mandate_across_bundles` ble målt direkte med de tre basene og et mandat med én
|
||||
approach per base:
|
||||
|
||||
```
|
||||
STEG 1 route_by_bundle → B-n100…: ['a0'] B-n200…: ['a1'] B-n500…: ['a2'] ✅ korrekt partisjon
|
||||
STEG 2 run_mandate_across_bundles → FileNotFoundError @ okf.py:433
|
||||
"IR projection not found in bundle: 'validator-input.json'"
|
||||
```
|
||||
|
||||
**Partisjonen virker perfekt.** Dispatchen gjør det ikke, fordi `run.py:1491` leser hver bases
|
||||
prosjekt fra **den basens egen** IR-projeksjon — som er selve multi-base-invariantens designvalg
|
||||
(«en kaller-oppgitt konstant kunne uansett bare vært riktig for én base av N»).
|
||||
|
||||
Konsekvensen er den viktigste innsikten i hele syretesten: **multi-base betyr N prosjekter, ikke
|
||||
1 prosjekt × N referansebaser.** Vegnormal-basene er *regelverk* — de har verken prosjekt eller
|
||||
kostbaseline, og skal ikke ha det. Ordrens mentale modell («ett veglysprosjekt + tre normalbaser
|
||||
som kontekst») er en **annen form**, og den finnes allerede — bare ikke i dispatchen:
|
||||
|
||||
| Dør | `bundle_dirs` betyr | Passer regelverk? |
|
||||
|---|---|---|
|
||||
| `run.run_mandate_across_bundles` (`run.py:1388`) | N **prosjektbaser** → N kjøringer | **Nei** — krever `validator-input.json` per base |
|
||||
| `explore.explore` (`explore.py:~840`) | N **lesekilder** for navigator/hypothesiser | **Ja** — målt, se under |
|
||||
|
||||
**Fil:linje:** `src/portfolio_optimiser/okf.py:431-433`, kalt fra `run.py:1491` (dispatchen),
|
||||
`run.py:609` (enkeltkjøringen) og `simulation.py` (demoen) — alle tre feiler på samme sted.
|
||||
|
||||
### MINOR-1 — importformen legger alt på ett nivå, så progressiv disclosure gir null lettelse
|
||||
|
||||
0 underkataloger, 0 nestede `index.md`, 1733 av 1733 konsepter lenket direkte fra rot. Navigasjonen
|
||||
har ingenting å utsette; hele korpuset er ett flatt nivå. Dette er en egenskap ved **kilden**, ikke
|
||||
ved `okf.py`.
|
||||
|
||||
> **Rettet 2026-08-26 (økt 65): tilskrivelsen var feil, og `vegnormal-okf` har rett.** Flatheten er
|
||||
> **Dør C** sin, ikke vegnormals emitterform. Verifisert mot kilden, ikke mot deres melding:
|
||||
> `llm-ingestion-okf` `src/llm_ingestion_okf/importer.py` (§6-index-blokka) kaller
|
||||
> `link_in_index(bundle, entry.path.name, _index_label(entry.concept_path))` per merget oppføring —
|
||||
> altså én flat lenke i rot-`index.md` for hvert konsept, uansett hvor nestet konseptstien er.
|
||||
> Vegnormals emitter skriver allerede et tonivåtre. Setningen over sto uendret som «vegnormal-okf
|
||||
> sin importform» til dette punktet.
|
||||
|
||||
Dette var også hele grunnen til BLOCKER-1: med nestede indekser kunne manageren åpnet én gren om
|
||||
gangen. Løsningen ble en annen — grener som **egne baser** (`vegnormal-okf` `8145c23`), med den
|
||||
målte begrunnelsen at basegrensen er der OKF-navigasjonen stopper, så ingen indeksstruktur INNE i en
|
||||
base senker prisen på å åpne den.
|
||||
|
||||
### NICE-1 — `read_bundle` nekter ukjent base ved navn, som lovet
|
||||
|
||||
```
|
||||
read_bundle("finnes-ikke") → ExplorationError: unknown knowledge base 'finnes-ikke';
|
||||
configured: B-n100-2023-…, B-n200-2024-…, B-n500-2024-…
|
||||
```
|
||||
|
||||
Nekten navngir det konfigurerte settet. **Fil:linje:** `explore.py:390-393`.
|
||||
|
||||
---
|
||||
|
||||
## 4. Hva punkt 5 faktisk viste — og hvor grensen for offline går
|
||||
|
||||
Ordren ba om at vakuiteten skulle måles, ikke antas. Målt, med `scripted_exploration_factory`
|
||||
(alle fem roller) mot de tre basene:
|
||||
|
||||
| Arm | Utfall |
|
||||
|---|---|
|
||||
| **B** — hypotese **uten** `bundle_id`, tre baser | `HypothesisParseError @ explore.py:732` — «a marked hypothesis must name its knowledge base when several are configured». **Multi-base-nekten fyrer korrekt mot et ekte korpus.** |
|
||||
| **C** — hypotese **med** `bundle_id`, tre baser | `stop=None`, **1 approach**, `bundle_id='B-n100-2023-uten-sources-importert'`, 2 ledger-runder, 6 modell-prompts. **Sløyfa fullfører og produserer et rutet mandat.** |
|
||||
|
||||
Og så det ærlige forbeholdet, som er poenget:
|
||||
|
||||
- `quick_validate`-kall: **0**
|
||||
- Prompt-strøm-forekomster av basenavnene: `N100` **2**, `N200` **0**, `N500` **0** — begge fra
|
||||
instruksjonene, ingen fra et verktøyresultat.
|
||||
|
||||
**Navigatoren åpnet aldri en base.** Den scriptede klienten returnerer tekst og emitterer ingen
|
||||
verktøykall — nøyaktig den grensen økt 56s måling allerede slo fast, nå bekreftet mot et eksternt
|
||||
korpus. Lesesømmen **selv** er derimot bevist mot disse basene, ved direkte kall (samme kontroll
|
||||
økt 56 måtte innføre da den oppdaget at armen lå utenfor gaten): `list_bundles()` → 3 baser,
|
||||
`read_bundle()` → 221 916 / 616 179 / 240 714 tegn.
|
||||
|
||||
**Konklusjon for punkt 5, uten pynt:** *plumbingen* er bevist ende-til-ende mot ekte eksterne
|
||||
baser — ruting, nekt, mandatform, artefaktskriving. *Verdien* er ikke bevist, og kan ikke bli det
|
||||
offline. **Syretesten trenger en levende modell.** Det er et funn, ikke en feil.
|
||||
|
||||
---
|
||||
|
||||
## 5. Hva som MÅ til for en live-kjøring
|
||||
|
||||
Målt i denne økten, ikke antatt:
|
||||
|
||||
| # | Mangler | Målt tilstand | Konkret |
|
||||
|---|---|---|---|
|
||||
| 1 | **Kontekstbudsjettet** | `list_bundles()` = 112 116 tokens; `read_bundle(N200)` = 250 785 | **Den harde blokkeringen.** Enten en modell med svært stort vindu, eller — mer realistisk — en bundle-form med nestede indekser slik at manageren kan åpne én gren. Kilden eies av `vegnormal-okf`. |
|
||||
| 2 | **Foundry-endepunkt** | `PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT` **ikke satt**, `FOUNDRY_PROJECT_ENDPOINT` **ikke satt** | Én av de to må eksporteres. Låst av operatørbeslutningen om `DisableLocalAuth` (ordre `20260821T094949Z`, `docs/2026-08-18-vurdering-azure-omdoeping.md`). **Ikke rørt her.** |
|
||||
| 3 | **Modell-map** | `PORTFOLIO_MODEL_MAP` ikke satt; `src/portfolio_optimiser/data/model_map.json` bærer `REPLACE-WITH-FOUNDRY-DEPLOYMENT` for alle azure-roller | `resolve_model("azure", r)` **nekter for alle fem roller**, inkl. `manager`/`navigator`/`hypothesiser`. Under `local` faller alle fem til `qwen3:4b` via `default` — utforskningsrollene er fortsatt ikke eksplisitt mappet (kjent ærlighets-grense fra økt 56). |
|
||||
| 4 | **En prosjektbase** | 0 av 3 Vegnormal-baser har `validator-input.json` eller `cost-baseline.json` | Kjøringen trenger et **prosjekt** å optimere. Vegnormal-basene er regelverket det optimeres *innenfor*. Riktig oppsett: `--bundle-dir <prosjektbase>` for pipelinen + de tre normalbasene som `explore(bundle_dirs=…)`-lesekilder — men det krever MAJOR-1 løst, siden CLI-en i dag sender **én** base til begge. |
|
||||
| 5 | **Offline-generalprøve** | `KeyError: 'navigator'` | MAJOR-2 må lukkes før en betalt kjøring, ellers er første live-kjøring også første gjennomkjøring. Repoets egen måleprotokoll: bevis så mye som mulig gratis, så en feil er attribuerbar. |
|
||||
|
||||
**Rekkefølge, uten å foregripe operatørens valg:** 5 → 1 → 2/3 → 4. Punkt 5 er gratis, punkt 1
|
||||
avgjør om vei A/B i det hele tatt er mulig med denne bundle-formen, og punktene 2–3 koster penger
|
||||
og er Azure-gatet.
|
||||
|
||||
---
|
||||
|
||||
## 6. Kommandologg
|
||||
|
||||
Hver tabellverdi over stammer fra én av disse, kjørt i denne økten:
|
||||
|
||||
```bash
|
||||
# pkt 1: konsepter, index-lenker, verdicts, cost-baseline
|
||||
find <base> -name '*.md' | wc -l ; grep -oE '\]\([^)]+\)' <base>/index.md | wc -l
|
||||
grep -lE '^type: *verdict' <base>/*.md | wc -l
|
||||
|
||||
# pkt 1+2: navigasjon, kontekst, tokens (instrument validert mot commons' tre fasittall)
|
||||
uv run --with tiktoken python # okf.navigate_bundle / okf.bundle_context / o200k_base
|
||||
|
||||
# pkt 3: CLI, fire --bundle-dir, begge rekkefølger
|
||||
uv run python -m portfolio_optimiser.run VEGLYS-FV-SOER --docs-dir … --bundle-dir … --live-dry-run
|
||||
|
||||
# pkt 3: bibliotekdøra
|
||||
python # mandate.route_by_bundle + run.run_mandate_across_bundles
|
||||
|
||||
# pkt 4: golden-regresjon
|
||||
uv run python -m portfolio_optimiser.simulation | shasum # ea8c534773acdbe41ae68f2c55724d69aaf8be4f
|
||||
|
||||
# pkt 5: CLI-en, og deretter explore() direkte med alle fem roller scriptet
|
||||
uv run python -m portfolio_optimiser.run … --explore … --explore-config … --scripted-replies …
|
||||
|
||||
# pkt 6: nestede index, med nav-golden-hierarchy som kjent-positiv kontroll
|
||||
|
||||
# regresjon
|
||||
uv run pytest -q # 1021 passed, 5 skipped, 166.00s
|
||||
```
|
||||
120
docs/2026-08-26-katalogkostnaden.md
Normal file
120
docs/2026-08-26-katalogkostnaden.md
Normal file
|
|
@ -0,0 +1,120 @@
|
|||
# Katalogkallet — `list_bundles()` fra 112 116 til 362 tokens
|
||||
|
||||
**Ordre:** `20260825T213645Z-9019120455-from-.claude`. **Dato:** 2026-08-26 (økt 65).
|
||||
**Kode:** `src/portfolio_optimiser/explore.py` (`_index_excerpt`, `navigator_tools`).
|
||||
**Gate:** `tests/test_catalogue_cost_loadbearing.py` (7 tester). **Suite:** 1066 passed / 5 skipped.
|
||||
**Golden `demo-transcript.stdout` byte-uendret** (`ea8c534773acdbe41ae68f2c55724d69aaf8be4f`).
|
||||
|
||||
**Eksponerings-grense (arvet fra syretesten, holdt):** dette repoet pusher til `open/`. Rapporten
|
||||
bærer kun tall, stier, kommandoer og egne observasjoner. Ingen bundle-fil er kopiert, og ingen
|
||||
kravtekst er gjengitt — alle konsept-tall er lengdemålinger, ikke innhold.
|
||||
|
||||
---
|
||||
|
||||
## 1. Instrumentet, validert før noe tall ble trodd
|
||||
|
||||
`tiktoken.get_encoding("o200k_base")` over `json.dumps(list_bundles(), ensure_ascii=False)`, kjørt
|
||||
med `uv run --with tiktoken` (`tiktoken` er fortsatt **ikke** en prosjekt-avhengighet). Samme
|
||||
instrument som syretesten 25.08.
|
||||
|
||||
**Kjent-positiv kontroll (Verifiseringsloven ansikt 4):** de tre flate Vegnormal-basene målte
|
||||
**201 196 tegn / 112 116 tokens** før endringen — tallet syretesten publiserte, reprodusert eksakt.
|
||||
Uten den kontrollen ville «etter»-tallet vært en måling ingen visste kunne treffe.
|
||||
|
||||
## 2. Før og etter, med nevner
|
||||
|
||||
| Katalog | Baser (nevner) | Tegn før | **Tokens før** | Tegn etter | **Tokens etter** | Endring |
|
||||
|---|---:|---:|---:|---:|---:|---:|
|
||||
| Tre flate baser (syretestens sett) | 3 | 201 196 | **112 116** | 877 | **362** | **−99,7 %** |
|
||||
| Grener, N100:2023 | 40 | 63 375 | **33 889** | 11 679 | **5 017** | −85,2 % |
|
||||
| Grener, N200:2024 | 99 | 134 667 | **71 726** | 28 894 | **12 396** | −82,7 % |
|
||||
| Grener, N500:2024 | 32 | 36 569 | **19 329** | 9 408 | **4 037** | −79,1 % |
|
||||
| **Alle grener samlet** | **171** | 234 611 | **124 942** | 49 981 | **21 448** | **−82,8 %** |
|
||||
| *kontroll:* commons' tre eksempelbaser | 3 | 11 050 | 3 472 | 989 | 321 | −90,8 % |
|
||||
|
||||
Per base: **731 → 125 tokens** i grenformen, **37 372 → 121** i den flate.
|
||||
|
||||
**Nevner-avvik mot ordren, uttalt:** ordren oppgir 34 grener for N100:2023. Målt på disk
|
||||
(`ls ~/repos/vegnormal-okf/build | grep -c '^B-n100-2023-gren-.*-importert$'`) er tallet **40**.
|
||||
N200:2024 = 99 og N500:2024 = 32 stemmer. Tallene over bruker den målte nevneren, ikke ordrens.
|
||||
|
||||
**Ordrens hypotese bekreftet:** grenformen lukket bundle-siden og gjorde katalogsiden **verre** —
|
||||
124 942 tokens over 171 grener mot 112 116 over tre flate baser. Etter endringen er hele
|
||||
maksimal-konfigurasjonen 21 448 tokens, altså under en femtedel av ett 128k-vindu.
|
||||
|
||||
## 3. Hva som ble endret, og hvorfor akkurat det
|
||||
|
||||
`list_bundles` returnerte `Bundle.index_summary` — **hele** rot-indeksens body — for **hver**
|
||||
konfigurert base samtidig, pluss ett JSON-objekt per ufulgt kryss-lenke. Begge vokser med korpuset.
|
||||
Prisen på å finne ut *hvilke baser som finnes* ble altså satt av hvor mye de basene *inneholder*.
|
||||
Det er progressiv disclosure snudd på hodet (målbilde §2/§4): katalogen er stigens billigste trinn,
|
||||
og den var loopens dyreste kall.
|
||||
|
||||
Hver oppføring er nå bundet ved konstruksjon: `id`, en **ordrett prefiks** av indeksbodyen på maks
|
||||
`_CATALOGUE_EXCERPT_CHARS` (200) tegn, `index_truncated`, `documents`, `verdict_count`,
|
||||
`cost_baseline`, `unreachable_links`. Kostnaden er **O(baser), aldri O(korpus)**.
|
||||
|
||||
**Et premiss ble felt FØR noe ble bygget på det.** «Indeksbodyen forteller en manager hva basen
|
||||
handler om» er **usant** for maskin-importerte baser: grenbasenes `index.md` har verken frontmatter
|
||||
eller prosa — den er en ren lenkeliste (målt: `B-n200-2024-gren-1-1-importert/index.md`, 959 bytes,
|
||||
første tegn er `-`). Feltet var altså ikke bare dyrt, det var dyrt **og** innholdsløst der. En
|
||||
avkortet prefiks taper ingenting en manager brukte.
|
||||
|
||||
Tre valg, hvert med sin begrunnelse:
|
||||
|
||||
- **Fast vindu, ikke en andel av basen.** En andel ville skalert med korpuset igjen — bare med en
|
||||
mindre konstant. Mutasjon M8 (`body[:len(body)//10]`) er rød mot nettopp den formen.
|
||||
- **Avkorting ANNONSERES, som felt.** `index_truncated` står **ved siden av** utdraget, ikke som en
|
||||
markør limt inn i det — samme grunn som `BudgetExceeded` bærer trippelen sin som felt (kø-(y)):
|
||||
en konsument som må re-parse prosa for å vite om den holder helheten, har fått en diagnostikk den
|
||||
ikke kan handle på. En base som **passer** blir ikke merket avkortet og får hele bodyen —
|
||||
omisjon, aldri en løgn i noen av retningene (M3 og M4 er begge røde, hver sin vei).
|
||||
- **En ufulgt kryss-lenke overlever som ANTALL.** Fakta fra økt 51 («et hopp er tolerert, men ikke
|
||||
lenger taust») står; per-lenke-detaljen blir liggende der den er handlingsbar — på
|
||||
`RunResult.skipped_links` / `DryRunReport.skipped_links` — og rir ikke med i et kall hvis hele
|
||||
jobb er å være billig.
|
||||
|
||||
Hele indeksen er fortsatt **ett** `read_file(id, "index.md")` unna. Dette er et disclosure-nivå,
|
||||
ikke datatap.
|
||||
|
||||
## 4. Gaten
|
||||
|
||||
Taket (500 tegn per base) bor i **testen**, ikke i `explore.py`: en test som importerte
|
||||
implementasjonens budsjett ville flyttet seg med det, og å heve budsjettet er nøyaktig den
|
||||
regresjonen fila finnes for.
|
||||
|
||||
Syv armer: (a) taket over mange store baser · (b) kostnaden sporer **ikke** korpus-størrelse ·
|
||||
(c) katalogen identifiserer fortsatt hver base — anti-vakuitets-armen, uten den består «returner
|
||||
bare id-ene» (a) perfekt · (d) avkorting annonsert, utdraget ordrett, begge retninger · (e) stigen
|
||||
er intakt, hele indeksen er ett kall unna · (f) ufulgte lenker overlever som antall ·
|
||||
(g) **kontrollen** — én bases rå indeksbody alene sprenger taket for hele katalogen, så en grønn (a)
|
||||
betyr at bindingen fyrte, ikke at fixturen var liten.
|
||||
|
||||
**Ni mutasjoner, alle røde mot HELE suiten, hver med sin egen signatur** (grønn kontroll 1066/5):
|
||||
|
||||
| # | Mutasjon | Røde tester |
|
||||
|---|---|---:|
|
||||
| M1 | ingen binding i det hele tatt (tilbake til hele bodyen) | 4 |
|
||||
| M2 | bundet, men vakuøst (utdraget tømt) | 2 |
|
||||
| M3 | avkorting ikke annonsert (stille kutt) | 1 |
|
||||
| M4 | over-annonsert (en hel indeks meldt avkortet) | 1 |
|
||||
| M5 | per-lenke-lista rir med igjen | 1 |
|
||||
| M6 | det ufulgte faktumet slettet | 1 |
|
||||
| M7 | bundet, men suffiks i stedet for ordrett prefiks | 2 |
|
||||
| M8 | en andel av basen i stedet for et fast vindu | 2 |
|
||||
| M9 | størrelses-signalet (`documents`) gjort konstant | 1 |
|
||||
|
||||
M9 ble kjørt fordi `documents` var et felt uten gate — et felt ingen test kan se, råtner.
|
||||
|
||||
## 5. Grenser, uttalt
|
||||
|
||||
- **MAJOR-1 (repeterbart `--bundle-dir`) var ikke nødvendig for dette.** Bindingen sitter i
|
||||
verktøykroppen, bak en uendret CLI-flate. Operatørbeslutning 26.08: repeterbart `--bundle-dir`
|
||||
er **nei** — multi-base forblir bibliotek-partisjonen (`run_mandate_across_bundles`).
|
||||
- **`navigate_bundle` kalles fortsatt per base per katalogkall**, altså leses hver fil på disk for å
|
||||
telle dokumenter og dommer. Det er I/O og veggklokke, ikke tokens, og er utenfor denne
|
||||
bestillingen — men det er ikke gratis ved 171 baser, og det er ikke målt her.
|
||||
- **Ingen levende modell har kalt det nye verktøyet.** Formen er bevist offline; at en manager
|
||||
faktisk velger bedre med et utdrag enn med hele indeksen er ikke målt (samme klasse som
|
||||
structured-output-grensen).
|
||||
- **N101 er ikke berørt** — utenfor bestillingen (operatørpresisering 26.08).
|
||||
230
docs/2026-08-29-maf-gjelden-omfang.md
Normal file
230
docs/2026-08-29-maf-gjelden-omfang.md
Normal file
|
|
@ -0,0 +1,230 @@
|
|||
# MAF-gjelden — omfang, ikke bygg (2026-08-29)
|
||||
|
||||
> **Ordre:** `20260825T214801Z-2660820373-from-.claude`, oppfølging av
|
||||
> [misjons-reviewen](2026-08-25-fable-misjonsreview.md) (25.08). **Mandat:** mål omfanget av fire
|
||||
> funn — F3, F15, F16, U16/U17/U19 — og lever kost/risiko/gevinst + en anbefalt rekkefølge. **Ingen
|
||||
> kode er endret.** Hvert tall under kommer fra en kommando kjørt i DENNE økten (verifiseringslogg
|
||||
> § 6); der reviewens tall er sitert direkte, er det merket.
|
||||
|
||||
## 0. Ferskhets-korreksjon FØR noe annet — reviewens telling er fire dager gammel
|
||||
|
||||
Reviewen (25.08) talte **5 ja / 5 delvis / 9 nei av 19**. Siden da har `c08ae91` (26.08, «U12 +
|
||||
asynkron U13, rad 3») landet checkpoint-persistens i `explore.py` (`checkpoint_storage=` på
|
||||
`_ALLOWED_CHECKPOINT_TYPES`, bak `--checkpoint-dir`). Frisk grep i dag (§ 6, linje 1–19) flytter
|
||||
**U12 fra nei til ja (opt-in)**:
|
||||
|
||||
**Ny telling, 29.08: 6 ja (U3, U4, U6, U12, U14, U18) · 5 delvis (U7, U8, U9, U11, U13) · 8 nei
|
||||
(U1, U2, U5, U10, U15, U16, U17, U19).** Av de åtte nei-radene er fem begrunnet i repoet (U1, U2,
|
||||
U5, U10, U15); tre er det ikke — **nøyaktig U16, U17, U19, uendret fra reviewen**, som er dette
|
||||
dokumentets fokus. Dette bekrefter at ordrens premiss («disse tre er uadresserte») fortsatt
|
||||
stemmer, men med et korrigert nevnertall — nok en instans av F12s advarsel («feil nevner har
|
||||
sirkulert»): selv reviewens eget tall råtner på fire dager.
|
||||
|
||||
## 1. F3 — `ExpeLContextProvider` er en MAF-hook som ikke bærer noe
|
||||
|
||||
**Bekreftet uendret** (linjenumre skiftet med commits siden 25.08, men koden og kommentarene er
|
||||
identiske i sak): `run.py:789-792` kaller `.format_fewshot()` og string-konkatenerer inn i
|
||||
`gen_context` FØR generering — dette er det som faktisk når prompten. `run.py:906-910` kjører
|
||||
`ExpeLContextProvider(...).before_run(...)` inn i en lokalt konstruert, kastet `SessionContext`
|
||||
**etter** at forslaget er generert — kommentaren sier det selv, ordrett: «this block's SessionContext
|
||||
is NOT what reaches the prompt» (`run.py:905`). Det andre kallet finnes utelukkende for å holde en
|
||||
regresjonsvakt i live (den to-argument `extend_instructions`-signaturen, «Critical Fase-1 GA-signature
|
||||
guard») — ikke for funksjon.
|
||||
|
||||
**Hvorfor det er vanskeligere enn å «bare koble den til» (målt § 6, linje 20-22):**
|
||||
`context_providers` er et konstruksjonsargument på `Agent`/`ChatAgent` (`_agents.py:385/676/1740`,
|
||||
også eksponert via `BaseChatClient.create_agent(context_providers=...)`, `_clients.py:577`) — det
|
||||
er **ikke** et argument `get_response()` tar. `generate_via_llm` (`generate.py:460`) kaller derimot
|
||||
`chat_client.get_response(...)` **direkte**, uten noen `Agent`-wrapper. Å la MAF-hooken faktisk bære
|
||||
injeksjonen krever at proposer-kallet flyttes fra rå `get_response()` til
|
||||
`chat_client.create_agent(context_providers=[...]).run(...)` (eller tilsvarende) — en strukturell
|
||||
endring av selve kall-stien Step 5 bygger på, ikke en lokal one-liner.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **Filer som røres** | `src/portfolio_optimiser/generate.py` (kall-sti, strukturert-output-wiring, retry/refine-løkke), `src/portfolio_optimiser/run.py:787-792/901-910` (fold + det dekorative kallet fjernes eller erstattes), `src/portfolio_optimiser/verdicts.py` (`ExpeLContextProvider`, mulig signaturendring) |
|
||||
| **Tester som må skrives/re-verifiseres** | `tests/test_step5_refine_loadbearing.py`, `tests/test_step5_history_loadbearing.py`, `tests/test_structured_output_loadbearing.py`, `tests/test_parse_failure_capture_loadbearing.py` (alle fire hviler på `generate_via_llm`s nåværende kall-form) |
|
||||
| **Oppgraderingsløp nødvendig?** | Nei isolert — `context_providers` finnes i installert 1.9.0 (verifisert § 6, linje 23). Uavhengig av F15. |
|
||||
| **Risiko** | **MEDIUM-HØY.** Rører den mest load-bearing løkka i repoet (budsjett-metering, strukturert-output-skjema, retry-historikk); en `Agent`-wrapper må bevise at den overflater SAMME usage/token-metadata `TokenMeter` leser i dag. |
|
||||
| **Gevinst** | Injeksjonen blir en egenskap ved GRAFEN i stedet for ved `run.py`s prosa — det U9 faktisk ba om. Fjerner det dekorative kallet (mindre kode, ingen «ser brukt ut»-forkledning). |
|
||||
| **Kost, isolert** | 1,5–2,5 økter (Opus 5/xhigh, TDD). Se § 4 for hvorfor F3 og F16 bør bygges SAMMEN. |
|
||||
|
||||
## 2. F15 — MAF-pinnen, re-verifisert EKSTERNT i dag (29.08, ikke 25.08)
|
||||
|
||||
**Reviewens tall var allerede fire dager gammelt da den ble skrevet; mitt tall er fra i dag.**
|
||||
PyPI (§ 6, linje 24-25): `agent-framework-core` siste er **1.16.0 (28.08 — GÅR I GÅR)**, ikke 1.15.0.
|
||||
Sekvensen siden 1.9.0: 1.10.0, 1.11.0, 1.12.0, 1.12.1, 1.13.0, 1.14.0, 1.15.0, **1.16.0**. Repoet er
|
||||
altså **SYV minor-releaser bak**, ikke seks. `agent-framework-orchestrations` siste er **1.1.1
|
||||
(21.08)**, og krever **`core<2,>=1.15.0`** — orchestrations kan IKKE oppgraderes uten å ta core med
|
||||
seg; det finnes ingen delvis bump.
|
||||
|
||||
**Ny, konkret ting reviewen ikke gjorde: jeg installerte 1.16.0 + orchestrations 1.1.1 i et isolert
|
||||
scratch-venv (`/tmp`, intet repo-treet berørt, slettet etter bruk) og diffet de to KONKRETE
|
||||
private-API-premissene `test_maf_version_guard.py` og `explore.py` navngir** (§ 6, linje 26-31):
|
||||
|
||||
- `_inner_get_response` / `_build_response_stream` (`agent_framework/_clients.py`) — **strukturelt
|
||||
uendret**, kun linjeskift på +3 (227→230 osv.). Samme nøkkelord-signatur.
|
||||
- `explore.py`s siterte `_magentic.py:1118`/`:1128-1131` (`speaker_known`/`next_speaker`-fallback-
|
||||
blokken) — **samme logikk, samme rekkefølge**, linjeskift +10 (filen vokste fra 1805 til 1810
|
||||
linjer). `next_speaker not in self._participant_registry.participants` — samme sjekk, samme sted
|
||||
relativt til resten av metoden.
|
||||
|
||||
Dette er en reell de-risking reviewen ikke hadde: de to konkrete tingene som gjør pinnen til en
|
||||
**test-time tripwire** (ikke bare en frys av bekvemmelighet) holder seg stabile over syv minor-
|
||||
releaser. `assert_supported_maf_version` (§ 6, linje 32) godtar for øvrig ALLEREDE `1.16.0` — guarden
|
||||
er en åpen range-sjekk (`major == 1 and minor >= 9`), ikke en eksakt pin; det er `uv.lock` som låser
|
||||
til nøyaktig `1.9.0`, ikke testen.
|
||||
|
||||
**Tre reelle breaking changes i vinduet (GitHub-releases, § 6, linje 33), rangert etter relevans for
|
||||
VÅR kode:**
|
||||
|
||||
1. **1.13.0 (30.07): «Workflow checkpoints are now fully replayable from initial input and
|
||||
human-in-the-loop responses, requiring architectural adjustments.»** Dette treffer DIREKTE
|
||||
`c08ae91`s checkpoint-wiring (`explore.py:478-493/1611`) — kode som er **tre dager gammel**
|
||||
relativt til en releasenote som endret akkurat det formatet. `tests/test_async_plan_review_loadbearing.py`
|
||||
(17 tester) må re-kjøres mot den nye formen FØR noen bump vurderes trygg.
|
||||
2. **1.15.0 (21.08): OTel GenAI-konvensjonene konsolidert** («stable and experimental modes with
|
||||
explicit message-event controls»). Treffer `tracing.py` (U14) og — fordi demoens pinnede stderr
|
||||
er en byte-fasit (P4) — potensielt `tests/golden/demo-transcript.stdout`/`.stderr`.
|
||||
`tests/test_tracing_loadbearing.py` (9 mutasjoner) må re-kjøres.
|
||||
3. **1.15.0: `MiddlewareFailure`** som nytt fatalt signal for function middleware. Lav risiko for
|
||||
oss (vi bruker `ChatMiddleware`/`FunctionMiddleware` til budsjett og observasjon, ikke til å
|
||||
blokkere), men bør sjekkes mot `budget.py:228`/`mcp_tools.py:197`.
|
||||
|
||||
**Er begrunnelsen for pinnen fortsatt gyldig?** **Delvis.** De to spesifikke private-API-premissene
|
||||
tripwiren finnes for, holder (målt i dag, ikke antatt). Det som IKKE lenger holder, er STATE.md:99s
|
||||
ordvalg «ENDELIG» — «endelig» beskriver en beslutning som ikke trenger re-evaluering; det denne
|
||||
målingen viser er at prisen for å UTSETTE fortsetter å vokse (syv releaser og stigende), mens selve
|
||||
oppgraderingsrisikoen på de to tingene som faktisk kan knekke oss, er lav og MÅLT lav. «Endelig» bør
|
||||
leses som «endelig for denne runden, re-vurder ved neste ordre», ikke som en frys uten utløpsdato.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **Filer som røres** | `pyproject.toml` (begge nedre grenser: `agent-framework-core>=1.15.0`, `agent-framework-orchestrations>=1.1.0` el. l.), `uv.lock` (`uv lock --upgrade-package agent-framework-core --upgrade-package agent-framework-orchestrations`), `tests/golden/demo-transcript.stdout`/`.stderr` (mulig regenerering + manuell diff-gjennomgang), `STATE.md`/`CLAUDE.md` (pin-språket) |
|
||||
| **Tester som må skrives/re-verifiseres** | Hele suiten (1080/5, ~200s) + spesielt `tests/test_async_plan_review_loadbearing.py`, `tests/test_tracing_loadbearing.py`, `tests/test_maf_version_guard.py` (bør IKKE trenge endring — se over) |
|
||||
| **Oppgraderingsløp nødvendig?** | **Ja, ETT sprang til 1.16.0** — orchestrations tvinger core≥1.15.0 uansett, så det finnes ingen billigere mellomstopp. |
|
||||
| **Risiko** | **MEDIUM.** De to harde premissene er målt stabile; tre navngitte breaking changes treffer nylig bygget/skjørt kode (checkpoint 3 dager gammel, OTel byte-fasit). Ikke en rewrite, men heller ikke en formalitet. |
|
||||
| **Gevinst** | Lukker syv-minor-gapet FØR det blir åtte/ni; fjerner den tyngste enkeltinnvendingen mot «eksperter på MAF»-påstanden; åpner orchestrations 1.1.1 (innhold ikke undersøkt her — egen sak); `_compaction.py` mister sine `@experimental`-dekoratorer i 1.16.0 (se § 4, U16 — mulig delvis relevant for U16, men Learn-dokumentasjonen kaller compaction fortsatt eksperimentell i prosa, så dette er IKKE en avklart «nå er U16 trygt» — kun et signal verdt å re-sjekke ETTER bump). |
|
||||
| **Kost, isolert** | 1–2 økter (Sonnet 5/xhigh for selve bumpen + re-kjøring; eskaler til Opus 5/xhigh hvis checkpoint- eller OTel-re-verifiseringen faktisk finner brudd). |
|
||||
|
||||
## 3. F16 — B1-premisset er halvveis foreldet i INSTALLERT 1.9.0 (bekreftet, også i 1.16.0)
|
||||
|
||||
**Bekreftet uendret, og bekreftet i BEGGE versjoner** (§ 6, linje 34-35): `class
|
||||
MiddlewareTermination` finnes i `agent_framework/_middleware.py` UTEN `@experimental`-markør, i
|
||||
både installert 1.9.0 og scratch 1.16.0 (som i tillegg legger til `MiddlewareFailure` ved siden av
|
||||
den — samme ikke-eksperimentelle status). `grep -rn "MiddlewareTermination" src/` gir **0 treff**
|
||||
(§ 6, linje 3) — registerets U8 («intercept av tool-calls → blokkerende validator») beskrev nøyaktig
|
||||
denne formen; repoet har den ikke. Vår gate (`validate_proposal`, CBC + Monte Carlo + baseline-
|
||||
avstemming) kalles i dag ETTER generering, i `run.py`, utenfor grafen (P10-avgjørelsen).
|
||||
|
||||
**Ordrens eget premiss («ikke et krav om å bygge om») stemmer, og det er viktig:** dette funnet ber
|
||||
IKKE om at validatoren flyttes inn i middleware — det ber om at **P10-avvisningen av «graf-adopsjon»
|
||||
re-vurderes MED dette premisset**. Å faktisk flytte tall-gaten inn i middleware ville krevd nøyaktig
|
||||
samme forutsetning som F3 (proposer-kallet må gå via `Agent`/`create_agent()`, ikke rå
|
||||
`get_response()`, for at noen middleware-pipeline skal ha noe å hekte seg på) — **F3 og F16 deler
|
||||
samme forutsetning-refaktor.** De bør IKKE bestilles som to uavhengige jobber; en løsning som bygger
|
||||
den ene kall-sti-endringen løser forutsetningen for begge.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **Kost — TIER 1 (bare re-vurdere P10, dokumentere)** | **Nær null.** Én linje i STATE/CLAUDE.md: «P10 (avvis graf-basert validering) står, MEN premisset var 'MAF har ingen blokkerende-svar-primitiv' — det er nå delvis usant (`MiddlewareTermination`); beslutningen opprettholdes av en ANNEN grunn: validatoren trenger IR-parsing + solver + Monte Carlo, som ikke er en ting man vil kjøre synkront inne i response-middleware uten videre design.» Kan gjøres i denne leveransen om operatøren ønsker det — se § 5. |
|
||||
| **Kost — TIER 2 (faktisk bygge om)** | Delt med F3s refaktor (se over). Isolert tillegg utover F3: `src/portfolio_optimiser/validator.py`-kallet må flyttes/wrappes som middleware, `tests/test_checker_gate_loadbearing.py` og `tests/test_s40_cost_baseline_loadbearing.py` må re-verifiseres mot den nye kall-formen. |
|
||||
| **Risiko** | TIER 1: ingen. TIER 2: **HØY** — rører selve den «obligatoriske, blokkerende» invarianten (CLAUDE.md «Arbeidsflyt (invarianter)», første linje). Enhver regresjon her er alvorlig per repoets eget prinsipp. |
|
||||
| **Gevinst** | TIER 1: ærlighet — P10 slutter å hvile på et utdatert premiss. TIER 2: «forslaget forlater aldri agenten uvalidert» blir en egenskap ved grafen, ikke ved `run.py`s disiplin — mer robust mot en fremtidig kalles feil, men samme FUNKSJONELLE utfall som i dag (suiten dekker begge like godt). |
|
||||
| **Kost, isolert** | TIER 1: 0 økter (kan gjøres nå). TIER 2: bygg SAMMEN med F3 — se § 4s samlede anslag, ikke additivt. |
|
||||
|
||||
## 4. U16/U17/U19 — uadresserte fordi ingen har sett, eller fordi de ikke passer?
|
||||
|
||||
Alle tre svares ut individuelt, per ordrens § 3 («et begrunnet 'nei' er et gyldig utfall»). To av de
|
||||
tre er små nok til at MÅLINGEN selv lukker raden (ordrens unntak) — gjort under, i dette dokumentet.
|
||||
|
||||
**U17 — `as_tool()` (agent-som-verktøy): IKKE PASSER. LUKKES HER.** Registerets egen kobling
|
||||
(`§15.5 A3`) knytter `as_tool()` til KRYSS-PROSJEKT agent-komposisjon («bruk in-process
|
||||
agent-as-tool først [for kryss-prosjekt], kun A2A ved sikkerhets-/compliance-grense»). Kryss-prosjekt
|
||||
er non-goal 2 i README («Not a portfolio-level reallocator … out of scope»). Uten kryss-prosjekt-
|
||||
komposisjon finnes det ingen naturlig bruksplass for «agent som verktøy for en annen agent» i denne
|
||||
arkitekturen — hvert prosjekt kjøres uavhengig. **Vedtak: NEI, begrunnet, permanent** (til non-goal 2
|
||||
eventuelt oppheves — usannsynlig, det er en README-forpliktelse). 0 treff bekreftet (§ 6, linje 15).
|
||||
|
||||
**U19 — Agent Harness: FORTSATT EKSPERIMENTELL, OGSÅ I 1.16.0. IKKE PASSER NÅ. LUKKES HER.**
|
||||
Scratch-venv-diffen (§ 6, linje 36) viser `@experimental(feature_id=ExperimentalFeature.HARNESS)` på
|
||||
`create_harness_agent`, `FileMemoryStore`/`_memory.py`, `_background_agents.py`, `_file_access.py` —
|
||||
like mange markører i 1.16.0 som i 1.9.0. Reviewens observasjon («installert pakke sier selv
|
||||
`[HARNESS] MemoryStore is experimental`») er IKKE et 1.9.0-artefakt som forsvinner ved en bump — det
|
||||
holder seg over syv minor-releaser, i motsetning til Magentic (som MISTET sin eksperimentelle status
|
||||
og DERFOR ble tatt i bruk, § 3 i reviewen). Funksjonelt overlapper harness («bakgrunnsagent med
|
||||
fil-tilgang og løkke») med det utforskningssløyfa (`explore.py`, Magentic + checkpoint + plan-review)
|
||||
allerede gjør med et ANNET, ikke-eksperimentelt sett byggeklosser. **Vedtak: NEI, begrunnet, for nå**
|
||||
— revisit-utløser: harness mister `@experimental`-markøren PÅ NYTT målt, ELLER et konkret behov
|
||||
oppstår som Magentic+checkpoint ikke dekker (f.eks. en agent som må overleve utenfor request/response-
|
||||
syklusen helt, ikke bare over dager via checkpoint). 0 treff bekreftet (§ 6, linje 17).
|
||||
|
||||
**U16 — Kontekst-kompaktering: EKTE GAP, INGEN HAR SETT PÅ DET. IKKE LUKKET, DEFERRED MED VILKÅR.**
|
||||
I motsetning til U17/U19 er dette IKKE en «passer ikke»-sak. Lange Magentic-utforskningsløp
|
||||
(`explore.py`) har i dag KUN token-taket (`ExplorationContract.max_tokens`) som vern mot
|
||||
kontekstvekst — det er et KOST-tak, ikke et vindus-tak; en løkke kan i prinsippet nå modellens
|
||||
kontekstvindu-grense (avvisning fra API-et) FØR den når token-BUDSJETTET, og de to feiler ulikt
|
||||
(én er en strukturert `BudgetExceeded`, den andre er en rå leverandørfeil). Ingen levende
|
||||
utforskningskjøring har vart lenge nok til å utløse dette (F5: null validerte forslag mot levende
|
||||
modell), så det er ikke observert i praksis — men det er heller ikke DESIGNET bort. Python-API-en er
|
||||
`compaction_strategy=` på `Agent`-konstruksjon eller frittstående `apply_compaction()`
|
||||
(`_compaction.py`), IKKE C#-navnet `CompactionProvider` registeret bruker (F17 — feil navn, samme
|
||||
konklusjon). Kost til å lukke det EKTE gapet (ikke bare dokumentere fraværet): en design-beslutning
|
||||
om HVOR i Magentic-løkka kompaktering skal anvendes uten å miste frø-hypotesene (§ C.6 dør 1s
|
||||
bevaringsregel — kompaktering må aldri kunne kutte en frø-tilnærming), pluss ny wiring + tester.
|
||||
**Vedtak: NEI, utsatt, begrunnelse skrevet ned HER** (tilfredsstiller ordrens unntak for U-rader) —
|
||||
revisit-utløser: en `--explore`-kjøring mot en levende modell treffer et kontekstvindu-avslag, ELLER
|
||||
operatøren ønsker å bygge lange (>10 runder) utforskningsløp proaktivt. **Kost hvis/når utløst:**
|
||||
uestimert i dag — krever egen scoping-økt (design + § C.6-bevis), IKKE inkludert i totalanslaget § 5.
|
||||
|
||||
## 5. Anbefalt rekkefølge og ærlig totalanslag
|
||||
|
||||
1. **F15 (versjonsbump 1.9.0→1.16.0).** Gjøres FØRST fordi de to spesifikke tingene som kunne gjort
|
||||
den farlig, er nå MÅLT stabile (§ 2), og fordi hvert videre arbeid på F3/F16 uansett må
|
||||
re-verifiseres mot HVILKEN core-versjon som til slutt kjører — å bygge F3/F16 mot 1.9.0 først
|
||||
betyr dobbelt verifiseringsarbeid hvis F15 kommer etterpå. **1–2 økter.**
|
||||
2. **F16 TIER 1 (re-vurder P10, dokumentér — kan gjøres NÅ, se under).**
|
||||
3. **F3 + F16 TIER 2, bygget SOM ÉN JOBB** (delt forutsetning: proposer-kallet flyttes fra rå
|
||||
`get_response()` til `Agent`-basert konstruksjon). Gjøres ETTER F15, slik at refaktoren skjer mot
|
||||
den versjonen som faktisk skal kjøre i produksjon. **2,5–4 økter** (Opus 5/xhigh, TDD, høy
|
||||
testtetthet — anbefal adversarial review før commit gitt at dette rører den obligatoriske
|
||||
validator-invarianten).
|
||||
4. **U16** — ingen bygg nå; revisit ved utløser (§ 4). **0 økter i dag.**
|
||||
5. **U17/U19** — LUKKET i dette dokumentet, ingen videre arbeid med mindre revisit-vilkårene inntreffer.
|
||||
|
||||
**Totalanslag for det operatøren FAKTISK må velge å betale for: 3,5–6 økter** (F15 + F3/F16 TIER 2).
|
||||
F16 TIER 1 og U17/U19 er reelt gratis — de er dokumentasjon, ikke bygg, og kan committes sammen med
|
||||
dette dokumentet uten å krysse ordrens «bygg ingenting»-grense (ingen kjørende kode eller pin røres).
|
||||
|
||||
**Det jeg IKKE anbefaler:** å bestille F3/F16 TIER 2 uten F15 først (dobbelt verifiseringsarbeid), og
|
||||
å bestille alle fire som én pakke (ordrens egen begrunnelse for hvorfor dette ble målt separat —
|
||||
ulik størrelse, ulik risiko — står; F15 er en ren versjons-øvelse, F3/F16 TIER 2 er et
|
||||
arkitektur-inngrep i den mest sensitive invarianten i repoet).
|
||||
|
||||
## 6. Verifiseringslogg (kommandoer kjørt i DENNE økten, 2026-08-29)
|
||||
|
||||
| # | Påstand | Kommando → resultat |
|
||||
|---|---|---|
|
||||
| 1-19 | Fresh U1-U19 grep-tally | `grep -rn <konstrukt> src/` per rad, § 0 — se full liste i øktloggen; U12 flyttet nei→ja siden `c08ae91` |
|
||||
| 20 | Suiten grønn i dag | `PYTHONIOENCODING=utf-8 uv run pytest -q` → **1080 passed, 5 skipped, 2 warnings in 196.69s** |
|
||||
| 21 | Ingen kildeendring siden reviewen berører F3/F16/U16/U17/U19 | `git log --oneline 6020f44..HEAD` → 3 commits (F2, katalogkall-kost, U12-checkpoint) — ingen treffer disse fem radene |
|
||||
| 22 | pyproject/uv.lock pin | `grep -n agent-framework pyproject.toml`; `grep -A1 'name = "agent-framework' uv.lock` → core/orch/foundry/openai alle pinnet eksakt |
|
||||
| 23 | F3 kodesti | `sed -n '740,920p' src/portfolio_optimiser/run.py`; `grep -n "class ExpeLContextProvider\|def before_run\|extend_instructions" src/portfolio_optimiser/verdicts.py` |
|
||||
| 24 | `context_providers` kun på Agent, ikke `get_response()` | `grep -n context_providers .venv/…/agent_framework/_agents.py` (7 treff, alle på `Agent`/`ChatAgent`-konstruktør) vs. `grep -n get_response src/portfolio_optimiser/generate.py` (rå klientkall) |
|
||||
| 25 | PyPI core siste versjon [ekstern] | `WebFetch pypi.org/pypi/agent-framework-core/json` → 1.16.0 (28.08); sekvens 1.10.0→1.16.0 |
|
||||
| 26 | PyPI orchestrations siste + dependency [ekstern] | `WebFetch pypi.org/pypi/agent-framework-orchestrations/json` → 1.1.1 (21.08), krever `core<2,>=1.15.0` |
|
||||
| 27 | GitHub releases changelog 1.13-1.16 [ekstern] | `WebFetch github.com/microsoft/agent-framework/releases` → breaking changes sitert § 2 |
|
||||
| 28-31 | Scratch-venv privat-API-diff | `uv venv /tmp/maf-scratch-1160 --python 3.12`; `uv pip install --python … "agent-framework-core==1.16.0" "agent-framework-orchestrations==1.1.1"`; `grep -n _inner_get_response\|_build_response_stream` mot begge versjoner av `_clients.py`; `sed -n '1110,1135p'` mot begge versjoner av `_magentic.py`; scratch-venv slettet etter bruk (`rm -rf /tmp/maf-scratch-1160`) |
|
||||
| 32 | Versjonsguarden godtar 1.16.0 allerede | Lest `tests/test_maf_version_guard.py:assert_supported_maf_version` — `major == 1 and minor >= 9`, ingen øvre minor-grense |
|
||||
| 33 | Tre breaking changes identifisert | Samme GitHub-fetch som # 27, kryssjekket mot `explore.py`/`tracing.py`/`budget.py` |
|
||||
| 34 | F16: MiddlewareTermination i 1.9.0 | `grep -rn MiddlewareTermination src/` → 0 treff; `grep -n "class MiddlewareTermination\|@experimental" .venv/…/agent_framework/_middleware.py` |
|
||||
| 35 | F16: MiddlewareTermination i 1.16.0 | Samme grep mot scratch-venv → identisk status (ikke eksperimentell), pluss ny `MiddlewareFailure` |
|
||||
| 36 | U19: harness eksperimentell i 1.16.0 | `grep -rn "@experimental" scratch-venv/…/agent_framework/_harness/*.py` → 16 treff, samme mønster som 1.9.0 |
|
||||
|
||||
**Ikke verifisert i denne økten:** innholdet i `orchestrations` 1.1.0/1.1.1 utover changelog-
|
||||
overskriftene (egen sak om operatøren vil vite HVA som vinnes der utover selve versjonstvangen);
|
||||
hvorvidt `_compaction.py`s manglende `@experimental`-dekoratorer i 1.16.0 betyr at Microsoft Learn-
|
||||
dokumentasjonens «currently experimental in Python»-advarsel (sitert i reviewen § 3.1) er utdatert
|
||||
eller om den fortsatt gjelder på et annet nivå (prosa vs. kode) — nevnt som åpent i § 4, ikke avgjort
|
||||
her; hvorvidt en faktisk 1.16.0-bump ville endret golden-transkriptets byte-fasit (krever å faktisk
|
||||
kjøre demoen mot den bumpede pinnen, som er byggearbeid utenfor denne ordren).
|
||||
|
|
@ -23,6 +23,15 @@ Systemet tar imot en ekspertdom på to steder, og de gjør forskjellige ting:
|
|||
|
||||
Form B er den som lærer systemet noe. Form A er en synkron snarvei.
|
||||
|
||||
**Ingen av dem er default, og det er poenget.** Utelater du begge flaggene, blir det ikke registrert
|
||||
noen dom i det hele tatt — kjøringen sier det høyt (`no expert verdict given; verdict key=…`), og
|
||||
ingenting går inn i læringsstoren. Fram til 1.1.0 defaultet `--decision` til `approved`, slik at
|
||||
hver flaggløs kjøring produserte en ekspertgodkjenning ingen ekspert hadde gitt, og bar den videre
|
||||
inn i neste prosjekts hypotese. Nøkkelen linja skriver ut er den du fører inn i `id`-feltet i Form
|
||||
B, og det er slik en dom avgitt dager senere finner tilbake til kandidaten den gjelder.
|
||||
**Halv dom nektes:** `--decision` og `--rationale` må gis sammen. Den manglende halvdelen er
|
||||
ekspertens å skrive, aldri systemets å fylle inn.
|
||||
|
||||
**`decision` er binær på kjørestien: `approved` eller `rejected`.** Det finnes ikke et tredje
|
||||
«godkjent med justering» her — en realiseringskorreksjon er en **godkjenning** der forbeholdet står
|
||||
i begrunnelsen. Det er med vilje: forbeholdet er fagkunnskap som skal leses, ikke en enum-verdi som
|
||||
|
|
|
|||
|
|
@ -74,9 +74,10 @@ layer does **not** produce: ingest materializes *source documents*, while the IR
|
|||
the *candidate measure*, and no connector can infer one from the other. A bundle may also ship an
|
||||
optional `cost-baseline.json`; without it the validator still runs, but unanchored to the
|
||||
project's real cost lines. Both are hand-authored today. For the IR projection the shape reference
|
||||
is `shared/examples/bygg-energi-mikro/validator-input.json`; **no bundled example ships a
|
||||
`cost-baseline.json`** (checked), so its shape — `{code: {quantity, unit_cost}}` — comes from
|
||||
`ir.CostBaseline` and the README. Writing them from ingested content is
|
||||
is `shared/examples/bygg-energi-mikro/validator-input.json`; for the cost baseline it is
|
||||
`shared/examples/veglys-fv-soer/cost-baseline.json` — **two bundled examples ship one** (that one
|
||||
and `tunnel-hauglia`, checked 2026-08-21), and its shape is `ir.CostBaseline`: a `project_id` plus
|
||||
an `items` map of `{code: {quantity, unit_cost}}`. Writing them from ingested content is
|
||||
unbuilt, and is not on the 90 %-principle side of the line: what candidate to propose is the
|
||||
agents' job, not the connector's.
|
||||
|
||||
|
|
|
|||
|
|
@ -45,6 +45,15 @@ The verdict is always the human's; the machine only ever translates and structur
|
|||
the gaps it names. Repeat until the base is comprehensive enough to be useful. This iteration
|
||||
loop is where most of the 1-2 weeks is spent.
|
||||
|
||||
## Composing the base for one specific run
|
||||
|
||||
This recipe describes the *process*. It does not say which categories of knowledge a given run
|
||||
needs, what each content type is for, or what happens when one is missing. That is covered, in
|
||||
Norwegian for the domain expert and the technical person together, in
|
||||
[`kunnskapsbase-for-en-kjoring.md`](kunnskapsbase-for-en-kjoring.md) — including a worked road
|
||||
project from the commission to a base that passes the dry-run check. The two documents are
|
||||
deliberately disjoint: phases and roles live here, composition lives there.
|
||||
|
||||
## Reading the bundle
|
||||
|
||||
Bundles are plain markdown, so **Obsidian or VS Code** are enough to read and navigate them — no
|
||||
|
|
|
|||
611
docs/kort-presentasjon.html
Normal file
611
docs/kort-presentasjon.html
Normal file
|
|
@ -0,0 +1,611 @@
|
|||
<!doctype html>
|
||||
<meta charset="utf-8">
|
||||
<title>Pengene inne i porteføljen</title>
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<style>
|
||||
:root {
|
||||
--ground: #F6F5F1; --surface: #FFFFFF; --surface-2: #EFEDE6;
|
||||
--ink: #22272B; --muted: #5C6570; --line: #D9D6CC;
|
||||
--accent: #C89B00; --accent-ink: #7A5F00; --steel: #35566F;
|
||||
--ok-bg: #E3F0E7; --ok-fg: #1F5C38;
|
||||
--warn-bg: #F6ECD4; --warn-fg: #7A5410;
|
||||
--bad-bg: #F5E0DD; --bad-fg: #8C3128;
|
||||
}
|
||||
@media (prefers-color-scheme: dark) {
|
||||
:root:not([data-theme="light"]) {
|
||||
--ground: #15181B; --surface: #1D2126; --surface-2: #23282E;
|
||||
--ink: #E9E7E1; --muted: #9AA3AC; --line: #343A41;
|
||||
--accent: #E3B93F; --accent-ink: #E3B93F; --steel: #8FB4D2;
|
||||
--ok-bg: #1E3327; --ok-fg: #8FCCA6;
|
||||
--warn-bg: #38301A; --warn-fg: #E0BE6A;
|
||||
--bad-bg: #3A2523; --bad-fg: #E09A92;
|
||||
}
|
||||
}
|
||||
:root[data-theme="dark"] {
|
||||
--ground: #15181B; --surface: #1D2126; --surface-2: #23282E;
|
||||
--ink: #E9E7E1; --muted: #9AA3AC; --line: #343A41;
|
||||
--accent: #E3B93F; --accent-ink: #E3B93F; --steel: #8FB4D2;
|
||||
--ok-bg: #1E3327; --ok-fg: #8FCCA6;
|
||||
--warn-bg: #38301A; --warn-fg: #E0BE6A;
|
||||
--bad-bg: #3A2523; --bad-fg: #E09A92;
|
||||
}
|
||||
|
||||
* { box-sizing: border-box; }
|
||||
html, body { height: 100%; }
|
||||
body {
|
||||
background: var(--ground); color: var(--ink);
|
||||
font-family: Charter, "Bitstream Charter", Cambria, Georgia, serif;
|
||||
margin: 0; overflow: hidden;
|
||||
}
|
||||
h1, h2, .kicker, .btn, .counter, .pill, .cmd, ul.plist {
|
||||
font-family: -apple-system, "Segoe UI", system-ui, "Helvetica Neue", Arial, sans-serif;
|
||||
}
|
||||
|
||||
.slide {
|
||||
display: none; position: absolute; inset: 0;
|
||||
padding: clamp(30px, 4.4vw, 72px) clamp(28px, 7vw, 116px) clamp(56px, 7vh, 88px);
|
||||
flex-direction: column; justify-content: center;
|
||||
}
|
||||
.slide.active { display: flex; }
|
||||
.inner { width: 100%; max-width: 1060px; margin: 0 auto; }
|
||||
|
||||
.kicker {
|
||||
font-size: clamp(0.68rem, 1vw, 0.8rem); font-weight: 650;
|
||||
text-transform: uppercase; letter-spacing: 0.12em;
|
||||
color: var(--accent-ink); margin-bottom: 16px;
|
||||
}
|
||||
h1 { font-size: clamp(2rem, 4.8vw, 3.5rem); font-weight: 650; letter-spacing: -0.02em; line-height: 1.09; margin: 0 0 22px; text-wrap: balance; }
|
||||
h2 { font-size: clamp(1.4rem, 3vw, 2.25rem); font-weight: 650; letter-spacing: -0.015em; line-height: 1.18; margin: 0 0 18px; text-wrap: balance; max-width: 26ch; }
|
||||
|
||||
/* selvbaerende brodtekst - hver slide skal kunne leses alene */
|
||||
.say {
|
||||
font-size: clamp(1rem, 1.55vw, 1.24rem);
|
||||
line-height: 1.6; max-width: 62ch; margin: 0 0 1rem;
|
||||
}
|
||||
.say.dim { color: var(--muted); }
|
||||
.say strong { font-weight: 650; }
|
||||
.say:last-child { margin-bottom: 0; }
|
||||
.lead-sub { font-size: clamp(1.05rem, 1.8vw, 1.4rem); color: var(--muted); max-width: 46ch; line-height: 1.5; }
|
||||
|
||||
code { font-family: ui-monospace, "SF Mono", Menlo, Consolas, monospace; font-size: 0.88em;
|
||||
background: var(--surface-2); border-radius: 3px; padding: 1px 5px; }
|
||||
|
||||
.cmd {
|
||||
background: var(--surface); border: 1px solid var(--line); border-left: 3px solid var(--accent);
|
||||
padding: 20px 26px; margin: 6px 0 20px;
|
||||
font-family: ui-monospace, "SF Mono", Menlo, Consolas, monospace;
|
||||
font-size: clamp(0.82rem, 1.35vw, 1.02rem); line-height: 2.0; overflow-x: auto; white-space: pre;
|
||||
}
|
||||
|
||||
.cards { display: grid; grid-template-columns: repeat(auto-fit, minmax(280px, 1fr)); gap: 16px; margin-top: 6px; }
|
||||
.card { background: var(--surface); border: 1px solid var(--line); border-top: 3px solid var(--accent); padding: 20px 22px; }
|
||||
.card .t { font-family: -apple-system, system-ui, sans-serif; font-weight: 650; font-size: clamp(0.98rem, 1.4vw, 1.1rem); display: block; margin-bottom: 12px; }
|
||||
.card.good { border-top-color: var(--ok-fg); }
|
||||
.card.good .t { color: var(--ok-fg); }
|
||||
.card.risk { border-top-color: var(--bad-fg); }
|
||||
.card.risk .t { color: var(--bad-fg); }
|
||||
ul.plist { list-style: none; padding: 0; margin: 0; display: grid; gap: 11px; }
|
||||
ul.plist li { font-size: clamp(0.86rem, 1.22vw, 1rem); line-height: 1.5; padding-left: 16px; position: relative; }
|
||||
ul.plist li::before { content: "—"; position: absolute; left: 0; color: var(--muted); }
|
||||
ul.plist .dim { color: var(--muted); }
|
||||
|
||||
.foot-note { color: var(--muted); font-size: clamp(0.82rem, 1.12vw, 0.96rem); margin-top: 20px; max-width: 64ch; line-height: 1.55; }
|
||||
|
||||
svg { display: block; max-width: 100%; height: auto; }
|
||||
.fig { margin: 2px 0 16px; }
|
||||
.svg-ink { fill: var(--ink); } .svg-muted { fill: var(--muted); } .svg-accent { fill: var(--accent-ink); }
|
||||
.svg-box { fill: var(--surface); stroke: var(--line); stroke-width: 1.5; }
|
||||
.svg-box-hi { fill: var(--surface-2); stroke: var(--accent); stroke-width: 2.5; }
|
||||
.svg-line { stroke: var(--muted); stroke-width: 1.8; fill: none; }
|
||||
.svg-line-hi { stroke: var(--accent); stroke-width: 2.5; fill: none; }
|
||||
text { font-family: -apple-system, "Segoe UI", system-ui, sans-serif; }
|
||||
|
||||
.bar-ui {
|
||||
position: fixed; left: 0; right: 0; bottom: 0;
|
||||
display: flex; align-items: center; justify-content: space-between;
|
||||
padding: 9px clamp(16px, 3vw, 32px); gap: 16px;
|
||||
border-top: 1px solid var(--line); background: var(--ground);
|
||||
font-family: -apple-system, system-ui, sans-serif; font-size: 0.78rem; color: var(--muted);
|
||||
}
|
||||
.btn { background: var(--surface); border: 1px solid var(--line); color: var(--ink);
|
||||
padding: 5px 13px; font-size: 0.8rem; cursor: pointer; border-radius: 4px; }
|
||||
.btn:hover { border-color: var(--accent); }
|
||||
.counter { font-variant-numeric: tabular-nums; letter-spacing: 0.03em; }
|
||||
.where { font-weight: 600; color: var(--accent-ink); }
|
||||
.nav { display: flex; gap: 10px; align-items: center; }
|
||||
.progress { position: fixed; top: 0; left: 0; height: 3px; background: var(--accent); transition: width 0.18s ease; z-index: 5; }
|
||||
@media print { .bar-ui, .progress { display: none; } }
|
||||
</style>
|
||||
|
||||
<div class="progress" id="prog"></div>
|
||||
|
||||
<!-- ============ 1 · TITTEL ============ -->
|
||||
<section class="slide" data-part="">
|
||||
<div class="inner">
|
||||
<div class="kicker">portfolio-optimiser · åpen kildekode</div>
|
||||
<h1>Pengene inne i porteføljen</h1>
|
||||
<p class="lead-sub">Et åpent rammeverk som leter etter kostnadsbesparelser i hvert enkelt prosjekt — og som ikke får lov til å godkjenne sine egne funn.</p>
|
||||
<p class="foot-note">Tolv sider. Kjøringstallene underveis er hentet fra en eksempelkjøring som følger med i repoet — du kan gjenta den selv, uten kostnad.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- ============ 2 · PROBLEMET ============ -->
|
||||
<section class="slide" data-part="Problemet">
|
||||
<div class="inner">
|
||||
<div class="kicker">Problemet</div>
|
||||
<h2>Besparelsene ligger inne i prosjekter som allerede er vedtatt</h2>
|
||||
<div class="fig">
|
||||
<svg viewBox="0 0 900 150" role="img" aria-label="En rekke vedtatte prosjekter, der noen bærer tiltak ingen har regnet på">
|
||||
<rect class="svg-box" x="6" y="14" width="104" height="76" rx="3"/>
|
||||
<rect class="svg-box" x="118" y="14" width="104" height="76" rx="3"/>
|
||||
<rect class="svg-box" x="230" y="14" width="104" height="76" rx="3"/>
|
||||
<rect class="svg-box" x="342" y="14" width="104" height="76" rx="3"/>
|
||||
<rect class="svg-box" x="454" y="14" width="104" height="76" rx="3"/>
|
||||
<rect class="svg-box" x="566" y="14" width="104" height="76" rx="3"/>
|
||||
<rect class="svg-box" x="678" y="14" width="104" height="76" rx="3"/>
|
||||
<rect class="svg-box" x="790" y="14" width="104" height="76" rx="3"/>
|
||||
<rect x="130" y="58" width="80" height="22" rx="2" fill="var(--warn-bg)" stroke="var(--accent)" stroke-width="1.5"/>
|
||||
<text x="170" y="74" font-size="11.5" font-weight="650" text-anchor="middle" fill="var(--warn-fg)">kr</text>
|
||||
<rect x="466" y="58" width="80" height="22" rx="2" fill="var(--warn-bg)" stroke="var(--accent)" stroke-width="1.5"/>
|
||||
<text x="506" y="74" font-size="11.5" font-weight="650" text-anchor="middle" fill="var(--warn-fg)">kr</text>
|
||||
<rect x="690" y="58" width="80" height="22" rx="2" fill="var(--warn-bg)" stroke="var(--accent)" stroke-width="1.5"/>
|
||||
<text x="730" y="74" font-size="11.5" font-weight="650" text-anchor="middle" fill="var(--warn-fg)">kr</text>
|
||||
<text class="svg-muted" x="6" y="124" font-size="13">vedtatte prosjekter — hvert med sitt budsjett og sine kostlinjer</text>
|
||||
<text class="svg-accent" x="894" y="124" font-size="13" font-weight="650" text-anchor="end">tiltak ingen har regnet på</text>
|
||||
<path class="svg-line-hi" d="M 806 114 C 776 104 748 96 736 86" marker-end="url(#m2)"/>
|
||||
<defs><marker id="m2" markerWidth="8" markerHeight="8" refX="7" refY="4" orient="auto"><path d="M0 0 L8 4 L0 8 z" class="svg-accent"/></marker></defs>
|
||||
</svg>
|
||||
</div>
|
||||
<p class="say">En portefølje er mange uavhengige prosjekter, hvert med sitt budsjett og sine kostlinjer. Erfaringen er at flere av dem rommer tiltak — en utskifting, en smartere styring, en annen metode — som ville spart penger uten å true det prosjektet skal levere.</p>
|
||||
<p class="say dim">Besparelsene står ikke i noen rapport. De må letes fram, kostlinje for kostlinje — og det er letingen som er dyr, ikke tiltakene.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- ============ 3 · HVORFOR DE IKKE HENTES ============ -->
|
||||
<section class="slide" data-part="Problemet">
|
||||
<div class="inner">
|
||||
<div class="kicker">Problemet</div>
|
||||
<h2>Å lete koster tid, folk og fagkompetanse — per prosjekt</h2>
|
||||
<div class="fig">
|
||||
<svg viewBox="0 0 900 156" role="img" aria-label="Én fagperson mot en lang rekke prosjekter som blekner bortover">
|
||||
<circle cx="60" cy="34" r="13" fill="none" stroke="var(--ink)" stroke-width="2"/>
|
||||
<path d="M60 47 V84 M60 58 L40 72 M60 58 L80 72 M60 84 L46 112 M60 84 L74 112" fill="none" stroke="var(--ink)" stroke-width="2" stroke-linecap="round"/>
|
||||
<text class="svg-muted" x="60" y="140" font-size="12.5" text-anchor="middle">én fagperson</text>
|
||||
<g opacity="1"><rect class="svg-box" x="150" y="22" width="72" height="64" rx="3"/></g>
|
||||
<g opacity="0.92"><rect class="svg-box" x="232" y="22" width="72" height="64" rx="3"/></g>
|
||||
<g opacity="0.84"><rect class="svg-box" x="314" y="22" width="72" height="64" rx="3"/></g>
|
||||
<g opacity="0.76"><rect class="svg-box" x="396" y="22" width="72" height="64" rx="3"/></g>
|
||||
<g opacity="0.68"><rect class="svg-box" x="478" y="22" width="72" height="64" rx="3"/></g>
|
||||
<g opacity="0.6"><rect class="svg-box" x="560" y="22" width="72" height="64" rx="3"/></g>
|
||||
<g opacity="0.52"><rect class="svg-box" x="642" y="22" width="72" height="64" rx="3"/></g>
|
||||
<g opacity="0.44"><rect class="svg-box" x="724" y="22" width="72" height="64" rx="3"/></g>
|
||||
<g opacity="0.36"><rect class="svg-box" x="806" y="22" width="72" height="64" rx="3"/></g>
|
||||
<path class="svg-line" d="M150 98 v6 H878 v-6"/>
|
||||
<text class="svg-muted" x="514" y="126" font-size="13" text-anchor="middle">dager med lesing og regning — per prosjekt</text>
|
||||
</svg>
|
||||
</div>
|
||||
<p class="say">Den som skal finne et slikt tiltak, må lese prosjektgrunnlaget, kjenne tiltakstypene og regne på tallene. Det er dager med arbeid for hvert eneste prosjekt — og det krever folk som allerede er opptatt med å levere.</p>
|
||||
<p class="say dim">De fleste leter allerede. Det som mangler er ikke vilje, men en leting som skalerer — over hele porteføljen, igjen og igjen.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- ============ 4 · HVORFOR KI ALENE IKKE LØSER DET ============ -->
|
||||
<section class="slide" data-part="Problemet">
|
||||
<div class="inner">
|
||||
<div class="kicker">Problemet</div>
|
||||
<h2>En språkmodell gir deg gjerne et pent tall ingen har regnet på</h2>
|
||||
<div class="fig">
|
||||
<svg viewBox="0 0 900 184" role="img" aria-label="Et pent notat med et stort beløp, og to ubesvarte spørsmål som peker på det">
|
||||
<rect class="svg-box" x="330" y="10" width="240" height="150" rx="4"/>
|
||||
<rect x="352" y="30" width="180" height="8" rx="2" fill="var(--line)"/>
|
||||
<rect x="352" y="46" width="196" height="8" rx="2" fill="var(--line)"/>
|
||||
<rect x="352" y="62" width="150" height="8" rx="2" fill="var(--line)"/>
|
||||
<text class="svg-ink" x="450" y="112" font-size="24" font-weight="700" text-anchor="middle">«Spar 2,1 mill.»</text>
|
||||
<text class="svg-muted" x="450" y="138" font-size="11.5" text-anchor="middle">grunnlag: ikke vedlagt</text>
|
||||
<text class="svg-muted" x="6" y="66" font-size="13">Hvilke kostlinjer hviler det på?</text>
|
||||
<path class="svg-line" d="M 214 62 H 320" marker-end="url(#m4)"/>
|
||||
<text class="svg-muted" x="894" y="118" font-size="13" text-anchor="end">Er beløpet i det hele tatt oppnåelig?</text>
|
||||
<path class="svg-line" d="M 646 114 H 580" marker-end="url(#m4)"/>
|
||||
<text class="svg-accent" x="450" y="180" font-size="13" font-weight="650" text-anchor="middle">et overbevisende notat er en påstand — ikke en besparelse</text>
|
||||
<defs><marker id="m4" markerWidth="8" markerHeight="8" refX="7" refY="4" orient="auto"><path d="M0 0 L8 4 L0 8 z" class="svg-muted"/></marker></defs>
|
||||
</svg>
|
||||
</div>
|
||||
<p class="say">KI-modeller leser fort og skriver overbevisende — det er nettopp problemet. Et velskrevet notat som lover «spar 2,1 millioner», er ikke en besparelse; det er en påstand. Skal funn brukes i beslutninger, må hvert tall kunne etterprøves.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- ============ 5 · GREPET ============ -->
|
||||
<section class="slide" data-part="Grepet">
|
||||
<div class="inner">
|
||||
<div class="kicker">Grepet</div>
|
||||
<h2>Maskinen får foreslå — aldri godkjenne seg selv</h2>
|
||||
<div class="fig">
|
||||
<svg viewBox="0 0 900 230" role="img" aria-label="Forslaget går til to uavhengige kontroller før det når fagpersonen">
|
||||
<rect class="svg-box" x="6" y="86" width="170" height="64" rx="3"/>
|
||||
<text class="svg-ink" x="91" y="112" font-size="15" font-weight="700" text-anchor="middle">Forslaget</text>
|
||||
<text class="svg-muted" x="91" y="132" font-size="12" text-anchor="middle">fra KI-debatten</text>
|
||||
<path class="svg-line" d="M176 104 C 220 96 230 70 274 64" marker-end="url(#m5)"/>
|
||||
<path class="svg-line" d="M176 132 C 220 140 230 172 274 178" marker-end="url(#m5)"/>
|
||||
<rect x="280" y="20" width="330" height="88" rx="3" fill="var(--warn-bg)" stroke="var(--accent)" stroke-width="2.5"/>
|
||||
<text x="445" y="46" font-size="15" font-weight="700" text-anchor="middle" fill="var(--warn-fg)">Regnekoden — blokkerer</text>
|
||||
<text x="445" y="68" font-size="12" text-anchor="middle" fill="var(--warn-fg)">avstemmer kostlinjene mot prosjektets faktiske tall</text>
|
||||
<text x="445" y="88" font-size="12" text-anchor="middle" fill="var(--warn-fg)">beregner taket for hva som er oppnåelig</text>
|
||||
<rect class="svg-box" x="280" y="140" width="330" height="76" rx="3"/>
|
||||
<text class="svg-ink" x="445" y="168" font-size="15" font-weight="700" text-anchor="middle">Utfordreren</text>
|
||||
<text class="svg-muted" x="445" y="190" font-size="12" text-anchor="middle">en KI-rolle med motsatt oppdrag: angripe begrunnelsen</text>
|
||||
<path class="svg-line" d="M612 64 C 660 64 672 100 700 112" marker-end="url(#m5)"/>
|
||||
<path class="svg-line" d="M612 178 C 660 178 672 142 700 136" marker-end="url(#m5)"/>
|
||||
<rect x="706" y="92" width="188" height="64" rx="3" fill="var(--ok-bg)" stroke="var(--ok-fg)" stroke-width="2.5"/>
|
||||
<text x="800" y="114" font-size="15" font-weight="700" text-anchor="middle" fill="var(--ok-fg)">Fagpersonen</text>
|
||||
<text x="800" y="132" font-size="11.5" text-anchor="middle" fill="var(--ok-fg)">ser bare det som</text>
|
||||
<text x="800" y="147" font-size="11.5" text-anchor="middle" fill="var(--ok-fg)">overlevde begge</text>
|
||||
<defs><marker id="m5" markerWidth="8" markerHeight="8" refX="7" refY="4" orient="auto"><path d="M0 0 L8 4 L0 8 z" class="svg-muted"/></marker></defs>
|
||||
</svg>
|
||||
</div>
|
||||
<p class="say">Hvert forslag må overleve to uavhengige kontroller: en <strong>deterministisk regnekode</strong> som ikke kan overtales, og en <strong>utfordrer</strong> som angriper resonnementet.</p>
|
||||
<p class="say dim">I eksempelkjøringen virker det slik: påstanden på 2 100 000 kr ble avvist fordi taket var 1 769 915 kr. Det korrigerte forslaget — 445 500 kr — slapp gjennom. Avvisningen er ikke en feil; den er systemet som virker.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- ============ 6 · ARKITEKTUREN SOM KRETSLOEP ============ -->
|
||||
<section class="slide" data-part="Slik virker det">
|
||||
<div class="inner">
|
||||
<div class="kicker">Slik virker det</div>
|
||||
<h2>Systemet er et kretsløp med fem deler</h2>
|
||||
<div class="fig">
|
||||
<svg viewBox="0 0 900 282" role="img" aria-label="Fem deler i et kretsløp: kunnskapsbasen, KI-debatten, regnekoden, fagpersonen og porten som skriver godkjente dommer tilbake i basen">
|
||||
<rect class="svg-box-hi" x="8" y="100" width="206" height="76" rx="3"/>
|
||||
<text class="svg-ink" x="111" y="126" font-size="14.5" font-weight="650" text-anchor="middle">Kunnskapsbasen</text>
|
||||
<text class="svg-muted" x="111" y="146" font-size="11" text-anchor="middle">dokumentene fagfolk eier</text>
|
||||
<text class="svg-muted" x="111" y="162" font-size="11" text-anchor="middle">start for hver kjøring</text>
|
||||
|
||||
<rect x="306" y="4" width="288" height="94" rx="3" fill="var(--surface-2)" stroke="var(--line)" stroke-width="1.5"/>
|
||||
<text class="svg-muted" x="450" y="24" font-size="13.5" font-weight="650" text-anchor="middle">KI-debatten</text>
|
||||
<rect class="svg-box" x="318" y="34" width="132" height="54" rx="3"/>
|
||||
<text class="svg-ink" x="384" y="56" font-size="13" font-weight="650" text-anchor="middle">Foreslår</text>
|
||||
<text class="svg-muted" x="384" y="74" font-size="10.5" text-anchor="middle">finner kandidattiltak</text>
|
||||
<rect class="svg-box" x="462" y="34" width="120" height="54" rx="3"/>
|
||||
<text class="svg-ink" x="522" y="56" font-size="13" font-weight="650" text-anchor="middle">Utfordrer</text>
|
||||
<text class="svg-muted" x="522" y="74" font-size="10.5" text-anchor="middle">angriper begrunnelsen</text>
|
||||
|
||||
<rect x="686" y="100" width="206" height="76" rx="3" fill="var(--warn-bg)" stroke="var(--accent)" stroke-width="2.5"/>
|
||||
<text x="789" y="126" font-size="14.5" font-weight="650" text-anchor="middle" fill="var(--warn-fg)">Regnekoden</text>
|
||||
<text x="789" y="146" font-size="11" text-anchor="middle" fill="var(--warn-fg)">avstemmer mot prosjektets tall</text>
|
||||
<text x="789" y="162" font-size="11" text-anchor="middle" fill="var(--warn-fg)">blokkerer resten</text>
|
||||
|
||||
<rect x="560" y="200" width="232" height="72" rx="3" fill="var(--ok-bg)" stroke="var(--ok-fg)" stroke-width="2.5"/>
|
||||
<text x="676" y="226" font-size="14.5" font-weight="650" text-anchor="middle" fill="var(--ok-fg)">Fagpersonen</text>
|
||||
<text x="676" y="246" font-size="11" text-anchor="middle" fill="var(--ok-fg)">ser bare det som overlevde</text>
|
||||
<text x="676" y="262" font-size="11" text-anchor="middle" fill="var(--ok-fg)">godkjenner eller avviser</text>
|
||||
|
||||
<rect class="svg-box-hi" x="222" y="200" width="228" height="72" rx="3"/>
|
||||
<text class="svg-ink" x="336" y="226" font-size="14.5" font-weight="650" text-anchor="middle">Porten</text>
|
||||
<text class="svg-muted" x="336" y="246" font-size="11" text-anchor="middle">bare en godkjent dom</text>
|
||||
<text class="svg-muted" x="336" y="262" font-size="11" text-anchor="middle">skrives inn i basen</text>
|
||||
|
||||
<path class="svg-line" d="M 150 100 C 198 66 240 56 296 60" marker-end="url(#mA)"/>
|
||||
<path class="svg-line" d="M 598 60 C 656 60 700 74 746 96" marker-end="url(#mA)"/>
|
||||
<path class="svg-line" d="M 788 176 C 788 190 782 196 770 197" marker-end="url(#mA)"/>
|
||||
<path class="svg-line" d="M 556 236 H 460" marker-end="url(#mA)"/>
|
||||
<path class="svg-line-hi" d="M 218 236 C 158 236 111 226 111 184" marker-end="url(#mAh)"/>
|
||||
<path class="svg-line-hi" d="M 682 138 C 640 134 610 120 594 102" stroke-dasharray="5 4" marker-end="url(#mAh)"/>
|
||||
<text class="svg-muted" x="674" y="184" font-size="11.5" text-anchor="end">avvist — nytt forsøk</text>
|
||||
|
||||
<text class="svg-muted" x="450" y="136" font-size="12.5" text-anchor="middle">ingenting når fagpersonen uten å ha passert regnekoden</text>
|
||||
<text class="svg-accent" x="450" y="156" font-size="12.5" font-weight="650" text-anchor="middle">ingenting går inn i basen uten en dom</text>
|
||||
|
||||
<defs>
|
||||
<marker id="mA" markerWidth="8" markerHeight="8" refX="7" refY="4" orient="auto"><path d="M0 0 L8 4 L0 8 z" class="svg-muted"/></marker>
|
||||
<marker id="mAh" markerWidth="8" markerHeight="8" refX="7" refY="4" orient="auto"><path d="M0 0 L8 4 L0 8 z" class="svg-accent"/></marker>
|
||||
</defs>
|
||||
</svg>
|
||||
</div>
|
||||
<p class="say">Alt starter i kunnskapsbasen. Derfra leser KI-modellene seg fram til et kandidattiltak og angriper hverandres begrunnelse. Regnekoden avstemmer tallene mot prosjektets egne kostlinjer og slipper bare gjennom det som lar seg bekrefte — fagpersonen får aldri se noe som ikke har passert der.</p>
|
||||
<p class="say dim">De to veiene tilbake er det som lukker kretsen: et avvist forslag går tilbake til modellene for et nytt forsøk, i et begrenset antall runder, og en dom et menneske har felt går gjennom porten og inn i basen igjen. Neste kjøring starter dermed på et litt bedre grunnlag enn den forrige.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- ============ 7 · KUNNSKAPSBASENE ============ -->
|
||||
<section class="slide" data-part="Slik virker det">
|
||||
<div class="inner">
|
||||
<div class="kicker">Slik virker det</div>
|
||||
<h2>Grunnlaget er flere kunnskapsbaser fagfolk kan lese og rette</h2>
|
||||
<div class="fig">
|
||||
<svg viewBox="0 0 900 202" role="img" aria-label="Prosjektets inngang peker til fem kunnskapsbaser, og hver base har sine egne dokumenter">
|
||||
<rect class="svg-box-hi" x="290" y="4" width="320" height="46" rx="3"/>
|
||||
<text class="svg-ink" x="450" y="25" font-size="14" font-weight="650" text-anchor="middle">Prosjektets inngang</text>
|
||||
<text class="svg-muted" x="450" y="42" font-size="11" text-anchor="middle">innholdsfortegnelsen som samler basene</text>
|
||||
<path class="svg-line" d="M450 50 v14"/>
|
||||
<path class="svg-line" d="M86 64 H814"/>
|
||||
<path class="svg-line" d="M86 64 v14" marker-end="url(#m6)"/>
|
||||
<path class="svg-line" d="M268 64 v14" marker-end="url(#m6)"/>
|
||||
<path class="svg-line" d="M450 64 v14" marker-end="url(#m6)"/>
|
||||
<path class="svg-line" d="M632 64 v14" marker-end="url(#m6)"/>
|
||||
<path class="svg-line" d="M814 64 v14" marker-end="url(#m6)"/>
|
||||
<rect class="svg-box" x="6" y="84" width="160" height="56" rx="3"/>
|
||||
<text class="svg-ink" x="86" y="106" font-size="13" font-weight="650" text-anchor="middle">Prosjektdata</text>
|
||||
<text class="svg-muted" x="86" y="122" font-size="10.5" text-anchor="middle">kostlinjer og rammer</text>
|
||||
<text class="svg-muted" x="86" y="134" font-size="10.5" text-anchor="middle">egen for hvert prosjekt</text>
|
||||
<rect class="svg-box" x="188" y="84" width="160" height="56" rx="3"/>
|
||||
<text class="svg-ink" x="268" y="106" font-size="13" font-weight="650" text-anchor="middle">Tiltakstyper</text>
|
||||
<text class="svg-muted" x="268" y="122" font-size="10.5" text-anchor="middle">hva som pleier å virke</text>
|
||||
<text class="svg-muted" x="268" y="134" font-size="10.5" text-anchor="middle">felles fagstoff</text>
|
||||
<rect class="svg-box" x="370" y="84" width="160" height="56" rx="3"/>
|
||||
<text class="svg-ink" x="450" y="106" font-size="13" font-weight="650" text-anchor="middle">Erfaringstall</text>
|
||||
<text class="svg-muted" x="450" y="122" font-size="10.5" text-anchor="middle">tidligere dommer</text>
|
||||
<text class="svg-muted" x="450" y="134" font-size="10.5" text-anchor="middle">vokser for hver runde</text>
|
||||
<rect class="svg-box" x="552" y="84" width="160" height="56" rx="3"/>
|
||||
<text class="svg-ink" x="632" y="106" font-size="13" font-weight="650" text-anchor="middle">Fagkilder</text>
|
||||
<text class="svg-muted" x="632" y="122" font-size="10.5" text-anchor="middle">håndbøker og normtall</text>
|
||||
<text class="svg-muted" x="632" y="134" font-size="10.5" text-anchor="middle">fageier oppdaterer</text>
|
||||
<rect class="svg-box" x="734" y="84" width="160" height="56" rx="3"/>
|
||||
<text class="svg-ink" x="814" y="106" font-size="13" font-weight="650" text-anchor="middle">Metoden</text>
|
||||
<text class="svg-muted" x="814" y="122" font-size="10.5" text-anchor="middle">hvordan det regnes</text>
|
||||
<text class="svg-muted" x="814" y="134" font-size="10.5" text-anchor="middle">fastsatt én gang</text>
|
||||
<path class="svg-line" d="M166 112 H188" stroke-dasharray="5 4"/>
|
||||
<path class="svg-line" d="M348 112 H370" stroke-dasharray="5 4"/>
|
||||
<path class="svg-line" d="M530 112 H552" stroke-dasharray="5 4"/>
|
||||
<path class="svg-line" d="M712 112 H734" stroke-dasharray="5 4"/>
|
||||
<rect class="svg-box" x="38" y="148" width="96" height="8" rx="1"/>
|
||||
<rect class="svg-box" x="38" y="159" width="96" height="8" rx="1"/>
|
||||
<rect class="svg-box" x="38" y="170" width="96" height="8" rx="1"/>
|
||||
<rect class="svg-box" x="220" y="148" width="96" height="8" rx="1"/>
|
||||
<rect class="svg-box" x="220" y="159" width="96" height="8" rx="1"/>
|
||||
<rect class="svg-box" x="220" y="170" width="96" height="8" rx="1"/>
|
||||
<rect class="svg-box" x="402" y="148" width="96" height="8" rx="1"/>
|
||||
<rect class="svg-box" x="402" y="159" width="96" height="8" rx="1"/>
|
||||
<rect class="svg-box" x="402" y="170" width="96" height="8" rx="1"/>
|
||||
<rect class="svg-box" x="584" y="148" width="96" height="8" rx="1"/>
|
||||
<rect class="svg-box" x="584" y="159" width="96" height="8" rx="1"/>
|
||||
<rect class="svg-box" x="584" y="170" width="96" height="8" rx="1"/>
|
||||
<rect class="svg-box" x="766" y="148" width="96" height="8" rx="1"/>
|
||||
<rect class="svg-box" x="766" y="159" width="96" height="8" rx="1"/>
|
||||
<rect class="svg-box" x="766" y="170" width="96" height="8" rx="1"/>
|
||||
<text class="svg-muted" x="450" y="197" font-size="12.5" text-anchor="middle">fem baser, hver med sine egne dokumenter — alle i klartekst, alle mulige å rette</text>
|
||||
<defs><marker id="m6" markerWidth="8" markerHeight="8" refX="7" refY="4" orient="auto"><path d="M0 0 L8 4 L0 8 z" class="svg-muted"/></marker></defs>
|
||||
</svg>
|
||||
</div>
|
||||
<p class="say">Maskinen graver ikke i løse dokumenthauger. Den navigerer <strong>flere kunnskapsbaser</strong> — wikier av korte tekstdokumenter som lenker til hverandre. Ett prosjekt trenger flere av dem samtidig: prosjektdata er prosjektets egne, fagkildene er felles og gjenbrukes fra prosjekt til prosjekt. Hver base har sin egen innholdsfortegnelse, sin egen eier og sin egen oppdateringstakt.</p>
|
||||
<p class="say dim">Alt står i klartekst: en fagperson kan åpne en base, se hva systemet «vet», og rette det som er feil. Prosjektets inngang avgjør hvilke baser som er med — maskinen når ingenting utenfor den. Eksempelet som følger med repoet er lite: der ligger alle fem som dokumenter i én base. I et virkelig prosjekt er hver av dem en base for seg.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- ============ 8 · ARKITEKTUREN ============ -->
|
||||
<section class="slide" data-part="Slik virker det">
|
||||
<div class="inner">
|
||||
<div class="kicker">Slik virker det</div>
|
||||
<h2>Fire lag — og bare det godkjente går tilbake</h2>
|
||||
<div class="fig">
|
||||
<svg viewBox="0 0 900 252" role="img" aria-label="Fire lag fra grunnlag til dom, med en returpil for godkjent kunnskap">
|
||||
<rect x="40" y="10" width="700" height="48" rx="3" fill="var(--surface-2)" stroke="var(--line)" stroke-width="1.5"/>
|
||||
<text class="svg-ink" x="58" y="40" font-size="14.5" font-weight="650">1 · Grunnlaget</text>
|
||||
<text class="svg-muted" x="724" y="40" font-size="12" text-anchor="end">kunnskapsbasen — dokumentene fagfolk eier</text>
|
||||
<rect class="svg-box" x="40" y="72" width="700" height="48" rx="3"/>
|
||||
<text class="svg-ink" x="58" y="102" font-size="14.5" font-weight="650">2 · Forslaget</text>
|
||||
<text class="svg-muted" x="724" y="102" font-size="12" text-anchor="end">KI-modeller foreslår og utfordrer hverandre</text>
|
||||
<rect x="40" y="134" width="700" height="48" rx="3" fill="var(--warn-bg)" stroke="var(--accent)" stroke-width="2"/>
|
||||
<text x="58" y="164" font-size="14.5" font-weight="650" fill="var(--warn-fg)">3 · Kontrollen</text>
|
||||
<text x="724" y="164" font-size="12" text-anchor="end" fill="var(--warn-fg)">regnekoden blokkerer alt den ikke kan bekrefte</text>
|
||||
<rect x="40" y="196" width="700" height="48" rx="3" fill="var(--ok-bg)" stroke="var(--ok-fg)" stroke-width="2"/>
|
||||
<text x="58" y="226" font-size="14.5" font-weight="650" fill="var(--ok-fg)">4 · Dommen</text>
|
||||
<text x="724" y="226" font-size="12" text-anchor="end" fill="var(--ok-fg)">fagpersonen dømmer det som står igjen</text>
|
||||
<path class="svg-line" d="M390 58 V 66" marker-end="url(#m7)"/>
|
||||
<path class="svg-line" d="M390 120 V 128" marker-end="url(#m7)"/>
|
||||
<path class="svg-line" d="M390 182 V 190" marker-end="url(#m7)"/>
|
||||
<path class="svg-line-hi" d="M 744 220 H 806 Q 826 220 826 200 V 54 Q 826 34 806 34 H 750" marker-end="url(#m7a)"/>
|
||||
<text class="svg-accent" x="838" y="118" font-size="12" font-weight="650">godkjent</text>
|
||||
<text class="svg-accent" x="838" y="134" font-size="12" font-weight="650">kunnskap</text>
|
||||
<text class="svg-accent" x="838" y="150" font-size="12" font-weight="650">tilbake</text>
|
||||
<defs>
|
||||
<marker id="m7" markerWidth="8" markerHeight="8" refX="7" refY="4" orient="auto"><path d="M0 0 L8 4 L0 8 z" class="svg-muted"/></marker>
|
||||
<marker id="m7a" markerWidth="8" markerHeight="8" refX="7" refY="4" orient="auto"><path d="M0 0 L8 4 L0 8 z" class="svg-accent"/></marker>
|
||||
</defs>
|
||||
</svg>
|
||||
</div>
|
||||
<p class="say">Grunnlaget mater forslagsarbeidet. Kontrollaget feller alt som ikke tåler etterregning. Fagpersonen dømmer det som står igjen. Og returpilen er poenget: en godkjent dom skrives tilbake i kunnskapsbasen, så neste kjøring starter klokere enn den forrige.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- ============ 9 · DE ÅTTE STEGENE ============ -->
|
||||
<section class="slide" data-part="Slik virker det">
|
||||
<div class="inner">
|
||||
<div class="kicker">Slik virker det</div>
|
||||
<h2>Arbeidsprosessen: åtte steg, med menneskene i begge ender</h2>
|
||||
<div class="fig">
|
||||
<svg viewBox="0 0 900 268" role="img" aria-label="Åtte steg på rad, med bånd som viser hvem som utfører hvert steg">
|
||||
<rect x="6" y="14" width="328" height="34" rx="3" fill="var(--surface-2)" stroke="var(--line)" stroke-width="1.5"/>
|
||||
<text class="svg-muted" x="170" y="36" font-size="14" font-weight="650" text-anchor="middle">KI-modellene arbeider</text>
|
||||
<rect x="342" y="14" width="104" height="34" rx="3" fill="var(--warn-bg)" stroke="var(--accent)" stroke-width="2"/>
|
||||
<text x="394" y="36" font-size="14" font-weight="650" text-anchor="middle" fill="var(--warn-fg)">Regnekoden</text>
|
||||
<rect x="454" y="14" width="216" height="34" rx="3" fill="var(--surface-2)" stroke="var(--line)" stroke-width="1.5"/>
|
||||
<text class="svg-muted" x="562" y="36" font-size="14" font-weight="650" text-anchor="middle">KI + regnekode</text>
|
||||
<rect x="678" y="14" width="216" height="34" rx="3" fill="var(--ok-bg)" stroke="var(--ok-fg)" stroke-width="2"/>
|
||||
<text x="786" y="36" font-size="14" font-weight="650" text-anchor="middle" fill="var(--ok-fg)">Fagpersonen</text>
|
||||
<g font-size="13">
|
||||
<rect class="svg-box" x="6" y="70" width="104" height="76" rx="3"/>
|
||||
<text class="svg-accent" x="58" y="96" font-size="19" font-weight="700" text-anchor="middle">1</text>
|
||||
<text class="svg-muted" x="58" y="122" text-anchor="middle">Les</text>
|
||||
<text class="svg-muted" x="58" y="138" text-anchor="middle">grunnlaget</text>
|
||||
<rect class="svg-box" x="118" y="70" width="104" height="76" rx="3"/>
|
||||
<text class="svg-accent" x="170" y="96" font-size="19" font-weight="700" text-anchor="middle">2</text>
|
||||
<text class="svg-muted" x="170" y="122" text-anchor="middle">Foreslå</text>
|
||||
<text class="svg-muted" x="170" y="138" text-anchor="middle">tiltak</text>
|
||||
<rect class="svg-box" x="230" y="70" width="104" height="76" rx="3"/>
|
||||
<text class="svg-accent" x="282" y="96" font-size="19" font-weight="700" text-anchor="middle">3</text>
|
||||
<text class="svg-muted" x="282" y="122" text-anchor="middle">Utfordre</text>
|
||||
<text class="svg-muted" x="282" y="138" text-anchor="middle">begrunnelsen</text>
|
||||
<rect x="342" y="70" width="104" height="76" rx="3" fill="var(--warn-bg)" stroke="var(--accent)" stroke-width="2.5"/>
|
||||
<text x="394" y="96" font-size="19" font-weight="700" text-anchor="middle" fill="var(--warn-fg)">4</text>
|
||||
<text x="394" y="122" text-anchor="middle" fill="var(--warn-fg)">Regn etter</text>
|
||||
<text x="394" y="138" text-anchor="middle" fill="var(--warn-fg)">— blokkerer</text>
|
||||
<rect class="svg-box" x="454" y="70" width="104" height="76" rx="3"/>
|
||||
<text class="svg-accent" x="506" y="96" font-size="19" font-weight="700" text-anchor="middle">5</text>
|
||||
<text class="svg-muted" x="506" y="122" text-anchor="middle">Forbedre</text>
|
||||
<text class="svg-muted" x="506" y="138" text-anchor="middle">og prøv igjen</text>
|
||||
<rect class="svg-box" x="566" y="70" width="104" height="76" rx="3"/>
|
||||
<text class="svg-accent" x="618" y="96" font-size="19" font-weight="700" text-anchor="middle">6</text>
|
||||
<text class="svg-muted" x="618" y="122" text-anchor="middle">Lever</text>
|
||||
<text class="svg-muted" x="618" y="138" text-anchor="middle">utfallet</text>
|
||||
<rect x="678" y="70" width="104" height="76" rx="3" fill="var(--ok-bg)" stroke="var(--ok-fg)" stroke-width="2.5"/>
|
||||
<text x="730" y="96" font-size="19" font-weight="700" text-anchor="middle" fill="var(--ok-fg)">7</text>
|
||||
<text x="730" y="122" text-anchor="middle" fill="var(--ok-fg)">Døm</text>
|
||||
<text x="730" y="138" text-anchor="middle" fill="var(--ok-fg)">resultatet</text>
|
||||
<rect x="790" y="70" width="104" height="76" rx="3" fill="var(--ok-bg)" stroke="var(--ok-fg)" stroke-width="2.5"/>
|
||||
<text x="842" y="96" font-size="19" font-weight="700" text-anchor="middle" fill="var(--ok-fg)">8</text>
|
||||
<text x="842" y="122" text-anchor="middle" fill="var(--ok-fg)">Lagre det</text>
|
||||
<text x="842" y="138" text-anchor="middle" fill="var(--ok-fg)">godkjente</text>
|
||||
</g>
|
||||
<path class="svg-line-hi" d="M506 152 V 176 Q 506 192 486 192 H 190 Q 170 192 170 176 V 152" marker-end="url(#m8)"/>
|
||||
<text class="svg-accent" x="338" y="212" font-size="13" font-weight="650" text-anchor="middle">avvisningen sendes tilbake — begrenset antall forsøk</text>
|
||||
<path class="svg-line-hi" d="M842 152 V 232 Q 842 250 820 250 H 80 Q 58 250 58 232 V 152" marker-end="url(#m8)"/>
|
||||
<text class="svg-accent" x="450" y="266" font-size="13" font-weight="650" text-anchor="middle">det fagpersonen godkjente blir grunnlag for neste kjøring</text>
|
||||
<defs><marker id="m8" markerWidth="8" markerHeight="8" refX="7" refY="4" orient="auto"><path d="M0 0 L8 4 L0 8 z" class="svg-accent"/></marker></defs>
|
||||
</svg>
|
||||
</div>
|
||||
<p class="say">Grønt er der fagfolkene står. Gult er regnekoden som eier tallene. KI-modellene gjør lese- og skrivearbeidet imellom — og de to pilene tilbake er de eneste veiene noe går i retur.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- ============ 10 · MENNESKENE ============ -->
|
||||
<section class="slide" data-part="Menneskene">
|
||||
<div class="inner">
|
||||
<div class="kicker">Menneskene</div>
|
||||
<h2>Dommen din blir systemets hukommelse</h2>
|
||||
<div class="fig">
|
||||
<svg viewBox="0 0 900 212" role="img" aria-label="En port slipper godkjente dommer inn i kunnskapsbasen og stopper rå maskinoutput">
|
||||
<rect x="6" y="28" width="250" height="56" rx="3" fill="var(--ok-bg)" stroke="var(--ok-fg)" stroke-width="2"/>
|
||||
<text x="131" y="52" font-size="14" font-weight="650" text-anchor="middle" fill="var(--ok-fg)">Dommen din — godkjent</text>
|
||||
<text x="131" y="70" font-size="11.5" text-anchor="middle" fill="var(--ok-fg)">med begrunnelse, sporbar</text>
|
||||
<rect x="6" y="124" width="250" height="56" rx="3" fill="var(--bad-bg)" stroke="var(--bad-fg)" stroke-width="2"/>
|
||||
<text x="131" y="148" font-size="14" font-weight="650" text-anchor="middle" fill="var(--bad-fg)">Rå maskinoutput</text>
|
||||
<text x="131" y="166" font-size="11.5" text-anchor="middle" fill="var(--bad-fg)">forslag ingen har dømt</text>
|
||||
<rect x="452" y="16" width="8" height="58" fill="var(--surface-2)" stroke="var(--accent)" stroke-width="1.5"/>
|
||||
<rect x="452" y="104" width="8" height="88" fill="var(--surface-2)" stroke="var(--accent)" stroke-width="1.5"/>
|
||||
<path class="svg-line-hi" d="M258 56 C 360 58 400 86 468 88 H 634" marker-end="url(#m9)"/>
|
||||
<path class="svg-line" d="M258 152 H 434" marker-end="url(#m9m)"/>
|
||||
<path d="M441 144 l14 16 M455 144 l-14 16" stroke="var(--bad-fg)" stroke-width="2.5" fill="none"/>
|
||||
<rect class="svg-box-hi" x="644" y="60" width="250" height="58" rx="3"/>
|
||||
<text class="svg-ink" x="769" y="84" font-size="14.5" font-weight="650" text-anchor="middle">Kunnskapsbasen</text>
|
||||
<text class="svg-muted" x="769" y="102" font-size="12" text-anchor="middle">neste kjøring starter her</text>
|
||||
<text class="svg-muted" x="450" y="208" font-size="12.5" text-anchor="middle">porten er lukket som utgangspunkt — det som ikke er dømt, kommer ikke inn</text>
|
||||
<defs>
|
||||
<marker id="m9" markerWidth="8" markerHeight="8" refX="7" refY="4" orient="auto"><path d="M0 0 L8 4 L0 8 z" class="svg-accent"/></marker>
|
||||
<marker id="m9m" markerWidth="8" markerHeight="8" refX="7" refY="4" orient="auto"><path d="M0 0 L8 4 L0 8 z" class="svg-muted"/></marker>
|
||||
</defs>
|
||||
</svg>
|
||||
</div>
|
||||
<p class="say">Når du godkjenner eller avviser et utfall, forsvinner ikke vurderingen i et referat. En port skriver den inn i kunnskapsbasen — og porten er lukket for alt annet: rå maskinoutput slipper aldri inn.</p>
|
||||
<p class="say dim">I eksempelkjøringen: første kjøring hentet 0 tidligere dommer for kandidaten. Etter én godkjenning og ett driftsnotat i innboksen hentet neste kjøring 3 — og begge de målte realiseringsgradene (0,79 og 0,66) var på plass i dens hypotesegrunnlag.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- ============ 11 · VERDIEN ============ -->
|
||||
<section class="slide" data-part="Verdien">
|
||||
<div class="inner">
|
||||
<div class="kicker">Verdien</div>
|
||||
<h2>Hva det gir, i størrelsesorden</h2>
|
||||
<div class="fig">
|
||||
<svg viewBox="0 0 900 212" role="img" aria-label="Fire søyler: kostlinja, den avviste påstanden, det beregnede taket og den validerte besparelsen">
|
||||
<text class="svg-muted" x="250" y="30" font-size="13" text-anchor="end">kostlinja tiltaket rører</text>
|
||||
<rect x="260" y="8" width="560" height="34" fill="var(--surface-2)" stroke="var(--line)" stroke-width="1.5"/>
|
||||
<text class="svg-ink" x="812" y="30" font-size="13" font-weight="650" text-anchor="end">4 386 150 kr</text>
|
||||
<text class="svg-muted" x="250" y="80" font-size="13" text-anchor="end">først påstått — avvist</text>
|
||||
<rect x="260" y="58" width="268" height="34" fill="var(--bad-bg)" stroke="var(--bad-fg)" stroke-width="1.5"/>
|
||||
<text x="536" y="80" font-size="13" font-weight="650" fill="var(--bad-fg)">2 100 000 kr</text>
|
||||
<text class="svg-muted" x="250" y="130" font-size="13" text-anchor="end">taket regnekoden fant</text>
|
||||
<rect x="260" y="108" width="226" height="34" fill="var(--warn-bg)" stroke="var(--accent)" stroke-width="1.5"/>
|
||||
<text x="494" y="130" font-size="13" font-weight="650" fill="var(--warn-fg)">1 769 915 kr</text>
|
||||
<text class="svg-muted" x="250" y="180" font-size="13" text-anchor="end">validert besparelse</text>
|
||||
<rect x="260" y="158" width="57" height="34" fill="var(--ok-bg)" stroke="var(--ok-fg)" stroke-width="1.5"/>
|
||||
<text x="325" y="180" font-size="13" font-weight="650" fill="var(--ok-fg)">445 500 kr — 10,2 % av linja</text>
|
||||
<text class="svg-muted" x="450" y="208" font-size="12.5" text-anchor="middle">tallene er fra eksempelkjøringen — utskriften er byte-låst fasit i repoet</text>
|
||||
</svg>
|
||||
</div>
|
||||
<p class="say">Dette er <strong>ett</strong> prosjekt og <strong>én</strong> kostlinje. Verdien av rammeverket er ikke ett enkelt tall, men tre egenskaper: letingen blir nesten gratis per prosjekt, bare etterprøvbare tall når fram til beslutning, og hver dom gjør neste leting bedre.</p>
|
||||
<p class="say dim">Størrelsesorden, ikke løfte: hva din portefølje rommer, vet ingen før letingen er gjort. Det nye er at selve letingen er blitt billig.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- ============ 12 · STATUS ============ -->
|
||||
<section class="slide" data-part="Status">
|
||||
<div class="inner">
|
||||
<div class="kicker">Status</div>
|
||||
<h2>Hva som er bevist — og hva som ikke er</h2>
|
||||
<div class="cards">
|
||||
<div class="card good">
|
||||
<span class="t">Bevist — målt, ikke ment</span>
|
||||
<ul class="plist">
|
||||
<li><strong>Rammeverket har kjørt mot en levende språkmodell</strong> — én kjøring 14. august 2026. Modellen svarte i den formen systemet bestiller, og regneporten avgjorde.</li>
|
||||
<li>Hele sløyfa kjører ende til ende offline, og læringen når fram på begge tidsskalaer. Utskriften er byte-låst som fasit i repoet.</li>
|
||||
<li>Regneporten feller reelt: 2 100 000-påstanden avvises og 445 500 valideres — i hver kjøring, hver gang.</li>
|
||||
<li>869 automatiske tester grønne (5 hoppes over), målt 14. august 2026.</li>
|
||||
<li>Installasjonen er målt, ikke antatt: dokumentert kommando, 65 pakker, exit 0.</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="card risk">
|
||||
<span class="t">Ikke bevist</span>
|
||||
<ul class="plist">
|
||||
<li>Ingen levende modell har fått et forslag <strong>gjennom</strong> regneporten. Den ene kjøringen ble avvist: <code>claimed saving 34500 exceeds P90 feasible 11488</code> — utfordreren godkjente resonnementet, regneporten avviste tallene.</li>
|
||||
<li>Modellen fant opp en kostnadslinje som ikke finnes i kunnskapsbasen. Porten stoppet forslaget — men på beløpet, ikke på at linja var oppdiktet.</li>
|
||||
<li>Om en ekte modell foreslår godt og dømmer klokt, er fortsatt et åpent spørsmål. Ingen målt gevinst fra en virkelig portefølje ennå.</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
<p class="say dim" style="margin-top:18px">Det som mangler, er nøyaktig det en første pilot finnes for å måle. Grunnmuren er bygget for at den målingen skal kunne stoles på.</p>
|
||||
<p class="foot-note">Demo-tallene ellers i presentasjonen er byte-låst fasit fra en skriptet kjøring (<code>tests/golden/demo-transcript.stdout</code>). Tallene fra den levende kjøringen hører bare til denne siden — målt 14. august 2026 mot <code>gpt-4.1-mini</code>, forbruk 15 306 tokens.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<div class="bar-ui">
|
||||
<span>← → for å bla · F for fullskjerm</span>
|
||||
<span class="where" id="where"></span>
|
||||
<span class="nav"><button class="btn" id="prev">◀</button><span class="counter" id="count"></span><button class="btn" id="next">▶</button></span>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
(function () {
|
||||
var slides = Array.prototype.slice.call(document.querySelectorAll('.slide'));
|
||||
var i = 0;
|
||||
var count = document.getElementById('count');
|
||||
var where = document.getElementById('where');
|
||||
var prog = document.getElementById('prog');
|
||||
|
||||
function show(n) {
|
||||
i = Math.max(0, Math.min(slides.length - 1, n));
|
||||
slides.forEach(function (s, k) { s.classList.toggle('active', k === i); });
|
||||
count.textContent = (i + 1) + ' / ' + slides.length;
|
||||
where.textContent = slides[i].getAttribute('data-part') || '';
|
||||
prog.style.width = ((i + 1) / slides.length * 100) + '%';
|
||||
try { location.hash = String(i + 1); } catch (e) {}
|
||||
}
|
||||
function go(d) { show(i + d); }
|
||||
|
||||
document.addEventListener('keydown', function (e) {
|
||||
var k = e.key;
|
||||
if (k === 'ArrowRight' || k === 'ArrowDown' || k === 'PageDown' || k === ' ' || k === 'Enter') { go(1); e.preventDefault(); }
|
||||
else if (k === 'ArrowLeft' || k === 'ArrowUp' || k === 'PageUp' || k === 'Backspace') { go(-1); e.preventDefault(); }
|
||||
else if (k === 'Home') { show(0); e.preventDefault(); }
|
||||
else if (k === 'End') { show(slides.length - 1); e.preventDefault(); }
|
||||
else if (k === 'f' || k === 'F') {
|
||||
if (document.fullscreenElement) { document.exitFullscreen(); }
|
||||
else if (document.documentElement.requestFullscreen) { document.documentElement.requestFullscreen(); }
|
||||
e.preventDefault();
|
||||
}
|
||||
});
|
||||
|
||||
document.getElementById('next').addEventListener('click', function () { go(1); });
|
||||
document.getElementById('prev').addEventListener('click', function () { go(-1); });
|
||||
|
||||
var start = null;
|
||||
document.addEventListener('touchstart', function (e) { start = e.changedTouches[0].clientX; }, {passive: true});
|
||||
document.addEventListener('touchend', function (e) {
|
||||
if (start === null) return;
|
||||
var dx = e.changedTouches[0].clientX - start;
|
||||
if (Math.abs(dx) > 55) { go(dx < 0 ? 1 : -1); }
|
||||
start = null;
|
||||
}, {passive: true});
|
||||
|
||||
var h = parseInt((location.hash || '').replace('#', ''), 10);
|
||||
show(isNaN(h) ? 0 : h - 1);
|
||||
})();
|
||||
</script>
|
||||
623
docs/kunnskapsbase-for-en-kjoring.md
Normal file
623
docs/kunnskapsbase-for-en-kjoring.md
Normal file
|
|
@ -0,0 +1,623 @@
|
|||
# Kunnskapsbasen for én konkret kjøring
|
||||
|
||||
Dette dokumentet er for **fagpersonen** som skal planlegge en kjøring mot et konkret prosjekt, og
|
||||
for **den tekniske personen** som skal bygge kunnskapsbasen den kjøringen leser. Den første jobben
|
||||
er alltid den samme: sette sammen en kunnskapsbase som gir riktig kontekst for *nettopp denne*
|
||||
kjøringen. Det er den jobben som beskrives her — fra bestillingen til en base som er klar til å
|
||||
kjøres.
|
||||
|
||||
Tre dokumenter står rundt dette:
|
||||
|
||||
- [knowledge-base-recipe.md](knowledge-base-recipe.md) *(engelsk)* beskriver **prosessen**: de to
|
||||
rollene, de fire fasene (inventar → skjelett → frø → iterer) og den ærlige forventningen om at en
|
||||
god base tar 1–2 uker. Det gjentas ikke her. Dette dokumentet svarer på det oppskriften ikke
|
||||
svarer på: *hvordan går man fra en gitt oppgave til en riktig sammensatt base?*
|
||||
- [bestille-en-kjoring.md](bestille-en-kjoring.md) beskriver oppdragsfila — det som kommer **inn**
|
||||
til kjøringen. Dette dokumentet møter den i den andre enden: basen oppdraget skal leses mot.
|
||||
- [ekspert-svar.md](ekspert-svar.md) beskriver dommen — det som kommer **ut**, og som gjør neste
|
||||
base bedre.
|
||||
|
||||
> **Merkekonvensjon.** Hver teknisk påstand i dokumentet er enten **VERIFISERT** (med filen,
|
||||
> linjen eller kommandoen som viser det — samlet i [verifiseringsloggen](#8-verifiseringslogg)
|
||||
> nederst) eller **ANTATT** (en tolkning eller en anbefaling som ikke følger av noe målt).
|
||||
> Konvensjoner som bare er observert i eksempelbasene, ikke normert i spesifikasjonen, er merket
|
||||
> **KONVENSJON**. Ingenting her er en påstand om at en levende modell ville resonnert godt over
|
||||
> basen; det er en påstand om hva basen må inneholde for at kjøringen skal kunne starte, lese
|
||||
> riktig og dømme mot ekte tall.
|
||||
|
||||
## Det korte svaret
|
||||
|
||||
**1. Hvilke kategorier trengs?** Fire, og de skiller seg på *hva de følger*:
|
||||
|
||||
| Kategori | Følger | Hvem eier den |
|
||||
|---|---|---|
|
||||
| **Prosjektlaget** | prosjektet / anlegget | prosjekteier og driftsorganisasjon |
|
||||
| **Faglaget** | fagområdet (veglys, tunnel, bygg …) | fagmiljøet |
|
||||
| **Erfaringslaget** | organisasjonen, over tid | fagekspertene som avgir dommer |
|
||||
| **Kjøringslaget** | denne ene bestillingen | bestilleren |
|
||||
|
||||
Det er **én kunnskapsbase per prosjekt** (VERIFISERT: `shared/method-spec.md` §2). Prosjekt-,
|
||||
fag- og erfaringslaget ligger *i* basen. Kjøringslaget ligger *ved siden av* — det er instruksen
|
||||
for én kjøring og lagres aldri som kunnskap. Faglaget deles logisk på tvers av prosjekter i samme
|
||||
fagområde, men **kopieres fysisk inn i hver base** — se [§2](#2-kategoriene-hva-følger-hva).
|
||||
|
||||
**2. Hvilke innholdstyper?** Seks markdown-typer og to tallfiler. Tabellen i
|
||||
[§4](#4-innholdstypene) sier for hver av dem hvem som eier den, hvilken form fagpersonen leverer
|
||||
den i, hva loopen bruker den til, og hva som skjer hvis den mangler. Den viktigste raden er
|
||||
**`cost-baseline.json`**: mangler den, starter kjøringen uten et ord — og validatoren dømmer da bare
|
||||
mot tall forslaget selv oppga (VERIFISERT ved kjøring, [§4.1](#41-den-skarpeste-mangelen-cost-baselinejson)).
|
||||
|
||||
**3. Hvordan ser det ut for et veiprosjekt?** [§5](#5-veiprosjektet-fylkesveg-sør-fra-bestilling-til-kjøreklar-base)
|
||||
går gjennom en veglysportefølje langs fylkesveg, fra oppdragsfila til en base som består
|
||||
kjøreklar-sjekken. Basen det ender i finnes og kjører (VERIFISERT: `shared/examples/veglys-fv-soer/`
|
||||
er den demoen bruker).
|
||||
|
||||
## 1. Hva kjøringen leser, og hvorfor det avgjør hva basen må inneholde
|
||||
|
||||
Hva basen må inneholde er ikke en smakssak. Det følger av hvordan loopen leser den.
|
||||
|
||||
**Kjøringen navigerer, den søker ikke.** Den starter i `index.md`, følger lenkene derfra, og
|
||||
leser hver fil den når. Det som står i brødteksten til `index.md` kommer **først og ordrett** inn
|
||||
i det agentene leser; deretter kommer hver lenket fil som en egen seksjon `## <type>: <tittel>`
|
||||
med hele brødteksten. Det finnes ingen stikkordssøk og ingen «relevante utdrag» — en fil leses
|
||||
helt, eller ikke i det hele tatt (VERIFISERT: `shared/method-spec.md` §3 Steg 1;
|
||||
`src/portfolio_optimiser/okf.py` `bundle_context`). Tre konsekvenser for den som bygger basen:
|
||||
|
||||
- **En fil uten lenke fra `index.md` finnes ikke** for kjøringen. Det gjelder også en fil som
|
||||
ligger i riktig mappe med riktig type (VERIFISERT: `shared/ingest-spec.md` §2: «a generated
|
||||
concept file without an index link is unreachable»).
|
||||
- **En lenke ut av basen følges aldri.** Målet må ligge inne i basens egen mappe; en lenke til en
|
||||
nabobase eller et delt fagbibliotek hoppes stille over, uten feilmelding (VERIFISERT:
|
||||
method-spec §3 Steg 1, «a target resolving outside the bundle is skipped»). Det er grunnen til
|
||||
at faglaget kopieres inn, ikke lenkes til.
|
||||
- **Mengde er ikke kvalitet.** Alt som lenkes leses i sin helhet, så ti sider støy koster like mye
|
||||
oppmerksomhet som ti sider substans. Hold filene stramme (ANTATT: en anbefaling, ikke et krav i
|
||||
koden).
|
||||
|
||||
**Én type holdes alltid utenfor lesekonteksten: `type: verdict`.** Tidligere dommer kommer aldri
|
||||
inn som vanlig tekst. De hentes for seg, nøklet på *kandidaten*, og legges foran hypoteseprompten
|
||||
som «tidligere dommer om lignende tiltak» (VERIFISERT: method-spec §3 Steg 1 «Experience fold»;
|
||||
`okf.py` `context_files` ekskluderer typen). Kandidaten de nøkles på leses fra
|
||||
`validator-input.json` — ikke fra bestillingen, og ikke fra hva agentene ender med å foreslå
|
||||
(VERIFISERT: `src/portfolio_optimiser/run.py:634-638`). Det er derfor den fila er påkrevd.
|
||||
|
||||
**Ved siden av markdown-filene ligger to tallfiler.** `validator-input.json` må finnes — uten den
|
||||
nekter kjøringen å starte (VERIFISERT ved kjøring, [§4.1](#41-den-skarpeste-mangelen-cost-baselinejson)).
|
||||
`cost-baseline.json` er valgfri, og det er nettopp problemet: uten den starter kjøringen som om alt
|
||||
var i orden.
|
||||
|
||||
**Målt på veglys-basen:** mappa har 9 filer. Én er `index.md`, én er dommen, to er tallfiler —
|
||||
og **5** er det kjøringen faktisk navigerer inn som kontekst (VERIFISERT:
|
||||
`tests/golden/demo-transcript.stdout` linje 13, «navigerte konseptfiler (5)»).
|
||||
|
||||
> **Teknisk note.** Formatet basen bygges i heter Open Knowledge Format (OKF); i koden og
|
||||
> spesifikasjonene kalles basen en *bundle*. Hver fil har en frontmatter (`---`-blokk) der
|
||||
> `type:` er det ene påkrevde feltet. Koden leser svært få av de andre feltene: `title` (til
|
||||
> seksjonsoverskriften og prosjektnavnet), `dimension` (når kjøringen er avgrenset til én
|
||||
> kostakse), og på dom-filer `decision`, `description`, `realization_rate`,
|
||||
> `expected_actual_saving_nok` samt den valgfrie nøkkel-trioen `affected_codes` /
|
||||
> `measure_type` / `claimed_saving_nok`. Felt som `resource`, `measure_id`, `tags`, `gap_source`
|
||||
> og `okf_version` i eksemplene leses **ikke** av koden — de er dokumentasjon for mennesker
|
||||
> (VERIFISERT: `grep` over `src/portfolio_optimiser/` gir null treff på dem). De er likevel verdt
|
||||
> å skrive: de gjør basen lesbar i Obsidian/VS Code, som er lesegrensesnittet i dag.
|
||||
|
||||
## 2. Kategoriene: hva følger hva
|
||||
|
||||
| Kategori | Innhold | Følger | Gjenbruk på tvers av prosjekter |
|
||||
|---|---|---|---|
|
||||
| **Prosjektlaget** | `index.md` · `type: project` · `type: hypothesis` · `validator-input.json` · `cost-baseline.json` | prosjektet | Nei. Skrives per prosjekt, fra prosjektets egne tall. |
|
||||
| **Faglaget** | `type: methodology` · `type: reference` | fagområdet | Logisk ja, fysisk nei: **kopieres inn** i hver base og tilpasses. |
|
||||
| **Erfaringslaget** | `type: verdict` (frø-dommer og promoterte dommer) · dom-innboksen utenfor basen | organisasjonen, over tid | Ja, gjennom dommene: en dom om et lignende tiltak i et annet prosjekt hentes inn når kandidatene ligner strukturelt. |
|
||||
| **Kjøringslaget** | oppdragsfila (`--mandate`) · dimensjonsavgrensning (`--dimension-config`) · eksterne tjenester (`--mcp-config`) · tak (runder/tokens) | denne ene bestillingen | Nei. Instruks for én kjøring; «den lagrer seg ikke som kunnskap» ([bestille-en-kjoring.md](bestille-en-kjoring.md)). |
|
||||
|
||||
**Deles metode- og litteraturlaget på tvers?** Svaret er todelt, og begge halvdeler er målt.
|
||||
|
||||
*Logisk* er det samme fagstoff: alle tre eksempelbasene (kontorbygg, veglys, tunnel) bærer en
|
||||
`metode-ipmvp-a.md` og en `kilder-*.md`, og alle tre bygger på samme M&V-rammeverk (IPMVP Option
|
||||
A). *Fysisk* er det tre ulike filer: 40, 81 og 98 linjer, med hver sin tittel — «for veglys — og
|
||||
hvorfor de andre opsjonene er stengt», «for tunnelstyring — anlegget måler inngangssignalet, ikke
|
||||
energien» (VERIFISERT: `wc -l` + `diff` over de tre). Begge veiprosjekt-basene sier det selv i
|
||||
`index.md`: «metode- og kildelaget er **materialisert inn her**, ikke lenket på tvers av bundler».
|
||||
|
||||
Grunnen er teknisk og ufravikelig: navigasjonen følger aldri en lenke ut av basen
|
||||
([§1](#1-hva-kjøringen-leser-og-hvorfor-det-avgjør-hva-basen-må-inneholde)). Men det er også
|
||||
faglig riktig: metoden *for veglys* er ikke metoden *for tunnel*. I veglys er ex-post stengt fordi
|
||||
anlegget mangler måler; i tunnel er ex-ante stengt fordi anlegget ble bygget før noen målte
|
||||
(VERIFISERT: de to `index.md`-filene). En delt fil ville måttet si begge deler og dermed ingen av
|
||||
dem.
|
||||
|
||||
Praktisk betyr det (ANTATT, anbefaling): fagmiljøet eier en **mal** per fagområde; hver base får
|
||||
en **tilpasset kopi**; når malen endres, er det en kjent jobb å gå gjennom kopiene. To kopier
|
||||
drifter — det er prisen, og den skal være uttalt, ikke skjult.
|
||||
|
||||
**Prosjektlaget er det som byttes ut.** Begge veiprosjekt-basene er bygget med fiktivt prosjektlag
|
||||
og ekte litteraturlag, og sier selv hvordan de er ment brukt: «en produksjons-deployer erstatter
|
||||
prosjektlaget med en ekte kunnskapsbase og beholder litteraturlaget» (VERIFISERT: begge
|
||||
`index.md`). Det er nøyaktig kategoriskillet over, skrevet fra eksemplenes side.
|
||||
|
||||
**Erfaringslaget er reservert.** Ingen automatisk kilde kan skrive en `type: verdict`-fil inn i
|
||||
basen; den eneste veien dit er en promotering av en dom et menneske har godkjent, eller en
|
||||
håndskrevet frø-dom (VERIFISERT: `shared/ingest-spec.md` §3, «The verdict layer is RESERVED»).
|
||||
Grunnen står i spesifikasjonen: hver dom-fil i basen leses som en *godkjent* dom, så en
|
||||
datakilde som kunne skrive dem ville sprøytet maskingenererte «godkjenninger» rett inn i
|
||||
læringen.
|
||||
|
||||
### Hvem eier hva
|
||||
|
||||
| Rolle | Leverer | Leverer ikke |
|
||||
|---|---|---|
|
||||
| **Prosjekteier / drift** | prosjektlaget: anleggsdata, kostlinjer, rammer, kandidat-tiltak | fag-metoden, dommene |
|
||||
| **Fagmiljøet** | faglaget: målemetode, verifisert litteratur, kjente gap-mekanismer | prosjekttall |
|
||||
| **Fageksperten(e)** | dommer — frø-dommer før første kjøring, innboks-dommer etter | innhold i basen direkte (de skriver innboksen; systemet leser) |
|
||||
| **Bestilleren** | oppdragsfila: hensikt, tilnærminger, suksesskriterium | tall i basen; bestillingen styrer hva som vurderes, aldri hva som godkjennes |
|
||||
| **Den tekniske personen** | bygger basen fra det de andre leverer: markdown, lenker, tallfilene, ev. ingest-manifest | fagvurderinger |
|
||||
|
||||
Fagpersonene leverer **filer i sine egne formater** — regneark, notater, PDF-er — aldri JSON eller
|
||||
skjema ([knowledge-base-recipe.md](knowledge-base-recipe.md), «Who does what»). Oversettelsen til
|
||||
basens format er den tekniske personens jobb, og i dag er den manuell
|
||||
([§6](#6-ærlighetsdelen-hva-som-ikke-finnes-i-dag)).
|
||||
|
||||
## 3. Avgjørelsesdelen: spørsmålene du stiller fagpersonen
|
||||
|
||||
Hvert spørsmål avgjør én konkret del av basen. Still dem i denne rekkefølgen; svaret på de fire
|
||||
første avgjør om kjøringen i det hele tatt kan starte.
|
||||
|
||||
**1. Hvilket prosjekt eller anlegg gjelder det, og hva skal det hete?**
|
||||
Svaret blir prosjekt-ID-en. Den må være identisk tre steder — kommandolinjen,
|
||||
`validator-input.json` og `cost-baseline.json` — ellers nektes kjøringen (VERIFISERT:
|
||||
`run.py` `_project_from_bundle`, «bundle project_id … != requested»). Velg en ID uten mellomrom og
|
||||
æøå; eksemplene bruker formen `VEGLYS-FV-SOER` (KONVENSJON).
|
||||
*Hvis svaret er «flere anlegg»:* én base per prosjekt. En portefølje er flere baser, kjørt i
|
||||
porteføljemodus.
|
||||
|
||||
**2. Hvilke kostlinjer kan tiltakene komme til å treffe — og har vi faktiske tall for dem?**
|
||||
Svaret blir `cost-baseline.json`: én linje per kostkode, med mengde og enhetspris. Hver kostkode
|
||||
et forslag viser til må finnes her, og mengden og prisen forslaget oppgir må ligge innenfor
|
||||
toleransen (`BASELINE_TOLERANCE_DEFAULT = 0.05`, altså 5 %) rundt baselinens verdi — ellers
|
||||
avvises forslaget (VERIFISERT: `src/portfolio_optimiser/validator.py` `_reconcile_against_baseline`).
|
||||
Derfor må fila dekke *alle* linjene bestillingens tilnærminger kan treffe, ikke bare den ene
|
||||
kandidaten som er projisert.
|
||||
*Hvis svaret er «vi har ikke tallene»:* kjøringen går likevel, og det er faren — se
|
||||
[§4.1](#41-den-skarpeste-mangelen-cost-baselinejson). Skaff tallene, eller skriv inn i bestillingens
|
||||
suksesskriterium at kjøringen er uforankret, så ingen leser et VALIDATED-utfall som mer enn det
|
||||
er (ANTATT: en arbeidsregel, ikke noe koden håndhever).
|
||||
|
||||
**3. Hvilket tiltak er «det ene» vi projiserer?**
|
||||
Basen holder én kandidat i `validator-input.json`: tiltaket, kostlinjene det virker på, den
|
||||
modellerte besparelsen og usikkerhetsbåndet. Denne kandidaten er **nøkkelen til tidligere
|
||||
dommer** — det er dens kostkoder, tiltakstype og beløpsstørrelse som avgjør hvilke dommer fra
|
||||
tidligere kjøringer som hentes inn (VERIFISERT: method-spec §3 Steg 1, rangeringsformelen;
|
||||
`run.py:634-638`). Den er ikke det som valideres i kjøringen — det er forslaget agentene lager
|
||||
(VERIFISERT: `generate.py:491` validerer kandidaten fra modellen).
|
||||
*Hvis fagpersonen vil ha flere tiltak vurdert:* det går gjennom bestillingen (spørsmål 5), og
|
||||
hvert tiltak bør ha sin egen `type: hypothesis`-fil i basen. Men nøkkelen til tidligere dommer er
|
||||
fortsatt den ene projiserte kandidaten — se
|
||||
[§6](#6-ærlighetsdelen-hva-som-ikke-finnes-i-dag).
|
||||
|
||||
**4. Hvilke harde rammer gjelder?**
|
||||
Minstekrav som setter et gulv ingen besparelse kan gå under, ting som ikke kan endres, budsjett-
|
||||
og anskaffelsesrammer. Svaret skrives inn i `type: project`-fila. Veglys-eksempelet har fire:
|
||||
lystekniske minstekrav, vedlikeholdsfaktoren, at nattslukking ikke kan antas, og at tiltak
|
||||
vurderes inne i porteføljen (VERIFISERT: `shared/examples/veglys-fv-soer/veglys-fv-soer.md`,
|
||||
«Rammer»). Agentene leser dem som tekst; koden håndhever dem ikke (ANTATT: det følger av at
|
||||
koden bare leser `title` fra fila, men er ikke målt mot en levende modell).
|
||||
|
||||
**5. Hvilke tilnærminger vil du ha vurdert, og hvorfor?**
|
||||
Svaret blir oppdragsfila — se [bestille-en-kjoring.md](bestille-en-kjoring.md). `description`-feltet
|
||||
mates ordrett inn til modellen; det er der fagkunnskapen om *hvorfor* tiltaket er verdt å prøve
|
||||
gjør en forskjell. Hver tilnærming bør ha et motstykke i basen: en `type: hypothesis`-fil med
|
||||
parametere, modellert besparelse og kjent usikkerhet (KONVENSJON: begge veiprosjekt-basene har
|
||||
én hypothesis-fil per kandidat-tiltak; koden krever det ikke).
|
||||
|
||||
**6. Hvordan måles og verifiseres en besparelse i dette faget?**
|
||||
Svaret blir `type: methodology`-fila. Den forteller agentene *hvorfor* modellert og faktisk
|
||||
besparelse kan sprike, og hvilken måleopsjon som er åpen for dette anlegget. For veglys er svaret
|
||||
«IPMVP Option A, ved eliminasjon» fordi anlegget mangler måler (VERIFISERT: veglys
|
||||
`metode-ipmvp-a.md`).
|
||||
*Hvis fagmiljøet har en mal:* kopier og tilpass. Tilpasningen er ikke pynt — den delen som
|
||||
forklarer hvilke opsjoner som er *stengt for dette anlegget* er prosjektspesifikk.
|
||||
|
||||
**7. Hva vet litteraturen om gapet mellom modellert og faktisk besparelse her?**
|
||||
Svaret blir `type: reference`-fila. Skill skarpt mellom det som er målt i vårt eget land/regime
|
||||
og det som er lånt fra andre program — veglys-eksempelet deler fila i «Del A — norsk materiale»
|
||||
og «Del B — lånt materiale», og sier hvorfor: «Å blande de to ville gjort et lånt tall til en
|
||||
norsk måling» (VERIFISERT: `kilder-veglys-realisering.md`).
|
||||
*Hvis svaret er «det finnes ingen norsk måling»:* skriv det. Et navngitt evidenshull er innhold;
|
||||
et oppdiktet tall er forurensning.
|
||||
|
||||
**8. Finnes det tidligere erfaring med lignende tiltak — en dom noen faktisk har avgitt?**
|
||||
Svaret blir en frø-dom: en `type: verdict`-fil med `decision`, en `description` som bærer
|
||||
begrunnelsen, og der det finnes, `realization_rate` og `expected_actual_saving_nok`. Det er disse
|
||||
feltene som når neste hypotese (VERIFISERT: `verdicts.py` `_verdict_rationale`).
|
||||
*Hvis svaret er nei:* det er lovlig å starte med tom erfaring. Demoen starter slik — «fersk
|
||||
kunnskapsbase, ingen tidligere dommer» (VERIFISERT: golden-transkriptet linje 11 og 14). Den
|
||||
første ekte dommen kommer etter første kjøring, via innboksen.
|
||||
|
||||
**9. Skal kjøringen avgrenses til én kostakse?**
|
||||
Hvis bestillingen gjelder for eksempel *energi* og ikke *vedlikehold*, settes en
|
||||
dimensjonsavgrensning (`--dimension-config`). Da leser agentene bare filer som er merket med den
|
||||
dimensjonen eller er umerket, og et forslag utenfor aksen avvises (VERIFISERT: `run.py:517-519`,
|
||||
`728-740`; `okf.py` `bundle_context`). Filene merkes med `dimension:` i frontmatter.
|
||||
*Hvis svaret er nei:* la det være. Ingen av eksempelbasene bruker `dimension:` (VERIFISERT:
|
||||
`grep` gir null treff under `shared/examples/`).
|
||||
|
||||
**10. Ligger kildene som data — regneark, database, tjeneste — eller som dokumenter?**
|
||||
Tabelldata kan hentes inn automatisk med et **ingest-manifest**: én JSON-fil som kobler en kilde
|
||||
(CSV-katalog, SQL-database, HTTP-endepunkt eller MCP-tjeneste) til basen, og skriver én
|
||||
markdown-fil per uttrekk (`ingest-<id>.md`) med provenance-stempel og lenke i `index.md`
|
||||
(VERIFISERT: `shared/ingest-spec.md` §2, §4–§7). Manifestet kan aldri skrive en dom-fil, og det
|
||||
kan aldri gi seg selv nettverkstilgang — den er et kjøreargument (VERIFISERT: §3, §8).
|
||||
*Det ingest IKKE gjør:* skrive `validator-input.json` eller `cost-baseline.json`. Begge er
|
||||
håndarbeid (VERIFISERT: [extending.md](extending.md), «An ingested bundle is not yet a runnable
|
||||
bundle»). Dokumenter i fri form (PDF, notat) oversettes manuelt til markdown i dag.
|
||||
|
||||
**11. Hvem skal avgi dommen etterpå — og etter hvilke kostkoder?**
|
||||
Svaret blir en innboks-mappe (`--verdict-dir`) eksperten skriver i, og eventuelt en rutingtabell
|
||||
som sier hvilken ekspert som får hvilke forslag, etter kostkode-prefiks (VERIFISERT:
|
||||
`src/portfolio_optimiser/hitl.py`, `route`). Innboksen skal være en annen mappe enn utboksen
|
||||
kjøringen skriver til — de har motsatt eierskap ([ekspert-svar.md](ekspert-svar.md)).
|
||||
|
||||
## 4. Innholdstypene
|
||||
|
||||
| Innhold | Eies av | Fagpersonen leverer | Brukes i loopen til | Hvis den mangler |
|
||||
|---|---|---|---|---|
|
||||
| `index.md` | teknisk person, tekst fra prosjekteier | et avsnitt om hva prosjektet er og hvorfor dette fagområdet er valgt | brødteksten er det **første** agentene leser, ordrett; lenkene bestemmer hva som ellers leses | kjøringen nektes — «a bundle has no entry point without it» (VERIFISERT: method-spec §3 Steg 1) |
|
||||
| `type: project` | prosjekteier / drift | anleggsregister, energibaseline, rammer — regneark og notat | kontekst: hva anlegget er, tallene det består av, gulvet ingen besparelse kan gå under | kjøringen starter; prosjektnavnet faller tilbake til ID-en (VERIFISERT: `run.py` `_project_from_bundle`); agentene resonnerer uten rammer (ANTATT) |
|
||||
| `type: hypothesis` | prosjekteier + fagmiljø | kandidat-tiltak med parametere, modellert besparelse, hva som er usikkert, hva som er utledet | kontekst: forslagene agentene forbedrer i stedet for å finne opp; «Mapping til validatoren» forklarer tallfilene | kjøringen starter; forslagene står på oppdragsfilas `description` alene (ANTATT) |
|
||||
| `type: methodology` | fagmiljøet | målemetode (M&V), hvilke opsjoner som er åpne og stengt for dette anlegget — notat eller mal | kontekst: hvorfor modellert ≠ faktisk, og hva som kan verifiseres etterpå | kjøringen starter; gap-resonnementet mangler i konteksten (ANTATT) |
|
||||
| `type: reference` | fagmiljøet | verifisert litteratur med kilde og årstall; eget materiale skilt fra lånt | kontekst: tallgrunnlaget for realiseringsgrad og gap-mekanismer | kjøringen starter; en realiseringskorreksjon har ingenting å forankres i (ANTATT) |
|
||||
| `type: verdict` (frø) | fageksperten | en tidligere dom: beslutning, begrunnelse, realiseringsgrad, forventet faktisk besparelse | **ikke** lest som kontekst; hentes som «tidligere dom» og legges foran hypoteseprompten | lovlig: første kjøring har null tidligere dommer (VERIFISERT: golden linje 14) |
|
||||
| `validator-input.json` | teknisk person, tall fra prosjekteier | den ene projiserte kandidaten: kostlinjer, modellert besparelse, usikkerhetsbånd | prosjekt-ID-sjekk ved oppstart; nøkkelen tidligere dommer hentes med | **kjøringen nektes**, rc 1: «IR projection not found in bundle» (VERIFISERT ved kjøring, §4.1) |
|
||||
| `cost-baseline.json` | prosjekteier / drift | prosjektets faktiske kostlinjer: kode → mengde × enhetspris | validatorens **steg 0**: hvert forslag avstemmes mot ekte linjer FØR noe annet regnes | **kjøringen starter, rc 0, uten melding** — gaten er uforankret (VERIFISERT ved kjøring, §4.1). Finnes fila men er feilformet: kjøringen nektes (VERIFISERT) |
|
||||
| oppdragsfila | bestilleren | hensikt, tilnærminger med begrunnelse, suksesskriterium | hva som vurderes; kunngjøring før og oppgjør etter | kjøringen går uten bestilling: systemet foreslår selv; ingen kunngjøring, intet oppgjør per tilnærming |
|
||||
| dom-innboksen | fageksperten | én JSON-fil per dom, etter kjøringen | merges inn før neste hypotese; lærer systemet noe | lovlig: ingen læring mellom kjøringene |
|
||||
| dimensjonskonfig | bestilleren | hvilken kostakse kjøringen gjelder | avgrenser lesekontekst og avviser forslag utenfor aksen | lovlig: hele basen leses, ingen avvisning på akse |
|
||||
| ingest-manifest | teknisk person | — (kobler en datakilde fagpersonen eier til basen) | skriver `ingest-*.md` med provenance før kjøringen | lovlig: alt er håndkuratert |
|
||||
| MCP-konfig | bestiller + teknisk person | hvilke interne tjenester agentene får spørre, med navngitte verktøy | oppslag under debatten; alt som kan kontaktes står i kunngjøringen | lovlig: null nettverkskall (VERIFISERT: [bestille-en-kjoring.md](bestille-en-kjoring.md)) |
|
||||
|
||||
De fire typene `project`, `hypothesis`, `methodology` og `reference` er **KONVENSJON**: de står
|
||||
i alle tre eksempelbasene, men ingen av de to spesifikasjonene nevner dem (VERIFISERT: `grep`
|
||||
over `shared/method-spec.md` og `shared/ingest-spec.md` gir null treff). Spesifikasjonen normerer
|
||||
bare at `type` er påkrevd, at `index.md` er inngangen, og at `verdict` er reservert. En base kan
|
||||
bruke andre typenavn; de renderes som `## <type>: <tittel>` uansett. Hold dere til de fire likevel
|
||||
— lesbarhet på tvers av baser er verdt mer enn friheten (ANTATT).
|
||||
|
||||
### 4.1 Den skarpeste mangelen: `cost-baseline.json`
|
||||
|
||||
Før denne fila fantes resonnerte hver del av validatoren bare om tall forslaget selv oppga, så
|
||||
«an internally consistent hallucination passes» (VERIFISERT: `README.md`, «How it is set up»).
|
||||
Fila forankrer gaten i prosjektets ekte kostlinjer: hvert forslag avstemmes mot dem i et steg 0
|
||||
*før* løseren, og to ting avvises — en kostkode prosjektet ikke har, og en ekte kode med mengde
|
||||
eller enhetspris utenfor 5 % av baselinens verdi (VERIFISERT: `validator.py:154-190`, `:210-214`).
|
||||
|
||||
Men fila er **valgfri** på bundle-stien, og fraværet er stille. Målt 2026-08-21 med
|
||||
`--live-dry-run` på fire kopier av veglys-basen, samme kommando, samme oppdragsfil:
|
||||
|
||||
| Variant | Utfall | rc | Melding |
|
||||
|---|---|---|---|
|
||||
| intakt | `LIVE-DRY-RUN OK` | 0 | — |
|
||||
| uten `validator-input.json` | nektet | 1 | `live-dry-run refused: IR projection not found in bundle: 'validator-input.json'` |
|
||||
| **uten `cost-baseline.json`** | **`LIVE-DRY-RUN OK`** | **0** | **`Cost baseline: NONE in the bundle — this run is un-anchored: …`** |
|
||||
| korrupt `cost-baseline.json` (`quantity: -1`, `unit_cost` mangler) | nektet | 1 | `live-dry-run refused: 2 validation errors for CostBaseline …` |
|
||||
|
||||
Kjeden som gir det utfallet: `okf.load_optional_cost_baseline` returnerer `None` når fila mangler
|
||||
(VERIFISERT: `okf.py:323-335`), `run.py:516` tar imot `None`, og `validate_proposal` hopper over
|
||||
steg 0 når baselinen er `None` (VERIFISERT: `validator.py:213`). Det er med vilje — en base skrevet
|
||||
før fila fantes skal fortsatt kjøre byte-identisk (VERIFISERT: `CLAUDE.md`, S4.0-raden).
|
||||
|
||||
**Fraværet er ikke lenger stille (2026-08-21).** Kjøringen sier det nå på to måter, og begge
|
||||
stammer fra den *samme* oppslagsverdien inne i kjøringen — ikke fra en ny lesing av basen:
|
||||
|
||||
- **Et strukturert felt:** `provenance.cost_baseline_anchored` er `true`/`false`, og følger med
|
||||
inn i utboksens `{run_id}-proposal.json`, som dumper hele stempelet (VERIFISERT:
|
||||
`tests/test_baseline_visibility_loadbearing.py`, armene (a) og (d)).
|
||||
- **Én linje på stdout:** `Cost baseline: NONE in the bundle — this run is un-anchored: …`,
|
||||
skrevet av tørrkjøringen, av den fulle enkeltkjøringen og per prosjekt i porteføljemodus. Er
|
||||
basen forankret, skrives **ingen linje i det hele tatt** — en linje for noe kjøringen ikke har
|
||||
utelates, samme regel som resten av kunngjøringen følger (VERIFISERT: kjørt 2026-08-21 mot to
|
||||
kopier av veglys-basen; intakt kopi er byte-uendret, kopi uten fila bærer linja).
|
||||
|
||||
Ankeringen er fortsatt **valgfri** — en base skrevet før fila fantes kjører uendret. Dette er
|
||||
synlighet, ikke en ny nekt. Demoen har sin egen, norske formulering
|
||||
(«KUNNSKAPSBASE: … — uten kostbaseline», `simulation.py:791-802`): to formuleringer med vilje, fordi
|
||||
demoen beskriver hva *basen erklærer* før kjøringen, mens linja over rapporterer hva *kjøringen
|
||||
gjorde*.
|
||||
|
||||
**Konsekvensen for planleggingen:** spørsmål 2 i [§3](#3-avgjørelsesdelen-spørsmålene-du-stiller-fagpersonen)
|
||||
er det eneste spørsmålet der et «vet ikke» ikke stopper noe — og derfor det eneste der svaret må
|
||||
dokumenteres utenfor systemet. En base uten `cost-baseline.json` bør ikke kalles kjøreklar av
|
||||
noen som vet hva fila gjør (ANTATT: en arbeidsregel; koden lar deg kjøre).
|
||||
|
||||
## 5. Veiprosjektet Fylkesveg Sør: fra bestilling til kjøreklar base
|
||||
|
||||
Eksempelet følger en veglysportefølje langs fylkesveg. Basen det ender i er
|
||||
`shared/examples/veglys-fv-soer/`, som er sjekket inn, kjører i demoen og er målt med
|
||||
kjøreklar-sjekken under. Prosjektlaget i den basen er **fiktivt** — porteføljen finnes ikke —
|
||||
mens litteraturlaget er ekte og kildebelagt (VERIFISERT: basens `index.md`). Det gjør den til et
|
||||
godt eksempel på nøyaktig det skillet [§2](#2-kategoriene-hva-følger-hva) handler om: et ekte
|
||||
prosjekt bytter ut prosjektlaget og beholder resten.
|
||||
|
||||
### 5.1 Bestillingen
|
||||
|
||||
En driftsleder i fylkeskommunen vil vite hva LED-utskifting gir på de eldste strekningene, og om
|
||||
styring oppå det er verdt noe. Oppdragsfila (VERIFISERT: akseptert av kjøringen, kunngjøringen
|
||||
under er dens faktiske utskrift):
|
||||
|
||||
```json
|
||||
{
|
||||
"objective": "Redusere energikostnaden i veglysporteføljen Fylkesveg Sør uten å gå under lystekniske minstekrav, med tiltak som kan bestilles i 2027.",
|
||||
"approaches": [
|
||||
{
|
||||
"id": "led-trinn-1",
|
||||
"label": "LED-utskifting av de 2 500 eldste HPS-punktene",
|
||||
"description": "Drift melder at armaturene på de eldste strekningene er fra før 2005 og byttes hyppig; vi vil vite hva ren armaturutskifting gir før styring vurderes."
|
||||
},
|
||||
{
|
||||
"id": "adaptiv-styring",
|
||||
"label": "Adaptiv styring på de LED-utskiftede punktene",
|
||||
"description": "Håndbok V124 tillater MF 0,85; vi tror nye anlegg overdimensjoneres og at marginen kan hentes ut med dimming, men har ingen måling."
|
||||
}
|
||||
],
|
||||
"allow_own_proposals": true,
|
||||
"success_criteria": "Minst ett tiltak som passerer validatoren og som driftsavdelingen kan stå inne for."
|
||||
}
|
||||
```
|
||||
|
||||
Bestillingen er den første målingen av basen: hver tilnærming nevner ting basen må kunne svare
|
||||
på — armaturalder, vedlikeholdsfaktor, lystekniske minstekrav, fravær av måling.
|
||||
|
||||
### 5.2 Spørsmålene, besvart for dette prosjektet
|
||||
|
||||
| # | Spørsmål | Svar for Fylkesveg Sør | Lander i |
|
||||
|---|---|---|---|
|
||||
| 1 | Prosjekt-ID | `VEGLYS-FV-SOER` — samme streng på kommandolinjen og i begge tallfiler | alle tre |
|
||||
| 2 | Kostlinjer med ekte tall | én linje: porteføljens årlige energikostnad, `ENERGI-VEGLYS-EL`, 4 386 150 kWh à 1,00 NOK. Investeringskostnad **bevisst utelatt** — ingen kilde gir NOK per lyspunkt | `cost-baseline.json` |
|
||||
| 3 | Den ene projiserte kandidaten | LED-utskifting trinn 1 (2 500 punkter, 114 → 70 W), modellert 445 500 NOK/år | `validator-input.json` |
|
||||
| 4 | Harde rammer | lystekniske minstekrav (1,0 cd/m², 5 lx), MF ≤ 0,85, nattslukking kan ikke antas, tiltak vurderes inne i porteføljen | `veglys-fv-soer.md` |
|
||||
| 5 | Tilnærminger | de to i oppdragsfila, pluss systemets egne | oppdragsfila + to `hypothesis`-filer |
|
||||
| 6 | Målemetode | IPMVP Option A, ved eliminasjon: umålt anlegg stenger B, C og D | `metode-ipmvp-a.md` |
|
||||
| 7 | Litteratur om gapet | norsk: baseline, regelverk og *årsaken* til at gapet ikke kan ses (mangler måler). Lånt: selve realiseringsgraden (amerikansk programlitteratur, 0,81) | `kilder-veglys-realisering.md` |
|
||||
| 8 | Tidligere erfaring | én frø-dom: godkjent med realiseringskorreksjon, rate 0,81, **merket som lån** | `verdict-veglys-fro.md` |
|
||||
| 9 | Avgrensning til kostakse | nei — porteføljen har én kostlinje | — |
|
||||
| 10 | Kilder som data | nei — anleggsregisteret er levert som tall i et notat; alt er håndkuratert | — |
|
||||
| 11 | Hvem dømmer | fylkets egen energirådgiver, etter kjøringen, via innboksen | `--verdict-dir` |
|
||||
|
||||
(Alle svar i kolonnen «Svar» er VERIFISERT mot filene i `shared/examples/veglys-fv-soer/`; kolonnen
|
||||
«Lander i» er VERIFISERT mot filnavnene der.)
|
||||
|
||||
### 5.3 Hva fagpersonene leverer
|
||||
|
||||
| Leveranse | Fra | Form de leverer i | Blir til |
|
||||
|---|---|---|---|
|
||||
| Anleggsregister: antall lyspunkter, armaturtype, installert effekt | drift | uttrekk fra anleggsdatabasen, regneark | `type: project` (energibaseline) + raden i `cost-baseline.json` |
|
||||
| Brenntimer og energipris | drift / økonomi | tabellverdi (Håndbok V124) + fakturagrunnlag | samme; prisbåndet i `validator-input.json` |
|
||||
| Kravgrunnlag: lystekniske minstekrav, vedlikeholdsfaktor | fagmiljø vegbelysning | henvisning til NMFV og Håndbok V124 | «Rammer» i `type: project` |
|
||||
| Kandidat-tiltak med parametere | drift + fagmiljø | notat: før/etter-effekt, antall, hva som er utledet | to `type: hypothesis`-filer |
|
||||
| M&V-praksis for umålte anlegg | fagmiljø | mal for IPMVP, tilpasset | `type: methodology` |
|
||||
| Litteratur om realiseringsgap, med kilde | fagmiljø | kildeliste med URL og årstall, merket norsk/lånt | `type: reference` |
|
||||
| Tidligere vurdering av LED på småveg | energirådgiver | kort notat: «forvent ~80 % av modellert, fordi …» | `type: verdict` (frø) |
|
||||
|
||||
Leveranseformene er ANTATT — de er det en slik leveranse rimelig ser ut som, ikke noe
|
||||
eksempelbasen dokumenterer. Det som er VERIFISERT er hva hver leveranse *blir til*.
|
||||
|
||||
### 5.4 Basen som bygges
|
||||
|
||||
```
|
||||
veglys-fv-soer/
|
||||
├── index.md type: index inngangen; lenker til alt under
|
||||
├── veglys-fv-soer.md type: project porteføljen, energibaselinen, rammene
|
||||
├── tiltak-led-utskifting.md type: hypothesis kandidat 1 — den som er projisert
|
||||
├── tiltak-adaptiv-styring.md type: hypothesis kandidat 2 — svakere kildebelagt, og merket slik
|
||||
├── metode-ipmvp-a.md type: methodology Option A, og hvorfor de andre er stengt
|
||||
├── kilder-veglys-realisering.md type: reference Del A norsk / Del B lånt
|
||||
├── verdict-veglys-fro.md type: verdict frø-dommen — holdes ute av lesekonteksten
|
||||
├── validator-input.json den projiserte kandidaten
|
||||
└── cost-baseline.json prosjektets ene kostlinje
|
||||
```
|
||||
|
||||
(VERIFISERT: `ls shared/examples/veglys-fv-soer/` og `grep '^type:'` over filene.)
|
||||
|
||||
`index.md` gjør to jobber i denne basen. Den første er navigasjon: seks lenker, én per fil, med
|
||||
type og én setning hver. Den andre er å si høyt hva som er fiktivt og hva som er ekte, og
|
||||
*hvorfor* domenet er valgt — at realiseringsgraden i norsk veglys er «strukturelt usynlig» fordi
|
||||
anlegget mangler måler. Begge deler går ordrett inn som det første agentene leser.
|
||||
|
||||
### 5.5 De to tallfilene — skrevet fra samme linje
|
||||
|
||||
Hele basens tallgrunnlag er én linje aritmetikk:
|
||||
|
||||
> 9 500 lyspunkter × 114 W × 4 050 t/år ÷ 1 000 = **4 386 150 kWh/år** à 1,00 NOK = 4 386 150 NOK/år
|
||||
|
||||
`cost-baseline.json` bærer den som `ENERGI-VEGLYS-EL: {quantity: 4386150, unit_cost: 1.0}`.
|
||||
`validator-input.json` bærer **nøyaktig samme** kode, mengde og pris i `affected_items`, pluss den
|
||||
modellerte besparelsen for trinn 1 (2 500 × 44 W × 4 050 t ÷ 1 000 = 445 500 kWh ≈ 445 500 NOK)
|
||||
og prisbåndet 0,70–1,40 NOK/kWh til risikosimuleringen (VERIFISERT: begge filene). De to er ikke
|
||||
«innenfor toleranse» — de er identiske, fordi begge er skrevet fra samme linje. Det er slik
|
||||
toleransen på 5 % skal lukkes: ved konstruksjon, ikke ved avstemming etterpå.
|
||||
|
||||
**Én beslutning i mappingen er verdt å lære av:** `affected_items` er *hele porteføljens*
|
||||
energikostnad, ikke de 2 500 berørte punktenes eget forbruk. Hadde det vært det siste, ville
|
||||
besparelsen vært 38,6 % av linjen — over validatorens 30 %-tak — og det riktige forslaget blitt
|
||||
avvist av en gate som målte feil størrelse (VERIFISERT: `tiltak-led-utskifting.md`, «Mapping til
|
||||
validatoren»). Kostlinjen skal være den linjen tiltaket *virker på* i regnskapet.
|
||||
|
||||
### 5.6 Frø-dommen
|
||||
|
||||
```yaml
|
||||
type: verdict
|
||||
decision: approved_with_adjustment
|
||||
realization_rate: 0.81
|
||||
modelled_saving_nok: 445500
|
||||
expected_actual_saving_nok: 360855
|
||||
description: "… brenntimene er et nasjonalt tabellanslag, ikke en målt kurve, og anlegget
|
||||
mangler måler — så avviket kan ikke oppdages i drift. Forventet faktisk besparelse settes til
|
||||
81 % av modellert, lånt fra belysnings-programlitteratur og merket som lån. …"
|
||||
provenance: "frø — AI-forfattet. Realiseringsgraden er LÅNT … Det finnes INGEN norsk ex-post-måling
|
||||
for veglys. Erstattes av ekte HITL i produksjon."
|
||||
```
|
||||
|
||||
(Utdrag; VERIFISERT: `verdict-veglys-fro.md`.) Det som når neste hypotese er `description` pluss
|
||||
`[realiseringsgrad=0.81; forventet_faktisk_NOK=360855]` (VERIFISERT: `verdicts.py`
|
||||
`_verdict_rationale`). `provenance`-feltet leses ikke av koden — men det er det som gjør at en
|
||||
fagperson som åpner basen ser at dommen er et frø og raten et lån. I et ekte prosjekt erstattes
|
||||
denne fila av energirådgiverens egen vurdering, i samme form.
|
||||
|
||||
### 5.7 Kjøreklar-sjekken
|
||||
|
||||
Det finnes ingen egen «valider basen»-kommando
|
||||
([§6](#6-ærlighetsdelen-hva-som-ikke-finnes-i-dag)). Det nærmeste er en tørrkjøring med
|
||||
bestillingen på plass:
|
||||
|
||||
```bash
|
||||
uv run python -m portfolio_optimiser.run VEGLYS-FV-SOER \
|
||||
--docs-dir shared/examples/veglys-fv-soer \
|
||||
--bundle-dir shared/examples/veglys-fv-soer \
|
||||
--mandate oppdrag.json \
|
||||
--live-dry-run
|
||||
```
|
||||
|
||||
Målt 2026-08-21 (VERIFISERT, rc 0):
|
||||
|
||||
```
|
||||
Run mandate for VEGLYS-FV-SOER
|
||||
Objective: Redusere energikostnaden i veglysporteføljen Fylkesveg Sør uten å gå under lystekniske minstekrav, med tiltak som kan bestilles i 2027.
|
||||
Evaluates: 2 expert-proposed approach(es) + the system's own proposals
|
||||
1. led-trinn-1 — LED-utskifting av de 2 500 eldste HPS-punktene
|
||||
2. adaptiv-styring — Adaptiv styring på de LED-utskiftede punktene
|
||||
Stops at: 3 rounds / 100000 tokens
|
||||
Contacts: no external services
|
||||
Success: Minst ett tiltak som passerer validatoren og som driftsavdelingen kan stå inne for.
|
||||
VEGLYS-FV-SOER: LIVE-DRY-RUN OK (profile=local, models={'proposer': 'qwen3:4b', 'checker': 'qwen3:4b'}, max_rounds=3, max_tokens=100000, top_k=3) — ingen modellkall gjort (stoppet før første debate.run)
|
||||
```
|
||||
|
||||
**Hva `OK` beviser:** basen åpner — `index.md` finnes, `validator-input.json` finnes og bærer
|
||||
riktig prosjekt-ID, `cost-baseline.json` *hvis den finnes* er velformet, bestillingen er gyldig,
|
||||
modellene er oppløst, takene er satt, og ingen ekstern tjeneste vil bli kontaktet. Alt dette
|
||||
skjer før kuttet, uten et eneste modellkall (VERIFISERT: `run.py:513-516` ligger før
|
||||
dry-run-returen på `:565-588`).
|
||||
|
||||
**Hva `OK` sier fra om:** en lenke kjøringen ikke klarte å følge. Tørrkjøringen skriver da én
|
||||
blokk under `OK`-linja — én linje per lenke, med filen lenken sto i, lenketeksten ordrett, og
|
||||
hvorfor: `missing` (målet ligger inne i basen, men det er ingen lesbar fil der — nesten alltid en
|
||||
skrivefeil i lenken) eller `outside-bundle` (målet peker ut av basen). Er alle lenkene fulgt,
|
||||
skrives ingen linje i det hele tatt. Målt på en base med begge feilene:
|
||||
|
||||
```
|
||||
Knowledge base: 2 cross-link(s) NOT followed — the agents never read the document(s) behind them:
|
||||
- index.md -> metodikk-som-aldri-ble-skrevet.md (missing)
|
||||
- index.md -> ../nabobase/index.md (outside-bundle)
|
||||
```
|
||||
|
||||
Selve toleransen er uendret: en brutt lenke stopper ikke kjøringen (OKF SPEC §4 krever det). Det
|
||||
som er nytt er at den ikke lenger er taus — før dette var «dokumentet ble aldri skrevet» og «lenken
|
||||
til det var feil» umulig å skille utenfra. Den samme blokka skrives på en full kjøring.
|
||||
|
||||
**Hva `OK` ikke beviser:** at `cost-baseline.json` *finnes* (se [§4.1](#41-den-skarpeste-mangelen-cost-baselinejson)
|
||||
— sjekk det med `ls`), at filene faktisk er lenket fra `index.md` (en fil ingen lenker til er
|
||||
unåbar, og da finnes det ingen lenke å rapportere), eller at innholdet er godt. `--docs-dir` må oppgis, men leses ikke på bundle-stien (VERIFISERT:
|
||||
`run.py:513-530` bruker bare `bundle_dir`; `:1504` krever flagget likevel). Pek den på samme
|
||||
mappe.
|
||||
|
||||
**Det offline ende-til-ende-beviset på denne basen** er demoen:
|
||||
`uv run python -m portfolio_optimiser.simulation`. Den kjører nøyaktig `veglys-fv-soer`, skriver
|
||||
«KUNNSKAPSBASE: veglys-fv-soer — kostbaseline erklært (ENERGI-VEGLYS-EL 4386150 x 1)», navigerer
|
||||
de fem konseptfilene, og viser at en dom avgitt etter kjøring A når kjøring B (VERIFISERT:
|
||||
`tests/golden/demo-transcript.stdout`, linjene 7, 13 og 50). Agent-svarene i demoen er
|
||||
skriptede; den beviser dataflyten, ikke modellens dømmekraft.
|
||||
|
||||
### 5.8 Etter kjøringen
|
||||
|
||||
Kjør med `--outbox-dir` og `--run-id`, så legger kjøringen igjen én artefakt per vurdert
|
||||
tilnærming. `hitl pending` viser hva som venter på dom; energirådgiveren skriver dommen som
|
||||
JSON-fil i innboksen etter [ekspert-svar.md](ekspert-svar.md). Neste kjøring med `--verdict-dir`
|
||||
leser den inn før hypotesen formes. Godkjennes den for promotering, blir den en ny
|
||||
`type: verdict`-fil i basen — og basen er blitt én versjon klokere.
|
||||
|
||||
## 6. Ærlighetsdelen: hva som ikke finnes i dag
|
||||
|
||||
Ingenting under er en feil i koden. Det er grensen for hva som er bygget, og det skal stå her
|
||||
så dokumentet ikke lover mer enn det som kan leveres.
|
||||
|
||||
- **Bundle-fabrikken er ikke bygget** (operatørbeslutning D-G / T0). Det finnes ingen verktøy
|
||||
som tar fagpersonens regneark og notater og lager en base. Oversettelsen til markdown,
|
||||
lenkene i `index.md` og begge tallfilene er den tekniske personens håndarbeid (VERIFISERT:
|
||||
[knowledge-base-recipe.md](knowledge-base-recipe.md), «Future / factory-dependent»).
|
||||
- **Fri-format-oversettelse av dommer finnes ikke.** Frø-dommer skrives direkte i det strenge
|
||||
formatet (VERIFISERT: samme sted). Den tiltenkte flyten — eksperten leverer fri tekst, en
|
||||
oversetter lager den strukturerte dommen, som alltid peker tilbake på originalen — er
|
||||
beskrevet, ikke bygget.
|
||||
- **Ingest har ingen kommandolinje.** `materialize` er et Python-API (VERIFISERT: ingen
|
||||
`__main__` i `src/portfolio_optimiser/ingest.py`), og «no bundle has yet been materialized from
|
||||
a live source» (VERIFISERT: `README.md`). Det som er målt er fil/CSV, SQL, HTTP og MCP mot
|
||||
innsjekkede fixtures.
|
||||
- **Tidligere dommer hentes én gang per kjøring, for den projiserte kandidaten** — ikke per
|
||||
tilnærming i bestillingen (VERIFISERT: `run.py:634-638`; én `expel_query` for hele
|
||||
kjøringen). En bestilling med to tilnærminger mot to ulike kostlinjer får tidligere dommer
|
||||
hentet for den kandidaten `validator-input.json` beskriver.
|
||||
- **Det finnes ingen «er basen klar?»-kommando.** `--live-dry-run` beviser at basen *åpner*, sier
|
||||
fra hvis `cost-baseline.json` mangler, og lister lenkene den ikke klarte å følge
|
||||
([§5.7](#57-kjøreklar-sjekken)) — men ingenting om innholdets kvalitet, og ingenting om filer
|
||||
ingen lenker til (de er unåbare, så det finnes ingen brutt lenke å rapportere). `preflight`
|
||||
sjekker Azure-konfigurasjon, ikke basen (VERIFISERT: `preflight.py` docstring).
|
||||
- **Uforankret kjøring er synlig, men ikke summert.** Feltet og linja finnes per kjøring
|
||||
([§4.1](#41-den-skarpeste-mangelen-cost-baselinejson)); det finnes ingen rapport som teller opp
|
||||
hvor mange kjøringer i et porteføljepass som gikk uforankret.
|
||||
- **Eksempelbasene er ikke ekte prosjekter.** Prosjektlaget er fiktivt; realiseringsgraden i alle
|
||||
tre frø-dommene er lånt fra utenlandsk programlitteratur fordi ingen norsk ex-post-måling
|
||||
finnes (VERIFISERT: `provenance`-feltet i de tre dom-filene).
|
||||
- **De fire innholdstypene er konvensjon**, ikke spesifikasjon
|
||||
([§4](#4-innholdstypene)).
|
||||
- **`docs/extending.md` er utdatert på ett punkt:** den sier at ingen eksempelbase shipper
|
||||
`cost-baseline.json`. Det var sant da den ble skrevet (2026-08-05); begge veiprosjekt-basene har
|
||||
fått fila siden (VERIFISERT: `ls`, begge datert 2026-08-09). Rettelsen er ikke gjort her, for
|
||||
den hører hjemme i det dokumentet.
|
||||
- **1–2 uker.** Oppskriften sier det, og ingenting i dette dokumentet korter det ned. Det som
|
||||
står her er hva ukene skal brukes til.
|
||||
|
||||
## 7. Punktet der utforskningen konkluderte med kode
|
||||
|
||||
Dokumentet pekte opprinnelig på ett punkt der det ikke holdt å beskrive: en kjøring som går uten
|
||||
`cost-baseline.json` burde si det — i kunngjøringen, i `provenance`, eller begge. Demoen gjorde det
|
||||
allerede; kjørestien gjorde det ikke.
|
||||
|
||||
**Det er nå bygget (2026-08-21), med begge deler:** et strukturert `cost_baseline_anchored`-felt på
|
||||
provenance-stempelet, og én linje på stdout når kjøringen er uforankret — se
|
||||
[§4.1](#41-den-skarpeste-mangelen-cost-baselinejson). Ankeringen forble valgfri, og
|
||||
golden-transkriptet er byte-uendret, fordi demoen kjører en base som *har* fila og linja derfor
|
||||
ikke fyrer der.
|
||||
|
||||
## 8. Verifiseringslogg
|
||||
|
||||
| Påstand | Kilde | Status |
|
||||
|---|---|---|
|
||||
| Én base per prosjekt | `shared/method-spec.md` §2 («one curated … knowledge bundle per project») | VERIFISERT |
|
||||
| Navigasjon fra `index.md`, flat rendering `## type: tittel`, verdict ekskludert | `shared/method-spec.md` §3 Steg 1; `src/portfolio_optimiser/okf.py` `bundle_context`, `context_files` | VERIFISERT |
|
||||
| Lenke ut av basen hoppes stille over | method-spec §3 Steg 1 | VERIFISERT |
|
||||
| Fil uten index-lenke er unåbar | `shared/ingest-spec.md` §2 | VERIFISERT |
|
||||
| Verdict-laget reservert; ingest kan ikke skrive det | ingest-spec §3, §4 (`okf_type` MUST NOT be `verdict`) | VERIFISERT |
|
||||
| Tidligere dommer nøkles på `validator-input.json`-kandidaten, én gang per kjøring | `run.py:634-638`; `verdicts.py` `bundle_candidate_features` | VERIFISERT |
|
||||
| Det som valideres er modellens forslag, ikke `validator-input.json` | `generate.py:491` | VERIFISERT |
|
||||
| Seks `type:`-verdier i drift: index 3 · project 4 · hypothesis 5 · methodology 3 · reference 3 · verdict 3 | `grep -h '^type:' shared/examples/*/*.md \| sort \| uniq -c` (2026-08-21) | VERIFISERT |
|
||||
| De fire ikke-reserverte typene er ikke nevnt i noen spesifikasjon | `grep` over `shared/method-spec.md` og `shared/ingest-spec.md` — null treff | VERIFISERT |
|
||||
| Metode-filene er tre ulike filer (40 / 81 / 98 linjer), materialisert per base | `wc -l`, `diff`; begge `index.md` («materialisert inn her») | VERIFISERT |
|
||||
| Koden leser ikke `resource`, `measure_id`, `gap_source`, `context_key`, `okf_version`, `tags` | `grep` over `src/portfolio_optimiser/*.py` — null treff | VERIFISERT |
|
||||
| `dimension:` brukes av ingen eksempelbase | `grep -rn '^dimension:' shared/examples/` — null treff | VERIFISERT |
|
||||
| Prosjekt-ID må stemme mellom kommandolinje og `validator-input.json` | `run.py` `_project_from_bundle` | VERIFISERT |
|
||||
| Prosjektnavn leses fra `type: project`-filas `title`, ellers ID | samme | VERIFISERT |
|
||||
| Steg 0 avviser ukjent kode og avvik > 5 % | `validator.py:154-190`; `BASELINE_TOLERANCE_DEFAULT = 0.05` | VERIFISERT |
|
||||
| Manglende `cost-baseline.json` → `None` → steg 0 hoppes over | `okf.py:323-335`; `run.py:516`; `validator.py:213` | VERIFISERT |
|
||||
| Fire tørrkjøringer: intakt rc 0 · uten IR rc 1 · uten baseline rc 0 uten melding · korrupt baseline rc 1 | kjørt 2026-08-21 på kopier i scratchpad, kommandoen i §5.7 | VERIFISERT |
|
||||
| Ingen artefakt bærer forankret/uforankret | `grep baseline src/portfolio_optimiser/provenance.py src/portfolio_optimiser/outbox.py` — null treff | VERIFISERT |
|
||||
| Demoen printer forankringsstatus | `simulation.py:791-802`; golden linje 7 | VERIFISERT |
|
||||
| `--live-dry-run` navigerer basen og laster begge tallfiler før kuttet | `run.py:513-516` vs `:565-588` | VERIFISERT |
|
||||
| `--docs-dir` påkrevd men ulest på bundle-stien | `run.py:513-530`, `:1504` | VERIFISERT |
|
||||
| Demoen kjører `veglys-fv-soer`, navigerer 5 konseptfiler, henter 0 så 3 dommer | `tests/golden/demo-transcript.stdout` linjene 7, 13, 14, 50 | VERIFISERT |
|
||||
| Frø-rationale = `description` + `[realiseringsgrad=…; forventet_faktisk_NOK=…]` | `verdicts.py` `_verdict_rationale` | VERIFISERT |
|
||||
| Dom-nøkkel-trioen: alle tre felt eller ingen | `verdicts.py` `_features_from_verdict_frontmatter`; `README.md` | VERIFISERT |
|
||||
| Veglys-tallene: 9 500 × 114 W × 4 050 t = 4 386 150 kWh; 445 500 NOK modellert; 10,2 % av total; 38,6 % av berørte punkter | `veglys-fv-soer.md`, `tiltak-led-utskifting.md`, begge JSON-filer | VERIFISERT |
|
||||
| Frø-dommen: rate 0,81, forventet 360 855, lånt | `verdict-veglys-fro.md` frontmatter | VERIFISERT |
|
||||
| Fabrikk og fri-format-oversettelse ikke bygget | `docs/knowledge-base-recipe.md`; `docs/plan/2026-07-14-revisjonspakke-DF-DI.md` §3 | VERIFISERT |
|
||||
| Ingest har ingen CLI; ingen base materialisert fra levende kilde | `grep __main__ src/portfolio_optimiser/ingest.py` — null treff; `README.md` «How it is set up» | VERIFISERT |
|
||||
| Ingest skriver ikke `validator-input.json` / `cost-baseline.json` | `docs/extending.md`, «Legg til en ingest-kilde» | VERIFISERT |
|
||||
| `docs/extending.md` sier ingen eksempelbase shipper `cost-baseline.json` | samme dokument; motbevist av `ls shared/examples/{veglys-fv-soer,tunnel-hauglia}/` | VERIFISERT (utdatert) |
|
||||
| Leveranseformene i §5.3; anbefalingene merket ANTATT | — | ANTATT |
|
||||
|
|
@ -69,7 +69,7 @@ nøkkel/nettverk · mypy/ruff grønne.
|
|||
## I3 — D7-speil: filkatalog/CSV **[GATET: annet repo]**
|
||||
|
||||
**Mål:** Samme kontrakt konsumert uendret fra commons i søskenrepoet
|
||||
(`/Users/ktg/repos/portfolio-optimiser-claude`) — beviser at ingest-spec-en faktisk er delt.
|
||||
(`../portfolio-optimiser-claude`) — beviser at ingest-spec-en faktisk er delt.
|
||||
|
||||
**Innhold:** forankres i søskenets STATE + egen `/trekbrief`. Commons-pull der; egen impl mot
|
||||
samme spec; speiltester av I2s load-bearing-sett; søskenets framework-guard utvides
|
||||
|
|
|
|||
|
|
@ -37,8 +37,8 @@
|
|||
|---|---|---|---|
|
||||
| A1 | `portfolio-optimiser/src/portfolio_optimiser/okf.py` | Py, 202 l | les + skriv + validér |
|
||||
| A2 | `portfolio-optimiser-claude/src/portfolio_optimiser_claude/okf.py` | Py, 134 l | **kun les** (mangler skrive-halvdelen/Steg 8-primitiver) |
|
||||
| B1 | `ktg-plugin-marketplace/okr/` (lib+scripts `.mjs`) | JS | index-generering + validering |
|
||||
| B2 | `ktg-plugin-marketplace/linkedin-studio/scripts/brain/` | TS | emit + conformance-test |
|
||||
| B1 | `plugin-marketplace/okr/` (lib+scripts `.mjs`) | JS | index-generering + validering |
|
||||
| B2 | `plugin-marketplace/linkedin-studio/scripts/brain/` | TS | emit + conformance-test |
|
||||
| C1 | `claude-code-llm-wiki/tools/wiki_ingest/bundle.py` | Py, 436 l | OKF-produsent (ingest→bundle) |
|
||||
| C2 | `llm-ingestion-pipeline-security/src/llm_ingestion_guard/okf.py` | Py, 579 l | defensiv validator, **reject-by-default** |
|
||||
|
||||
|
|
|
|||
|
|
@ -25,7 +25,7 @@ kodelokasjoner (revisjonspakke §0.2, verifisert mot ground truth 2026-07-14):
|
|||
| A2 | `portfolio-optimiser-claude/.../okf.py` | 134 l | **kun les** — mangler skrive-halvdelen |
|
||||
| C1 | `claude-code-llm-wiki/tools/wiki_ingest/bundle.py` | 436 l | OKF-produsent (ingest→bundle) |
|
||||
| C2 | `llm-ingestion-pipeline-security/.../okf.py` | 579 l | defensiv validator, reject-by-default |
|
||||
| B1/B2 | `ktg-plugin-marketplace/` (`.mjs`/`.ts`) | JS/TS | index-generering + conformance |
|
||||
| B1/B2 | `plugin-marketplace/` (`.mjs`/`.ts`) | JS/TS | index-generering + conformance |
|
||||
|
||||
≥4 uavhengige Python-`parse_frontmatter` + 2 i JS. Konsekvensene (F-INT-2): duplisert
|
||||
vedlikehold, og at Claude-søskenet (A2) **mangler skrive-halvdelen** A1 har — så det ikke kan
|
||||
|
|
@ -93,7 +93,7 @@ UNDER det D7-sammenligningen måler, så delt kode her forurenser ikke sammenlig
|
|||
mangler i dag** (Steg 8-primitiver). Dette er hovedgevinsten for D-Gs «minst like godt»-mål.
|
||||
3. **`claude-code-llm-wiki` (C1)** → senere; produsent-stien (ingest→bundle) legges på
|
||||
`okf-core` + fabrikken.
|
||||
4. **`ktg-plugin-marketplace` (B1/B2)** → forblir JS/TS (kan ikke dele Python-kode), men
|
||||
4. **`plugin-marketplace` (B1/B2)** → forblir JS/TS (kan ikke dele Python-kode), men
|
||||
konsumerer **formatprøven** som delt data.
|
||||
5. **`llm-ingestion-guard` (C2)** → migreres IKKE; forblir eget repo og trekkes inn som
|
||||
**avhengighet** i fabrikkens sikkerhetsvask.
|
||||
|
|
|
|||
|
|
@ -175,7 +175,7 @@ andre enden — å avgi dommen etterpå — er `docs/ekspert-svar.md`.
|
|||
**Feiler live-kjøringen på scenen: ikke debug. Vis fila.**
|
||||
|
||||
```
|
||||
cat /Users/ktg/repos/portfolio-optimiser/tests/golden/demo-transcript.stdout
|
||||
cat tests/golden/demo-transcript.stdout
|
||||
```
|
||||
|
||||
**Stien er absolutt med vilje.** Prosaen under peker på feil katalog som en sannsynlig årsak — og en
|
||||
|
|
@ -307,7 +307,7 @@ til **hakene** — de er det eneste som ellers ville havnet i fila etter `Y`.
|
|||
Et `| wc -l` alene kan altså ikke skille dem, og lest som «taggen landet ikke» er det en abort
|
||||
på feil grunnlag — dagen etter at push-en faktisk lyktes. **Exit 128: vent til porten svarer
|
||||
igjen, og bruk en probe — ikke en klokke:** `ssh -T git@git.fromaitochitta.com` → `Hi there,
|
||||
ktg!`. Det er dette som kom tilbake da forbindelsen løsnet 08-11; **ingen fast ventetid er
|
||||
<user>!`. Det er dette som kom tilbake da forbindelsen løsnet 08-11; **ingen fast ventetid er
|
||||
målt**, så et tall her ville vært gjettet. Kjør så kommandoen på nytt. `git tag -l` er lokal og
|
||||
svarer uansett; den skiller «tagget lokalt» fra «nådde origin» uten å røre nettverket.
|
||||
**Ble PUSH-en selv rate-limitet (punkt 10), er retry både riktig og trygt** — målt: `git push`
|
||||
|
|
|
|||
652
docs/plan/2026-08-23-magentic-utforskningssloeyfe.md
Normal file
652
docs/plan/2026-08-23-magentic-utforskningssloeyfe.md
Normal file
|
|
@ -0,0 +1,652 @@
|
|||
# MAF-utnyttelsesrevisjon U1–U15 + plan for Magentic-basert utforskningssløyfe
|
||||
|
||||
> **Ordre** `20260823T125528Z-1876024635-from-.claude` · **Økt 53**, 2026-08-23 · Fable 5/xhigh uten
|
||||
> advisor (operatør-overstyring). Planarbeid: ingen kode i `src/`, ingen tester. **Hvert tall i dette
|
||||
> dokumentet kommer fra en kommando som ble kjørt denne økten** (vedlegg V1–V3); alt som ikke ble
|
||||
> målt er merket «ikke målt» eller «spike».
|
||||
|
||||
## 0. Sammendrag — de seks setningene som bærer resten
|
||||
|
||||
1. **Målingen:** av `.claude`s ni «null i src/»-påstander holder 7 ordrett; U5 og U11 trenger
|
||||
presisering (artefaktet/sitatbæringen finnes, MAF-konstruktet gjør det ikke). Nevner: 32 filer.
|
||||
2. **Magentic-status:** hele Learn-siden hentet — **ingen experimental-advarsel for Python**
|
||||
(eneste «experimental» er et C#-pragma), og installert kode har ingen `@experimental`-markør
|
||||
på Magentic. F14-sitatet «untested … outside of the original Magentic-One design» **står
|
||||
fortsatt**. PyPI latest er 1.1.1 (Production/Stable), men bare 1.0.0/1.0.1 kjører på core 1.9.0.
|
||||
3. **Designet:** Magentic legges **OVER** den normative 8-stegs-sløyfa, ikke inni Steg 3. Utforskningen
|
||||
er en **mandat-former**: prompt + bundles (+ fagpersonens frø-hypoteser) → `Mandate` →
|
||||
`run_project(mandate=…)` **uendret**. Validatoren gater hvert tall i nøyaktig samme gate som i dag.
|
||||
4. **Fire ting ble MÅLT på installert 1.0.0, ikke antatt:** en Magentic-workflow er single-use
|
||||
(høylytt `RuntimeError`), men **builder og manager-instans blør** (4/4 kontaminering, worker
|
||||
hoppet over, fabrikkert sluttsvar); plan-review-rundturen **virker**, men har **ingen
|
||||
revise-cap**; `max_round_count` gir en **kanonisk streng, aldri et typet signal**;
|
||||
`MagenticResetSignal` **nullstiller ikke** deltakernes sesjon (orphan-attributt i 1.0.0).
|
||||
5. **U14 er ikke «én funksjon»:** `setup_observability` finnes ikke; inngangen er
|
||||
`agent_framework.observability.configure_otel_providers`, og **`opentelemetry-sdk` er ikke
|
||||
installert** — kallet krasjer med `ModuleNotFoundError`. U14 er en avhengighetsbeslutning.
|
||||
6. **Rekkefølgen:** S0–S6-spikes (1 økt) → U14 (1) → U4+U13 synkron (2) → U12 + asynkron U13 (1)
|
||||
→ U15 som egen sammenligning (1). U1/U2/U5/U11 avvises med begrunnelse. Seks økter.
|
||||
|
||||
---
|
||||
|
||||
## A. U1–U15-måling (nevner: 32 `.py`-filer i `src/`; kontroll: 117 i `tests/`, 6 i `spikes/`)
|
||||
|
||||
Metode: `grep -rEn --include='*.py' "<regex>" src` per kapabilitet, samme regex mot `tests/` og
|
||||
`spikes/` som kjent-positiv-kontroll, og — for hver regex som ga 0 i alle tre — en tredje kjøring
|
||||
mot installert MAF (`.venv/lib/python3.12/site-packages/agent_framework{,_orchestrations}`) som
|
||||
beviser at spørringen KAN finne (Verifiseringsloven ansikt 4). Skript og rå output: vedlegg V1.
|
||||
En uavhengig andre sveip (Opus-agent, 35 tokens, samme nevner) ga identiske nuller (V3).
|
||||
|
||||
| ID | Kapabilitet (§15.1) | src | tests | spikes | venv-kontroll | Status | Belegg / spørring |
|
||||
|----|----|---:|---:|---:|---:|----|----|
|
||||
| U1 | `ConcurrentBuilder` fan-out | 0 | 4 | 4 | — (tests/spikes finner) | **IKKE** — erstattet av håndrullet bølge | `run.py:1262` `await asyncio.gather(` (D-D). Regex `ConcurrentBuilder\|BuildConcurrent\|FanIn` |
|
||||
| U2 | `WorkflowBuilder` graf | 0 | 0 | 0 | 21 filer | **IKKE** | Regex `WorkflowBuilder\|@executor\|class .*\(Executor\)\|WorkflowContext\|SwitchCase\|add_edge\|add_fan_out` |
|
||||
| U3 | `GroupChatBuilder` debatt | 10 | 7 | 7 | — | **LANDET** | `workflow.py:24` import · `:104` `GroupChatBuilder(` · `:108` `termination_condition=` · `:113` `.with_max_rounds(max_rounds)` |
|
||||
| U4 | Magentic | 0 | 4 | 9 | 4 filer | **IKKE** | tests-treff: `tests/spikes/test_b_footguns.py`, `tests/spikes/test_imports.py`, `tests/test_method_spec_loadbearing.py:40` (framework-navn-svarteliste for spec-prosa — ikke bruk). `.claude`s «kun spikes/b_footguns.py» er riktig for BRUK |
|
||||
| U5 | Agent Skills (`SkillsProvider`) | 0 | 0 | 0 | 3 filer | **IKKE** som MAF-konstrukt; **DELVIS** som artefakt | `persona.py:3`/`:47` leser `shared/skills/expert-reviewer/SKILL.md` via EGEN `load_persona_example`, aldri `SkillsProvider`. Installert core: `@experimental(feature_id=ExperimentalFeature.SKILLS)` på `SkillsProvider` (`_skills.py:1718`) |
|
||||
| U6 | MCP-tools | 6 | 6 | 0 | — | **LANDET** (opt-in) | `mcp_tools.py:42-43` import · `:154` `MCPStdioTool(` · `:170` `MCPStreamableHTTPTool(`. Ingest-sømmen bruker rå `mcp.ClientSession` (`ingest_mcp.py:144-179`) |
|
||||
| U7 | Function tools | 4 | 10 | 0 | — | **LANDET** | `datasource.py:18` `from agent_framework import FunctionTool, tool` · `:65` · `:69` `@tool(` |
|
||||
| U8 | Middleware | 16 | 20 | 0 | — | **LANDET** | `budget.py:27` · `:228` `class BudgetMiddleware(ChatMiddleware)` · `run.py:635/:645` |
|
||||
| U9 | `ContextProvider` | 8 | 11 | 4 | — | **LANDET** | `verdicts.py:29` · `:319` `class ExpeLContextProvider(ContextProvider)` · `run.py:724/:843` |
|
||||
| U10 | Vektorlager | 2 | 19 | 0 | — | **DELVIS** (egen, ikke MAF) | `semretrieval.py:355` `save_vector_store` · `:393` `load_vector_store` — numpy brute-force (D-C). 0 på `AzureAISearch\|cosmos\|qdrant\|redis\|VectorStore` |
|
||||
| U11 | Citation-aware RAG | 26 | 35 | 0 | — | **DELVIS** (egen, med MAF-typer) | `provenance.py:18` `from agent_framework import Annotation, TextSpanRegion` · `:78` `to_annotations()` («display only, NOT the source of truth»); `datasource.py:1` «citation-bearing data source». 0 på `AIContextProvider\|TextSearchProvider\|agentic_retrieval\|KnowledgeAgent` |
|
||||
| U12 | Checkpointing | 3 (kun prosa) | 0 | 0 | 14 filer | **FRAVALGT** — §A.1 | `run.py:22`, `workflow.py:14`, `:115`: «no checkpoint — research 01: durable resume is fragile». 0 kode-treff på `checkpoint_storage\|CheckpointStorage` |
|
||||
| U13 | HITL-gates | 3 | 0 | 0 | — | **DELVIS, og UØVD** | `workflow.py:116` `builder.with_request_info(agents=[agents[-1]])` bak `enable_layer1_hitl=False` (`:78`, `run.py:524`; ingen CLI-flagg). ENESTE test: `tests/test_workflow.py:54` `test_layer1_hitl_option_builds` — bygger, kjører aldri gaten, sender aldri et svar. `run.py` awaiter `debate.run` til ferdig (`:684-686`) og leser aldri strømmen. 0 på `RequestInfoEvent\|approval_mode\|MagenticPlanReview` i src OG tests |
|
||||
| U14 | OpenTelemetry | 0 | 0 | 1 | 4 filer | **IKKE** | Regex `opentelemetry\|setup_observability\|OBSERVABILITY\|get_tracer\|start_as_current_span`. `uv pip list`: KUN `opentelemetry-api 1.42.1` (transitiv) — ingen sdk, ingen exporter |
|
||||
| U15 | Evaluering | 0 | 0 | 0 | 2 filer | **IKKE** | Regex `evaluate_workflow\|LocalEvaluator\|Evaluator\|_evaluation`. `evaluate_workflow` finnes (`_evaluation.py:1833`) under `@experimental(feature_id=ExperimentalFeature.EVALS)` (`:1832`); 20 `@experimental` i modulen |
|
||||
|
||||
Utenfor de 15 (samme kjøring): U16 `CompactionProvider` 0/0/0 (venv 4) · U17 `as_tool(` 0/0/0
|
||||
(venv 1) · U18 `FoundryChatClient` **LANDET** `backends.py:150` · U19 Harness 0/0/0 (venv 1).
|
||||
|
||||
**Mot `.claude`s ni null-påstander:** 7 av 9 holder ordrett (U1, U2, U4, U12, U13, U14, U15). 2 av 9
|
||||
trenger presisering: **U5** og **U11**. De 6 landet-påstandene holder (U3, U6, U7, U8, U9,
|
||||
U10-delvis). Aliasing/re-eksport er utelukket: alle 14 `agent_framework`-importlinjer i `src/` er
|
||||
listet i V3 og hver er et direkte navn; `from agent_framework_orchestrations import` → 0 treff
|
||||
(importstien i bruk er `agent_framework.orchestrations`, `workflow.py:24`).
|
||||
|
||||
### A.1 U12 — fravalgt på et premiss som har råtnet, eller som holder?
|
||||
|
||||
Premisset i koden: «research 01: durable checkpoint resume is fragile» (`workflow.py:14`).
|
||||
**«research 01» finnes ikke som dokument i repoet** (`grep -rn -i "research.01" docs/ STATE*.md`
|
||||
→ 0). Nærmeste dokumenterte grunnlag: kapabilitetskartet
|
||||
(`docs/research/2026-06-24-maf-capability-map.md:54`): «superstep-granular (not mid-LLM-call);
|
||||
resume needs identical graph. Defer past MVP unless needed»; egnethetsplanen 08-09 P9/P10
|
||||
(«pass-nivå-gjenopptakelse er levert via spend-fila; ingen checkpointing»); `docs/extending.md:315`
|
||||
(«A run either completes or is re-run; the async verdict inbox is the resumable boundary»).
|
||||
|
||||
**Vurdering:** begrunnelsen holder **for dagens debatt** (sekunder–minutter; re-kjøring er billigere
|
||||
enn gjenopptakelse). Den holder **ikke** for målbildet: en utforskning som stopper i
|
||||
`MagenticPlanReviewRequest` og venter på en fagperson i dager kan ikke holde en prosess i live, og
|
||||
«re-kjør» kaster planen fagpersonen skulle revidere. **Premisset er ikke råttent — det er utenfor
|
||||
sitt gyldighetsområde.** Det som ble MÅLT i installert core 1.9.0 denne økten (V2 §7): mekanismen
|
||||
FINNES og dekker Magentic-tilstand — `FileCheckpointStorage` (`_checkpoint.py:239`), «restore then
|
||||
send» (`run(responses=…, checkpoint_id=…)` eksplisitt tillatt, `_workflow.py:877`), pending
|
||||
`request_info` re-emitteres ved resume (`_runner_context.py:414-426`), og `MagenticOrchestrator.
|
||||
on_checkpoint_save` skriver kontekst (runde/stall/reset-tellere), ledgere og managerens hele sesjon
|
||||
(`_magentic.py:1266-1284`, `:746`). «Skjør» er i dag konkret: base64-pickle av tilstand med
|
||||
allow-list (`_checkpoint.py:242-250`), graf-signatur må matche (`_runner.py:275-279`), og
|
||||
manager-state-feil ved lagring/gjenoppretting **svelges med warning** (`_magentic.py:1281-1282`,
|
||||
`:1320-1321`). Ingen ende-til-ende-resume med pending plan review er kjørt → **spike S4** (§D).
|
||||
|
||||
---
|
||||
|
||||
## B. Magentic-status mot primærkilden (hele siden hentet 2026-08-23)
|
||||
|
||||
Kilde: `microsoft_docs_fetch` mot
|
||||
`learn.microsoft.com/en-us/agent-framework/workflows/orchestrations/magentic` — hele siden.
|
||||
|
||||
1. **Ingen experimental-advarsel for Python.** Eneste «experimental» på siden: C#-blokken
|
||||
`#pragma warning disable MAAIW001 // Magentic types are experimental` (.NET-diagnostikk).
|
||||
`.claude`s fravær er **bekreftet**, ikke bare ikke-motbevist.
|
||||
2. **F14-sitatet står ordrett:** «it is untested how well the Magentic orchestration will perform
|
||||
outside of the original Magentic-One design», og tipset «consider using the Group Chat pattern
|
||||
instead» for enklere koordinering. F14s VURDERING var riktig lest; operatøren har flyttet
|
||||
PREMISSET (målbildet krever planlegging), ikke sitatet. G8 i §15.3 er halvt riktig: funksjonelle
|
||||
workflows er fortsatt `ExperimentalFeature.FUNCTIONAL_WORKFLOWS`; Magentic er ikke markert.
|
||||
3. **Installert kode:** `agent_framework/_feature_stage.py:43-65` — `ExperimentalFeature` har 13
|
||||
medlemmer (DECLARATIVE_AGENTS, EVALS, FILE_HISTORY, FIDES, FOUNDRY_TOOLS, FOUNDRY_PREVIEW_TOOLS,
|
||||
FUNCTIONAL_WORKFLOWS, HARNESS, MCP_LONG_RUNNING_TASKS, MCP_SKILLS, PROGRESSIVE_TOOLS, SKILLS,
|
||||
TO_PROMPT_AGENT). **Magentic er ikke der**; `grep -n -i "experimental\|feature_stage"
|
||||
agent_framework_orchestrations/*.py` → 0.
|
||||
4. **PyPI** (`curl https://pypi.org/pypi/agent-framework-orchestrations/json`): latest **1.1.1**
|
||||
(2026-08-21), `Development Status :: 5 - Production/Stable`, 26 utgivelser. Krav per versjon:
|
||||
1.0.0/1.0.1 → core ≥1.9.0; 1.0.2/1.1.0 → ≥1.13.0; 1.1.1 → ≥1.15.0. Core latest 1.15.0 (21.08).
|
||||
5. **Installert = pinnet:** `uv pip list` → orchestrations 1.0.0, core 1.9.0; `uv.lock:61-63` låser
|
||||
1.0.0; `pyproject.toml:17` `>=1.0.0`, `:12` core `>=1.9.0,<2` (`tests/test_maf_version_guard.py`).
|
||||
6. **Learn-siden beskriver LATEST API** (`manager_agent=`, `enable_plan_review=`,
|
||||
`intermediate_output_from=`). Installert 1.0.0 har de samme konstruktør-navnene (V2 §2) — men
|
||||
builder-docstringen i 1.0.0 beskriver et HITL-API (`MagenticHumanInterventionRequest`,
|
||||
`with_human_input_on_stall`) som **ikke finnes i koden** (kun i docstrings, `_magentic.py:1388-1559`).
|
||||
Designet i §C er skrevet mot det som er målt installert, aldri mot docstring eller Learn.
|
||||
7. **Core 1.12.0-notatet (21.07, samme minutt som orchestrations 1.0.1):** «Prevent the Magentic
|
||||
manager from duplicating conversation history (#6297)». Om 1.0.1 bærer fiksen er **ikke
|
||||
verifisert** — det er spike S0.
|
||||
|
||||
---
|
||||
|
||||
## C. Design: den utforskende sløyfa
|
||||
|
||||
### C.0 Beslutningen — Magentic OVER sløyfa, ikke INNI Steg 3
|
||||
|
||||
Kjernespørsmålet: Magentic gir manageren frihet til å velge vei; validatoren gater utfallet. Er de
|
||||
forenlige? **Ja — på én betingelse: utforskningen produserer KANDIDATER, aldri FORSLAG.**
|
||||
|
||||
Tre fakta avgjør formen:
|
||||
|
||||
1. **Metodespesifikasjonen er normativ og commons-eid** (`shared/method-spec.md`, pull-only). §3
|
||||
Steg 3 KREVER en to-rolle maker-checker-debatt, rundetak og fersk tilstand per kjøring; Steg 2
|
||||
krever «exactly one candidate measure» per proposer-kall; §8 «Never an unbounded loop, anywhere».
|
||||
En Magentic-sløyfe som ERSTATTER Steg 3 bryter spec-en og krever commons-amendment FØR bygging —
|
||||
og den ville flyttet validatoren INN i grafen. Sømkartet (V3) er entydig: alt som gjør et svar
|
||||
etterprøvbart ligger UTENFOR workflow-grafen og ETTER `debate.run` — generering `run.py:748-759`,
|
||||
validator `generate.py:491`, provenance `run.py:793`, checker-override `:816`, dimensjonsgate
|
||||
`:826`, domfangst `:849`, outbox `:857`.
|
||||
2. **`Mandate` finnes og ER «en prompt om hva som skal gjøres»** (`mandate.py:55-88`): `objective` +
|
||||
`approaches[{id, label, description}]` + `allow_own_proposals` + `success_criteria`. En
|
||||
`Approach.description` er «the expert's own prose reason … fed to the proposer VERBATIM»
|
||||
(`mandate.py:44-49`, `generate.py:308-317`) — altså «hypoteser fagpersoner kommer med».
|
||||
`_evaluate_mandate` (`run.py:292-344`) kjører hver approach gjennom `generate_via_llm` →
|
||||
`validate_proposal` under SAMME meter, og rapporterer `not_evaluated` for det budsjettet ikke
|
||||
rakk (aldri utelatt). `announce` (`mandate.py:130-183`) sier høyt hva kjøringen skal gjøre FØR
|
||||
første betalte kall — repoets eksisterende før-kjøring-menneskesjekk.
|
||||
3. **Validatoren er ren og offline** (`validate_proposal`, `validator.py:194-260`; CBC via PuLP,
|
||||
`_MC_SAMPLES = 512`, ingen `agent_framework`-import) og kan eksponeres som `FunctionTool` (U7)
|
||||
uten å endre hva den er.
|
||||
|
||||
Derfor: **utforskningen er en MANDAT-FORMER.** Inn: operatørens prompt + tilgjengelige bundles
|
||||
(+ valgfrie frø-approaches fra fagpersoner). Ut: en `Mandate` hvis `approaches` er hypotesene
|
||||
utforskningen fant verdt å teste, med managerens begrunnelse som `description`, og `bundle_id` per
|
||||
approach. Deretter `run_project(bundle_dir=…, mandate=…)` **uendret**: debatt (Steg 3) → generering
|
||||
(Steg 2) → deterministisk validator (Steg 4, blokkerende, stempler `validator_decision`) →
|
||||
checker-gate → outbox/HITL (Steg 6–8). Ingen linje i den kjeden endres. Manageren velger VEI i et
|
||||
rom som ikke kan skrive til outbox eller wiki; validatoren gater hvert TALL som forlater rommet, i
|
||||
samme gate som i dag.
|
||||
|
||||
**Tre nivåer av garanti (uttalt, så ingen leser mer inn i det enn det er):**
|
||||
|
||||
| Nivå | Hva | Hvor | Stempel? |
|
||||
|---|---|---|---|
|
||||
| 1 | `quick_validate`-verktøyet i utforskningen | samme `validate_proposal`, samme baseline, kalt av hypothesiser-agenten | **NEI** — rådgivende tall i en utforskningslogg, aldri provenance |
|
||||
| 2 | Pipelinen per approach | `run.py:748` `_evaluate` → `validate_proposal` | **JA** — `ProvenanceStamp.validator_decision` |
|
||||
| 3 | Skriverettigheter | kun pipelinen når outbox (`outbox.write_outbox`) og wiki (`promote_verdict`, gated) | utforskningens sluttsvar er RÅ (`{run_id}-exploration.json`), aldri et forslag |
|
||||
|
||||
Ærlighetsregelen (§1) følger direkte: managerens «final answer» presenteres ALDRI som et resultat.
|
||||
Resultatet er det `_evaluate_mandate` returnerer.
|
||||
|
||||
### C.1 Manager + deltakersett (mot installert 1.0.0-API, V2 §2–3)
|
||||
|
||||
| Rolle | Type | Gjør | Verktøy / kanal |
|
||||
|---|---|---|---|
|
||||
| **manager** | `MagenticBuilder(manager_agent_factory=…)` → `StandardMagenticManager(agent=…)` med EGNE ledger-prompter (konstruktør-parametre `task_ledger_*_prompt`, `progress_ledger_prompt`, `final_answer_prompt`) | planlegger, velger neste taler, fører progress ledger, replanlegger | ingen — men **`BudgetMiddleware` PÅ manager-agenten** (C.2) |
|
||||
| **navigator** | `Agent` | leser bundles PROGRESSIVT (§3 Steg 1) og svarer med sitater | `list_bundles()` · `read_bundle(id)` → `okf.navigate_bundle` + `Bundle.index_summary()`/`bundle_context` · `read_file(id, path)` innenfor `retrieval.safe_resolve` |
|
||||
| **hypothesiser** | `Agent` | former én kandidat om gangen som IR-JSON (gjenbruker `generate._build_messages` + `proposal_response_format()`) | `quick_validate(ir_json)` → `{decision, reason, p10, p50, p90}` (nivå 1) |
|
||||
| **expert_liaison** | menneske/persona | svarer på spørsmål, reviderer retning | plan review (`MagenticPlanReviewRequest.approve()/.revise(feedback)`) — og, etter spike S3b, `AgentApprovalExecutor` som deltaker for svar MIDT i kjøringen |
|
||||
|
||||
**Hvorfor `StandardMagenticManager`, ikke egen subklasse:** egen manager = egen ledger-parsing = egen
|
||||
kopi av Magentic-One-logikken (kø-(p): den andre kopien drifter). Prompt-overstyring dekker behovet
|
||||
(norsk domene, IR-disiplin, «aldri oppgi et tall uten `quick_validate`»). **Målt grunn til å
|
||||
overvåke:** manageren parser ledgeren med en håndrullet JSON-skraper (`_extract_json`,
|
||||
`_magentic.py:406-449`, `ast.literal_eval`-fallback), 3 forsøk (`:588-590`, `:710-723`) → `RuntimeError`;
|
||||
`MagenticProgressLedgerItem.answer` er ikke typesjekket per felt (`:300-302`). Egen manager FØRST når en
|
||||
målt mangel krever det.
|
||||
|
||||
**Hvorfor `manager_agent_factory=` og ikke `manager_agent=`:** målt E4 (V2 §6) — `manager_agent=`
|
||||
konstruerer manageren eagerly (`:1683`) og gir SAMME instans til hvert `build()` (`:1729-1730`);
|
||||
factory-varianten kalles per build (`:1732-1738`).
|
||||
|
||||
### C.2 Progress ledger som styringssignal, koblet mot `budget.py`
|
||||
|
||||
Ledgeren (`_magentic.py:310-314`, navn verifisert): `is_request_satisfied` / `is_in_loop` /
|
||||
`is_progress_being_made` / `next_speaker` / `instruction_or_question`. Stall-regelen (`:1109-1118`):
|
||||
`not is_progress_being_made or is_in_loop` → `stall_count += 1`, ellers **dekrementer** (ikke
|
||||
nullstill); `stall_count > max_stall_count` (strengt `>`) → `_reset_and_replan` (`:1149`): kontekst
|
||||
nullstilles, `reset_count += 1`, replan, plan review på nytt med `is_stalled=True` hvis aktivert.
|
||||
|
||||
To signaler, to vakter, aldri slått sammen (S3.4-invarianten):
|
||||
|
||||
| Signal | Kilde | Vakt | Utfall |
|
||||
|---|---|---|---|
|
||||
| **Semantisk** — stall | ledger | `max_stall_count` → reset+replan → `max_reset_count` | `exploration_stop="stalled"` — strukturert; IKKE krasj, IKKE suksess |
|
||||
| **Ressurs** — tokens | `BudgetMiddleware` (`budget.py:228`) på HVER agent **inkl. manager** | pre-call-guard `:242-244` | `BudgetExceeded(kind="tokens")` → 429-kanalen |
|
||||
| **Ressurs** — runder | Magentic `max_round_count` (= `Budget.max_rounds`) | intern (`:1238`) | **oversettes** av vårt lag til `BudgetExceeded(kind="exploration_rounds", limit, observed)` |
|
||||
|
||||
**Målt grunn til oversettelsen (E5, V2 §4):** `max_round_count`/`max_reset_count` gir **ingen
|
||||
exception** — en kanonisk assistent-melding «Workflow terminated due to reaching maximum round
|
||||
count.» + `_terminated=True` (`:1222-1256`), uten `prepare_final_answer`. Ved transporten er det
|
||||
uskillbart fra suksess. Vårt lag leser derfor `round_count` fra `MagenticContext` (eksponert via
|
||||
`PROGRESS_LEDGER_UPDATED`-eventene, ett per runde) og `orchestrator._terminated`, og produserer det
|
||||
typede stoppet SELV — slik `BudgetExceeded`/429 alt gjør for debatten. Strengen pinnes mot den
|
||||
installerte konstanten (`ROUND_CAP_LOGGER`-presedensen), aldri fri-tekst-matchet.
|
||||
|
||||
**`tick_round` er IKKE koblet til orkestreringsrunder i dag** (sømkart: to kallere, begge i
|
||||
`generate.py:390/:455`; `run.py:632` gir meteret `max(max_rounds*4, 4)` for GENERERINGS-forsøk).
|
||||
Utforskningen får derfor sin egen `Budget` (tokens + runder) i en egen `TokenMeter`, og `run_project`
|
||||
sitt meter er urørt. Stall-telleren er IKKE en budsjettvakt og kobles ikke inn i `TokenMeter`.
|
||||
|
||||
Hver runde logges (ledger + `next_speaker` + `instruction_or_question`) til
|
||||
`{run_id}-exploration.json`, så en stoppet utforskning er lesbar uansett hvilken vakt som fyrte.
|
||||
|
||||
**⚠ Spike S2:** manageren er et `Agent` (`StandardMagenticManager(agent=…)`, `:528`; ett
|
||||
`create_session()` i `__init__`, `:572`; alle kall via `agent.run(…, session=…)`, `:601`), så
|
||||
agent-nivå `ChatMiddleware` SKAL fyre — men det er **ikke målt** for manager-stien, og en feil her
|
||||
gjør «harde token-tak» til en løgn for den mest pratsomme deltakeren. Kontroll: bar
|
||||
`BaseChatClient` no-op-er middleware (`simulation.py:373-375`) — spiken bruker `ScriptedChatClient`.
|
||||
|
||||
### C.3 Grensene — B4 bygges ikke bort
|
||||
|
||||
| Dagens | Magentic (installert) | Regel |
|
||||
|---|---|---|
|
||||
| `with_max_rounds(max_rounds)` (`workflow.py:113`) | `max_round_count` (ctor-kwarg; default `None` = ubegrenset, `:542`) | PÅKREVD, positiv, fail-fast (G1/B4/A4) |
|
||||
| — | `max_stall_count` (default 3, `:540`) | PÅKREVD, positiv — ingen arv av defaulten |
|
||||
| — | `max_reset_count` (default `None`, `:541`) | PÅKREVD, ≥0 |
|
||||
| termination-safety-net (`:108`) | manageren avslutter ved `is_request_satisfied` (`:1104-1107`) **eller** ved ukjent `next_speaker` (`:1128-1131` — stille sluttsvar!) | rundetaket ER nettet; ukjent taler logges som avvik |
|
||||
| — | plan review: **ingen revise-cap, revises telles ikke som runder** (målt E6: 2 manager-kall per revise, 0 ledger-kall) | NY `max_plan_revisions` i kontrakten; hver revise `tick_round()`-es |
|
||||
|
||||
`ExplorationContract` (pydantic, fail-fast, speiler `contracts.TerminationContract`): `max_rounds`,
|
||||
`max_tokens`, `max_stall_count`, `max_reset_count`, `max_plan_revisions`, `enable_plan_review`.
|
||||
Ingen defaults: en utforskning uten uttalte grenser nekter å starte.
|
||||
|
||||
### C.4 B7 — tilstandsisolasjon. MÅLT (V2 §6; skript i V1)
|
||||
|
||||
| Eksperiment | Resultat | Konsekvens |
|
||||
|---|---|---|
|
||||
| **E1** én bygget workflow, to `.run()` | run 2 → `RuntimeError: This Magentic workflow has already been completed…` (`_magentic.py:914-918`), **0 modellkall** | single-use — STERKERE enn GroupChat 1.9.0s stille tomme kjøringer [2,0,0]; kan ikke korrumpere, for den kan ikke kjøre |
|
||||
| **E2** to workflows, ÉN delt `StandardMagenticManager` | run 2s manager-kall ser run 1s oppgave **4/4**; meldingstall vokser 2→…→23→**26**→39 over kjøringsgrensen; run 2s første ledger svarte `satisfied=True` og **workeren ble aldri kalt (0)** — et fabrikkert sluttsvar | ikke bare kost: **en plausibel fasit produsert av null arbeid** |
|
||||
| **E3** kontroll, fersk manager per build | **0/5** kontaminering, 5 kall inkl. `ledger_UNSAT` (workeren KALT) | 5-vs-4-kallformen er diskriminatoren |
|
||||
| **E4** én `MagenticBuilder`, to `.build()` | samme 4/4-blødning som E2 (`:1683`, `:1729-1730`) | «ny workflow» er ikke nok — **ny BUILDER per oppgave**, eller `manager_agent_factory=` |
|
||||
| **E7** `MagenticResetSignal` | `_magentic.py:1366` skriver `self._agent_thread` — lest av INGENTING (1 treff repo-vidt; den levende er `AgentExecutor._session`, 7 treff). Målt: cache/full_conversation tømt, **sesjonsidentitet uendret** | stall-replan gir fersk manager-ledger men **gamle deltakere**; docstringens løfte (`:1341-1344`) holder ikke i 1.0.0 |
|
||||
|
||||
**Regel:** per utforskning **ny builder → ny build → én kjøring**, fersk klient per agent (som
|
||||
`fresh_workflow`/`_default_factory` alt gjør, `run.py:436-440`). Stol aldri på stall-reset for å
|
||||
tømme deltakerminne. `fresh_exploration_workflow()` speiler `fresh_workflow()` (`workflow.py:74`).
|
||||
|
||||
### C.5 HITL-spranget: fra JA/NEI til «be om svar, bruke svarene»
|
||||
|
||||
**Installert flyt (målt E6, V2 §5):** `with_plan_review()`/`enable_plan_review=True` →
|
||||
`ctx.request_info(MagenticPlanReviewRequest{plan, current_progress, is_stalled},
|
||||
MagenticPlanReviewResponse)` (`:1041-1048`) → event `type="request_info"` med `request_id` →
|
||||
kjøringen STOPPER uten output → `workflow.run(responses={request_id: req.approve()})` eller
|
||||
`req.revise("…")` (`_workflow.py:899-908`) → ved revise: feedback inn i historikken, `manager.replan`,
|
||||
`REPLANNED`-event, **ny review-forespørsel** (`:1014-1031`) → ved approve: løkka kjører.
|
||||
Dette er målbildets «be om svar, bruke svarene og jobbe videre» ordrett — og det virker på 1.0.0.
|
||||
|
||||
**Ingen landingsplass i dag** (sømkart §4): `with_request_info` bygges men ingenting konsumerer
|
||||
strømmen; `hitl.py` er et offline inspeksjonsverktøy (id-join outbox − inbox, `hitl.py:159`), ikke
|
||||
en forespørsel/svar-mekanisme. Koblingen mot den asynkrone køa **gjenbruker formen, ikke filene:**
|
||||
|
||||
| Eksisterende | Utforskningen |
|
||||
|---|---|
|
||||
| outbox `{run_id}-outcome.json` med `verdict_id` | **NY artefaktklasse** `{run_id}-plan-review.json` = `{request_id, plan, current_progress, is_stalled, checkpoint_id}` — samme RAW-lag, samme mappe, byte-deterministisk via `outbox._dump` |
|
||||
| inbox `{id}.json` (`verdicts.write_verdict`) | `{request_id}.json` = `{decision: approve\|revise, feedback}` — eksperten SKRIVER, systemet LESER (§3 Steg 7-rolledelingen) |
|
||||
| `hitl.py pending` (id-join) | samme join over den nye klassen; `python -m portfolio_optimiser.hitl pending` viser begge |
|
||||
| `FeedbackContract.decision ∈ {approved, rejected}` | **IKKE gjenbrukt** — en plan-revisjon er ikke en dom; eget lite pydantic-skjema |
|
||||
| `simulation`-personaen (`load_persona_example`) | spiller eksperten SYNKRONT i demoen: `revise("Test også …")` én gang, så `approve()` |
|
||||
|
||||
To tidsskalaer, som Steg 7: **synkron** (persona/operatør i samme prosess — demoen og første
|
||||
leveranse) og **asynkron** (fil-innboks over dager → krever U12: `checkpoint_id` i artefaktet,
|
||||
`run(responses=…, checkpoint_id=…, checkpoint_storage=…)`). Det er DÉT som flytter U12 fra «NEI» til
|
||||
«nå» — og spike S4 avgjør om 1.9.0 bærer det.
|
||||
|
||||
**Svar MIDT i kjøringen (ikke bare ved plan):** plan review er det ENESTE `request_info`
|
||||
orkestratoren selv reiser (4 treff i `_magentic.py`, 0 i basen). Men `AgentApprovalExecutor`
|
||||
(`_orchestration_request_info.py:168`, `propagate_request=True` `:201`) kan stå som DELTAKER:
|
||||
når manageren velger `expert_liaison`, blir dens output en `request_info`, og mennesket svarer med
|
||||
`AgentRequestInfoResponse.approve()`/`.from_strings(…)` (`:55-79`) — svaret blir deltakerens
|
||||
output i historikken. **Ikke målt** → spike S3b. Lander den, er «stille spørsmål … be om svar»
|
||||
dekket både ved plan og underveis.
|
||||
|
||||
### C.6 «Hypoteser fagpersoner kommer med» — tre inngangsdører, i kostnadsrekkefølge
|
||||
|
||||
1. **I mandatet (finnes):** `Approach{label, description}` → proposeren verbatim. Utforskningen tar
|
||||
frø-approaches som INPUT og **MÅ bevare dem i output-mandatet** (en hypotese fagpersonen ba om
|
||||
forsvinner aldri stille — `not_evaluated`-regelen anvendt på utforskning).
|
||||
2. **I plan review (U13):** `revise("Test hypotesen at …")` → replan med teksten i task-ledgeren.
|
||||
3. **Underveis (S3b):** `expert_liaison`-deltakeren — manageren SPØR, fagpersonen svarer, svaret
|
||||
blir kontekst for neste runde.
|
||||
4. **I kunnskapsbasen (senere, commons-sak):** `type: hypothesis`-konseptfil navigatoren finner.
|
||||
Krever OKF-vokabular-beslutning hos commons — **ikke i denne planen**.
|
||||
|
||||
### C.7 OKF-bundles som inngang — hva manageren må vite
|
||||
|
||||
Manageren planlegger mot en **katalog**, aldri innholdet (§3 Steg 1). Per bundle: `id` ·
|
||||
`Bundle.index_summary()` (`okf.py:154`) · kandidat-features (`verdicts.bundle_candidate_features`,
|
||||
`:482`) · **om `cost-baseline.json` finnes** (`okf.load_optional_cost_baseline`, `:411`) — avgjørende:
|
||||
en uforankret bundle kan ikke quick-valideres mot stage 0, og manageren skal vite hvilke hypoteser
|
||||
som KAN testes deterministisk · antall `type: verdict`-filer (`Bundle.verdicts`). Navigatoren åpner
|
||||
én bundle om gangen på managerens instruks. **Multi-bundle er NYTT:** `run_project` tar ÉN
|
||||
`bundle_dir` (`run.py:514`; `bundle_dirs` → 0 treff); utforskningen tar `bundle_dirs`, hver
|
||||
approach bærer `bundle_id`, og pipelinen kjøres per bundle som i dag (`run_portfolio`-formen).
|
||||
|
||||
### C.8 Modulplassering (for byggeøktene, ikke for denne)
|
||||
|
||||
`src/portfolio_optimiser/explore.py` (NY): `ExplorationContract`, `ExplorationResult{mandate,
|
||||
ledger_log, stop, plan_reviews}`, `fresh_exploration_workflow(...)`, `explore(...)`. Importerer
|
||||
`agent_framework.orchestrations` — **aldri** fra `okf.py`/`mandate.py`/`hitl.py` (MAF-fri-guarden
|
||||
`test_okf_is_maf_free`). Verktøyene (`quick_validate`, navigator-tools) bor i `explore.py` og
|
||||
wrapper de rene modulene. `run.py` får ett nytt kallsted (`--explore "<prompt>"` → `explore()` →
|
||||
`run_project(mandate=…)`), `hosting.py` ett nytt whitelistet felt (`explore_prompt`), og
|
||||
`simulation.py` ett nytt scenario. Demo-transkriptet (`tests/golden/demo-transcript.stdout`) er
|
||||
**byte-uendret** fordi utforskningen er opt-in.
|
||||
|
||||
---
|
||||
|
||||
## D. Prioritert vei gjennom de ni gapene
|
||||
|
||||
### D.0 Versjonsbeslutningen (FØR noe bygges — én avgjørelse, ikke en glidning)
|
||||
|
||||
**Anbefaling: bygg mot 1.0.x (installert) i denne planen.** (1) core-bumpen 1.9→1.15 er seks
|
||||
minor-versjoner over privat-API-premisser `test_maf_version_guard` vokter, og tre målte
|
||||
1.9-egenskaper (`ExperimentalWarning`-paret, single-use GroupChat [2,0,0], middleware-no-op på bar
|
||||
klient) må re-måles; (2) en bump er en egen beslutning, ikke et biprodukt av sløyfa; (3) **1.0.1 er et
|
||||
patch-hopp på samme core** som kan tas gratis
|
||||
hvis `uv lock --upgrade-package agent-framework-orchestrations` gir 1.0.1 og suiten er grønn — og
|
||||
som KAN bære #6297 (§B pkt. 7) og/eller E7-fiksen. Det er spike S0. Bumpen til 1.1.x/core 1.15 er
|
||||
en EGEN økt etter denne planen (bonus: `InvocationsHostServer` krever core ≥1.13) — ikke en forutsetning.
|
||||
|
||||
### D.1 Spikes (økt 54 — alle i `spikes/e_magentic.py` + `tests/spikes/`, aldri `src/`)
|
||||
|
||||
| # | Spørsmål | Måling | Felles hvis |
|
||||
|---|---|---|---|
|
||||
| S0 | Gir 1.0.1 samme API og fjerner den E2/E4/E7? | `uv lock --upgrade-package agent-framework-orchestrations` → `uv pip list`; re-kjør V1-skriptene `magentic_b7*.py` + `magentic_reset.py`; `uv run pytest -q` | E7 står → bli på 1.0.0, noter |
|
||||
| S1 | B7 i repoets egen form | E1–E4 + E7 portert til `spikes/` med `ScriptedChatClient` (ikke agentens `FakeClient`) | 4/4 på delt builder, 0/5 på fersk |
|
||||
| S2 | Fyrer `BudgetMiddleware` på MANAGERENS kall? | manager-`Agent(middleware=[BudgetMiddleware(meter)])`, `Budget(max_tokens=1, …)`; assert `BudgetExceeded` propagerer ut av `workflow.run` OG at `meter.tokens > 0` etter ett kall | middleware fyrer ikke → egen klient-wrapper; plan-endring |
|
||||
| S3 | Plan-review-rundtur i repoets form | E6 portert: request_info-event → `run(responses=…)` med `revise` så `approve`; tell manager-kall per revise (målt 2) | rundturen feiler på 1.0.0 |
|
||||
| S3b | `AgentApprovalExecutor` som deltaker | svar injiseres via `from_strings("svar")`, assert svaret står i neste manager-prompt | ikke nåbar → kun plan review (dør 1+2) |
|
||||
| S4 | Resume med pending plan review | `with_checkpointing(FileCheckpointStorage(tmp))`, kjør til review, **ny prosess** (subprosess), `run(responses=…, checkpoint_id=…, checkpoint_storage=…)`; assert `request_info` re-emittert + sluttsvar | feiler → U12 synkron-først, asynkron etter planen |
|
||||
| S5 | `quick_validate`-latens | `validate_proposal` ×20 på `bygg-energi-baseline-mikro` med `time.perf_counter` | > 2 s/kall → budsjettér kall i kontrakten |
|
||||
| S6 | OTEL gratis? | `uv add opentelemetry-sdk` i scratch-venv; `configure_otel_providers(enable_console_exporters=True)`; kjør demoen; tell spans | ingen `workflow.run`-span → U14 er større enn én økt |
|
||||
|
||||
### D.2 Rekkefølgen
|
||||
|
||||
| # | Hva | Gap | Load-bearing for målbildet? | Økt | Modell |
|
||||
|---|---|---|---|---|---|
|
||||
| 0 | **Spikes S0–S6** | U4/U13/U12/U8/U14 | JA — hver er en antakelse planen ellers bygger blindt på | 54 | Opus 5/xhigh |
|
||||
| 1 | **U14 OpenTelemetry** — `opentelemetry-sdk` som deklarert dep, `configure_otel_providers` bak `PORTFOLIO_OTEL=…` (console default; OTLP KUN med eksplisitt `OTEL_EXPORTER_OTLP_ENDPOINT` — «ingen stille egress»), spans på DAGENS pipeline først; egne events for `PLAN_CREATED`/`REPLANNED`/`PROGRESS_LEDGER_UPDATED` (målt: `_magentic.py` emitterer null spans selv) | U14 | JA for «stole på» — og baselinen U15 måler mot | 55 | Opus 5/high |
|
||||
| 2 | **U4 + U13 synkron** — `explore.py` (C.8), `ExplorationContract`, mandat-former, `quick_validate`, navigator-tools, plan review med persona; `simulation`-scenario; `--explore` i CLI | U4, U13 | JA — dette ER målbildet | 56–57 | Opus 5/xhigh |
|
||||
| 3 | **U12 + asynkron U13** — `FileCheckpointStorage`, `{run_id}-plan-review.json` + innboks-svar, `hitl.py pending` utvidet, `--resume <run_id>` | U12, U13 | JA for svar over dager — **betinget av S4** | 58 | Opus 5/xhigh |
|
||||
| 4 | **U15** — EGEN sammenligning på golden-bundelen: utforskning vs. debatt på tre tall (validerte kandidater, tokens, runder), skrevet til `docs/` som måling | U15 | JA for «finner den bedre tiltak» — men MAF-evals AVVISES (eksperimentell i kode, «wrong shape», kapabilitetskart :58) | 59 | Opus 5/high |
|
||||
| — | U1 `ConcurrentBuilder` | U1 | NEI — bølgemodellen er målt, bærer budsjett-reservasjon (S3.4); Concurrent har ingen runder å reservere mot | etter planen, om noensinne | — |
|
||||
| — | U2 `WorkflowBuilder` | U2 | NEI — Magentic ER graf-laget; direkte adopsjon gir null ny kapabilitet | nei | — |
|
||||
| — | U5 `SkillsProvider` | U5 | NEI — `ExperimentalFeature.SKILLS`; egen loader virker; commons eier innholdet | etter planen | — |
|
||||
| — | U11 MAF RAG-providere | U11 | NEI — «navigate, never stuff» forbyr query-time retrieval; sitatbæring finnes | avvist med begrunnelse | — |
|
||||
|
||||
Seks økter (54–59); faller S4, sparer pkt. 3 én økt.
|
||||
|
||||
### D.3 For og mot de fire kandidatene ordren navngir
|
||||
|
||||
**U4 + U13 tidlig — FOR:** selve målbildet; alt annet er instrumentering av status quo. **MOT:** hviler
|
||||
på umålte antakelser (S2, S3b, S4) og på en `request_info`-sti repoet ALDRI har kjørt. Derfor spikes
|
||||
først — ikke omvendt. **Plass #2 og #3.**
|
||||
|
||||
**U14 tidlig — FOR:** (1) repoets deploy-gate; (2) uten sporing er en utforskning en svart boks — og
|
||||
Magentic-koden gir selv null spans, så ledger-eventene må vi logge; (3) LITE i kode: én funksjon +
|
||||
env; (4) instrumentert FØR utforskningen gir U15 sin baseline. **MOT (målt):** det er **ikke gratis i
|
||||
avhengigheter** — `opentelemetry-sdk` + exporter må deklareres (wheel-vekt, `uv sync`, handover-
|
||||
pakka), og `ENABLE_INSTRUMENTATION` er default `True` (`observability.py:697`) så spans LAGES alt i
|
||||
dag og kastes. OTLP-exporter er egress og må være opt-in bak eksplisitt endepunkt. **Plass #1,
|
||||
betinget av S6.**
|
||||
|
||||
**U15 — FOR:** uten måling kan vi ikke påstå at sløyfa er bedre; ærlighetsregelen forbyr påstanden.
|
||||
**MOT `evaluate_workflow`:** eksperimentell i kode (20 dekoratorer) og feil form (kvalitetsscore
|
||||
offline, ikke domeneobjekt). **Konklusjon:** evaluering JA, som egen sammenligning; MAF-flaten
|
||||
NEI inntil `EVALS` forlater `ExperimentalFeature`. **Plass #4** — den trenger noe å måle.
|
||||
|
||||
**U12 — FOR:** asynkron plan review over dager er umulig uten; mekanismen dekker Magentic-tilstand
|
||||
(målt i kode, §A.1). **MOT:** ende-til-ende-resume på 1.9.0 er umålt, restore-feil svelges, og en
|
||||
synkron persona-variant leverer målbildet i demoen uten den. **Konklusjon:** S4 avgjør; bygg i #3 hvis
|
||||
grønn, ellers synkron-først og U12 etter planen, sammen med core-bumpen.
|
||||
|
||||
### D.4 Det som IKKE gjøres (uttalt)
|
||||
|
||||
- Ingen endring i `shared/method-spec.md`. Utforskningen er et lag OVER §3; en amendment
|
||||
(«Step 0 — Explore, optional») foreslås til commons via coord (operatørbeslutning, §F) — planen
|
||||
avhenger ikke av svaret.
|
||||
- Ingen D7-speiling nå: po-claude er PARKERT (STATE); utforskningslaget meldes dit når parkeringen
|
||||
oppheves, på linje med Steg-5-returtypen og demo-innboksen som også venter.
|
||||
- Ingen erstatning av GroupChat-debatten (Steg 3 står). Ingen `open/`-push, versjonsbump, Azure.
|
||||
- Ingen egen `MagenticManagerBase`-subklasse før en målt mangel krever det.
|
||||
|
||||
---
|
||||
|
||||
## E. Verifisering — testbare kriterier (kommandoer, ikke «sjekk at det virker»)
|
||||
|
||||
**Denne økten (planen selv):**
|
||||
|
||||
| # | Kriterium | Kommando → forventet |
|
||||
|---|---|---|
|
||||
| E1 | Nevneren er 32 | `find src -name '*.py' \| wc -l` → `32` |
|
||||
| E2 | Hver null-spørring kan finne | `bash scratchpad/u_measure.sh` (V1) → `venv_files` ≥ 1 for hver av de 9 null-regexene |
|
||||
| E3 | Magentic ikke kode-markert | `grep -c -i "experimental" .venv/lib/python3.12/site-packages/agent_framework_orchestrations/_magentic.py` → `0`; `grep -n "SKILLS\|EVALS" …/agent_framework/_feature_stage.py` → `:54`, `:64` |
|
||||
| E4 | Installert = lås | `uv pip list \| grep orchestrations` → `1.0.0`; `sed -n '61,63p' uv.lock` → `version = "1.0.0"` |
|
||||
| E5 | Single-use + builder-blødning reproduserbar | `uv run python scratchpad/magentic_b7.py` → `E1 run2 … RuntimeError`, `E2 VERDICT … 4/4`, `E3 VERDICT … 0/5` |
|
||||
| E6 | Plan-review-rundtur reproduserbar | `uv run python scratchpad/magentic_b7b.py` → `E6 after APPROVE outputs: ['FINAL: …']`, `E5 outputs: ['Workflow terminated due to reaching maximum round count.']` |
|
||||
| E7 | OTEL-sdk mangler | `.venv/bin/python -c "import opentelemetry.sdk"` → `ModuleNotFoundError` |
|
||||
| E8 | Planen er dok-gate-klassifisert | `uv run pytest -q tests/test_doc_constant_sync_loadbearing.py` → grønn (datert sti = arkiv-klasse) |
|
||||
| E9 | Treet urørt utenom planen | `git status --porcelain` → kun `docs/plan/2026-08-23-…md` + den fremmede `presentasjon`-HTML-en |
|
||||
|
||||
**Byggeøktene (kriterier som FELLES når sømmen detaches — load-bearing-metoden):**
|
||||
|
||||
| Leveranse | Rødt-først-kriterium |
|
||||
|---|---|
|
||||
| S1/B7 | test som deler builder mellom to utforskninger er RØD (4/4); `fresh_exploration_workflow` gjør den grønn (0/N) |
|
||||
| S2/budsjett | `Budget(max_tokens=1)` på manager-agenten → `BudgetExceeded` ut av `explore()`; detach middleware på manageren → testen rød |
|
||||
| C.2/runder | `max_rounds=1` → `BudgetExceeded(kind="exploration_rounds", limit=1, observed=1)`; detach oversettelsen → kjøringen «lykkes» med den kanoniske strengen → rød |
|
||||
| C.3/kontrakt | `ExplorationContract(max_stall_count=None)` → `ValidationError` ved konstruksjon |
|
||||
| C.5/plan review | persona-`revise` én gang → `REPLANNED` i loggen og `plan_reviews == 2`; `max_plan_revisions=1` + persona som alltid reviderer → typet stopp, aldri evig løkke |
|
||||
| C.6/frø | frø-approach `id="fagperson-1"` finnes i output-`Mandate.approaches` uansett hva manageren fant; detach bevaringen → rød |
|
||||
| C.0/nivå 3 | `explore()` skriver ingenting til outbox/wiki: `ls outbox` før/etter → kun `{run_id}-exploration.json`; `git status shared/` ren |
|
||||
| U12/S4 | subprosess-resume: pending `request_info` re-emittert + sluttsvar; detach `checkpoint_storage` → rød |
|
||||
| U14 | demoen under `PORTFOLIO_OTEL=console` → ≥1 `workflow.run`-span på stderr; uten variabelen → stdout byte-identisk med `tests/golden/demo-transcript.stdout` |
|
||||
| Golden | `uv run pytest -q` grønn; `shasum tests/golden/demo-transcript.stdout` uendret |
|
||||
|
||||
---
|
||||
|
||||
## F. Nøkkelantakelser, risiko og åpne beslutninger
|
||||
|
||||
**Alle åtte radene er MÅLT i økt 54** (ordre `20260823T162224Z`, spikes S0–S6). Måleapparatet er
|
||||
`spikes/e_magentic.py` + `tests/spikes/test_e_magentic.py` (16 tester), kjørt mot orchestrations
|
||||
**1.0.1** på core 1.9.0; hele suiten 920 passed / 5 skipped, golden-transkriptet uendret
|
||||
(`ea8c534773acdbe41ae68f2c55724d69aaf8be4f`).
|
||||
|
||||
| # | Antakelse | Test | Status |
|
||||
|---|---|---|---|
|
||||
| A1 | Agent-nivå `ChatMiddleware` fyrer på managerens kall | S2 | **MÅLT GRØNN.** `BudgetMiddleware` på manager-agenten krediterer meteret (`meter.tokens == 8` etter ett kall) og stopper kjøringen. Detach-kontroll: uten middleware fullfører SAMME 1-token-budsjett. Planens budsjettgaranti står. |
|
||||
| A2 | `BudgetExceeded` reist inni en Magentic-deltaker propagerer ut av `workflow.run` | S2 | **MÅLT GRØNN, og som repoets EGEN type** — ikke pakket i en `ExceptionGroup`: `isinstance(exc, BudgetExceeded)`, `kind="tokens"`, `limit=1`, `observed=8`. Trippelen kø-(y) leser er intakt, så 429-kanalen kan brukes uendret. |
|
||||
| A3 | `request_info`-rundtur | S3 | **MÅLT GRØNN på 1.0.1.** Review stopper kjøringen uten output (manageren har da kun kalt `facts`+`plan`); `revise` koster nøyaktig 2 manager-kall (`facts_update`, `plan_update`), **null ledger-kall og null runder**, og **spør på nytt**; `approve` kjører løkka til sluttsvar. Bekrefter at `max_plan_revisions` MÅ inn i kontrakten — ellers er en alltid-reviderende ekspert et ubundet forbruk. |
|
||||
| A4 | Resume med pending plan review i NY prosess | S4 | **MÅLT GRØNN.** Foreldreprosessen stopper på review og etterlater checkpoints; en subprosess (`spikes/e_magentic_resume.py`) som aldri så kjøringen svarer fra checkpointen alene og driver workflowen til sluttsvar. **Pris, ikke forutsett:** `FileCheckpointStorage` NEKTER å deserialisere `MagenticPlanReviewRequest`/`…Response` uten at begge navngis i `allowed_checkpoint_types` — uten det er checkpoint-fila uleselig og listingen TOM, altså en resume som feiler som et FRAVÆR. Begge prosesser må deklarere dem. |
|
||||
| A5 | `AgentApprovalExecutor` som deltaker gir svar midt i kjøringen | S3b | **MÅLT GRØNN — dør 3 i C.6 står.** Ekspertens ord når både liaisonen og en senere manager-prompt. **To rundturer per menneskesvar:** `from_strings([svar])` mater svaret tilbake INN i liaisonen og gjenopptar IKKE manageren (målt: null manager-kall mellom de to forespørslene); først `approve()` sender liaisonens output videre. **Kostnad:** `AgentApprovalExecutor` er IKKE re-eksportert fra `agent_framework.orchestrations` (svartypen ER det) — døra koster i dag en privat-API-import. |
|
||||
| A6 | Persona-manuset kan drive en Magentic-manager (ledger-JSON) offline | S1/S3 | **MÅLT GRØNN med repoets `ScriptedChatClient`** (ikke lenger en ad-hoc `FakeClient`). Ett forbehold funnet: selectoren får den SAMMENSLÅTTE prompten, så ett av fem manager-kall bærer to markører — rutingen må teste senere-stadium-markøren først. Og en `next_speaker` som ikke matcher en deltaker gir **stille sluttsvar uten at noen ble spurt** (`_magentic.py:1128-1131`), målt da liaison-spiken først het `worker`. |
|
||||
| A7 | Utforskning som opt-in holder demo-transkriptet byte-uendret | `shasum` | **MÅLT uendret** gjennom hele økten (`ea8c534…`). |
|
||||
| A8 | 1.0.1 er API-identisk med 1.0.0 | S0 | **MÅLT: API-identisk, ATFERD ikke.** Diffen er upstream-regresjonsfiksen #4371: `StandardMagenticManager.__init__` mistet `self._session = agent.create_session()`, og hvert manager-kall mynter nå en engangs-sesjon. **Konsekvens: E2 OG E4 er BORTE** (4/4 → **0/5** begge). E1 (single-use `RuntimeError`, null kall) og E7 (`MagenticResetSignal` skriver til orphan-attributtet `_agent_thread`, `:1369`, sesjonsidentitet uendret) står. Manager-sesjonen fjernet også fra checkpoint-state, konsistent med at manageren nå er tilstandsløs per kall. |
|
||||
|
||||
**Målt korreksjon til E-tabellen (U12/S4):** kriteriet «detach `checkpoint_storage` → rød» er
|
||||
FEIL — å fjerne `checkpoint_storage=` fra `run()` lar HELE suiten stå grønn (920 passed), fordi
|
||||
`.with_checkpointing(...)` på builderen allerede ga workflowen lageret. De to bærende punktene er
|
||||
`checkpoint_id=` (fjernes → rød) og builderens `.with_checkpointing(...)` (fjernes → rød).
|
||||
|
||||
**Load-bearing MÅLT** (mot HELE suiten, grønn kontroll 920/5): `manager_keeps_persistent_session`
|
||||
konstant `True` (2 røde — E2+E4 alene) · aldri fest `BudgetMiddleware` på manageren (3 røde, mens
|
||||
detach-armen forblir grønn) · detach markør-registreringen (1 rød — S3b-positiven alene, kontrollen
|
||||
grønn) · flipp `_route`-rekkefølgen så `pre-survey` testes først (4 røde) · resume uten
|
||||
`checkpoint_id=` (1 rød) · builder uten `.with_checkpointing()` (1 rød) · tom
|
||||
`_ALLOWED_CHECKPOINT_TYPES` (1 rød) · og den falsifiserte: resume uten `checkpoint_storage=` (**0
|
||||
røde** — funnet over).
|
||||
|
||||
**S5 — `quick_validate`-latens: median 13,6 ms** over 20 kall på `bygg-energi-baseline-mikro`
|
||||
(forankret baseline + assumption-bånd, så stage 0 + CBC + 512-sample Monte Carlo er alle med).
|
||||
Langt under 2 s-terskelen: verktøyet kan kalles fritt i løkka, og kontrakten trenger ingen
|
||||
egen latens-post. Båndet er med med vilje — uten det faller `_monte_carlo` tilbake på
|
||||
`item.unit_cost`, alle draw blir identiske, og tallet ville underrapportert den ekte kostnaden.
|
||||
|
||||
**S6 — OTEL er gratis, men IKKE via `enable_console_exporters`.** I et scratch-venv pinnet til
|
||||
samme stack (core 1.9.0 / orch 1.0.1 / `opentelemetry-sdk` 1.44.0; **ingenting lagt til
|
||||
`pyproject.toml`**) gir `configure_otel_providers(enable_console_exporters=True)` **2
|
||||
`workflow.run`-spans** — pluss `workflow.build`, `executor.process`, `edge_group.process`,
|
||||
`message.send`, `invoke_agent`, `chat synthetic` — men de skrives til **stdout**, som ville
|
||||
ødelagt golden-transkriptet. Med `exporters=[ConsoleSpanExporter(out=sys.stderr)]` kommer de 2
|
||||
`workflow.run`-spanene på **stderr** og demoens stdout er **byte-identisk med fasiten** (samme
|
||||
shasum). U14 er altså én økts arbeid — forutsatt at exporteren konstrueres eksplisitt mot stderr,
|
||||
aldri via flagget.
|
||||
|
||||
**U14 er LANDET (økt 55, ordre `20260823T165757Z-6762213199`).** Operatøren tok alle fire
|
||||
beslutningene som anbefalt 23.08, og bekreftet samme dag at **låsen på orchestrations 1.0.1 er
|
||||
ENDELIG, ikke midlertidig** — spike-ordrens «revert hvis E7 står» er overstyrt av den senere
|
||||
beslutningen, som betinget på grønn suite (920/5, målt) og ikke på E7. Levert:
|
||||
|
||||
* `opentelemetry-sdk>=1.42,<2` som DEKLARERT dependency (beslutning 2). Én ny pakke, ikke to:
|
||||
`ConsoleSpanExporter` bor inne i sdk-en. `opentelemetry-api` fulgte med fra 1.42.1 til 1.44.0 —
|
||||
målt uskadelig (golden `ea8c534…` uendret før og etter).
|
||||
* `src/portfolio_optimiser/tracing.py` — `PORTFOLIO_OTEL` på truthiness, modusene `console`
|
||||
(spans → stderr) og `otlp` (kun mot et navngitt endepunkt), `tracing_notice` som eneste renderer.
|
||||
* Wiret i TRE inngangspunkter: `run.main`, `simulation.main`, `hosting.main`.
|
||||
* **S6-forbeholdet er implementert som skrevet:** `ConsoleSpanExporter(out=…)`, aldri
|
||||
`enable_console_exporters=True`. Målt: demoen under `PORTFOLIO_OTEL=console` gir `workflow.run`-
|
||||
spans på stderr og stdout BYTE-IDENTISK med fasiten; uten variabelen null spans.
|
||||
* **To ting E-tabellen ikke forutså, begge lukket:** (i) MAF bygger env-avledede OTLP-exportere
|
||||
UBETINGET, så `console` NEKTER når en `OTEL_EXPORTER_OTLP_*_ENDPOINT` er satt — ellers var
|
||||
«console» en usann påstand om hvor kjøringens innhold tok veien; (ii) `enable_console_exporters`
|
||||
faller tilbake på `ENABLE_CONSOLE_EXPORTERS` i miljøet, så den pinnes eksplisitt `False` i begge
|
||||
moduser. Begge er MÅLT som defektklasser, ikke antatt.
|
||||
* **IKKE bygget, med grunn:** `PLAN_CREATED`/`REPLANNED`/`PROGRESS_LEDGER_UPDATED` — de hører til
|
||||
sløyfa U4 bygger, og en emitter uten kallsted er en form gjettet i stedet for målt. De landes i
|
||||
økt 56–57, på sømmen U14 nettopp la. OTLP-exporter-PAKKENE er bevisst ikke deklarert (egress +
|
||||
grpc/protobuf-vekt i et publisert wheel); uttalt ærlighets-grense i README/DEPLOY/env.template.
|
||||
|
||||
Suite etter U14: **943 passed / 5 skipped**, golden `ea8c534…` uendret, `mypy src` + `ruff` rene.
|
||||
|
||||
**Beslutninger som trenger operatøren (speiles i avslutningsblokken):**
|
||||
1. **Versjon — AVGJORT 23.08: låsen står på 1.0.1, ENDELIG.** Bakgrunnen: ordrens «felles hvis» sa
|
||||
at hvis E7 står i 1.0.1, revert til 1.0.0. E7 STÅR — men 1.0.1 fikset noe større enn det som ble
|
||||
håpet (E2/E4, «en plausibel fasit produsert av null arbeid»). Operatøren aksepterte 23.08 at
|
||||
gevinsten veier tyngre enn ordrens bokstav, og at betingelsen som gjelder er «grønn suite»
|
||||
(målt 920/5), ikke E7. **Ikke midlertidig, og ikke å gjenåpne uten en ny måling.** Bumpen til
|
||||
1.1.x / core 1.15 er fortsatt en EGEN økt etter denne planen.
|
||||
2. **U14 som deklarert avhengighet — AVGJORT 23.08: ja.** Landet i økt 55; se «U14 er LANDET»
|
||||
over. Kostnaden ble ÉN pakke (`opentelemetry-sdk`), ikke to.
|
||||
3. **HITL — AVGJORT 23.08: synkron-først.** Merk at ordrens BEGRUNNELSE er utdatert mens
|
||||
konklusjonen står: den sier «A4 er umålt og kun kode-lest», men A4 ble MÅLT GRØNN i økt 54, så
|
||||
asynkron er teknisk mulig. Valget er dermed ren rekkefølge, ikke risiko — asynkron U13 forblir
|
||||
pkt. 3 i rekkefølgen, betinget av S4 som nå er grønn.
|
||||
4. **Commons-amendment «Step 0 — Explore» — AVGJORT 23.08: sendt.** `.claude` sendte forslaget til
|
||||
`portfolio-optimiser-commons` via coord. Vi venter IKKE: planen slår selv fast at ingen amendment
|
||||
er påkrevd, og «Step 0» er foreslått som VALGFRI. Kommer den, er den en tilføyelse.
|
||||
|
||||
---
|
||||
|
||||
## Vedlegg V1 — målingsskript og rå output (U1–U19)
|
||||
|
||||
Skript (`scratchpad/u_measure.sh`, kjørt 2026-08-23; kopiert hit ordrett så målingen kan gjentas):
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
cd "$(git rev-parse --show-toplevel)"
|
||||
SRC_N=$(find src -name '*.py' | wc -l | tr -d ' ')
|
||||
TST_N=$(find tests -name '*.py' | wc -l | tr -d ' ')
|
||||
SPK_N=$(find spikes -name '*.py' | wc -l | tr -d ' ')
|
||||
echo "DENOMINATOR src=$SRC_N tests=$TST_N spikes=$SPK_N"
|
||||
m() { s=$(grep -rEn --include='*.py' "$2" src | wc -l | tr -d ' ')
|
||||
t=$(grep -rEn --include='*.py' "$2" tests | wc -l | tr -d ' ')
|
||||
k=$(grep -rEn --include='*.py' "$2" spikes | wc -l | tr -d ' ')
|
||||
printf "%-4s %-60s src=%-4s tests=%-4s spikes=%s\n" "$1" "$2" "$s" "$t" "$k"
|
||||
[ "$s" != "0" ] && grep -rEn --include='*.py' "$2" src | head -12; }
|
||||
m U1 'ConcurrentBuilder|BuildConcurrent|FanIn'
|
||||
m U1b 'asyncio\.gather|create_task\('
|
||||
m U2 'WorkflowBuilder|@executor|class .*\(Executor\)|WorkflowContext|SwitchCase|add_edge|add_fan_out'
|
||||
m U3 'GroupChatBuilder|with_max_rounds|termination_condition|RoundRobinGroupChatManager'
|
||||
m U4 'Magentic'
|
||||
m U5 'SkillsProvider|McpSkillsSource|AgentSkillsProvider|agent_framework.*skills'
|
||||
m U6 'MCPStdioTool|MCPStreamableHTTPTool|MCPWebsocketTool|as_mcp_server'
|
||||
m U7 'FunctionTool|ai_function|@tool|tools=\['
|
||||
m U8 'Middleware|middleware='
|
||||
m U9 'ContextProvider|HistoryProvider|context_providers='
|
||||
m U10 'AzureAISearch|azure\.search|cosmos|qdrant|redis|VectorStore|vector_store|Qdrant|Chroma|chromadb'
|
||||
m U11 'AIContextProvider|TextSearchProvider|citation|SourceName|SourceLink|agentic_retrieval|KnowledgeAgent'
|
||||
m U12 'checkpoint|Checkpoint'
|
||||
m U13 'approval_mode|request_info|RequestInfoEvent|RequestPort|MagenticPlanReview|ApprovalRequired'
|
||||
m U14 'opentelemetry|setup_observability|OBSERVABILITY|get_tracer|start_as_current_span'
|
||||
m U15 'evaluate_workflow|LocalEvaluator|Evaluator|_evaluation|agent_framework.*evaluation'
|
||||
m U16 'CompactionProvider|SlidingWindow'
|
||||
m U17 'as_tool\(|AsAIFunction'
|
||||
m U18 'FoundryChatClient'
|
||||
m U19 'HarnessAgent|ShellExecutor|FileMemoryStore|ToolApprovalAgent'
|
||||
```
|
||||
|
||||
Rå tellinger: `DENOMINATOR src=32 tests=117 spikes=6` · U1 0/4/4 · U1b 1/5/0 · U2 0/0/0 · U3 10/7/7 ·
|
||||
U4 0/4/9 · U5 0/0/0 · U5b(`SKILL\.md`) 1/5/0 · U6 6/6/0 · U6b(rå mcp) 7/6/0 · U7 4/10/0 · U8 16/20/0 ·
|
||||
U9 8/11/4 · U10 2/19/0 · U10b(numpy/cosine) 49/102/1 · U11 26/35/0 · U12 3/0/0 · U13 3/0/0 ·
|
||||
U14 0/0/1 · U15 0/0/0 · U16 0/0/0 · U17 0/0/0 · U18 7/8/0 · U19 0/0/0.
|
||||
|
||||
Kjent-positiv-kontroll mot installert MAF (`grep -rEl <regex> agent_framework agent_framework_orchestrations | wc -l`):
|
||||
U2 → 21 · U5 → 3 · U14 → 4 · U15 → 2 · U16 → 4 · U17 → 1 · U19 → 1 · `checkpoint_storage|CheckpointStorage`
|
||||
→ 14 · `MagenticBuilder|StandardMagenticManager` → 4.
|
||||
|
||||
B7-/HITL-målingsskriptene (`magentic_b7.py`, `magentic_b7b.py`, `magentic_reset.py`) ligger i
|
||||
øktens scratchpad og porteres til `spikes/e_magentic.py` i økt 54 (S1/S3) — de committes ikke
|
||||
herfra, fordi de bruker en ad-hoc `FakeClient` og ikke repoets `ScriptedChatClient`.
|
||||
|
||||
## Vedlegg V2 — installert Magentic-API (orchestrations 1.0.0, `_magentic.py`, 1802 linjer)
|
||||
|
||||
- **Eksport** (`agent_framework_orchestrations/__init__.py:43-62`): `MagenticBuilder`,
|
||||
`MagenticContext`, `MagenticManagerBase`, `MagenticOrchestrator`, `MagenticOrchestratorEvent`/
|
||||
`…EventType`, `MagenticPlanReviewRequest`/`…Response`, `MagenticProgressLedger`/`…Item`,
|
||||
`MagenticResetSignal`, `StandardMagenticManager`, `MagenticAgentExecutor`. **Ikke i koden:**
|
||||
`MagenticHumanIntervention*`, `with_human_input_on_stall` (kun docstrings).
|
||||
- **`MagenticBuilder(*, participants, manager=|manager_factory=|manager_agent=|manager_agent_factory=
|
||||
(nøyaktig én), task_ledger_*_prompt…, progress_ledger_prompt, final_answer_prompt,
|
||||
max_stall_count=UNSET→3, max_reset_count=None, max_round_count=None, enable_plan_review=False,
|
||||
checkpoint_storage=None, output_from, intermediate_output_from)`** (`:1396-1422`). Metoder:
|
||||
`with_plan_review(enable=True)` (`:1516`), `with_checkpointing(storage)` (`:1564`), `build()`
|
||||
(`:1772`; bidireksjonale kanter orkestrator↔deltaker `:1796-1797`). Grensene bor på MANAGEREN
|
||||
(`:1693-1695`, lest tilbake `:905/:1115/:1240`); `manager=` + prompt/limit-argumenter → warning og
|
||||
**stille ignorert** (`:1661-1677`). Ikke-agent-`Executor` som deltaker aksepteres (`:1498-1499`)
|
||||
men får placeholder-beskrivelse (`_base_group_chat_orchestrator.py:115`).
|
||||
- **`StandardMagenticManager(agent, …, max_stall_count=3, max_reset_count=None,
|
||||
max_round_count=None, progress_ledger_retry_count=None→3)`** (`:528-544`). Én `AgentSession` i
|
||||
`__init__` (`:572`), alle kall `agent.run(…, session=…)` (`:601`). Ledger-parsing: fri tekst +
|
||||
`_extract_json` (`:406-449`), dataklasser ikke pydantic (`:288-333`).
|
||||
- **Løkke:** runde-teller ×2 (`:1075-1076`), grense-sjekk FØR inkrement på `_round_index` (`:1238`);
|
||||
stall `:1109-1118`; reset `:1149-1185`; **grense → kanonisk streng + `_terminated`, ingen
|
||||
exception** (`:1222-1256`); sluttsvar `:1206-1220`; ukjent `next_speaker` → stille sluttsvar
|
||||
(`:1128-1131`); single-use-guard `:914-918`.
|
||||
- **Plan review:** request `:829-853` (`plan`, `current_progress`, `is_stalled`; `.approve()`,
|
||||
`.revise(feedback)`), response `:800-826` (`review: list[Message]`, tom = approve `:1010`),
|
||||
reises `:1041-1048`, svares `workflow.run(responses={id: …})` (`_workflow.py:899-908`; ukjent id
|
||||
→ `ValueError` `:946`), revise-håndtering `:1014-1031` (ingen cap, ingen runde-telling).
|
||||
Per-tur-HITL via `AgentApprovalExecutor` (`_orchestration_request_info.py:168`).
|
||||
- **Checkpoint:** `CheckpointStorage`-protokoll `_checkpoint.py:119`, `InMemoryCheckpointStorage`
|
||||
`:192`, `FileCheckpointStorage(storage_path, *, allowed_checkpoint_types)` `:239`;
|
||||
`WorkflowCheckpoint.pending_request_info_events` `:81`; `run(checkpoint_id=, checkpoint_storage=)`
|
||||
`_workflow.py:701-712`, `responses`+`checkpoint_id` tillatt `:877`; superstep-granularitet
|
||||
`_runner.py:143-144`; re-emittering `_runner_context.py:414-426`; Magentic-state
|
||||
`_magentic.py:1266-1284` + manager-sesjon `:746`; feil svelges `:1281-1282`/`:1320-1321`.
|
||||
- **Observability:** `_magentic.py` → 0 spans. Runtime-spans `workflow.build`/`workflow.run`/
|
||||
`executor.process`/`edge_group.process`/`message.send` (`observability.py:253-276`).
|
||||
Inngang `configure_otel_providers(*, enable_sensitive_data, enable_console_exporters, exporters,
|
||||
views, vs_code_extension_port, env_file_path, env_file_encoding)` (`:1168`), `enable_instrumentation`
|
||||
(`:1132`); env `ENABLE_INSTRUMENTATION` (default True, `:697`), `ENABLE_SENSITIVE_DATA`,
|
||||
`ENABLE_CONSOLE_EXPORTERS`, `OTEL_EXPORTER_OTLP_ENDPOINT`/`_PROTOCOL`/`_HEADERS`,
|
||||
`OTEL_SERVICE_NAME`. Ikke re-eksportert fra `agent_framework` (`hasattr` → False).
|
||||
`APPLICATIONINSIGHTS_CONNECTION_STRING` → ikke funnet. `opentelemetry-sdk`: **ikke installert**.
|
||||
- **Evals/Skills:** `evaluate_workflow(*, workflow, workflow_result|queries, evaluators, …)`
|
||||
(`_evaluation.py:1833`, `@experimental(EVALS)` `:1832`); `SkillsProvider(ContextProvider)`
|
||||
(`_skills.py:1719`, `@experimental(SKILLS)` `:1718`; `from_paths` `:1861`).
|
||||
|
||||
## Vedlegg V3 — kommandoer kjørt (utvalg; fullstendige lister i øktens agent-transkripter)
|
||||
|
||||
```
|
||||
find src -name '*.py' | sort | wc -l # 32
|
||||
uv pip list | grep -i -E "agent.framework|^mcp |opentelemetry|azure-ai|anyio"
|
||||
sed -n '61,75p' uv.lock ; grep -n "agent-framework" pyproject.toml
|
||||
curl -s https://pypi.org/pypi/agent-framework-orchestrations/json # + /1.0.0 … /1.1.1/json
|
||||
sed -n '1,80p' .venv/.../agent_framework/_feature_stage.py
|
||||
grep -rn "@experimental|@preview|feature_stage(" .venv/.../agent_framework/*.py .../_workflows/*.py
|
||||
grep -n -i "experimental|feature_stage" .venv/.../agent_framework_orchestrations/*.py # 0
|
||||
bash scratchpad/u_measure.sh # V1
|
||||
grep -rEl <9 null-regexer> .venv/.../agent_framework .venv/.../agent_framework_orchestrations | wc -l
|
||||
grep -rn -i "research.01" STATE.md STATE-arkiv-2026-08-21.local.md docs/ # 0
|
||||
grep -rn "enable_layer1_hitl|layer1" src tests ; grep -rEn 'request_info|RequestInfoEvent' tests # 0
|
||||
grep -rn "agent_framework" src/portfolio_optimiser/*.py # 14 importlinjer
|
||||
microsoft_docs_fetch learn.microsoft.com/en-us/agent-framework/workflows/orchestrations/magentic
|
||||
uv run python scratchpad/magentic_b7.py ; … magentic_b7b.py ; … magentic_reset.py # E1–E7
|
||||
.venv/bin/python -c "import opentelemetry.sdk" # ModuleNotFoundError
|
||||
```
|
||||
475
docs/presentasjon-bygge-kunnskapsbase.html
Normal file
475
docs/presentasjon-bygge-kunnskapsbase.html
Normal file
|
|
@ -0,0 +1,475 @@
|
|||
<!doctype html>
|
||||
<html lang="no">
|
||||
<meta charset="utf-8">
|
||||
<title>Slik bygger du kunnskapsbasen</title>
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<style>
|
||||
:root { color-scheme: light; }
|
||||
* { box-sizing: border-box; }
|
||||
body { margin:0; background:#fff; color:#000;
|
||||
font:17px/1.6 -apple-system, "Segoe UI", system-ui, sans-serif; }
|
||||
.slide { display:none; min-height:100vh; padding:6vh 6vw 14vh; max-width:52rem; margin:0 auto; }
|
||||
.slide.on { display:block; }
|
||||
h1 { font-size:2.1rem; margin:0 0 .5em; line-height:1.2; }
|
||||
h2 { font-size:1.5rem; margin:0 0 .9em; line-height:1.3; }
|
||||
.kicker { color:#666; text-transform:uppercase; letter-spacing:.1em;
|
||||
font-size:.72rem; margin:0 0 1.4em; }
|
||||
.step { display:inline-block; border:2px solid #000; border-radius:4px;
|
||||
padding:.05em .55em; font-weight:700; margin-right:.5em; }
|
||||
p { margin:0 0 1em; }
|
||||
ol, ul { padding-left:1.4em; margin:0 0 1em; }
|
||||
li { margin:.5em 0; }
|
||||
.lead { font-size:1.15rem; color:#444; }
|
||||
table { border-collapse:collapse; width:100%; margin:1.2em 0; font-size:.94rem; }
|
||||
th, td { border-bottom:1px solid #ddd; text-align:left; padding:.5em .6em; vertical-align:top; }
|
||||
th { color:#666; font-weight:600; font-size:.78rem; text-transform:uppercase; letter-spacing:.04em; }
|
||||
.done { border:2px solid #000; padding:.7em 1em; margin:1.4em 0; font-size:.95rem; }
|
||||
.done b { display:block; font-size:.72rem; text-transform:uppercase; letter-spacing:.08em;
|
||||
color:#666; margin-bottom:.25em; }
|
||||
.note { border-left:3px solid #ccc; padding:.3em 0 .3em 1.1em; color:#444;
|
||||
margin:1.4em 0; font-size:.95rem; }
|
||||
figure { margin:1.6em 0; }
|
||||
figure svg { width:100%; height:auto; display:block; }
|
||||
figcaption { font-size:.82rem; color:#666; margin-top:.6em; text-align:center; }
|
||||
nav { position:fixed; bottom:0; left:0; right:0; padding:.7em 6vw;
|
||||
background:#fff; border-top:1px solid #ddd;
|
||||
display:flex; gap:1em; align-items:center; font-size:.85rem; }
|
||||
button { font:inherit; padding:.3em .9em; cursor:pointer; border:1px solid #bbb;
|
||||
background:#fff; color:#000; border-radius:4px; }
|
||||
#pos { color:#666; margin-left:auto; }
|
||||
.d { fill:none; stroke:#000; stroke-width:2; }
|
||||
.dt { fill:#000; font:13px -apple-system,"Segoe UI",system-ui,sans-serif; }
|
||||
.dt-s { fill:#444; font:11px -apple-system,"Segoe UI",system-ui,sans-serif; }
|
||||
.dim { stroke:#bbb; }
|
||||
.dimt { fill:#999; font:12px -apple-system,"Segoe UI",system-ui,sans-serif; }
|
||||
.fill { fill:#000; }
|
||||
.hatch { fill:#eee; stroke:#000; stroke-width:2; }
|
||||
</style>
|
||||
|
||||
<section class="slide on">
|
||||
<p class="kicker">Optimaliseringsprosjekt</p>
|
||||
<h1>Slik bygger du kunnskapsbasen</h1>
|
||||
<p class="lead">Ni steg fra ingenting til en kunnskapsbase som kan kjøres. Hvert steg sier hvem
|
||||
som gjør det, hva som leveres, og når du er ferdig med det.</p>
|
||||
<p>Kunnskapsbasen er den lille, kuraterte samlingen systemet leser før det foreslår tiltak.
|
||||
Uten den vet systemet ingenting om anlegget ditt.</p>
|
||||
|
||||
<figure>
|
||||
<svg viewBox="0 0 720 118" role="img" aria-label="Ni steg fordelt på fire faser">
|
||||
<rect class="d" x="8" y="26" width="150" height="52" rx="4"/>
|
||||
<text class="dt" x="83" y="48" text-anchor="middle" font-weight="700">Ramme</text>
|
||||
<text class="dt-s" x="83" y="66" text-anchor="middle">steg 1–3</text>
|
||||
<path class="d" d="M158 52 h20"/><path class="fill" d="M186 52 l-10 -5 v10 z"/>
|
||||
|
||||
<rect class="d" x="186" y="26" width="150" height="52" rx="4"/>
|
||||
<text class="dt" x="261" y="48" text-anchor="middle" font-weight="700">Innhold</text>
|
||||
<text class="dt-s" x="261" y="66" text-anchor="middle">steg 4–6</text>
|
||||
<path class="d" d="M336 52 h20"/><path class="fill" d="M364 52 l-10 -5 v10 z"/>
|
||||
|
||||
<rect class="d" x="364" y="26" width="150" height="52" rx="4"/>
|
||||
<text class="dt" x="439" y="48" text-anchor="middle" font-weight="700">Bind sammen</text>
|
||||
<text class="dt-s" x="439" y="66" text-anchor="middle">steg 7–8</text>
|
||||
<path class="d" d="M514 52 h20"/><path class="fill" d="M542 52 l-10 -5 v10 z"/>
|
||||
|
||||
<rect class="d" x="542" y="26" width="170" height="52" rx="4"/>
|
||||
<text class="dt" x="627" y="48" text-anchor="middle" font-weight="700">Kjør og lær</text>
|
||||
<text class="dt-s" x="627" y="66" text-anchor="middle">steg 9</text>
|
||||
|
||||
<text class="dimt" x="360" y="108" text-anchor="middle">samlet: én til to uker, og det tunge er steg 2</text>
|
||||
</svg>
|
||||
</figure>
|
||||
</section>
|
||||
|
||||
<section class="slide">
|
||||
<p class="kicker">Før du starter</p>
|
||||
<h2>To roller, og de kan ikke bytte plass</h2>
|
||||
|
||||
<figure>
|
||||
<svg viewBox="0 0 720 165" role="img" aria-label="Fagpersonen eier innholdet, den tekniske personen eier formen">
|
||||
<rect class="d" x="8" y="14" width="340" height="130" rx="4"/>
|
||||
<text class="dt" x="178" y="42" text-anchor="middle" font-weight="700">FAGPERSONEN</text>
|
||||
<path class="d dim" d="M40 54 h276"/>
|
||||
<text class="dt-s" x="178" y="78" text-anchor="middle">eier innholdet og korrektheten</text>
|
||||
<text class="dt-s" x="178" y="98" text-anchor="middle">tallene, rammene, tiltakene, dommene</text>
|
||||
<text class="dt-s" x="178" y="126" text-anchor="middle">leverer i den formen de allerede jobber i</text>
|
||||
|
||||
<rect class="d" x="372" y="14" width="340" height="130" rx="4"/>
|
||||
<text class="dt" x="542" y="42" text-anchor="middle" font-weight="700">DEN TEKNISKE PERSONEN</text>
|
||||
<path class="d dim" d="M404 54 h276"/>
|
||||
<text class="dt-s" x="542" y="78" text-anchor="middle">eier formen og strukturen</text>
|
||||
<text class="dt-s" x="542" y="98" text-anchor="middle">oversetter til dokumenter og tall</text>
|
||||
<text class="dt-s" x="542" y="126" text-anchor="middle">finner aldri opp et tall selv</text>
|
||||
</svg>
|
||||
</figure>
|
||||
|
||||
<p>Fagpersonen leverer regneark, notater og henvisninger — ikke ferdige dokumenter. Den tekniske
|
||||
personen oversetter. <strong>Regelen som ikke kan brytes: den tekniske personen utleder aldri et
|
||||
tall fagpersonen ikke har oppgitt.</strong> Mangler et tall, står det som mangler.</p>
|
||||
<div class="note">Det finnes ingen automatikk som gjør denne jobben. Oversettelsen er håndarbeid,
|
||||
og det er derfor det tar én til to uker.</div>
|
||||
</section>
|
||||
|
||||
<section class="slide">
|
||||
<p class="kicker">Ramme</p>
|
||||
<h2><span class="step">1</span> Bestem hvilket anlegg det gjelder</h2>
|
||||
<p>Én kunnskapsbase dekker <strong>ett</strong> prosjekt eller anlegg. Har du flere, blir det
|
||||
flere baser — ikke én stor.</p>
|
||||
<p><strong>Gjør dette:</strong> velg et navn på anlegget og skriv det ned ett sted. Navnet skal
|
||||
brukes helt likt overalt siden, uten mellomrom og uten æ, ø og å.</p>
|
||||
<table>
|
||||
<tr><th>Hvem</th><th>Leverer</th></tr>
|
||||
<tr><td>Prosjekteier</td><td>hvilket anlegg, og hva det skal hete</td></tr>
|
||||
</table>
|
||||
<div class="done"><b>Ferdig når</b>Navnet er bestemt, og alle er enige om at basen dekker akkurat
|
||||
dette anlegget — ikke to naboanlegg «mens vi først er i gang».</div>
|
||||
</section>
|
||||
|
||||
<section class="slide">
|
||||
<p class="kicker">Ramme</p>
|
||||
<h2><span class="step">2</span> Skaff de faktiske kostnadstallene</h2>
|
||||
<p>Dette er det tunge steget, og det som avgjør om resten er verdt noe. Systemet skal senere
|
||||
kontrollere hvert forslag mot ekte kostnadslinjer — mengde ganger enhetspris.</p>
|
||||
|
||||
<figure>
|
||||
<svg viewBox="0 0 720 150" role="img" aria-label="Med kostnadstall avvises et oppdiktet forslag; uten dem passerer det">
|
||||
<text class="dt" x="8" y="22" font-weight="700">MED ekte tall</text>
|
||||
<rect class="d" x="8" y="32" width="120" height="40" rx="4"/>
|
||||
<text class="dt-s" x="68" y="56" text-anchor="middle">Oppdiktet forslag</text>
|
||||
<path class="d" d="M128 52 h30"/><path class="fill" d="M168 52 l-11 -5 v10 z"/>
|
||||
<rect class="d" x="168" y="32" width="150" height="40" rx="4" stroke-width="3"/>
|
||||
<text class="dt-s" x="243" y="56" text-anchor="middle">Avstemmes mot fasit</text>
|
||||
<path class="d" d="M318 52 h30"/><path class="fill" d="M358 52 l-11 -5 v10 z"/>
|
||||
<text class="dt" x="370" y="57" font-weight="700">AVVIST</text>
|
||||
|
||||
<text class="dt" x="8" y="112" font-weight="700">UTEN ekte tall</text>
|
||||
<rect class="d" x="8" y="122" width="120" height="0" rx="4"/>
|
||||
<rect class="d" x="8" y="102" width="120" height="40" rx="4"/>
|
||||
<text class="dt-s" x="68" y="126" text-anchor="middle">Oppdiktet forslag</text>
|
||||
<path class="d" d="M128 122 h30"/><path class="fill" d="M168 122 l-11 -5 v10 z"/>
|
||||
<rect class="d dim" x="168" y="102" width="150" height="40" rx="4" stroke-dasharray="6 5"/>
|
||||
<text class="dimt" x="243" y="126" text-anchor="middle">ingenting å avstemme mot</text>
|
||||
<path class="d" d="M318 122 h30"/><path class="fill" d="M358 122 l-11 -5 v10 z"/>
|
||||
<text class="dt" x="370" y="127" font-weight="700">GODKJENT</text>
|
||||
<text class="dimt" x="440" y="127">— og kjøringen ser helt normal ut</text>
|
||||
</svg>
|
||||
</figure>
|
||||
|
||||
<p><strong>Gjør dette:</strong> list opp hver kostnadslinje tiltakene kan komme til å treffe, med
|
||||
mengde og enhetspris. Ikke bare linjen til det tiltaket du tror på — alle linjene som er i spill.</p>
|
||||
<p><strong>Ta ikke med tall du ikke har.</strong> Har dere ingen kilde på hva noe koster per enhet,
|
||||
skal linjen utelates. En utledet verdi hører ikke hjemme her.</p>
|
||||
<div class="done"><b>Ferdig når</b>Hver linje har en mengde og en enhetspris, og du kan si hvor
|
||||
hvert tall kom fra. Får du ikke tak i tallene: skriv uttrykkelig at kjøringen blir uforankret, så
|
||||
ingen leser et godkjent-resultat som mer enn det er.</div>
|
||||
</section>
|
||||
|
||||
<section class="slide">
|
||||
<p class="kicker">Ramme</p>
|
||||
<h2><span class="step">3</span> Skriv ned hva som allerede er gjort</h2>
|
||||
<p>Systemet vet ingenting om anlegget utover det basen sier. Er noe allerede bygget om, og det
|
||||
ikke står noe sted, vil systemet foreslå det på nytt — og love en besparelse som allerede er tatt ut.</p>
|
||||
|
||||
<figure>
|
||||
<svg viewBox="0 0 720 130" role="img" aria-label="Gjennomførte tiltak hører i anleggsbeskrivelsen, tidligere vurderinger i erfaringslaget">
|
||||
<rect class="d" x="8" y="14" width="340" height="100" rx="4"/>
|
||||
<text class="dt" x="178" y="42" text-anchor="middle" font-weight="700">Gjennomførte tiltak</text>
|
||||
<text class="dt-s" x="178" y="66" text-anchor="middle">hva som faktisk er bygget om,</text>
|
||||
<text class="dt-s" x="178" y="84" text-anchor="middle">når, og på hvor mye av anlegget</text>
|
||||
<text class="dt-s" x="178" y="106" text-anchor="middle">→ inn i anleggsbeskrivelsen</text>
|
||||
|
||||
<rect class="d" x="372" y="14" width="340" height="100" rx="4"/>
|
||||
<text class="dt" x="542" y="42" text-anchor="middle" font-weight="700">Tidligere vurderinger</text>
|
||||
<text class="dt-s" x="542" y="66" text-anchor="middle">hva fagfolk mente om et forslag,</text>
|
||||
<text class="dt-s" x="542" y="84" text-anchor="middle">og hvor mye som faktisk ble realisert</text>
|
||||
<text class="dt-s" x="542" y="106" text-anchor="middle">→ inn i erfaringslaget (steg 7)</text>
|
||||
</svg>
|
||||
</figure>
|
||||
|
||||
<p><strong>Gjør dette:</strong> lag en kort liste over gjennomførte tiltak, og sørg for at
|
||||
kostnadstallene fra steg 2 viser <em>dagens</em> situasjon — ikke situasjonen før forrige tiltak.</p>
|
||||
<div class="done"><b>Ferdig når</b>Ingen kan peke på et tiltak som er gjennomført uten å stå i
|
||||
basen, og kostnadstallene stemmer med hva anlegget bruker i dag.</div>
|
||||
</section>
|
||||
|
||||
<section class="slide">
|
||||
<p class="kicker">Innhold</p>
|
||||
<h2><span class="step">4</span> Beskriv anlegget og rammene</h2>
|
||||
<p>Ett kort dokument som svarer: hva er dette anlegget, hva består det av, og hva kan
|
||||
<em>ikke</em> fravikes?</p>
|
||||
<p><strong>Rammene er den viktigste halvdelen.</strong> Uten dem foreslår systemet tiltak som
|
||||
bryter krav ingen har fortalt det om — og fagpersonen må bruke tid på å avvise det samme igjen
|
||||
og igjen.</p>
|
||||
<table>
|
||||
<tr><th>Ta med</th><th>Eksempeltype</th></tr>
|
||||
<tr><td>Hva anlegget består av</td><td>antall enheter, alder, tilstand, kapasitet</td></tr>
|
||||
<tr><td>Dagens forbruk eller kostnad</td><td>gulvet ingen besparelse kan gå under</td></tr>
|
||||
<tr><td>Krav som ikke kan fravikes</td><td>minstekrav, sikkerhetskrav, avtalefestede nivåer</td></tr>
|
||||
<tr><td>Antakelser som ikke holder</td><td>«vi kan ikke forutsette at X er mulig her»</td></tr>
|
||||
</table>
|
||||
<div class="done"><b>Ferdig når</b>En fagperson som ikke kjenner anlegget kan lese dokumentet og
|
||||
vite hva som er lov å foreslå.</div>
|
||||
</section>
|
||||
|
||||
<section class="slide">
|
||||
<p class="kicker">Innhold</p>
|
||||
<h2><span class="step">5</span> Skriv tiltaksnotatene</h2>
|
||||
<p>Ett kort notat per kandidat-tiltak. Systemet skal <em>forbedre</em> disse, ikke finne opp sine
|
||||
egne fra bunnen — så jo mer konkrete de er, jo bedre blir svaret.</p>
|
||||
<p><strong>Hvert notat skal ha:</strong> hva tiltaket går ut på, hvilke kostnadslinjer det virker
|
||||
på, hvor mye det er beregnet å spare, hva som er usikkert, og hva som er utledet framfor målt.</p>
|
||||
<p><strong>Trenger du hjelp til å finne kandidatene?</strong> For de fleste anleggstyper finnes
|
||||
det et ganske stabilt sett tiltakstyper — utskifting til nyere teknologi, behovsstyring framfor
|
||||
fast drift, tilstandsbasert vedlikehold framfor intervallbasert, og levetidsforlengelse framfor
|
||||
utskifting. Fagmiljøet har listen for sin type; jobben er å avgjøre hvilke som er aktuelle her.</p>
|
||||
<div class="note"><strong>Filteret som avgjør:</strong> et tiltak må kunne uttrykkes som en
|
||||
kostnadslinje — en mengde ganger en enhetspris. Kan det ikke det, kan systemet foreslå det, men
|
||||
ikke kontrollere det. Da er svaret verdt akkurat like mye som et vanlig godt råd.</div>
|
||||
<div class="done"><b>Ferdig når</b>Hvert notat peker på kostnadslinjer som finnes i steg 2, og
|
||||
det er tydelig merket hva som er målt og hva som er antatt.</div>
|
||||
</section>
|
||||
|
||||
<section class="slide">
|
||||
<p class="kicker">Innhold</p>
|
||||
<h2><span class="step">6</span> Velg det ene tiltaket som skal regnes på</h2>
|
||||
<p>Én kandidat får en egen, presis form som den deterministiske kontrollen kan konsumere:
|
||||
kostnadslinjene den virker på, den beregnede besparelsen, og et usikkerhetsbånd.</p>
|
||||
<p><strong>Denne kandidaten er også nøkkelen til tidligere erfaring</strong> — det er dens
|
||||
kostnadslinjer og størrelsesorden som avgjør hvilke tidligere dommer systemet henter fram.</p>
|
||||
|
||||
<figure>
|
||||
<svg viewBox="0 0 720 120" role="img" aria-label="Den regnede kandidaten og kostnadsgrunnlaget skrives fra samme regnestykke">
|
||||
<rect class="d" x="200" y="8" width="320" height="40" rx="4" stroke-width="3"/>
|
||||
<text class="dt" x="360" y="33" text-anchor="middle" font-weight="700">Ett regnestykke</text>
|
||||
<path class="d" d="M300 48 C 300 68, 180 68, 180 82"/><path class="fill" d="M180 92 l-5 -11 h10 z"/>
|
||||
<path class="d" d="M420 48 C 420 68, 540 68, 540 82"/><path class="fill" d="M540 92 l-5 -11 h10 z"/>
|
||||
<rect class="d" x="50" y="92" width="260" height="24" rx="4"/>
|
||||
<text class="dt-s" x="180" y="109" text-anchor="middle">Kostnadsgrunnlaget (steg 2)</text>
|
||||
<rect class="d" x="410" y="92" width="260" height="24" rx="4"/>
|
||||
<text class="dt-s" x="540" y="109" text-anchor="middle">Den regnede kandidaten</text>
|
||||
</svg>
|
||||
<figcaption>Samme kostnadslinje, samme mengde, samme enhetspris — skrevet ut fra ett regnestykke.</figcaption>
|
||||
</figure>
|
||||
|
||||
<div class="done"><b>Ferdig når</b>Kandidaten og kostnadsgrunnlaget bærer nøyaktig samme tall for
|
||||
de linjene de deler. Er de skrevet inn hver for seg, blir et lite avvik bare slurv — ikke et signal.</div>
|
||||
</section>
|
||||
|
||||
<section class="slide">
|
||||
<p class="kicker">Bind sammen</p>
|
||||
<h2><span class="step">7</span> Legg inn metode, kilder og første erfaring</h2>
|
||||
<p>Tre korte dokumenter som gjør forskjellen mellom et tall og et troverdig tall.</p>
|
||||
<table>
|
||||
<tr><th>Dokument</th><th>Hvem</th><th>Svarer på</th></tr>
|
||||
<tr><td><strong>Metode</strong></td><td>fagmiljøet</td>
|
||||
<td>hvordan skal effekten måles i etterkant — og hvilke metoder er stengt for dette anlegget?</td></tr>
|
||||
<tr><td><strong>Kilder</strong></td><td>fagmiljøet</td>
|
||||
<td>hva sier litteraturen om avviket mellom beregnet og faktisk effekt? Med årstall og opphav.</td></tr>
|
||||
<tr><td><strong>Første erfaring</strong></td><td>fageksperten</td>
|
||||
<td>hva pleier å skje? «Forvent rundt X prosent av beregnet, fordi …»</td></tr>
|
||||
</table>
|
||||
<p><strong>Skill eget materiale fra lånt.</strong> Har dere ingen egne målinger og må låne et tall
|
||||
fra litteraturen, skal det stå uttrykkelig at det er lånt. Det er ikke en svakhet — det er det som
|
||||
gjør at neste person kan vurdere om lånet holder.</p>
|
||||
<div class="done"><b>Ferdig når</b>Hvert tall som ikke er målt hos dere, har en navngitt kilde
|
||||
ved siden av seg.</div>
|
||||
</section>
|
||||
|
||||
<section class="slide">
|
||||
<p class="kicker">Bind sammen</p>
|
||||
<h2><span class="step">8</span> Skriv innholdsfortegnelsen som binder alt</h2>
|
||||
<p>Systemet <strong>navigerer</strong>, det søker ikke. Det starter i innholdsfortegnelsen, følger
|
||||
lenkene derfra, og leser hvert dokument det når — helt, eller ikke i det hele tatt.</p>
|
||||
|
||||
<figure>
|
||||
<svg viewBox="0 0 720 180" role="img" aria-label="Innholdsfortegnelsen lenker til dokumentene; et dokument uten lenke nås aldri">
|
||||
<rect class="d" x="30" y="60" width="130" height="56" rx="4"/>
|
||||
<text class="dt" x="95" y="83" text-anchor="middle">Innholds-</text>
|
||||
<text class="dt" x="95" y="101" text-anchor="middle">fortegnelse</text>
|
||||
|
||||
<rect class="d" x="290" y="14" width="180" height="34" rx="4"/>
|
||||
<text class="dt-s" x="380" y="35" text-anchor="middle">Anlegg og rammer</text>
|
||||
<rect class="d" x="290" y="58" width="180" height="34" rx="4"/>
|
||||
<text class="dt-s" x="380" y="79" text-anchor="middle">Tiltaksnotater</text>
|
||||
<rect class="d" x="290" y="102" width="180" height="34" rx="4"/>
|
||||
<text class="dt-s" x="380" y="123" text-anchor="middle">Metode og kilder</text>
|
||||
|
||||
<path class="d" d="M160 82 C 210 82, 230 31, 280 31"/><path class="fill" d="M290 31 l-11 -5 v10 z"/>
|
||||
<path class="d" d="M160 88 h120"/><path class="fill" d="M290 75 l-11 -5 v10 z"/>
|
||||
<path class="d" d="M160 94 C 210 94, 230 119, 280 119"/><path class="fill" d="M290 119 l-11 -5 v10 z"/>
|
||||
|
||||
<rect class="hatch" x="30" y="132" width="180" height="34" rx="4" stroke-dasharray="5 4"/>
|
||||
<text class="dimt" x="120" y="153" text-anchor="middle">dokument uten lenke — finnes ikke</text>
|
||||
|
||||
<rect class="d dim" x="530" y="58" width="160" height="34" rx="4" stroke-dasharray="5 4"/>
|
||||
<text class="dimt" x="610" y="79" text-anchor="middle">annen base</text>
|
||||
<path class="d dim" d="M470 75 h44" stroke-dasharray="6 5"/>
|
||||
<path class="d" d="M480 63 l22 24 M502 63 l-22 24" stroke-width="2.5"/>
|
||||
<text class="dimt" x="610" y="112" text-anchor="middle">lenker ut følges aldri</text>
|
||||
</svg>
|
||||
</figure>
|
||||
|
||||
<p><strong>Gjør dette:</strong> skriv et kort avsnitt om hva prosjektet er og hvorfor dette
|
||||
fagområdet er valgt — det er det aller første systemet leser — og lenk til hvert eneste dokument
|
||||
du vil at det skal se.</p>
|
||||
<div class="done"><b>Ferdig når</b>Hvert dokument du har laget er lenket. Et dokument uten lenke
|
||||
eksisterer ikke, uansett hvor det ligger. Og trenger du fagstoff fra en annen base: kopier det
|
||||
inn — lenker ut av basen følges aldri, men tørrkjøringen lister dem, så du ser at det skjedde.</div>
|
||||
</section>
|
||||
|
||||
<section class="slide">
|
||||
<p class="kicker">Kjør og lær</p>
|
||||
<h2><span class="step">9</span> Sjekk at basen åpner, så kjør</h2>
|
||||
<p>Kjør en tørrkjøring først. Den går hele veien fram til det første kallet mot språkmodellen og
|
||||
stopper der — den koster ingenting, og kan kjøres hver gang basen endres.</p>
|
||||
<table>
|
||||
<tr><th>Tørrkjøringen beviser</th><th>Den beviser ikke</th></tr>
|
||||
<tr><td>at basen åpner</td><td>at innholdet er godt</td></tr>
|
||||
<tr><td>at navnet stemmer overalt</td><td>at forslagene blir kloke</td></tr>
|
||||
<tr><td>at kostnadsgrunnlaget er lesbart</td><td>at alt du laget faktisk er lenket</td></tr>
|
||||
<tr><td>hvilke lenker den ikke klarte å følge</td><td>—</td></tr>
|
||||
<tr><td>at ingen ekstern tjeneste kontaktes</td><td>—</td></tr>
|
||||
</table>
|
||||
<p>Mangler kostnadsgrunnlaget, sier tørrkjøringen fra. Er en lenke brutt, stopper den fortsatt
|
||||
ikke kjøringen — men den skrives ut, med dokumentet den pekte på og hvorfor den ikke ble fulgt
|
||||
(<code>missing</code> = ingen fil der, <code>outside-bundle</code> = peker ut av basen). Er alle
|
||||
lenker fulgt, står det ingenting. Den feilen den fortsatt ikke kan se, er dokumentet
|
||||
<em>ingen</em> lenker til — da finnes det ingen lenke å rapportere.</p>
|
||||
<div class="done"><b>Ferdig når</b>Tørrkjøringen er grønn, den lister ingen lenker den ikke fulgte,
|
||||
og du har lest gjennom listen over dokumenter den faktisk fant.</div>
|
||||
</section>
|
||||
|
||||
<section class="slide">
|
||||
<p class="kicker">Kjør og lær</p>
|
||||
<h2>Etter kjøringen: dommen tilbake i basen</h2>
|
||||
|
||||
<figure>
|
||||
<svg viewBox="0 0 720 210" role="img" aria-label="Kjøring gir forslag, fagperson dømmer, dommen leses av neste kjøring">
|
||||
<rect class="d" x="30" y="24" width="150" height="46" rx="4"/>
|
||||
<text class="dt" x="105" y="52" text-anchor="middle">Kjøring</text>
|
||||
<path class="d" d="M180 47 h64"/><path class="fill" d="M254 47 l-11 -5 v10 z"/>
|
||||
|
||||
<rect class="d" x="254" y="24" width="180" height="46" rx="4"/>
|
||||
<text class="dt" x="344" y="44" text-anchor="middle">Forslag + kontroll</text>
|
||||
<text class="dt-s" x="344" y="62" text-anchor="middle">ett per vurdert tilnærming</text>
|
||||
<path class="d" d="M434 47 h64"/><path class="fill" d="M508 47 l-11 -5 v10 z"/>
|
||||
|
||||
<rect class="d" x="508" y="24" width="182" height="46" rx="4" stroke-width="3"/>
|
||||
<text class="dt" x="599" y="44" text-anchor="middle" font-weight="700">Fagperson dømmer</text>
|
||||
<text class="dt-s" x="599" y="62" text-anchor="middle">godkjent / avvist / korrigert</text>
|
||||
|
||||
<path class="d" d="M599 70 v42 H344"/><path class="fill" d="M344 112 l11 -5 v10 z"/>
|
||||
<rect class="d" x="254" y="112" width="180" height="46" rx="4"/>
|
||||
<text class="dt" x="344" y="140" text-anchor="middle">Innboks</text>
|
||||
|
||||
<path class="d" d="M254 135 H105 V76"/><path class="fill" d="M105 70 l-5 11 h10 z"/>
|
||||
<text class="dt-s" x="118" y="105">leses før neste forslag</text>
|
||||
|
||||
<text class="dimt" x="360" y="192" text-anchor="middle">systemet leser innboksen — mennesket skriver den</text>
|
||||
</svg>
|
||||
</figure>
|
||||
|
||||
<p>Neste kjøring leser dommene <em>før</em> den former sitt forslag. Dommen påvirker altså det
|
||||
neste forslaget, ikke bare arkivet. Godkjennes en dom for varig bruk, løftes den inn i basen.</p>
|
||||
<div class="note"><strong>Regelen som ikke kan brytes:</strong> systemet leser innboksen,
|
||||
mennesket skriver den. En kjøring lagrer aldri sin egen vurdering tilbake som kunnskap — ellers
|
||||
ville systemet lært av seg selv.</div>
|
||||
</section>
|
||||
|
||||
<section class="slide">
|
||||
<p class="kicker">Underveis</p>
|
||||
<h2>Må systemet spørre andre fagsystemer?</h2>
|
||||
<p>Utgangspunktet er <strong>ingen integrasjon</strong>: uten konfigurasjon gjøres null
|
||||
nettverkskall. Trenger du likevel data fra et annet system, er det to helt ulike veier — og valget
|
||||
avgjøres av <em>når</em> svaret trengs.</p>
|
||||
<table>
|
||||
<tr><th></th><th>Hent på forhånd</th><th>Slå opp underveis</th></tr>
|
||||
<tr><td>Når</td><td>før kjøringen</td><td>mens forslaget formes</td></tr>
|
||||
<tr><td>Blir</td><td>et dokument i basen</td><td>et verktøy systemet kan kalle</td></tr>
|
||||
<tr><td>Fordel</td><td>fagpersonen kan lese og korrigere det først</td><td>fanger opp noe som endrer seg</td></tr>
|
||||
<tr><td>Krever</td><td>opphav og dato på dokumentet</td><td>uttrykkelig liste over tillatte kall</td></tr>
|
||||
<tr><td>Nettverk under kjøring</td><td>null</td><td>ja, og alt navngis på forhånd</td></tr>
|
||||
</table>
|
||||
<p><strong>Velg «hent på forhånd» hvis du kan.</strong> Det er billigere, det kan kvalitetssikres
|
||||
av et menneske, og det gjør at du kan si nøyaktig hva en kjøring har rørt.</p>
|
||||
</section>
|
||||
|
||||
<section class="slide">
|
||||
<p class="kicker">Sjekkliste</p>
|
||||
<h2>Hvem leverer hva</h2>
|
||||
<table>
|
||||
<tr><th>Steg</th><th>Hvem</th><th>Leveranse</th></tr>
|
||||
<tr><td>1</td><td>Prosjekteier</td><td>anlegg og navn</td></tr>
|
||||
<tr><td><strong>2</strong></td><td><strong>Drift / økonomi</strong></td><td><strong>kostnadslinjer med mengde og enhetspris</strong></td></tr>
|
||||
<tr><td>3</td><td>Drift</td><td>liste over allerede gjennomførte tiltak</td></tr>
|
||||
<tr><td>4</td><td>Prosjekteier + fagmiljø</td><td>anleggsbeskrivelse og rammer</td></tr>
|
||||
<tr><td>5</td><td>Drift + fagmiljø</td><td>kandidat-tiltak med parametere</td></tr>
|
||||
<tr><td>6</td><td>Teknisk person</td><td>den ene kandidaten i presis form</td></tr>
|
||||
<tr><td>7</td><td>Fagmiljø + fagekspert</td><td>metode, kilder, første erfaring</td></tr>
|
||||
<tr><td>8</td><td>Teknisk person</td><td>innholdsfortegnelsen med alle lenker</td></tr>
|
||||
<tr><td>9</td><td>Teknisk person</td><td>tørrkjøring, så kjøring</td></tr>
|
||||
</table>
|
||||
<p>Steg 2 er markert fordi det er det som stopper prosjekter. Alt annet kan skrives på en dag
|
||||
hvis fagpersonene er tilgjengelige.</p>
|
||||
</section>
|
||||
|
||||
<section class="slide">
|
||||
<p class="kicker">Fallgruver</p>
|
||||
<h2>De fem feilene som koster mest</h2>
|
||||
<ol>
|
||||
<li><strong>Ingen kostnadstall.</strong> Kjøringen går, resultatet ser normalt ut, og kontrollen
|
||||
dømmer bare mot tall forslaget fant på selv. Dette er den dyreste feilen, fordi den er usynlig.</li>
|
||||
<li><strong>Gjennomførte tiltak står ikke i basen.</strong> Systemet foreslår dem på nytt, og
|
||||
besparelsen dobbelttelles.</li>
|
||||
<li><strong>Dokumenter uten lenke.</strong> De leses aldri, og ingen feilmelding sier fra —
|
||||
for det finnes ingen lenke å klage på. Er lenken der, men feil, sier tørrkjøringen nå fra;
|
||||
er den aldri skrevet, er dokumentet usynlig. Symptomet er at forslagene virker uinformerte
|
||||
uten at noen skjønner hvorfor.</li>
|
||||
<li><strong>For mye materiale.</strong> Alt som lenkes leses i sin helhet. Ti sider støy koster
|
||||
like mye oppmerksomhet som ti sider substans.</li>
|
||||
<li><strong>Utledede tall.</strong> Et tall ingen kan peke på en kilde for, forurenser alt som
|
||||
bygger på det. Mangler et tall, skal det stå som manglende.</li>
|
||||
</ol>
|
||||
</section>
|
||||
|
||||
<section class="slide">
|
||||
<p class="kicker">Til slutt</p>
|
||||
<h2>Hva dette lover — og ikke lover</h2>
|
||||
<ul>
|
||||
<li><strong>Én til to uker</strong> for en god base. Det tunge er ikke teknikken, men å få tak i
|
||||
kostnadstallene og å få en fagperson til å skrive ned rammene sine.</li>
|
||||
<li><strong>Det finnes ingen automatikk</strong> som lager basen av regnearkene deres. Det er
|
||||
håndarbeid, og det er derfor det tar tid.</li>
|
||||
<li><strong>Det finnes ingen «er basen klar»-knapp.</strong> Tørrkjøringen er det nærmeste: den
|
||||
sier fra om kostnadsgrunnlaget mangler og lister lenkene den ikke klarte å følge, men den sier
|
||||
ingenting om et dokument ingen har lenket til, og ingenting om innholdet er godt.</li>
|
||||
<li><strong>Systemet dømmer ikke.</strong> Det foreslår og kontrollerer regnestykket.
|
||||
Vurderingen av om tiltaket er klokt, gjør fagpersonen — og den vurderingen er det basen lærer av.</li>
|
||||
</ul>
|
||||
<div class="note">Basen blir bedre for hver kjøring, men bare hvis noen faktisk avgir dommen
|
||||
etterpå. En base ingen dømmer imot, står stille.</div>
|
||||
</section>
|
||||
|
||||
<nav>
|
||||
<button id="prev">← Forrige</button>
|
||||
<button id="next">Neste →</button>
|
||||
<span id="pos"></span>
|
||||
</nav>
|
||||
|
||||
<script>
|
||||
const slides = document.querySelectorAll('.slide');
|
||||
let i = 0;
|
||||
function show(n) {
|
||||
i = Math.max(0, Math.min(slides.length - 1, n));
|
||||
slides.forEach((s, k) => s.classList.toggle('on', k === i));
|
||||
document.getElementById('pos').textContent = (i + 1) + ' / ' + slides.length;
|
||||
window.scrollTo(0, 0);
|
||||
}
|
||||
document.getElementById('prev').onclick = () => show(i - 1);
|
||||
document.getElementById('next').onclick = () => show(i + 1);
|
||||
document.addEventListener('keydown', e => {
|
||||
if (e.key === 'ArrowRight' || e.key === 'PageDown' || e.key === ' ') show(i + 1);
|
||||
if (e.key === 'ArrowLeft' || e.key === 'PageUp') show(i - 1);
|
||||
});
|
||||
show(0);
|
||||
</script>
|
||||
</html>
|
||||
609
docs/presentasjon-fagpersonens-bidrag.html
Normal file
609
docs/presentasjon-fagpersonens-bidrag.html
Normal file
|
|
@ -0,0 +1,609 @@
|
|||
<!DOCTYPE html>
|
||||
<html lang="no">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>Hva løsningen trenger fra deg</title>
|
||||
<style>
|
||||
:root { color-scheme: light; }
|
||||
* { box-sizing: border-box; }
|
||||
body { margin:0; background:#fff; color:#000;
|
||||
font:17px/1.6 -apple-system, "Segoe UI", system-ui, sans-serif; }
|
||||
.slide { display:none; min-height:100vh; padding:6vh 6vw 14vh; max-width:52rem; margin:0 auto; }
|
||||
.slide.on { display:block; }
|
||||
h1 { font-size:2.1rem; margin:0 0 .5em; line-height:1.2; }
|
||||
h2 { font-size:1.5rem; margin:0 0 .9em; line-height:1.3; }
|
||||
h3 { font-size:1.02rem; margin:1.6em 0 .5em; }
|
||||
.kicker { color:#666; text-transform:uppercase; letter-spacing:.1em;
|
||||
font-size:.72rem; margin:0 0 1.4em; }
|
||||
.step { display:inline-block; border:2px solid #000; border-radius:4px;
|
||||
padding:.05em .55em; font-weight:700; margin-right:.5em; }
|
||||
p { margin:0 0 1em; }
|
||||
ol, ul { padding-left:1.4em; margin:0 0 1em; }
|
||||
li { margin:.5em 0; }
|
||||
.lead { font-size:1.15rem; color:#444; }
|
||||
table { border-collapse:collapse; width:100%; margin:1.2em 0; font-size:.94rem; }
|
||||
th, td { border-bottom:1px solid #ddd; text-align:left; padding:.5em .6em; vertical-align:top; }
|
||||
th { color:#666; font-weight:600; font-size:.78rem; text-transform:uppercase; letter-spacing:.04em; }
|
||||
.done { border:2px solid #000; padding:.7em 1em; margin:1.4em 0; font-size:.95rem; }
|
||||
.done b { display:block; font-size:.72rem; text-transform:uppercase; letter-spacing:.08em;
|
||||
color:#666; margin-bottom:.25em; }
|
||||
.note { border-left:3px solid #ccc; padding:.3em 0 .3em 1.1em; color:#444;
|
||||
margin:1.4em 0; font-size:.95rem; }
|
||||
.ask { border-left:5px solid #000; padding:.2em 0 .2em 1.1em; margin:1.3em 0;
|
||||
font-size:1.06rem; font-weight:600; }
|
||||
code { font:.92em ui-monospace, SFMono-Regular, Menlo, monospace; }
|
||||
figure { margin:1.6em 0; }
|
||||
figure svg { width:100%; height:auto; display:block; }
|
||||
figcaption { font-size:.82rem; color:#666; margin-top:.6em; text-align:center; }
|
||||
nav { position:fixed; bottom:0; left:0; right:0; padding:.7em 6vw;
|
||||
background:#fff; border-top:1px solid #ddd;
|
||||
display:flex; gap:1em; align-items:center; font-size:.85rem; }
|
||||
button { font:inherit; padding:.3em .9em; cursor:pointer; border:1px solid #bbb;
|
||||
background:#fff; color:#000; border-radius:4px; }
|
||||
#pos { color:#666; margin-left:auto; }
|
||||
.d { fill:none; stroke:#000; stroke-width:2; }
|
||||
.dt { fill:#000; font:13px -apple-system,"Segoe UI",system-ui,sans-serif; }
|
||||
.dt-s { fill:#444; font:11px -apple-system,"Segoe UI",system-ui,sans-serif; }
|
||||
.dim { stroke:#bbb; }
|
||||
.dimt { fill:#999; font:12px -apple-system,"Segoe UI",system-ui,sans-serif; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
|
||||
<!-- 1 -->
|
||||
<section class="slide on">
|
||||
<p class="kicker">Kostnadskutt i veg- og tunnelprosjekter</p>
|
||||
<h1>Hva løsningen trenger fra deg</h1>
|
||||
<p class="lead">Du er fagpersonen. Løsningen finner ingen besparelser uten det du vet —
|
||||
og den kan ikke gjette seg til det.</p>
|
||||
|
||||
<p>Denne presentasjonen er de <b>sju spørsmålene</b> du blir stilt før en kjøring bestilles,
|
||||
hva du skal svare, og hva du må skaffe på forhånd. Eksemplene er hentet fra
|
||||
<b>utbedringsprosjekter på veg</b> og <b>tunnelprosjekter</b>.</p>
|
||||
|
||||
<table>
|
||||
<tr><th>Spørsmål</th><th>Du leverer</th><th>Tid</th></tr>
|
||||
<tr><td>1. Hvilke hypoteser skal vurderes?</td><td>3–6 tiltak du tror på, med begrunnelse</td><td>1 møte</td></tr>
|
||||
<tr><td>2. Hva koster det i dag?</td><td>kostnadslinjer: mengde × enhetspris</td><td><b>dager til uker</b></td></tr>
|
||||
<tr><td>3. Hvilke interne data finnes?</td><td>uttrekk og registre dere allerede har</td><td>timer</td></tr>
|
||||
<tr><td>4. Hvilke eksterne data trengs?</td><td>normer, priser, erfaringstall — og hvilke <b>åpne API-er</b> som er relevante</td><td>timer</td></tr>
|
||||
<tr><td>5. Hva kan ikke fravikes?</td><td>krav, minstenivåer, avtaler</td><td>1 møte</td></tr>
|
||||
<tr><td>6. Hva er allerede gjort?</td><td>liste over gjennomførte tiltak</td><td>timer</td></tr>
|
||||
<tr><td>7. Hva pleier å skje i praksis?</td><td>din erfaring med kalkyle vs. virkelighet</td><td>1 møte</td></tr>
|
||||
</table>
|
||||
|
||||
<div class="note">Spørsmål 2 er det eneste som pleier å ta uker. Alt annet kan besvares
|
||||
på en dag hvis du er tilgjengelig.</div>
|
||||
</section>
|
||||
|
||||
<!-- 2 -->
|
||||
<section class="slide">
|
||||
<p class="kicker">Før spørsmålene</p>
|
||||
<h2>Hva løsningen gjør — og hvorfor du blir spurt</h2>
|
||||
|
||||
<p>Løsningen leser en liten, kuratert kunnskapsbase om <b>ett prosjekt</b>, foreslår
|
||||
kostnadsreduserende tiltak, og <b>kontrollerer regnestykket deterministisk</b> mot prosjektets
|
||||
faktiske kostnadslinjer før noe forslag slipper ut.</p>
|
||||
|
||||
<figure>
|
||||
<svg viewBox="0 0 720 150" role="img" aria-label="Fra din kunnskap til din dom">
|
||||
<rect class="d" x="4" y="30" width="160" height="70" rx="4"/>
|
||||
<text class="dt" x="84" y="58" text-anchor="middle">Din kunnskap</text>
|
||||
<text class="dt-s" x="84" y="78" text-anchor="middle">tall, rammer, hypoteser</text>
|
||||
|
||||
<path class="d" d="M168 65 h44"/><path class="d" d="M204 58 l8 7 l-8 7"/>
|
||||
|
||||
<rect class="d" x="216" y="30" width="160" height="70" rx="4"/>
|
||||
<text class="dt" x="296" y="58" text-anchor="middle">Forslag</text>
|
||||
<text class="dt-s" x="296" y="78" text-anchor="middle">flere agenter, flere runder</text>
|
||||
|
||||
<path class="d" d="M380 65 h44"/><path class="d" d="M416 58 l8 7 l-8 7"/>
|
||||
|
||||
<rect class="d" x="428" y="30" width="130" height="70" rx="4"/>
|
||||
<text class="dt" x="493" y="52" text-anchor="middle">Kontroll</text>
|
||||
<text class="dt-s" x="493" y="70" text-anchor="middle">avstemmer mot</text>
|
||||
<text class="dt-s" x="493" y="86" text-anchor="middle">ekte kostnadslinjer</text>
|
||||
|
||||
<path class="d" d="M562 65 h42"/><path class="d" d="M596 58 l8 7 l-8 7"/>
|
||||
|
||||
<rect class="d" x="608" y="30" width="108" height="70" rx="4"/>
|
||||
<text class="dt" x="662" y="58" text-anchor="middle">Din dom</text>
|
||||
<text class="dt-s" x="662" y="78" text-anchor="middle">ja / nei / justert</text>
|
||||
|
||||
<path class="dim" d="M662 104 v18 h-578 v-18" fill="none"/>
|
||||
<path class="dim" d="M84 111 l-6 -8 M84 111 l6 -8" fill="none"/>
|
||||
<text class="dimt" x="373" y="140" text-anchor="middle">dommen din går inn i neste kjøring</text>
|
||||
</svg>
|
||||
</figure>
|
||||
|
||||
<p><b>Kontrollen kan avgjøre om et tall er mulig. Den kan ikke avgjøre om tiltaket er klokt.</b>
|
||||
Det gjør du, etterpå — og det er den vurderingen løsningen lærer av.</p>
|
||||
|
||||
<div class="note">Derfor er spørsmålene under ikke en kartlegging. De er de fire tingene
|
||||
kontrollen ikke kan finne på egen hånd: <b>hva noe koster</b>, <b>hva som ikke er lov</b>,
|
||||
<b>hva som allerede er gjort</b>, og <b>hva som pleier å skje i drift</b>.</div>
|
||||
</section>
|
||||
|
||||
<!-- 3 -->
|
||||
<section class="slide">
|
||||
<p class="kicker">Før spørsmålene</p>
|
||||
<h2>Din rolle, og hva du <em>ikke</em> skal gjøre</h2>
|
||||
|
||||
<table>
|
||||
<tr><th></th><th>Du — fagpersonen</th><th>Den tekniske personen</th></tr>
|
||||
<tr><td><b>Eier</b></td><td>innholdet og korrektheten</td><td>formen og strukturen</td></tr>
|
||||
<tr><td><b>Leverer</b></td><td>tallene, rammene, tiltakene, dommene</td><td>oversettelsen til dokumenter og datafiler</td></tr>
|
||||
<tr><td><b>Format</b></td><td>det du allerede jobber i: regneark, notat, uttrekk, PDF</td><td>markdown og JSON</td></tr>
|
||||
<tr><td><b>Aldri</b></td><td>skriver systemfiler eller skjema</td><td>utleder et tall du ikke har oppgitt</td></tr>
|
||||
</table>
|
||||
|
||||
<p><b>Du skal aldri levere ferdige dokumenter.</b> Lever et regneark, et notat, et skjermbilde
|
||||
fra fagsystemet, en henvisning til en håndbok. Oversettelsen er ikke din jobb.</p>
|
||||
|
||||
<div class="done"><b>Regelen som ikke kan brytes</b>
|
||||
Mangler et tall, står det som <b>manglende</b>. Det utledes ikke, og det rundes ikke av til
|
||||
noe som «virker rimelig». Et tall ingen kan peke på en kilde for, forurenser alt som bygger på det.</div>
|
||||
|
||||
<div class="note">Det finnes ingen automatikk som lager kunnskapsbasen av regnearkene deres.
|
||||
Oversettelsen er håndarbeid, og det er derfor forberedelsen tar én til to uker.</div>
|
||||
</section>
|
||||
|
||||
<!-- 4 -->
|
||||
<section class="slide">
|
||||
<p class="kicker">Spørsmål 1 av 7</p>
|
||||
<h2><span class="step">1</span> Hvilke hypoteser vil du at løsningen skal vurdere?</h2>
|
||||
|
||||
<div class="ask">«Hvis du fikk én uke til å lete etter penger i dette prosjektet —
|
||||
hvor ville du sett først, og hvorfor?»</div>
|
||||
|
||||
<p>Løsningen <b>forbedrer</b> hypotesene dine framfor å finne opp sine egne fra bunnen. Jo mer
|
||||
konkrete de er, jo bedre blir svaret. Den foreslår også sitt eget i tillegg — men dine går først.</p>
|
||||
|
||||
<p><b>Fire tiltakstyper dekker det meste.</b> Bruk dem som huskeliste, ikke som fasit:</p>
|
||||
|
||||
<table>
|
||||
<tr><th>Tiltakstype</th><th>Utbedring på veg</th><th>Tunnel</th></tr>
|
||||
<tr><td><b>Ny teknologi erstatter gammel</b></td>
|
||||
<td>LED i veglys · nye rekkverkstyper med lengre levetid</td>
|
||||
<td>LED-armaturer · frekvensstyrte vifter</td></tr>
|
||||
<tr><td><b>Behovsstyring framfor fast drift</b></td>
|
||||
<td>vinterdrift utløst av målestasjon/prognose framfor fast rode-utkalling</td>
|
||||
<td>ventilasjon styrt på målt CO/NO₂ framfor fast drift · finere dimmetrinn på dagsonen</td></tr>
|
||||
<tr><td><b>Tilstandsbasert framfor intervallbasert</b></td>
|
||||
<td>dekkefornyelse etter målt spor og jevnhet framfor fast syklus · grøfterens etter tilstand</td>
|
||||
<td>vask og renhold etter målt tilsmussing framfor fast frekvens</td></tr>
|
||||
<tr><td><b>Levetidsforlengelse framfor utskifting</b></td>
|
||||
<td>forsegling eller tynndekke framfor full reasfaltering · reparasjon framfor bytte av rekkverk</td>
|
||||
<td>rehabilitering av eksisterende installasjon framfor full utskifting</td></tr>
|
||||
</table>
|
||||
|
||||
<p><b>En femte som ofte glemmes:</b> <i>redusert omfang</i> — å utbedre mindre der tilstanden
|
||||
ikke krever mer. Den er ofte den største, og den er alltid den vanskeligste å foreslå.</p>
|
||||
|
||||
<div class="done"><b>Ferdig når</b>
|
||||
Du har 3–6 hypoteser, hver med én setning om <b>hvorfor</b> du tror på den. Begrunnelsen mates
|
||||
ordrett inn til løsningen — det er der fagkunnskapen din faktisk gjør en forskjell.</div>
|
||||
</section>
|
||||
|
||||
<!-- 5 -->
|
||||
<section class="slide">
|
||||
<p class="kicker">Spørsmål 1, fortsatt</p>
|
||||
<h2>Filteret som avgjør om en hypotese er brukbar</h2>
|
||||
|
||||
<p>Et tiltak må kunne uttrykkes som en <b>kostnadslinje</b> — en mengde ganger en enhetspris.
|
||||
Kan det ikke det, kan løsningen foreslå det, men <b>ikke kontrollere det</b>. Da er svaret verdt
|
||||
akkurat like mye som et vanlig godt råd.</p>
|
||||
|
||||
<table>
|
||||
<tr><th></th><th>Hypotese</th><th>Blir til</th></tr>
|
||||
<tr><td>✔</td><td>Færre vinterutkallinger med prognosestyring</td><td>antall utkallinger × kr per utkalling</td></tr>
|
||||
<tr><td>✔</td><td>Lengre intervall mellom tunnelvask</td><td>antall vask per år × kr per vask</td></tr>
|
||||
<tr><td>✔</td><td>Finere dimming av tunnelbelysningen</td><td>kWh per år × kr per kWh</td></tr>
|
||||
<tr><td>✔</td><td>Tynndekke framfor full reasfaltering</td><td>m² × kr per m²</td></tr>
|
||||
<tr><td>✘</td><td>«Bedre samhandling med entreprenøren»</td><td>ingen mengde, ingen enhetspris</td></tr>
|
||||
<tr><td>✘</td><td>«Tidligere involvering av fagressurser»</td><td>ingen mengde, ingen enhetspris</td></tr>
|
||||
</table>
|
||||
|
||||
<p>De to nederste kan godt være riktige. De hører bare hjemme et annet sted enn her.</p>
|
||||
|
||||
<div class="done"><b>Gjør dette</b>
|
||||
Skriv om hver hypotese til formen «<b>noe</b> ganger <b>en pris</b>». Klarer du det ikke, spør
|
||||
deg selv hva som faktisk endrer seg i regnskapet — svaret er som regel mengden.</div>
|
||||
</section>
|
||||
|
||||
<!-- 6 -->
|
||||
<section class="slide">
|
||||
<p class="kicker">Spørsmål 2 av 7 — det tunge</p>
|
||||
<h2><span class="step">2</span> Hva koster dette i dag?</h2>
|
||||
|
||||
<div class="ask">«For hver hypotese: hvilken kostnadslinje treffer den, hva er mengden,
|
||||
og hva er enhetsprisen — og hvor kommer tallet fra?»</div>
|
||||
|
||||
<p>Dette er det steget som stopper prosjekter. Uten ekte kostnadslinjer har kontrollen
|
||||
<b>ingenting å avstemme mot</b>: et internt konsistent, oppdiktet forslag går rett gjennom, og
|
||||
kjøringen ser helt normal ut.</p>
|
||||
|
||||
<table>
|
||||
<tr><th>Prosjekttype</th><th>Typiske linjer</th><th>Formen</th></tr>
|
||||
<tr><td rowspan="4"><b>Utbedring veg</b></td>
|
||||
<td>dekkefornyelse</td><td>m² × kr/m²</td></tr>
|
||||
<tr><td>vinterdrift</td><td>utkallinger/år × kr per utkalling, eller km × kr/km</td></tr>
|
||||
<tr><td>veglys, energi</td><td>kWh/år × kr/kWh</td></tr>
|
||||
<tr><td>grøfterens, kantklipp, rekkverk</td><td>løpemeter × kr/lm</td></tr>
|
||||
<tr><td rowspan="4"><b>Tunnel</b></td>
|
||||
<td>belysning, energi</td><td>kWh/år × kr/kWh</td></tr>
|
||||
<tr><td>ventilasjon, energi</td><td>kWh/år × kr/kWh</td></tr>
|
||||
<tr><td>vask og renhold</td><td>vask/år × kr per vask</td></tr>
|
||||
<tr><td>utskifting av komponenter</td><td>antall × kr/stk</td></tr>
|
||||
</table>
|
||||
|
||||
<p><b>Ta med alle linjene som er i spill</b> — ikke bare linjen til det tiltaket du tror mest på.
|
||||
Et forslag som viser til en kostnadskode som ikke finnes i grunnlaget, blir avvist.</p>
|
||||
|
||||
<div class="done"><b>Ferdig når</b>
|
||||
Hver linje har en mengde og en enhetspris, og du kan si <b>hvor hvert tall kom fra</b>.
|
||||
Får du ikke tak i tallene: si det uttrykkelig, så ingen leser et godkjent-resultat som mer enn det er.</div>
|
||||
|
||||
<div class="note"><b>Vanligste fellene:</b> et tall som er fordelt fra en større post uten at det
|
||||
står · et tall fra før forrige ombygging · en enhetspris uten årstall, som ikke kan prisjusteres ·
|
||||
en investeringskostnad som dekker <em>hele</em> anlegget mens tiltaket bytter én del.</div>
|
||||
</section>
|
||||
|
||||
<!-- 7 -->
|
||||
<section class="slide">
|
||||
<p class="kicker">Spørsmål 3 av 7</p>
|
||||
<h2><span class="step">3</span> Hvilke interne data finnes — og hvem henter dem?</h2>
|
||||
|
||||
<div class="ask">«Hvilket system holder dette tallet i dag, og kan noen hente det ut for meg?»</div>
|
||||
|
||||
<p>Ikke spør om integrasjoner. Spør per tall. Fem kategorier dekker det meste:</p>
|
||||
|
||||
<table>
|
||||
<tr><th>Kategori</th><th>Utbedring veg</th><th>Tunnel</th><th>Uten den</th></tr>
|
||||
<tr><td><b>A. Kostnad og regnskap</b><br><span class="dimt">faktura, kalkyle, kontraktspriser</span></td>
|
||||
<td>enhetspriser fra driftskontrakt, sluttkostnad fra tilsvarende prosjekt</td>
|
||||
<td>energifaktura, priser fra siste elektroanbud</td>
|
||||
<td><b>kontrollen er uforankret</b></td></tr>
|
||||
<tr><td><b>B. Objekt og mengde</b><br><span class="dimt">hva anlegget består av</span></td>
|
||||
<td>km veg, m² dekke, antall stikkrenner, meter rekkverk, alder og tilstand</td>
|
||||
<td>antall armaturer og effekt, antall vifter og pumper, lengde, antall løp</td>
|
||||
<td>ingen mengde å gange med</td></tr>
|
||||
<tr><td><b>C. Bruk og driftsprofil</b><br><span class="dimt">hvor mye, hvor ofte, hvor lenge</span></td>
|
||||
<td>ÅDT, antall vinterutkallinger, saltmengde, klippefrekvens</td>
|
||||
<td>brenntimer, driftstimer vifter, vaskefrekvens, trafikkfordeling</td>
|
||||
<td>årsforbruket kan ikke regnes</td></tr>
|
||||
<tr><td><b>D. Historikk</b><br><span class="dimt">hva som er gjort</span></td>
|
||||
<td>utførte dekkefornyelser med årstall og strekning</td>
|
||||
<td>utskiftinger, oppgraderinger, rehabiliteringer med årstall</td>
|
||||
<td>tiltak foreslås på nytt, gevinst dobbelttelles</td></tr>
|
||||
<tr><td><b>E. Kontrakt og avtale</b><br><span class="dimt">hva som er bundet</span></td>
|
||||
<td>driftskontraktens omfang og løpetid, opsjoner</td>
|
||||
<td>serviceavtaler, garantiperioder</td>
|
||||
<td>tiltak foreslås som ikke kan bestilles</td></tr>
|
||||
</table>
|
||||
|
||||
<div class="done"><b>Gjør dette</b>
|
||||
For hver kategori: skriv ned <b>hvilket system eller regneark</b> tallet ligger i, og
|
||||
<b>hvem</b> som kan hente det ut. Et CSV-uttrekk eller et skjermbilde er nok — det trengs
|
||||
ingen integrasjon for å komme i gang.</div>
|
||||
</section>
|
||||
|
||||
<!-- 8 -->
|
||||
<section class="slide">
|
||||
<p class="kicker">Spørsmål 4 av 7</p>
|
||||
<h2><span class="step">4</span> Hvilke eksterne data er nyttige?</h2>
|
||||
|
||||
<p>Eksterne data brukes til to ting: å <b>begrense</b> hva som er lov, og å <b>kalibrere</b> hva
|
||||
som er realistisk. Fire kategorier:</p>
|
||||
|
||||
<p><b>Det aller meste av dette finnes allerede som åpne API-er.</b> Tilgang er sjelden problemet.
|
||||
Å vite <em>hvilke</em> kilder som er relevante for akkurat dine tiltak, er det — og det er en
|
||||
fagvurdering. Neste side er den jobben.</p>
|
||||
|
||||
<table>
|
||||
<tr><th>Kategori</th><th>Hva det er</th><th>Hva det gjør i kjøringen</th></tr>
|
||||
<tr><td><b>F. Normer og krav</b></td>
|
||||
<td>håndbøker og vegnormaler som gjelder tiltaket — for tunnelbelysning
|
||||
f.eks. Håndbok V124 og N500, med paragraf</td>
|
||||
<td>setter gulvet ingen besparelse kan gå under; hindrer forslag som bryter krav</td></tr>
|
||||
<tr><td><b>G. Priser og indekser</b></td>
|
||||
<td>kraftpris og nettleie, prisindekser, markedspriser fra sammenlignbare anbud</td>
|
||||
<td>gjør enhetsprisen etterprøvbar og prisjusterbar</td></tr>
|
||||
<tr><td><b>H. Erfaringstall for realisering</b></td>
|
||||
<td>litteratur og evalueringer om <b>gapet mellom beregnet og faktisk</b> effekt</td>
|
||||
<td>korrigerer den beregnede besparelsen ned til det som pleier å komme ut</td></tr>
|
||||
<tr><td><b>I. Produkt- og leverandørdata</b></td>
|
||||
<td>effekt, levetid, garanti, dokumenterte ytelser</td>
|
||||
<td>gir parametere til tiltaksnotatene i stedet for antakelser</td></tr>
|
||||
</table>
|
||||
|
||||
<p><b>Kategori H er den som er vanskeligst å skaffe og som betyr mest.</b> Finnes det ingen norsk
|
||||
måling på ditt område, er det helt i orden å låne et tall fra utenlandsk litteratur — men da skal
|
||||
det stå uttrykkelig <b>at det er lånt</b>, og fra hva.</p>
|
||||
|
||||
<div class="done"><b>Ferdig når</b>
|
||||
Hvert tall som ikke er målt hos dere, har en <b>navngitt kilde med årstall</b> ved siden av seg —
|
||||
og det er tydelig merket hva som er eget materiale og hva som er lånt.</div>
|
||||
|
||||
</section>
|
||||
|
||||
<!-- 8b -->
|
||||
<section class="slide">
|
||||
<p class="kicker">Spørsmål 4, fortsatt — nesten det viktigste</p>
|
||||
<h2>Åpne API-er og MCP-servere: jobben er å peke ut de riktige</h2>
|
||||
|
||||
<div class="ask">«Hvilke åpne datakilder bruker fagmiljøet ditt allerede —
|
||||
og hvilke skulle du ønske du hadde hatt?»</div>
|
||||
|
||||
<p>Det tekniske er sjelden flaskehalsen. Det finnes ferdige MCP-servere for en rekke offentlige
|
||||
norske datakilder, og et hvilket som helst åpent REST-API kan pakkes som én. <b>Det som mangler,
|
||||
er noen som kan si hvilke kilder som er verdt å koble til.</b> Det er deg.</p>
|
||||
|
||||
<h3>Fem spørsmål per kandidatkilde</h3>
|
||||
<table>
|
||||
<tr><th>Spør</th><th>Hvorfor det avgjør</th></tr>
|
||||
<tr><td><b>1.</b> Hvilket tall i regnestykket svarer den på?</td>
|
||||
<td>En kilde som ikke treffer en kostnadslinje eller en parameter, tilfører støy — ikke presisjon.</td></tr>
|
||||
<tr><td><b>2.</b> Er den autoritativ for nettopp det tallet?</td>
|
||||
<td>Ville du sitert den i en rapport? Hvis ikke, skal den ikke ligge til grunn her heller.</td></tr>
|
||||
<tr><td><b>3.</b> Hvor ofte endrer tallet seg?</td>
|
||||
<td>Sjelden ⇒ hent på forhånd. I løpet av dager ⇒ argument for oppslag underveis.</td></tr>
|
||||
<tr><td><b>4.</b> Hvem eier tilgangen, og koster den noe?</td>
|
||||
<td>Åpent uten nøkkel · åpent med registrering · lukket og krever avtale. Tre helt ulike tidslinjer.</td></tr>
|
||||
<tr><td><b>5.</b> Hvilken lisens har dataene?</td>
|
||||
<td>Avgjør om resultatet kan deles videre, og med hvem.</td></tr>
|
||||
</table>
|
||||
|
||||
<h3>To måter en kilde kommer inn — og de er ikke likeverdige</h3>
|
||||
<table>
|
||||
<tr><th></th><th>Hent på forhånd</th><th>Slå opp underveis (MCP)</th></tr>
|
||||
<tr><td><b>Når</b></td><td>før kjøringen</td><td>mens forslaget formes</td></tr>
|
||||
<tr><td><b>Blir</b></td><td>et dokument i kunnskapsbasen, med opphav og dato</td><td>et verktøy løsningen kan kalle selv</td></tr>
|
||||
<tr><td><b>Fordel</b></td><td>du kan lese og korrigere dataene <em>først</em></td><td>fanger opp noe som endrer seg</td></tr>
|
||||
<tr><td><b>Krever</b></td><td>at noen henter uttrekket</td><td>uttrykkelig liste over tillatte oppslag</td></tr>
|
||||
<tr><td><b>Nettverk under kjøring</b></td><td>null</td><td>ja — og alt navngis på forhånd</td></tr>
|
||||
</table>
|
||||
|
||||
<p><b>Velg «hent på forhånd» når du kan.</b> Det er billigere, det kan kvalitetssikres av et
|
||||
menneske før det brukes, og det gjør at du etterpå kan si nøyaktig hva en kjøring har rørt.
|
||||
Uten eksplisitt oppsett gjør en kjøring <b>null</b> nettverkskall.</p>
|
||||
|
||||
<div class="done"><b>Ferdig når</b>
|
||||
Du har en navngitt liste: <b>kilde · hvilket tall den svarer på · hvor ofte det endrer seg ·
|
||||
hvem som eier tilgangen</b>. Fem treffsikre kilder slår femti mulige.</div>
|
||||
|
||||
<div class="note"><b>Advarsel:</b> flere kilder gjør ikke svaret bedre av seg selv. Alt som kobles
|
||||
til, blir lest. Ti kilder som ikke treffer et tall i regnestykket, koster like mye oppmerksomhet
|
||||
som ti som gjør det.</div>
|
||||
</section>
|
||||
|
||||
<!-- 9 -->
|
||||
<section class="slide">
|
||||
<p class="kicker">Spørsmål 5 av 7</p>
|
||||
<h2><span class="step">5</span> Hva kan ikke fravikes?</h2>
|
||||
|
||||
<div class="ask">«Hvilke krav, nivåer og avtaler er det ingen besparelse som kan gå under —
|
||||
og avviker noe av det hos dere?»</div>
|
||||
|
||||
<p>Rammene er den viktigste halvdelen av anleggsbeskrivelsen. Uten dem foreslår løsningen tiltak
|
||||
som bryter krav ingen har fortalt den om, og du bruker tid på å avvise det samme igjen og igjen.</p>
|
||||
|
||||
<table>
|
||||
<tr><th>Type ramme</th><th>Utbedring veg</th><th>Tunnel</th></tr>
|
||||
<tr><td><b>Fagkrav med minstenivå</b></td>
|
||||
<td>krav til friksjon, jevnhet, sporddybde, siktforhold</td>
|
||||
<td>lystekniske minstekrav i sonene, luftkvalitetskrav, hysteresetid ved nivåendring</td></tr>
|
||||
<tr><td><b>Sikkerhetskrav</b></td>
|
||||
<td>rekkverksklasser, arbeidsvarsling</td>
|
||||
<td>krav til nødbelysning, ventilasjon ved brann, redundans</td></tr>
|
||||
<tr><td><b>Antakelser som ikke holder</b></td>
|
||||
<td>«vi kan ikke forutsette at strekningen kan stenges»</td>
|
||||
<td>«vi kan ikke forutsette nattstenging for arbeid»</td></tr>
|
||||
<tr><td><b>Kontraktsbundet</b></td>
|
||||
<td>driftskontraktens omfang ut avtaleperioden</td>
|
||||
<td>serviceavtaler, garantibetingelser</td></tr>
|
||||
<tr><td><b>Budsjett og anskaffelse</b></td>
|
||||
<td colspan="2">hva som kan bestilles i hvilket år, terskelverdier</td></tr>
|
||||
</table>
|
||||
|
||||
<div class="done"><b>Ferdig når</b>
|
||||
En fagperson som <em>ikke</em> kjenner anlegget kan lese listen og vite hva som er lov å foreslå.
|
||||
Er et krav strengere hos dere enn i normen — si det. Det er nøyaktig det du vet og normen ikke sier.</div>
|
||||
</section>
|
||||
|
||||
<!-- 10 -->
|
||||
<section class="slide">
|
||||
<p class="kicker">Spørsmål 6 av 7</p>
|
||||
<h2><span class="step">6</span> Hva er allerede gjort — og hva er allerede vurdert?</h2>
|
||||
|
||||
<div class="ask">«Hva er bygget om de siste årene, når, og på hvor stor del av anlegget?
|
||||
Og har noen vurdert et av disse tiltakene før?»</div>
|
||||
|
||||
<p>Løsningen vet ingenting om anlegget utover det kunnskapsbasen sier. Står et gjennomført tiltak
|
||||
ingen steder, blir det <b>foreslått på nytt</b> — med en besparelse som allerede er tatt ut.</p>
|
||||
|
||||
<table>
|
||||
<tr><th>Du leverer</th><th>Hva det hindrer</th></tr>
|
||||
<tr><td><b>Gjennomførte tiltak</b> — hva, når, på hvor mye av anlegget</td>
|
||||
<td>dobbelttelling av en gevinst som allerede er hentet</td></tr>
|
||||
<tr><td><b>Kostnadstall som viser dagens situasjon</b>, ikke situasjonen før forrige tiltak</td>
|
||||
<td>at kontrollen avstemmer mot et grunnlag som ikke finnes lenger</td></tr>
|
||||
<tr><td><b>Tidligere vurderinger</b> — hva fagfolk mente om et forslag, og hvorfor</td>
|
||||
<td>at samme diskusjon tas om igjen fra null</td></tr>
|
||||
<tr><td><b>Tiltak som ble forsøkt og ikke virket</b>, med begrunnelse</td>
|
||||
<td>den dyreste gjentakelsen av alle</td></tr>
|
||||
</table>
|
||||
|
||||
<div class="done"><b>Ferdig når</b>
|
||||
Ingen i rommet kan peke på et gjennomført tiltak som ikke står i basen, og kostnadstallene
|
||||
stemmer med det anlegget faktisk bruker i dag.</div>
|
||||
</section>
|
||||
|
||||
<!-- 11 -->
|
||||
<section class="slide">
|
||||
<p class="kicker">Spørsmål 7 av 7 — det mest verdifulle</p>
|
||||
<h2><span class="step">7</span> Hva pleier å skje mellom kalkyle og virkelighet?</h2>
|
||||
|
||||
<div class="ask">«Når dere har gjort noe slikt før — hvor mye av den beregnede besparelsen
|
||||
kom faktisk ut? Og hva var det som spiste resten?»</div>
|
||||
|
||||
<p>Kontrollen kan avgjøre om et tall er <b>mulig</b>. Den kan ikke avgjøre om anlegget faktisk
|
||||
leverer det. Der stopper regnestykket og din erfaring begynner — og det er den <b>eneste</b>
|
||||
kunnskapen i hele prosessen som ikke kan hentes fra et system.</p>
|
||||
|
||||
<p><b>Et godt svar navngir mekanismen, ikke bare tallet.</b> Eksempel fra tunnelbelysning, der
|
||||
tre kjente mekanismer trekker gevinsten ned:</p>
|
||||
|
||||
<table>
|
||||
<tr><th>Mekanisme</th><th>Hvorfor den spiser gevinst</th></tr>
|
||||
<tr><td>Påkrevd forsinkelse ved nivåendring</td>
|
||||
<td>holder anlegget på det <em>høyere</em> nivået gjennom svingninger — asymmetrisk i energi</td></tr>
|
||||
<tr><td>Den delen av tiltaket som ikke blir implementert</td>
|
||||
<td>halve gevinsten kan ligge i en del som rutinemessig faller ut av leveransen</td></tr>
|
||||
<tr><td>Kalibrering med sikkerhetsmargin</td>
|
||||
<td>systematisk og ensrettet: ingen driftsorganisasjon justerer seg til for lite lys</td></tr>
|
||||
</table>
|
||||
|
||||
<p>De samme spørsmålene på vegsiden: <i>Ble den nye driftsrutinen faktisk fulgt hele vinteren?
|
||||
Ble tilstandsmålingene brukt til å styre, eller bare rapportert? Hvor mye av tynndekket måtte
|
||||
gjøres om igjen innen tre år?</i></p>
|
||||
|
||||
<div class="done"><b>Ferdig når</b>
|
||||
Du har sagt, med egne ord: «forvent rundt <b>X</b> prosent av det som er beregnet, fordi <b>Y</b>.»
|
||||
Har du et tall fra et eget prosjekt — det er den enkeltleveransen som forbedrer basen mest.
|
||||
Har du det ikke, si det: et navngitt kunnskapshull er innhold, et oppdiktet tall er forurensning.</div>
|
||||
</section>
|
||||
|
||||
<!-- 12 -->
|
||||
<section class="slide">
|
||||
<p class="kicker">Kvalitet</p>
|
||||
<h2>Fire krav til hvert tall du leverer</h2>
|
||||
|
||||
<table>
|
||||
<tr><th>Krav</th><th>Hvorfor</th></tr>
|
||||
<tr><td><b>1. Kilde.</b> Hvor kom tallet fra — system, faktura, håndbok, notat?</td>
|
||||
<td>Et tall uten kilde kan ikke etterprøves, og da kan heller ikke resultatet det.</td></tr>
|
||||
<tr><td><b>2. Årstall.</b> Hvilket år gjelder det for?</td>
|
||||
<td>Et beløp uten årstall kan ikke prisjusteres. Da er det ubrukelig, uansett hvor riktig det var.</td></tr>
|
||||
<tr><td><b>3. Målt eller antatt.</b> Er dette avlest, eller er det anslått?</td>
|
||||
<td>Begge deler er brukbart. Å forveksle dem er ikke.</td></tr>
|
||||
<tr><td><b>4. Omfang.</b> Hva dekker tallet — hele anlegget, eller den delen tiltaket treffer?</td>
|
||||
<td>Feil omfang er den vanligste grunnen til at et riktig tiltak blir avvist.</td></tr>
|
||||
</table>
|
||||
|
||||
<div class="done"><b>Den enkleste formen</b>
|
||||
Én linje per tall: <code>hva · verdi · enhet · kilde · år · målt/antatt</code>.
|
||||
Et regneark med de seks kolonnene er en fullgod leveranse.</div>
|
||||
</section>
|
||||
|
||||
<!-- 13 -->
|
||||
<section class="slide">
|
||||
<p class="kicker">Fallgruver</p>
|
||||
<h2>De fem feilene som koster mest</h2>
|
||||
|
||||
<ol>
|
||||
<li><b>Ingen ekte kostnadstall.</b> Kjøringen går, resultatet ser normalt ut, og kontrollen
|
||||
dømmer bare mot tall forslaget fant på selv. Den dyreste feilen, fordi den er usynlig.</li>
|
||||
|
||||
<li><b>Gjennomførte tiltak står ikke i basen.</b> De foreslås på nytt, og besparelsen
|
||||
dobbelttelles.</li>
|
||||
|
||||
<li><b>Utledede tall.</b> Et tall ingen kan peke på en kilde for, forurenser alt som bygger
|
||||
på det. Mangler et tall, skal det stå som manglende.</li>
|
||||
|
||||
<li><b>For mye materiale.</b> Alt som legges inn, leses i sin helhet. Ti sider støy koster like
|
||||
mye oppmerksomhet som ti sider substans. Lever det som er relevant, ikke alt som finnes.</li>
|
||||
|
||||
<li><b>Ingen som dømmer etterpå.</b> Kjøringen produserer et forslag ingen svarer på, og
|
||||
løsningen lærer ingenting. En base ingen dømmer imot, står stille.</li>
|
||||
</ol>
|
||||
</section>
|
||||
|
||||
<!-- 14 -->
|
||||
<section class="slide">
|
||||
<p class="kicker">Etterpå</p>
|
||||
<h2>Din dom er produktet — ikke forslaget</h2>
|
||||
|
||||
<p>Etter kjøringen får du hvert vurderte tiltak tilbake, ett for ett, med kontrollens begrunnelse.
|
||||
Du svarer én av tre ting:</p>
|
||||
|
||||
<table>
|
||||
<tr><th>Svar</th><th>Når</th><th>Hva du skriver</th></tr>
|
||||
<tr><td><b>Godkjent</b></td><td>tallet står seg som det er</td>
|
||||
<td>kort. En lang begrunnelse for et enkelt ja gir bare støy.</td></tr>
|
||||
<tr><td><b>Godkjent med korreksjon</b></td><td>regnestykket stemmer, men drift leverer mindre</td>
|
||||
<td><b>det vanligste ekte svaret</b> — og det som bærer mest læring: hvor mye, og hvorfor.</td></tr>
|
||||
<tr><td><b>Avvist</b></td><td>virkeligheten rundt tallet holder ikke</td>
|
||||
<td>hvorfor. «Ikke gjennomførbart» lærer ingenting; «forutsetningen om X holder ikke her, fordi Y» gjør det.</td></tr>
|
||||
</table>
|
||||
|
||||
<p><b>Skriv hvorfor, ikke hva.</b> Begrunnelsen er det eneste som bærer fagkunnskap videre til
|
||||
neste kjøring — på dette prosjektet og på liknende prosjekter senere.</p>
|
||||
|
||||
<div class="note">Et avvist forslag er ikke en feilet kjøring. En avvisning med en god begrunnelse
|
||||
er ofte mer verdt enn en godkjenning, fordi den lukker en retning for godt.</div>
|
||||
</section>
|
||||
|
||||
<!-- 15 -->
|
||||
<section class="slide">
|
||||
<p class="kicker">Ta med denne</p>
|
||||
<h2>Sjekkliste: dette skaffer du før kjøringen</h2>
|
||||
|
||||
<h3>Blokkerende — uten disse kjøres det ikke</h3>
|
||||
<ul>
|
||||
<li>Hvilket prosjekt eller anlegg det gjelder, med ett entydig navn</li>
|
||||
<li>Kostnadslinjene tiltakene kan treffe: <b>mengde × enhetspris</b>, med kilde og årstall</li>
|
||||
<li>3–6 hypoteser, hver med én setning om hvorfor</li>
|
||||
<li>Liste over hva som allerede er gjennomført, med årstall</li>
|
||||
<li>Kravene som ikke kan fravikes</li>
|
||||
<li>Navnet på den som skal avgi dommen etterpå</li>
|
||||
</ul>
|
||||
|
||||
<h3>Gjør svaret vesentlig bedre</h3>
|
||||
<ul>
|
||||
<li>Objekt- og mengdedata: antall, effekt, alder, tilstand</li>
|
||||
<li>Driftsprofil: timer, frekvenser, ÅDT, utkallinger</li>
|
||||
<li>Erfaringstall for realiseringsgrad — eget eller lånt, merket hvilket</li>
|
||||
<li><b>Navngitte åpne datakilder</b>: hva hver av dem svarer på, og hvem som eier tilgangen</li>
|
||||
<li>Produkt- og leverandørdata for de aktuelle tiltakene</li>
|
||||
<li>Tidligere vurderinger, inkludert de som endte i nei</li>
|
||||
</ul>
|
||||
|
||||
<h3>Avklares med den tekniske personen</h3>
|
||||
<ul>
|
||||
<li>Hvilke systemer tallene hentes fra, og hvem som henter dem</li>
|
||||
<li>Om en kilde skal hentes på forhånd eller slås opp underveis</li>
|
||||
<li>Om noe skal kontaktes under kjøring — og hvem som godkjenner det</li>
|
||||
</ul>
|
||||
|
||||
<div class="done"><b>Realistisk tidsbruk</b>
|
||||
Én til to uker. Det tunge er ikke teknikken — det er å få tak i kostnadstallene og å få skrevet
|
||||
ned rammene.</div>
|
||||
</section>
|
||||
|
||||
<nav>
|
||||
<button id="prev">← Forrige</button>
|
||||
<button id="next">Neste →</button>
|
||||
<span id="pos"></span>
|
||||
</nav>
|
||||
|
||||
<script>
|
||||
const slides = document.querySelectorAll('.slide');
|
||||
let i = 0;
|
||||
function show(n) {
|
||||
i = Math.max(0, Math.min(slides.length - 1, n));
|
||||
slides.forEach((s, k) => s.classList.toggle('on', k === i));
|
||||
document.getElementById('pos').textContent = (i + 1) + ' / ' + slides.length;
|
||||
window.scrollTo(0, 0);
|
||||
}
|
||||
document.getElementById('prev').onclick = () => show(i - 1);
|
||||
document.getElementById('next').onclick = () => show(i + 1);
|
||||
document.addEventListener('keydown', e => {
|
||||
if (e.key === 'ArrowRight' || e.key === 'PageDown' || e.key === ' ') show(i + 1);
|
||||
if (e.key === 'ArrowLeft' || e.key === 'PageUp') show(i - 1);
|
||||
});
|
||||
show(0);
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
|
|
@ -148,7 +148,7 @@ derfor re-kjøringen som et erklært protokollavvik — det absorberes ikke stil
|
|||
kjøring 1 førsteklasses, jf. tabellen over.
|
||||
|
||||
**Konfig-diff mellom feilet og vellykket kjøring (publisert):**
|
||||
`git -C /Users/ktg/repos/portfolio-optimiser-claude diff 0238507..7637c6f --stat` viser 7 filer
|
||||
`git -C ../portfolio-optimiser-claude diff 0238507..7637c6f --stat` viser 7 filer
|
||||
endret, 253 innsettinger, 25 slettinger (`artifacts.py`, `loop.py`, `run_s10.py`, `sdk_client.py`
|
||||
+ tre testfiler), samlet i fiks-commiten `7637c6f` «fix(run): S10 del 2 — post-mortem:
|
||||
stopp-artefakt, SDK-isolasjon, raw-JSON-direktiv». Endringene er harness-fikser (SDK-isolasjon
|
||||
|
|
@ -248,8 +248,8 @@ Klasser: **reproduce** = kommandoen kan kjøres på nytt og skal gi samme svar;
|
|||
**recompute-from-artifact** = tallet avledes av committede artefakter (live-tall re-kjøres aldri,
|
||||
jf. §4.2/§4.5); **disclosed-non-reproducible** = kilden er erklært og ikke-reproduserbar.
|
||||
|
||||
Kommandoer kjøres fra MAF-repoets rot (`/Users/ktg/repos/portfolio-optimiser`); søskenet er
|
||||
`/Users/ktg/repos/portfolio-optimiser-claude`.
|
||||
Kommandoer kjøres fra MAF-repoets rot (dette repoet); søskenet er sjekket ut som nabokatalog
|
||||
(`../portfolio-optimiser-claude`).
|
||||
|
||||
| # | Påstand | Kommando / kilde | Klasse |
|
||||
|---|---|---|---|
|
||||
|
|
@ -278,7 +278,7 @@ Kommandoer kjøres fra MAF-repoets rot (`/Users/ktg/repos/portfolio-optimiser`);
|
|||
python3 - <<'EOF'
|
||||
import json, pathlib
|
||||
g = json.loads(pathlib.Path("shared/examples/bygg-energi-mikro/golden.json").read_text())["validator"]
|
||||
r = json.loads(pathlib.Path("/Users/ktg/repos/portfolio-optimiser-claude/runs/s10/run_result.json").read_text())["outcome"]
|
||||
r = json.loads(pathlib.Path("../portfolio-optimiser-claude/runs/s10/run_result.json").read_text())["outcome"]
|
||||
for k in ("p10", "p50", "p90"):
|
||||
assert g[k] == r[k], (k, g[k], r[k])
|
||||
print("OK: golden == run_result (bit-for-bit p10/p50/p90)")
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@
|
|||
**Dato:** 2026-07-04
|
||||
**Program:** ingest-konnektorer for portfolio-optimiser, avsluttende steg (I7).
|
||||
**Omfang:** begge referanseimplementasjonene — MAF (dette repoet) og Claude Agent
|
||||
SDK-søsknet (D7, [`portfolio-optimiser-claude`](https://git.fromaitochitta.com/ktg/portfolio-optimiser-claude),
|
||||
SDK-søsknet (D7, [`portfolio-optimiser-claude`](https://git.fromaitochitta.com/open/portfolio-optimiser-claude),
|
||||
sitert *read-only*).
|
||||
**Bindende føring:** ærlighets-regelen — [målbilde §0 + §11 «Stack-mapping (ærlighets-avgrenset)»](../plan/2026-07-03-maalbilde-ingest-lag.md).
|
||||
Rapporten skiller **kapabilitet-implementert** fra **kjørt-mot-live**, og holder to «live»-akser
|
||||
|
|
|
|||
13
env.template
13
env.template
|
|
@ -28,3 +28,16 @@ PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT= # tenant-specific Foundry project endpoint
|
|||
# Hosted only: Foundry injects FOUNDRY_PROJECT_ENDPOINT itself, and the framework reads it as a
|
||||
# fallback. The name above wins whenever it holds a non-empty value, so an explicit export is never
|
||||
# shadowed by a platform default — and an exported-but-empty name falls through rather than masking.
|
||||
|
||||
# --- Tracing (U14) — OFF unless this variable holds a value ---
|
||||
# Unset or empty: OpenTelemetry providers are NOT configured at all. Spans are still made (MAF's
|
||||
# ENABLE_INSTRUMENTATION defaults to true) and simply discarded, so nothing can leave the process.
|
||||
# console -> spans are written to STDERR. stdout is byte-identical to an untraced run, which is
|
||||
# what keeps the pinned demo transcript usable. Refused if any OTEL_EXPORTER_OTLP_*
|
||||
# endpoint is also set: those exporters are built unconditionally, so "console"
|
||||
# would otherwise be a false statement about where the run's contents went.
|
||||
# otlp -> spans are EXPORTED OVER THE NETWORK, and only to an endpoint you named below.
|
||||
# Requires an OTLP exporter package, which this framework deliberately does NOT
|
||||
# declare (`pip install opentelemetry-exporter-otlp-proto-grpc`).
|
||||
# PORTFOLIO_OTEL=console
|
||||
# OTEL_EXPORTER_OTLP_ENDPOINT= # required by PORTFOLIO_OTEL=otlp; egress — name it yourself
|
||||
|
|
|
|||
33
llms.txt
Normal file
33
llms.txt
Normal file
|
|
@ -0,0 +1,33 @@
|
|||
# portfolio-optimiser
|
||||
|
||||
> A generic, open Python framework on Microsoft Agent Framework (MAF) that finds cost savings
|
||||
> inside each project of a portfolio of independent projects. A swarm of agents proposes candidate
|
||||
> measures; a mandatory deterministic validator (solver + Monte Carlo) decides the numbers; domain
|
||||
> experts judge the outcomes; and the system learns from their verdicts across runs. Solo-maintained,
|
||||
> fork-and-own, MIT. Python >=3.10, installed from source with `uv`.
|
||||
|
||||
One start command. The clone is the documented path — the walkthrough in the README points at
|
||||
files in the tree:
|
||||
|
||||
```bash
|
||||
git clone https://git.fromaitochitta.com/open/portfolio-optimiser.git
|
||||
cd portfolio-optimiser
|
||||
uv sync
|
||||
```
|
||||
|
||||
## Docs
|
||||
|
||||
- [README.md](README.md): what the framework is, an offline walkthrough of the whole chain (no API key, no network), and the task API.
|
||||
- [shared/method-spec.md](shared/method-spec.md): the framework-neutral method specification the implementation is measured against.
|
||||
- [docs/extending.md](docs/extending.md): the extension points — domains, data sources, ingest connectors — and what is deliberately unbuilt.
|
||||
- [docs/knowledge-base-recipe.md](docs/knowledge-base-recipe.md): how to build a knowledge-base bundle the run path can navigate.
|
||||
- [docs/kunnskapsbase-for-en-kjoring.md](docs/kunnskapsbase-for-en-kjoring.md): what one concrete run's knowledge base must contain (Norwegian).
|
||||
- [docs/bestille-en-kjoring.md](docs/bestille-en-kjoring.md): what a requester supplies to order a run (Norwegian).
|
||||
- [DEPLOY.md](DEPLOY.md): running the hosted entry point (`python main.py`) and the environment it needs.
|
||||
- [CLAUDE.md](CLAUDE.md): the invariant ledger — each design decision, the measurement that forced it, and the test that turns red when it is undone.
|
||||
|
||||
## Optional
|
||||
|
||||
- [CHANGELOG.md](CHANGELOG.md): release history.
|
||||
- [SECURITY.md](SECURITY.md): how to report a vulnerability, and what is supported.
|
||||
- [CONTRIBUTING.md](CONTRIBUTING.md): issues welcome as signals; pull requests are not accepted.
|
||||
9
main.py
9
main.py
|
|
@ -1,9 +1,10 @@
|
|||
"""Hosted-agent process entry (Fase 4d): the ONE file the container starts.
|
||||
"""Hosted-agent process entry (Fase 4d): the ONE file that starts the service.
|
||||
|
||||
Thin by design — the runtime contract lives in ``portfolio_optimiser.hosting`` (packaged
|
||||
and tested); this file is the scaffold-conventional ``main.py`` that the Dockerfile ``CMD``
|
||||
and ``azure.yaml`` point at. It is part of the git archive (the build context), not of the
|
||||
wheel: a wheel consumer starts the same server with
|
||||
and tested); this file is the scaffold-conventional ``main.py`` that ``python main.py``
|
||||
starts, which is the single start command DEPLOY.md prints (14.08: the delivery is runnable
|
||||
Python, and no image ``CMD`` exists to be a second copy of it). It is part of the git archive,
|
||||
not of the wheel: a wheel consumer starts the same server with
|
||||
``python -c "from portfolio_optimiser.hosting import main; main()"``.
|
||||
"""
|
||||
|
||||
|
|
|
|||
|
|
@ -28,7 +28,18 @@ dependencies = [
|
|||
# `uv run mypy src` outright. Raise both floors together or not at all.
|
||||
"numpy>=2.0,<2.3",
|
||||
"llm-ingestion-okf", # Door A ingest (§4–§6) — the shared implementation of shared/ingest-spec.md; zero runtime deps, MAF-free (D7)
|
||||
"llm-ingestion-guard", # Door A content gate — scans materialized concepts before they reach the bundle; zero runtime deps, MAF-free (D7)
|
||||
"llm-ingestion-guard",
|
||||
# U14 (økt 55, operatørbeslutning 23.08): the tracing seam's ONE new runtime dependency.
|
||||
# `opentelemetry-api` was already here transitively via agent-framework-core, but the SDK is
|
||||
# what `configure_otel_providers()` needs — without it MAF raises ModuleNotFoundError, so the
|
||||
# seam cannot land as an opt-in without declaring it. The console exporter this framework uses
|
||||
# (`opentelemetry.sdk.trace.export.ConsoleSpanExporter`) ships INSIDE this package, so console
|
||||
# mode costs exactly one declaration. The OTLP exporter packages are deliberately NOT declared:
|
||||
# they are egress, they drag grpc/protobuf into a published wheel, and MAF already raises a
|
||||
# named ImportError telling the operator which one to install. Stated honesty limit, not an
|
||||
# oversight — see tracing.py.
|
||||
"opentelemetry-sdk>=1.42,<2",
|
||||
# Door A content gate — scans materialized concepts before they reach the bundle; zero runtime deps, MAF-free (D7)
|
||||
]
|
||||
|
||||
# Console entry points — the install surface a fresh clone gets from `uv sync` (P4 pkt. 5).
|
||||
|
|
|
|||
35
scripts/make-handover-package.sh
Executable file
35
scripts/make-handover-package.sh
Executable file
|
|
@ -0,0 +1,35 @@
|
|||
#!/usr/bin/env bash
|
||||
# Build the external handover package (Fase 5): one archive a receiver deploys into their own
|
||||
# Microsoft Foundry, without cloning this repository or having an account on our Forgejo.
|
||||
#
|
||||
# The archive is git's view of HEAD -- TRACKED FILES ONLY. That is deliberate and it is the whole
|
||||
# exposure control: STATE.md is gitignored, *.local.md is gitignored, .env is gitignored, so they
|
||||
# cannot enter the archive. A filter maintained here would be a second copy of that rule, and the
|
||||
# second copy is the one that drifts (kø-(p)).
|
||||
#
|
||||
# What the receiver gets is therefore HEAD itself -- never a hand-curated selection. That also
|
||||
# decided how the 14.08 directive ("runnable Python, no container wrapper") was carried out: the
|
||||
# Dockerfile and azure.yaml were removed from the TREE, not filtered out here. A filter would have
|
||||
# been a curation step deciding what a receiver sees, i.e. exactly the second copy this comment
|
||||
# exists to forbid. Absence is asserted in tests/test_handover_package_loadbearing.py.
|
||||
#
|
||||
# Usage: scripts/make-handover-package.sh [dest-dir] (default: dist/)
|
||||
# Gated by tests/test_handover_package_loadbearing.py.
|
||||
set -euo pipefail
|
||||
|
||||
DEST="${1:-dist}"
|
||||
mkdir -p "$DEST"
|
||||
|
||||
# Version from pyproject.toml -- the one place the build stamps it. Read, never hardcoded: a second
|
||||
# copy here would go stale at the next bump exactly like the README wheel filename did (Fase 3).
|
||||
VERSION=$(grep -m1 '^version = ' pyproject.toml | cut -d'"' -f2)
|
||||
if [ -z "$VERSION" ]; then
|
||||
echo "make-handover-package: could not read version from pyproject.toml" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
OUT="$DEST/portfolio-optimiser-foundry-$VERSION.zip"
|
||||
|
||||
git archive --format=zip --output "$OUT" HEAD
|
||||
|
||||
echo "$OUT"
|
||||
646
spikes/e_magentic.py
Normal file
646
spikes/e_magentic.py
Normal file
|
|
@ -0,0 +1,646 @@
|
|||
"""Spike E — the Magentic exploration loop, measured BEFORE it is built
|
||||
(order 20260823T162224Z; plan ``docs/plan/2026-08-23-magentic-utforskningssloeyfe.md`` § D.1).
|
||||
|
||||
Nothing here is production code and nothing here is wired into ``src/``. Each function is one
|
||||
measurement whose outcome moves a row of the plan's § F assumption table from "umålt" to a fact.
|
||||
|
||||
**The client is the repo's own ``ScriptedChatClient``, not an ad-hoc fake.** The scratch scripts
|
||||
this spike ports used a bare ``BaseChatClient``, on which ``BudgetMiddleware`` is silently a
|
||||
no-op (measured, ``simulation.py:373-375``) — so a budget claim proved with one would have proved
|
||||
nothing. ``ScriptedChatClient`` subclasses the LAYERED ``OpenAIChatCompletionClient``, which is
|
||||
what makes S2 a real measurement.
|
||||
|
||||
**The budget types are the PRODUCTION ones** (``portfolio_optimiser.budget``), deliberately NOT
|
||||
``spikes/_harness.py``'s private copy. The harness copy is exactly why ``tick_round``'s
|
||||
``observed`` went four raise-sites without coverage (kø-(y)); S2's whole question is whether the
|
||||
SHIPPED middleware reaches the manager, and only the shipped object can answer it.
|
||||
|
||||
**Routing is on the joined prompt blob, in a fixed priority order, and that order is
|
||||
load-bearing.** ``ScriptedChatClient``'s selector receives the concatenation of every message in
|
||||
the call, so one manager call carries two markers (the plan prompt is built on a history that
|
||||
still holds the pre-survey text — measured: 1 ambiguous call in 5). Checking the later-stage
|
||||
marker first resolves it; the five-kind call shape each experiment asserts is what proves the
|
||||
routing stayed correct.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import statistics
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from collections.abc import Callable, Sequence
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from agent_framework import Agent, FileCheckpointStorage
|
||||
from agent_framework.orchestrations import (
|
||||
AgentRequestInfoResponse,
|
||||
MagenticAgentExecutor,
|
||||
MagenticBuilder,
|
||||
MagenticPlanReviewResponse,
|
||||
MagenticResetSignal,
|
||||
StandardMagenticManager,
|
||||
)
|
||||
|
||||
# ``AgentApprovalExecutor`` is the ONE name S3b needs that the package does not re-export
|
||||
# (measured: ``hasattr(agent_framework.orchestrations, "AgentApprovalExecutor")`` is False while
|
||||
# its response type IS public). Reaching into the private module is therefore part of the S3b
|
||||
# FINDING, not an oversight: door 3 of § C.6 currently costs a private-API dependency, and that
|
||||
# is a fact the operator's decision needs to carry.
|
||||
from agent_framework_orchestrations._orchestration_request_info import AgentApprovalExecutor
|
||||
|
||||
import portfolio_optimiser
|
||||
from portfolio_optimiser.budget import Budget, BudgetExceeded, BudgetMiddleware, TokenMeter
|
||||
from portfolio_optimiser.ir import SavingsProposal
|
||||
from portfolio_optimiser.okf import load_ir_projection, load_optional_cost_baseline
|
||||
from portfolio_optimiser.simulation import ScriptedChatClient
|
||||
from portfolio_optimiser.validator import validate_proposal
|
||||
|
||||
TASK_ALPHA = "TASK-ALPHA: find the saving in the alpha project."
|
||||
TASK_BETA = "TASK-BETA: find the saving in the beta project."
|
||||
WORKER_SENTINEL = "WORKER-SAW"
|
||||
|
||||
|
||||
def _ledger(*, satisfied: bool, speaker: str) -> str:
|
||||
"""A progress ledger naming ``speaker`` as the next talker.
|
||||
|
||||
The name is a PARAMETER because an unknown ``next_speaker`` is a silent footgun: the
|
||||
orchestrator does not error on it, it quietly produces a final answer with zero participant
|
||||
work (``_magentic.py:1128-1131``). Hard-coding "worker" here made the S3b run — whose
|
||||
participant is the ``expert_liaison`` — finish without ever asking anyone (measured).
|
||||
"""
|
||||
return json.dumps(
|
||||
{
|
||||
"is_request_satisfied": {
|
||||
"reason": "the participant replied" if satisfied else "no one has spoken yet",
|
||||
"answer": satisfied,
|
||||
},
|
||||
"is_in_loop": {"reason": "no", "answer": False},
|
||||
"is_progress_being_made": {"reason": "yes", "answer": True},
|
||||
"next_speaker": {"reason": "it does the work", "answer": speaker},
|
||||
"instruction_or_question": {
|
||||
"reason": "done" if satisfied else "kick off",
|
||||
"answer": "none" if satisfied else "Do the work now.",
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ExplorationCallRecord:
|
||||
"""One manager call, reduced to the facts every verdict here reads.
|
||||
|
||||
``sees_marker`` is the HITL half (S3b): whether a caller-supplied sentinel — an expert's
|
||||
answer injected mid-run — had reached this manager prompt. It defaults to ``False`` so the
|
||||
contamination experiments, which supply no marker, are unchanged.
|
||||
"""
|
||||
|
||||
kind: str
|
||||
messages: int
|
||||
sees_alpha: bool
|
||||
sees_beta: bool
|
||||
sees_marker: bool = False
|
||||
|
||||
|
||||
def _route(blob: str, speaker: str) -> tuple[str, str]:
|
||||
"""Map a manager prompt blob to ``(kind, reply)``.
|
||||
|
||||
Order matters: the later-stage marker is tested FIRST because an earlier stage's text is
|
||||
still present in the joined blob. Reversing two of these silently changes which prompt a
|
||||
kind is attributed to, which is why every experiment asserts the resulting call shape.
|
||||
"""
|
||||
if "provide the final answer" in blob:
|
||||
return "final", "FINAL: the worker did it."
|
||||
if "pure JSON format" in blob:
|
||||
if WORKER_SENTINEL in blob:
|
||||
return "ledger_SAT", _ledger(satisfied=True, speaker=speaker)
|
||||
return "ledger_UNSAT", _ledger(satisfied=False, speaker=speaker)
|
||||
if "went wrong on this last run" in blob:
|
||||
return "plan_update", "PLAN-UPDATE: ask the worker again."
|
||||
if "rewrite the following fact sheet" in blob:
|
||||
return "facts_update", "FACTS-UPDATE: still nothing."
|
||||
if "bullet-point plan" in blob:
|
||||
return "plan", "PLAN: - ask the worker"
|
||||
if "pre-survey" in blob:
|
||||
return "facts", "FACTS: nothing given."
|
||||
return "unknown", "{}"
|
||||
|
||||
|
||||
def _manager_client(
|
||||
records: list[ExplorationCallRecord],
|
||||
*,
|
||||
marker: str | None = None,
|
||||
speaker: str = "worker",
|
||||
) -> ScriptedChatClient:
|
||||
"""A manager-shaped scripted client appending one record per call to ``records``.
|
||||
|
||||
``records`` is CALLER-owned (the parse-failure-capture precedent): the evidence must survive
|
||||
however the run ended, including a run cut short by ``BudgetExceeded`` mid-way.
|
||||
"""
|
||||
|
||||
def _select(blob: str, _role: str) -> str:
|
||||
kind, reply = _route(blob, speaker)
|
||||
records.append(
|
||||
ExplorationCallRecord(
|
||||
kind=kind,
|
||||
messages=blob.count("\n") + 1,
|
||||
sees_alpha="TASK-ALPHA" in blob,
|
||||
sees_beta="TASK-BETA" in blob,
|
||||
sees_marker=marker is not None and marker in blob,
|
||||
)
|
||||
)
|
||||
return reply
|
||||
|
||||
return ScriptedChatClient(reply_selector=_select, role="manager")
|
||||
|
||||
|
||||
def _worker_client(seen: list[str]) -> ScriptedChatClient:
|
||||
def _select(blob: str, _role: str) -> str:
|
||||
which = "ALPHA" if "TASK-ALPHA" in blob else ("BETA" if "TASK-BETA" in blob else "NOTHING")
|
||||
seen.append(which)
|
||||
return f"{WORKER_SENTINEL}-{which}"
|
||||
|
||||
return ScriptedChatClient(reply_selector=_select, role="worker")
|
||||
|
||||
|
||||
def _manager_agent(client: ScriptedChatClient, *, middleware: Sequence[Any] | None = None) -> Agent:
|
||||
return Agent(
|
||||
client,
|
||||
"You are the Magentic manager.",
|
||||
name="manager",
|
||||
description="plans the work",
|
||||
middleware=middleware,
|
||||
)
|
||||
|
||||
|
||||
def _worker_agent(client: ScriptedChatClient) -> Agent:
|
||||
return Agent(client, "You are the worker.", name="worker", description="does the work")
|
||||
|
||||
|
||||
async def _run(workflow: Any, task: str) -> dict[str, Any]:
|
||||
try:
|
||||
result = await workflow.run(task)
|
||||
outputs = [str(o) for o in (result.get_outputs() or [])]
|
||||
return {"ok": True, "outputs": outputs}
|
||||
except Exception as exc: # noqa: BLE001 - the failure mode IS the measurement
|
||||
return {"ok": False, "error": type(exc).__name__, "message": str(exc)}
|
||||
|
||||
|
||||
def manager_keeps_persistent_session() -> bool:
|
||||
"""Does the INSTALLED ``StandardMagenticManager`` hold one ``AgentSession`` for its whole
|
||||
life, or mint a throwaway one per call?
|
||||
|
||||
This is the single structural property that decides E2 and E4. Orchestrations 1.0.0 assigns
|
||||
``self._session = self._agent.create_session()`` in ``__init__``; 1.0.1 removed that line and
|
||||
creates the session inside the call instead (upstream regression fix #4371). Probing the
|
||||
attribute rather than the version string states the CAUSE, and keeps the spike honest across
|
||||
a version the plan has not seen.
|
||||
"""
|
||||
manager = StandardMagenticManager(agent=_manager_agent(_manager_client([])))
|
||||
return hasattr(manager, "_session")
|
||||
|
||||
|
||||
async def single_use_second_run() -> dict[str, Any]:
|
||||
"""E1: build one Magentic workflow, run it twice. Measures the second run's outcome and
|
||||
what it COST — a refusal that still made model calls would be a different finding."""
|
||||
records: list[ExplorationCallRecord] = []
|
||||
worker_seen: list[str] = []
|
||||
workflow = MagenticBuilder(
|
||||
participants=[_worker_agent(_worker_client(worker_seen))],
|
||||
manager_agent=_manager_agent(_manager_client(records)),
|
||||
max_round_count=6,
|
||||
).build()
|
||||
|
||||
first = await _run(workflow, TASK_ALPHA)
|
||||
manager_before, worker_before = len(records), len(worker_seen)
|
||||
second = await _run(workflow, TASK_BETA)
|
||||
return {
|
||||
"first_ok": first["ok"],
|
||||
"second_error": second.get("error"),
|
||||
"second_message": second.get("message", ""),
|
||||
"manager_calls_added": len(records) - manager_before,
|
||||
"worker_calls_added": len(worker_seen) - worker_before,
|
||||
}
|
||||
|
||||
|
||||
async def _two_runs(
|
||||
build_second: Callable[[list[ExplorationCallRecord]], Any],
|
||||
*,
|
||||
shared_records: bool,
|
||||
) -> tuple[int, int, list[str]]:
|
||||
"""Run ALPHA, then BETA, and report how many of run 2's manager calls still see ALPHA.
|
||||
|
||||
``shared_records`` says whether run 2's manager is the same object as run 1's — when it is,
|
||||
run 2's records are the tail of one list; when it is not, they are their own list. Both
|
||||
shapes reduce to the same verdict triple so the callers stay comparable.
|
||||
"""
|
||||
records: list[ExplorationCallRecord] = []
|
||||
first_worker: list[str] = []
|
||||
first = MagenticBuilder(
|
||||
participants=[_worker_agent(_worker_client(first_worker))],
|
||||
manager_agent=_manager_agent(_manager_client(records)),
|
||||
max_round_count=6,
|
||||
).build()
|
||||
await _run(first, TASK_ALPHA)
|
||||
split = len(records)
|
||||
|
||||
second_records = records if shared_records else []
|
||||
second = build_second(second_records)
|
||||
await _run(second, TASK_BETA)
|
||||
|
||||
tail = records[split:] if shared_records else second_records
|
||||
bled = sum(1 for record in tail if record.sees_alpha)
|
||||
return bled, len(tail), [record.kind for record in tail]
|
||||
|
||||
|
||||
async def shared_manager_contamination() -> tuple[int, int, list[str]]:
|
||||
"""E2: two builders sharing ONE ``StandardMagenticManager`` instance.
|
||||
|
||||
Built directly (not via ``_two_runs``) because run 1 must go through the SAME manager object,
|
||||
which ``manager=`` accepts and ``manager_agent=`` does not.
|
||||
"""
|
||||
records: list[ExplorationCallRecord] = []
|
||||
shared = StandardMagenticManager(
|
||||
agent=_manager_agent(_manager_client(records)), max_round_count=6
|
||||
)
|
||||
|
||||
first = MagenticBuilder(
|
||||
participants=[_worker_agent(_worker_client([]))], manager=shared
|
||||
).build()
|
||||
await _run(first, TASK_ALPHA)
|
||||
split = len(records)
|
||||
|
||||
second = MagenticBuilder(
|
||||
participants=[_worker_agent(_worker_client([]))], manager=shared
|
||||
).build()
|
||||
await _run(second, TASK_BETA)
|
||||
|
||||
tail = records[split:]
|
||||
return sum(1 for r in tail if r.sees_alpha), len(tail), [r.kind for r in tail]
|
||||
|
||||
|
||||
async def shared_builder_contamination() -> tuple[int, int, list[str]]:
|
||||
"""E4: ONE ``MagenticBuilder``, ``.build()`` twice. ``manager_agent=`` constructs the manager
|
||||
eagerly and hands the same instance to every build — the accidental route into E2."""
|
||||
records: list[ExplorationCallRecord] = []
|
||||
builder = MagenticBuilder(
|
||||
participants=[_worker_agent(_worker_client([]))],
|
||||
manager_agent=_manager_agent(_manager_client(records)),
|
||||
max_round_count=6,
|
||||
)
|
||||
|
||||
await _run(builder.build(), TASK_ALPHA)
|
||||
split = len(records)
|
||||
await _run(builder.build(), TASK_BETA)
|
||||
|
||||
tail = records[split:]
|
||||
return sum(1 for r in tail if r.sees_alpha), len(tail), [r.kind for r in tail]
|
||||
|
||||
|
||||
async def fresh_manager_contamination() -> tuple[int, int, list[str]]:
|
||||
"""E3, the control and the mitigation: a fresh builder, agent and client per exploration."""
|
||||
|
||||
def _second(records: list[ExplorationCallRecord]) -> Any:
|
||||
return MagenticBuilder(
|
||||
participants=[_worker_agent(_worker_client([]))],
|
||||
manager_agent=_manager_agent(_manager_client(records)),
|
||||
max_round_count=6,
|
||||
).build()
|
||||
|
||||
return await _two_runs(_second, shared_records=False)
|
||||
|
||||
|
||||
async def reset_signal_resets_participant_session() -> dict[str, Any]:
|
||||
"""E7: does ``MagenticResetSignal`` actually give the participant a clean session?
|
||||
|
||||
Measured no: the fresh session is written to ``_agent_thread``, which nothing reads, while
|
||||
the live ``_session`` keeps its identity. The stall-replan path therefore hands the manager a
|
||||
clean ledger and the participants their old memory.
|
||||
"""
|
||||
executor = MagenticAgentExecutor(_worker_agent(_worker_client([])))
|
||||
session_before = id(executor._session)
|
||||
executor._cache.append("sentinel") # type: ignore[arg-type]
|
||||
executor._full_conversation.append("sentinel") # type: ignore[arg-type]
|
||||
|
||||
await executor.handle_magentic_reset(MagenticResetSignal(), None) # type: ignore[arg-type]
|
||||
|
||||
return {
|
||||
"cache_cleared": len(executor._cache) == 0,
|
||||
"conversation_cleared": len(executor._full_conversation) == 0,
|
||||
"session_identity_changed": id(executor._session) != session_before,
|
||||
"orphan_attribute_written": hasattr(executor, "_agent_thread"),
|
||||
}
|
||||
|
||||
|
||||
async def manager_budget_enforced(
|
||||
*, max_tokens: int, attach: bool, return_exception: bool = False
|
||||
) -> dict[str, Any]:
|
||||
"""S2: put the SHIPPED ``BudgetMiddleware`` on the manager agent and see whether the typed
|
||||
refusal leaves ``workflow.run``.
|
||||
|
||||
``attach=False`` is the detach control: the same one-token budget with no middleware must
|
||||
stop nothing. Without that arm the positive test would pass on any implementation in which
|
||||
something, anything, raised.
|
||||
"""
|
||||
meter = TokenMeter(Budget(max_tokens=max_tokens, max_rounds=8))
|
||||
middleware = [BudgetMiddleware(meter)] if attach else None
|
||||
workflow = MagenticBuilder(
|
||||
participants=[_worker_agent(_worker_client([]))],
|
||||
manager_agent=_manager_agent(_manager_client([]), middleware=middleware),
|
||||
max_round_count=6,
|
||||
).build()
|
||||
|
||||
raised: str | None = None
|
||||
kind: str | None = None
|
||||
exception: BaseException | None = None
|
||||
completed = False
|
||||
try:
|
||||
result = await workflow.run(TASK_ALPHA)
|
||||
completed = bool(result.get_outputs())
|
||||
except BudgetExceeded as exc:
|
||||
raised, kind, exception = type(exc).__name__, exc.kind, exc
|
||||
except Exception as exc: # noqa: BLE001 - a DIFFERENT exception type is itself the finding
|
||||
raised, exception = type(exc).__name__, exc
|
||||
|
||||
payload: dict[str, Any] = {
|
||||
"raised": raised,
|
||||
"kind": kind,
|
||||
"completed": completed,
|
||||
"meter_tokens": meter.tokens,
|
||||
}
|
||||
if return_exception:
|
||||
payload["exception"] = exception
|
||||
return payload
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# S3 / S3b — the two HITL doors (plan § C.5, § C.6)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _pending_requests(result: Any) -> list[Any]:
|
||||
return [event for event in result if event.type == "request_info"]
|
||||
|
||||
|
||||
async def plan_review_round_trip() -> dict[str, Any]:
|
||||
"""S3: ``enable_plan_review=True`` → the run stops with a ``MagenticPlanReviewRequest`` and no
|
||||
output → ``revise(...)`` replans and asks AGAIN → ``approve()`` lets the loop run.
|
||||
|
||||
The manager-call count PER revise is the number the contract needs: a revise costs model
|
||||
calls but is not counted as a round by the orchestration (measured: no ledger call), so an
|
||||
uncapped reviser is an unbounded spend the plan's ``max_plan_revisions`` has to bound.
|
||||
"""
|
||||
records: list[ExplorationCallRecord] = []
|
||||
workflow = MagenticBuilder(
|
||||
participants=[_worker_agent(_worker_client([]))],
|
||||
manager_agent=_manager_agent(_manager_client(records)),
|
||||
max_round_count=6,
|
||||
enable_plan_review=True,
|
||||
).build()
|
||||
|
||||
first = await workflow.run(TASK_ALPHA)
|
||||
pending = _pending_requests(first)
|
||||
stopped_without_output = not first.get_outputs()
|
||||
review_kinds = [r.kind for r in records]
|
||||
|
||||
before_revise = len(records)
|
||||
revised = await workflow.run(
|
||||
responses={pending[0].request_id: MagenticPlanReviewResponse.revise("Test the LED case.")}
|
||||
)
|
||||
revise_calls = [r.kind for r in records[before_revise:]]
|
||||
pending_after_revise = _pending_requests(revised)
|
||||
|
||||
approve_id = (
|
||||
pending_after_revise[0].request_id if pending_after_revise else pending[0].request_id
|
||||
)
|
||||
approved = await workflow.run(responses={approve_id: MagenticPlanReviewResponse.approve()})
|
||||
|
||||
return {
|
||||
"pending_before_review": len(pending),
|
||||
"request_type": type(pending[0].data).__name__ if pending else None,
|
||||
"is_stalled": bool(pending[0].data.is_stalled) if pending else None,
|
||||
"stopped_without_output": stopped_without_output,
|
||||
"kinds_before_review": review_kinds,
|
||||
"revise_manager_calls": revise_calls,
|
||||
"pending_after_revise": len(pending_after_revise),
|
||||
"outputs_after_approve": [str(o) for o in (approved.get_outputs() or [])],
|
||||
}
|
||||
|
||||
|
||||
async def expert_liaison_answer_round_trip(
|
||||
*, answer: str, probe: str | None = None
|
||||
) -> dict[str, Any]:
|
||||
"""S3b: an ``AgentApprovalExecutor`` standing in as the ``expert_liaison`` PARTICIPANT — the
|
||||
third door of § C.6, the one that lets the manager ask a question MID-run.
|
||||
|
||||
Measured shape, two round-trips per human turn: the manager picks the liaison → its output
|
||||
becomes a ``request_info`` → ``from_strings([answer])`` feeds the human's words back INTO the
|
||||
liaison, which runs again → a second ``request_info`` → ``approve()`` forwards the liaison's
|
||||
(now informed) output to the manager, which resumes. ``from_strings`` alone does NOT resume
|
||||
the manager: measured zero manager calls between the two requests.
|
||||
|
||||
``probe`` is the CONTROL knob: the manager's prompts are scanned for IT instead of for
|
||||
``answer``, so a sentinel the expert never sent must come back absent. Without that arm,
|
||||
"the manager saw the answer" could equally mean "the scanner matches anything".
|
||||
"""
|
||||
records: list[ExplorationCallRecord] = []
|
||||
liaison_prompts: list[str] = []
|
||||
|
||||
def _liaison_reply(blob: str, _role: str) -> str:
|
||||
liaison_prompts.append(blob)
|
||||
if answer in blob:
|
||||
return f"{WORKER_SENTINEL}-LIAISON heard: {answer}"
|
||||
return f"{WORKER_SENTINEL}-LIAISON has no expert input yet."
|
||||
|
||||
liaison = AgentApprovalExecutor(
|
||||
Agent(
|
||||
ScriptedChatClient(reply_selector=_liaison_reply, role="expert_liaison"),
|
||||
"You relay the expert's answers.",
|
||||
name="expert_liaison",
|
||||
description="asks the human expert",
|
||||
)
|
||||
)
|
||||
workflow = MagenticBuilder(
|
||||
participants=[liaison],
|
||||
manager_agent=_manager_agent(
|
||||
_manager_client(records, marker=probe or answer, speaker="expert_liaison")
|
||||
),
|
||||
max_round_count=6,
|
||||
).build()
|
||||
|
||||
first = await workflow.run(TASK_ALPHA)
|
||||
asked = _pending_requests(first)
|
||||
|
||||
before_answer = len(records)
|
||||
answered = await workflow.run(
|
||||
responses={asked[0].request_id: AgentRequestInfoResponse.from_strings([answer])}
|
||||
)
|
||||
manager_calls_on_answer = [r.kind for r in records[before_answer:]]
|
||||
second = _pending_requests(answered)
|
||||
|
||||
approved = (
|
||||
await workflow.run(responses={second[0].request_id: AgentRequestInfoResponse.approve()})
|
||||
if second
|
||||
else answered
|
||||
)
|
||||
|
||||
return {
|
||||
"reachable": bool(asked),
|
||||
"request_type": type(asked[0].data).__name__ if asked else None,
|
||||
"manager_calls_between_requests": manager_calls_on_answer,
|
||||
"second_request": len(second),
|
||||
"liaison_saw_answer": any(answer in prompt for prompt in liaison_prompts),
|
||||
"manager_saw_answer": any(r.sees_marker for r in records),
|
||||
"outputs": [str(o) for o in (approved.get_outputs() or [])],
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# S4 — resume a pending plan review in a NEW PROCESS (plan U12)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def checkpoint_until_plan_review(storage_dir: str) -> dict[str, Any]:
|
||||
"""Run until the plan review stops the workflow, leaving checkpoints on disk.
|
||||
|
||||
This is the FIRST half of S4 and runs in the parent process; the resume half must run in a
|
||||
separate interpreter (``spikes.e_magentic_resume``), because a resume that quietly rode on
|
||||
live in-process objects would prove nothing about the asynchronous file inbox U12 needs.
|
||||
"""
|
||||
records: list[ExplorationCallRecord] = []
|
||||
workflow = _plan_review_workflow(records, storage_dir)
|
||||
result = await workflow.run(TASK_ALPHA)
|
||||
pending = _pending_requests(result)
|
||||
checkpoints = await _checkpoint_storage(storage_dir).list_checkpoints(
|
||||
workflow_name=workflow.name
|
||||
)
|
||||
return {
|
||||
"request_id": pending[0].request_id if pending else None,
|
||||
"checkpoint_ids": [c.checkpoint_id for c in checkpoints],
|
||||
"outputs": [str(o) for o in (result.get_outputs() or [])],
|
||||
}
|
||||
|
||||
|
||||
# Measured, and a real cost of the asynchronous HITL door: ``FileCheckpointStorage`` refuses to
|
||||
# unpickle a plan-review request unless its type is declared. Without this the checkpoint file is
|
||||
# written but comes back UNREADABLE ("Checkpoint deserialization blocked for type ..."), and the
|
||||
# listing is empty — a resume that fails as an absence rather than as an error, which is exactly
|
||||
# the shape the fourth face of the verification law warns about. Both processes must declare it.
|
||||
_ALLOWED_CHECKPOINT_TYPES = [
|
||||
"agent_framework_orchestrations._magentic:MagenticPlanReviewRequest",
|
||||
"agent_framework_orchestrations._magentic:MagenticPlanReviewResponse",
|
||||
]
|
||||
|
||||
|
||||
def _checkpoint_storage(storage_dir: str) -> FileCheckpointStorage:
|
||||
return FileCheckpointStorage(storage_dir, allowed_checkpoint_types=_ALLOWED_CHECKPOINT_TYPES)
|
||||
|
||||
|
||||
def _plan_review_workflow(records: list[ExplorationCallRecord], storage_dir: str) -> Any:
|
||||
"""The workflow BOTH processes build — identical construction, so the only thing carried
|
||||
across the process boundary is the checkpoint on disk."""
|
||||
return (
|
||||
MagenticBuilder(
|
||||
participants=[_worker_agent(_worker_client([]))],
|
||||
manager_agent=_manager_agent(_manager_client(records)),
|
||||
max_round_count=6,
|
||||
enable_plan_review=True,
|
||||
)
|
||||
.with_checkpointing(_checkpoint_storage(storage_dir))
|
||||
.build()
|
||||
)
|
||||
|
||||
|
||||
async def resume_from_checkpoint(
|
||||
storage_dir: str, *, request_id: str, checkpoint_id: str
|
||||
) -> dict[str, Any]:
|
||||
"""The SECOND half of S4, called by ``spikes.e_magentic_resume`` in a fresh interpreter.
|
||||
|
||||
**Measured, and it contradicts the plan's E-table:** ``checkpoint_storage=`` on ``run()`` is
|
||||
NOT the load-bearing seam here — removing it leaves the whole suite green (920 passed),
|
||||
because ``.with_checkpointing(...)`` on the builder already gave this workflow its storage.
|
||||
The two arguments that ARE load-bearing are ``checkpoint_id=`` (drop it → red) and the
|
||||
builder's ``.with_checkpointing(...)`` (drop it → red). It is passed anyway, explicitly,
|
||||
because an exploration layer that builds its workflow WITHOUT checkpointing and resumes by
|
||||
handing storage in at call time is a legitimate second shape — but a criterion that names it
|
||||
as the detach point would be a gate that cannot go red.
|
||||
"""
|
||||
records: list[ExplorationCallRecord] = []
|
||||
workflow = _plan_review_workflow(records, storage_dir)
|
||||
result = await workflow.run(
|
||||
responses={request_id: MagenticPlanReviewResponse.approve()},
|
||||
checkpoint_id=checkpoint_id,
|
||||
checkpoint_storage=_checkpoint_storage(storage_dir),
|
||||
)
|
||||
return {
|
||||
"manager_kinds": [r.kind for r in records],
|
||||
"pending_after_resume": len(_pending_requests(result)),
|
||||
"outputs": [str(o) for o in (result.get_outputs() or [])],
|
||||
}
|
||||
|
||||
|
||||
def run_resume_subprocess(
|
||||
storage_dir: str, *, request_id: str, checkpoint_id: str
|
||||
) -> dict[str, Any]:
|
||||
"""Launch ``spikes.e_magentic_resume`` in a FRESH interpreter and parse its one JSON line.
|
||||
|
||||
``sys.executable`` is this venv's Python, and the repo root is the working directory, so the
|
||||
child imports the same tree the parent did without any path juggling. A non-zero exit is
|
||||
surfaced with the child's stderr attached: a resume that failed must read as a failed resume,
|
||||
never as an empty result.
|
||||
"""
|
||||
completed = subprocess.run(
|
||||
[sys.executable, "-m", "spikes.e_magentic_resume", storage_dir, request_id, checkpoint_id],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
cwd=str(Path(__file__).resolve().parents[1]),
|
||||
)
|
||||
if completed.returncode != 0:
|
||||
raise RuntimeError(
|
||||
f"resume subprocess exited {completed.returncode}: {completed.stderr.strip()}"
|
||||
)
|
||||
return dict(json.loads(completed.stdout.strip().splitlines()[-1]))
|
||||
|
||||
|
||||
def micro_bundle_dir() -> str:
|
||||
"""The repo's own anchored micro bundle — the one bundle that ships BOTH a cost baseline and
|
||||
an IR projection, so a latency number measured here is measured through the WHOLE gate
|
||||
(stage 0 reconciliation + CBC solve + 512-sample Monte Carlo), not a subset of it."""
|
||||
return str(
|
||||
Path(portfolio_optimiser.__file__).parent
|
||||
/ "data"
|
||||
/ "bundles"
|
||||
/ "bygg-energi-baseline-mikro"
|
||||
)
|
||||
|
||||
|
||||
def micro_proposal() -> SavingsProposal:
|
||||
"""The bundle's IR projection as a validated ``SavingsProposal``, carrying an assumption band.
|
||||
|
||||
The band is not decoration: without one, ``validator._monte_carlo`` falls back to each item's
|
||||
own ``unit_cost`` and every draw is identical — a cheaper computation than any real hypothesis
|
||||
would trigger, so a latency measured without it would understate the in-loop cost.
|
||||
"""
|
||||
projection = dict(load_ir_projection(micro_bundle_dir()))
|
||||
projection.pop("_note", None)
|
||||
item = projection["affected_items"][0]
|
||||
unit_cost = float(item["unit_cost"])
|
||||
projection["assumptions"] = {item["code"]: (unit_cost * 0.9, unit_cost * 1.1)}
|
||||
return SavingsProposal.model_validate(projection)
|
||||
|
||||
|
||||
def validator_latency_seconds(*, runs: int) -> tuple[float, int]:
|
||||
"""S5: median wall-clock of one ``validate_proposal`` against the micro reference bundle —
|
||||
the cost the hypothesiser's ``quick_validate`` tool would pay per call, per hypothesis."""
|
||||
baseline = load_optional_cost_baseline(micro_bundle_dir())
|
||||
proposal = micro_proposal()
|
||||
|
||||
timings: list[float] = []
|
||||
for _ in range(runs):
|
||||
start = time.perf_counter()
|
||||
validate_proposal(proposal, baseline=baseline)
|
||||
timings.append(time.perf_counter() - start)
|
||||
return statistics.median(timings), len(timings)
|
||||
39
spikes/e_magentic_resume.py
Normal file
39
spikes/e_magentic_resume.py
Normal file
|
|
@ -0,0 +1,39 @@
|
|||
"""S4's second half: resume a pending Magentic plan review in a **fresh interpreter**.
|
||||
|
||||
A resume proved inside the parent process would prove nothing about U12. The whole point of the
|
||||
asynchronous HITL time-scale is that the expert answers days later, in a process that never saw
|
||||
the run — so the only thing allowed to cross the boundary is the checkpoint on disk. Running the
|
||||
resume as its own module, launched by ``subprocess``, is what makes that boundary real
|
||||
(``tests/test_hosting_loadbearing.py`` set this precedent for the hosted shim).
|
||||
|
||||
Usage (the test supplies the arguments; nothing here is wired into ``src/``)::
|
||||
|
||||
python -m spikes.e_magentic_resume <storage_dir> <request_id> <checkpoint_id>
|
||||
|
||||
Writes ONE line of JSON to stdout: the resumed run's manager calls, remaining requests and
|
||||
outputs. Anything else on stdout would make the parent's parse the fragile part of the spike.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import sys
|
||||
|
||||
from spikes.e_magentic import resume_from_checkpoint
|
||||
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
if len(argv) != 4:
|
||||
print(f"usage: {argv[0]} <storage_dir> <request_id> <checkpoint_id>", file=sys.stderr)
|
||||
return 2
|
||||
storage_dir, request_id, checkpoint_id = argv[1], argv[2], argv[3]
|
||||
result = asyncio.run(
|
||||
resume_from_checkpoint(storage_dir, request_id=request_id, checkpoint_id=checkpoint_id)
|
||||
)
|
||||
print(json.dumps(result, sort_keys=True))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main(sys.argv))
|
||||
|
|
@ -107,7 +107,11 @@ class Contracts(BaseModel):
|
|||
data_source: DataSourceContract
|
||||
model_map: ModelMapContract
|
||||
termination: TerminationContract
|
||||
feedback: FeedbackContract
|
||||
#: ``None`` when the run carries no expert verdict at all (F2, non-goal 3). A run nobody
|
||||
#: reviewed has no feedback to validate; a run that DOES claim one is validated exactly as
|
||||
#: before, so a half-given verdict still fails fast here rather than being completed for the
|
||||
#: expert further down.
|
||||
feedback: FeedbackContract | None
|
||||
|
||||
|
||||
def _bundled_model_map() -> dict[str, Any]:
|
||||
|
|
@ -119,17 +123,19 @@ def _bundled_model_map() -> dict[str, Any]:
|
|||
def load_contracts(
|
||||
data_source: dict[str, Any],
|
||||
termination: dict[str, Any],
|
||||
feedback: dict[str, Any],
|
||||
feedback: dict[str, Any] | None,
|
||||
*,
|
||||
model_map: dict[str, Any] | None = None,
|
||||
) -> Contracts:
|
||||
"""Validate ALL contracts at startup (fail-fast, before any chat-client is built). Raises
|
||||
``pydantic.ValidationError`` on the first malformed contract. ``model_map`` defaults to the
|
||||
bundled ``data/model_map.json`` (the same file Step 8 ships)."""
|
||||
bundled ``data/model_map.json`` (the same file Step 8 ships). ``feedback`` is ``None`` for a
|
||||
run nobody reviewed — the ONE place the expert-verdict shape is validated, so a caller that
|
||||
supplies half a verdict is refused here by field name (F2)."""
|
||||
raw_map = _bundled_model_map() if model_map is None else model_map
|
||||
return Contracts(
|
||||
data_source=DataSourceContract(**data_source),
|
||||
model_map=ModelMapContract(**raw_map),
|
||||
termination=TerminationContract(**termination),
|
||||
feedback=FeedbackContract(**feedback),
|
||||
feedback=None if feedback is None else FeedbackContract(**feedback),
|
||||
)
|
||||
|
|
|
|||
1667
src/portfolio_optimiser/explore.py
Normal file
1667
src/portfolio_optimiser/explore.py
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -22,11 +22,12 @@ Two entry points, because the LLM call is async while ``validator.self_repair``
|
|||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections.abc import Callable
|
||||
from collections.abc import Callable, Mapping
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
from agent_framework import BaseChatClient, Message
|
||||
from pydantic import ValidationError
|
||||
from pydantic import BaseModel, ValidationError
|
||||
|
||||
from portfolio_optimiser.budget import TokenMeter
|
||||
from portfolio_optimiser.ir import CostBaseline, SavingsProposal
|
||||
|
|
@ -44,6 +45,212 @@ class GenerationError(RuntimeError):
|
|||
"""No parseable proposal could be produced within the attempt budget."""
|
||||
|
||||
|
||||
class StructuredOutputUnsupported(TypeError):
|
||||
"""A schema node cannot be expressed in the provider's strict structured-output subset.
|
||||
|
||||
Fail-closed, and deliberately so (mirrors ``write_concept_file`` / ``promote_verdict``:
|
||||
validation, never repair). The alternative — silently dropping what cannot be expressed — would
|
||||
stop commissioning a field without saying so, and the field it would have dropped first is
|
||||
``assumptions``, whose absence makes the Monte Carlo falsifier inert while it still reports
|
||||
percentiles. A schema this module cannot express is a decision for a human, not a default.
|
||||
"""
|
||||
|
||||
|
||||
#: Type-specific JSON Schema keywords the provider's structured-output subset does NOT support,
|
||||
#: transcribed from Azure's published table (Structured outputs -> "Unsupported type-specific
|
||||
#: keywords", https://learn.microsoft.com/azure/foundry/openai/how-to/structured-outputs), which
|
||||
#: states it is the same subset OpenAI accepts.
|
||||
#:
|
||||
#: ``exclusiveMinimum``/``exclusiveMaximum`` are NOT literally in that table — it names
|
||||
#: ``minimum maximum multipleOf`` — but they are the same family, and pydantic emits them for
|
||||
#: ``Field(gt=...)``/``Field(lt=...)``, which is exactly how this repo's IR spells its bounds. Being
|
||||
#: stricter than the table costs nothing here: every constraint stripped is re-applied by pydantic in
|
||||
#: ``_parse_ir`` and by ``validate_proposal``. The schema's job is SHAPE; the validator's job is
|
||||
#: VALUES. ``default`` is stripped for a different reason — strict mode requires every property to be
|
||||
#: required, so a default can never apply.
|
||||
UNSUPPORTED_SCHEMA_KEYWORDS = frozenset(
|
||||
{
|
||||
# String
|
||||
"minLength",
|
||||
"maxLength",
|
||||
"pattern",
|
||||
"format",
|
||||
# Number
|
||||
"minimum",
|
||||
"maximum",
|
||||
"multipleOf",
|
||||
"exclusiveMinimum",
|
||||
"exclusiveMaximum",
|
||||
# Objects
|
||||
"patternProperties",
|
||||
"unevaluatedProperties",
|
||||
"propertyNames",
|
||||
"minProperties",
|
||||
"maxProperties",
|
||||
# Arrays
|
||||
"unevaluatedItems",
|
||||
"contains",
|
||||
"minContains",
|
||||
"maxContains",
|
||||
"minItems",
|
||||
"maxItems",
|
||||
"uniqueItems",
|
||||
# Meaningless once every property is required
|
||||
"default",
|
||||
}
|
||||
)
|
||||
|
||||
#: The strict-legal stand-in for ``SavingsProposal.assumptions``.
|
||||
#:
|
||||
#: The IR spells the uncertainty bands as ``dict[str, tuple[float, float]]`` — a free-form map whose
|
||||
#: values are tuples. Neither half is expressible: strict mode requires ``additionalProperties:
|
||||
#: false`` in every object (so a map with arbitrary keys cannot be described), and tuples arrive as
|
||||
#: ``prefixItems``, which is outside the supported type list. Dropping the field instead would be
|
||||
#: silent damage: ``validator._monte_carlo`` falls back to the item's stated ``unit_cost`` for every
|
||||
#: code with no band, so with no bands at all the samples are identical and P10 == P50 == P90 — the
|
||||
#: stochastic falsifier goes inert while still reporting percentiles.
|
||||
#:
|
||||
#: So the WIRE carries an array of named entries and ``_parse_ir`` folds it back into the IR's map.
|
||||
#: The IR itself is untouched; the entry names spell out what the tuple positions mean, which the
|
||||
#: model would otherwise have to guess.
|
||||
_ASSUMPTIONS_WIRE_NODE: dict[str, Any] = {
|
||||
"type": "array",
|
||||
"description": (
|
||||
"Uncertainty band per affected cost line: the low and high unit cost the true price is "
|
||||
"expected to fall between. The band MUST enclose that item's own unit_cost. Omit an entry "
|
||||
"for a line whose unit cost is certain; an empty list means no uncertainty is claimed."
|
||||
),
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"code": {"type": "string"},
|
||||
"low_unit_cost": {"type": "number"},
|
||||
"high_unit_cost": {"type": "number"},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
#: Dotted paths (from the root model's own properties) whose node is replaced before sanitising.
|
||||
_PROPOSAL_SCHEMA_OVERRIDES: Mapping[str, dict[str, Any]] = {"assumptions": _ASSUMPTIONS_WIRE_NODE}
|
||||
|
||||
|
||||
def _sanitise_schema_node(node: Any, *, path: str, overrides: Mapping[str, dict[str, Any]]) -> Any:
|
||||
"""Rewrite one JSON Schema node into the strict subset, or raise ``StructuredOutputUnsupported``.
|
||||
|
||||
An override is applied FIRST, so a declared replacement is what gets checked and emitted — that
|
||||
is how the one inexpressible node in this repo's IR (``assumptions``) is expressed rather than
|
||||
excused. The replacement is then sanitised by the same code as everything else, so an override
|
||||
cannot smuggle in an illegal node.
|
||||
"""
|
||||
if not isinstance(node, Mapping):
|
||||
return node
|
||||
if path in overrides:
|
||||
node = overrides[path]
|
||||
|
||||
if "prefixItems" in node:
|
||||
raise StructuredOutputUnsupported(
|
||||
f"{path or '<root>'}: tuple types (prefixItems) are outside the strict subset"
|
||||
)
|
||||
for combinator in ("oneOf", "allOf"):
|
||||
if combinator in node:
|
||||
raise StructuredOutputUnsupported(
|
||||
f"{path or '<root>'}: {combinator} is outside the strict subset (anyOf is the "
|
||||
"only supported combinator)"
|
||||
)
|
||||
if isinstance(node.get("additionalProperties"), Mapping):
|
||||
raise StructuredOutputUnsupported(
|
||||
f"{path or '<root>'}: a free-form map cannot be expressed — strict mode requires "
|
||||
"additionalProperties: false in every object. Declare an override that spells the "
|
||||
"entries out as an array."
|
||||
)
|
||||
|
||||
out: dict[str, Any] = {}
|
||||
for key, value in node.items():
|
||||
if key in UNSUPPORTED_SCHEMA_KEYWORDS:
|
||||
continue
|
||||
if key == "properties" and isinstance(value, Mapping):
|
||||
out[key] = {
|
||||
name: _sanitise_schema_node(
|
||||
sub, path=f"{path}.{name}" if path else name, overrides=overrides
|
||||
)
|
||||
for name, sub in value.items()
|
||||
}
|
||||
elif key == "$defs" and isinstance(value, Mapping):
|
||||
out[key] = {
|
||||
name: _sanitise_schema_node(sub, path=f"$defs.{name}", overrides=overrides)
|
||||
for name, sub in value.items()
|
||||
}
|
||||
elif key == "items":
|
||||
out[key] = _sanitise_schema_node(value, path=f"{path}[]", overrides=overrides)
|
||||
elif key == "anyOf" and isinstance(value, list):
|
||||
out[key] = [_sanitise_schema_node(sub, path=path, overrides=overrides) for sub in value]
|
||||
else:
|
||||
out[key] = value
|
||||
|
||||
if "properties" in out:
|
||||
# Strict mode's two structural demands, applied to EVERY object rather than the root only:
|
||||
# no undeclared keys, and every declared key required.
|
||||
out["additionalProperties"] = False
|
||||
out["required"] = sorted(out["properties"])
|
||||
return out
|
||||
|
||||
|
||||
def strict_json_schema(
|
||||
model: type[BaseModel], *, overrides: Mapping[str, dict[str, Any]] | None = None
|
||||
) -> dict[str, Any]:
|
||||
"""Derive a strict-structured-output schema from ``model``'s own pydantic schema.
|
||||
|
||||
DERIVED rather than hand-written on purpose: a hand-written copy of a shape that already exists
|
||||
in ``ir.py`` is the second copy that drifts (kø-(p)), and it drifts silently — the model would
|
||||
keep being commissioned for the old shape. ``$defs``/``$ref`` are kept (the published subset
|
||||
supports definitions), so nested models need no inlining.
|
||||
"""
|
||||
schema = _sanitise_schema_node(model.model_json_schema(), path="", overrides=overrides or {})
|
||||
assert isinstance(schema, dict) # a model's root schema is always an object
|
||||
return schema
|
||||
|
||||
|
||||
def proposal_response_format() -> dict[str, Any]:
|
||||
"""The ``response_format`` mapping commissioning a ``SavingsProposal`` from the proposer.
|
||||
|
||||
A MAPPING, not the ``type[BaseModel]`` the option also accepts, and the reason is measured: given
|
||||
a class, the client converts it with ``type_to_response_format_param``, which emits ``minimum`` /
|
||||
``exclusiveMinimum`` / ``minItems`` / ``prefixItems`` and an ``assumptions`` node whose
|
||||
``additionalProperties`` is a schema — four things the published subset rules out. Our own
|
||||
mapping is the only way to control what reaches the wire.
|
||||
|
||||
ONE mapping serves both wired profiles (measured against agent-framework-openai 1.8.2 /
|
||||
agent-framework-foundry 1.8.2): the Chat Completions client passes it through verbatim, and the
|
||||
Responses client — which ``FoundryChatClient`` delegates to — converts this exact envelope into
|
||||
``text.format``.
|
||||
"""
|
||||
return {
|
||||
"type": "json_schema",
|
||||
"json_schema": {
|
||||
"name": SavingsProposal.__name__,
|
||||
"strict": True,
|
||||
"schema": strict_json_schema(SavingsProposal, overrides=_PROPOSAL_SCHEMA_OVERRIDES),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ParseFailure:
|
||||
"""One model reply that did NOT parse into the typed IR, kept VERBATIM (Fase 1b, funn 1).
|
||||
|
||||
``text`` is the reply exactly as the model produced it — never truncated, stripped or
|
||||
summarised. It is the thing the run PAID for and the only evidence of *why* the reply did not
|
||||
parse; a paraphrase would make the next paid run a guess again, which is the defect this type
|
||||
exists to close. ``error`` names the parse error itself (``json.JSONDecodeError`` vs a pydantic
|
||||
``ValidationError`` are very different diagnoses: leaked prose vs a wrong-shaped object).
|
||||
|
||||
Collected into a CALLER-OWNED sink rather than returned — see ``generate_via_llm``.
|
||||
"""
|
||||
|
||||
text: str
|
||||
error: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class GenerationResult:
|
||||
"""What one ``generate_via_llm`` call produced: the outcome, and the falsification history that
|
||||
|
|
@ -119,6 +326,37 @@ def _build_messages(
|
|||
return [Message(role="user", contents=[prompt])]
|
||||
|
||||
|
||||
def _normalise_assumptions(data: dict[str, Any]) -> None:
|
||||
"""Fold the WIRE's array-of-entries assumption bands back into the IR's ``code -> (low, high)``
|
||||
map, in place.
|
||||
|
||||
ADDITIVE, never a replacement: a reply that already uses the IR's map form (every scripted reply
|
||||
in the suite, and any model that answers without honouring the schema) is left untouched. A
|
||||
malformed entry is raised as ``ValueError`` rather than ``KeyError`` on purpose — ``ValueError``
|
||||
is what ``_fetch_parsed`` catches, so a bad band is captured as the parse failure it is instead
|
||||
of escaping the loop and killing the run.
|
||||
"""
|
||||
entries = data.get("assumptions")
|
||||
if not isinstance(entries, list):
|
||||
return
|
||||
bands: dict[str, tuple[Any, Any]] = {}
|
||||
for entry in entries:
|
||||
if (
|
||||
not isinstance(entry, Mapping)
|
||||
or not {
|
||||
"code",
|
||||
"low_unit_cost",
|
||||
"high_unit_cost",
|
||||
}
|
||||
<= entry.keys()
|
||||
):
|
||||
raise ValueError(
|
||||
f"each assumption entry needs code, low_unit_cost and high_unit_cost; got {entry!r}"
|
||||
)
|
||||
bands[entry["code"]] = (entry["low_unit_cost"], entry["high_unit_cost"])
|
||||
data["assumptions"] = bands
|
||||
|
||||
|
||||
def _parse_ir(text: str, project: Project) -> SavingsProposal:
|
||||
"""Parse the model's structured reply into the typed IR. Raises on malformed/text-leaked
|
||||
output (JSON error or Pydantic ``ValidationError``)."""
|
||||
|
|
@ -126,6 +364,7 @@ def _parse_ir(text: str, project: Project) -> SavingsProposal:
|
|||
if not isinstance(data, dict):
|
||||
raise ValueError("reply is not a JSON object")
|
||||
data.setdefault("project_id", project.id)
|
||||
_normalise_assumptions(data)
|
||||
return SavingsProposal(**data)
|
||||
|
||||
|
||||
|
|
@ -163,6 +402,7 @@ async def generate_via_llm(
|
|||
max_attempts: int = 3,
|
||||
baseline: CostBaseline | None = None,
|
||||
approach: Approach | None = None,
|
||||
parse_failures: list[ParseFailure] | None = None,
|
||||
) -> GenerationResult:
|
||||
"""Async LLM path: non-streaming chat -> parse -> validate, with TWO bounded retry kinds,
|
||||
the meter checked in this loop:
|
||||
|
|
@ -189,6 +429,19 @@ async def generate_via_llm(
|
|||
falsified per ATTEMPT like any other rejection — and its reason feeds the next attempt's prompt
|
||||
through the SAME informed-refinement path (Step 5), which is why no new loop appears here.
|
||||
|
||||
``parse_failures`` (Fase 1b, funn 1) is a CALLER-OWNED sink: every reply that fails to parse is
|
||||
appended to it VERBATIM, at the moment it fails. It is an out-parameter and not part of the
|
||||
return value ON PURPOSE, and the reason is measured rather than stylistic. ``meter.tick_round``
|
||||
raises ``BudgetExceeded`` inside the inner fetch loop, so on the path this capture exists for —
|
||||
a model whose replies never parse, which burns the round ledger — this function raises and
|
||||
returns NOTHING. That is exactly the live Fase-1b failure. A field on ``GenerationResult`` (the
|
||||
Step-5 ``refinements`` shape) would be blind to it, as would any artefact written by the caller
|
||||
*after* a successful return. The sink mirrors ``meter`` instead: a caller-owned accumulator this
|
||||
loop mutates, whose contents the caller still holds however the loop ended. Step 5's "a returned
|
||||
value cannot be silently lost by a caller that forgets to pass a collector" governs a value that
|
||||
REACHES the caller; here it does not, so the rule is cited and departed from deliberately. That
|
||||
a caller can forget is answered by a test on the wiring, not by a shape that cannot work.
|
||||
|
||||
Returns a ``GenerationResult``: the ``ValidatedProposal | Rejection`` outcome plus every
|
||||
rejection that was fed back into a later attempt's prompt. Surfacing that history changes
|
||||
nothing about the loop's BOUND — ``max_attempts`` and ``meter.tick_round`` are exactly as
|
||||
|
|
@ -200,11 +453,24 @@ async def generate_via_llm(
|
|||
# Parse-robust: a malformed/text-leaked reply is retried; the meter caps total work.
|
||||
while True:
|
||||
meter.tick_round() # between-attempt bound (BudgetExceeded over cap)
|
||||
reply = await chat_client.get_response(messages) # non-streaming
|
||||
# Fase 1b, funn 1b: hand the model a GRAMMAR, not a prose request. The prompt's
|
||||
# "Respond with ONLY a JSON object" line stays — a provider that ignores
|
||||
# ``response_format`` (or a local model that does not implement it) must still be told
|
||||
# what is wanted, and the parse-retry below remains the backstop either way.
|
||||
reply = await chat_client.get_response( # non-streaming
|
||||
messages, options={"response_format": proposal_response_format()}
|
||||
)
|
||||
_charge_usage(meter, reply)
|
||||
try:
|
||||
return _parse_ir(reply.text, project)
|
||||
except (ValidationError, ValueError, TypeError):
|
||||
except (ValidationError, ValueError, TypeError) as exc:
|
||||
# Capture BEFORE the retry: this reply was paid for, and once ``continue`` runs the
|
||||
# only record of what the model actually said is gone (Fase 1b, funn 1). Verbatim —
|
||||
# the operator is diagnosing a format failure, so any shortening removes evidence.
|
||||
if parse_failures is not None:
|
||||
parse_failures.append(
|
||||
ParseFailure(text=reply.text, error=f"{type(exc).__name__}: {exc}")
|
||||
)
|
||||
continue
|
||||
|
||||
last: Rejection | None = None
|
||||
|
|
|
|||
|
|
@ -165,6 +165,138 @@ def pending(outbox_dir: str, verdict_dir: str) -> list[PendingProposal]:
|
|||
return sorted(unjudged, key=lambda p: (p.run_id, p.approach_id, p.verdict_id))
|
||||
|
||||
|
||||
# --- U12: the pending PLAN REVIEWS of parked explorations, and the expert's answer ---------------
|
||||
# The same registry shape as ``pending`` above, one time-scale earlier: there the outbox holds a
|
||||
# PROPOSAL awaiting a verdict, here it holds a QUESTION awaiting a decision. Both live in this
|
||||
# MAF-free module because both are read by an operator tool that must not drag the framework in.
|
||||
|
||||
#: The closed answer vocabulary, identical to the terminal door's (``explore.terminal_plan_reviewer``).
|
||||
#: Two words, matched structurally — a file cannot be re-asked, so anything else is a refusal.
|
||||
_PLAN_REVIEW_ANSWERS = frozenset({"approve", "revise"})
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PendingPlanReview:
|
||||
"""One parked exploration still waiting on a human. ``plan`` is carried because a registry that
|
||||
only counted questions could not be used to answer one."""
|
||||
|
||||
run_id: str
|
||||
request_id: str
|
||||
index: int
|
||||
plan: str
|
||||
is_stalled: bool
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PlanReviewAnswer:
|
||||
"""The expert's decision, as read off a file. ``feedback`` is empty exactly when approving —
|
||||
the same encoding ``explore.PlanReviewDecision`` uses, kept plain so this module stays
|
||||
MAF-free and the adapter between them lives at ONE call site."""
|
||||
|
||||
run_id: str
|
||||
request_id: str
|
||||
decision: str
|
||||
feedback: str
|
||||
|
||||
|
||||
class PlanReviewAnswerError(ValueError):
|
||||
"""An answer file that cannot be read as a decision.
|
||||
|
||||
Fail-closed, and deliberately NOT the tolerant rule the verdict inbox uses. A dropped verdict
|
||||
that will not parse is one opinion missing from a fold; an unreadable plan-review answer is the
|
||||
one thing standing between a suspended run and a plan nobody signed. A ``ValueError`` so the
|
||||
CLI's existing structured-refusal arm surfaces it as ``rc 1`` rather than a traceback."""
|
||||
|
||||
|
||||
def _answer_path(review_dir: str, run_id: str) -> Path:
|
||||
return Path(review_dir) / f"{run_id}-plan-review-answer.json"
|
||||
|
||||
|
||||
def read_plan_review_question(outbox_dir: str, run_id: str) -> dict[str, Any] | None:
|
||||
"""The open question of ``run_id``, or ``None`` when there is none. Tolerant: an outbox with no
|
||||
such file simply has no parked review."""
|
||||
return _load_json_dict(Path(outbox_dir) / f"{run_id}-plan-review.json")
|
||||
|
||||
|
||||
def load_plan_review_answer(review_dir: str, run_id: str, *, request_id: str) -> PlanReviewAnswer:
|
||||
"""Read the expert's answer to ONE named review, fail-closed at every step.
|
||||
|
||||
``request_id`` is a required argument rather than something read off the file and trusted: two
|
||||
reviews of one run share a file name, so an answer left over from the previous round would
|
||||
otherwise be applied to a plan the expert never saw. A mismatch is REFUSED by name — it is a
|
||||
stale answer, not an absent one, and the two need different words.
|
||||
|
||||
A missing file raises rather than returning ``None``: "not answered yet" is the normal state of
|
||||
this door, and the caller asking to resume has already said it believes otherwise."""
|
||||
path = _answer_path(review_dir, run_id)
|
||||
data = _load_json_dict(path)
|
||||
if data is None:
|
||||
raise PlanReviewAnswerError(
|
||||
f"no answer for plan review {request_id} of run {run_id!r} in {review_dir!r} "
|
||||
f"(expected {path.name}): the review is still waiting on a human"
|
||||
)
|
||||
found = str(data.get("request_id", ""))
|
||||
if found != request_id:
|
||||
raise PlanReviewAnswerError(
|
||||
f"the answer in {path.name} answers plan review {found!r}, but the open review of run "
|
||||
f"{run_id!r} is {request_id!r}. Refused: an answer to another question is not an "
|
||||
f"answer to this one"
|
||||
)
|
||||
decision = str(data.get("decision", ""))
|
||||
if decision not in _PLAN_REVIEW_ANSWERS:
|
||||
raise PlanReviewAnswerError(
|
||||
f"{path.name} answers {decision!r}, which is outside the vocabulary "
|
||||
f"{sorted(_PLAN_REVIEW_ANSWERS)}. Refused, never read as a sign-off"
|
||||
)
|
||||
feedback = str(data.get("feedback", ""))
|
||||
if decision == "revise" and not feedback.strip():
|
||||
raise PlanReviewAnswerError(
|
||||
f"{path.name} answers 'revise' with nothing to revise: the manager would be asked to "
|
||||
f"replan against an empty instruction. Say what to change, or answer 'approve'"
|
||||
)
|
||||
return PlanReviewAnswer(
|
||||
run_id=run_id, request_id=request_id, decision=decision, feedback=feedback
|
||||
)
|
||||
|
||||
|
||||
def pending_plan_reviews(outbox_dir: str, review_dir: str) -> list[PendingPlanReview]:
|
||||
"""Every parked plan review whose OWN answer has not landed, sorted by ``run_id``.
|
||||
|
||||
Mirrors ``pending``: an outbox artefact joined against an inbox, with the join on the key each
|
||||
side names. The key here is ``request_id`` — an answer to a different review leaves this one
|
||||
pending rather than quietly clearing it, which is the same fail-closed rule the resume path
|
||||
applies and for the same reason.
|
||||
|
||||
Tolerant on the READ side (an unreadable file in either folder is not a question and not an
|
||||
answer), fail-closed on the DECIDE side (``load_plan_review_answer``). The registry says who is
|
||||
waiting; it never decides what they said."""
|
||||
waiting: list[PendingPlanReview] = []
|
||||
directory = Path(outbox_dir)
|
||||
if not directory.is_dir():
|
||||
return waiting
|
||||
for file in sorted(directory.glob("*-plan-review.json")):
|
||||
data = _load_json_dict(file)
|
||||
if data is None:
|
||||
continue
|
||||
run_id = str(data.get("run_id", ""))
|
||||
request_id = str(data.get("request_id", ""))
|
||||
if not run_id or not request_id:
|
||||
continue
|
||||
answer = _load_json_dict(_answer_path(review_dir, run_id))
|
||||
if answer is not None and str(answer.get("request_id", "")) == request_id:
|
||||
continue
|
||||
waiting.append(
|
||||
PendingPlanReview(
|
||||
run_id=run_id,
|
||||
request_id=request_id,
|
||||
index=int(data.get("index", 0)),
|
||||
plan=str(data.get("plan", "")),
|
||||
is_stalled=bool(data.get("is_stalled", False)),
|
||||
)
|
||||
)
|
||||
return sorted(waiting, key=lambda p: (p.run_id, p.index))
|
||||
|
||||
|
||||
# --- Routing config: self-contained dimension→expert table (fail-fast) ----------------------------
|
||||
# A minimal MVP stand-in for the S3.5 dimension catalog (kept DISTINCT — see the plan's Non-Goals).
|
||||
# Field names mirror ``dimension.Dimension`` so the two reconcile cleanly when S3.5 lands. No ``label``
|
||||
|
|
|
|||
|
|
@ -38,7 +38,22 @@ unknown field is a 400 naming the field — the permissive-schema trap (valg-doc
|
|||
to our own surface. Error mapping is honest: ``ValueError`` (pydantic contract violations
|
||||
subclass it) → 400; any other failure → 500 ``{error_type, error}`` (mirrors
|
||||
``RunFailure``); a ``Rejection`` is a SUCCESSFUL run → 200 with ``outcome_type:
|
||||
"rejected"`` — the negative outcome belongs to the payload, never to the transport. The
|
||||
"rejected"`` — the negative outcome belongs to the payload, never to the transport.
|
||||
|
||||
``BudgetExceeded`` gets its OWN arm → 429, for the same reason ``BudgetStop`` is kept out of
|
||||
``stop_reason`` (S3.4): a cap that fires is the feature working (``Budget`` exists so a run can
|
||||
never hang unbounded), and answering it on the crash channel makes "it did not work"
|
||||
unreadable — the first live run died exactly here and the surface said 500, the same thing it
|
||||
says when the endpoint falls over. It is NOT 200 either: unlike a ``Rejection``, which is a run
|
||||
that CONCLUDED, an exhausted budget produced no proposal, and a 2xx would let an automated
|
||||
caller record "analysed" for a run that analysed nothing. 429 because the condition arises from
|
||||
an ALLOWANCE — ``max_rounds``/``max_tokens`` are whitelisted request fields and raising them is
|
||||
the caller's own remedy — never from a server fault. The ``kind``/``limit``/``observed`` triple
|
||||
is carried as STRUCTURE, not flattened into ``str(exc)`` (kø-(y): it describes one ledger and
|
||||
answering "which cap bound, and by how much" is the operational question), and ``error_type``
|
||||
is deliberately absent — that key belongs to the failure channel. Honesty limit, stated: no
|
||||
``Retry-After``. Retrying an unchanged body hits the same cap; the remedy is a larger allowance
|
||||
or accepting the stop, and a header promising time would be a lie. The
|
||||
platform's injected headers (``x-agent-user-id``/``x-agent-foundry-call-id``) are absent
|
||||
locally by contract and unused here; forwarding the call-id on outgoing Foundry calls has
|
||||
no seam in ``backends.py`` today and is deliberately not built (90 %-prinsippet).
|
||||
|
|
@ -50,17 +65,46 @@ import asyncio
|
|||
import json
|
||||
import os
|
||||
import signal
|
||||
import sys
|
||||
from collections.abc import Mapping
|
||||
from typing import Any
|
||||
|
||||
from portfolio_optimiser.budget import BudgetExceeded
|
||||
from portfolio_optimiser.explore import ExplorationContract, explore
|
||||
from portfolio_optimiser.outbox import outcome_payload
|
||||
from portfolio_optimiser.run import RunResult, run_project
|
||||
from portfolio_optimiser.tracing import configure_tracing, tracing_notice
|
||||
|
||||
DEFAULT_PORT = 8088
|
||||
_HOSTED_DEFAULT_PROFILE = "azure"
|
||||
_REQUIRED_FIELDS = ("project_id", "docs_dir", "verdict_input")
|
||||
_OPTIONAL_FIELDS = ("bundle_dir", "profile", "max_rounds", "max_tokens", "top_k")
|
||||
_ALLOWED_FIELDS = frozenset(_REQUIRED_FIELDS + _OPTIONAL_FIELDS)
|
||||
_REASONS = {200: "OK", 400: "Bad Request", 404: "Not Found", 500: "Internal Server Error"}
|
||||
_REQUIRED_FIELDS = ("project_id", "docs_dir")
|
||||
#: ``verdict_input`` is OPTIONAL since F2 (non-goal 3). It used to be required, which forced an
|
||||
#: external caller to invent an expert verdict just to get a run at all — a field that could not be
|
||||
#: filled honestly, on the surface handed over 2026-08-14. The move is a pure WIDENING: a caller
|
||||
#: that still sends it is unaffected, and one that omits it now gets a run whose verdict is
|
||||
#: honestly absent.
|
||||
_OPTIONAL_FIELDS = (
|
||||
"bundle_dir",
|
||||
"profile",
|
||||
"max_rounds",
|
||||
"max_tokens",
|
||||
"top_k",
|
||||
"verdict_input",
|
||||
)
|
||||
#: Fields this surface CONSUMES rather than forwards (U4). They are not ``run_project``
|
||||
#: parameters — the exploration runs first and hands ``run_project`` a ``Mandate`` — so passing one
|
||||
#: through would be a ``TypeError`` answered as a 500. The whitelist is therefore a THREE-way
|
||||
#: partition, and the Fase 4e proof gained a negative half to match: every forwarded field must be
|
||||
#: a real ``run_project`` parameter, and every consumed field must not be.
|
||||
_CONSUMED_FIELDS = ("explore_prompt", "explore_contract")
|
||||
_ALLOWED_FIELDS = frozenset(_REQUIRED_FIELDS + _OPTIONAL_FIELDS + _CONSUMED_FIELDS)
|
||||
_REASONS = {
|
||||
200: "OK",
|
||||
400: "Bad Request",
|
||||
404: "Not Found",
|
||||
429: "Too Many Requests",
|
||||
500: "Internal Server Error",
|
||||
}
|
||||
|
||||
|
||||
class InvocationRefused(ValueError):
|
||||
|
|
@ -75,10 +119,15 @@ def resolve_port() -> int:
|
|||
return int(os.environ.get("PORT") or DEFAULT_PORT)
|
||||
|
||||
|
||||
def _run_kwargs(payload: Any) -> tuple[str, dict[str, Any]]:
|
||||
def _run_kwargs(payload: Any) -> tuple[str, dict[str, Any], dict[str, Any]]:
|
||||
"""Whitelist the JSON payload onto ``run_project``'s signature. Everything not named in
|
||||
the whitelist — including server-side seams like ``outbox_dir``, ``client_factory`` or
|
||||
``verdict_dir`` — is refused by name, never silently dropped."""
|
||||
``verdict_dir`` — is refused by name, never silently dropped.
|
||||
|
||||
Returns ``(project_id, forwarded_kwargs, consumed)``. The consumed half is split out HERE
|
||||
rather than filtered at the call site so there is one place that decides which fields reach
|
||||
``run_project``: a consumed field left in ``kwargs`` is an argument the signature does not have,
|
||||
which the container answers as a 500 for what is really a wiring mistake."""
|
||||
if not isinstance(payload, dict):
|
||||
raise InvocationRefused("body must be a JSON object")
|
||||
unknown = sorted(set(payload) - _ALLOWED_FIELDS)
|
||||
|
|
@ -87,9 +136,60 @@ def _run_kwargs(payload: Any) -> tuple[str, dict[str, Any]]:
|
|||
missing = [field for field in _REQUIRED_FIELDS if field not in payload]
|
||||
if missing:
|
||||
raise InvocationRefused(f"missing required field(s): {', '.join(missing)}")
|
||||
kwargs: dict[str, Any] = {k: payload[k] for k in payload if k != "project_id"}
|
||||
consumed = {k: payload[k] for k in _CONSUMED_FIELDS if k in payload}
|
||||
kwargs: dict[str, Any] = {
|
||||
k: payload[k] for k in payload if k != "project_id" and k not in _CONSUMED_FIELDS
|
||||
}
|
||||
kwargs.setdefault("profile", _HOSTED_DEFAULT_PROFILE)
|
||||
return payload["project_id"], kwargs
|
||||
return payload["project_id"], kwargs, consumed
|
||||
|
||||
|
||||
async def _shaped_mandate(consumed: Mapping[str, Any], kwargs: Mapping[str, Any]) -> Any:
|
||||
"""Run the U4 exploration this invocation asked for and return the mandate it shaped.
|
||||
|
||||
Every refusal here is the CALLER's error and therefore a ``ValueError`` (the 400 arm), by name.
|
||||
That placement is deliberate rather than incidental: ``explore()`` refuses two of these itself,
|
||||
but ``ExplorationError`` is a ``RuntimeError``, so leaving them to the loop would answer a
|
||||
caller's configuration mistake on the crash channel — the same conflation ``BudgetExceeded``
|
||||
was given its own 429 to end.
|
||||
|
||||
``enable_plan_review`` is refused outright. The U13 door is SYNCHRONOUS: it blocks the loop on
|
||||
a human or persona, and an HTTP request has neither — the invocation would hang rather than
|
||||
answer. The library API is where that door opens."""
|
||||
prompt = consumed.get("explore_prompt")
|
||||
raw_contract = consumed.get("explore_contract")
|
||||
if prompt is None:
|
||||
raise InvocationRefused(
|
||||
"explore_contract without explore_prompt: the bounds describe an exploration that "
|
||||
"would never run"
|
||||
)
|
||||
if raw_contract is None:
|
||||
raise InvocationRefused(
|
||||
"explore_prompt without explore_contract: an exploration's bounds are never defaulted "
|
||||
"(an omitted cap falls back to an unbounded loop)"
|
||||
)
|
||||
if not kwargs.get("bundle_dir"):
|
||||
raise InvocationRefused(
|
||||
"explore_prompt without bundle_dir: the exploration navigates knowledge bases, and "
|
||||
"with none configured it would spend its budget reading nothing"
|
||||
)
|
||||
if not isinstance(raw_contract, dict):
|
||||
raise InvocationRefused("explore_contract must be a JSON object")
|
||||
contract = ExplorationContract(**raw_contract) # ValidationError subclasses ValueError -> 400
|
||||
if contract.enable_plan_review:
|
||||
raise InvocationRefused(
|
||||
"explore_contract sets enable_plan_review, but this surface has no reviewer to answer "
|
||||
"it: the synchronous plan review would block the request on nobody, and would block "
|
||||
"the event loop that answers /readiness while doing it. The operator door is the CLI's "
|
||||
"--plan-review (or explore(..., plan_reviewer=...) in-process)"
|
||||
)
|
||||
result = await explore(
|
||||
str(prompt),
|
||||
contract=contract,
|
||||
bundle_dirs=(kwargs["bundle_dir"],),
|
||||
profile=kwargs["profile"],
|
||||
)
|
||||
return result.mandate
|
||||
|
||||
|
||||
def _response_payload(result: RunResult) -> dict[str, Any]:
|
||||
|
|
@ -97,7 +197,10 @@ def _response_payload(result: RunResult) -> dict[str, Any]:
|
|||
**outcome_payload(
|
||||
result.outcome,
|
||||
checker_verdict=result.checker_verdict,
|
||||
verdict_id=result.verdict.id,
|
||||
# The candidate's KEY, not evidence that anybody decided (F2): identical to the
|
||||
# captured verdict's id whenever one was given, and still the id under which a later
|
||||
# expert verdict on this candidate will arrive when none was.
|
||||
verdict_id=result.verdict_key,
|
||||
),
|
||||
"proposal": result.outcome.proposal.model_dump(),
|
||||
"provenance": result.provenance.model_dump(),
|
||||
|
|
@ -108,13 +211,33 @@ def _response_payload(result: RunResult) -> dict[str, Any]:
|
|||
async def invoke(payload: Any) -> dict[str, Any]:
|
||||
"""One invocation: validate → ``run_project`` → outbox-shaped response payload.
|
||||
``run_project`` is resolved through this module's namespace at call time (the test
|
||||
seam). ``live_dry_run`` is not on the whitelist, so the union narrows to RunResult."""
|
||||
project_id, kwargs = _run_kwargs(payload)
|
||||
seam). ``live_dry_run`` is not on the whitelist, so the union narrows to RunResult.
|
||||
|
||||
With ``explore_prompt`` the exploration runs FIRST and its mandate is what the pipeline then
|
||||
evaluates — level 2 and 3 of the guarantee table are unchanged, and the exploration itself
|
||||
still writes nothing."""
|
||||
project_id, kwargs, consumed = _run_kwargs(payload)
|
||||
if consumed:
|
||||
kwargs["mandate"] = await _shaped_mandate(consumed, kwargs)
|
||||
result = await run_project(project_id, **kwargs)
|
||||
assert isinstance(result, RunResult)
|
||||
return _response_payload(result)
|
||||
|
||||
|
||||
def _budget_payload(exc: BudgetExceeded) -> dict[str, Any]:
|
||||
"""The exhausted-budget body: the ledger's own triple, plus the human line for the log.
|
||||
|
||||
The ``budget_exhausted`` key's PRESENCE is the discriminator — it is not folded into
|
||||
``outcome_type`` (whose values, ``validated``/``rejected``, mean "the run concluded and
|
||||
here is the verdict") for the same reason ``BudgetStop`` was given its own field instead of
|
||||
widening ``stop_reason``. Nor could it be: ``outcome_payload`` is the ONE copy of that fork
|
||||
and takes a ``ValidatedProposal | Rejection``, neither of which an exhausted run has."""
|
||||
return {
|
||||
"budget_exhausted": {"kind": exc.kind, "limit": exc.limit, "observed": exc.observed},
|
||||
"error": str(exc),
|
||||
}
|
||||
|
||||
|
||||
def _http_response(status: int, content_type: str, body: bytes) -> bytes:
|
||||
head = (
|
||||
f"HTTP/1.1 {status} {_REASONS[status]}\r\n"
|
||||
|
|
@ -171,6 +294,10 @@ async def _respond(method: str, path: str, body: bytes) -> bytes:
|
|||
return _json_response(400, {"error": "body is not valid JSON"})
|
||||
try:
|
||||
return _json_response(200, await invoke(payload))
|
||||
except BudgetExceeded as exc:
|
||||
# A cap that fired, not a failure — its own channel, and the triple kept as
|
||||
# structure rather than re-parsed out of the message by whoever reads this.
|
||||
return _json_response(429, _budget_payload(exc))
|
||||
except ValueError as exc:
|
||||
# The caller's error: InvocationRefused + run_project's fail-fast contract
|
||||
# violations (pydantic ValidationError subclasses ValueError).
|
||||
|
|
@ -217,5 +344,15 @@ async def _serve_until_sigterm() -> None:
|
|||
|
||||
def main() -> None:
|
||||
"""Serve the hosted-agent contract until SIGTERM (bind 0.0.0.0 — the platform
|
||||
terminates TLS in front of us), then exit 0."""
|
||||
terminates TLS in front of us), then exit 0.
|
||||
|
||||
U14: the tracing seam is installed before the loop starts and announced on stderr, which in a
|
||||
container IS the log. This is the entry where "an organisation must be able to see what the run
|
||||
did" is actually cashed — the demo is a scripted proof, not the product. A malformed
|
||||
``PORTFOLIO_OTEL`` propagates: a server whose telemetry cannot be configured as asked must not
|
||||
start and then look healthy on ``/readiness``."""
|
||||
setup = configure_tracing()
|
||||
notice = tracing_notice(setup)
|
||||
if notice is not None:
|
||||
print(notice, file=sys.stderr)
|
||||
asyncio.run(_serve_until_sigterm())
|
||||
|
|
|
|||
|
|
@ -29,6 +29,7 @@ Two refusals are load-bearing, both at construction time:
|
|||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Sequence
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Literal
|
||||
|
|
@ -50,6 +51,14 @@ class Approach(BaseModel):
|
|||
id: str = Field(min_length=1)
|
||||
label: str = Field(min_length=1)
|
||||
description: str = ""
|
||||
#: Which knowledge base this approach belongs to (§ C.7), by the base's id — the directory's
|
||||
#: BASENAME, exactly as ``explore._bundle_index`` names it. DEFAULTS to empty, meaning "no base
|
||||
#: named": a legitimate statement when the run has only one base to name, and what keeps every
|
||||
#: mandate written before multi-base existed valid and dispatchable unchanged.
|
||||
#:
|
||||
#: The field is a ROUTING key, never a claim about content. It says which pipeline the approach
|
||||
#: must be evaluated in, and ``route_by_bundle`` is the one place that reads it.
|
||||
bundle_id: str = ""
|
||||
|
||||
|
||||
class Mandate(BaseModel):
|
||||
|
|
@ -88,6 +97,85 @@ class Mandate(BaseModel):
|
|||
return self
|
||||
|
||||
|
||||
class MandateRoutingError(ValueError):
|
||||
"""A commission that cannot be executed against the bases it was given (§ C.7).
|
||||
|
||||
**A ``ValueError`` by construction, and that is a measurement rather than a taxonomy note.**
|
||||
økt 57 paid for the opposite: ``ExplorationError`` is a ``RuntimeError`` and therefore fell
|
||||
outside ``run.main``'s ``(ValueError, FileNotFoundError, ValidationError)`` refusal tuple and
|
||||
outside ``hosting``'s 400 arm, so a caller's configuration mistake would have left as a
|
||||
traceback on one surface and a 500 — the crash channel — on the other. A routing refusal is
|
||||
exactly that class of caller mistake, so it is born inside both nets instead of being
|
||||
retrofitted into them later.
|
||||
"""
|
||||
|
||||
|
||||
def route_by_bundle(mandate: Mandate, bundle_ids: Sequence[str]) -> tuple[tuple[str, Mandate], ...]:
|
||||
"""Partition one commission into one sub-mandate PER knowledge base (§ C.7).
|
||||
|
||||
This is the whole of "multi-base", and it is deliberately a partition rather than a widening.
|
||||
``run_project`` derives four single-valued things from THE bundle it is given — the project
|
||||
(``_project_from_bundle`` fail-fasts when the bundle's own ``project_id`` is not the requested
|
||||
one), the validator's stage-0 cost baseline, the agents' read context and the ExpeL query key —
|
||||
and returns ONE ``RunResult`` with ONE ``ProvenanceStamp``. A second ``bundle_dir`` on that
|
||||
signature would force a silent pick-one for all four. § C.7 says the same thing in its own
|
||||
words: *pipelinen kjøres per bundle som i dag* — N calls, not one call taking N.
|
||||
|
||||
Order is taken from ``bundle_ids``, never from first appearance among the approaches, so the
|
||||
dispatch's spend order is a property of how the run was configured rather than of how a model
|
||||
happened to sequence its hypotheses.
|
||||
|
||||
**Fail-fast on a commission that cannot be executed as written**, mirroring ``load_mandate``'s
|
||||
contract for exactly its reason: a run must never proceed on a *silently degraded* commission,
|
||||
because the coverage report would then describe work nobody ordered. Two ways that happens, and
|
||||
both refuse by name rather than resolving by position (the S3.2 key-collision class):
|
||||
|
||||
* an approach naming a base that was not configured;
|
||||
* an approach naming NO base while more than one is configured — with a single base there is no
|
||||
other value the field could take, so resolving it there is the only answer rather than a
|
||||
guess, and it is what keeps every pre-multi-base mandate dispatchable unchanged.
|
||||
|
||||
A base that no approach names is NOT run: a run costs money and the commission ordered nothing
|
||||
for it. The single-base case keeps its own rule above, so an own-proposals-only mandate still
|
||||
reaches the one base it could possibly mean.
|
||||
|
||||
:raises MandateRoutingError: no bases configured, or an approach that cannot be routed.
|
||||
"""
|
||||
if not bundle_ids:
|
||||
raise MandateRoutingError(
|
||||
"a mandate cannot be routed against zero knowledge bases: an empty plan reads as "
|
||||
"'there was nothing to do', which is indistinguishable from a commission that was "
|
||||
"fully evaluated against nothing"
|
||||
)
|
||||
|
||||
known = tuple(bundle_ids)
|
||||
sole = known[0] if len(known) == 1 else None
|
||||
grouped: dict[str, list[Approach]] = {bundle_id: [] for bundle_id in known}
|
||||
for approach in mandate.approaches:
|
||||
target = approach.bundle_id or sole
|
||||
if target is None:
|
||||
raise MandateRoutingError(
|
||||
f"approach {approach.id!r} names no knowledge base and {len(known)} are "
|
||||
f"configured ({', '.join(known)}); which one it belongs to is not something this "
|
||||
"layer may decide on the expert's behalf"
|
||||
)
|
||||
if target not in grouped:
|
||||
raise MandateRoutingError(
|
||||
f"approach {approach.id!r} names knowledge base {target!r}, which is not "
|
||||
f"configured for this run; configured: {', '.join(known)}"
|
||||
)
|
||||
grouped[target].append(approach)
|
||||
|
||||
return tuple(
|
||||
(
|
||||
bundle_id,
|
||||
mandate.model_copy(update={"approaches": tuple(grouped[bundle_id])}),
|
||||
)
|
||||
for bundle_id in known
|
||||
if grouped[bundle_id] or sole is not None
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ApproachOutcome:
|
||||
"""What became of ONE commissioned approach — one row of the run's coverage report.
|
||||
|
|
|
|||
|
|
@ -19,6 +19,12 @@ fields. A target that fails to resolve for ANY reason (missing file, invalid pat
|
|||
escape) is silently skipped, never raised. Path-safety reuses ``retrieval.safe_resolve`` (also pure
|
||||
stdlib): each cross-link is canonicalised and boundary-checked against the bundle dir, fail-closed
|
||||
— the SOLE in-/out-of-bundle test.
|
||||
|
||||
Skipped is not SILENT, though: every link the walk could not follow is recorded on
|
||||
``Bundle.skipped`` as a ``SkippedLink`` (which file it was written in, the link text verbatim, and
|
||||
which of the two reasons applied). The tolerance is unchanged — nothing raises — but a bundle whose
|
||||
other half was never reached is no longer indistinguishable from one where those documents were
|
||||
never written.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
|
@ -28,7 +34,7 @@ import posixpath
|
|||
import re
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from typing import Any, Literal
|
||||
|
||||
from portfolio_optimiser.ir import CostBaseline
|
||||
from portfolio_optimiser.retrieval import PathSecurityError, safe_resolve
|
||||
|
|
@ -94,12 +100,55 @@ class BundleFile:
|
|||
body: str
|
||||
|
||||
|
||||
#: Why navigation did not follow a cross-link. TWO values, because the two mean different things
|
||||
#: to whoever has to fix the bundle: ``outside-bundle`` is a target that resolves OUTSIDE the bundle
|
||||
#: root (frequently a deliberate link to a neighbouring base), ``missing`` is a target that resolves
|
||||
#: INSIDE it with no readable file there (almost always a typo in the link). Collapsing them into
|
||||
#: one "skipped" would answer neither question. De-duplication is NOT among them: a repeated link
|
||||
#: and a cycle are correct navigation, never a skip.
|
||||
SkipReason = Literal["outside-bundle", "missing"]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SkippedLink:
|
||||
"""One cross-link the walk did NOT follow, and why.
|
||||
|
||||
STRUCTURED rather than a rendered string, for the reason ``BudgetExceeded`` carries
|
||||
``kind``/``limit``/``observed`` as fields (kø-(y)): "which document is missing" and "why is it
|
||||
missing" are two separate operative questions, and a caller that has to re-parse prose to tell
|
||||
them apart has been handed a diagnostic it cannot act on.
|
||||
|
||||
``target`` is the link text VERBATIM as written in the source file, never the resolved path: the
|
||||
operator fixing the bundle edits that text, and a normalised form would send them looking for a
|
||||
string their file does not contain."""
|
||||
|
||||
#: Bundle-relative name of the file the link was written in.
|
||||
from_file: str
|
||||
#: The link target exactly as it appears in that file.
|
||||
target: str
|
||||
reason: SkipReason
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Bundle:
|
||||
"""A navigated OKF bundle: ``index.md`` plus every cross-linked file that resolves."""
|
||||
"""A navigated OKF bundle: ``index.md`` plus every cross-linked file that resolves — and, in
|
||||
``skipped``, every cross-link that did not.
|
||||
|
||||
``skipped`` DEFAULTS to the empty tuple, and the default is the honest reading rather than a
|
||||
convenience: an empty trace is a positive statement ("every cross-link was followed"), in the
|
||||
same class as ``ProvenanceStamp.external_calls`` ("nothing outside this process was contacted").
|
||||
That is the opposite of ``ProvenanceStamp.cost_baseline_anchored``, which is REQUIRED precisely
|
||||
because both of its defaults would lie. The difference is what each absent value would assert:
|
||||
a missing bool has to claim something about an event, while a missing trace asserts only that
|
||||
the event list is empty — which is exactly what a construction with no skips means."""
|
||||
|
||||
dir: str
|
||||
files: tuple[BundleFile, ...]
|
||||
#: Every link navigation could not follow, in walk order. Read by ``run`` to render the one line
|
||||
#: a run prints about its own reachability; NEVER read by ``bundle_context``, whose rendering is
|
||||
#: built from ``index_summary`` + ``context_files`` alone — which is what keeps the commons-owned
|
||||
#: nav-golden fasit byte-identical.
|
||||
skipped: tuple[SkippedLink, ...] = ()
|
||||
|
||||
@property
|
||||
def index_summary(self) -> str:
|
||||
|
|
@ -165,29 +214,54 @@ def _resolve_target(bundle_dir: str, from_name: str, target: str) -> tuple[str,
|
|||
return None
|
||||
|
||||
|
||||
def _walk(bundle_dir: str, current: BundleFile, files: list[BundleFile], seen: set[str]) -> None:
|
||||
def _walk(
|
||||
bundle_dir: str,
|
||||
current: BundleFile,
|
||||
files: list[BundleFile],
|
||||
seen: set[str],
|
||||
skipped: list[SkippedLink],
|
||||
) -> None:
|
||||
"""Follow ``current``'s cross-links depth-first in first-seen order, appending each newly
|
||||
reached file and recursing into it. De-duplication is on the CANONICAL RESOLVED path (so
|
||||
``./a.md``, ``a.md`` and ``/a.md`` are one entry), which is also what terminates cycles."""
|
||||
``./a.md``, ``a.md`` and ``/a.md`` are one entry), which is also what terminates cycles.
|
||||
|
||||
A link that cannot be followed is still SKIPPED, never raised (OKF §4) — the tolerance is the
|
||||
spec — but it is now RECORDED in ``skipped``, with the reason distinguishing the two cases.
|
||||
The dedup branch records NOTHING: a repeated link and a cycle are correct navigation, and an
|
||||
implementation that logged every ``continue`` would report a healthy bundle as half-unread.
|
||||
|
||||
A caller-owned accumulator rather than a return value, for the reason ``generate``'s
|
||||
parse-failure sink is one: the recursion is depth-first over an unbounded tree, so every frame
|
||||
appends into the SAME list and the walk's shape stays unchanged."""
|
||||
for target in _LINK_RE.findall(current.body):
|
||||
resolved = _resolve_target(bundle_dir, current.name, target)
|
||||
if resolved is None:
|
||||
# The target left the bundle. Often deliberate (a link to a neighbouring base), so it is
|
||||
# reported rather than refused — the tolerance is unchanged.
|
||||
skipped.append(
|
||||
SkippedLink(from_file=current.name, target=target, reason="outside-bundle")
|
||||
)
|
||||
continue
|
||||
rel, canonical = resolved
|
||||
if canonical in seen:
|
||||
continue
|
||||
continue # de-duplication / cycle termination: correct navigation, NOT a skip
|
||||
seen.add(canonical)
|
||||
linked = _load_file(bundle_dir, rel)
|
||||
if linked is None:
|
||||
continue # broken link: tolerated, never raised (OKF §4)
|
||||
# In-bundle, but nothing readable is there: broken link, tolerated, never raised (§4).
|
||||
# Recorded once per resolved target — the ``seen`` entry above absorbs repeats.
|
||||
skipped.append(SkippedLink(from_file=current.name, target=target, reason="missing"))
|
||||
continue
|
||||
files.append(linked)
|
||||
_walk(bundle_dir, linked, files, seen)
|
||||
_walk(bundle_dir, linked, files, seen, skipped)
|
||||
|
||||
|
||||
def navigate_bundle(bundle_dir: str) -> Bundle:
|
||||
"""Navigate the OKF bundle from ``index.md``: parse the root index, then follow intra-bundle
|
||||
``.md`` cross-links RECURSIVELY, depth-first in first-seen link order, reading each reached
|
||||
file's frontmatter + body. Fully deterministic. Broken / escaping links are skipped (§4).
|
||||
file's frontmatter + body. Fully deterministic. Broken / escaping links are skipped (§4) — and
|
||||
RECORDED on the returned ``Bundle.skipped``, so "this document was never written" and "the link
|
||||
to it was wrong" stop looking identical from the outside.
|
||||
|
||||
Navigation follows LINKS ONLY — a directory is never enumerated. Hence the missing-``index.md``
|
||||
error binds the bundle ROOT alone (a bundle has no entry point without it); an intermediate
|
||||
|
|
@ -197,10 +271,11 @@ def navigate_bundle(bundle_dir: str) -> Bundle:
|
|||
if index is None:
|
||||
raise ValueError(f"OKF bundle has no readable {_INDEX_NAME}: {bundle_dir!r}")
|
||||
files: list[BundleFile] = [index]
|
||||
skipped: list[SkippedLink] = []
|
||||
root = _resolve_target(bundle_dir, _INDEX_NAME, _INDEX_NAME)
|
||||
seen = {root[1]} if root is not None else set()
|
||||
_walk(bundle_dir, index, files, seen)
|
||||
return Bundle(dir=bundle_dir, files=tuple(files))
|
||||
_walk(bundle_dir, index, files, seen, skipped)
|
||||
return Bundle(dir=bundle_dir, files=tuple(files), skipped=tuple(skipped))
|
||||
|
||||
|
||||
def bundle_context(bundle: Bundle, *, dimension: str | None = None) -> str:
|
||||
|
|
@ -251,17 +326,30 @@ class IngestStampError(ValueError):
|
|||
forged it could be silently deleted by a later ingest run."""
|
||||
|
||||
|
||||
def _carries_complete_ingest_stamp(frontmatter: dict[str, str]) -> bool:
|
||||
"""Whether ``frontmatter`` carries BOTH halves of the ingest ownership stamp: ``generated:
|
||||
true`` together with a non-empty ``ingest_manifest`` reference (ingest-spec §7).
|
||||
_YAML_TRUE_LITERALS = frozenset({"true", "yes", "on"})
|
||||
"""Every scalar a real YAML reader parses to boolean ``True`` (measured with PyYAML's ``safe_load``
|
||||
core-schema resolver: ``true``/``yes``/``on``, any case, are bool; the same resolver reads bare
|
||||
``y``/``n`` and ``1``/``0`` as string/int, never bool — so those are deliberately EXCLUDED here.
|
||||
Widening past what a YAML reader actually resolves would over-block curated content no ingest
|
||||
pipeline ever produces, on a form nothing downstream would honour as the stamp either."""
|
||||
|
||||
The test is on the COMPLETE stamp, never on the individual field names — curated content may
|
||||
legitimately carry a single provenance field, and a verbatim round-trip of one half must keep
|
||||
working. Values are compared the way ``parse_frontmatter`` yields them (line-oriented strings,
|
||||
quotes retained), so surrounding quotes and case are normalised away here."""
|
||||
|
||||
def _carries_complete_ingest_stamp(frontmatter: dict[str, str]) -> bool:
|
||||
"""Whether ``frontmatter`` carries BOTH halves of the ingest ownership stamp: a ``generated``
|
||||
value a YAML reader would read as boolean ``True`` (``_YAML_TRUE_LITERALS``) together with a
|
||||
non-empty ``ingest_manifest`` reference (ingest-spec §7).
|
||||
|
||||
FAIL-CLOSED on the value literal: the field previously matched only the exact string ``"true"``,
|
||||
so a pinned ingest writer emitting any other YAML-1.1 truthy form (``yes``, ``on``) would have
|
||||
slipped the stamp past this gate undetected — inert only by the accident of the pinned writer's
|
||||
current output, per the CLAUDE.md ingest-stamp invariant. The test is on the COMPLETE stamp,
|
||||
never on the individual field names — curated content may legitimately carry a single provenance
|
||||
field, and a verbatim round-trip of one half must keep working. Values are compared the way
|
||||
``parse_frontmatter`` yields them (line-oriented strings, quotes retained), so surrounding quotes
|
||||
and case are normalised away here."""
|
||||
generated = str(frontmatter.get("generated", "")).strip().strip('"').lower()
|
||||
manifest = str(frontmatter.get("ingest_manifest", "")).strip().strip('"')
|
||||
return generated == "true" and bool(manifest)
|
||||
return generated in _YAML_TRUE_LITERALS and bool(manifest)
|
||||
|
||||
|
||||
def write_concept_file(bundle_dir: str, name: str, frontmatter: dict[str, str], body: str) -> Path:
|
||||
|
|
|
|||
|
|
@ -31,6 +31,7 @@ called by ``run_project``, not a public authoring API (contrast ``verdicts.write
|
|||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections.abc import Mapping, Sequence
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
|
|
@ -131,6 +132,92 @@ def outcome_payload(
|
|||
}
|
||||
|
||||
|
||||
def write_parse_failures(
|
||||
outbox_dir: str,
|
||||
run_id: str,
|
||||
*,
|
||||
failures: Sequence[Mapping[str, str]],
|
||||
) -> Path:
|
||||
"""Write ``{run_id}-parse-failures.json`` — the raw model replies that did NOT parse into the
|
||||
typed IR (Fase 1b, funn 1) — and return its path.
|
||||
|
||||
**This is the only outbox artefact written from a ``finally``**, because it is the only one whose
|
||||
subject is a run that may never finish: the measured 1b failure exhausted the round ledger inside
|
||||
the generation loop and left ``run_project`` as a ``BudgetExceeded``, so the proposal/outcome
|
||||
writers below were never reached. An artefact that recorded parse failures only for runs that
|
||||
survived them would be silent for exactly the runs that need it.
|
||||
|
||||
Takes plain mappings (the caller flattens ``generate.ParseFailure``), so this module stays
|
||||
MAF-free — ``generate`` imports ``agent_framework``, and importing it here would drag MAF into
|
||||
the RAW output layer.
|
||||
|
||||
Byte-determinism is NOT claimed for this file, unlike its two neighbours: its content is a live
|
||||
model's prose, which is not reproducible by construction. It uses the same ``_dump`` form for
|
||||
consistency of reading, not to pin bytes. The caller writes it only when there is at least one
|
||||
failure, so the file's PRESENCE is itself the signal that something did not parse."""
|
||||
directory = Path(outbox_dir)
|
||||
directory.mkdir(parents=True, exist_ok=True)
|
||||
path = directory / f"{run_id}-parse-failures.json"
|
||||
path.write_text(
|
||||
_dump({"run_id": run_id, "parse_failures": [dict(f) for f in failures]}),
|
||||
encoding="utf-8",
|
||||
)
|
||||
return path
|
||||
|
||||
|
||||
def write_exploration(
|
||||
outbox_dir: str,
|
||||
run_id: str,
|
||||
*,
|
||||
payload: Mapping[str, Any],
|
||||
) -> Path:
|
||||
"""Write ``{run_id}-exploration.json`` — what the U4 exploration did before the pipeline ran
|
||||
(§ C.2) — and return its path.
|
||||
|
||||
Takes an already-rendered plain mapping (``explore.trace_payload``) for the reason
|
||||
``write_parse_failures`` takes plain mappings: ``explore`` imports ``agent_framework``, and
|
||||
importing it here would drag MAF into the RAW output layer. The ONE renderer lives beside the
|
||||
dataclasses it renders; this writer only decides bytes and a filename.
|
||||
|
||||
Byte-deterministic like its neighbours (the caller supplies ``run_id``; no wall-clock), and
|
||||
written even when the exploration RAISED — the caller writes it from a ``finally``, because a
|
||||
capped exploration is precisely the one whose per-round ledger a reader needs."""
|
||||
directory = Path(outbox_dir)
|
||||
directory.mkdir(parents=True, exist_ok=True)
|
||||
path = directory / f"{run_id}-exploration.json"
|
||||
path.write_text(_dump({"run_id": run_id, **dict(payload)}), encoding="utf-8")
|
||||
return path
|
||||
|
||||
|
||||
def write_plan_review(
|
||||
outbox_dir: str,
|
||||
run_id: str,
|
||||
*,
|
||||
payload: Mapping[str, Any],
|
||||
) -> Path:
|
||||
"""Write ``{run_id}-plan-review.json`` — the open question of a PARKED exploration (U12) — and
|
||||
return its path.
|
||||
|
||||
This is the outbox half of the asynchronous HITL door: the run writes the question, the expert
|
||||
writes the answer into a separate review INBOX, days later. The two folders are never the same
|
||||
one, for the reason the verdict inbox is never the outbox — a run that read its own output as
|
||||
input would be answering itself.
|
||||
|
||||
Takes an already-rendered plain mapping (``explore.parked_payload``) for the reason
|
||||
``write_exploration`` does: ``explore`` imports ``agent_framework`` and this layer stays
|
||||
MAF-free. Byte-deterministic like its neighbours.
|
||||
|
||||
**Last write wins**, exactly one open question per run: a revision produces a NEW review of a
|
||||
REPLANNED plan, and leaving the superseded one on disk would let an expert answer a question
|
||||
the loop has already moved past. Staleness is caught anyway — the answer names the
|
||||
``request_id`` it answers — but the file should not invite it."""
|
||||
directory = Path(outbox_dir)
|
||||
directory.mkdir(parents=True, exist_ok=True)
|
||||
path = directory / f"{run_id}-plan-review.json"
|
||||
path.write_text(_dump({"run_id": run_id, **dict(payload)}), encoding="utf-8")
|
||||
return path
|
||||
|
||||
|
||||
def write_run_config(
|
||||
config_dir: str,
|
||||
run_id: str,
|
||||
|
|
|
|||
|
|
@ -2,9 +2,11 @@
|
|||
call before the operator pays for one.
|
||||
|
||||
Runs ``python -m portfolio_optimiser.preflight --profile azure``. It checks, purely offline
|
||||
(config/string/env only — NO client construction, NO network, NO auto-login): (1) the endpoint env
|
||||
``PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT`` is set and shaped like a Foundry project endpoint
|
||||
(``https://`` + host ``*.services.ai.azure.com``); (2) the effective model-map (honoring
|
||||
(config/string/env only — NO client construction, NO network, NO auto-login): (1) an endpoint is set
|
||||
under EITHER name the run path accepts (``PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT`` first, then the
|
||||
platform-injected ``FOUNDRY_PROJECT_ENDPOINT`` — same tuple, imported from ``backends``) and is
|
||||
shaped like a Foundry project endpoint (``https://`` + host ``*.services.ai.azure.com``); (2) the
|
||||
effective model-map (honoring
|
||||
``PORTFOLIO_MODEL_MAP``) is structurally valid (``ModelMapContract``); (3) no azure deployment is
|
||||
still a ``REPLACE-WITH-*`` placeholder (via ``resolve_model`` — the SAME seam the run path uses, so
|
||||
preflight and run never validate different maps).
|
||||
|
|
@ -26,10 +28,14 @@ from dataclasses import dataclass
|
|||
|
||||
from pydantic import ValidationError
|
||||
|
||||
from portfolio_optimiser.backends import Profile, _load_effective_map, resolve_model
|
||||
from portfolio_optimiser.backends import _ENDPOINT_ENVS, Profile, _load_effective_map, resolve_model
|
||||
from portfolio_optimiser.contracts import ModelMapContract
|
||||
|
||||
_ENDPOINT_ENV = "PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT"
|
||||
# Fase 5 — the SAME tuple the run path resolves against, imported rather than restated. A second
|
||||
# copy here is how the gate and the run path came to know different variable names in the first
|
||||
# place: preflight refused a hosted container's platform-injected endpoint that backends.py would
|
||||
# have accepted (measured from the extracted handover package, 14.08).
|
||||
_ENDPOINT_ENV = _ENDPOINT_ENVS[0]
|
||||
_FOUNDRY_HOST_SUFFIX = ".services.ai.azure.com"
|
||||
_ROLES = ("default", "proposer", "checker")
|
||||
# Exact operator-facing disclaimer marker (Norwegian, per docs-language convention). The docs note
|
||||
|
|
@ -52,12 +58,28 @@ class PreflightRefusal:
|
|||
reason: str
|
||||
|
||||
|
||||
def _resolve_endpoint() -> str | None:
|
||||
"""First NON-EMPTY of ``_ENDPOINT_ENVS`` — ours first, the platform-injected name as fallback.
|
||||
Precedence over VALUES, not declarations: an exported-but-empty own name falls through instead
|
||||
of shadowing a real injected one into a refusal (the 4b rule, same seam as ``backends.py``)."""
|
||||
for name in _ENDPOINT_ENVS:
|
||||
value = os.environ.get(name)
|
||||
if value:
|
||||
return value
|
||||
return None
|
||||
|
||||
|
||||
def _endpoint_error() -> str | None:
|
||||
"""Return an actionable reason if the endpoint env is missing/misshapen, else ``None``. Pure
|
||||
string work — no ``urllib`` (both the NFR and the offline grep-guard forbid it)."""
|
||||
endpoint = os.environ.get(_ENDPOINT_ENV)
|
||||
endpoint = _resolve_endpoint()
|
||||
if not endpoint:
|
||||
return f"{_ENDPOINT_ENV} er ikke satt (påkrevd for azure-profilen)"
|
||||
# Name BOTH: the operator on a laptop and the operator in a hosted container are looking
|
||||
# for different variables (the fail-fast in ``backends.py`` says the same thing).
|
||||
return (
|
||||
f"{_ENDPOINT_ENVS[0]} (eller plattformens injiserte {_ENDPOINT_ENVS[1]}) "
|
||||
"er ikke satt (påkrevd for azure-profilen)"
|
||||
)
|
||||
if not endpoint.startswith("https://"):
|
||||
return f"{_ENDPOINT_ENV} må være en https://-URL, fikk: {endpoint!r}"
|
||||
# Host = between the scheme and the first '/', minus any port; lowercased. Do NOT require the
|
||||
|
|
|
|||
|
|
@ -57,6 +57,20 @@ class ProvenanceStamp(BaseModel):
|
|||
role: str
|
||||
validator_decision: Literal["validated", "rejected"]
|
||||
token_usage: int
|
||||
#: Was the deterministic gate ANCHORED to the project's own cost lines? ``True`` means
|
||||
#: ``validate_proposal`` ran its stage-0 reconciliation (every ``affected_item`` checked against
|
||||
#: a real ``CostBaseline`` line, within tolerance, BEFORE the solver); ``False`` means the
|
||||
#: bundle shipped no ``cost-baseline.json``, so that stage was SKIPPED and the gate reasoned
|
||||
#: only about numbers the proposal itself supplied. Anchoring stays OPTIONAL (a pre-amendment
|
||||
#: bundle is legitimately un-anchored) — this field does not gate anything, it makes the skip
|
||||
#: legible. A STRUCTURED field rather than prose, for the reason ``BudgetExceeded`` carries
|
||||
#: ``kind``/``limit``/``observed`` as fields (kø-(y)): "was the falsifier anchored" is an
|
||||
#: operative question that must be readable by machine.
|
||||
#:
|
||||
#: REQUIRED, with no default, because both defaults lie: ``True`` would let a constructor that
|
||||
#: forgot claim an anchoring that never happened, and ``False`` would under-claim a real one.
|
||||
#: A binary fact about a falsifier has no honest default.
|
||||
cost_baseline_anchored: bool
|
||||
#: External service calls the run made (B4). EMPTY is a positive statement — "nothing outside
|
||||
#: this process was contacted" — not an absent field, which is why it is always serialized.
|
||||
external_calls: list[ExternalCall] = Field(default_factory=list)
|
||||
|
|
|
|||
|
|
@ -45,7 +45,12 @@ class Project:
|
|||
currency: str
|
||||
cost_items: tuple[CostItem, ...]
|
||||
docs_dir: str # absolute path to this project's bundled cost-docs folder (config-driven)
|
||||
verdict_input: dict[str, str] # SYNTHETIC Layer-2 expert decision/rationale (config-driven)
|
||||
#: The Layer-2 expert decision/rationale, when an expert actually gave one. OPTIONAL and
|
||||
#: ``None`` by default (F2, non-goal 3): a project nobody has reviewed has no honest value
|
||||
#: here, and the previous required field made silence indistinguishable from an approval.
|
||||
#: The shipped reference fixture DOES carry values, and labels them SYNTHETIC at every point —
|
||||
#: that dataset's honesty is F5's subject (measuring the mission claim), not F2's.
|
||||
verdict_input: dict[str, str] | None = None
|
||||
bundle_dir: str | None = None # abs path to an OKF bundle backing this project (Fase 2a S2.0)
|
||||
verdict_dir: str | None = None # abs path to this project's async verdict inbox (Fase 2a S2.0)
|
||||
|
||||
|
|
@ -59,8 +64,9 @@ def load_reference_projects() -> tuple[Project, ...]:
|
|||
|
||||
Each project's ``docs_dir`` is stored in the JSON relative to the package ``data/`` root
|
||||
and resolved here to an absolute filesystem path; ``verdict_input`` carries the SYNTHETIC
|
||||
Layer-2 expert decision/rationale. Missing keys raise ``KeyError`` (fail-fast, matching the
|
||||
existing loader contract)."""
|
||||
Layer-2 expert decision/rationale when the row declares one, and is ``None`` when it does not
|
||||
(F2). Missing keys raise ``KeyError`` (fail-fast, matching the existing loader contract) —
|
||||
``verdict_input`` is deliberately NOT among them."""
|
||||
resource = files("portfolio_optimiser").joinpath(_DATA_RESOURCE)
|
||||
raw = json.loads(resource.read_text(encoding="utf-8"))
|
||||
|
||||
|
|
@ -87,7 +93,9 @@ def load_reference_projects() -> tuple[Project, ...]:
|
|||
for c in p["cost_items"]
|
||||
),
|
||||
docs_dir=str(files("portfolio_optimiser").joinpath(f"data/{p['docs_dir']}")),
|
||||
verdict_input=p["verdict_input"],
|
||||
# ``.get`` — never fail-fast: a reference row without a reviewed verdict is now legal
|
||||
# (F2), and a missing key must mean "nobody reviewed it", not a load error.
|
||||
verdict_input=p.get("verdict_input"),
|
||||
bundle_dir=_resolve(p.get("bundle_dir")),
|
||||
verdict_dir=_resolve(p.get("verdict_dir")),
|
||||
)
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
|
|
@ -39,10 +39,21 @@ from agent_framework import (
|
|||
from agent_framework_openai import OpenAIChatCompletionClient
|
||||
|
||||
from portfolio_optimiser import okf
|
||||
from portfolio_optimiser.explore import (
|
||||
HYPOTHESIS_MARKER,
|
||||
HYPOTHESISER_ROLE,
|
||||
MANAGER_ROLE,
|
||||
NAVIGATOR_ROLE,
|
||||
ExplorationContract,
|
||||
ExplorationResult,
|
||||
ExplorationTrace,
|
||||
explore,
|
||||
)
|
||||
from portfolio_optimiser.ir import AffectedItem, CostBaseline, CostBaselineLine, SavingsProposal
|
||||
from portfolio_optimiser.persona import load_persona_example
|
||||
from portfolio_optimiser.run import RunResult, run_project
|
||||
from portfolio_optimiser.shared_root import shared_root
|
||||
from portfolio_optimiser.tracing import configure_tracing, tracing_notice
|
||||
from portfolio_optimiser.validator import ValidatedProposal
|
||||
from portfolio_optimiser.verdicts import (
|
||||
Verdict,
|
||||
|
|
@ -568,6 +579,9 @@ async def simulate_learning_loop(
|
|||
)
|
||||
|
||||
# Gate-promote the persona verdict from the raw output layer into the OKF wiki (Steg 8).
|
||||
# Narrowed, not defaulted: this demo ALWAYS hands Run A a persona verdict (``verdict_input``
|
||||
# above), so ``None`` here would mean the wiring was cut — an assertion, not a fallback (F2).
|
||||
assert run_a.verdict is not None, "Run A recorded no persona verdict — the wiring is detached"
|
||||
promoted_path = promote_verdict(
|
||||
copy_s,
|
||||
run_a.verdict,
|
||||
|
|
@ -631,6 +645,198 @@ async def simulate_learning_loop(
|
|||
)
|
||||
|
||||
|
||||
#: The U4 walkthrough's prompt and the direction its scripted hypothesiser commits to. The LABEL
|
||||
#: must be absent from the knowledge base — checked at call time, refused rather than assumed —
|
||||
#: because a label the base already states would reach the hypothesis prompt as ordinary navigated
|
||||
#: context, and the scenario would demonstrate nothing (``simulate_learning_loop``'s two-marker
|
||||
#: guard, in the one form this scenario can go vacuous).
|
||||
_EXPLORE_PROMPT = "Finn den rimeligste besparelsen som kan testes i denne kunnskapsbasen."
|
||||
_EXPLORE_LABEL = "styring etter tilstedevaerelse"
|
||||
_EXPLORE_RATIONALE = (
|
||||
"kunnskapsbasen beskriver armaturer som staar paa hele driftsdoegnet, saa en styringsgevinst "
|
||||
"kan testes mot de samme kostlinjene som selve retrofitten"
|
||||
)
|
||||
#: The walkthrough's stated bounds. Written out rather than defaulted for the reason
|
||||
#: ``ExplorationContract`` has no defaults at all: the demo is the one place a reader looks to see
|
||||
#: what bounding an exploration actually looks like.
|
||||
_EXPLORE_CONTRACT = ExplorationContract(
|
||||
max_rounds=4,
|
||||
max_tokens=100_000,
|
||||
max_stall_count=1,
|
||||
max_reset_count=1,
|
||||
max_plan_revisions=0,
|
||||
enable_plan_review=False,
|
||||
)
|
||||
|
||||
|
||||
def _exploration_manager_reply(ledgers: Sequence[str]) -> Callable[[str, str], str]:
|
||||
"""Route a manager prompt to its scripted reply by STAGE, consuming ``ledgers`` in order.
|
||||
|
||||
Keyed on the orchestrator's own prompt text rather than on the project id (the
|
||||
``scripted_proposer`` rule), and that is not an exception to it: the manager is asked five
|
||||
DIFFERENT questions — facts, plan, progress ledger, replan, final answer — and which one it is
|
||||
being asked is the only thing a reply can be keyed on. The project id is constant across all
|
||||
five and would select nothing."""
|
||||
remaining = list(ledgers)
|
||||
|
||||
def _select(blob: str, _role: str) -> str:
|
||||
if "provide the final answer" in blob:
|
||||
return "FINAL: utforskningen er ferdig."
|
||||
if "pure JSON format" in blob:
|
||||
return remaining.pop(0) if remaining else _exploration_ledger(satisfied=True)
|
||||
if "went wrong on this last run" in blob:
|
||||
return "PLAN-UPDATE: revidert plan."
|
||||
if "rewrite the following fact sheet" in blob:
|
||||
return "FACTS-UPDATE: reviderte fakta."
|
||||
if "bullet-point plan" in blob:
|
||||
return "PLAN: - la hypotesiseren forme en retning"
|
||||
if "pre-survey" in blob:
|
||||
return "FACTS: kunnskapsbasen er forankret."
|
||||
return "{}"
|
||||
|
||||
return _select
|
||||
|
||||
|
||||
def _exploration_ledger(*, satisfied: bool) -> str:
|
||||
"""One progress ledger naming a REAL participant. The name is what makes the run non-vacuous:
|
||||
a ``next_speaker`` matching nobody makes the orchestrator answer having asked no one."""
|
||||
return json.dumps(
|
||||
{
|
||||
"is_request_satisfied": {"reason": "r", "answer": satisfied},
|
||||
"is_in_loop": {"reason": "r", "answer": False},
|
||||
"is_progress_being_made": {"reason": "r", "answer": True},
|
||||
"next_speaker": {"reason": "r", "answer": HYPOTHESISER_ROLE},
|
||||
"instruction_or_question": {"reason": "r", "answer": "Form én retning."},
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def scripted_exploration_factory(
|
||||
sink: list[str], *, label: str, rationale: str, ledgers: Sequence[str]
|
||||
) -> Callable[[str], BaseChatClient]:
|
||||
"""A role-keyed client factory covering BOTH the exploration's roles and the pipeline's.
|
||||
|
||||
One factory, because the walkthrough is one continuous story: the same call that shapes the
|
||||
mandate hands it to the run that evaluates it. The pipeline's roles fall through to the
|
||||
existing ``_proposer_reply`` / ``_CHECKER_APPROVE`` scaffolding, so nothing about the debate
|
||||
changes."""
|
||||
pipeline = scripted_factory({"proposer": _proposer_reply, "checker": _CHECKER_APPROVE}, sink)
|
||||
hypothesis_line = f"{HYPOTHESIS_MARKER} " + json.dumps({"label": label, "rationale": rationale})
|
||||
|
||||
def factory(role: str) -> BaseChatClient:
|
||||
if role == MANAGER_ROLE:
|
||||
return ScriptedChatClient(
|
||||
sink=sink, role=role, reply_selector=_exploration_manager_reply(ledgers)
|
||||
)
|
||||
if role == HYPOTHESISER_ROLE:
|
||||
return ScriptedChatClient(hypothesis_line, sink, role=role)
|
||||
if role == NAVIGATOR_ROLE:
|
||||
return ScriptedChatClient("NAVIGATOR: leste indeksen.", sink, role=role)
|
||||
return pipeline(role)
|
||||
|
||||
return factory
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ExplorationSimulationResult:
|
||||
"""The trace of one U4 walkthrough: what the loop shaped, and whether the pipeline used it.
|
||||
|
||||
There is deliberately NO ``label_in_bundle`` field. The vacuity guard below raises before a
|
||||
result exists, so such a field could only ever be ``False`` — an assertion on it would be green
|
||||
against every implementation, which is the class this scenario's guard exists to prevent. The
|
||||
guard IS the control; a second, always-true restatement of it would only make the real one
|
||||
easier to discount."""
|
||||
|
||||
exploration: ExplorationResult
|
||||
trace: ExplorationTrace
|
||||
run: RunResult
|
||||
label: str
|
||||
label_in_generation_prompt: bool
|
||||
generation_prompts: list[str]
|
||||
|
||||
|
||||
async def simulate_exploration(
|
||||
bundle_dir: str,
|
||||
work_dir: str,
|
||||
*,
|
||||
project_id: str = _PROJECT_ID,
|
||||
prompt: str = _EXPLORE_PROMPT,
|
||||
label: str = _EXPLORE_LABEL,
|
||||
rationale: str = _EXPLORE_RATIONALE,
|
||||
contract: ExplorationContract | None = None,
|
||||
max_rounds: int = 3,
|
||||
) -> ExplorationSimulationResult:
|
||||
"""Walk U4 offline: prompt + knowledge base -> ``Mandate`` -> the pipeline that evaluates it.
|
||||
|
||||
A THIRD scenario beside ``simulate_learning_loop``, and deliberately not part of it. The
|
||||
demo's stdout and stderr are both byte-pinned fixtures (``tests/golden/demo-transcript.*``),
|
||||
and the learning walkthrough's claim — that a verdict crosses two runs — is a different claim
|
||||
from this one. Folding them together would make each harder to read and would move a pinned
|
||||
transcript for a reason unrelated to what it pins. This scenario is reachable by NAME only;
|
||||
``main()`` does not call it.
|
||||
|
||||
Honesty limit, the same one §1 states for the rest of the demo: every reply is scripted, so
|
||||
what is shown is that the plumbing closes — that a shaped direction really becomes an
|
||||
``Approach`` the proposer is asked about — never that a live model would shape a good one.
|
||||
"""
|
||||
copy = Path(work_dir) / "explore-bundle"
|
||||
shutil.copytree(bundle_dir, copy)
|
||||
copy_s = str(copy)
|
||||
|
||||
# The vacuity guard. A label the base already states would reach the hypothesis prompt as
|
||||
# navigated context whether or not the exploration ran, so the scenario's own assertion would
|
||||
# hold against an implementation that never wired the mandate at all.
|
||||
context = okf.bundle_context(okf.navigate_bundle(copy_s))
|
||||
if label in context:
|
||||
raise ValueError(
|
||||
f"label {label!r} already appears in the knowledge base, so it would reach the "
|
||||
"hypothesis prompt as ordinary context and the walkthrough would demonstrate nothing; "
|
||||
"choose a direction the base does not already state"
|
||||
)
|
||||
|
||||
sink: list[str] = []
|
||||
trace = ExplorationTrace()
|
||||
exploration = await explore(
|
||||
prompt,
|
||||
contract=contract or _EXPLORE_CONTRACT,
|
||||
bundle_dirs=(copy_s,),
|
||||
client_factory=scripted_exploration_factory(
|
||||
sink,
|
||||
label=label,
|
||||
rationale=rationale,
|
||||
ledgers=[_exploration_ledger(satisfied=False), _exploration_ledger(satisfied=True)],
|
||||
),
|
||||
trace=trace,
|
||||
)
|
||||
|
||||
run = cast(
|
||||
RunResult,
|
||||
await run_project(
|
||||
project_id,
|
||||
"local",
|
||||
docs_dir=copy_s,
|
||||
bundle_dir=copy_s,
|
||||
verdict_input={"decision": "approved", "rationale": "ekspert-persona (sim)"},
|
||||
store=VerdictStore(verdicts=[]),
|
||||
client_factory=scripted_exploration_factory(
|
||||
sink, label=label, rationale=rationale, ledgers=[]
|
||||
),
|
||||
mandate=exploration.mandate,
|
||||
max_rounds=max_rounds,
|
||||
),
|
||||
)
|
||||
|
||||
prompts = _generation_prompts(sink)
|
||||
return ExplorationSimulationResult(
|
||||
exploration=exploration,
|
||||
trace=trace,
|
||||
run=run,
|
||||
label=label,
|
||||
label_in_generation_prompt=any(label in p for p in prompts),
|
||||
generation_prompts=prompts,
|
||||
)
|
||||
|
||||
|
||||
def _outcome_line(result: RunResult) -> str:
|
||||
o = result.outcome
|
||||
if isinstance(o, ValidatedProposal):
|
||||
|
|
@ -769,6 +975,9 @@ def _run_trace_lines(result: RunResult, *, marker: str, marker_in_prompt: bool)
|
|||
# captured while the run is live. The long one — a verdict file dropped into a folder after the
|
||||
# run — is printed by ``main`` between the two runs, because that is when it happens.
|
||||
lines.append(" Steg 7 — SVAR PÅ TILBAKEMELDING (ekspert-persona, kort løkke i kjøringen)")
|
||||
# Same narrowing as above: every simulated run records a persona verdict, so ``None`` is a
|
||||
# detached wiring rather than a state this renderer has to have a word for.
|
||||
assert result.verdict is not None, "the simulated run recorded no persona verdict"
|
||||
lines.append(f" dom: {result.verdict.decision}")
|
||||
lines.append(f" begrunnelse: {_clip(result.verdict.rationale, 300)}")
|
||||
return lines
|
||||
|
|
@ -817,6 +1026,16 @@ def main(argv: list[str] | None = None) -> int: # pragma: no cover - console tr
|
|||
# library consumer of this module keeps its own logging configuration.
|
||||
quiet_expected_round_cap_notice()
|
||||
|
||||
# U14: install OpenTelemetry providers if — and only if — PORTFOLIO_OTEL asks for it, BEFORE
|
||||
# any workflow runs (MAF's own "call once at startup, before telemetry is captured"). Absent
|
||||
# the variable this configures nothing at all, which is what keeps the pinned stdout AND the
|
||||
# four-line pinned stderr byte-identical; console mode writes spans to stderr, never stdout.
|
||||
# The announcement goes to stderr for the same reason.
|
||||
setup = configure_tracing()
|
||||
notice = tracing_notice(setup)
|
||||
if notice is not None:
|
||||
print(notice, file=sys.stderr)
|
||||
|
||||
work = tempfile.mkdtemp(prefix="po-sim-")
|
||||
# THE call site (P3, GO): the demo runs the DELIVERED bundle, which ships its own
|
||||
# `cost-baseline.json` — so the gate is anchored on numbers a domain team wrote, not on numbers
|
||||
|
|
|
|||
265
src/portfolio_optimiser/tracing.py
Normal file
265
src/portfolio_optimiser/tracing.py
Normal file
|
|
@ -0,0 +1,265 @@
|
|||
"""U14 — the opt-in OpenTelemetry seam: one variable, two sinks, and no silent egress.
|
||||
|
||||
A run of this framework already *makes* spans: MAF's ``ENABLE_INSTRUMENTATION`` defaults to True
|
||||
(``observability.py:697``), so every workflow, executor and chat call is instrumented — and, with
|
||||
no provider configured, every one of those spans is discarded. This module is the provider, and
|
||||
nothing else. It exists because the exploration loop being built on top of it (U4) hands a manager
|
||||
the freedom to choose its own next step, and the programme's ordering rule is that nothing which
|
||||
grants that freedom lands before the freedom can be *watched*. An organisation cannot be asked to
|
||||
trust an autonomous exploration it cannot read afterwards.
|
||||
|
||||
**The contract, in one paragraph.** ``PORTFOLIO_OTEL`` is read on truthiness. Absent or empty:
|
||||
``configure_otel_providers`` is not called at all, so nothing is configured and nothing can leave —
|
||||
"off" means off, not "on, exporting to nowhere". ``console``: spans are written to **stderr**, so a
|
||||
traced run and an untraced run print byte-identical stdout and the pinned demo transcript survives.
|
||||
``otlp``: spans go over the network, and ONLY when the operator has named a destination in one of
|
||||
the standard ``OTEL_EXPORTER_OTLP_*_ENDPOINT`` variables. Anything else is refused by name.
|
||||
|
||||
**Two rules here are measurements, not preferences.** ``configure_otel_providers`` composes its
|
||||
exporter list in a fixed order (``observability.py:849``):
|
||||
|
||||
1. exporters derived from the standard ``OTEL_EXPORTER_OTLP_*`` variables — **unconditionally**,
|
||||
2. the exporters passed in as ``exporters=``,
|
||||
3. a ``ConsoleSpanExporter()`` — default sink ``sys.stdout`` — when ``enable_console_exporters``
|
||||
is true, taken from the argument *or*, if that is ``None``, from ``ENABLE_CONSOLE_EXPORTERS``
|
||||
in the environment.
|
||||
|
||||
Step 3 is why ``enable_console_exporters=False`` is passed explicitly in **both** modes: left to
|
||||
the environment, an operator with that variable exported gets a span dump on stdout, which spike
|
||||
S6 measured as destroying the golden transcript outright. Step 1 is why console mode **refuses**
|
||||
when an OTLP endpoint variable is present: the word "console" would otherwise be a false statement
|
||||
about where the run's contents went. The refusal names the variable and leaves it alone —
|
||||
validation, never repair, the same rule ``write_concept_file`` and ``load_optional_cost_baseline``
|
||||
follow. Unsetting an operator's environment behind their back would be a fix that hides its own
|
||||
cause.
|
||||
|
||||
**What is deliberately absent.** The OTLP exporter *packages*
|
||||
(``opentelemetry-exporter-otlp-proto-grpc`` / ``-http``) are not declared dependencies. They are
|
||||
egress, they drag grpc and protobuf into a published wheel for a mode that is off by default, and
|
||||
MAF already raises an ``ImportError`` that names the package to install. Stated honesty limit:
|
||||
``PORTFOLIO_OTEL=otlp`` works only after the operator installs one of them.
|
||||
|
||||
**The ``PLAN_CREATED`` / ``REPLANNED`` / ``PROGRESS_LEDGER_UPDATED`` events now exist** (U4, økt
|
||||
56). They were held back here in økt 55 on the ground that an emitter written before its call site
|
||||
is a shape guessed rather than measured; the call site is ``explore._absorb``, and the events are
|
||||
recorded on the exploration span this module's ``exploration_tracer`` hands out. Nothing about the
|
||||
contract above changed: with tracing off there is no provider, so those events are discarded like
|
||||
every other span this process makes.
|
||||
|
||||
MAF-touching by construction, so this module never enters the framework-neutral context layer
|
||||
(``okf.py``); the ``test_okf_is_maf_free`` guard keeps that boundary.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from collections.abc import Callable, Mapping
|
||||
from dataclasses import dataclass
|
||||
from typing import IO, Any, Final
|
||||
|
||||
#: The one variable that turns tracing on. Read on TRUTHINESS, never presence (the 4b rule): an
|
||||
#: exported-but-empty value is a shell accident, and treating it as a request would turn
|
||||
#: ``export PORTFOLIO_OTEL=`` into a fail-fast on an unknown mode.
|
||||
TRACING_ENV: Final = "PORTFOLIO_OTEL"
|
||||
|
||||
MODE_CONSOLE: Final = "console"
|
||||
MODE_OTLP: Final = "otlp"
|
||||
|
||||
#: The closed set. A value outside it is refused by name rather than falling back to off — an
|
||||
#: operator who asked for a trace and mistyped would otherwise get the black box this seam exists
|
||||
#: to remove, arrived at by accident and without a word.
|
||||
_MODES: Final = (MODE_CONSOLE, MODE_OTLP)
|
||||
|
||||
#: Every standard variable that makes ``_get_exporters_from_env`` construct a NETWORK exporter.
|
||||
#: All four are checked, not just the base one: a run configured only via
|
||||
#: ``OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`` exports exactly the signal this seam is about.
|
||||
_OTLP_ENDPOINT_ENVS: Final = (
|
||||
"OTEL_EXPORTER_OTLP_ENDPOINT",
|
||||
"OTEL_EXPORTER_OTLP_TRACES_ENDPOINT",
|
||||
"OTEL_EXPORTER_OTLP_METRICS_ENDPOINT",
|
||||
"OTEL_EXPORTER_OTLP_LOGS_ENDPOINT",
|
||||
)
|
||||
|
||||
|
||||
class TracingConfigError(ValueError):
|
||||
"""A tracing request that cannot be honoured as stated.
|
||||
|
||||
``ValueError`` so the hosted flate maps it to 400 through the existing arm rather than needing
|
||||
a new one: it is a malformed request about this process's configuration, not a crash.
|
||||
"""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TracingSetup:
|
||||
"""What a run resolved about its own tracing — the single source both callers read.
|
||||
|
||||
``mode`` is ``None`` when tracing is off. ``endpoints`` carries ``(variable, value)`` pairs and
|
||||
is non-empty only in OTLP mode, so the announcement can name the variable the operator edits
|
||||
rather than only the URL. Both the configuring and the announcing descend from this one value:
|
||||
a renderer that re-read the environment would be a second resolution of the same rule, free to
|
||||
disagree with the run it describes (the ``cost_baseline_notice`` rule).
|
||||
"""
|
||||
|
||||
mode: str | None
|
||||
endpoints: tuple[tuple[str, str], ...]
|
||||
|
||||
|
||||
def declared_otlp_endpoints(env: Mapping[str, str]) -> tuple[tuple[str, str], ...]:
|
||||
"""Return the OTLP endpoint variables the environment actually declares, in a fixed order.
|
||||
|
||||
Truthiness again: an exported-but-empty endpoint variable declares nothing, and MAF's own
|
||||
``os.getenv`` reads would skip it too, so treating it as a declaration would make console mode
|
||||
refuse over a destination that does not exist.
|
||||
"""
|
||||
return tuple(
|
||||
(name, env[name].strip()) for name in _OTLP_ENDPOINT_ENVS if env.get(name, "").strip()
|
||||
)
|
||||
|
||||
|
||||
def resolve_tracing_mode(env: Mapping[str, str]) -> str | None:
|
||||
"""Read ``PORTFOLIO_OTEL`` into the closed mode set, or ``None`` when tracing is off."""
|
||||
raw = env.get(TRACING_ENV, "").strip()
|
||||
if not raw:
|
||||
return None
|
||||
if raw not in _MODES:
|
||||
raise TracingConfigError(
|
||||
f"{TRACING_ENV}={raw!r} is not a tracing mode. Allowed: "
|
||||
+ ", ".join(repr(mode) for mode in _MODES)
|
||||
+ f". Unset {TRACING_ENV} to run without tracing."
|
||||
)
|
||||
return raw
|
||||
|
||||
|
||||
def configure_tracing(
|
||||
*,
|
||||
env: Mapping[str, str] | None = None,
|
||||
stream: IO[str] | None = None,
|
||||
configure: Callable[..., None] | None = None,
|
||||
) -> TracingSetup:
|
||||
"""Install OpenTelemetry providers for this process if — and only if — asked to.
|
||||
|
||||
Call ONCE, at process startup, before any telemetry is captured (MAF's own instruction; a
|
||||
second call is a no-op behind its ``_executed_setup`` guard). Returns what was resolved, so the
|
||||
caller can announce it without re-reading anything.
|
||||
|
||||
``configure`` is injectable for the same reason ``run._default_factory`` is: it is the one seam
|
||||
a test can observe without installing global providers into the pytest process. The real proof
|
||||
that the seam works is a subprocess running the actual demo, not this argument.
|
||||
"""
|
||||
env = os.environ if env is None else env
|
||||
mode = resolve_tracing_mode(env)
|
||||
endpoints = declared_otlp_endpoints(env)
|
||||
|
||||
if mode is None:
|
||||
# Not "configure with nothing" — NOT CALLING is what makes "off" mean nothing can leave.
|
||||
# A call with an empty exporter list would still install providers and re-read every
|
||||
# OTEL_EXPORTER_OTLP_* variable in the ambient environment.
|
||||
return TracingSetup(mode=None, endpoints=())
|
||||
|
||||
if configure is None: # pragma: no cover - trivial default resolution
|
||||
from agent_framework.observability import configure_otel_providers
|
||||
|
||||
configure = configure_otel_providers
|
||||
|
||||
if mode == MODE_CONSOLE:
|
||||
if endpoints:
|
||||
named = ", ".join(name for name, _ in endpoints)
|
||||
raise TracingConfigError(
|
||||
f"{TRACING_ENV}={MODE_CONSOLE} promises that spans stay in this process, but the "
|
||||
f"environment declares a network exporter: {named}. OpenTelemetry exporters are "
|
||||
"built from those variables unconditionally, so the run would also ship its spans "
|
||||
f"over the wire. Unset them, or ask for {TRACING_ENV}={MODE_OTLP} and say so."
|
||||
)
|
||||
# Imported here rather than at module scope: the exporter is constructed only in this
|
||||
# branch, and every importer of the CLI would otherwise pay for a mode that is off by
|
||||
# default.
|
||||
from opentelemetry.sdk.trace.export import ConsoleSpanExporter
|
||||
|
||||
# `out=stream` is the whole of console mode's safety. The default sink is sys.stdout, and
|
||||
# stdout is byte-pinned by tests/golden/demo-transcript.stdout.
|
||||
exporters: list[Any] | None = [ConsoleSpanExporter(out=stream or _default_stream())]
|
||||
else:
|
||||
if not endpoints:
|
||||
raise TracingConfigError(
|
||||
f"{TRACING_ENV}={MODE_OTLP} was requested but no endpoint is declared. Set one of: "
|
||||
+ ", ".join(_OTLP_ENDPOINT_ENVS)
|
||||
+ ". Configuring providers with nowhere to export would produce a run that looks "
|
||||
"traced and is not."
|
||||
)
|
||||
# No exporter of our own: MAF builds them from the standard variables, and a hand-rolled
|
||||
# second one would be the duplicate free to drift from the OTel spec.
|
||||
exporters = None
|
||||
|
||||
# `enable_console_exporters=False` is EXPLICIT in both modes, and load-bearing in both: left as
|
||||
# None it falls back to ENABLE_CONSOLE_EXPORTERS in the environment, whose console exporter
|
||||
# writes to stdout.
|
||||
configure(enable_console_exporters=False, exporters=exporters)
|
||||
return TracingSetup(mode=mode, endpoints=endpoints)
|
||||
|
||||
|
||||
def _default_stream() -> IO[str]:
|
||||
"""``sys.stderr`` resolved at CALL time, not import time.
|
||||
|
||||
The demo replaces neither, but a caller that redirects ``sys.stderr`` before startup should get
|
||||
the redirected one — an import-time binding would have captured whatever was current when the
|
||||
module was first imported.
|
||||
"""
|
||||
import sys
|
||||
|
||||
return sys.stderr
|
||||
|
||||
|
||||
def tracing_notice(setup: TracingSetup) -> str | None:
|
||||
"""Render what a run says about its own tracing, or ``None`` when there is nothing to say.
|
||||
|
||||
ONE renderer with N call sites, never N copies of the wording (kø-(p)), and it takes the
|
||||
already-resolved ``TracingSetup`` rather than an environment: the printed line and the
|
||||
providers that were installed then descend from the same single resolution.
|
||||
|
||||
``None`` when tracing is off — omission, never an empty row (``mandate.announce``'s rule, the
|
||||
one ``cost_baseline_notice`` and ``skipped_links_notice`` follow). Here it is load-bearing past
|
||||
style: the pinned demo stderr is four lines, and a "tracing: off" row would have made it five.
|
||||
|
||||
The OTLP form names the VARIABLE beside the value, because the variable is what the operator
|
||||
edits — and prints one row per declared endpoint rather than only the first, so a run exporting
|
||||
logs and traces to different collectors declares both. English, like every other line this CLI
|
||||
prints.
|
||||
"""
|
||||
if setup.mode is None:
|
||||
return None
|
||||
if setup.mode == MODE_CONSOLE:
|
||||
return (
|
||||
f" Tracing: {TRACING_ENV}={MODE_CONSOLE} — OpenTelemetry spans are written to stderr; "
|
||||
"nothing leaves this process"
|
||||
)
|
||||
rows = "\n".join(f" {name} = {value}" for name, value in setup.endpoints)
|
||||
return (
|
||||
f" Tracing: {TRACING_ENV}={MODE_OTLP} — OpenTelemetry spans are EXPORTED OVER THE NETWORK "
|
||||
f"to the endpoints declared below\n{rows}"
|
||||
)
|
||||
|
||||
|
||||
#: The instrumentation scope every exploration span is created under. One name, so a collector
|
||||
#: can select this framework's own spans apart from MAF's (``invoke_agent``, ``workflow.run``)
|
||||
#: without matching on span names that MAF owns and may rename.
|
||||
EXPLORATION_TRACER_NAME: Final = "portfolio_optimiser.explore"
|
||||
|
||||
|
||||
def exploration_tracer() -> Any:
|
||||
"""The tracer the exploration loop records its decisions on.
|
||||
|
||||
``get_tracer`` is safe to call whether or not a provider was installed: with none, OpenTelemetry
|
||||
hands back a no-op tracer and every span and event is discarded. That is the SAME shape MAF's
|
||||
own instrumentation already has (``ENABLE_INSTRUMENTATION`` defaults to True and its spans are
|
||||
thrown away for want of a provider), and it is what lets the exploration emit unconditionally.
|
||||
Gating emission on ``PORTFOLIO_OTEL`` would be a second resolution of a rule this module owns,
|
||||
free to disagree with the providers actually installed.
|
||||
|
||||
A FUNCTION rather than a module-level tracer, and the reason is ordering: ``configure_tracing``
|
||||
runs at process startup, and a tracer bound at import time would have been taken from the
|
||||
global provider that existed BEFORE it — a no-op one, permanently. It is also the seam a test
|
||||
substitutes a local provider through, without installing anything globally.
|
||||
"""
|
||||
from opentelemetry import trace
|
||||
|
||||
return trace.get_tracer(EXPLORATION_TRACER_NAME)
|
||||
|
|
@ -10,11 +10,13 @@ from __future__ import annotations
|
|||
|
||||
import json
|
||||
from collections.abc import Callable, Sequence
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from agent_framework import BaseChatClient
|
||||
|
||||
from portfolio_optimiser.reference_domain import load_reference_projects
|
||||
from portfolio_optimiser.run import RunResult
|
||||
from portfolio_optimiser.simulation import ScriptedChatClient
|
||||
from portfolio_optimiser.verdicts import VerdictStore, seed_store
|
||||
|
||||
|
|
@ -192,3 +194,50 @@ def docs_dir(tmp_path) -> str:
|
|||
encoding="utf-8",
|
||||
)
|
||||
return str(d)
|
||||
|
||||
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
# Fase 1b — the FULL-RUN contract, in ONE copy (kø-(p): a second copy of an assertion drifts).
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
#: What the paid run must fell, stated as an assertion rather than as prose. The open honesty limit
|
||||
#: after økt 37 is narrow and specific: *"that the emitted structured schema IS accepted by the LIVE
|
||||
#: endpoint is NOT verified — the tests prove conformance with the DOCUMENTED subset, not
|
||||
#: acceptance."* So the contract asserts schema ACCEPTANCE, never model JUDGEMENT.
|
||||
#:
|
||||
#: The discriminator is an artefact this repo already owns: ``{run_id}-parse-failures.json`` is
|
||||
#: written if and ONLY if some reply failed to parse (økt 35 invariant — "the file's presence is the
|
||||
#: signal"). Its ABSENCE beside a RunResult therefore proves that every generation reply came back
|
||||
#: in the requested shape, which is exactly what "the live endpoint honoured the schema" means.
|
||||
#:
|
||||
#: ``validator_decision`` is the second half: it mirrors the VALIDATOR alone (never the checker), so
|
||||
#: reading it proves the deterministic gate actually ran on a parsed candidate. Both ``validated``
|
||||
#: and ``rejected`` satisfy the contract — a P90 rejection is a run that CONCLUDED, and demanding
|
||||
#: ``validated`` would be asserting that the model reasons well, which no schema can promise and
|
||||
#: which one paid run could not establish anyway.
|
||||
def assert_full_run_contract(result: RunResult, outbox_dir: str | Path, run_id: str) -> None:
|
||||
"""Assert the Fase 1b full-run contract on a completed ``run_project`` result.
|
||||
|
||||
Two things, and deliberately nothing else:
|
||||
|
||||
1. **No parse-failure artefact** — every reply parsed, i.e. the live endpoint accepted the
|
||||
emitted ``response_format`` schema. This is the honesty limit being felled.
|
||||
2. **The validator was reached and decided** — ``provenance.validator_decision`` is one of the
|
||||
two decisions the deterministic gate emits.
|
||||
|
||||
Deliberately NOT asserted: ``checker_verdict``, token counts, ``validated_count``, or the
|
||||
content of the proposal. Those are model-judgement claims, and one run cannot carry them.
|
||||
"""
|
||||
artefact = Path(outbox_dir) / f"{run_id}-parse-failures.json"
|
||||
if artefact.exists():
|
||||
# Quote the evidence in the failure message: the whole point of økt 35 was that the operator
|
||||
# should never again have to guess WHY a reply did not parse.
|
||||
raise AssertionError(
|
||||
f"the endpoint did NOT honour the structured schema — {artefact.name} exists.\n"
|
||||
f"{artefact.read_text(encoding='utf-8')[:2000]}"
|
||||
)
|
||||
assert result.provenance.validator_decision in {"validated", "rejected"}, (
|
||||
"the deterministic validator never decided — the run did not reach the gate with a "
|
||||
f"parsed candidate (validator_decision={result.provenance.validator_decision!r})"
|
||||
)
|
||||
|
|
|
|||
260
tests/spikes/test_e_magentic.py
Normal file
260
tests/spikes/test_e_magentic.py
Normal file
|
|
@ -0,0 +1,260 @@
|
|||
"""Spike E tests — Magentic exploration loop, MEASURED before anything is built
|
||||
(order 20260823T162224Z, plan § D.1 spikes S0–S6).
|
||||
|
||||
Every test here pins ONE binary outcome the plan's § F assumption table needs. Where an
|
||||
outcome is version-dependent (the manager's session lifetime changed in orchestrations
|
||||
1.0.1, upstream regression fix #4371), the test asserts against the STRUCTURAL probe of the
|
||||
installed manager — never a version string — and carries a non-vacuity control so a zero
|
||||
can never be the absence of a run.
|
||||
|
||||
Pattern: tests/spikes/test_b_footguns.py.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from portfolio_optimiser.budget import BudgetExceeded
|
||||
from spikes.e_magentic import (
|
||||
ExplorationCallRecord,
|
||||
checkpoint_until_plan_review,
|
||||
expert_liaison_answer_round_trip,
|
||||
fresh_manager_contamination,
|
||||
manager_budget_enforced,
|
||||
manager_keeps_persistent_session,
|
||||
plan_review_round_trip,
|
||||
reset_signal_resets_participant_session,
|
||||
run_resume_subprocess,
|
||||
shared_builder_contamination,
|
||||
shared_manager_contamination,
|
||||
single_use_second_run,
|
||||
validator_latency_seconds,
|
||||
)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# S1 — B7 state isolation (E1–E4, E7) in the repo's own form
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_e1_second_run_of_one_built_workflow_is_refused() -> None:
|
||||
"""E1: a built Magentic workflow is SINGLE-USE. The second ``.run()`` raises and makes
|
||||
ZERO model calls — stronger than GroupChat 1.9.0's silent empty [2, 0, 0] re-run, because
|
||||
a workflow that cannot run cannot fabricate an answer."""
|
||||
result = await single_use_second_run()
|
||||
assert result["first_ok"] is True
|
||||
assert result["second_error"] == "RuntimeError"
|
||||
assert "already been completed" in result["second_message"]
|
||||
# The refusal is FREE: nothing was spent proving it.
|
||||
assert result["manager_calls_added"] == 0
|
||||
assert result["worker_calls_added"] == 0
|
||||
|
||||
|
||||
async def test_e3_fresh_manager_per_build_never_contaminates() -> None:
|
||||
"""E3 — the CONTROL, and the invariant the plan's C.4 rule rests on: a fresh manager
|
||||
(fresh builder, fresh agent, fresh client) per exploration leaks nothing across runs, on
|
||||
EVERY measured version. This is what ``fresh_exploration_workflow`` will implement."""
|
||||
bled, total, kinds = await fresh_manager_contamination()
|
||||
assert total >= 4, f"only {total} manager calls — the control would be reporting on nothing"
|
||||
assert bled == 0
|
||||
# The five-call shape (facts, plan, ledger_UNSAT, ledger_SAT, final) is the discriminator:
|
||||
# a contaminated run answers satisfied on its FIRST ledger and never calls the worker.
|
||||
assert kinds == ["facts", "plan", "ledger_UNSAT", "ledger_SAT", "final"]
|
||||
|
||||
|
||||
async def test_e2_shared_manager_bleed_tracks_the_persistent_session() -> None:
|
||||
"""E2: two workflows sharing ONE ``StandardMagenticManager``. Whether run 2's manager still
|
||||
sees run 1's task is decided by ONE structural property of the installed manager — whether
|
||||
it holds a persistent ``AgentSession`` built in ``__init__`` (1.0.0) or creates a throwaway
|
||||
one per call (1.0.1, #4371). The probe reads that property, never a version string."""
|
||||
bled, total, kinds = await shared_manager_contamination()
|
||||
assert total >= 4, f"only {total} manager calls — a zero here would be vacuous"
|
||||
if manager_keeps_persistent_session():
|
||||
assert bled == total, "a persistent manager session must bleed EVERY run-2 call"
|
||||
# The fabricated-answer signature: satisfied on the first ledger, worker never called.
|
||||
assert "ledger_UNSAT" not in kinds
|
||||
else:
|
||||
assert bled == 0, "a per-call manager session must leak nothing"
|
||||
assert kinds == ["facts", "plan", "ledger_UNSAT", "ledger_SAT", "final"]
|
||||
|
||||
|
||||
async def test_e4_shared_builder_bleed_tracks_the_persistent_session() -> None:
|
||||
"""E4: ONE ``MagenticBuilder``, two ``.build()`` calls. ``manager_agent=`` constructs the
|
||||
manager eagerly and hands the SAME instance to every build, so E4 is E2 reached by the
|
||||
route a caller is most likely to take by accident. Same discriminator."""
|
||||
bled, total, kinds = await shared_builder_contamination()
|
||||
assert total >= 4, f"only {total} manager calls — a zero here would be vacuous"
|
||||
if manager_keeps_persistent_session():
|
||||
assert bled == total
|
||||
assert "ledger_UNSAT" not in kinds
|
||||
else:
|
||||
assert bled == 0
|
||||
assert kinds == ["facts", "plan", "ledger_UNSAT", "ledger_SAT", "final"]
|
||||
|
||||
|
||||
async def test_e7_reset_signal_does_not_reset_the_participant_session() -> None:
|
||||
"""E7: ``MagenticResetSignal`` clears the cache and the conversation but writes the fresh
|
||||
session to ``_agent_thread`` — an attribute the executor never reads. Stall-replan therefore
|
||||
gives a fresh manager ledger and STALE participants. Measured on 1.0.0 and still true on
|
||||
1.0.1: never rely on the reset to empty participant memory."""
|
||||
result = await reset_signal_resets_participant_session()
|
||||
assert result["cache_cleared"] is True
|
||||
assert result["conversation_cleared"] is True
|
||||
assert result["session_identity_changed"] is False # the whole finding
|
||||
assert result["orphan_attribute_written"] is True
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# S2 — does BudgetMiddleware fire on the MANAGER's calls? (plan § F / A1, A2)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_s2_budget_middleware_fires_on_the_manager_path() -> None:
|
||||
"""A1 + A2: the manager is the most talkative participant, and the plan's hard token cap is
|
||||
a lie if agent-level ``ChatMiddleware`` does not reach it. Two halves, both required: the
|
||||
typed ``BudgetExceeded`` must LEAVE ``workflow.run`` (A2), and the meter must have been
|
||||
CHARGED (A1) — a refusal with a zero meter would prove only that something raised."""
|
||||
result = await manager_budget_enforced(max_tokens=1, attach=True)
|
||||
assert result["raised"] == "BudgetExceeded"
|
||||
assert result["kind"] == "tokens"
|
||||
assert result["meter_tokens"] > 0, "the middleware never charged — it did not run"
|
||||
assert result["completed"] is False
|
||||
|
||||
|
||||
async def test_s2_control_detaching_the_manager_middleware_lets_the_run_finish() -> None:
|
||||
"""The detach control the order requires: with no middleware on the manager the SAME
|
||||
one-token budget stops nothing and the run completes. Without this the test above could
|
||||
pass on an implementation where anything at all raised."""
|
||||
result = await manager_budget_enforced(max_tokens=1, attach=False)
|
||||
assert result["raised"] is None
|
||||
assert result["completed"] is True
|
||||
assert result["meter_tokens"] == 0
|
||||
|
||||
|
||||
async def test_s2_budget_exceeded_is_the_repo_type_not_a_look_alike() -> None:
|
||||
"""The exception that leaves ``workflow.run`` must be the repo's own ``BudgetExceeded``
|
||||
carrying ``kind``/``limit``/``observed`` — the triple the 429 channel reads (kø-(y)). An
|
||||
orchestration layer that wrapped it in an ``ExceptionGroup`` would make the exploration
|
||||
layer's error mapping a fiction, so the object itself is asserted, not just its name."""
|
||||
result = await manager_budget_enforced(max_tokens=1, attach=True, return_exception=True)
|
||||
exc = result["exception"]
|
||||
assert isinstance(exc, BudgetExceeded)
|
||||
assert exc.kind == "tokens"
|
||||
assert exc.limit == 1
|
||||
assert exc.observed > exc.limit # limit and observed must not be the same number
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# S5 — quick_validate latency (plan C.0 level 1: the tool the hypothesiser calls)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_s5_validator_latency_is_affordable_as_an_in_loop_tool() -> None:
|
||||
"""S5: ``validate_proposal`` is what ``quick_validate`` wraps. If a single call costs
|
||||
seconds, every hypothesis the manager tests costs wall-clock the contract has to budget."""
|
||||
median, samples = validator_latency_seconds(runs=20)
|
||||
assert samples == 20
|
||||
assert median > 0.0, "a zero median means the clock never moved — nothing was measured"
|
||||
assert median < 2.0, f"median {median:.3f}s per validate_proposal — budget it in the contract"
|
||||
|
||||
|
||||
def test_call_record_is_a_plain_readable_row() -> None:
|
||||
"""The record type carries the four facts every contamination verdict is computed from;
|
||||
a spike whose evidence cannot be printed is a spike nobody can re-check."""
|
||||
record = ExplorationCallRecord(kind="facts", messages=2, sees_alpha=True, sees_beta=False)
|
||||
assert record.kind == "facts"
|
||||
assert record.sees_alpha is True
|
||||
|
||||
|
||||
@pytest.mark.parametrize("attach", [True, False])
|
||||
async def test_s2_arms_disagree(attach: bool) -> None:
|
||||
"""Both arms of S2 run under one parametrisation too, so a future refactor that made the
|
||||
two arms identical shows up as a shared outcome rather than as two green tests."""
|
||||
result = await manager_budget_enforced(max_tokens=1, attach=attach)
|
||||
assert result["completed"] is not attach
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# S3 / S3b — the two HITL doors (plan § C.5 / § C.6, assumptions A3 and A5)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_s3_plan_review_round_trip_revises_then_approves() -> None:
|
||||
"""A3: the request/response round-trip the plan's synchronous HITL rests on.
|
||||
|
||||
Four facts in one run: the review STOPS the workflow before any output; a ``revise`` replans
|
||||
and asks AGAIN (so an always-revising expert is an unbounded loop unless the contract caps
|
||||
it); the revise costs manager calls but NO ledger call (it is not a round); and an
|
||||
``approve`` lets the loop finish."""
|
||||
result = await plan_review_round_trip()
|
||||
assert result["pending_before_review"] == 1
|
||||
assert result["request_type"] == "MagenticPlanReviewRequest"
|
||||
assert result["is_stalled"] is False
|
||||
assert result["stopped_without_output"] is True
|
||||
# Before the review the manager has only surveyed and planned -- it never reached a ledger.
|
||||
assert result["kinds_before_review"] == ["facts", "plan"]
|
||||
# The measured cost of one revise: two manager calls, zero ledger calls, zero rounds.
|
||||
assert result["revise_manager_calls"] == ["facts_update", "plan_update"]
|
||||
assert not any(k.startswith("ledger") for k in result["revise_manager_calls"])
|
||||
# ... and it asks again -- this is why max_plan_revisions must exist in the contract.
|
||||
assert result["pending_after_revise"] == 1
|
||||
assert result["outputs_after_approve"] == ["FINAL: the worker did it."]
|
||||
|
||||
|
||||
async def test_s3b_expert_liaison_answer_reaches_the_manager() -> None:
|
||||
"""A5: ``AgentApprovalExecutor`` as the ``expert_liaison`` PARTICIPANT — door 3 of § C.6.
|
||||
|
||||
Reachable, and the expert's words do become context: the liaison sees them, and the manager
|
||||
sees them in a later prompt. The cost is two round-trips per human turn -- ``from_strings``
|
||||
feeds the answer back into the liaison and does NOT resume the manager (measured: zero
|
||||
manager calls between the two requests); only ``approve`` forwards the output."""
|
||||
answer = "EXPERT-SAYS-TEST-THE-LED-RETROFIT"
|
||||
result = await expert_liaison_answer_round_trip(answer=answer)
|
||||
assert result["reachable"] is True
|
||||
assert result["manager_calls_between_requests"] == [] # from_strings alone resumes nothing
|
||||
assert result["second_request"] == 1 # ... it asks again, with the answer folded in
|
||||
assert result["liaison_saw_answer"] is True
|
||||
assert result["manager_saw_answer"] is True # the point: it becomes context for the next round
|
||||
assert result["outputs"] == ["FINAL: the worker did it."]
|
||||
|
||||
|
||||
async def test_s3b_control_an_unsent_sentinel_never_reaches_the_manager() -> None:
|
||||
"""The control that makes the assertion above non-vacuous. Same run, same expert answer --
|
||||
but the manager's prompts are scanned for a sentinel the expert NEVER sent. It must come
|
||||
back absent. Without this arm, ``manager_saw_answer is True`` would be equally consistent
|
||||
with a scanner that matches anything, and door 3 would look proven when it was not."""
|
||||
result = await expert_liaison_answer_round_trip(
|
||||
answer="EXPERT-SAYS-TEST-THE-LED-RETROFIT", probe="NEVER-SENT-SENTINEL-XYZ"
|
||||
)
|
||||
assert result["reachable"] is True
|
||||
assert result["liaison_saw_answer"] is True # the run really happened
|
||||
assert result["manager_saw_answer"] is False # ... and the unsent sentinel is nowhere in it
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# S4 — resume a pending plan review in a NEW PROCESS (plan U12, assumption A4)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_s4_pending_plan_review_resumes_in_a_fresh_process(tmp_path: Path) -> None:
|
||||
"""A4: the asynchronous HITL time-scale. The parent runs until the plan review stops it and
|
||||
leaves checkpoints on disk; a SEPARATE interpreter -- which never saw the run -- answers the
|
||||
pending request from the checkpoint alone and drives the workflow to its final answer.
|
||||
|
||||
In-process resume would prove nothing here: U12's whole claim is that the expert can answer
|
||||
days later from a file inbox."""
|
||||
storage = tmp_path / "checkpoints"
|
||||
storage.mkdir()
|
||||
first = await checkpoint_until_plan_review(str(storage))
|
||||
assert first["request_id"], "no plan review was raised -- nothing to resume"
|
||||
assert first["outputs"] == []
|
||||
assert first["checkpoint_ids"], "no checkpoint was written -- the resume would be vacuous"
|
||||
|
||||
resumed = run_resume_subprocess(
|
||||
str(storage), request_id=first["request_id"], checkpoint_id=first["checkpoint_ids"][-1]
|
||||
)
|
||||
assert resumed["outputs"] == ["FINAL: the worker did it."]
|
||||
assert resumed["pending_after_resume"] == 0
|
||||
# The fresh process really did the remaining work -- not a replay of a cached answer.
|
||||
assert any(k.startswith("ledger") for k in resumed["manager_kinds"])
|
||||
689
tests/test_async_plan_review_loadbearing.py
Normal file
689
tests/test_async_plan_review_loadbearing.py
Normal file
|
|
@ -0,0 +1,689 @@
|
|||
"""U12 + asynchronous U13 (plan § D.2 row 3) — a plan review a human answers over DAYS.
|
||||
|
||||
F4 (økt 63) made "still spørsmål, be om svar, bruke svarene" reachable from the CLI, but only
|
||||
SYNCHRONOUSLY: ``terminal_plan_reviewer`` blocks the loop on a human at a terminal, so the answer
|
||||
has to arrive while the process is alive. Målbilde §3's time-scale is the other one — the expert
|
||||
answers days later, in a process that never saw the run — and that is impossible without carrying
|
||||
the workflow's state to disk.
|
||||
|
||||
**The measured trap this row is built around** (§ F row A4, and confirmed here against the
|
||||
INSTALLED source rather than the plan's prose): ``FileCheckpointStorage.list_checkpoints``
|
||||
(``_workflows/_checkpoint.py:386-388``) swallows a deserialisation failure into a
|
||||
``logger.warning`` and returns an EMPTY list. Without ``MagenticPlanReviewRequest`` and
|
||||
``MagenticPlanReviewResponse`` in ``allowed_checkpoint_types``, a resume therefore fails as an
|
||||
ABSENCE — "nothing to resume" — not as an error. A test asserting "the listing is empty, so there
|
||||
is nothing to resume" would be GREEN against exactly that defect, which is why every test here
|
||||
asserts that the resume DID something instead.
|
||||
|
||||
**The discriminator, in both halves.** A door that writes a question file and a resume that reads
|
||||
an answer file both pass "the expert was asked" while failing the målbilde. So the goal test drives
|
||||
the F4 T1 shape across TWO process boundaries: ``revise`` written into an inbox on day 1 must reach
|
||||
the manager, make it replan, and produce a SECOND question about the NEW plan — which an
|
||||
always-approve resume, or one that discards the answer, cannot produce.
|
||||
|
||||
The witness is the artefacts, never scraped stdout: ``{run_id}-plan-review.json`` is the question
|
||||
and ``{run_id}-exploration.json`` is the record of what was decided.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
from portfolio_optimiser import explore as ex
|
||||
from portfolio_optimiser import hitl, run
|
||||
|
||||
_REPO = Path(__file__).resolve().parents[1]
|
||||
_BUNDLE_DIR = _REPO / "shared" / "examples" / "bygg-energi-mikro"
|
||||
_PID = "BYGG-KONTOR-NORD"
|
||||
_RUN_ID = "async-review"
|
||||
|
||||
_PROPOSER_REPLY = json.dumps(
|
||||
{
|
||||
"measure": "LED-retrofit",
|
||||
"affected_items": [{"code": "ENERGI-TOTAL-EL", "quantity": 300000, "unit_cost": 1.0}],
|
||||
"claimed_saving_nok": 30000,
|
||||
}
|
||||
)
|
||||
_MANAGER_REPLY = json.dumps(
|
||||
{
|
||||
"is_request_satisfied": {"reason": "r", "answer": True},
|
||||
"is_in_loop": {"reason": "r", "answer": False},
|
||||
"is_progress_being_made": {"reason": "r", "answer": True},
|
||||
"next_speaker": {"reason": "r", "answer": "hypothesiser"},
|
||||
"instruction_or_question": {"reason": "r", "answer": "go"},
|
||||
}
|
||||
)
|
||||
_REPLIES = {
|
||||
"proposer": _PROPOSER_REPLY,
|
||||
"checker": "VERDICT: APPROVE",
|
||||
"manager": _MANAGER_REPLY,
|
||||
"navigator": "NAVIGATOR: read the index.",
|
||||
"hypothesiser": "HYPOTHESIS: " + json.dumps({"label": "Night setback", "rationale": "y"}),
|
||||
}
|
||||
|
||||
_FEEDBACK = "Also test night setback on the ventilation."
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# Fixture plumbing: the two operator surfaces, driven exactly as an operator would drive them
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _config_file(tmp_path: Path, **overrides: Any) -> str:
|
||||
path = tmp_path / "exploration.json"
|
||||
path.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"max_rounds": 4,
|
||||
"max_tokens": 200_000,
|
||||
"max_stall_count": 2,
|
||||
"max_reset_count": 1,
|
||||
"max_plan_revisions": 2,
|
||||
"enable_plan_review": True,
|
||||
**overrides,
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
return str(path)
|
||||
|
||||
|
||||
def _replies_file(tmp_path: Path) -> str:
|
||||
path = tmp_path / "replies.json"
|
||||
path.write_text(json.dumps(_REPLIES), encoding="utf-8")
|
||||
return str(path)
|
||||
|
||||
|
||||
def _park_argv(tmp_path: Path, **config: Any) -> list[str]:
|
||||
"""Day 1: explore with the ASYNCHRONOUS door armed. No reviewer at this terminal."""
|
||||
return [
|
||||
_PID,
|
||||
"--docs-dir",
|
||||
str(_BUNDLE_DIR),
|
||||
"--bundle-dir",
|
||||
str(_BUNDLE_DIR),
|
||||
"--explore",
|
||||
"Find the cheapest saving.",
|
||||
"--explore-config",
|
||||
_config_file(tmp_path, **config),
|
||||
"--scripted-replies",
|
||||
_replies_file(tmp_path),
|
||||
"--outbox-dir",
|
||||
str(tmp_path / "outbox"),
|
||||
"--checkpoint-dir",
|
||||
str(tmp_path / "checkpoints"),
|
||||
"--run-id",
|
||||
_RUN_ID,
|
||||
]
|
||||
|
||||
|
||||
def _resume_argv(tmp_path: Path) -> list[str]:
|
||||
"""Day N: a process that never saw the run, resuming from the checkpoint and the answer."""
|
||||
return [
|
||||
_PID,
|
||||
"--docs-dir",
|
||||
str(_BUNDLE_DIR),
|
||||
"--bundle-dir",
|
||||
str(_BUNDLE_DIR),
|
||||
"--scripted-replies",
|
||||
_replies_file(tmp_path),
|
||||
"--outbox-dir",
|
||||
str(tmp_path / "outbox"),
|
||||
"--checkpoint-dir",
|
||||
str(tmp_path / "checkpoints"),
|
||||
"--review-inbox",
|
||||
str(tmp_path / "review-inbox"),
|
||||
"--resume",
|
||||
_RUN_ID,
|
||||
]
|
||||
|
||||
|
||||
def _question(tmp_path: Path) -> dict[str, Any]:
|
||||
path = tmp_path / "outbox" / f"{_RUN_ID}-plan-review.json"
|
||||
assert path.exists(), "a parked exploration must leave the question where an expert can read it"
|
||||
return json.loads(path.read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
def _artefact(tmp_path: Path) -> dict[str, Any]:
|
||||
path = tmp_path / "outbox" / f"{_RUN_ID}-exploration.json"
|
||||
assert path.exists(), "the exploration artefact must be written even when the run parked"
|
||||
return json.loads(path.read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
def _answer(tmp_path: Path, **payload: Any) -> None:
|
||||
"""The expert's side of the loop: a file dropped into the review inbox, days later."""
|
||||
inbox = tmp_path / "review-inbox"
|
||||
inbox.mkdir(parents=True, exist_ok=True)
|
||||
body = {"run_id": _RUN_ID, "request_id": _question(tmp_path)["request_id"], **payload}
|
||||
(inbox / f"{_RUN_ID}-plan-review-answer.json").write_text(json.dumps(body), encoding="utf-8")
|
||||
|
||||
|
||||
def _resume_in_a_fresh_process(tmp_path: Path) -> subprocess.CompletedProcess[str]:
|
||||
"""The resume runs in its OWN interpreter, because an in-process resume would prove nothing.
|
||||
|
||||
The whole claim of U12 is that the only thing crossing the boundary is what is on disk. The
|
||||
subprocess precedent is ``spikes/e_magentic_resume.py`` and ``test_hosting_loadbearing.py``.
|
||||
"""
|
||||
return subprocess.run(
|
||||
[sys.executable, "-m", "portfolio_optimiser.run", *_resume_argv(tmp_path)],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
cwd=str(_REPO),
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# 1. THE GOAL — asked on day 1, answered on day N, and the answer USED
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_an_answer_written_days_later_reaches_the_manager_and_produces_a_new_question(
|
||||
tmp_path, capsys
|
||||
) -> None:
|
||||
"""T1: the whole row, across two process boundaries.
|
||||
|
||||
Day 1 parks at the plan review. The expert writes ``revise`` into the inbox. A FRESH
|
||||
interpreter resumes from the checkpoint alone — and because the feedback reached the manager,
|
||||
the manager replans and asks AGAIN about the NEW plan.
|
||||
|
||||
RED against a resume that discards the answer, and RED against one that always approves: both
|
||||
yield a single review and no second question. RED against an in-process-only door: there is no
|
||||
checkpoint for the child to resume from.
|
||||
"""
|
||||
assert run.main(_park_argv(tmp_path)) == 0, capsys.readouterr().err
|
||||
|
||||
first = _question(tmp_path)
|
||||
assert first["index"] == 0
|
||||
assert first["request_id"], "without the request id the answer can never be routed back"
|
||||
|
||||
_answer(tmp_path, decision="revise", feedback=_FEEDBACK)
|
||||
completed = _resume_in_a_fresh_process(tmp_path)
|
||||
assert completed.returncode == 0, completed.stderr
|
||||
|
||||
second = _question(tmp_path)
|
||||
assert second["index"] == 1, (
|
||||
"a revision must produce a SECOND question about the replanned plan — a resume that "
|
||||
"discarded the answer would finish, or park again on the SAME review"
|
||||
)
|
||||
assert second["request_id"] != first["request_id"], (
|
||||
"the replanned review is a new request; reusing the old id would route the next answer "
|
||||
"into a request the orchestrator has already retired"
|
||||
)
|
||||
|
||||
reviews = _artefact(tmp_path)["plan_reviews"]
|
||||
assert [r["decision"] for r in reviews] == ["revise"], (
|
||||
"the record must carry the decision the expert actually made"
|
||||
)
|
||||
assert reviews[0]["feedback"] == _FEEDBACK, (
|
||||
"what a human told the loop is worth nothing paraphrased"
|
||||
)
|
||||
|
||||
|
||||
def test_the_answer_is_carried_all_the_way_to_a_completed_exploration(tmp_path, capsys) -> None:
|
||||
"""T2: the control for T1 — the door does not only re-ask, it can also FINISH.
|
||||
|
||||
A gate that could only ever park again would be a hang wearing a loop's clothes. Answering
|
||||
``approve`` must let the exploration conclude and the pipeline run, leaving NO open question.
|
||||
"""
|
||||
assert run.main(_park_argv(tmp_path)) == 0, capsys.readouterr().err
|
||||
_answer(tmp_path, decision="approve")
|
||||
|
||||
completed = _resume_in_a_fresh_process(tmp_path)
|
||||
assert completed.returncode in (0, 1), completed.stderr
|
||||
assert "Traceback" not in completed.stderr, completed.stderr
|
||||
|
||||
artefact = _artefact(tmp_path)
|
||||
assert artefact["completed"] is True, (
|
||||
"an approved review must let the exploration conclude, not park again"
|
||||
)
|
||||
assert [r["decision"] for r in artefact["plan_reviews"]] == ["approve"]
|
||||
assert (
|
||||
hitl.pending_plan_reviews(str(tmp_path / "outbox"), str(tmp_path / "review-inbox")) == []
|
||||
), "a concluded exploration leaves no question waiting for anybody"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# 2. THE MEASURED TRAP — a checkpoint that cannot be read back
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_the_parked_checkpoint_can_actually_be_read_back(tmp_path, capsys) -> None:
|
||||
"""T3: the trap, asserted POSITIVELY.
|
||||
|
||||
``list_checkpoints`` turns a blocked deserialisation into an empty list, so the failure mode
|
||||
is silence. This asserts the opposite of silence: the id the question file names resolves to a
|
||||
checkpoint that loads. RED the moment ``_ALLOWED_CHECKPOINT_TYPES`` stops naming both types.
|
||||
"""
|
||||
import asyncio
|
||||
|
||||
assert run.main(_park_argv(tmp_path)) == 0, capsys.readouterr().err
|
||||
checkpoint_id = _question(tmp_path)["checkpoint_id"]
|
||||
assert checkpoint_id, "parking on an unreadable checkpoint is an unanswerable question"
|
||||
|
||||
storage = ex.checkpoint_storage(str(tmp_path / "checkpoints"))
|
||||
loaded = asyncio.run(storage.load(checkpoint_id))
|
||||
assert loaded.checkpoint_id == checkpoint_id
|
||||
|
||||
|
||||
def test_a_park_with_no_readable_checkpoint_refuses_instead_of_writing_a_dead_question(
|
||||
tmp_path, capsys, monkeypatch
|
||||
) -> None:
|
||||
"""T4: fail LOUDLY where the framework fails silently.
|
||||
|
||||
If the listing comes back empty there is nothing to resume from, and writing the question
|
||||
anyway would hand an expert a review whose answer can never be applied — the fourth face of
|
||||
the verification law, built into our own surface. Simulated by emptying the allow-list, which
|
||||
is exactly what produces an empty listing in the installed source.
|
||||
|
||||
It leaves as a RAISE, not an rc-1 refusal, and that is the consistent call rather than a
|
||||
softer one: argv was fine and the loop had already spent, so this is the run failing — the
|
||||
same channel ``BudgetExceeded`` and an unreadable marked hypothesis use. What the door owes is
|
||||
that it fails LOUDLY where the framework fails silently, and that no dead question is left
|
||||
behind for somebody to answer into the void.
|
||||
"""
|
||||
monkeypatch.setattr(ex, "_ALLOWED_CHECKPOINT_TYPES", ())
|
||||
|
||||
with pytest.raises(ex.CheckpointUnreadable):
|
||||
run.main(_park_argv(tmp_path))
|
||||
|
||||
capsys.readouterr()
|
||||
assert not (tmp_path / "outbox" / f"{_RUN_ID}-plan-review.json").exists(), (
|
||||
"a question nobody can answer must not be written at all"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# 3. FAIL-CLOSED ON THE EXPERT'S OWN INPUT (the F4 rule, on a file instead of a terminal)
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_an_answer_outside_the_vocabulary_is_refused_never_read_as_a_sign_off(
|
||||
tmp_path, capsys
|
||||
) -> None:
|
||||
"""T5: the closed vocabulary survives the move from stdin to a file.
|
||||
|
||||
``terminal_plan_reviewer`` re-asks anything it does not recognise; a file cannot be re-asked,
|
||||
so the only honest answer is a refusal. Reading it as approval would sign a plan nobody signed.
|
||||
"""
|
||||
assert run.main(_park_argv(tmp_path)) == 0, capsys.readouterr().err
|
||||
_answer(tmp_path, decision="looks fine to me")
|
||||
|
||||
completed = _resume_in_a_fresh_process(tmp_path)
|
||||
|
||||
assert completed.returncode == 1, completed.stdout
|
||||
assert "looks fine to me" in completed.stderr, completed.stderr
|
||||
assert _artefact(tmp_path)["plan_reviews"] == [], (
|
||||
"an unreadable answer must not be recorded as a decision"
|
||||
)
|
||||
|
||||
|
||||
def test_a_revision_with_nothing_to_revise_is_refused(tmp_path, capsys) -> None:
|
||||
"""T6: ``revise`` without feedback is the same defect wearing a valid token.
|
||||
|
||||
``PlanReviewDecision.revise`` refuses an empty revision at the library door too, and that is
|
||||
exactly why the assertion here is on the SHAPE of the failure rather than on the exit code:
|
||||
measured, removing the inbox guard still gives rc 1 and still puts the word "revise" on
|
||||
stderr — as a TRACEBACK out of the library. A test that stopped at those two facts could not
|
||||
tell a fail-closed door from an unhandled exception, so it asserts the structured refusal.
|
||||
"""
|
||||
assert run.main(_park_argv(tmp_path)) == 0, capsys.readouterr().err
|
||||
_answer(tmp_path, decision="revise", feedback=" ")
|
||||
|
||||
completed = _resume_in_a_fresh_process(tmp_path)
|
||||
|
||||
assert completed.returncode == 1, completed.stdout
|
||||
assert "Traceback" not in completed.stderr, completed.stderr
|
||||
assert "run refused" in completed.stderr, completed.stderr
|
||||
assert "revise" in completed.stderr.lower(), completed.stderr
|
||||
|
||||
|
||||
def test_an_answer_to_a_different_review_is_refused_never_applied_to_this_one(
|
||||
tmp_path, capsys
|
||||
) -> None:
|
||||
"""T7: staleness is a refusal, not a silent misapplication.
|
||||
|
||||
Two reviews of one run share a file name, so the answer names the ``request_id`` it answers.
|
||||
An answer left over from the previous round must not be applied to the current question — that
|
||||
would sign off a plan the expert never saw.
|
||||
"""
|
||||
assert run.main(_park_argv(tmp_path)) == 0, capsys.readouterr().err
|
||||
_answer(tmp_path, decision="approve")
|
||||
inbox = tmp_path / "review-inbox" / f"{_RUN_ID}-plan-review-answer.json"
|
||||
stale = json.loads(inbox.read_text(encoding="utf-8"))
|
||||
stale["request_id"] = "a-request-from-last-week"
|
||||
inbox.write_text(json.dumps(stale), encoding="utf-8")
|
||||
|
||||
completed = _resume_in_a_fresh_process(tmp_path)
|
||||
|
||||
assert completed.returncode == 1, completed.stdout
|
||||
assert "a-request-from-last-week" in completed.stderr, completed.stderr
|
||||
|
||||
|
||||
def test_a_resume_with_no_answer_yet_refuses_before_spending_anything(tmp_path, capsys) -> None:
|
||||
"""T8: "not answered yet" is the normal state of this door, and it must be cheap.
|
||||
|
||||
The hoist rule from økt 57: a refusal that fires AFTER the model calls is indistinguishable
|
||||
from one that fires before, by exit code alone — so this asserts that NOTHING was spent.
|
||||
"""
|
||||
assert run.main(_park_argv(tmp_path)) == 0, capsys.readouterr().err
|
||||
before = json.loads(
|
||||
(tmp_path / "outbox" / f"{_RUN_ID}-plan-review.json").read_text(encoding="utf-8")
|
||||
)
|
||||
|
||||
completed = _resume_in_a_fresh_process(tmp_path)
|
||||
|
||||
assert completed.returncode == 1, completed.stdout
|
||||
after = json.loads(
|
||||
(tmp_path / "outbox" / f"{_RUN_ID}-plan-review.json").read_text(encoding="utf-8")
|
||||
)
|
||||
assert after == before, (
|
||||
"a resume with no answer must not touch the run at all — an unchanged question file is "
|
||||
"what proves the exploration was never restarted"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# 4. THE PENDING REGISTRY (hitl.py) — who is still waiting on whom
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_a_parked_review_is_pending_until_its_own_answer_lands(tmp_path, capsys) -> None:
|
||||
"""T9: ``hitl.pending_plan_reviews`` is the machine-readable "still waiting", mirroring
|
||||
``hitl.pending`` for proposals: an outbox question whose answer is not yet in the inbox.
|
||||
|
||||
The join is on ``request_id``, so an answer to a DIFFERENT review leaves the question pending
|
||||
rather than quietly clearing it — the same fail-closed rule the resume applies.
|
||||
"""
|
||||
outbox, inbox = str(tmp_path / "outbox"), str(tmp_path / "review-inbox")
|
||||
assert run.main(_park_argv(tmp_path)) == 0, capsys.readouterr().err
|
||||
|
||||
waiting = hitl.pending_plan_reviews(outbox, inbox)
|
||||
assert [p.run_id for p in waiting] == [_RUN_ID]
|
||||
assert waiting[0].plan, "an expert cannot answer a review that does not show them the plan"
|
||||
|
||||
_answer(tmp_path, decision="approve")
|
||||
inbox_file = tmp_path / "review-inbox" / f"{_RUN_ID}-plan-review-answer.json"
|
||||
wrong = json.loads(inbox_file.read_text(encoding="utf-8"))
|
||||
wrong["request_id"] = "someone-elses-review"
|
||||
inbox_file.write_text(json.dumps(wrong), encoding="utf-8")
|
||||
assert hitl.pending_plan_reviews(outbox, inbox) == waiting, (
|
||||
"an answer to another review must not clear this one"
|
||||
)
|
||||
|
||||
_answer(tmp_path, decision="approve")
|
||||
assert hitl.pending_plan_reviews(outbox, inbox) == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# 5. THE BUDGET MUST SPAN THE SUSPENSION, NOT RESTART WITH IT
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_a_resumed_exploration_does_not_get_a_fresh_budget(tmp_path, capsys) -> None:
|
||||
"""T10: the hole a park would otherwise open.
|
||||
|
||||
Both budget channels live in the process: a fresh ``TokenMeter`` and an empty ledger mean a
|
||||
resumed exploration could spend its whole cap AGAIN, once per park — unbounded consumption
|
||||
behind guards that all look satisfied (the S3.4 class). The suspended state therefore carries
|
||||
what was already spent, and the resume starts from it.
|
||||
|
||||
RED when the carry-over is detached: the resumed run then reports a spend of its own calls
|
||||
only, and the ledger restarts at round 1.
|
||||
"""
|
||||
assert run.main(_park_argv(tmp_path)) == 0, capsys.readouterr().err
|
||||
parked = ex.load_parked(_question(tmp_path))
|
||||
assert parked.tokens_spent > 0, (
|
||||
"the parked run made model calls; a zero here would make the assertion below vacuous"
|
||||
)
|
||||
|
||||
_answer(tmp_path, decision="approve")
|
||||
assert _resume_in_a_fresh_process(tmp_path).returncode in (0, 1)
|
||||
|
||||
artefact = _artefact(tmp_path)
|
||||
assert artefact["tokens_spent"] > parked.tokens_spent, (
|
||||
"the resumed exploration must add to the day-1 spend, never start over from zero"
|
||||
)
|
||||
|
||||
|
||||
def test_the_revision_cap_is_counted_across_the_suspension_not_restarted_by_it(
|
||||
tmp_path, capsys
|
||||
) -> None:
|
||||
"""T11: the hole the carry-over closes, and the reason it exists at all.
|
||||
|
||||
A revise costs two manager calls, emits no ledger and consumes no round (§ F, A3), so
|
||||
``max_plan_revisions`` is the ONLY bound on it. With ``max_plan_revisions=1`` a second revision
|
||||
must be refused — and refused on the strength of what the FIRST process did, which only the
|
||||
carried ``plan_reviews`` can say.
|
||||
|
||||
RED when the carry-over is detached: every leg then counts zero prior revisions, the cap never
|
||||
binds, and the run parks a third time. That mutation left the entire suite green before this
|
||||
test existed, which is exactly the vacuous-gate class it was written against.
|
||||
"""
|
||||
assert run.main(_park_argv(tmp_path, max_plan_revisions=1)) == 0, capsys.readouterr().err
|
||||
|
||||
_answer(tmp_path, decision="revise", feedback=_FEEDBACK)
|
||||
assert _resume_in_a_fresh_process(tmp_path).returncode == 0
|
||||
assert _question(tmp_path)["index"] == 1, "the first revision must be applied"
|
||||
|
||||
_answer(tmp_path, decision="revise", feedback="And once more.")
|
||||
second = _resume_in_a_fresh_process(tmp_path)
|
||||
|
||||
assert second.returncode in (0, 1), second.stderr
|
||||
assert "Traceback" not in second.stderr, second.stderr
|
||||
artefact = _artefact(tmp_path)
|
||||
assert artefact["stop"] == "plan_revisions_exhausted", (
|
||||
"the second revision is over the cap and must STOP the exploration, never be sent"
|
||||
)
|
||||
assert (
|
||||
hitl.pending_plan_reviews(str(tmp_path / "outbox"), str(tmp_path / "review-inbox")) == []
|
||||
), "a stopped exploration must not leave a third question waiting for anybody"
|
||||
|
||||
|
||||
def test_the_review_history_survives_every_leg_not_just_the_last(tmp_path, capsys) -> None:
|
||||
"""T16: the carried ``plan_reviews`` are the RECORD, and the record is the only witness.
|
||||
|
||||
Written because the mutation that detaches ``trace.plan_reviews.extend(parked.plan_reviews)``
|
||||
left the ENTIRE suite green (measured, økt 64): the revision cap counts
|
||||
``parked.plan_reviews`` DIRECTLY, so it binds either way, and the first two legs cannot tell
|
||||
the difference — a park with an empty carried history and one with none look identical until
|
||||
there are two reviews to carry. An unmeasured seam is this repo's recurring defect class, so
|
||||
the gate is the THIRD leg, where the two implementations finally diverge.
|
||||
|
||||
RED when the carry-over is detached: the artefact then records only the review the LAST
|
||||
process saw — the day-1 revision vanishes from the run's own history — and the next question
|
||||
is numbered 1 again, so two distinct reviews of one run share an index.
|
||||
"""
|
||||
second_feedback = "And check the pumps while you are at it."
|
||||
assert run.main(_park_argv(tmp_path, max_plan_revisions=2)) == 0, capsys.readouterr().err
|
||||
assert _question(tmp_path)["index"] == 0
|
||||
|
||||
_answer(tmp_path, decision="revise", feedback=_FEEDBACK)
|
||||
assert _resume_in_a_fresh_process(tmp_path).returncode == 0
|
||||
assert _question(tmp_path)["index"] == 1, "the first revision must be applied"
|
||||
|
||||
_answer(tmp_path, decision="revise", feedback=second_feedback)
|
||||
third = _resume_in_a_fresh_process(tmp_path)
|
||||
assert third.returncode == 0, third.stderr
|
||||
|
||||
assert _question(tmp_path)["index"] == 2, (
|
||||
"the third question is the third review of this run — a resume that dropped the carried "
|
||||
"history would number it 1 again, and two different plans would share one index"
|
||||
)
|
||||
reviews = _artefact(tmp_path)["plan_reviews"]
|
||||
assert [r["feedback"] for r in reviews] == [_FEEDBACK, second_feedback], (
|
||||
"the record must carry every decision the expert made, not only the most recent one: "
|
||||
"what a human told the loop on day 1 is not superseded by what they said on day 2"
|
||||
)
|
||||
|
||||
|
||||
def test_what_the_first_process_found_survives_into_the_resumed_mandate(tmp_path) -> None:
|
||||
"""T12: the other half of the carry-over — the loop's own findings.
|
||||
|
||||
A plan review can fire mid-run after a stall, and everything the loop found before it would be
|
||||
lost if the mandate were minted from only what the resuming process observed. Driven through a
|
||||
CRAFTED parked state (the ``budget_stop`` precedent) because the scripted manager never stalls,
|
||||
so the mid-run park has no reachable path today — but the carry-over that serves it does.
|
||||
|
||||
RED when the ledger and hypotheses are dropped on resume: the mandate then names only what the
|
||||
second leg saw, and the returned ledger restarts at the resumed round.
|
||||
"""
|
||||
import asyncio
|
||||
import dataclasses
|
||||
|
||||
from portfolio_optimiser.simulation import scripted_factory
|
||||
|
||||
factory = scripted_factory(_REPLIES, [])
|
||||
contract = ex.load_exploration_contract(_config_file(tmp_path))
|
||||
checkpoints = str(tmp_path / "checkpoints")
|
||||
|
||||
with pytest.raises(ex.PlanReviewParked) as caught:
|
||||
asyncio.run(
|
||||
ex.explore(
|
||||
"Find the cheapest saving.",
|
||||
contract=contract,
|
||||
bundle_dirs=(str(_BUNDLE_DIR),),
|
||||
client_factory=factory,
|
||||
checkpoint_dir=checkpoints,
|
||||
)
|
||||
)
|
||||
|
||||
carried = dataclasses.replace(
|
||||
caught.value.parked,
|
||||
hypotheses=("HYPOTHESIS: " + json.dumps({"label": "Carried", "rationale": "found first"}),),
|
||||
ledger=(
|
||||
ex.LedgerEntry(
|
||||
round_index=1,
|
||||
is_request_satisfied=False,
|
||||
is_in_loop=False,
|
||||
is_progress_being_made=True,
|
||||
next_speaker="hypothesiser",
|
||||
instruction_or_question="keep going",
|
||||
speaker_known=True,
|
||||
),
|
||||
),
|
||||
)
|
||||
result = asyncio.run(
|
||||
ex.resume_exploration(
|
||||
carried,
|
||||
ex.PlanReviewDecision.approve(),
|
||||
checkpoint_dir=checkpoints,
|
||||
client_factory=factory,
|
||||
)
|
||||
)
|
||||
|
||||
assert "Carried" in [a.label for a in result.mandate.approaches], (
|
||||
"a hypothesis the FIRST process found must reach the mandate the second one mints"
|
||||
)
|
||||
assert result.ledger_log[0].instruction_or_question == "keep going", (
|
||||
"the resumed ledger must continue the suspended run's, not restart it"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# 6. THE CLI PARTITION — two doors onto one review, and the preconditions
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_the_two_review_doors_are_refused_together(tmp_path, capsys) -> None:
|
||||
"""T11: ``--plan-review`` and ``--checkpoint-dir`` are two sources of one answer.
|
||||
|
||||
Refused rather than ranked, for the reason ``--explore`` + ``--mandate`` is: silently
|
||||
preferring one would mean an operator who asked for the asynchronous door got the synchronous
|
||||
one, and found out by being blocked at a terminal.
|
||||
"""
|
||||
rc = run.main([*_park_argv(tmp_path), "--plan-review"])
|
||||
|
||||
err = capsys.readouterr().err
|
||||
assert rc == 1
|
||||
assert "--plan-review" in err and "--checkpoint-dir" in err, err
|
||||
|
||||
|
||||
def test_an_asynchronous_door_without_somewhere_to_put_the_question_is_refused_early(
|
||||
tmp_path, capsys
|
||||
) -> None:
|
||||
"""T12: the økt-57 hoist. The question artefact IS the door — without ``--outbox-dir`` and
|
||||
``--run-id`` the exploration would spend a full budget and then have nowhere to say what it
|
||||
was waiting for. Refused before the first model call; the empty checkpoint dir is what proves
|
||||
nothing ran.
|
||||
"""
|
||||
argv = [a for a in _park_argv(tmp_path) if a not in ("--outbox-dir", str(tmp_path / "outbox"))]
|
||||
|
||||
rc = run.main(argv)
|
||||
|
||||
err = capsys.readouterr().err
|
||||
assert rc == 1
|
||||
assert "--outbox-dir" in err and "--checkpoint-dir" in err, err
|
||||
assert not (tmp_path / "checkpoints").exists() or not list(
|
||||
(tmp_path / "checkpoints").iterdir()
|
||||
), "the refusal must fire before the exploration starts, not after it has paid"
|
||||
|
||||
|
||||
def test_a_review_with_neither_door_is_still_refused_and_says_which_two_exist(
|
||||
tmp_path, capsys
|
||||
) -> None:
|
||||
"""T13: the F4 refusal is WIDENED, never weakened.
|
||||
|
||||
``enable_plan_review`` with no reviewer at all is still a hang. The message must now name BOTH
|
||||
doors, because an operator told only about ``--plan-review`` cannot find the asynchronous one.
|
||||
"""
|
||||
argv = [
|
||||
a
|
||||
for a in _park_argv(tmp_path)
|
||||
if a not in ("--checkpoint-dir", str(tmp_path / "checkpoints"))
|
||||
]
|
||||
|
||||
rc = run.main(argv)
|
||||
|
||||
err = capsys.readouterr().err
|
||||
assert rc == 1
|
||||
assert "--plan-review" in err and "--checkpoint-dir" in err, err
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"drop,expected",
|
||||
[
|
||||
(("--checkpoint-dir",), "--checkpoint-dir"),
|
||||
(("--review-inbox",), "--review-inbox"),
|
||||
],
|
||||
)
|
||||
def test_a_resume_names_the_coordinate_it_is_missing(
|
||||
tmp_path, capsys, drop: tuple[str, ...], expected: str
|
||||
) -> None:
|
||||
"""T14: a resume needs the checkpoints AND the inbox. Refused by NAME rather than falling
|
||||
through to a generic message, so an operator is told which of the two to add.
|
||||
"""
|
||||
argv = _resume_argv(tmp_path)
|
||||
for flag in drop:
|
||||
i = argv.index(flag)
|
||||
del argv[i : i + 2]
|
||||
|
||||
rc = run.main(argv)
|
||||
|
||||
err = capsys.readouterr().err
|
||||
assert rc == 1
|
||||
assert expected in err, err
|
||||
|
||||
|
||||
def test_resume_and_explore_are_refused_together(tmp_path, capsys) -> None:
|
||||
"""T15: two sources of one exploration. ``--resume`` continues the exploration recorded in the
|
||||
parked file; ``--explore`` starts a new one. Merging them would silently drop one prompt.
|
||||
"""
|
||||
argv = [
|
||||
*_resume_argv(tmp_path),
|
||||
"--explore",
|
||||
"A different question entirely.",
|
||||
"--explore-config",
|
||||
_config_file(tmp_path),
|
||||
]
|
||||
|
||||
rc = run.main(argv)
|
||||
|
||||
err = capsys.readouterr().err
|
||||
assert rc == 1
|
||||
assert "--resume" in err and "--explore" in err, err
|
||||
327
tests/test_baseline_visibility_loadbearing.py
Normal file
327
tests/test_baseline_visibility_loadbearing.py
Normal file
|
|
@ -0,0 +1,327 @@
|
|||
"""An UN-ANCHORED run says so — in a machine-readable field AND in one line on stdout.
|
||||
|
||||
S4.0 made the deterministic gate anchorable: when a bundle ships ``cost-baseline.json`` the
|
||||
validator's stage 0 reconciles every ``affected_item`` against the project's own cost lines BEFORE
|
||||
the solver, and when it does not, that stage is simply skipped (``None`` = pre-amendment behaviour,
|
||||
which is what keeps every commons-owned golden bundle running). The anchoring stayed OPTIONAL on
|
||||
purpose — and that is not what this file changes.
|
||||
|
||||
What it changes is that the skip was INVISIBLE. Measured (session 48, ``9d149b3``): four
|
||||
``--live-dry-run``s over copies of the veglys bundle — intact rc 0 · without ``validator-input.json``
|
||||
rc 1 · **without ``cost-baseline.json`` rc 0 with no message at all** · corrupt baseline rc 1. And
|
||||
``grep baseline provenance.py outbox.py`` returned 0 hits, so neither the stamp nor the outbox
|
||||
artefacts carried it either. An operator could therefore run the whole gate un-anchored, read a
|
||||
clean rc 0, and have nothing anywhere to tell them the fabrication stage never ran.
|
||||
|
||||
Two teeth, both small:
|
||||
|
||||
1. ``ProvenanceStamp.cost_baseline_anchored`` — a REQUIRED bool, no default. "Was the gate
|
||||
anchored" is a binary fact about a falsifier, in the same class as ``BudgetExceeded``'s
|
||||
``kind``/``limit``/``observed`` (kø-(y)): it must be readable by machine, not inferred from
|
||||
prose. It carries no default because BOTH defaults lie — ``True`` would let a forgetful
|
||||
constructor claim an anchoring that never happened, ``False`` would under-claim a real one — and
|
||||
a binary fact with no honest default is exactly what a required field is for. It reaches the
|
||||
outbox for free: ``outbox.write_proposal`` dumps the whole stamp.
|
||||
2. ``run.cost_baseline_notice`` — ONE renderer, rendering ONE line when the run is un-anchored and
|
||||
``None`` when it is anchored. Omission, not an empty row, mirrors ``mandate.announce``'s rule
|
||||
that a line for something the run does not have is left out rather than rendered blank.
|
||||
|
||||
**The line is rendered from the run's OWN resolution, never from a second read of the bundle**
|
||||
(kø-(p)). ``run_project`` is the single place that calls ``okf.load_optional_cost_baseline`` on the
|
||||
run path; the fact leaves the run as a typed field on ``DryRunReport`` and on ``ProvenanceStamp``,
|
||||
and ``main`` prints from that. Rendering it inside ``mandate.announce`` was MEASURED and rejected:
|
||||
``announce`` fires only when ``--mandate`` is given, so the very runs this file exists for — the
|
||||
four bare dry-runs above, none of which had a mandate — would still have printed nothing.
|
||||
|
||||
Arms:
|
||||
(a) the provenance field is ``False`` on an un-anchored bundle run and ``True`` on an anchored one,
|
||||
end-to-end through ``run_project`` (+ the road path, which is anchored by construction);
|
||||
(b) the notice EXISTS un-anchored and is ABSENT anchored — asserted on a sentinel that the anchored
|
||||
branch cannot contain, because it prints no line at all (never a substring both branches share:
|
||||
the 08-09 class);
|
||||
(c) both CLI surfaces carry it — ``--live-dry-run`` and the full run;
|
||||
(d) the outbox artefact carries the field.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from conftest import SyntheticUsageChatClient
|
||||
|
||||
from portfolio_optimiser import run
|
||||
from portfolio_optimiser.ir import AffectedItem, SavingsProposal
|
||||
from portfolio_optimiser.provenance import Citation, ProvenanceStamp
|
||||
from portfolio_optimiser.retrieval import TextSpan
|
||||
from portfolio_optimiser.run import (
|
||||
DryRunReport,
|
||||
PortfolioResult,
|
||||
RunResult,
|
||||
cost_baseline_notice,
|
||||
run_project,
|
||||
)
|
||||
from portfolio_optimiser.validator import Rejection
|
||||
from portfolio_optimiser.verdicts import ProposalFeatures, Verdict, VerdictStore
|
||||
|
||||
_DATA = Path(__file__).resolve().parents[1] / "src" / "portfolio_optimiser" / "data" / "bundles"
|
||||
#: The ONLY repo-local bundle shipping a ``cost-baseline.json`` (S4.0 fixture) -> anchored.
|
||||
BASELINE_BUNDLE = _DATA / "bygg-energi-baseline-mikro"
|
||||
#: A bundle written before the amendment -> legitimately un-anchored (this is the case under test).
|
||||
PRE_AMENDMENT_BUNDLE = _DATA / "bygg-energi-mikro-a"
|
||||
|
||||
_VERDICT_INPUT = {"decision": "approved", "rationale": "expert reviewed (sim)"}
|
||||
|
||||
#: A reply that reconciles against the S4.0 fixture's own line, so the ANCHORED control run reaches
|
||||
#: a proposal rather than being rejected by stage 0 — the arms below are about visibility, and a
|
||||
#: control that died in the gate would not exercise the stamp.
|
||||
_REPLY = json.dumps(
|
||||
{
|
||||
"measure": "LED-retrofit",
|
||||
"affected_items": [{"code": "ENERGI-TOTAL-EL", "quantity": 180000, "unit_cost": 1.0}],
|
||||
"claimed_saving_nok": 30000,
|
||||
}
|
||||
)
|
||||
|
||||
#: The word the un-anchored line carries and the anchored branch cannot: it prints NO line at all.
|
||||
_SENTINEL = "un-anchored"
|
||||
|
||||
|
||||
def _factory(reply: str = _REPLY):
|
||||
def factory(role: str):
|
||||
return SyntheticUsageChatClient(default_reply=reply)
|
||||
|
||||
return factory
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _isolate_model_env(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Hermetic env (mirrors ``test_scripted_cli_door_loadbearing``): the operator's Foundry
|
||||
overrides must not reach the CLI arms."""
|
||||
monkeypatch.delenv("PORTFOLIO_MODEL_MAP", raising=False)
|
||||
monkeypatch.delenv("PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT", raising=False)
|
||||
|
||||
|
||||
# --- Arm (a): the structured field ----------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_provenance_records_an_unanchored_bundle_run(fresh_store) -> None:
|
||||
"""RED: a bundle with no ``cost-baseline.json`` stamps ``cost_baseline_anchored=False``. Detach
|
||||
the wiring (stamp a constant, or drop the field) and the run again records nothing about the
|
||||
skipped stage."""
|
||||
result = await run_project(
|
||||
"BYGG-ENERGI-MIKRO-A",
|
||||
"local",
|
||||
docs_dir=str(PRE_AMENDMENT_BUNDLE),
|
||||
bundle_dir=str(PRE_AMENDMENT_BUNDLE),
|
||||
verdict_input=_VERDICT_INPUT,
|
||||
client_factory=_factory(),
|
||||
store=fresh_store,
|
||||
)
|
||||
assert result.provenance.cost_baseline_anchored is False
|
||||
|
||||
|
||||
async def test_provenance_records_an_anchored_bundle_run(fresh_store) -> None:
|
||||
"""Causality control: the SAME code path over a bundle that DOES ship a baseline stamps
|
||||
``True``. Without this the arm above would pass on a constant ``False``."""
|
||||
result = await run_project(
|
||||
"BYGG-ENERGI-BASELINE-MIKRO",
|
||||
"local",
|
||||
docs_dir=str(BASELINE_BUNDLE),
|
||||
bundle_dir=str(BASELINE_BUNDLE),
|
||||
verdict_input=_VERDICT_INPUT,
|
||||
client_factory=_factory(),
|
||||
store=fresh_store,
|
||||
)
|
||||
assert result.provenance.cost_baseline_anchored is True
|
||||
|
||||
|
||||
async def test_road_path_is_anchored_by_construction(docs_dir, fresh_store) -> None:
|
||||
"""The road path derives its baseline from the reference project's own ``cost_items``, so it is
|
||||
ALWAYS anchored — the stamp says so rather than leaving the reader to know it."""
|
||||
result = await run_project(
|
||||
"FV42-GSV-E1",
|
||||
"local",
|
||||
docs_dir=docs_dir,
|
||||
verdict_input=_VERDICT_INPUT,
|
||||
client_factory=_factory(
|
||||
json.dumps(
|
||||
{
|
||||
"measure": "Reduce scope",
|
||||
"affected_items": [{"code": "05.2", "quantity": 4300.0, "unit_cost": 215.0}],
|
||||
"claimed_saving_nok": 200000.0,
|
||||
}
|
||||
)
|
||||
),
|
||||
store=fresh_store,
|
||||
)
|
||||
assert result.provenance.cost_baseline_anchored is True
|
||||
|
||||
|
||||
def test_the_field_has_no_default() -> None:
|
||||
"""A binary fact about a falsifier gets no default: both defaults lie (see the module docstring),
|
||||
so a stamp that forgot to say must not construct at all."""
|
||||
with pytest.raises(Exception):
|
||||
ProvenanceStamp( # type: ignore[call-arg]
|
||||
citations=[],
|
||||
model="m",
|
||||
role="proposer",
|
||||
validator_decision="validated",
|
||||
token_usage=0,
|
||||
)
|
||||
|
||||
|
||||
# --- Arm (b): the renderer ------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_notice_is_rendered_only_when_unanchored() -> None:
|
||||
"""One renderer, two branches that share NO wording: un-anchored returns a line carrying the
|
||||
sentinel, anchored returns ``None`` (omitted, never an empty row — ``announce``'s rule)."""
|
||||
unanchored = cost_baseline_notice(False)
|
||||
assert unanchored is not None
|
||||
assert _SENTINEL in unanchored
|
||||
assert cost_baseline_notice(True) is None
|
||||
|
||||
|
||||
def test_dry_run_report_carries_the_anchoring(fresh_store) -> None:
|
||||
"""The dry-run type is the carrier for the surface the order measured: a run that stops before
|
||||
the first model call still knows whether the gate would have been anchored."""
|
||||
assert "cost_baseline_anchored" in DryRunReport.__dataclass_fields__
|
||||
|
||||
|
||||
# --- Arm (c): both CLI surfaces -------------------------------------------------------------------
|
||||
|
||||
|
||||
def _dry_run(bundle: Path, project_id: str) -> list[str]:
|
||||
return [
|
||||
project_id,
|
||||
"--docs-dir",
|
||||
str(bundle),
|
||||
"--bundle-dir",
|
||||
str(bundle),
|
||||
"--live-dry-run",
|
||||
]
|
||||
|
||||
|
||||
def test_cli_dry_run_announces_an_unanchored_bundle(capsys) -> None:
|
||||
"""RED (the measured defect, verbatim): ``--live-dry-run`` over a bundle without
|
||||
``cost-baseline.json`` exits 0 — and now SAYS the gate is un-anchored instead of exiting
|
||||
silently."""
|
||||
rc = run.main(_dry_run(PRE_AMENDMENT_BUNDLE, "BYGG-ENERGI-MIKRO-A"))
|
||||
assert rc == 0
|
||||
assert _SENTINEL in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_cli_dry_run_says_nothing_when_the_bundle_is_anchored(capsys) -> None:
|
||||
"""Control: the anchored bundle prints NO baseline line at all. Lines for what a run does not
|
||||
have are omitted (``announce``); a run that IS anchored has nothing to warn about."""
|
||||
rc = run.main(_dry_run(BASELINE_BUNDLE, "BYGG-ENERGI-BASELINE-MIKRO"))
|
||||
assert rc == 0
|
||||
out = capsys.readouterr().out
|
||||
assert _SENTINEL not in out
|
||||
assert "Cost baseline" not in out
|
||||
|
||||
|
||||
def test_cli_full_run_announces_an_unanchored_bundle(tmp_path, capsys) -> None:
|
||||
"""The full-run surface too, through the offline scripted door — so the notice is a property of
|
||||
a RUN, not of the dry-run branch alone."""
|
||||
replies = tmp_path / "replies.json"
|
||||
replies.write_text(
|
||||
json.dumps({"proposer": _REPLY, "checker": "Holder. VERDICT: APPROVE"}), encoding="utf-8"
|
||||
)
|
||||
rc = run.main(
|
||||
[
|
||||
"BYGG-ENERGI-MIKRO-A",
|
||||
"--docs-dir",
|
||||
str(PRE_AMENDMENT_BUNDLE),
|
||||
"--bundle-dir",
|
||||
str(PRE_AMENDMENT_BUNDLE),
|
||||
"--scripted-replies",
|
||||
str(replies),
|
||||
]
|
||||
)
|
||||
assert rc == 0
|
||||
assert _SENTINEL in capsys.readouterr().out
|
||||
|
||||
|
||||
# --- Arm (d): the outbox artefact -----------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_outbox_proposal_carries_the_anchoring(tmp_path, fresh_store) -> None:
|
||||
"""The outbox needed no change of its own: the artefact dumps the whole stamp, so the field
|
||||
lands in ``{run_id}-proposal.json`` the moment it exists on the stamp."""
|
||||
await run_project(
|
||||
"BYGG-ENERGI-MIKRO-A",
|
||||
"local",
|
||||
docs_dir=str(PRE_AMENDMENT_BUNDLE),
|
||||
bundle_dir=str(PRE_AMENDMENT_BUNDLE),
|
||||
verdict_input=_VERDICT_INPUT,
|
||||
client_factory=_factory(),
|
||||
store=fresh_store,
|
||||
outbox_dir=str(tmp_path),
|
||||
run_id="vis-1",
|
||||
)
|
||||
payload = json.loads((tmp_path / "vis-1-proposal.json").read_text(encoding="utf-8"))
|
||||
assert payload["provenance"]["cost_baseline_anchored"] is False
|
||||
|
||||
|
||||
# --- Arm (e): the portfolio surface (DEFENSIVE, and said out loud) --------------------------------
|
||||
|
||||
|
||||
def _unanchored_run() -> RunResult:
|
||||
"""One ``RunResult`` whose stamp says the gate was un-anchored."""
|
||||
proposal = SavingsProposal(
|
||||
project_id="P",
|
||||
measure="m",
|
||||
affected_items=[AffectedItem(code="05.2", quantity=1.0, unit_cost=1.0)],
|
||||
claimed_saving_nok=1.0,
|
||||
assumptions={},
|
||||
)
|
||||
return RunResult(
|
||||
outcome=Rejection(proposal=proposal, reason="r"),
|
||||
provenance=ProvenanceStamp(
|
||||
citations=[Citation(file="f.md", locator=TextSpan(0, 1), snippet="x")],
|
||||
model="synthetic",
|
||||
role="proposer",
|
||||
validator_decision="rejected",
|
||||
token_usage=1,
|
||||
cost_baseline_anchored=False,
|
||||
),
|
||||
verdict=Verdict(
|
||||
id="v1",
|
||||
proposal_features=ProposalFeatures(
|
||||
affected_codes=frozenset({"05.2"}), measure_type="m", claimed_saving_nok=1.0
|
||||
),
|
||||
decision="rejected",
|
||||
rationale="r",
|
||||
),
|
||||
retrieved=[],
|
||||
store=VerdictStore([]),
|
||||
debate_output="",
|
||||
)
|
||||
|
||||
|
||||
def test_portfolio_surface_announces_an_unanchored_run(monkeypatch, capsys) -> None:
|
||||
"""The portfolio branch reports per project, because anchoring is a per-project fact.
|
||||
|
||||
Driven by a CRAFTED ``PortfolioResult`` (the ``budget_stop`` precedent in
|
||||
``test_portfolio_cli_offline_loadbearing``), and for the same measured reason: no reference
|
||||
project sets ``bundle_dir``, so every portfolio run today takes the road path and is anchored by
|
||||
construction. This arm is therefore DEFENSIVE — it guards the surface for the day a bundle-backed
|
||||
project is wired into a pass, rather than covering a path reachable now."""
|
||||
|
||||
async def _fake(*_args, **_kwargs) -> PortfolioResult:
|
||||
return PortfolioResult(
|
||||
runs=(_unanchored_run(),),
|
||||
store=VerdictStore([]),
|
||||
validated_count=0,
|
||||
rejected_count=1,
|
||||
sum_claimed_saving_nok=0.0,
|
||||
sum_token_usage=1,
|
||||
)
|
||||
|
||||
monkeypatch.setattr(run, "run_portfolio", _fake)
|
||||
rc = run.main(["--portfolio"])
|
||||
assert rc == 0
|
||||
assert _SENTINEL in capsys.readouterr().out
|
||||
183
tests/test_catalogue_cost_loadbearing.py
Normal file
183
tests/test_catalogue_cost_loadbearing.py
Normal file
|
|
@ -0,0 +1,183 @@
|
|||
"""The catalogue call costs O(bases), never O(corpus) — and what it drops, it SAYS it dropped.
|
||||
|
||||
Measured 2026-08-25 (session 60, ``docs/2026-08-25-syretest-vei-ab.md``) and re-measured 26.08 with
|
||||
the same instrument (``tiktoken`` ``o200k_base``, run through ``uv run --with tiktoken``, validated
|
||||
first against the three commons example bundles whose numbers commons itself publishes):
|
||||
|
||||
list_bundles() over the three flat Vegnormal bases -> 201 196 chars / 112 116 tokens
|
||||
list_bundles() over the 171 branch bases -> 234 611 chars / 124 942 tokens
|
||||
|
||||
The branch form (``vegnormal-okf`` ``8145c23``) closed the *bundle* side — ``read_bundle`` fell 82-92
|
||||
percent — and made the *catalogue* side WORSE, exactly as that repo predicted: one call now costs
|
||||
more than a 128k window, before the manager has read a single document.
|
||||
|
||||
The cause is in this repo. ``list_bundles`` returned ``Bundle.index_summary`` — the WHOLE root index
|
||||
body — for EVERY configured base at once, plus one JSON object per unfollowed cross-link. Both grow
|
||||
with the corpus, so the price of *finding out which bases exist* was set by how much those bases
|
||||
contain. That is the opposite of progressive disclosure (målbilde §2/§4): the catalogue is the
|
||||
cheapest rung of the ladder, and it was the most expensive.
|
||||
|
||||
**A MEASURED premise, felled before anything was built on it:** "the index body tells a manager what
|
||||
the base is about" is FALSE for machine-imported bases. The branch bases' ``index.md`` carries no
|
||||
frontmatter and no prose — it is a pure link list (measured: ``B-n200-2024-gren-1-1-importert``,
|
||||
959 bytes, first byte is ``-``). So the old field was not merely expensive, it was expensive AND
|
||||
uninformative there; a truncated prefix loses nothing a manager was using.
|
||||
|
||||
**The ceiling lives in this file, not in ``explore.py``.** A test that imported the implementation's
|
||||
budget would move with it, and raising the budget is precisely the regression this file exists to
|
||||
catch. 500 characters per base is the number, chosen with headroom over the measured entry so that
|
||||
ordinary field growth does not force a rewrite, and small enough that all 171 branch bases fit in
|
||||
well under a tenth of the window they used to need.
|
||||
|
||||
What the arms pin, and what each one refuses:
|
||||
|
||||
(a) the bound itself, over many large bases — refuses the unbounded form;
|
||||
(b) cost does NOT track corpus size — the same base with ten times the index costs the SAME, which
|
||||
is the property "O(bases), not O(corpus)" stated directly rather than inferred from (a);
|
||||
(c) the catalogue still IDENTIFIES what it lists — without this, "return only the ids" passes (a)
|
||||
perfectly and delivers a manager nothing to choose between (the repo's vacuous-gate class);
|
||||
(d) truncation is ANNOUNCED, never silent, and the excerpt is a VERBATIM prefix — validation, never
|
||||
repair (``write_concept_file``'s rule). A base whose index FITS is not marked truncated and gets
|
||||
its whole body: omission, never a lie in either direction;
|
||||
(e) the ladder is intact — ``read_file(id, "index.md")`` still returns the COMPLETE index, so the
|
||||
bound is a disclosure level, not data loss;
|
||||
(f) an unreachable link survives as a COUNT — the fact stays visible (session 51's "a skip is
|
||||
tolerated but no longer silent"), while the unbounded per-link detail does not ride along in a
|
||||
call whose whole job is to be cheap. The detail is still carried where it is actionable, on
|
||||
``RunResult.skipped_links`` / ``DryRunReport.skipped_links``;
|
||||
(g) the CONTROL — the ceiling is proved to discriminate. One base's raw index body alone exceeds the
|
||||
budget for the entire catalogue, so a green (a) means the bound fired, not that the fixture was
|
||||
small (Verifiseringsloven face 4: a gate that can only pass proves nothing).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from portfolio_optimiser.explore import navigator_tools
|
||||
|
||||
#: Characters per base the catalogue may cost. Test-owned on purpose — see the module docstring.
|
||||
_CEILING_CHARS_PER_BASE = 500
|
||||
|
||||
#: Big enough that the old form blew the ceiling by two orders of magnitude (arm (g) measures it).
|
||||
_MANY_CONCEPTS = 300
|
||||
|
||||
|
||||
def _write_base(root: Path, name: str, concepts: int, *, dangling: int = 0) -> str:
|
||||
base = root / name
|
||||
base.mkdir(parents=True)
|
||||
lines = [f"# {name}", "", "Denne basen dekker et avgrenset fagområde.", ""]
|
||||
for i in range(concepts):
|
||||
lines.append(f"- [konsept-{i:04d}](konsept-{i:04d}.md)")
|
||||
(base / f"konsept-{i:04d}.md").write_text(
|
||||
f"---\ntype: concept\n---\n\n# Konsept {i}\n\nInnhold.\n", encoding="utf-8"
|
||||
)
|
||||
for i in range(dangling):
|
||||
lines.append(f"- [borte-{i:04d}](borte-{i:04d}.md)")
|
||||
(base / "index.md").write_text(
|
||||
"---\ntype: index\n---\n\n" + "\n".join(lines) + "\n", encoding="utf-8"
|
||||
)
|
||||
return str(base)
|
||||
|
||||
|
||||
def _catalogue(dirs: list[str]) -> list[dict[str, object]]:
|
||||
tools = {t.name: t for t in navigator_tools(tuple(dirs))}
|
||||
return tools["list_bundles"].func()
|
||||
|
||||
|
||||
def _blob(entries: object) -> str:
|
||||
return json.dumps(entries, ensure_ascii=False)
|
||||
|
||||
|
||||
def test_catalogue_cost_is_bounded_per_base(tmp_path: Path) -> None:
|
||||
"""(a) Many large bases, one call: the payload stays under a per-base ceiling."""
|
||||
dirs = [_write_base(tmp_path, f"base-{n}", _MANY_CONCEPTS) for n in range(5)]
|
||||
|
||||
blob = _blob(_catalogue(dirs))
|
||||
|
||||
assert len(blob) <= len(dirs) * _CEILING_CHARS_PER_BASE, (
|
||||
f"catalogue cost {len(blob)} chars over {len(dirs)} bases exceeds the ceiling "
|
||||
f"{len(dirs) * _CEILING_CHARS_PER_BASE}"
|
||||
)
|
||||
|
||||
|
||||
def test_catalogue_cost_does_not_track_corpus_size(tmp_path: Path) -> None:
|
||||
"""(b) Ten times the index, the same price. O(bases), not O(corpus)."""
|
||||
small = _write_base(tmp_path, "small", 30)
|
||||
large = _write_base(tmp_path, "large", 300)
|
||||
|
||||
entries = {str(e["id"]): e for e in _catalogue([small, large])}
|
||||
small_entry, large_entry = entries["small"], entries["large"]
|
||||
|
||||
# The excerpt is a fixed window, so ten times the index yields the same number of characters.
|
||||
# (The counts beside it grow by a DIGIT, which is honest and logarithmic — this arm is about
|
||||
# the field that used to grow linearly, and asserting on the whole blob would measure that
|
||||
# digit instead of the property.)
|
||||
assert len(str(large_entry["index_excerpt"])) == len(str(small_entry["index_excerpt"]))
|
||||
assert len(_blob(large_entry)) <= _CEILING_CHARS_PER_BASE
|
||||
|
||||
|
||||
def test_catalogue_still_identifies_every_base(tmp_path: Path) -> None:
|
||||
"""(c) The anti-vacuity arm: bounded is not the same as empty."""
|
||||
dirs = [_write_base(tmp_path, f"base-{n}", 50) for n in range(3)]
|
||||
|
||||
entries = _catalogue(dirs)
|
||||
|
||||
assert sorted(str(e["id"]) for e in entries) == ["base-0", "base-1", "base-2"]
|
||||
for entry in entries:
|
||||
excerpt = str(entry["index_excerpt"])
|
||||
assert excerpt.strip(), "a catalogue that says nothing about a base cannot be chosen from"
|
||||
assert str(entry["id"]) in excerpt
|
||||
# How big the base is, is part of choosing one: with the index body no longer riding along,
|
||||
# this count is the only thing left that says what read_bundle would cost.
|
||||
assert entry["documents"] == 50
|
||||
|
||||
|
||||
def test_truncation_is_announced_and_the_excerpt_is_verbatim(tmp_path: Path) -> None:
|
||||
"""(d) Both directions: a cut index SAYS it was cut; a whole one is not marked, and is whole."""
|
||||
from portfolio_optimiser import okf
|
||||
|
||||
large = _write_base(tmp_path, "large", 300)
|
||||
tiny = _write_base(tmp_path, "tiny", 1)
|
||||
|
||||
entries = {str(e["id"]): e for e in _catalogue([large, tiny])}
|
||||
|
||||
assert entries["large"]["index_truncated"] is True
|
||||
body = okf.navigate_bundle(large).index_summary
|
||||
assert body.startswith(str(entries["large"]["index_excerpt"]))
|
||||
assert len(str(entries["large"]["index_excerpt"])) < len(body)
|
||||
|
||||
assert entries["tiny"]["index_truncated"] is False
|
||||
assert str(entries["tiny"]["index_excerpt"]) == okf.navigate_bundle(tiny).index_summary
|
||||
|
||||
|
||||
def test_the_full_index_is_still_one_call_away(tmp_path: Path) -> None:
|
||||
"""(e) The bound is a disclosure LEVEL, not data loss."""
|
||||
large = _write_base(tmp_path, "large", 300)
|
||||
tools = {t.name: t for t in navigator_tools((large,))}
|
||||
|
||||
whole = tools["read_file"].func(bundle_id="large", path="index.md")
|
||||
|
||||
assert whole.count("- [konsept-") == 300
|
||||
|
||||
|
||||
def test_unreachable_links_survive_as_a_count(tmp_path: Path) -> None:
|
||||
"""(f) The fact stays; the unbounded per-link detail does not ride along."""
|
||||
base = _write_base(tmp_path, "holes", 20, dangling=40)
|
||||
|
||||
entry = _catalogue([base])[0]
|
||||
|
||||
assert entry["unreachable_links"] == 40
|
||||
assert len(_blob([entry])) <= _CEILING_CHARS_PER_BASE
|
||||
|
||||
|
||||
def test_control_the_unbounded_form_would_blow_the_ceiling(tmp_path: Path) -> None:
|
||||
"""(g) The ceiling discriminates — proved, not assumed."""
|
||||
from portfolio_optimiser import okf
|
||||
|
||||
large = _write_base(tmp_path, "large", _MANY_CONCEPTS)
|
||||
|
||||
body = okf.navigate_bundle(large).index_summary
|
||||
|
||||
assert len(body) > 5 * _CEILING_CHARS_PER_BASE
|
||||
|
|
@ -52,6 +52,7 @@ _LIVE_DOCS = (
|
|||
"docs/ekspert-svar.md",
|
||||
"docs/bestille-en-kjoring.md",
|
||||
"docs/knowledge-base-recipe.md",
|
||||
"docs/kunnskapsbase-for-en-kjoring.md",
|
||||
)
|
||||
|
||||
# A dated path segment marks a point-in-time record. ``docs/fase1-spikes/`` is the one archive whose
|
||||
|
|
|
|||
766
tests/test_explore_callsites_loadbearing.py
Normal file
766
tests/test_explore_callsites_loadbearing.py
Normal file
|
|
@ -0,0 +1,766 @@
|
|||
"""U4 + U13, part 2 — the CALL SITES. Load-bearing proofs for the seams econ 56 left open.
|
||||
|
||||
Three things are proved here, and each of them is a seam a mutation can detach.
|
||||
|
||||
**1. The trace is a CALLER-OWNED accumulator, for the reason the parse-failure sink is one
|
||||
(Fase 1b, funn 1).** ``explore()`` raises ``BudgetExceeded`` on its round cap, and a token cap
|
||||
fires from inside the middleware mid-run — on both paths ``ExplorationResult`` never returns, so a
|
||||
``ledger_log`` that existed only as a return value would be destroyed by exactly the endings § C.2
|
||||
requires the artefact to be readable after ("så en stoppet utforskning er lesbar uansett hvilken
|
||||
vakt som fyrte"). The accumulator the caller holds survives however the loop ended.
|
||||
``ExplorationResult.ledger_log`` is BUILT FROM that accumulator rather than alongside it: two lists
|
||||
holding one fact is the kø-(p) drift class, one layer up.
|
||||
|
||||
**2. The CLI door refuses everything it cannot honour, by name.** ``--explore`` and ``--mandate``
|
||||
are two sources of ONE mandate and are REFUSED together rather than merged: ``explore()`` sets the
|
||||
objective from the prompt and hardcodes ``allow_own_proposals=True``, so composing them would
|
||||
silently overwrite three fields an operator wrote by hand. The refusal names the library API
|
||||
(``explore(seed_approaches=…)``) because § C.6 door 1 is a real need this surface does not serve.
|
||||
|
||||
**3. ``{run_id}-exploration.json`` is written from a ``finally``**, so the run that most needs the
|
||||
evidence — the one a cap cut short — is the one that has it.
|
||||
|
||||
The client is the repo's own ``ScriptedChatClient`` throughout (a bare ``BaseChatClient`` no-ops
|
||||
``BudgetMiddleware``), and every tool assertion calls the tool's ``func`` DIRECTLY: measured in
|
||||
econ 56, a scripted run returns TEXT and never emits a tool call, so no scripted exploration
|
||||
reaches a tool body and a gate that only drove ``explore()`` would be vacuous.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from agent_framework import BaseChatClient
|
||||
|
||||
import portfolio_optimiser
|
||||
from portfolio_optimiser import explore, hosting, okf, run, simulation
|
||||
from portfolio_optimiser.budget import BudgetExceeded
|
||||
from portfolio_optimiser.explore import ExplorationContract, ExplorationTrace
|
||||
from portfolio_optimiser.mandate import Approach, Mandate
|
||||
from portfolio_optimiser.simulation import ScriptedChatClient
|
||||
|
||||
_BUNDLE_DIR = Path(__file__).resolve().parents[1] / "shared" / "examples" / "bygg-energi-mikro"
|
||||
_PID = "BYGG-KONTOR-NORD"
|
||||
|
||||
#: The hypothesiser's marked line. The label is the marker the end-to-end arm looks for on stdout:
|
||||
#: it appears nowhere in the bundle, in the reference projects or in any other test, so its presence
|
||||
#: in the settlement can only have come through the mandate the exploration shaped.
|
||||
_LABEL = "SENTINEL-EXPLORE-7c1d33"
|
||||
|
||||
_ENERGY_REPLY = (
|
||||
'{"measure":"LED-retrofit av kontorbelysning","affected_items":'
|
||||
'[{"code":"ENERGI-TOTAL-EL","quantity":300000,"unit_cost":1.0}],"claimed_saving_nok":30000}'
|
||||
)
|
||||
|
||||
_CONTRACT_JSON: dict[str, Any] = {
|
||||
"max_rounds": 4,
|
||||
"max_tokens": 100_000,
|
||||
"max_stall_count": 2,
|
||||
"max_reset_count": 1,
|
||||
"max_plan_revisions": 0,
|
||||
"enable_plan_review": False,
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _isolate_model_env(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Hermetic env: no arm here may read the operator's Foundry configuration."""
|
||||
monkeypatch.delenv("PORTFOLIO_MODEL_MAP", raising=False)
|
||||
monkeypatch.delenv("PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT", raising=False)
|
||||
|
||||
|
||||
def _ledger_json(*, satisfied: bool, speaker: str = "hypothesiser") -> str:
|
||||
return json.dumps(
|
||||
{
|
||||
"is_request_satisfied": {"reason": "r", "answer": satisfied},
|
||||
"is_in_loop": {"reason": "r", "answer": False},
|
||||
"is_progress_being_made": {"reason": "r", "answer": True},
|
||||
"next_speaker": {"reason": "r", "answer": speaker},
|
||||
"instruction_or_question": {"reason": "r", "answer": "Shape one hypothesis."},
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _manager_script(ledgers: list[str]) -> Callable[[str, str], str]:
|
||||
"""Route a manager prompt blob to its scripted reply (the econ-56 helper, verbatim in shape).
|
||||
|
||||
The stage ORDER is load-bearing (§ F, A6): the selector sees the CONCATENATION of the call's
|
||||
messages, so a later-stage prompt still carries the earlier stage's text.
|
||||
"""
|
||||
|
||||
def _select(blob: str, _role: str) -> str:
|
||||
if "provide the final answer" in blob:
|
||||
return "FINAL: exploration done."
|
||||
if "pure JSON format" in blob:
|
||||
return ledgers.pop(0) if ledgers else _ledger_json(satisfied=True)
|
||||
if "went wrong on this last run" in blob:
|
||||
return "PLAN-UPDATE: revised plan."
|
||||
if "rewrite the following fact sheet" in blob:
|
||||
return "FACTS-UPDATE: revised facts."
|
||||
if "bullet-point plan" in blob:
|
||||
return "PLAN: - ask the hypothesiser"
|
||||
if "pre-survey" in blob:
|
||||
return "FACTS: the bundle is anchored."
|
||||
return "{}"
|
||||
|
||||
return _select
|
||||
|
||||
|
||||
def _hypothesis_line(label: str, rationale: str) -> str:
|
||||
return f"{explore.HYPOTHESIS_MARKER} " + json.dumps({"label": label, "rationale": rationale})
|
||||
|
||||
|
||||
def _factory(
|
||||
*,
|
||||
ledgers: list[str],
|
||||
hypothesiser: list[str],
|
||||
fallback: str = "ok",
|
||||
sink: list[str] | None = None,
|
||||
) -> Callable[[str], BaseChatClient]:
|
||||
"""One fresh ``ScriptedChatClient`` per role — the exploration's three plus everyone else.
|
||||
|
||||
``fallback`` serves the roles the PIPELINE builds (proposer/checker), so one factory can drive
|
||||
an exploration and the run it hands its mandate to. That is what makes the end-to-end arm an
|
||||
end-to-end arm rather than two half-proofs.
|
||||
"""
|
||||
|
||||
def factory(role: str) -> BaseChatClient:
|
||||
if role == explore.MANAGER_ROLE:
|
||||
return ScriptedChatClient(sink=sink, reply_selector=_manager_script(ledgers), role=role)
|
||||
if role == explore.HYPOTHESISER_ROLE:
|
||||
replies = list(hypothesiser)
|
||||
|
||||
def _hyp(_blob: str, _role: str) -> str:
|
||||
return replies.pop(0) if replies else "nothing further."
|
||||
|
||||
return ScriptedChatClient(sink=sink, reply_selector=_hyp, role=role)
|
||||
if role == explore.NAVIGATOR_ROLE:
|
||||
return ScriptedChatClient("NAVIGATOR: index read.", sink, role=role)
|
||||
return ScriptedChatClient(fallback, sink, role=role)
|
||||
|
||||
return factory
|
||||
|
||||
|
||||
def _contract(**overrides: Any) -> ExplorationContract:
|
||||
return ExplorationContract(**{**_CONTRACT_JSON, **overrides})
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# 1. The caller-owned accumulator (the funn-1 sink shape, applied to the exploration)
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _micro_bundle_dir() -> str:
|
||||
return str(
|
||||
Path(portfolio_optimiser.__file__).parent
|
||||
/ "data"
|
||||
/ "bundles"
|
||||
/ "bygg-energi-baseline-mikro"
|
||||
)
|
||||
|
||||
|
||||
def test_quick_validate_verdicts_reach_the_callers_trace() -> None:
|
||||
"""T1: every advisory verdict the hypothesiser asked for is recorded where the caller can read
|
||||
it — the ONE thing ``ExplorationResult`` deliberately does not carry.
|
||||
|
||||
Called DIRECTLY, because a scripted run never reaches a tool body (measured, econ 56): an arm
|
||||
that drove ``explore()`` and then asserted on an empty list would be green against every
|
||||
implementation, including one with no sink at all.
|
||||
|
||||
Detach point: drop the ``sink`` append in ``quick_validate_tool`` → RED.
|
||||
"""
|
||||
base = _micro_bundle_dir()
|
||||
projection = dict(okf.load_ir_projection(base))
|
||||
projection.pop("_note", None)
|
||||
|
||||
trace = ExplorationTrace()
|
||||
validate = explore.quick_validate_tool((base,), sink=trace.quick_validations)
|
||||
|
||||
honest = validate.func(
|
||||
bundle_id="bygg-energi-baseline-mikro", proposal_json=json.dumps(projection)
|
||||
)
|
||||
invented = dict(projection)
|
||||
invented["affected_items"] = [
|
||||
{**dict(projection["affected_items"][0]), "code": "CODE-THAT-DOES-NOT-EXIST"}
|
||||
]
|
||||
validate.func(bundle_id="bygg-energi-baseline-mikro", proposal_json=json.dumps(invented))
|
||||
|
||||
assert len(trace.quick_validations) == 2, "both calls must be recorded, in call order"
|
||||
first, second = trace.quick_validations
|
||||
assert first.bundle_id == "bygg-energi-baseline-mikro"
|
||||
assert first.verdict == honest, "the recorded verdict must be the one the tool ANSWERED"
|
||||
assert second.verdict["decision"] == "rejected"
|
||||
assert "CODE-THAT-DOES-NOT-EXIST" in second.proposal_json
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_returned_ledger_log_is_the_traces_own_entries() -> None:
|
||||
"""T2: ``ExplorationResult.ledger_log`` is BUILT FROM the accumulator, never alongside it.
|
||||
|
||||
Two lists holding one fact drift (kø-(p)), and a drifted pair would let the returned result and
|
||||
the written artefact describe different runs.
|
||||
|
||||
Detach point: accumulate rounds in a second local list → RED.
|
||||
"""
|
||||
trace = ExplorationTrace()
|
||||
result = await explore.explore(
|
||||
"Find a saving.",
|
||||
contract=_contract(),
|
||||
bundle_dirs=(str(_BUNDLE_DIR),),
|
||||
client_factory=_factory(
|
||||
ledgers=[_ledger_json(satisfied=False), _ledger_json(satisfied=True)],
|
||||
hypothesiser=[_hypothesis_line(_LABEL, "because the bundle says so")],
|
||||
),
|
||||
trace=trace,
|
||||
)
|
||||
|
||||
assert result.stop is None
|
||||
assert len(trace.ledger) == 2
|
||||
assert tuple(trace.ledger) == result.ledger_log
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_trace_survives_the_budget_exception_that_destroys_the_result() -> None:
|
||||
"""T3: the round cap raises, and the caller STILL holds every round the loop recorded.
|
||||
|
||||
This is the whole reason the accumulator is caller-owned. The round cap leaves as a typed
|
||||
``BudgetExceeded`` (econ 56), so nothing is returned — and § C.2 requires the artefact to be
|
||||
readable no matter which guard fired.
|
||||
|
||||
Detach point: return the log only, keeping no caller-visible accumulator → RED.
|
||||
"""
|
||||
trace = ExplorationTrace()
|
||||
with pytest.raises(BudgetExceeded) as excinfo:
|
||||
await explore.explore(
|
||||
"Find a saving.",
|
||||
contract=_contract(max_rounds=2),
|
||||
bundle_dirs=(str(_BUNDLE_DIR),),
|
||||
client_factory=_factory(
|
||||
ledgers=[_ledger_json(satisfied=False), _ledger_json(satisfied=False)],
|
||||
hypothesiser=["still thinking."],
|
||||
),
|
||||
trace=trace,
|
||||
)
|
||||
|
||||
assert excinfo.value.kind == "exploration_rounds"
|
||||
assert len(trace.ledger) == 2, (
|
||||
"the rounds the exploration DID record were destroyed with the result — the artefact a "
|
||||
"capped run needs most would be empty"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# 2. The CLI door — every refusal by name, never a silent merge or a silent drop
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _config_file(tmp_path: Path, **overrides: Any) -> str:
|
||||
path = tmp_path / "exploration.json"
|
||||
path.write_text(json.dumps({**_CONTRACT_JSON, **overrides}), encoding="utf-8")
|
||||
return str(path)
|
||||
|
||||
|
||||
def _mandate_file(tmp_path: Path) -> str:
|
||||
path = tmp_path / "mandate.json"
|
||||
path.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"objective": "cut energy cost",
|
||||
"approaches": [{"id": "a1", "label": "LED", "description": "swap the fittings"}],
|
||||
"allow_own_proposals": False,
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
return str(path)
|
||||
|
||||
|
||||
def _base_argv(tmp_path: Path) -> list[str]:
|
||||
return [
|
||||
_PID,
|
||||
"--docs-dir",
|
||||
str(_BUNDLE_DIR),
|
||||
"--bundle-dir",
|
||||
str(_BUNDLE_DIR),
|
||||
"--explore",
|
||||
"Find the cheapest saving.",
|
||||
"--explore-config",
|
||||
_config_file(tmp_path),
|
||||
]
|
||||
|
||||
|
||||
def test_an_exploration_config_without_an_exploration_is_refused_by_name(tmp_path, capsys) -> None:
|
||||
"""T4: ``--explore-config`` alone would be loaded and then dropped on the floor — the exact
|
||||
silent-ignore ``--embedder-config requires --semantic-retrieval`` exists to prevent.
|
||||
|
||||
Detach point: drop the refusal → RED.
|
||||
"""
|
||||
rc = run.main(
|
||||
[_PID, "--docs-dir", str(_BUNDLE_DIR), "--explore-config", _config_file(tmp_path)]
|
||||
)
|
||||
assert rc == 1
|
||||
assert "--explore-config" in capsys.readouterr().err
|
||||
|
||||
|
||||
def test_an_exploration_without_its_bounds_is_refused_rather_than_defaulted(
|
||||
tmp_path, capsys
|
||||
) -> None:
|
||||
"""T5: ``--explore`` alone is refused — the CLI may not invent bounds.
|
||||
|
||||
Every ``ExplorationContract`` field is required WITHOUT a default precisely because
|
||||
``MagenticBuilder`` falls back to unbounded, and a CLI that supplied its own numbers would undo
|
||||
that decision one layer up.
|
||||
"""
|
||||
rc = run.main(
|
||||
[_PID, "--docs-dir", str(_BUNDLE_DIR), "--bundle-dir", str(_BUNDLE_DIR), "--explore", "go"]
|
||||
)
|
||||
assert rc == 1
|
||||
assert "--explore-config" in capsys.readouterr().err
|
||||
|
||||
|
||||
def test_explore_and_mandate_are_two_sources_of_one_mandate_and_are_refused_together(
|
||||
tmp_path, capsys
|
||||
) -> None:
|
||||
"""T6: the decision, made deliberately and stated: REFUSE, never merge.
|
||||
|
||||
``explore()`` takes the objective from the prompt and hardcodes ``allow_own_proposals=True``, so
|
||||
composing the two would silently overwrite fields the operator wrote by hand. The message names
|
||||
the library door (``seed_approaches``) so the refusal teaches instead of only forbidding.
|
||||
|
||||
Detach point: let one source silently win → RED.
|
||||
"""
|
||||
rc = run.main(_base_argv(tmp_path) + ["--mandate", _mandate_file(tmp_path)])
|
||||
assert rc == 1
|
||||
err = capsys.readouterr().err
|
||||
assert "--explore" in err and "--mandate" in err
|
||||
assert "seed_approaches" in err, "the refusal must name the door that DOES serve door 1"
|
||||
|
||||
|
||||
def test_explore_and_live_dry_run_contradict_and_are_refused(tmp_path, capsys) -> None:
|
||||
"""T7: ``--live-dry-run`` stops before the first model call; an exploration IS model calls."""
|
||||
rc = run.main(_base_argv(tmp_path) + ["--live-dry-run"])
|
||||
assert rc == 1
|
||||
assert "--live-dry-run" in capsys.readouterr().err
|
||||
|
||||
|
||||
def test_an_exploration_with_no_knowledge_base_is_refused(tmp_path, capsys) -> None:
|
||||
"""T8: without ``--bundle-dir`` the navigator has nothing to open — the loop would run, cost
|
||||
tokens and read nothing. Refused rather than run empty (the ``--semantic-retrieval`` shape).
|
||||
|
||||
Detach point: drop the requirement → RED.
|
||||
"""
|
||||
rc = run.main(
|
||||
[
|
||||
_PID,
|
||||
"--docs-dir",
|
||||
str(_BUNDLE_DIR),
|
||||
"--explore",
|
||||
"go",
|
||||
"--explore-config",
|
||||
_config_file(tmp_path),
|
||||
]
|
||||
)
|
||||
assert rc == 1
|
||||
assert "--bundle-dir" in capsys.readouterr().err
|
||||
|
||||
|
||||
def test_a_plan_review_nobody_can_answer_is_refused_at_the_cli(tmp_path, capsys) -> None:
|
||||
"""T9: ``enable_plan_review`` is the U13 SYNCHRONOUS door, and a run must never stop at a
|
||||
review nobody offered to answer.
|
||||
|
||||
Refused HERE rather than left to ``explore()``: ``ExplorationError`` is a ``RuntimeError``, so
|
||||
it is outside ``main()``'s ``(ValueError, FileNotFoundError, ValidationError)`` refusal tuple
|
||||
and would leave as a traceback instead of the rc-1 line every other misconfiguration produces.
|
||||
|
||||
**The assertion names wording unique to THIS branch.** Since F4 the CLI has a second refusal
|
||||
carrying ``enable_plan_review`` (``--plan-review`` against a config that asks for no review),
|
||||
so asserting on the shared token would pass against a surface missing this branch entirely —
|
||||
the økt-57 mutation, in the form this repo keeps meeting it.
|
||||
|
||||
Detach point: let the flag through to ``explore()`` → RED (traceback, not rc 1).
|
||||
"""
|
||||
rc = run.main(
|
||||
_base_argv(tmp_path)[:-1] + [_config_file(tmp_path, enable_plan_review=True)],
|
||||
)
|
||||
assert rc == 1
|
||||
err = capsys.readouterr().err
|
||||
assert "no reviewer was offered" in err
|
||||
assert "--plan-review" in err, "the refusal must name the door that answers it (F4)"
|
||||
|
||||
|
||||
def test_explore_belongs_to_single_project_mode(tmp_path, capsys) -> None:
|
||||
"""T10: portfolio mode is a documented partition, and ``--explore`` is on the single-project
|
||||
side of it — one exploration shapes ONE mandate against ONE knowledge base.
|
||||
|
||||
The assertion names ``--portfolio``, and that was MEASURED rather than chosen: asserting only
|
||||
that the message mentions ``--explore`` passed against an implementation with no partition
|
||||
entry at all, because the run then fell through to ``--explore requires --bundle-dir``, which
|
||||
names ``--explore`` too. Two refusals sharing a substring is this repo's "assert never on
|
||||
wording two branches share" rule, caught by its own mutation.
|
||||
|
||||
Detach point: drop ``--explore`` from the portfolio ``single_only`` partition → RED.
|
||||
"""
|
||||
rc = run.main(["--portfolio", "--explore", "go", "--explore-config", _config_file(tmp_path)])
|
||||
assert rc == 1
|
||||
err = capsys.readouterr().err
|
||||
assert "--explore" in err and "--portfolio" in err
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def _explored_main(monkeypatch: pytest.MonkeyPatch) -> list[str]:
|
||||
"""Inject the role-dispatching scripted factory into the seam ``main()`` resolves through.
|
||||
|
||||
``main()`` passes no ``client_factory``, and ``explore()`` imports ``run._default_factory``
|
||||
lazily at call time, so this ONE patch covers both the exploration and the pipeline it feeds —
|
||||
which is what makes the arm below end-to-end rather than a wiring spy.
|
||||
"""
|
||||
sink: list[str] = []
|
||||
factory = _factory(
|
||||
ledgers=[_ledger_json(satisfied=False), _ledger_json(satisfied=True)],
|
||||
hypothesiser=[_hypothesis_line(_LABEL, "the index says the fittings are old")],
|
||||
fallback=_ENERGY_REPLY,
|
||||
sink=sink,
|
||||
)
|
||||
monkeypatch.setattr("portfolio_optimiser.run._default_factory", lambda profile: factory)
|
||||
return sink
|
||||
|
||||
|
||||
def test_the_shaped_mandate_reaches_the_pipeline(tmp_path, capsys, _explored_main) -> None:
|
||||
"""T11: the approach the hypothesiser shaped is SETTLED by the run — the whole point of (1).
|
||||
|
||||
The settlement is printed only for a run that HAS a mandate, and the label appears nowhere in
|
||||
the bundle or the reference projects, so it can have reached stdout only by travelling
|
||||
prompt → ``explore()`` → ``Mandate`` → ``run_project(mandate=…)`` → ``settle``.
|
||||
|
||||
Detach point: drop ``mandate=`` from the exploring branch's ``run_project`` call → RED.
|
||||
"""
|
||||
rc = run.main(_base_argv(tmp_path))
|
||||
assert rc == 0
|
||||
out = capsys.readouterr().out
|
||||
assert _LABEL in out, "the exploration's mandate never reached the pipeline's settlement"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# 3. The artefact — written from a ``finally``, because a capped run is what it exists for
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_the_exploration_artefact_carries_the_rounds_and_the_advisory_verdicts(
|
||||
tmp_path, _explored_main
|
||||
) -> None:
|
||||
"""T12: ``{run_id}-exploration.json`` holds the per-round ledger AND the ``quick_validate``
|
||||
verdicts — the level-1 evidence ``ExplorationResult`` deliberately does not carry (§ C.2).
|
||||
|
||||
Detach point: drop the artefact write → RED.
|
||||
"""
|
||||
outbox = tmp_path / "outbox"
|
||||
rc = run.main(_base_argv(tmp_path) + ["--outbox-dir", str(outbox), "--run-id", "r1"])
|
||||
assert rc == 0
|
||||
|
||||
payload = json.loads((outbox / "r1-exploration.json").read_text(encoding="utf-8"))
|
||||
assert payload["run_id"] == "r1"
|
||||
assert payload["completed"] is True
|
||||
assert payload["stop"] is None
|
||||
assert [row["round_index"] for row in payload["rounds"]] == [1, 2]
|
||||
assert payload["rounds"][-1]["is_request_satisfied"] is True
|
||||
assert payload["rounds"][0]["next_speaker"] == "hypothesiser"
|
||||
assert "quick_validations" in payload
|
||||
|
||||
|
||||
def test_the_artefact_is_written_even_when_the_exploration_was_cut_short(
|
||||
tmp_path, monkeypatch
|
||||
) -> None:
|
||||
"""T13: a capped exploration is the run whose evidence matters MOST, and it is the one that
|
||||
returns nothing — so the write lives in a ``finally`` (the ``write_parse_failures`` precedent).
|
||||
|
||||
``completed`` is a field rather than an inference: with no result there is no ``stop``, and a
|
||||
``stop: null`` that meant BOTH "concluded normally" and "we never found out" would be the kind
|
||||
of silence this repo writes required fields to close.
|
||||
|
||||
Detach point: move the write out of the ``finally`` → RED.
|
||||
"""
|
||||
factory = _factory(
|
||||
ledgers=[_ledger_json(satisfied=False), _ledger_json(satisfied=False)],
|
||||
hypothesiser=["still thinking."],
|
||||
fallback=_ENERGY_REPLY,
|
||||
)
|
||||
monkeypatch.setattr("portfolio_optimiser.run._default_factory", lambda profile: factory)
|
||||
|
||||
outbox = tmp_path / "outbox"
|
||||
with pytest.raises(BudgetExceeded):
|
||||
run.main(
|
||||
[
|
||||
_PID,
|
||||
"--docs-dir",
|
||||
str(_BUNDLE_DIR),
|
||||
"--bundle-dir",
|
||||
str(_BUNDLE_DIR),
|
||||
"--explore",
|
||||
"go",
|
||||
"--explore-config",
|
||||
_config_file(tmp_path, max_rounds=2),
|
||||
"--outbox-dir",
|
||||
str(outbox),
|
||||
"--run-id",
|
||||
"r2",
|
||||
]
|
||||
)
|
||||
|
||||
payload = json.loads((outbox / "r2-exploration.json").read_text(encoding="utf-8"))
|
||||
assert payload["completed"] is False
|
||||
assert payload["stop"] is None
|
||||
assert len(payload["rounds"]) == 2
|
||||
|
||||
|
||||
def test_the_artefact_payload_is_byte_deterministic() -> None:
|
||||
"""T14 (control): the same trace renders the same bytes, so the artefact is diff-stable like
|
||||
every other outbox file. Drives the renderer directly — the CLI arms above prove it is CALLED,
|
||||
this proves what it produces."""
|
||||
trace = ExplorationTrace()
|
||||
trace.ledger.append(
|
||||
explore.LedgerEntry(
|
||||
round_index=1,
|
||||
is_request_satisfied=True,
|
||||
is_in_loop=False,
|
||||
is_progress_being_made=True,
|
||||
next_speaker="hypothesiser",
|
||||
instruction_or_question="Shape one hypothesis.",
|
||||
speaker_known=True,
|
||||
)
|
||||
)
|
||||
trace.quick_validations.append(
|
||||
explore.QuickValidation(
|
||||
bundle_id="b", proposal_json="{}", verdict={"decision": "unparseable"}
|
||||
)
|
||||
)
|
||||
first = explore.trace_payload(trace, stop=None, completed=True)
|
||||
second = explore.trace_payload(trace, stop=None, completed=True)
|
||||
assert json.dumps(first, sort_keys=True) == json.dumps(second, sort_keys=True)
|
||||
|
||||
|
||||
def test_a_seeded_mandate_still_leads_the_shaped_one() -> None:
|
||||
"""T15 (control for T6's refusal): the library door the refusal names actually works.
|
||||
|
||||
A refusal that pointed at a door which did not open would be worse than no message at all.
|
||||
"""
|
||||
seed = Approach(id="expert-1", label="expert's own", description="the domain expert asked")
|
||||
minted = explore._mint_approaches((seed,), [(_LABEL, "shaped in the loop", "")])
|
||||
assert [a.id for a in minted] == ["expert-1", "hypothesis-1"]
|
||||
assert isinstance(Mandate(objective="o", approaches=minted, allow_own_proposals=True), Mandate)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# 4. The hosted surface — a THREE-way whitelist, and the Fase 4e rule extended to cover it
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _hosted_payload(**extra: Any) -> dict[str, Any]:
|
||||
return {
|
||||
"project_id": _PID,
|
||||
"docs_dir": str(_BUNDLE_DIR),
|
||||
"verdict_input": {"decision": "approved", "rationale": "expert reviewed (explore)"},
|
||||
# LOCAL, never the hosted AZURE default: the AZURE arm resolves a Foundry deployment name
|
||||
# from the model map before any client is built, so it cannot complete offline.
|
||||
"profile": "local",
|
||||
**extra,
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def _hosted_backend(monkeypatch: pytest.MonkeyPatch) -> list[str]:
|
||||
"""The scripted backend behind a hosted invocation, plus the prompt sink that proves it ran.
|
||||
|
||||
``client_factory`` is refused by the invocations whitelist on purpose — the caller of a hosted
|
||||
agent never chooses the server's model client — so ``run._default_factory`` is the only
|
||||
injection point the surface leaves, and ``explore()`` resolves through the same one.
|
||||
"""
|
||||
sink: list[str] = []
|
||||
factory = _factory(
|
||||
ledgers=[_ledger_json(satisfied=False), _ledger_json(satisfied=True)],
|
||||
hypothesiser=[_hypothesis_line(_LABEL, "the index says the fittings are old")],
|
||||
fallback=_ENERGY_REPLY,
|
||||
sink=sink,
|
||||
)
|
||||
monkeypatch.setattr("portfolio_optimiser.run._default_factory", lambda profile: factory)
|
||||
return sink
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_hosted_exploration_shapes_the_mandate_the_run_evaluates(_hosted_backend) -> None:
|
||||
"""H1: ``explore_prompt`` over the hosted surface reaches the pipeline as a mandate.
|
||||
|
||||
Driven through ``hosting.invoke`` and the REAL ``run_project`` (Fase 4e): every other
|
||||
invocations test hands ``invoke`` a recorder that swallows ``**kwargs`` and therefore cannot
|
||||
see whether a new field composes with the signature at all.
|
||||
|
||||
The proof is the PROMPT, not the status code: an ``Approach``'s description reaches the
|
||||
proposer VERBATIM, and this label exists nowhere in the bundle or the reference projects — so
|
||||
finding it in a generation prompt means it travelled prompt → ``explore()`` → ``Mandate`` →
|
||||
``run_project(mandate=…)``.
|
||||
|
||||
Detach point: stop passing the shaped mandate into ``run_project`` → RED.
|
||||
"""
|
||||
body = await hosting.invoke(
|
||||
_hosted_payload(
|
||||
bundle_dir=str(_BUNDLE_DIR),
|
||||
explore_prompt="Find the cheapest saving.",
|
||||
explore_contract=dict(_CONTRACT_JSON),
|
||||
)
|
||||
)
|
||||
|
||||
assert body["outcome_type"] in {"validated", "rejected"}
|
||||
assert any(_LABEL in prompt for prompt in _hosted_backend), (
|
||||
"the shaped approach never reached a prompt — the hosted door does not wire the mandate"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_consumed_field_is_never_forwarded_to_run_project() -> None:
|
||||
"""H2: the whitelist is a THREE-way partition, and the consumed half is proved NEGATIVELY.
|
||||
|
||||
``explore_prompt``/``explore_contract`` are accepted by the surface and consumed BY it — they
|
||||
are not ``run_project`` parameters, and forwarding one would be a ``TypeError`` answered as a
|
||||
500. The positive half of Fase 4e (every forwarded field reaches the real signature) cannot
|
||||
see that; without this arm a field sliding from consumed to forwarded is exactly the drift 4e
|
||||
exists to catch.
|
||||
|
||||
Detach point: build ``kwargs`` from the whole payload again → RED.
|
||||
"""
|
||||
import inspect
|
||||
|
||||
_, kwargs, consumed = hosting._run_kwargs(
|
||||
_hosted_payload(
|
||||
bundle_dir=str(_BUNDLE_DIR),
|
||||
explore_prompt="p",
|
||||
explore_contract=dict(_CONTRACT_JSON),
|
||||
)
|
||||
)
|
||||
assert set(hosting._CONSUMED_FIELDS).isdisjoint(kwargs), (
|
||||
"a consumed field was forwarded to run_project, which does not take it"
|
||||
)
|
||||
assert set(consumed) == set(hosting._CONSUMED_FIELDS)
|
||||
|
||||
parameters = inspect.signature(run.run_project).parameters
|
||||
assert set(hosting._CONSUMED_FIELDS).isdisjoint(parameters), (
|
||||
"a CONSUMED field is a run_project parameter — it belongs in the forwarded half"
|
||||
)
|
||||
for name in (*hosting._REQUIRED_FIELDS, *hosting._OPTIONAL_FIELDS):
|
||||
# project_id is positional; every other forwarded field must be a real keyword.
|
||||
assert name in parameters, f"whitelisted field {name!r} is not a run_project parameter"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
("payload", "expected"),
|
||||
[
|
||||
pytest.param(
|
||||
{"bundle_dir": str(_BUNDLE_DIR), "explore_contract": dict(_CONTRACT_JSON)},
|
||||
"explore_prompt",
|
||||
id="bounds-without-an-exploration",
|
||||
),
|
||||
pytest.param(
|
||||
{"bundle_dir": str(_BUNDLE_DIR), "explore_prompt": "p"},
|
||||
"explore_contract",
|
||||
id="exploration-without-bounds",
|
||||
),
|
||||
pytest.param(
|
||||
{"explore_prompt": "p", "explore_contract": dict(_CONTRACT_JSON)},
|
||||
"bundle_dir",
|
||||
id="exploration-without-a-knowledge-base",
|
||||
),
|
||||
pytest.param(
|
||||
{
|
||||
"bundle_dir": str(_BUNDLE_DIR),
|
||||
"explore_prompt": "p",
|
||||
"explore_contract": {**_CONTRACT_JSON, "enable_plan_review": True},
|
||||
},
|
||||
"enable_plan_review",
|
||||
id="a-review-nobody-can-answer",
|
||||
),
|
||||
],
|
||||
)
|
||||
async def test_the_hosted_door_refuses_by_name_on_the_callers_channel(payload, expected) -> None:
|
||||
"""H3: each hosted refusal names the field, and each is a ``ValueError`` — the 400 arm.
|
||||
|
||||
``enable_plan_review`` is the one that had to be refused HERE rather than in ``explore()``:
|
||||
``ExplorationError`` is a ``RuntimeError``, so leaving it to the loop would answer a caller's
|
||||
configuration mistake on the crash channel (500), which is where a fallen-over endpoint lives.
|
||||
A synchronous plan review would also block the HTTP request on a reviewer that does not exist.
|
||||
|
||||
Detach point: drop any one of the four guards → RED.
|
||||
"""
|
||||
with pytest.raises(ValueError) as excinfo:
|
||||
await hosting.invoke(_hosted_payload(**payload))
|
||||
assert expected in str(excinfo.value)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# 5. The demo scenario — a THIRD entry, reachable only by name
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_demo_scenario_lets_a_shaped_direction_reach_the_hypothesis(tmp_path) -> None:
|
||||
"""S1: the offline walkthrough of U4 — a prompt and a knowledge base become a mandate, and the
|
||||
direction the loop shaped reaches the proposer VERBATIM.
|
||||
|
||||
The same honesty limit the rest of the demo carries applies here and is worth restating: this
|
||||
proves the plumbing and that the data flow closes, NOT that a live model would shape a good
|
||||
direction. Every reply is scripted.
|
||||
|
||||
Detach point: drop ``mandate=`` from the scenario's ``run_project`` call → RED.
|
||||
"""
|
||||
result = await simulation.simulate_exploration(str(_BUNDLE_DIR), str(tmp_path), max_rounds=3)
|
||||
|
||||
assert [a.label for a in result.exploration.mandate.approaches] == [result.label]
|
||||
assert result.label_in_generation_prompt, (
|
||||
"the shaped direction never reached the hypothesis prompt — the demo would show a mandate "
|
||||
"the pipeline ignored"
|
||||
)
|
||||
assert result.trace.ledger, "the exploration recorded no rounds"
|
||||
|
||||
|
||||
def test_an_outbox_without_a_run_id_is_refused_before_the_exploration_spends_anything(
|
||||
tmp_path, _explored_main
|
||||
) -> None:
|
||||
"""T16: an argv that cannot finish is refused BEFORE the loop costs anything.
|
||||
|
||||
``run_project`` refuses ``outbox_dir`` without ``run_id`` at its very first statement, which is
|
||||
early enough for every path that existed before U4. The exploration runs AHEAD of that call, so
|
||||
without this guard the run spends its whole exploration budget on model calls and only then
|
||||
refuses — and the artefact write is skipped too, so not even the evidence of what was spent
|
||||
survives. Exactly the hoist ``main()`` already performs twice ("an incomplete argv is refused
|
||||
BEFORE the honesty banner could claim a scripted run happened").
|
||||
|
||||
The assertion is that NO model call happened, not merely that rc is 1: a refusal that arrives
|
||||
after the spend looks identical at the exit code.
|
||||
|
||||
Detach point: drop the guard from the exploration block → RED.
|
||||
"""
|
||||
rc = run.main(_base_argv(tmp_path) + ["--outbox-dir", str(tmp_path / "outbox")])
|
||||
assert rc == 1
|
||||
assert not _explored_main, (
|
||||
"the exploration made model calls before the run was refused — the budget was spent on an "
|
||||
"argv that could never finish"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_direction_the_base_already_states_is_refused_as_vacuous(tmp_path) -> None:
|
||||
"""S2: a label the knowledge base ALREADY contains is refused, not demonstrated.
|
||||
|
||||
Exactly the guard ``simulate_learning_loop`` raises on when its two markers coincide: the
|
||||
scenario's whole claim is that the direction came from the LOOP, and a label the bundle states
|
||||
on its own would reach the prompt as ordinary context — a demonstration that demonstrates
|
||||
nothing, which is this repo's vacuous-gate class in demo form.
|
||||
|
||||
Detach point: drop the guard → RED.
|
||||
"""
|
||||
stated = "LED-retrofit" # present in the bundle's own text
|
||||
with pytest.raises(ValueError) as excinfo:
|
||||
await simulation.simulate_exploration(str(_BUNDLE_DIR), str(tmp_path), label=stated)
|
||||
assert stated in str(excinfo.value)
|
||||
992
tests/test_explore_loadbearing.py
Normal file
992
tests/test_explore_loadbearing.py
Normal file
|
|
@ -0,0 +1,992 @@
|
|||
"""U4 + U13-synchronous (økt 56) — the Magentic exploration loop as a MANDATE-FORMER.
|
||||
|
||||
**What this loop is, and what it deliberately is not.** ``explore()`` puts a Magentic manager
|
||||
*over* the normative pipeline, never inside it: the manager is free to choose which knowledge base
|
||||
to open and which hypothesis to shape next, and what leaves that freedom is a
|
||||
``mandate.Mandate`` — a list of approaches worth *testing*. It is never a proposal. Every number
|
||||
that survives is still gated by ``validate_proposal`` inside ``run_project``, in the same blocking
|
||||
gate as today, and the exploration itself can write to neither the outbox nor the wiki. Step 3's
|
||||
maker-checker debate is untouched (``shared/method-spec.md`` §3 is commons-owned and normative).
|
||||
|
||||
**Everything asserted here was measured before it was built** (plan
|
||||
``docs/plan/2026-08-23-magentic-utforskningssloeyfe.md`` § F, spikes S0–S6 in økt 54, plus three
|
||||
probes run at the head of økt 56):
|
||||
|
||||
* a plan-review ``revise`` costs two manager calls, **zero** rounds, and asks *again* — so an
|
||||
always-revising expert is unbounded spend under a round cap that never ticks. That is the whole
|
||||
reason ``max_plan_revisions`` is a required contract field rather than a nicety.
|
||||
* the round cap and the reset cap **raise nothing**. Both end the run with a canonical assistant
|
||||
message and a normal-looking result (measured: ``max_round_count=2`` → two ledger events and
|
||||
``'Workflow terminated due to reaching maximum round count.'``; a stalling ledger with
|
||||
``max_reset_count=1`` → one ``REPLANNED`` event and ``'…maximum reset count.'``). At the
|
||||
transport both are indistinguishable from success, so this layer produces the typed stop itself.
|
||||
* a ``next_speaker`` naming nobody produces a **silent final answer with zero participant work**
|
||||
(``_magentic.py:1128-1131``) — a plausible answer produced by no work at all, which is the
|
||||
hazard class E2 was retired for. The names are therefore validated, never assumed.
|
||||
|
||||
**The client is the repo's own ``ScriptedChatClient``.** A bare ``BaseChatClient`` silently no-ops
|
||||
``BudgetMiddleware`` (measured, ``simulation.py:373-375``), so a budget claim proved against one
|
||||
would prove nothing.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from agent_framework import BaseChatClient
|
||||
from opentelemetry.sdk.trace import TracerProvider
|
||||
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
|
||||
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
|
||||
from pydantic import ValidationError
|
||||
|
||||
import portfolio_optimiser
|
||||
from portfolio_optimiser import explore, okf
|
||||
from portfolio_optimiser.budget import Budget, BudgetExceeded, TokenMeter
|
||||
from portfolio_optimiser.explore import ExplorationContract
|
||||
from portfolio_optimiser.mandate import Approach
|
||||
from portfolio_optimiser.simulation import ScriptedChatClient
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# C.3 — the contract: an exploration without stated bounds refuses to start
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
_FULL_CONTRACT = {
|
||||
"max_rounds": 4,
|
||||
"max_tokens": 5_000,
|
||||
"max_stall_count": 2,
|
||||
"max_reset_count": 1,
|
||||
"max_plan_revisions": 1,
|
||||
"enable_plan_review": True,
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("omitted", sorted(_FULL_CONTRACT))
|
||||
def test_every_bound_is_required_with_no_default(omitted: str) -> None:
|
||||
"""T1: each of the six fields is REQUIRED — dropping any one refuses construction.
|
||||
|
||||
Not a style point. ``MagenticBuilder`` defaults ``max_round_count`` to ``None`` (unbounded)
|
||||
and ``max_reset_count`` to ``None`` (unlimited), and inheriting either would give this repo
|
||||
the one thing ``shared/method-spec.md`` §8 forbids outright: a loop with no stated end. A
|
||||
default here would also be a claim about the operator's intent that nobody made — the same
|
||||
ground on which ``ProvenanceStamp.cost_baseline_anchored`` is required without one.
|
||||
"""
|
||||
payload = {k: v for k, v in _FULL_CONTRACT.items() if k != omitted}
|
||||
with pytest.raises(ValidationError):
|
||||
ExplorationContract(**payload)
|
||||
|
||||
|
||||
def test_full_contract_constructs() -> None:
|
||||
"""T2: the control for T1 — the complete payload IS valid.
|
||||
|
||||
Without it, T1 would pass on a model that refuses everything, which is the vacuous-gate class
|
||||
this repo has paid for six times.
|
||||
"""
|
||||
contract = ExplorationContract(**_FULL_CONTRACT)
|
||||
assert contract.max_rounds == 4
|
||||
assert contract.enable_plan_review is True
|
||||
|
||||
|
||||
def test_a_revision_cap_without_plan_review_is_refused_not_ignored() -> None:
|
||||
"""T3: ``max_plan_revisions > 0`` with ``enable_plan_review=False`` refuses.
|
||||
|
||||
A plan revision can only arise from a plan review — with the review off, the cap bounds an
|
||||
event that cannot occur, and a caller who set it believes they bounded something. This repo
|
||||
refuses a setting that cannot take effect rather than dropping it silently (the same partition
|
||||
``--embedder-config requires --semantic-retrieval`` enforces on the CLI).
|
||||
"""
|
||||
with pytest.raises(ValidationError):
|
||||
ExplorationContract(**{**_FULL_CONTRACT, "enable_plan_review": False})
|
||||
|
||||
|
||||
def test_review_off_with_zero_revisions_is_the_coherent_form() -> None:
|
||||
"""T4: the control for T3 — review off and the cap at ``0`` is a consistent statement, and
|
||||
must construct. Without this arm T3 would pass on a model that simply forbade
|
||||
``enable_plan_review=False`` outright, which is a different (and wrong) rule.
|
||||
"""
|
||||
contract = ExplorationContract(
|
||||
**{**_FULL_CONTRACT, "enable_plan_review": False, "max_plan_revisions": 0}
|
||||
)
|
||||
assert contract.enable_plan_review is False
|
||||
assert contract.max_plan_revisions == 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# The scripted stand-ins. ScriptedChatClient, never a bare BaseChatClient: the latter no-ops
|
||||
# BudgetMiddleware (measured, simulation.py:373-375), so a budget assertion made against one
|
||||
# would assert nothing.
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
PROMPT = "Find the cheapest saving available in the energy bundle."
|
||||
|
||||
|
||||
def _ledger_json(
|
||||
*, satisfied: bool, speaker: str, instruction: str = "Shape one hypothesis."
|
||||
) -> str:
|
||||
"""A progress ledger naming ``speaker``.
|
||||
|
||||
The name is a PARAMETER, never a literal, because a ``next_speaker`` matching no participant
|
||||
is the measured footgun this module defends against: the orchestrator does not error, it
|
||||
quietly emits a final answer having asked nobody (``_magentic.py:1128-1131``).
|
||||
"""
|
||||
return json.dumps(
|
||||
{
|
||||
"is_request_satisfied": {"reason": "r", "answer": satisfied},
|
||||
"is_in_loop": {"reason": "r", "answer": False},
|
||||
"is_progress_being_made": {"reason": "r", "answer": True},
|
||||
"next_speaker": {"reason": "r", "answer": speaker},
|
||||
"instruction_or_question": {"reason": "r", "answer": instruction},
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _stalling_ledger_json(speaker: str) -> str:
|
||||
"""A ledger reporting NO progress and a loop — the two flags that drive ``stall_count`` up."""
|
||||
return json.dumps(
|
||||
{
|
||||
"is_request_satisfied": {"reason": "r", "answer": False},
|
||||
"is_in_loop": {"reason": "circles", "answer": True},
|
||||
"is_progress_being_made": {"reason": "none", "answer": False},
|
||||
"next_speaker": {"reason": "r", "answer": speaker},
|
||||
"instruction_or_question": {"reason": "r", "answer": "Try again."},
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _manager_script(
|
||||
ledgers: list[str], calls: list[str] | None = None
|
||||
) -> Callable[[str, str], str]:
|
||||
"""Route a manager prompt blob to its scripted reply, consuming ``ledgers`` in order.
|
||||
|
||||
The ORDER of these tests is load-bearing and was measured (§ F, A6): the selector receives the
|
||||
CONCATENATION of every message in the call, so a later-stage prompt still carries the earlier
|
||||
stage's text — one manager call in five carries two markers. Testing the later stage FIRST is
|
||||
what resolves it; reversing two of these silently reattributes a reply to the wrong stage.
|
||||
"""
|
||||
|
||||
def _select(blob: str, _role: str) -> str:
|
||||
if calls is not None:
|
||||
calls.append(blob[:40])
|
||||
if "provide the final answer" in blob:
|
||||
return "FINAL: exploration done."
|
||||
if "pure JSON format" in blob:
|
||||
return ledgers.pop(0) if ledgers else _ledger_json(satisfied=True, speaker="navigator")
|
||||
if "went wrong on this last run" in blob:
|
||||
return "PLAN-UPDATE: revised plan."
|
||||
if "rewrite the following fact sheet" in blob:
|
||||
return "FACTS-UPDATE: revised facts."
|
||||
if "bullet-point plan" in blob:
|
||||
return "PLAN: - ask the hypothesiser"
|
||||
if "pre-survey" in blob:
|
||||
return "FACTS: the bundle is anchored."
|
||||
return "{}"
|
||||
|
||||
return _select
|
||||
|
||||
|
||||
def _factory(
|
||||
*, ledgers: list[str], hypothesiser: list[str], navigator: str = "NAVIGATOR: index read."
|
||||
) -> Callable[[str], BaseChatClient]:
|
||||
"""One fresh ``ScriptedChatClient`` per role, exactly as the real factory hands out one per
|
||||
role. ``hypothesiser`` is a list consumed in order, so a run can shape several candidates."""
|
||||
|
||||
def factory(role: str) -> BaseChatClient:
|
||||
if role == explore.MANAGER_ROLE:
|
||||
return ScriptedChatClient(reply_selector=_manager_script(ledgers), role=role)
|
||||
if role == explore.HYPOTHESISER_ROLE:
|
||||
replies = list(hypothesiser)
|
||||
|
||||
def _hyp(_blob: str, _role: str) -> str:
|
||||
return replies.pop(0) if replies else "nothing further."
|
||||
|
||||
return ScriptedChatClient(reply_selector=_hyp, role=role)
|
||||
return ScriptedChatClient(navigator, role=role)
|
||||
|
||||
return factory
|
||||
|
||||
|
||||
def _hypothesis_line(label: str, rationale: str) -> str:
|
||||
return f"{explore.HYPOTHESIS_MARKER} " + json.dumps({"label": label, "rationale": rationale})
|
||||
|
||||
|
||||
#: The no-review base every stop test derives from. ``enable_plan_review`` and
|
||||
#: ``max_plan_revisions`` move together — ``ExplorationContract`` refuses them apart — so a test
|
||||
#: about round or stall behaviour has to say so explicitly rather than inherit ``_FULL_CONTRACT``.
|
||||
_NO_REVIEW = {**_FULL_CONTRACT, "enable_plan_review": False, "max_plan_revisions": 0}
|
||||
|
||||
_CONTRACT = ExplorationContract(
|
||||
max_rounds=6,
|
||||
max_tokens=100_000,
|
||||
max_stall_count=2,
|
||||
max_reset_count=1,
|
||||
max_plan_revisions=0,
|
||||
enable_plan_review=False,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# C.0 / C.6 — the exploration is a MANDATE-FORMER, and a seed never disappears
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_hypotheses_become_the_mandate_in_the_order_they_were_shaped() -> None:
|
||||
"""T5: what the hypothesiser MARKED becomes ``Mandate.approaches``, rationale VERBATIM.
|
||||
|
||||
The rationale is the half a model cannot re-derive from cost data — ``mandate.Approach``
|
||||
already feeds ``description`` to the proposer verbatim (``generate._build_messages``), so
|
||||
paraphrasing it here would drop precisely the part the exploration exists to carry forward.
|
||||
"""
|
||||
ledgers = [
|
||||
_ledger_json(satisfied=False, speaker=explore.HYPOTHESISER_ROLE),
|
||||
_ledger_json(satisfied=True, speaker=explore.HYPOTHESISER_ROLE),
|
||||
]
|
||||
result = await explore.explore(
|
||||
PROMPT,
|
||||
contract=_CONTRACT,
|
||||
bundle_dirs=(),
|
||||
client_factory=_factory(
|
||||
ledgers=ledgers,
|
||||
hypothesiser=[
|
||||
"Looking at the bundle.\n"
|
||||
+ _hypothesis_line("LED retrofit", "the fixtures are 1990s fluorescent")
|
||||
],
|
||||
),
|
||||
)
|
||||
|
||||
assert [a.label for a in result.mandate.approaches] == ["LED retrofit"]
|
||||
assert result.mandate.approaches[0].description == "the fixtures are 1990s fluorescent"
|
||||
assert result.mandate.objective == PROMPT
|
||||
assert result.stop is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_seed_approach_survives_whatever_the_manager_found() -> None:
|
||||
"""T6: an expert's own hypothesis is in the output mandate, FIRST, untouched.
|
||||
|
||||
Door 1 of § C.6, and the ``not_evaluated`` rule applied one stage earlier: a direction the
|
||||
domain expert asked for may never vanish because an autonomous loop preferred its own. Seeds
|
||||
lead so the pipeline reaches them before spending its budget on discovered ones.
|
||||
"""
|
||||
seed = Approach(id="fagperson-1", label="Night setback", description="the expert's own words")
|
||||
ledgers = [
|
||||
_ledger_json(satisfied=False, speaker=explore.HYPOTHESISER_ROLE),
|
||||
_ledger_json(satisfied=True, speaker=explore.HYPOTHESISER_ROLE),
|
||||
]
|
||||
result = await explore.explore(
|
||||
PROMPT,
|
||||
contract=_CONTRACT,
|
||||
bundle_dirs=(),
|
||||
seed_approaches=(seed,),
|
||||
client_factory=_factory(
|
||||
ledgers=ledgers,
|
||||
hypothesiser=[_hypothesis_line("LED retrofit", "fluorescent fixtures")],
|
||||
),
|
||||
)
|
||||
|
||||
assert [a.id for a in result.mandate.approaches] == ["fagperson-1", "hypothesis-1"]
|
||||
assert result.mandate.approaches[0] == seed
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_an_exploration_that_shaped_nothing_still_returns_the_seeds() -> None:
|
||||
"""T7: the control for T6 — with the hypothesiser silent, the seed is still the mandate.
|
||||
|
||||
This is what makes T6 a statement about PRESERVATION rather than about ordering: a test that
|
||||
only ever saw seeds alongside discoveries could not tell "seeds are kept" from "seeds sort
|
||||
first".
|
||||
"""
|
||||
seed = Approach(id="fagperson-1", label="Night setback", description="the expert's own words")
|
||||
result = await explore.explore(
|
||||
PROMPT,
|
||||
contract=_CONTRACT,
|
||||
bundle_dirs=(),
|
||||
seed_approaches=(seed,),
|
||||
client_factory=_factory(
|
||||
ledgers=[_ledger_json(satisfied=True, speaker=explore.NAVIGATOR_ROLE)],
|
||||
hypothesiser=[],
|
||||
),
|
||||
)
|
||||
|
||||
assert result.mandate.approaches == (seed,)
|
||||
assert result.mandate.allow_own_proposals is True
|
||||
|
||||
|
||||
def test_zero_resets_is_refused_because_it_silently_explores_nothing() -> None:
|
||||
"""T8: ``max_reset_count=0`` refuses — MEASURED, not reasoned.
|
||||
|
||||
The orchestrator's limit check is ``reset_count >= max_reset_count`` (``_magentic.py:1243``)
|
||||
and ``reset_count`` starts at zero, so a cap of zero is already met before the first round.
|
||||
Measured against the installed stack: the run makes only the ``facts`` and ``plan`` manager
|
||||
calls, emits **zero** progress-ledger events, and returns
|
||||
``'Workflow terminated due to reaching maximum reset count.'`` — an exploration that explored
|
||||
nothing, reported as a stall that never happened. An operator writing "allow no resets" would
|
||||
get "do no work", quietly. So it is refused at construction, where the reason can be said.
|
||||
"""
|
||||
with pytest.raises(ValidationError):
|
||||
ExplorationContract(**{**_FULL_CONTRACT, "max_reset_count": 0})
|
||||
|
||||
|
||||
def test_zero_stalls_is_allowed_because_it_means_something() -> None:
|
||||
"""T9: the control for T8 — ``max_stall_count=0`` is a real setting and must construct.
|
||||
|
||||
The stall check is STRICT (``stall_count > max_stall_count``, ``:1118``) and the counter is
|
||||
incremented before it, so zero means "reset on the first round that reports no progress".
|
||||
That is strictness, not self-defeat, and refusing both zeroes on symmetry would have banned a
|
||||
usable configuration on the strength of a measurement about a different field.
|
||||
"""
|
||||
contract = ExplorationContract(**{**_FULL_CONTRACT, "max_stall_count": 0})
|
||||
assert contract.max_stall_count == 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# C.2 / C.3 — three endings the orchestration reports as if they were success
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_round_cap_leaves_as_a_typed_budget_stop() -> None:
|
||||
"""T10: the round cap becomes ``BudgetExceeded(kind="exploration_rounds")``.
|
||||
|
||||
Measured (§ F, E5, re-measured at the head of this økt): ``max_round_count`` raises NOTHING.
|
||||
The run ends with the assistant message ``'Workflow terminated due to reaching maximum round
|
||||
count.'`` and a result that ``get_outputs()`` answers like any other — at the transport it is
|
||||
indistinguishable from a finished exploration. Left alone, a caller would read a run that
|
||||
explored two rounds of a six-round question as a completed answer. The triple is the one
|
||||
kø-(y) defends: WHICH cap bound, what it was, and how far the run actually got.
|
||||
"""
|
||||
contract = ExplorationContract(**{**_NO_REVIEW, "max_rounds": 2})
|
||||
with pytest.raises(BudgetExceeded) as excinfo:
|
||||
await explore.explore(
|
||||
PROMPT,
|
||||
contract=contract,
|
||||
bundle_dirs=(),
|
||||
client_factory=_factory(
|
||||
ledgers=[_ledger_json(satisfied=False, speaker=explore.HYPOTHESISER_ROLE)] * 4,
|
||||
hypothesiser=[_hypothesis_line("LED", "worth a look")] * 4,
|
||||
),
|
||||
)
|
||||
|
||||
assert excinfo.value.kind == "exploration_rounds"
|
||||
assert excinfo.value.limit == 2
|
||||
assert excinfo.value.observed == 2
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_request_satisfied_on_the_last_allowed_round_is_success() -> None:
|
||||
"""T11: the discriminator for T10 — reaching the cap is not the same as being cut off by it.
|
||||
|
||||
Both runs end with exactly ``max_rounds`` progress-ledger events, so a check written on the
|
||||
count alone would raise on this one too and turn a completed exploration into a budget error.
|
||||
What separates them is the LAST ledger's ``is_request_satisfied``, which is also what the
|
||||
orchestrator itself branches on (``:1106``). Without this arm, T10 would pass on an
|
||||
implementation that refuses every exploration that uses its whole allowance.
|
||||
"""
|
||||
contract = ExplorationContract(**{**_NO_REVIEW, "max_rounds": 2})
|
||||
result = await explore.explore(
|
||||
PROMPT,
|
||||
contract=contract,
|
||||
bundle_dirs=(),
|
||||
client_factory=_factory(
|
||||
ledgers=[
|
||||
_ledger_json(satisfied=False, speaker=explore.HYPOTHESISER_ROLE),
|
||||
_ledger_json(satisfied=True, speaker=explore.HYPOTHESISER_ROLE),
|
||||
],
|
||||
hypothesiser=[_hypothesis_line("LED", "worth a look")],
|
||||
),
|
||||
)
|
||||
|
||||
assert len(result.ledger_log) == contract.max_rounds
|
||||
assert result.stop is None
|
||||
assert [a.label for a in result.mandate.approaches] == ["LED"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_stalling_out_is_a_typed_value_never_an_exception() -> None:
|
||||
"""T12: stall → reset → out of resets is ``stop="stalled"``, and the run still returns.
|
||||
|
||||
Kept as a VALUE while the round cap RAISES, and the split is S3.4's, not a preference: a
|
||||
stalled exploration is an outcome (the manager tried and got nowhere), whereas an exhausted
|
||||
round or token cap is resource exhaustion. Fusing them would leave a caller unable to tell
|
||||
"there was nothing here" from "we could not afford to look".
|
||||
"""
|
||||
contract = ExplorationContract(
|
||||
**{**_NO_REVIEW, "max_rounds": 6, "max_stall_count": 1, "max_reset_count": 1}
|
||||
)
|
||||
result = await explore.explore(
|
||||
PROMPT,
|
||||
contract=contract,
|
||||
bundle_dirs=(),
|
||||
client_factory=_factory(
|
||||
ledgers=[_stalling_ledger_json(explore.HYPOTHESISER_ROLE)] * 6,
|
||||
hypothesiser=["still nothing."] * 6,
|
||||
),
|
||||
)
|
||||
|
||||
assert result.stop == "stalled"
|
||||
assert len(result.ledger_log) < contract.max_rounds
|
||||
assert all(entry.is_in_loop for entry in result.ledger_log)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_ledger_naming_nobody_withholds_what_the_run_produced() -> None:
|
||||
"""T13: a ``next_speaker`` matching no participant stops the exploration and drops its finds.
|
||||
|
||||
The measured footgun (``_magentic.py:1128-1131``): the orchestrator neither raises nor retries
|
||||
on an unknown speaker — it logs a warning and jumps to ``_prepare_final_answer``. The run
|
||||
therefore returns a plausible answer that no participant was asked for. This is the shape E2
|
||||
was retired over ("a plausible verdict produced by zero work"), so the mandate is NOT built
|
||||
from what such a run said it found.
|
||||
|
||||
The scripted run reaches the bad ledger on round TWO, after a good round in which the
|
||||
hypothesiser really did commit to a direction. That ordering is what makes the assertion
|
||||
sharp: with the bad ledger first, nobody would ever have spoken and "nothing was carried
|
||||
forward" would be true of any implementation at all.
|
||||
"""
|
||||
seed = Approach(id="fagperson-1", label="Night setback", description="expert's own")
|
||||
result = await explore.explore(
|
||||
PROMPT,
|
||||
contract=_CONTRACT,
|
||||
bundle_dirs=(),
|
||||
seed_approaches=(seed,),
|
||||
client_factory=_factory(
|
||||
ledgers=[
|
||||
_ledger_json(satisfied=False, speaker=explore.HYPOTHESISER_ROLE),
|
||||
_ledger_json(satisfied=False, speaker="a-name-nobody-answers-to"),
|
||||
],
|
||||
hypothesiser=[_hypothesis_line("LED retrofit", "fluorescent fixtures")],
|
||||
),
|
||||
)
|
||||
|
||||
assert result.stop == "unknown_speaker"
|
||||
assert result.ledger_log[0].speaker_known is True
|
||||
assert result.ledger_log[-1].speaker_known is False
|
||||
# The seed survives — preservation is unconditional (§ C.6 door 1) — while the loop's own
|
||||
# find does not, because nothing stands behind the turn that ended the run.
|
||||
assert result.mandate.approaches == (seed,)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# C.2 — the token cap covers the MANAGER, which is the loop's most talkative agent
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_token_cap_binds_the_manager_before_any_participant_speaks() -> None:
|
||||
"""T14: a one-token budget stops the exploration on the MANAGER's own first call.
|
||||
|
||||
Agent-level ``ChatMiddleware`` does fire on the manager's calls (§ F, A1, measured green), and
|
||||
the manager talks more than anyone else in a Magentic loop — it extracts facts, writes the
|
||||
plan, and writes a progress ledger every single round. A cap fastened only to the participants
|
||||
would be a cap in name.
|
||||
|
||||
The assertion is deliberately not "something raised". ``kind == "tokens"`` separates it from
|
||||
the round-cap stop, ``meter.tokens == 8`` shows the charge came from a call that was actually
|
||||
made and metered, and the EMPTY ledger log shows it landed before the loop had run a single
|
||||
round — which is exactly what a manager-attached middleware does and a participant-only one
|
||||
cannot.
|
||||
"""
|
||||
meter = TokenMeter(Budget(max_tokens=1, max_rounds=6))
|
||||
contract = ExplorationContract(**{**_NO_REVIEW, "max_tokens": 1})
|
||||
|
||||
with pytest.raises(BudgetExceeded) as excinfo:
|
||||
await explore.explore(
|
||||
PROMPT,
|
||||
contract=contract,
|
||||
bundle_dirs=(),
|
||||
meter=meter,
|
||||
client_factory=_factory(
|
||||
ledgers=[_ledger_json(satisfied=True, speaker=explore.NAVIGATOR_ROLE)],
|
||||
hypothesiser=[],
|
||||
),
|
||||
)
|
||||
|
||||
assert excinfo.value.kind == "tokens"
|
||||
assert meter.tokens == 8, "the manager's own call must have been charged to the meter"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# C.5 / U13 — the synchronous plan review, and the cap the measurement forced
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _reviewer(script: list[explore.PlanReviewDecision], seen: list[explore.PlanReviewRequest]):
|
||||
def review(request: explore.PlanReviewRequest) -> explore.PlanReviewDecision:
|
||||
seen.append(request)
|
||||
return script.pop(0) if script else explore.PlanReviewDecision.approve()
|
||||
|
||||
return review
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_revision_reaches_the_manager_and_the_review_is_asked_again() -> None:
|
||||
"""T15: revise → replan → asked AGAIN → approve → the loop runs.
|
||||
|
||||
This is målbilde's "ask the question, use the answer, carry on" on the installed stack: the
|
||||
expert's words go into the manager's history, the manager replans, and the human is asked to
|
||||
sign off on the NEW plan rather than the old one. Both round trips are recorded, in order,
|
||||
with the feedback verbatim — an audit of what a human actually told an autonomous loop is
|
||||
worth nothing paraphrased.
|
||||
"""
|
||||
seen: list[explore.PlanReviewRequest] = []
|
||||
contract = ExplorationContract(
|
||||
**{**_FULL_CONTRACT, "enable_plan_review": True, "max_plan_revisions": 2}
|
||||
)
|
||||
result = await explore.explore(
|
||||
PROMPT,
|
||||
contract=contract,
|
||||
bundle_dirs=(),
|
||||
plan_reviewer=_reviewer(
|
||||
[explore.PlanReviewDecision.revise("Also test night setback.")], seen
|
||||
),
|
||||
client_factory=_factory(
|
||||
ledgers=[
|
||||
_ledger_json(satisfied=False, speaker=explore.HYPOTHESISER_ROLE),
|
||||
_ledger_json(satisfied=True, speaker=explore.HYPOTHESISER_ROLE),
|
||||
],
|
||||
hypothesiser=[_hypothesis_line("Night setback", "the expert asked for it")],
|
||||
),
|
||||
)
|
||||
|
||||
assert [r.decision for r in result.plan_reviews] == ["revise", "approve"]
|
||||
assert result.plan_reviews[0].feedback == "Also test night setback."
|
||||
assert len(seen) == 2, "a revision must produce a SECOND review, not resume silently"
|
||||
assert seen[1].plan != "", "the second review must show the revised plan"
|
||||
assert result.stop is None
|
||||
assert [a.label for a in result.mandate.approaches] == ["Night setback"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_an_always_revising_reviewer_is_stopped_by_the_cap() -> None:
|
||||
"""T16: the cap terminates a reviewer that never signs off — the reason it exists.
|
||||
|
||||
Measured (§ F, A3): a revise costs two manager calls, emits NO progress ledger and consumes
|
||||
NO round, then asks again. The round cap therefore never ticks, and without
|
||||
``max_plan_revisions`` this is an unbounded spend under caps that all look satisfied —
|
||||
precisely what ``shared/method-spec.md`` §8 forbids. The stop is typed and the exploration
|
||||
still returns; the reviewer's last (refused) revision is recorded, because the record is of
|
||||
what the human decided and ``stop`` is what says it was not applied.
|
||||
"""
|
||||
seen: list[explore.PlanReviewRequest] = []
|
||||
always_revise = [explore.PlanReviewDecision.revise(f"Again #{n}.") for n in range(10)]
|
||||
contract = ExplorationContract(
|
||||
**{**_FULL_CONTRACT, "enable_plan_review": True, "max_plan_revisions": 1}
|
||||
)
|
||||
result = await explore.explore(
|
||||
PROMPT,
|
||||
contract=contract,
|
||||
bundle_dirs=(),
|
||||
plan_reviewer=_reviewer(always_revise, seen),
|
||||
client_factory=_factory(
|
||||
ledgers=[_ledger_json(satisfied=True, speaker=explore.NAVIGATOR_ROLE)],
|
||||
hypothesiser=[],
|
||||
),
|
||||
)
|
||||
|
||||
assert result.stop == "plan_revisions_exhausted"
|
||||
assert [r.decision for r in result.plan_reviews] == ["revise", "revise"]
|
||||
assert result.ledger_log == (), "the loop must never have run: the plan was never approved"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_reviewer_that_signs_off_at_once_is_not_capped() -> None:
|
||||
"""T17: the control for T16 — the same cap, a reviewer that approves, and no stop.
|
||||
|
||||
Without it, T16 would pass on an implementation that refuses every plan review it is given,
|
||||
which would stop the runaway loop and every legitimate one with it.
|
||||
"""
|
||||
seen: list[explore.PlanReviewRequest] = []
|
||||
contract = ExplorationContract(
|
||||
**{**_FULL_CONTRACT, "enable_plan_review": True, "max_plan_revisions": 1}
|
||||
)
|
||||
result = await explore.explore(
|
||||
PROMPT,
|
||||
contract=contract,
|
||||
bundle_dirs=(),
|
||||
plan_reviewer=_reviewer([], seen),
|
||||
client_factory=_factory(
|
||||
ledgers=[_ledger_json(satisfied=True, speaker=explore.NAVIGATOR_ROLE)],
|
||||
hypothesiser=[],
|
||||
),
|
||||
)
|
||||
|
||||
assert result.stop is None
|
||||
assert [r.decision for r in result.plan_reviews] == ["approve"]
|
||||
assert len(result.ledger_log) >= 1, "an approved plan must let the loop actually run"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# C.0 level 3 — the exploration has no write access, and level 1 is advisory
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _tree(root: Path) -> dict[str, bytes]:
|
||||
return {
|
||||
str(p.relative_to(root)): p.read_bytes() for p in sorted(root.rglob("*")) if p.is_file()
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_an_exploration_leaves_the_knowledge_base_byte_identical(tmp_path: Path) -> None:
|
||||
"""T18: ``explore()`` writes NOTHING — not to the base, not anywhere under it.
|
||||
|
||||
Level 3 of the guarantee table: only the pipeline may write an outbox artefact, and only the
|
||||
gated ``promote_verdict`` may write to the wiki. An exploration that could write would be a
|
||||
route around the gate that makes an answer checkable — and, promoting into the base it reads,
|
||||
the self-contamination loop the Step-8 gate exists to prevent.
|
||||
|
||||
Compared BYTE for byte over the whole subtree rather than by listing names, so a rewritten
|
||||
``index.md`` of the same length would still fail.
|
||||
|
||||
**The tools are exercised DIRECTLY, and that is a correction, not thoroughness.** A first
|
||||
version of this test drove only ``explore()`` — and a mutation that made ``read_bundle`` write
|
||||
a file into the base it reads left the WHOLE suite green (measured: 974 passed). A
|
||||
``ScriptedChatClient`` returns text and never emits a tool call, so no scripted run reaches a
|
||||
tool body: the read surface, which is the only place a write could plausibly come from, was
|
||||
outside the gate entirely.
|
||||
"""
|
||||
base = tmp_path / "bygg-energi-baseline-mikro"
|
||||
shutil.copytree(
|
||||
Path(portfolio_optimiser.__file__).parent / "data" / "bundles" / base.name, base
|
||||
)
|
||||
before = _tree(tmp_path)
|
||||
|
||||
result = await explore.explore(
|
||||
PROMPT,
|
||||
contract=_CONTRACT,
|
||||
bundle_dirs=(str(base),),
|
||||
client_factory=_factory(
|
||||
# Three ledgers, and the third is what makes the second one matter: the orchestrator
|
||||
# tests ``is_request_satisfied`` BEFORE it reads ``next_speaker`` (``:1106``), so a
|
||||
# satisfied ledger naming the hypothesiser never actually asks it anything.
|
||||
ledgers=[
|
||||
_ledger_json(satisfied=False, speaker=explore.NAVIGATOR_ROLE),
|
||||
_ledger_json(satisfied=False, speaker=explore.HYPOTHESISER_ROLE),
|
||||
_ledger_json(satisfied=True, speaker=explore.HYPOTHESISER_ROLE),
|
||||
],
|
||||
hypothesiser=[_hypothesis_line("LED retrofit", "fluorescent fixtures")],
|
||||
),
|
||||
)
|
||||
|
||||
assert result.mandate.approaches[0].label == "LED retrofit"
|
||||
|
||||
# Every read tool, called on the same base, with model-shaped arguments.
|
||||
tools = {t.name: t for t in explore.navigator_tools((str(base),))}
|
||||
assert tools["list_bundles"].func()[0]["id"] == base.name
|
||||
assert tools["read_bundle"].func(bundle_id=base.name) != ""
|
||||
assert tools["read_file"].func(bundle_id=base.name, path="index.md") != ""
|
||||
explore.quick_validate_tool((str(base),)).func(
|
||||
bundle_id=base.name, proposal_json=json.dumps(_micro_projection())
|
||||
)
|
||||
|
||||
assert _tree(tmp_path) == before
|
||||
|
||||
|
||||
def _micro_bundle_dir() -> str:
|
||||
return str(
|
||||
Path(portfolio_optimiser.__file__).parent
|
||||
/ "data"
|
||||
/ "bundles"
|
||||
/ "bygg-energi-baseline-mikro"
|
||||
)
|
||||
|
||||
|
||||
def _micro_projection() -> dict[str, Any]:
|
||||
projection = dict(okf.load_ir_projection(_micro_bundle_dir()))
|
||||
projection.pop("_note", None)
|
||||
return projection
|
||||
|
||||
|
||||
def test_quick_validate_reports_the_real_verdict_and_says_whether_it_was_anchored() -> None:
|
||||
"""T19: the in-loop check is the SAME validator, and it declares its own anchoring.
|
||||
|
||||
Level 1 is advisory but never fake: it runs ``validate_proposal`` against the base's own
|
||||
``cost-baseline.json``, so stage 0 reconciliation is live and a fabricated cost line is caught
|
||||
in the loop rather than three steps later. ``anchored`` rides along for the reason
|
||||
``ProvenanceStamp.cost_baseline_anchored`` is a required field — a verdict reached without the
|
||||
project's real cost lines is a weaker claim, and one that does not say so is a silence.
|
||||
"""
|
||||
base = _micro_bundle_dir()
|
||||
projection = _micro_projection()
|
||||
validate = explore.quick_validate_tool((base,))
|
||||
|
||||
honest = validate.func(
|
||||
bundle_id="bygg-energi-baseline-mikro", proposal_json=json.dumps(projection)
|
||||
)
|
||||
assert honest["decision"] == "validated"
|
||||
assert honest["anchored"] is True
|
||||
assert honest["p90"] >= honest["p50"] >= honest["p10"]
|
||||
|
||||
# A cost code the project does not have is refused by stage 0 — the one stage that can tell a
|
||||
# fabricated line from a real one, and the reason `anchored` is worth reporting at all.
|
||||
invented = dict(projection)
|
||||
invented["affected_items"] = [
|
||||
{**dict(projection["affected_items"][0]), "code": "CODE-THAT-DOES-NOT-EXIST"}
|
||||
]
|
||||
fabricated = validate.func(
|
||||
bundle_id="bygg-energi-baseline-mikro", proposal_json=json.dumps(invented)
|
||||
)
|
||||
assert fabricated["decision"] == "rejected"
|
||||
assert "CODE-THAT-DOES-NOT-EXIST" in fabricated["reason"]
|
||||
|
||||
|
||||
def test_an_unknown_knowledge_base_is_refused_by_name() -> None:
|
||||
"""T20: a tool call naming a base nobody configured refuses, and says what IS configured.
|
||||
|
||||
Model-chosen arguments are untrusted input. Answering an unknown id with an empty result would
|
||||
let the manager conclude the base is empty rather than absent — the fourth face of the
|
||||
verification law, arrived at through a tool rather than a query.
|
||||
"""
|
||||
validate = explore.quick_validate_tool(("/tmp/base-a",))
|
||||
with pytest.raises(explore.ExplorationError) as excinfo:
|
||||
validate.func(bundle_id="base-b", proposal_json="{}")
|
||||
assert "base-a" in str(excinfo.value)
|
||||
|
||||
|
||||
def test_two_bases_with_the_same_name_are_refused() -> None:
|
||||
"""T21: duplicate ids refuse at construction — the S3.2 key-collision class, one layer up.
|
||||
|
||||
The id is how the manager names a base. Two bases answering to one name would let it read A
|
||||
while believing it read B, and every quotation it produced afterwards would be attributed to
|
||||
the wrong project.
|
||||
"""
|
||||
with pytest.raises(explore.ExplorationError):
|
||||
explore.navigator_tools(("/tmp/one/shared-name", "/tmp/two/shared-name"))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# U14 — the three events the tracing seam was landed for, now that they have a call site
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _recording_tracer() -> tuple[Any, InMemorySpanExporter]:
|
||||
"""A REAL OpenTelemetry tracer over an in-memory exporter — not a spy.
|
||||
|
||||
A recorder standing in for ``add_event`` would prove that this module calls something shaped
|
||||
like OTel; this proves the events survive the actual SDK, with the attribute types it will
|
||||
accept. The provider is LOCAL and is never installed globally, so the pytest process keeps
|
||||
whatever tracing configuration it had (the same restraint U14's own tests exercise).
|
||||
"""
|
||||
provider = TracerProvider()
|
||||
exporter = InMemorySpanExporter()
|
||||
provider.add_span_processor(SimpleSpanProcessor(exporter))
|
||||
return provider.get_tracer("test"), exporter
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_three_orchestrator_events_reach_the_trace(monkeypatch: Any) -> None:
|
||||
"""T22: ``plan_created``, ``replanned`` and ``progress_ledger_updated`` are recorded.
|
||||
|
||||
These are the events U14 deliberately did NOT build in økt 55 — "an emitter with no call site
|
||||
is a shape guessed instead of measured". This is the call site. A Magentic manager decides
|
||||
which base to open and who speaks next; without these, the only trace of that reasoning is
|
||||
MAF's own ``invoke_agent`` spans, which say a call happened and nothing about what it decided.
|
||||
|
||||
The ledger event carries the decision fields rather than a rendered sentence, for the reason
|
||||
``SkippedLink`` is structured and ``BudgetExceeded`` carries three fields: "who was asked" and
|
||||
"was the request satisfied" are separate operative questions, and a reader who has to re-parse
|
||||
prose to tell them apart has a trace they cannot query.
|
||||
"""
|
||||
tracer, exporter = _recording_tracer()
|
||||
# Patched where the name is BOUND (the ``hosting.run_project`` precedent): ``explore``
|
||||
# imports it by name, so patching ``tracing`` would leave that binding untouched and this
|
||||
# test would quietly measure nothing.
|
||||
monkeypatch.setattr(explore, "exploration_tracer", lambda: tracer)
|
||||
|
||||
seen: list[explore.PlanReviewRequest] = []
|
||||
contract = ExplorationContract(
|
||||
**{**_FULL_CONTRACT, "enable_plan_review": True, "max_plan_revisions": 2}
|
||||
)
|
||||
await explore.explore(
|
||||
PROMPT,
|
||||
contract=contract,
|
||||
bundle_dirs=(),
|
||||
plan_reviewer=_reviewer([explore.PlanReviewDecision.revise("Test night setback.")], seen),
|
||||
client_factory=_factory(
|
||||
ledgers=[
|
||||
_ledger_json(satisfied=False, speaker=explore.HYPOTHESISER_ROLE),
|
||||
_ledger_json(satisfied=True, speaker=explore.HYPOTHESISER_ROLE),
|
||||
],
|
||||
hypothesiser=[_hypothesis_line("Night setback", "the expert asked")],
|
||||
),
|
||||
)
|
||||
|
||||
spans = exporter.get_finished_spans()
|
||||
assert [s.name for s in spans] == [explore.EXPLORATION_SPAN]
|
||||
events = [(e.name, dict(e.attributes or {})) for e in spans[0].events]
|
||||
names = [name for name, _ in events]
|
||||
assert names.count("plan_created") == 1
|
||||
assert names.count("replanned") == 1, "the human's revision must be visible in the trace"
|
||||
assert names.count("progress_ledger_updated") == 2
|
||||
|
||||
ledger_events = [attrs for name, attrs in events if name == "progress_ledger_updated"]
|
||||
assert [a["round_index"] for a in ledger_events] == [1, 2]
|
||||
assert [a["next_speaker"] for a in ledger_events] == [explore.HYPOTHESISER_ROLE] * 2
|
||||
assert [a["is_request_satisfied"] for a in ledger_events] == [False, True]
|
||||
assert all(a["speaker_known"] for a in ledger_events)
|
||||
|
||||
|
||||
#: A complete exploration in a CHILD interpreter. The stdout/stderr question cannot be answered
|
||||
#: in-process: ``ConsoleSpanExporter``'s ``out`` default is bound when
|
||||
#: ``opentelemetry.sdk.trace.export`` is first imported, so under pytest it is whatever stdout was
|
||||
#: at COLLECTION time — and ``capsys``, which replaces ``sys.stdout`` later, never sees it. That is
|
||||
#: not a testing quirk to work around; it is precisely the fact U14 exists for, and the reason
|
||||
#: ``configure_tracing`` passes ``out=`` explicitly instead of trusting the default. Measured: a
|
||||
#: mutation routing exploration spans to that default left an in-process ``capsys`` assertion
|
||||
#: GREEN while the spans really were on stdout.
|
||||
_CHILD_EXPLORATION = """
|
||||
import asyncio, json, sys
|
||||
from portfolio_optimiser import explore
|
||||
from portfolio_optimiser.simulation import ScriptedChatClient
|
||||
from portfolio_optimiser.tracing import configure_tracing
|
||||
|
||||
configure_tracing()
|
||||
|
||||
LEDGER = json.dumps({
|
||||
"is_request_satisfied": {"reason": "r", "answer": True},
|
||||
"is_in_loop": {"reason": "r", "answer": False},
|
||||
"is_progress_being_made": {"reason": "r", "answer": True},
|
||||
"next_speaker": {"reason": "r", "answer": "navigator"},
|
||||
"instruction_or_question": {"reason": "r", "answer": "none"},
|
||||
})
|
||||
|
||||
def _select(blob, _role):
|
||||
if "provide the final answer" in blob:
|
||||
return "FINAL: done."
|
||||
if "pure JSON format" in blob:
|
||||
return LEDGER
|
||||
if "bullet-point plan" in blob:
|
||||
return "PLAN: - ask the navigator"
|
||||
if "pre-survey" in blob:
|
||||
return "FACTS: none."
|
||||
return "{}"
|
||||
|
||||
def factory(role):
|
||||
if role == explore.MANAGER_ROLE:
|
||||
return ScriptedChatClient(reply_selector=_select, role=role)
|
||||
return ScriptedChatClient("ok", role=role)
|
||||
|
||||
contract = explore.ExplorationContract(
|
||||
max_rounds=4, max_tokens=100000, max_stall_count=2,
|
||||
max_reset_count=1, max_plan_revisions=0, enable_plan_review=False,
|
||||
)
|
||||
result = asyncio.run(
|
||||
explore.explore("probe", contract=contract, bundle_dirs=(), client_factory=factory)
|
||||
)
|
||||
assert result.stop is None, result.stop
|
||||
print("EXPLORATION-OK", file=sys.stderr)
|
||||
"""
|
||||
|
||||
|
||||
def _run_child(**env: str) -> subprocess.CompletedProcess[str]:
|
||||
return subprocess.run(
|
||||
[sys.executable, "-c", _CHILD_EXPLORATION],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
cwd=str(Path(__file__).resolve().parent.parent),
|
||||
env={**os.environ, **env},
|
||||
)
|
||||
|
||||
|
||||
def test_an_untraced_exploration_writes_nothing_to_stdout_or_stderr() -> None:
|
||||
"""T23: in a real process, with tracing off, an exploration prints NOTHING.
|
||||
|
||||
A subprocess and not ``capsys``, for the reason recorded above ``_CHILD_EXPLORATION`` — and the
|
||||
stakes are the pinned artefacts: ``tests/golden/demo-transcript.stdout`` is byte-fixed and the
|
||||
demo's stderr is fixed at four lines, so one stray span dump would break both.
|
||||
|
||||
``EXPLORATION-OK`` on stderr is the control. Without it, "stdout was empty" would be equally
|
||||
true of a child that crashed on import, which is the fourth face of the verification law: an
|
||||
absence is only evidence once you have shown the measurement could have found something.
|
||||
"""
|
||||
proc = _run_child(PORTFOLIO_OTEL="")
|
||||
assert proc.returncode == 0, proc.stderr
|
||||
assert "EXPLORATION-OK" in proc.stderr, "the child must really have run an exploration"
|
||||
assert proc.stdout == ""
|
||||
# Not an exact-equality assertion on stderr: MAF emits two ``ExperimentalWarning`` lines while
|
||||
# importing, under every run form, and they are the same pair the demo's pinned stderr already
|
||||
# carries. What must be absent is TRACE data, so that is what is asserted.
|
||||
assert '"name": "exploration"' not in proc.stderr
|
||||
assert "progress_ledger_updated" not in proc.stderr
|
||||
|
||||
|
||||
def test_a_traced_exploration_puts_its_span_on_stderr_and_leaves_stdout_clean() -> None:
|
||||
"""T24: the positive arm — ``PORTFOLIO_OTEL=console`` and the exploration span is on STDERR.
|
||||
|
||||
This is what the whole U14 seam was landed for, now carrying the events U4 gave it a call site
|
||||
for. Both halves are asserted: the span and its ``progress_ledger_updated`` event ARE exported
|
||||
(so tracing is real), and stdout is STILL empty (so the byte-pinned transcript survives a
|
||||
traced run). Asserting only the first would pass on an exporter writing to stdout — which is
|
||||
OpenTelemetry's own default, and therefore the mistake actually available to make.
|
||||
"""
|
||||
proc = _run_child(PORTFOLIO_OTEL="console")
|
||||
assert proc.returncode == 0, proc.stderr
|
||||
assert "EXPLORATION-OK" in proc.stderr
|
||||
assert proc.stdout == "", "a traced run must not put one byte on stdout"
|
||||
assert '"name": "exploration"' in proc.stderr
|
||||
assert "progress_ledger_updated" in proc.stderr
|
||||
|
||||
|
||||
def test_a_marked_line_that_will_not_parse_is_a_hard_error() -> None:
|
||||
"""T24: the marker is what makes fail-closed affordable here.
|
||||
|
||||
Most hypothesiser turns legitimately are not hypotheses — the agent reasons out loud — so
|
||||
"parse every turn or fail" would refuse a normal exploration. The marker separates a turn that
|
||||
is not a claim from a claim that cannot be read. The second is the run's own product coming
|
||||
back unreadable, so it raises (``write_concept_file``'s rule: validation, never repair) rather
|
||||
than following the tolerant RAW-inbox rule, which belongs to folders anyone may drop files in.
|
||||
"""
|
||||
with pytest.raises(explore.HypothesisParseError):
|
||||
explore._parse_hypotheses([f"{explore.HYPOTHESIS_MARKER} not json at all"], ())
|
||||
with pytest.raises(explore.HypothesisParseError):
|
||||
explore._parse_hypotheses([f'{explore.HYPOTHESIS_MARKER} {{"label": "no rationale"}}'], ())
|
||||
|
||||
|
||||
def test_unmarked_prose_is_not_a_failure() -> None:
|
||||
"""T25: the control for T24 — ordinary reasoning yields no hypothesis and no error.
|
||||
|
||||
Without it, T24 would pass on an implementation that refused every hypothesiser turn that was
|
||||
not a hypothesis, which would make the loop unusable and the strictness meaningless.
|
||||
"""
|
||||
assert (
|
||||
explore._parse_hypotheses(["I looked at the index and nothing stands out yet."], ()) == []
|
||||
)
|
||||
|
||||
|
||||
def test_a_review_nobody_can_answer_is_refused_before_the_first_model_call() -> None:
|
||||
"""T26: plan review without a reviewer refuses; a reviewer without plan review refuses too.
|
||||
|
||||
The first would hang: the workflow stops at a ``request_info`` and nothing ever answers it, and
|
||||
a hang is the one failure mode that reports nothing at all. The second is the silent-ignore the
|
||||
repo's flag partition forbids — a caller who supplied a reviewer believes a human is in the
|
||||
loop. Both are refused BEFORE anything is built, so neither costs a model call.
|
||||
"""
|
||||
with pytest.raises(explore.ExplorationError):
|
||||
asyncio.run(
|
||||
explore.explore(
|
||||
PROMPT,
|
||||
contract=ExplorationContract(**_FULL_CONTRACT),
|
||||
bundle_dirs=(),
|
||||
client_factory=_factory(ledgers=[], hypothesiser=[]),
|
||||
)
|
||||
)
|
||||
with pytest.raises(explore.ExplorationError):
|
||||
asyncio.run(
|
||||
explore.explore(
|
||||
PROMPT,
|
||||
contract=ExplorationContract(**_NO_REVIEW),
|
||||
bundle_dirs=(),
|
||||
plan_reviewer=lambda _r: explore.PlanReviewDecision.approve(),
|
||||
client_factory=_factory(ledgers=[], hypothesiser=[]),
|
||||
)
|
||||
)
|
||||
86
tests/test_full_run_live.py
Normal file
86
tests/test_full_run_live.py
Normal file
|
|
@ -0,0 +1,86 @@
|
|||
"""Fase 1b, last step — GATED live run over the WHOLE ``run_project`` path (måleprotokoll §4.4).
|
||||
|
||||
NOT default CI, and NOT gated like its two siblings. This is the expensive arm: it drives the
|
||||
complete vertical slice — bundle navigation, the maker/checker debate, generation under
|
||||
``response_format``, and the deterministic validator — against a real Foundry deployment.
|
||||
|
||||
**Why a THIRD environment variable, and why it is load-bearing.** ``test_foundry_profile_live.py``
|
||||
(client-level probe) and ``test_portfolio_live.py`` (``run_portfolio`` fan-out) both skip on exactly
|
||||
``PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT`` + ``PORTFOLIO_FOUNDRY_DEPLOYMENT``. Reusing that pair here
|
||||
would mean the moment an operator exports the two variables to run the CHEAP one-word probe, this
|
||||
full run fires too — collapsing the måleprotokoll's whole point (§1: *"bevis så mye som mulig før
|
||||
det dyre trinnet, så en feil er attribuerbar"*) into a single step, and spending money on a rung
|
||||
whose predecessors have not been shown green. ``PORTFOLIO_LIVE_FULL_RUN`` is therefore a separate,
|
||||
deliberate opt-in, read on **truthiness, not presence** (the Fase 4b invariant: an exported-but-empty
|
||||
value is a shell accident, not a decision).
|
||||
|
||||
``PORTFOLIO_MODEL_MAP`` is part of the skip condition for a different reason — attribution. ``run.py``
|
||||
stamps provenance with the deployment NAME before any client is built (målt 4e), so without the map
|
||||
the run fails for a CONFIGURATION reason while looking exactly like a model failure. Skipping is
|
||||
honest; failing there would misattribute.
|
||||
|
||||
**What this asserts is narrow on purpose** — see ``conftest.assert_full_run_contract``. The claim
|
||||
being felled is *"the emitted structured schema is accepted by the live endpoint"* (økt 37's stated
|
||||
honesty limit), NOT "the model proposes well". A validator REJECTION passes this test: the run
|
||||
reached the deterministic gate with a parsed candidate, which is the whole question. The contract's
|
||||
ability to discriminate is proven offline and for free by
|
||||
``tests/test_live_full_run_contract.py`` — the paid call here is the measurement, not the proof that
|
||||
the instrument works.
|
||||
|
||||
Outcomes are pre-registered in ``docs/2026-08-14-fase1b-forste-levende-kjoring.md`` §5, written
|
||||
BEFORE the run, so the write-up cannot be negotiated after the fact.
|
||||
|
||||
The round/token caps are the SAME ones the first live run died on. They are deliberately not
|
||||
raised: if the ledger fires again that is information, and raising it spends more on a path that may
|
||||
still be broken.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from conftest import assert_full_run_contract
|
||||
|
||||
from portfolio_optimiser.run import RunResult, run_project
|
||||
from portfolio_optimiser.verdicts import VerdictStore
|
||||
|
||||
BUNDLE_DIR = Path(__file__).resolve().parents[1] / "shared" / "examples" / "bygg-energi-mikro"
|
||||
_PROJECT_ID = "BYGG-KONTOR-NORD"
|
||||
|
||||
_ENDPOINT = os.environ.get("PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT")
|
||||
_DEPLOYMENT = os.environ.get("PORTFOLIO_FOUNDRY_DEPLOYMENT")
|
||||
_MODEL_MAP = os.environ.get("PORTFOLIO_MODEL_MAP")
|
||||
#: Truthiness, not presence (Fase 4b): ``PORTFOLIO_LIVE_FULL_RUN=`` must NOT arm a paid run.
|
||||
_OPTED_IN = bool(os.environ.get("PORTFOLIO_LIVE_FULL_RUN"))
|
||||
|
||||
_SKIP = not (_ENDPOINT and _DEPLOYMENT and _MODEL_MAP and _OPTED_IN)
|
||||
|
||||
|
||||
@pytest.mark.skipif(
|
||||
_SKIP,
|
||||
reason=(
|
||||
"paid full run not armed (set PORTFOLIO_LIVE_FULL_RUN=1 alongside "
|
||||
"PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT + PORTFOLIO_FOUNDRY_DEPLOYMENT + PORTFOLIO_MODEL_MAP)"
|
||||
),
|
||||
)
|
||||
async def test_full_run_reaches_the_validator_on_a_live_model(tmp_path: Path) -> None:
|
||||
"""The whole slice against a real deployment: every generation reply must come back in the
|
||||
requested shape, and the deterministic validator must decide on it."""
|
||||
outbox_dir = tmp_path / "outbox"
|
||||
run_id = "live-full-001"
|
||||
|
||||
result = await run_project(
|
||||
_PROJECT_ID,
|
||||
"azure",
|
||||
docs_dir=str(BUNDLE_DIR),
|
||||
bundle_dir=str(BUNDLE_DIR),
|
||||
verdict_input={"decision": "approved", "rationale": "expert reviewed (live 1b)"},
|
||||
store=VerdictStore(verdicts=[]),
|
||||
outbox_dir=str(outbox_dir),
|
||||
run_id=run_id,
|
||||
)
|
||||
|
||||
assert isinstance(result, RunResult)
|
||||
assert_full_run_contract(result, outbox_dir, run_id)
|
||||
428
tests/test_handover_package_loadbearing.py
Normal file
428
tests/test_handover_package_loadbearing.py
Normal file
|
|
@ -0,0 +1,428 @@
|
|||
"""Fase 5 — the external handover package: one archive a stranger can deploy into their own
|
||||
Microsoft Foundry with minimal friction.
|
||||
|
||||
**Why an archive and not "clone the repo".** The receiving party is not a contributor: they get a
|
||||
tree, set two environment variables and run it. The package is ``git archive HEAD`` itself — never a
|
||||
hand-curated copy, which would be the second copy that drifts (the kø-(p) rule applied to a
|
||||
deliverable).
|
||||
|
||||
**Runnable Python, no container wrapper (operator directive 14.08, from an external trial).** What a
|
||||
receiver gets is a Python tree they install and start themselves — ``uv sync --frozen`` +
|
||||
``python main.py`` — and the ``Dockerfile``/``azure.yaml`` pair that used to ride along was removed
|
||||
from the tree rather than filtered out of the archive. Filtering would have meant a curation
|
||||
mechanism deciding what a receiver sees, i.e. a SECOND copy of "what is delivered" free to drift
|
||||
from HEAD (the kø-(p) rule, which is the very reason this package is ``git archive HEAD``). Removing
|
||||
the files keeps the archive uncurated and makes the absence a property of HEAD — which is what the
|
||||
gate below can actually measure.
|
||||
|
||||
Four seams are gated here, and each one is a way the handover fails in the receiver's hands rather
|
||||
than in ours:
|
||||
|
||||
1. **Completeness** — a tree missing ``uv.lock`` resolves different versions than every measurement
|
||||
in this repo ran against; missing ``shared/`` gives a receiver with no example knowledge base.
|
||||
2. **Exposure** — ``STATE.md``, ``*.local.md`` and ``.env`` must never leave this machine. The
|
||||
archive is built from tracked files only, so this is a property of the BUILDER; the control below
|
||||
proves the check looks for names that could actually appear.
|
||||
3. **The deploy contract is written down** — the packaged ``model_map.json`` ships
|
||||
``REPLACE-WITH-*`` placeholders and ``backends.py`` fail-fasts on them, so a receiver who sets
|
||||
only an endpoint gets a container that answers ``/readiness`` and fails every ``/invocations``.
|
||||
4e measured that requirement and called it "et deploy-krav ingen rad hadde skrevet ned"; DEPLOY.md
|
||||
is that row, and this test is what keeps it written.
|
||||
4. **Python-only delivery** — the archive carries no container/azd wrapper, and DEPLOY.md starts the
|
||||
service the way the tree actually supports: as a Python process. A gate that merely stopped
|
||||
REQUIRING ``Dockerfile`` could not tell "removed" from "still shipped", so the check is positive
|
||||
(absence, asserted) rather than an omission.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import subprocess
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
_REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
_SCRIPT = _REPO_ROOT / "scripts" / "make-handover-package.sh"
|
||||
|
||||
# Deploy-critical members. Each one is load-bearing for a receiver, not decoration:
|
||||
# the entry point, the locked resolution, the packaged data and the example base. `Dockerfile` and
|
||||
# `azure.yaml` were REQUIRED here until 14.08; they are gone from the tree, and their absence is now
|
||||
# asserted below instead of their presence.
|
||||
_REQUIRED_MEMBERS = (
|
||||
"main.py",
|
||||
"pyproject.toml",
|
||||
"uv.lock",
|
||||
"DEPLOY.md",
|
||||
"src/portfolio_optimiser/hosting.py",
|
||||
"src/portfolio_optimiser/backends.py",
|
||||
"src/portfolio_optimiser/data/model_map.json",
|
||||
"shared/examples/bygg-energi-mikro/index.md",
|
||||
)
|
||||
|
||||
# Names that must NEVER reach a stranger. Local-only continuity, operator config, secrets.
|
||||
_FORBIDDEN_SUFFIXES = (".local.md", ".env")
|
||||
_FORBIDDEN_NAMES = ("STATE.md",)
|
||||
|
||||
# The container/azd wrapper. Matched on archive MEMBER NAMES, never on prose: the documents may
|
||||
# explain that no image is shipped, and a gate that read the word out of a sentence would be red on
|
||||
# exactly the prose it protects (this repo's 08-09 defect class).
|
||||
_CONTAINER_WRAPPER_NAMES = ("Dockerfile", ".dockerignore", "docker-compose.yml", "azure.yaml")
|
||||
|
||||
# The start command a receiver is told to run. With no image CMD left, the document IS the one copy —
|
||||
# and it names the entry point whose serve/SIGTERM behaviour test_hosting_loadbearing measures.
|
||||
_PYTHON_START = "python main.py"
|
||||
|
||||
# Command invocations that would put a container step back into the documented path. Fragments, not
|
||||
# the bare word: "no container image is shipped" must stay sayable.
|
||||
_WRAPPER_COMMANDS = ("docker build", "docker run", "azd up", "azd deploy", "azd provision")
|
||||
|
||||
|
||||
def _members_named(names: list[str], wanted: tuple[str, ...]) -> list[str]:
|
||||
"""Archive members whose basename is one of ``wanted`` — the matcher both the assertion and its
|
||||
control run through, so a matcher that silently matches nothing cannot pass unnoticed."""
|
||||
return [n for n in names if Path(n).name in wanted]
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def package(tmp_path_factory: pytest.TempPathFactory) -> zipfile.ZipFile:
|
||||
"""Build the real package by running the real script — the packaging config is itself a seam
|
||||
(the 4a precedent: an ``uv build`` in the fixture, never a simulated one)."""
|
||||
dest = tmp_path_factory.mktemp("handover")
|
||||
result = subprocess.run(
|
||||
[str(_SCRIPT), str(dest)],
|
||||
cwd=_REPO_ROOT,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
assert result.returncode == 0, f"builder failed: {result.stderr}"
|
||||
archives = sorted(dest.glob("*.zip"))
|
||||
assert len(archives) == 1, f"expected exactly one archive, got {archives}"
|
||||
return zipfile.ZipFile(archives[0])
|
||||
|
||||
|
||||
def test_package_carries_every_deploy_critical_file(package: zipfile.ZipFile) -> None:
|
||||
"""Detach point: drop a member from the archive → RED. A receiver cannot supply what we omit."""
|
||||
names = set(package.namelist())
|
||||
missing = [m for m in _REQUIRED_MEMBERS if m not in names]
|
||||
assert not missing, f"handover package is missing {missing}"
|
||||
|
||||
|
||||
def test_package_leaks_no_local_or_secret_files(package: zipfile.ZipFile) -> None:
|
||||
"""Detach point: build from the working tree instead of tracked files → RED (STATE.md appears).
|
||||
|
||||
The control is the point: a filter that matched nothing would make this gate green forever, so
|
||||
we first prove the archive is populated and that the suffixes we forbid are ones the repo
|
||||
actually produces (``STATE.md`` exists on this machine, untracked-by-design)."""
|
||||
names = package.namelist()
|
||||
assert len(names) > 50, "archive suspiciously small — the check below would be vacuous"
|
||||
assert (_REPO_ROOT / "STATE.md").exists(), (
|
||||
"control: STATE.md must exist locally, else this gate cannot discriminate"
|
||||
)
|
||||
leaked = [
|
||||
n for n in names if Path(n).name in _FORBIDDEN_NAMES or n.endswith(_FORBIDDEN_SUFFIXES)
|
||||
]
|
||||
assert not leaked, f"handover package leaks local-only files: {leaked}"
|
||||
|
||||
|
||||
def test_deploy_doc_names_both_required_env_vars(package: zipfile.ZipFile) -> None:
|
||||
"""Detach point: remove either variable from DEPLOY.md → RED.
|
||||
|
||||
Line-anchored, not substring: ``PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT`` CONTAINS
|
||||
``FOUNDRY_PROJECT_ENDPOINT``, so a naive substring assert on the platform-injected name is
|
||||
satisfied by our own (the 08-09 defect class, measured twice before in this repo)."""
|
||||
doc = package.read("DEPLOY.md").decode("utf-8")
|
||||
lines = doc.splitlines()
|
||||
|
||||
assert any("PORTFOLIO_MODEL_MAP" in line for line in lines), (
|
||||
"DEPLOY.md must name PORTFOLIO_MODEL_MAP — without it the container fail-fasts on the "
|
||||
"REPLACE-WITH-* placeholders"
|
||||
)
|
||||
# The injected name must appear on a line that is NOT merely our own name.
|
||||
injected_lines = [
|
||||
line
|
||||
for line in lines
|
||||
if "FOUNDRY_PROJECT_ENDPOINT" in line.replace("PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT", "")
|
||||
]
|
||||
assert injected_lines, "DEPLOY.md must name the platform-injected FOUNDRY_PROJECT_ENDPOINT"
|
||||
|
||||
|
||||
def test_deploy_doc_states_the_placeholder_requirement(package: zipfile.ZipFile) -> None:
|
||||
"""Detach point: drop the placeholder warning → RED. The packaged map ships REPLACE-WITH-*, so a
|
||||
receiver who is not told will deploy a container that fails every invocation."""
|
||||
doc = package.read("DEPLOY.md").decode("utf-8")
|
||||
packaged_map = package.read("src/portfolio_optimiser/data/model_map.json").decode("utf-8")
|
||||
|
||||
# Control: the requirement is only real while the packaged map actually ships placeholders.
|
||||
assert "REPLACE-WITH-" in packaged_map, (
|
||||
"control: packaged model_map no longer has placeholders — this gate would be vacuous"
|
||||
)
|
||||
assert "REPLACE-WITH-" in doc, (
|
||||
"DEPLOY.md must state that the packaged deployment ids are placeholders"
|
||||
)
|
||||
|
||||
|
||||
def test_package_ships_runnable_python_and_no_container_wrapper(package: zipfile.ZipFile) -> None:
|
||||
"""Detach point: put ``Dockerfile`` or ``azure.yaml`` back into HEAD → RED.
|
||||
|
||||
The 14.08 operator directive is that the deliverable is runnable Python. Removing the wrapper
|
||||
from the TREE (rather than filtering it out of the archive) is what makes that measurable here:
|
||||
the package is ``git archive HEAD``, so absence in the archive IS absence in what we ship.
|
||||
|
||||
Two controls, because a name-matcher that matches nothing would make this green forever:
|
||||
the run path a receiver actually needs must be present, and the matcher must be shown to match
|
||||
a member this archive really has."""
|
||||
names = package.namelist()
|
||||
|
||||
# Control 1 — the Python run path is what replaces the image. If these are missing, "no
|
||||
# container" would just mean "nothing to run".
|
||||
for member in ("main.py", "pyproject.toml", "uv.lock"):
|
||||
assert member in names, f"the runnable-Python path is incomplete: {member} is not packaged"
|
||||
|
||||
# Control 2 — the matcher matches by basename against THIS archive, proven on a member we know
|
||||
# is there. Without it, a matcher comparing full paths would find nothing and pass silently.
|
||||
assert _members_named(names, ("main.py",)) == ["main.py"], (
|
||||
"control: the member matcher found nothing for a member the archive demonstrably has"
|
||||
)
|
||||
|
||||
wrappers = _members_named(names, _CONTAINER_WRAPPER_NAMES)
|
||||
assert not wrappers, (
|
||||
f"the handover package ships a container/azd wrapper: {wrappers}. The delivery is runnable "
|
||||
"Python (operator directive 14.08); a wrapper here is a second, unmeasured way to start it."
|
||||
)
|
||||
|
||||
|
||||
def test_receiver_documents_start_the_service_as_a_python_process(package: zipfile.ZipFile) -> None:
|
||||
"""Detach point: tell the receiver to build an image again in either document → RED.
|
||||
|
||||
Both documents ride inside the archive, so both are the receiver's instructions. Each is checked
|
||||
LINE-ANCHORED with its own positive control first: a negative assertion on a document the
|
||||
extractor failed to read is a gate that can only be green."""
|
||||
deploy = package.read("DEPLOY.md").decode("utf-8").splitlines()
|
||||
readme = package.read("README.md").decode("utf-8").splitlines()
|
||||
|
||||
# Positive control on DEPLOY.md: with no image CMD left, this document carries the ONE copy of
|
||||
# the start command, and it must name the entry point the hosting tests actually exercise.
|
||||
assert [line for line in deploy if _PYTHON_START in line], (
|
||||
f"DEPLOY.md no longer prints the start command ({_PYTHON_START!r}) — the receiver has "
|
||||
"nothing to run, and the one copy of the start command is gone"
|
||||
)
|
||||
# Positive control on README.md: proves this document was read and searched at all.
|
||||
assert [line for line in readme if "uv sync" in line], (
|
||||
"control: README.md has no install line — the check below would be searching nothing"
|
||||
)
|
||||
|
||||
for label, lines in (("DEPLOY.md", deploy), ("README.md", readme)):
|
||||
offenders = [
|
||||
line for line in lines if any(fragment in line for fragment in _WRAPPER_COMMANDS)
|
||||
]
|
||||
assert not offenders, (
|
||||
f"{label} instructs a container/azd build step that this package no longer ships: "
|
||||
f"{offenders}"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The content gap (ordre 20260818T103716Z). Every gate above reads archive MEMBER
|
||||
# NAMES. None reads what the members SAY -- which is exactly how a Foundry resource
|
||||
# group, resource, project and host name reached an external organisation in the
|
||||
# 14:24 build on 14.08 without one of 869 tests noticing. The archive is
|
||||
# ``git archive HEAD`` by construction, so a content gate here is a REFUSAL, never a
|
||||
# filter: it never removes anything from the archive, it says the tree is not
|
||||
# deliverable. That keeps the kø-(p) rule intact -- there is still exactly one copy
|
||||
# of "what a receiver gets", and it is HEAD.
|
||||
#
|
||||
# One list, never patterns spread through the code. Each row carries its own
|
||||
# known-positive SAMPLE, so a pattern that can no longer find anything cannot be
|
||||
# added: ``test_every_content_pattern_can_find`` runs the whole list against its own
|
||||
# samples. Every sample is BUILT BY CONCATENATION so this file's own source bytes
|
||||
# contain no literal match -- otherwise the gate would be red on the file that
|
||||
# defines it, and the only sane fix would be a hole in the gate at exactly the place
|
||||
# a secret could hide.
|
||||
#
|
||||
# HONESTY LIMIT, stated because it decided the design: this gate catches STRUCTURE, so it
|
||||
# catches the host (`<label>.services.ai.azure.com`) and misses the resource group and the
|
||||
# project name, which are free-form strings with no shape to match. In the 14.08 leak those
|
||||
# two were only spotted because they sat in the same table as the host. Closing that gap
|
||||
# would take a name list -- the second copy of the exposure rule, free to drift, which this
|
||||
# package's `git archive HEAD` shape exists to forbid. A structural gate that catches the one
|
||||
# recoverable-by-anyone coordinate is worth more than a name list nobody prunes.
|
||||
#
|
||||
# And it is DELAYED BY ONE COMMIT: `git archive HEAD` reads HEAD, not the working tree, so a
|
||||
# coordinate added in an uncommitted edit is invisible here until it is committed (funn 35,
|
||||
# already true of every other assertion in this file).
|
||||
_GUID_ZERO = "-".join(("0" * 8, "0" * 4, "0" * 4, "0" * 4, "0" * 12))
|
||||
|
||||
_SECRET_CONTENT_PATTERNS: tuple[tuple[str, re.Pattern[str], str], ...] = (
|
||||
(
|
||||
# A tenant's own Foundry/OpenAI host. The label must START with an alphanumeric,
|
||||
# which is what keeps the repo's placeholder (`<resource>.`) and wildcard
|
||||
# (`*.services.ai.azure.com`) forms out: neither `>` nor `*` is a label character.
|
||||
"azure-ai-host",
|
||||
re.compile(
|
||||
r"[A-Za-z0-9][A-Za-z0-9-]*\.(?:services\.ai|openai|cognitiveservices)\.azure\.com"
|
||||
),
|
||||
"https://" + "sample" + ".services.ai.azure.com/api/projects/p",
|
||||
),
|
||||
(
|
||||
# Subscription id in its structural context. A bare GUID is NOT a coordinate --
|
||||
# `53ca6127-db72-4b80-b1b0-d745d6d5456d` is Azure's PUBLIC built-in role definition
|
||||
# id for Foundry User, identical in every tenant, and it is quoted in DEPLOY.md.
|
||||
"arm-subscription-scope",
|
||||
re.compile(r"/subscriptions/[0-9a-fA-F]{8}-[0-9a-fA-F-]{27}"),
|
||||
"/subscriptions/" + _GUID_ZERO + "/resourceGroups/rg",
|
||||
),
|
||||
(
|
||||
# An absolute path into somebody's home directory: a machine layout, and usually a
|
||||
# username with it. Zero on HEAD since 6d2837f; six lines in the delivered 14:24 zip.
|
||||
"absolute-home-path",
|
||||
re.compile(r"/(?:Users|home)/[A-Za-z0-9._-]+/"),
|
||||
"/Users/" + "someone" + "/repos/thing",
|
||||
),
|
||||
(
|
||||
"bearer-jwt",
|
||||
re.compile(r"ey" + r"J[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}"),
|
||||
"ey" + "J" + "abcdefghij.klmnopqrst.uvwxyz0123",
|
||||
),
|
||||
(
|
||||
"vendor-api-key",
|
||||
re.compile(r"\b(?:sk-[A-Za-z0-9]{20,}|gh[pousr]_[A-Za-z0-9]{20,}|AKIA[0-9A-Z]{16})\b"),
|
||||
"sk-" + "A" * 24,
|
||||
),
|
||||
(
|
||||
# Foundry/Cognitive Services keys are 32 hex characters. Git hashes are 7 or 40, and
|
||||
# a 32-run inside a 40-char hash is excluded by the boundary look-arounds.
|
||||
"cognitive-services-key",
|
||||
re.compile(r"(?<![0-9a-fA-F])[0-9a-fA-F]{32}(?![0-9a-fA-F])"),
|
||||
"a" * 32,
|
||||
),
|
||||
(
|
||||
"private-key-block",
|
||||
re.compile(r"-----BEGIN (?:RSA |EC |OPENSSH )?" + "PRIVATE KEY-----"),
|
||||
"-----BEGIN " + "PRIVATE KEY-----",
|
||||
),
|
||||
)
|
||||
|
||||
# Matched text that is deliberately NOT a coordinate. Every entry needs a reason, and every
|
||||
# entry is checked for being STILL REACHED (``test_accepted_content_literals_stay_minimal``)
|
||||
# -- an allowlist nobody prunes is the second copy of the exposure rule, free to drift, which
|
||||
# is the very thing this package's "git archive HEAD" shape exists to forbid.
|
||||
_ACCEPTED_CONTENT_LITERALS: tuple[tuple[str, str], ...] = (
|
||||
("x.services.ai.azure.com", "preflight/backends test dummy: one-letter label, not a resource"),
|
||||
("platform.services.ai.azure.com", "hosted-backend test dummy for the platform-injected value"),
|
||||
("x.openai.azure.com", "preflight test dummy for the WRONG Azure surface"),
|
||||
("wrong.openai.azure.com", "preflight test dummy for the WRONG Azure surface"),
|
||||
)
|
||||
|
||||
# The commit the external organisation actually received (the 14:24 build of 14.08). It is the
|
||||
# known-positive control on REAL data: the scanner has to find what did leak, not only what a
|
||||
# synthetic sample can be made to contain.
|
||||
_LEAKED_COMMIT = "77076b9"
|
||||
|
||||
|
||||
def _scan_text(text: str) -> list[tuple[str, str]]:
|
||||
"""Return ``(pattern label, matched text)`` for everything in ``text`` that is not on the
|
||||
accepted list. Detection only -- nothing is rewritten, nothing is removed."""
|
||||
findings: list[tuple[str, str]] = []
|
||||
for label, pattern, _sample in _SECRET_CONTENT_PATTERNS:
|
||||
for match in pattern.finditer(text):
|
||||
hit = match.group(0)
|
||||
if any(hit in accepted for accepted, _why in _ACCEPTED_CONTENT_LITERALS):
|
||||
continue
|
||||
findings.append((label, hit))
|
||||
return findings
|
||||
|
||||
|
||||
def _scan_archive(archive: zipfile.ZipFile) -> tuple[list[str], int, int]:
|
||||
"""Scan every member of ``archive``. Returns ``(findings, files read, files skipped)``.
|
||||
|
||||
The denominator is returned, not discarded: "nothing found" over an unknown number of files
|
||||
is not zero, it is unmeasured."""
|
||||
findings: list[str] = []
|
||||
read = skipped = 0
|
||||
for name in archive.namelist():
|
||||
if name.endswith("/"):
|
||||
continue
|
||||
try:
|
||||
text = archive.read(name).decode("utf-8")
|
||||
except UnicodeDecodeError:
|
||||
skipped += 1
|
||||
continue
|
||||
read += 1
|
||||
findings.extend(f"{name}: {label}: {hit}" for label, hit in _scan_text(text))
|
||||
return findings, read, skipped
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def leaked_package(tmp_path_factory: pytest.TempPathFactory) -> zipfile.ZipFile:
|
||||
"""The archive as it was at the commit that reached an external organisation."""
|
||||
dest = tmp_path_factory.mktemp("leaked") / "leaked.zip"
|
||||
subprocess.run(
|
||||
["git", "archive", "--format=zip", "--output", str(dest), _LEAKED_COMMIT],
|
||||
cwd=_REPO_ROOT,
|
||||
check=True,
|
||||
capture_output=True,
|
||||
)
|
||||
return zipfile.ZipFile(dest)
|
||||
|
||||
|
||||
def test_every_content_pattern_can_find(package: zipfile.ZipFile) -> None:
|
||||
"""Control: each row of the pattern list is proven able to match before any of them is
|
||||
trusted to report nothing. A pattern that silently matches nothing makes the gate below
|
||||
green forever -- this repo's recurring vacuity class."""
|
||||
for label, pattern, sample in _SECRET_CONTENT_PATTERNS:
|
||||
assert pattern.search(sample), f"pattern {label!r} does not match its own sample"
|
||||
|
||||
# And the samples must not be literal in this file, or the gate would be red on itself and
|
||||
# the only fix would be excluding the file that defines the patterns.
|
||||
own_source = package.read("tests/test_handover_package_loadbearing.py").decode("utf-8")
|
||||
assert not _scan_text(own_source), "the pattern samples are literal in this file's source"
|
||||
|
||||
|
||||
def test_content_scan_finds_what_actually_leaked(leaked_package: zipfile.ZipFile) -> None:
|
||||
"""Control on REAL data: point the scanner at the tree the external organisation received
|
||||
and it must report the Foundry coordinates and the absolute home paths. Without this the
|
||||
gate below would only prove that HEAD is clean, never that the scanner can see."""
|
||||
findings, read, _skipped = _scan_archive(leaked_package)
|
||||
assert read > 300, f"only {read} members read from the leaked archive -- control is vacuous"
|
||||
|
||||
labels = {finding.split(": ")[1] for finding in findings}
|
||||
assert "azure-ai-host" in labels, f"the leaked Foundry host was not found; got {findings}"
|
||||
assert "absolute-home-path" in labels, f"the leaked home paths were not found; got {findings}"
|
||||
|
||||
|
||||
def test_package_leaks_no_secret_content(package: zipfile.ZipFile) -> None:
|
||||
"""The gate. Detach point: put a tenant coordinate, an ARM subscription scope, an absolute
|
||||
home path or a credential back into any tracked file -> RED.
|
||||
|
||||
Refusal, not filtering: what fails here is the TREE, and the fix is to change HEAD."""
|
||||
findings, read, skipped = _scan_archive(package)
|
||||
assert read > 300, f"only {read} members read -- this gate would be reporting on nothing"
|
||||
assert skipped <= 1, f"unexpectedly many undecodable members ({skipped}) went unscanned"
|
||||
assert not findings, (
|
||||
f"the handover package leaks secret content ({read} members read, {skipped} skipped): "
|
||||
f"{findings}"
|
||||
)
|
||||
|
||||
|
||||
def test_accepted_content_literals_stay_minimal(package: zipfile.ZipFile) -> None:
|
||||
"""Every accepted literal must still be reachable in the archive. An entry that matches
|
||||
nothing is a standing exemption for a string the repo no longer has -- the second copy of
|
||||
the exposure rule, quietly drifting away from what is actually shipped."""
|
||||
corpus = "\n".join(
|
||||
archive_text
|
||||
for name in package.namelist()
|
||||
if not name.endswith("/")
|
||||
for archive_text in _decoded(package, name)
|
||||
)
|
||||
stale = [literal for literal, _why in _ACCEPTED_CONTENT_LITERALS if literal not in corpus]
|
||||
assert not stale, f"accepted content literals no longer occur in the package: {stale}"
|
||||
|
||||
|
||||
def _decoded(archive: zipfile.ZipFile, name: str) -> list[str]:
|
||||
try:
|
||||
return [archive.read(name).decode("utf-8")]
|
||||
except UnicodeDecodeError:
|
||||
return []
|
||||
|
|
@ -33,6 +33,9 @@ _PROVENANCE = ProvenanceStamp(
|
|||
role="proposer",
|
||||
validator_decision="validated",
|
||||
token_usage=8,
|
||||
# These fixtures stand in for an ordinary complete run; the road path is anchored by
|
||||
# construction, so ``True`` is the honest value here. The un-anchored case has its own file.
|
||||
cost_baseline_anchored=True,
|
||||
)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -31,6 +31,9 @@ _PROVENANCE = ProvenanceStamp(
|
|||
role="proposer",
|
||||
validator_decision="validated",
|
||||
token_usage=8,
|
||||
# These fixtures stand in for an ordinary complete run; the road path is anchored by
|
||||
# construction, so ``True`` is the honest value here. The un-anchored case has its own file.
|
||||
cost_baseline_anchored=True,
|
||||
)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -18,11 +18,11 @@ Load-bearing surface pinned here:
|
|||
* The server is asyncio on the ONE loop (NG1: ``test_no_thread_or_process_path_exists_under_src``
|
||||
ratchets src/ thread-free) — these tests run client and server as coroutines on the SAME loop,
|
||||
which only works because nothing in the server blocks it.
|
||||
* Root ``main.py`` is the ONE process entry (Dockerfile CMD + azure.yaml point at it): the
|
||||
* Root ``main.py`` is the ONE process entry (``python main.py``, the command DEPLOY.md prints): the
|
||||
subprocess test is the ONLY test that catches a detached shim or a detached SIGTERM handler
|
||||
(P4-presedensen: entry-point-mutasjoner fanges aldri av in-process-tester).
|
||||
|
||||
Fase 4e closes two gaps the above leaves open, and both are about things a stand-in cannot see:
|
||||
Fase 4e closed two gaps the above leaves open, and both are about things a stand-in cannot see:
|
||||
|
||||
* **The whitelist composes with the REAL ``run_project``.** Every test above hands ``invoke`` a
|
||||
stand-in that swallows ``**kwargs``, so the whitelist could name a field ``run_project`` does not
|
||||
|
|
@ -33,14 +33,11 @@ Fase 4e closes two gaps the above leaves open, and both are about things a stand
|
|||
PURPOSE (a caller must never choose the server's model client), so patching the factory the run
|
||||
falls back to is the only injection point this surface leaves — the same argument
|
||||
``test_run_cli_loadbearing`` makes for ``main()``.
|
||||
* **The deployment artifacts are raw-text-gated.** ``Dockerfile`` and ``azure.yaml`` are the two
|
||||
files that decide whether the image the platform pulls can run at all, and NO test executes
|
||||
them here (``docker build``/``azd deploy`` are operator-gated). A raw-text gate is therefore the
|
||||
only mechanism available: it pins ``--platform linux/amd64`` (measured required — spike §1.4;
|
||||
an arm64 image built on this Intel-free-of-charge assumption would fail only in the cloud) and
|
||||
the ONE-copy rule for the start command (the image's ``CMD``; ``azure.yaml`` carries no
|
||||
``startupCommand`` to drift from it). Guard-tester leser kildefiler som RÅ TEKST — reformulate
|
||||
the prose around them, never the strings they pin.
|
||||
* **The deployment artifacts were raw-text-gated** — until 14.08, when the operator directive
|
||||
after an external trial made the delivery runnable Python and the two artifacts were removed
|
||||
from the tree. The gate is deleted, not weakened; see the note where it stood, below the
|
||||
end-to-end test. The start command now has exactly one copy left, in DEPLOY.md, and
|
||||
``tests/test_handover_package_loadbearing.py`` is what keeps it there.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
|
@ -101,6 +98,9 @@ _PROVENANCE = ProvenanceStamp(
|
|||
role="proposer",
|
||||
validator_decision="validated",
|
||||
token_usage=8,
|
||||
# These fixtures stand in for an ordinary complete run; the road path is anchored by
|
||||
# construction, so ``True`` is the honest value here. The un-anchored case has its own file.
|
||||
cost_baseline_anchored=True,
|
||||
)
|
||||
_VALIDATED = ValidatedProposal(
|
||||
proposal=_PROPOSAL, p10=100.0, p50=150.0, p90=200.0, nominal_feasible=180.0
|
||||
|
|
@ -234,7 +234,9 @@ async def test_invocations_wires_payload_into_run_project(
|
|||
assert body["outcome_type"] == "validated"
|
||||
assert body["p90"] == 200.0
|
||||
assert body["checker_verdict"] == "approve"
|
||||
assert body["verdict_id"] == "vid-hosted"
|
||||
# F2: the response quotes the candidate's verdict KEY, not the captured verdict's id — the two
|
||||
# coincide on a real run that recorded one, and only the key survives a run nobody reviewed.
|
||||
assert body["verdict_id"] == _result(_VALIDATED).verdict_key
|
||||
assert body["proposal"]["measure"] == "LED-retrofit av kontorbelysning"
|
||||
assert body["provenance"]["model"] == "synthetic"
|
||||
|
||||
|
|
@ -271,7 +273,9 @@ async def test_unknown_field_is_refused_never_repaired(
|
|||
assert len(recorder.calls) == 1
|
||||
|
||||
|
||||
@pytest.mark.parametrize("missing", ["project_id", "docs_dir", "verdict_input"])
|
||||
# ``verdict_input`` is deliberately ABSENT from this list since F2: it is optional, and the arms
|
||||
# proving the relaxation live in ``tests/test_ungiven_verdict_loadbearing.py``.
|
||||
@pytest.mark.parametrize("missing", ["project_id", "docs_dir"])
|
||||
async def test_missing_required_field_is_400(
|
||||
served: str, monkeypatch: pytest.MonkeyPatch, missing: str
|
||||
) -> None:
|
||||
|
|
@ -320,8 +324,11 @@ async def test_contract_violation_is_400_and_run_failure_is_500(
|
|||
) -> None:
|
||||
"""ValueError (pydantic contract violations subclass it) is the CALLER's error → 400; any
|
||||
other failure is an honest 500 carrying {error_type, error} (mirrors RunFailure's shape).
|
||||
BudgetExceeded is RuntimeError, so it lands in the 500 arm — with observed != limit so the
|
||||
two can never be conflated by an echo (kø-(y))."""
|
||||
|
||||
The 500 witness is a NON-budget RuntimeError on purpose. It used to be ``BudgetExceeded``,
|
||||
which is what made this test the one that pinned exhaustion to the crash channel; the two
|
||||
now have separate arms, and this half is what keeps the budget arm NARROW — RED if it is
|
||||
widened to catch ``Exception`` and route every failure to 429."""
|
||||
monkeypatch.setattr(
|
||||
hosting, "run_project", _Recorder(error=ValueError("docs_dir does not exist"))
|
||||
)
|
||||
|
|
@ -329,12 +336,61 @@ async def test_contract_violation_is_400_and_run_failure_is_500(
|
|||
assert status == 400
|
||||
assert "docs_dir does not exist" in body["error"]
|
||||
|
||||
monkeypatch.setattr(hosting, "run_project", _Recorder(error=BudgetExceeded("tokens", 100, 173)))
|
||||
monkeypatch.setattr(
|
||||
hosting, "run_project", _Recorder(error=RuntimeError("chat client fell over"))
|
||||
)
|
||||
status, body = await _post(served, "/invocations", _PAYLOAD)
|
||||
assert status == 500
|
||||
assert body["error_type"] == "BudgetExceeded"
|
||||
assert "limit=100" in body["error"]
|
||||
assert "173" in body["error"]
|
||||
assert body["error_type"] == "RuntimeError"
|
||||
assert "chat client fell over" in body["error"]
|
||||
assert "budget_exhausted" not in body
|
||||
|
||||
|
||||
async def test_budget_exhaustion_is_not_the_failure_channel(
|
||||
served: str, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""Budget exhaustion is a DESIGNED terminal state — the cap firing IS the feature working
|
||||
(``Budget``: fail-fast, never an unbounded loop) — so it must not share a channel with a
|
||||
crash. The first live run died exactly here (``rounds limit=12 observed=13``) and the hosted
|
||||
surface answered 500, i.e. the same thing it says when the model endpoint falls over.
|
||||
|
||||
* NOT 500: nothing broke.
|
||||
* NOT 200: unlike a ``Rejection`` — which is a run that CONCLUDED, and therefore belongs in
|
||||
the payload — an exhausted budget produced no proposal at all. A 2xx would let an automated
|
||||
caller record "analysed" for a run that analysed nothing.
|
||||
* 429: the condition arises from an ALLOWANCE (``max_rounds``/``max_tokens`` are whitelisted
|
||||
request fields, and the raise is the caller's own remedy), never from a server fault.
|
||||
* ``error_type`` is ABSENT: that key belongs to the failure channel, and a caller switching
|
||||
on its presence must not find it on a run that did not fail.
|
||||
|
||||
RED when the arm is detached (falls through to 500) or relabelled to any other status."""
|
||||
monkeypatch.setattr(hosting, "run_project", _Recorder(error=BudgetExceeded("rounds", 12, 13)))
|
||||
|
||||
status, body = await _post(served, "/invocations", _PAYLOAD)
|
||||
|
||||
assert status == 429
|
||||
assert "error_type" not in body
|
||||
|
||||
|
||||
async def test_budget_stop_triple_survives_as_structure(
|
||||
served: str, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""kø-(y): ``kind``/``limit``/``observed`` describe ONE ledger and are ONE structured stop
|
||||
event. ``str(exc)`` flattens them into prose the caller has to re-parse to learn WHICH cap
|
||||
bound and how far past it the run got — which is the whole operational question (raise
|
||||
``max_rounds``? raise ``max_tokens``? give up?).
|
||||
|
||||
Built with ``observed != limit`` deliberately: at an exactly-exhausted cap the two coincide,
|
||||
and a test written there cannot tell a faithful implementation from one that echoes the limit
|
||||
back as the observed value. RED when the payload carries only the message string."""
|
||||
monkeypatch.setattr(hosting, "run_project", _Recorder(error=BudgetExceeded("rounds", 12, 13)))
|
||||
|
||||
status, body = await _post(served, "/invocations", _PAYLOAD)
|
||||
|
||||
assert status == 429
|
||||
assert body["budget_exhausted"] == {"kind": "rounds", "limit": 12, "observed": 13}
|
||||
# The human-readable line stays alongside the structure — an operator reading a log needs it.
|
||||
assert body["error"] == "budget exceeded: rounds limit=12 observed=13"
|
||||
|
||||
|
||||
async def test_readiness_answers_while_an_invocation_is_in_flight(
|
||||
|
|
@ -381,8 +437,8 @@ def _blocking_get(url: str) -> tuple[int, str]:
|
|||
|
||||
|
||||
def test_main_entrypoint_serves_and_stops_on_sigterm() -> None:
|
||||
"""Root main.py is the ONE process entry (Dockerfile CMD + azure.yaml point at it): started
|
||||
as a subprocess it must serve /readiness and exit 0 on SIGTERM. This is the only test that
|
||||
"""Root main.py is the ONE process entry (``python main.py``, the command DEPLOY.md prints):
|
||||
started as a subprocess it must serve /readiness and exit 0 on SIGTERM. This is the only test that
|
||||
catches a shim that stops calling hosting.main() or a detached SIGTERM handler."""
|
||||
with socket.socket() as probe:
|
||||
probe.bind(("127.0.0.1", 0))
|
||||
|
|
@ -415,7 +471,7 @@ def test_main_entrypoint_serves_and_stops_on_sigterm() -> None:
|
|||
proc.wait()
|
||||
|
||||
|
||||
# --- Fase 4e: the whitelist against the REAL run_project, and the deployment artifacts ----------
|
||||
# --- Fase 4e: the whitelist against the REAL run_project -----------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
|
|
@ -455,8 +511,15 @@ async def test_invocations_answers_through_the_real_run_project(
|
|||
to break that are invisible to a recorder and fatal in a container: naming a field
|
||||
``run_project`` does not take, and passing it an argument twice. Both surface here as a 500.
|
||||
|
||||
The payload names EVERY whitelisted field, and the coverage assertion below is what keeps that
|
||||
true: a field added to the whitelist later cannot slip past this test unexercised."""
|
||||
The payload names every FORWARDED field, and the coverage assertion below is what keeps that
|
||||
true: a field added to the whitelist later cannot slip past this test unexercised.
|
||||
|
||||
The whitelist became a THREE-way partition when U4's ``explore_prompt`` arrived: those fields
|
||||
are accepted by the surface and CONSUMED by it (the exploration runs first and hands
|
||||
``run_project`` a mandate), so they can never satisfy a "reaches ``run_project``" assertion.
|
||||
The consumed half has its own, negative proof in
|
||||
``tests/test_explore_callsites_loadbearing.py`` — without it a field sliding from consumed to
|
||||
forwarded is exactly the drift 4e exists to catch."""
|
||||
payload = {
|
||||
"project_id": _BUNDLE_PID,
|
||||
"docs_dir": str(_BUNDLE_DIR),
|
||||
|
|
@ -470,10 +533,14 @@ async def test_invocations_answers_through_the_real_run_project(
|
|||
"max_tokens": 100_000,
|
||||
"top_k": 3,
|
||||
}
|
||||
assert set(payload) == hosting._ALLOWED_FIELDS, (
|
||||
"the payload must exercise every whitelisted field — a field the whitelist accepts but "
|
||||
assert set(payload) == set(hosting._REQUIRED_FIELDS) | set(hosting._OPTIONAL_FIELDS), (
|
||||
"the payload must exercise every FORWARDED field — a field the whitelist forwards but "
|
||||
"this test never sends is a field no test proves ``run_project`` accepts"
|
||||
)
|
||||
assert hosting._ALLOWED_FIELDS == set(payload) | set(hosting._CONSUMED_FIELDS), (
|
||||
"the whitelist is a three-way partition and this arm covers the forwarded half; a field "
|
||||
"in neither half would be accepted by the surface with nothing proving what it does"
|
||||
)
|
||||
|
||||
status, body = await _post(served, "/invocations", payload)
|
||||
|
||||
|
|
@ -491,51 +558,9 @@ async def test_invocations_answers_through_the_real_run_project(
|
|||
assert _scripted_backend, "the scripted backend was never called — no real run happened"
|
||||
|
||||
|
||||
def test_deployment_artifacts_pin_the_measured_platform_and_one_start_command() -> None:
|
||||
"""RAW-TEXT GATE on the two files that decide whether the hosted image runs at all.
|
||||
|
||||
Neither is executed by any test: ``docker build`` and ``azd deploy`` are operator-gated, so a
|
||||
regression in them is invisible to the whole suite until it fails in the cloud. What the gate
|
||||
pins is exactly what was MEASURED, and nothing about the prose around it:
|
||||
|
||||
* ``--platform linux/amd64`` — the platform requires x86_64 (spike §1.4). Dropping it makes the
|
||||
image inherit the builder's architecture, which on an arm64 laptop yields an image that
|
||||
builds green locally and cannot start in the cloud. The flag lives in the documented build
|
||||
command (a Dockerfile cannot set the build platform for its own invocation), so pinning the
|
||||
documented string is the only gate available — and it is worth having precisely because
|
||||
nothing else re-derives it.
|
||||
* ONE copy of the start command: the image's ``CMD`` names ``main.py``, and ``azure.yaml``
|
||||
declares NO ``startupCommand``. Two copies drift (kø-(p)-regelen); this is the pair that
|
||||
keeps there being one.
|
||||
|
||||
``env:`` is checked for the same reason: ``FOUNDRY_PROJECT_ENDPOINT`` is injected by the
|
||||
platform, and redeclaring it here could shadow the injected value — which is the failure mode
|
||||
Fase 4b's endpoint precedence exists to avoid, undone from the config side."""
|
||||
dockerfile = _REPO_ROOT / "Dockerfile"
|
||||
azure_yaml = _REPO_ROOT / "azure.yaml"
|
||||
assert dockerfile.is_file(), "the hosted image has no build definition"
|
||||
assert azure_yaml.is_file(), "azd has no project definition to deploy"
|
||||
|
||||
docker_text = dockerfile.read_text(encoding="utf-8")
|
||||
azure_text = azure_yaml.read_text(encoding="utf-8")
|
||||
|
||||
assert "--platform linux/amd64" in docker_text, (
|
||||
"the Dockerfile no longer names the measured build platform; an image built without it "
|
||||
"inherits the builder's architecture and cannot start on the hosting platform"
|
||||
)
|
||||
assert "CMD" in docker_text and "main.py" in docker_text, (
|
||||
"the image's CMD is the ONE copy of the start command and must name main.py"
|
||||
)
|
||||
# Line-anchored: a mention inside a comment is prose, a top-level key is a declaration.
|
||||
azure_keys = [line.split(":")[0].strip() for line in azure_text.splitlines()]
|
||||
assert "startupCommand" not in azure_keys, (
|
||||
"azure.yaml declares a startupCommand — a SECOND copy of the start command, free to drift "
|
||||
"from the image's CMD"
|
||||
)
|
||||
assert "env" not in azure_keys, (
|
||||
"azure.yaml declares an env block — FOUNDRY_PROJECT_ENDPOINT is injected by the platform "
|
||||
"and must never be redeclared here"
|
||||
)
|
||||
assert "host: azure.ai.agent" in azure_text and "kind: hosted" in azure_text, (
|
||||
"azure.yaml no longer declares the hosted-agent host this whole entrypoint targets"
|
||||
)
|
||||
# The 4e raw-text gate on ``Dockerfile``/``azure.yaml`` lived here until 14.08. It pinned
|
||||
# ``--platform linux/amd64`` and the one-copy rule for the image's ``CMD``. Both files were removed
|
||||
# from the tree that day (operator directive: the delivery is runnable Python), and a gate that pins
|
||||
# a surface we no longer ship is deleted with it rather than weakened into something that can only
|
||||
# be green. What replaced it lives in ``tests/test_handover_package_loadbearing.py``: the package
|
||||
# must carry NO container/azd wrapper and must document the Python start command.
|
||||
|
|
|
|||
77
tests/test_ingest_stamp_fail_closed_loadbearing.py
Normal file
77
tests/test_ingest_stamp_fail_closed_loadbearing.py
Normal file
|
|
@ -0,0 +1,77 @@
|
|||
"""RED-FIRST: the ingest ownership stamp check must be FAIL-CLOSED on the *value*, not just on
|
||||
field presence. Before this fix ``_carries_complete_ingest_stamp`` compared ``generated`` against
|
||||
the single literal string ``"true"`` — a YAML-1.1 truthy form such as ``yes``/``on`` slipped the
|
||||
gate open even though a real YAML reader reads it as boolean ``True`` (measured with PyYAML's
|
||||
``safe_load`` core-schema resolver, the same resolver family ``ruamel``/most YAML tooling ships:
|
||||
``yes``/``no``/``true``/``false``/``on``/``off``, any case, are read as bool; bare ``y``/``n`` and
|
||||
``1``/``0`` are NOT — they resolve to string/int). ``okf.py`` stays a hand-rolled, PyYAML-free
|
||||
line parser (D7 constraint), so the vocabulary is a plain literal set, not a real parser call."""
|
||||
|
||||
import pytest
|
||||
|
||||
from portfolio_optimiser import okf
|
||||
|
||||
|
||||
def test_yaml_truthy_form_yes_is_recognized_as_the_stamp(tmp_path) -> None:
|
||||
"""``generated: yes`` is read as boolean True by a real YAML reader, so it is the SAME stamp
|
||||
as ``generated: true`` and must be refused identically."""
|
||||
fm = {"type": "reference", "generated": "yes", "ingest_manifest": "bygg@0123456789abcdef"}
|
||||
with pytest.raises(okf.IngestStampError):
|
||||
okf.write_concept_file(str(tmp_path), "forged.md", fm, "body\n")
|
||||
assert not (tmp_path / "forged.md").exists() # refused, not silently repaired
|
||||
|
||||
|
||||
def test_yaml_truthy_form_on_mixed_case_is_recognized_as_the_stamp(tmp_path) -> None:
|
||||
"""Case-insensitivity must hold for every truthy form, not only ``true``/``True``/``TRUE``."""
|
||||
fm = {"type": "reference", "generated": "On", "ingest_manifest": "bygg@0123456789abcdef"}
|
||||
with pytest.raises(okf.IngestStampError):
|
||||
okf.write_concept_file(str(tmp_path), "forged.md", fm, "body\n")
|
||||
assert not (tmp_path / "forged.md").exists()
|
||||
|
||||
|
||||
def test_literal_true_still_refused_control(tmp_path) -> None:
|
||||
"""Control: the literal the gate has always recognized must still raise — otherwise the two
|
||||
assertions above could pass against a gate that raises unconditionally, proving nothing."""
|
||||
fm = {"type": "reference", "generated": "true", "ingest_manifest": "bygg@0123456789abcdef"}
|
||||
with pytest.raises(okf.IngestStampError):
|
||||
okf.write_concept_file(str(tmp_path), "forged.md", fm, "body\n")
|
||||
assert not (tmp_path / "forged.md").exists()
|
||||
|
||||
|
||||
def test_non_yaml_boolean_forms_are_not_treated_as_the_stamp(tmp_path) -> None:
|
||||
"""``1`` and bare ``y`` are NOT read as boolean True by a real YAML reader (measured: PyYAML
|
||||
resolves them to int ``1`` and string ``"y"``, never bool) — widening the true-vocabulary to
|
||||
include them would over-block curated content no ingest pipeline ever produces, and no YAML
|
||||
reader would read as the stamp either. Both write through unchanged, same as any other
|
||||
unrecognized ``generated`` value paired with a manifest."""
|
||||
for value in ("1", "y", "maybe"):
|
||||
name = f"curated-{value}.md"
|
||||
okf.write_concept_file(
|
||||
str(tmp_path),
|
||||
name,
|
||||
{"type": "reference", "generated": value, "ingest_manifest": "bygg@0123456789abcdef"},
|
||||
"body\n",
|
||||
)
|
||||
assert (tmp_path / name).exists()
|
||||
|
||||
|
||||
def test_half_stamp_remains_legal_generated_alone(tmp_path) -> None:
|
||||
"""CLAUDE.md's ingest-stamp invariant: either half of the stamp alone is still legal curated
|
||||
content — even when that half is written in a non-``true`` truthy form."""
|
||||
okf.write_concept_file(str(tmp_path), "a.md", {"type": "reference", "generated": "yes"}, "b\n")
|
||||
assert (tmp_path / "a.md").exists()
|
||||
|
||||
|
||||
def test_half_stamp_remains_legal_manifest_alone(tmp_path) -> None:
|
||||
okf.write_concept_file(
|
||||
str(tmp_path),
|
||||
"b.md",
|
||||
{"type": "reference", "ingest_manifest": "bygg@0123456789abcdef"},
|
||||
"c\n",
|
||||
)
|
||||
assert (tmp_path / "b.md").exists()
|
||||
|
||||
|
||||
def test_unstamped_file_written_as_before(tmp_path) -> None:
|
||||
okf.write_concept_file(str(tmp_path), "c.md", {"type": "reference"}, "d\n")
|
||||
assert (tmp_path / "c.md").exists()
|
||||
146
tests/test_live_full_run_contract.py
Normal file
146
tests/test_live_full_run_contract.py
Normal file
|
|
@ -0,0 +1,146 @@
|
|||
"""The Fase 1b full-run contract must DISCRIMINATE — proven offline, for free (Fase 1b, last step).
|
||||
|
||||
``tests/test_full_run_live.py`` spends real money on a real endpoint, exactly once, and cannot be
|
||||
run red-then-green: the paid call is the MEASUREMENT, not the proof that the measuring instrument
|
||||
works. This file is that proof, and it costs nothing.
|
||||
|
||||
The repo's own rule — *"en test som ikke kan skille to implementasjoner beviser ingenting"* — has
|
||||
been violated by this project's own measuring instruments three times (økt 27's slide sweep read
|
||||
``section``'s 100vh and reported the same number for all eleven slides; økt 37's T3 asserted on a
|
||||
schema the scripted client ignores). An assertion that can only ever pass is the same defect class,
|
||||
and a LIVE assertion is the worst place to discover it, because a green result there is precisely
|
||||
what the operator would act on.
|
||||
|
||||
So both arms drive the SAME ``assert_full_run_contract`` helper the live test uses (ONE copy, in
|
||||
``conftest`` — a second copy would drift, kø-(p)), over the SAME bundle and the SAME project as the
|
||||
live run, through the canonical ``ScriptedChatClient``'s ``reply_selector`` seam:
|
||||
|
||||
* T1 — a run in which ONE reply failed to parse must make the contract FAIL. This is the arm that
|
||||
matters: it is the offline stand-in for "the live endpoint ignored ``response_format``", which is
|
||||
the single outcome the paid run exists to rule out. Per the repo rule for negative asserts, it
|
||||
first PROVES the event happened (the artefact exists) rather than inferring it from the failure.
|
||||
* T2 — the CONTROL: a run in which every reply parsed must make the contract PASS. Without it, a
|
||||
helper that raised unconditionally would satisfy T1 and the live test could then only ever be red,
|
||||
which is the mirror-image vacuity.
|
||||
|
||||
Together they show the contract keys on the ACTUAL discriminator (the parse-failure artefact), not
|
||||
on something both runs share.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from conftest import assert_full_run_contract
|
||||
|
||||
from portfolio_optimiser.budget import Budget, TokenMeter
|
||||
from portfolio_optimiser.run import RunResult, run_project
|
||||
from portfolio_optimiser.simulation import ScriptedChatClient
|
||||
from portfolio_optimiser.verdicts import VerdictStore
|
||||
|
||||
BUNDLE_DIR = Path(__file__).resolve().parents[1] / "shared" / "examples" / "bygg-energi-mikro"
|
||||
_PROJECT_ID = "BYGG-KONTOR-NORD"
|
||||
_VERDICT_INPUT = {"decision": "approved", "rationale": "expert reviewed (contract probe)"}
|
||||
|
||||
#: The line ``generate._build_messages`` puts in EVERY generation prompt and nowhere else — the one
|
||||
#: identifier separating a generation call from a debate turn (borrowed from
|
||||
#: ``test_parse_failure_capture_loadbearing``, whose seam this file shares).
|
||||
_GENERATION_MARK = "Respond with ONLY a JSON object"
|
||||
|
||||
#: Prose where an object was requested: what a model that ignored ``response_format`` returns.
|
||||
_MALFORMED = (
|
||||
"Sure! Here is what I found for this project.\n\n"
|
||||
"The main opportunity looks like demand-controlled lighting, worth roughly 30 000 NOK.\n"
|
||||
"Let me know if you want that as JSON."
|
||||
)
|
||||
|
||||
#: BYGG-KONTOR-NORD: affected total 300000 x 1.0 -> degenerate Monte Carlo P90 = 90000, so a claim
|
||||
#: of 30000 validates (same arithmetic as ``test_parse_failure_capture_loadbearing``).
|
||||
_VALID_REPLY = (
|
||||
'{"measure":"Behovsstyrt belysning i fellesarealer","affected_items":'
|
||||
'[{"code":"ENERGI-TOTAL-EL","quantity":300000,"unit_cost":1.0}],'
|
||||
'"claimed_saving_nok":30000}'
|
||||
)
|
||||
|
||||
|
||||
def _malformed_then_valid(failures: int) -> Callable[[str, str], str]:
|
||||
"""Fail to parse ``failures`` times, then answer with a proposal that validates. Keyed on the
|
||||
GENERATION prompt only, so debate turns — which are never parsed — do not consume the counter."""
|
||||
seen = {"n": 0}
|
||||
|
||||
def _select(blob: str, _role: str) -> str:
|
||||
if _GENERATION_MARK not in blob:
|
||||
return "ok"
|
||||
seen["n"] += 1
|
||||
return _MALFORMED if seen["n"] <= failures else _VALID_REPLY
|
||||
|
||||
return _select
|
||||
|
||||
|
||||
async def _run(select: Callable[[str, str], str], outbox_dir: Path, run_id: str) -> RunResult:
|
||||
def factory(role: str) -> ScriptedChatClient:
|
||||
return ScriptedChatClient(role=role, reply_selector=select, default_reply="ok")
|
||||
|
||||
result = await run_project(
|
||||
_PROJECT_ID,
|
||||
"local",
|
||||
docs_dir=str(BUNDLE_DIR),
|
||||
bundle_dir=str(BUNDLE_DIR),
|
||||
verdict_input=_VERDICT_INPUT,
|
||||
store=VerdictStore(verdicts=[]),
|
||||
client_factory=factory,
|
||||
outbox_dir=str(outbox_dir),
|
||||
run_id=run_id,
|
||||
meter=TokenMeter(Budget(max_tokens=10**9, max_rounds=8)),
|
||||
)
|
||||
assert isinstance(result, RunResult)
|
||||
return result
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------------------------
|
||||
# T1 — the arm that matters: an unparsed reply must FAIL the contract.
|
||||
# --------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_contract_fails_when_a_reply_did_not_parse(tmp_path: Path) -> None:
|
||||
"""The offline stand-in for "the live endpoint ignored the schema". The run itself SUCCEEDS —
|
||||
the second reply validates — so the contract cannot be keying on the run's outcome; the only
|
||||
thing separating this from T2 is the parse-failure artefact.
|
||||
|
||||
RED (i.e. this test fails) on a contract that ignores the artefact and merely checks that a
|
||||
RunResult came back — which is what ``test_portfolio_live.py``'s ``len(runs) == 1`` does, and
|
||||
the reason that existing gated test could not carry this claim."""
|
||||
outbox_dir = tmp_path / "outbox"
|
||||
run_id = "contract-dirty"
|
||||
|
||||
result = await _run(_malformed_then_valid(1), outbox_dir, run_id)
|
||||
|
||||
# Prove the event happened FIRST — a negative assert that merely observes a failure cannot tell
|
||||
# "the contract caught the artefact" from "the contract failed for some unrelated reason".
|
||||
artefact = outbox_dir / f"{run_id}-parse-failures.json"
|
||||
assert artefact.exists(), "precondition: this arm must actually produce a parse failure"
|
||||
assert result.provenance.validator_decision in {"validated", "rejected"}, (
|
||||
"precondition: the run must otherwise CONCLUDE, so the artefact is the only difference"
|
||||
)
|
||||
|
||||
with pytest.raises(AssertionError, match="did NOT honour the structured schema"):
|
||||
assert_full_run_contract(result, outbox_dir, run_id)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------------------------
|
||||
# T2 — the CONTROL: a clean run must PASS.
|
||||
# --------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_contract_passes_when_every_reply_parsed(tmp_path: Path) -> None:
|
||||
"""Without this control, a helper that raised unconditionally would satisfy T1, and the live
|
||||
test could then only ever be red — the mirror image of a check that can only ever be green."""
|
||||
outbox_dir = tmp_path / "outbox"
|
||||
run_id = "contract-clean"
|
||||
|
||||
result = await _run(_malformed_then_valid(0), outbox_dir, run_id)
|
||||
|
||||
assert not (outbox_dir / f"{run_id}-parse-failures.json").exists()
|
||||
assert_full_run_contract(result, outbox_dir, run_id)
|
||||
733
tests/test_multibase_loadbearing.py
Normal file
733
tests/test_multibase_loadbearing.py
Normal file
|
|
@ -0,0 +1,733 @@
|
|||
"""U4 + U13, del 3 — MULTI-BASE (``Approach.bundle_id``, plan § C.7).
|
||||
|
||||
**The premise this file corrects, measured before anything was built.** § C.7 and økt 56's own
|
||||
honesty limit read as though the deliverable were "``run_project`` accepts more than one
|
||||
``bundle_dir``". It cannot, and the refusal is structural rather than stylistic: on the bundle
|
||||
path ``run_project`` derives FOUR single-valued things from THE bundle — the project
|
||||
(``_project_from_bundle``, which fail-fasts when ``validator-input.json``'s ``project_id`` is not
|
||||
the requested one), the validator's stage-0 cost baseline (S4.0's whole point being that the gate
|
||||
is anchored to THAT project's real cost lines), the agents' read context, and the ExpeL query key
|
||||
— and it returns ONE ``RunResult`` with ONE ``ProvenanceStamp``. A second ``bundle_dir`` would
|
||||
force a silent pick-one for all four, which is the guessed-shape class this repo refuses.
|
||||
|
||||
§ C.7's own sentence says the same thing once read closely: *"pipelinen kjøres per bundle som i dag
|
||||
(``run_portfolio``-formen)"* — N calls, one per base, not one call taking N. So the delivered shape
|
||||
is: each approach RECORDS its base, a pure router PARTITIONS the mandate by base, and a thin
|
||||
dispatch runs the existing ``run_project`` once per base. **No existing caller's signature changes**
|
||||
— CLI, hosting and simulation each still pass one base, and each still may.
|
||||
|
||||
Three seams, each with its own detach signature:
|
||||
|
||||
* ``mandate.Approach.bundle_id`` — the field, defaulting to ``""`` so every mandate written before
|
||||
this session stays valid;
|
||||
* ``mandate.route_by_bundle`` — the partition, fail-fast on a commission that cannot be executed as
|
||||
written (``load_mandate``'s rule: a run must never proceed on a silently degraded commission);
|
||||
* ``run.run_mandate_across_bundles`` — the dispatch, whose ``project_id`` per base comes from THAT
|
||||
base's own IR projection and never from a caller-supplied constant.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import shutil
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from agent_framework import BaseChatClient
|
||||
|
||||
from portfolio_optimiser import explore, run as run_module
|
||||
from portfolio_optimiser.budget import BudgetRefused, PortfolioBudget, PortfolioMeter
|
||||
from portfolio_optimiser.explore import ExplorationContract, ExplorationError, HypothesisParseError
|
||||
from portfolio_optimiser.mandate import (
|
||||
Approach,
|
||||
Mandate,
|
||||
MandateRoutingError,
|
||||
route_by_bundle,
|
||||
)
|
||||
from portfolio_optimiser.run import RunResult, run_mandate_across_bundles
|
||||
from portfolio_optimiser.simulation import ScriptedChatClient
|
||||
from portfolio_optimiser.verdicts import VerdictStore
|
||||
|
||||
_EXAMPLES = Path(__file__).resolve().parents[1] / "shared" / "examples"
|
||||
#: Three bases with three DISTINCT project ids — which is what makes "the project comes from the
|
||||
#: base, not from the caller" a claim a test can actually falsify.
|
||||
_BYGG = _EXAMPLES / "bygg-energi-mikro" # BYGG-KONTOR-NORD
|
||||
_TUNNEL = _EXAMPLES / "tunnel-hauglia" # TUNNEL-HAUGLIA
|
||||
_VEGLYS = _EXAMPLES / "veglys-fv-soer" # VEGLYS-FV-SOER
|
||||
|
||||
_VERDICT_INPUT = {"decision": "approved", "rationale": "expert reviewed (sim)"}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# 1. The field. mandate.py stays pydantic + stdlib (D7-portable, test_okf_is_maf_free).
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_an_approach_records_which_knowledge_base_it_belongs_to() -> None:
|
||||
"""T1: ``Approach`` carries ``bundle_id``, and it DEFAULTS to the empty string.
|
||||
|
||||
The default is what keeps every mandate JSON written before this session valid, and every
|
||||
existing ``Approach(...)`` constructor call unaffected — the same reason ``RunResult.coverage``
|
||||
defaults. Empty means "no base named", which is a legitimate statement when the run has only
|
||||
one base to name.
|
||||
"""
|
||||
assert Approach(id="a", label="A").bundle_id == ""
|
||||
assert Approach(id="a", label="A", bundle_id="tunnel-hauglia").bundle_id == "tunnel-hauglia"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# 2. The router. Pure, framework-neutral, and fail-fast on a commission it cannot execute.
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _mandate(*approaches: Approach, own: bool = True) -> Mandate:
|
||||
return Mandate(objective="find savings", approaches=approaches, allow_own_proposals=own)
|
||||
|
||||
|
||||
def test_the_router_partitions_the_mandate_one_sub_mandate_per_named_base() -> None:
|
||||
"""T2: each base gets a sub-mandate carrying ONLY its own approaches, in ``bundle_ids`` order.
|
||||
|
||||
Order is taken from the configured bases rather than from first appearance among the
|
||||
approaches, so the dispatch's spend order is a property of how the run was configured and not
|
||||
of how a model happened to sequence its hypotheses.
|
||||
"""
|
||||
a = Approach(id="a", label="A", bundle_id="tunnel-hauglia")
|
||||
b = Approach(id="b", label="B", bundle_id="veglys-fv-soer")
|
||||
c = Approach(id="c", label="C", bundle_id="tunnel-hauglia")
|
||||
|
||||
routed = route_by_bundle(_mandate(a, b, c), ("veglys-fv-soer", "tunnel-hauglia"))
|
||||
|
||||
assert [bundle_id for bundle_id, _ in routed] == ["veglys-fv-soer", "tunnel-hauglia"]
|
||||
assert [ap.id for ap in routed[0][1].approaches] == ["b"]
|
||||
assert [ap.id for ap in routed[1][1].approaches] == ["a", "c"]
|
||||
# The commission's own fields travel with every partition: each sub-run is still working on the
|
||||
# same objective and under the same "and/or your own" permission.
|
||||
assert routed[0][1].objective == "find savings"
|
||||
assert routed[0][1].allow_own_proposals is True
|
||||
|
||||
|
||||
def test_a_single_base_absorbs_every_unassigned_approach() -> None:
|
||||
"""T3: with exactly ONE base configured, an approach naming none routes to it.
|
||||
|
||||
Not a guess — with one configured base there is no other value the field could take, and
|
||||
``_bundle_index`` already guarantees the id is unique. This is what keeps a single-base
|
||||
mandate (every mandate that exists today) dispatchable unchanged.
|
||||
"""
|
||||
routed = route_by_bundle(_mandate(Approach(id="a", label="A")), ("tunnel-hauglia",))
|
||||
|
||||
assert [bundle_id for bundle_id, _ in routed] == ["tunnel-hauglia"]
|
||||
assert [ap.id for ap in routed[0][1].approaches] == ["a"]
|
||||
|
||||
|
||||
def test_an_unassigned_approach_among_several_bases_is_refused_not_guessed() -> None:
|
||||
"""T4: the discriminator for T3 — with TWO bases, an unnamed approach REFUSES the dispatch.
|
||||
|
||||
Silently sending it to the first base would evaluate a commissioned direction against a
|
||||
project nobody asked about and report it as done. Refusing is ``load_mandate``'s rule applied
|
||||
one layer on: a run must never proceed on a silently degraded commission, because the coverage
|
||||
report would then describe work nobody ordered.
|
||||
"""
|
||||
with pytest.raises(MandateRoutingError) as exc:
|
||||
route_by_bundle(_mandate(Approach(id="a", label="A")), ("tunnel-hauglia", "veglys-fv-soer"))
|
||||
|
||||
# The REPR, never the bare letter: "a" is a substring of almost any English sentence, so an
|
||||
# assertion on it would hold against a refusal raised for an entirely different reason.
|
||||
assert "'a'" in str(exc.value)
|
||||
|
||||
|
||||
def test_an_approach_naming_an_unconfigured_base_is_refused_by_name() -> None:
|
||||
"""T5: an approach whose ``bundle_id`` matches no configured base refuses, naming both.
|
||||
|
||||
Resolving it by position instead would be the S3.2 key-collision class: an approach evaluated
|
||||
against a base it does not belong to, with nothing in the report saying so.
|
||||
"""
|
||||
approach = Approach(id="a", label="A", bundle_id="does-not-exist")
|
||||
with pytest.raises(MandateRoutingError) as exc:
|
||||
route_by_bundle(_mandate(approach), ("tunnel-hauglia", "veglys-fv-soer"))
|
||||
|
||||
message = str(exc.value)
|
||||
assert "does-not-exist" in message
|
||||
assert "tunnel-hauglia" in message
|
||||
|
||||
|
||||
def test_the_routing_refusal_is_a_value_error() -> None:
|
||||
"""T6: ``MandateRoutingError`` subclasses ``ValueError`` — a TYPE claim, not a taxonomy note.
|
||||
|
||||
økt 57 measured the cost of getting this wrong the other way: ``ExplorationError`` is a
|
||||
``RuntimeError`` and therefore fell outside ``run.main``'s ``(ValueError, FileNotFoundError,
|
||||
ValidationError)`` refusal tuple and outside hosting's 400 arm, so a caller's configuration
|
||||
mistake would have left as a traceback and a 500. A routing refusal is exactly that class of
|
||||
caller mistake, so it is born inside both nets rather than retrofitted into them.
|
||||
"""
|
||||
assert issubclass(MandateRoutingError, ValueError)
|
||||
|
||||
|
||||
def test_routing_against_no_base_at_all_is_refused() -> None:
|
||||
"""T7: zero configured bases refuses rather than returning an empty plan.
|
||||
|
||||
An empty plan reads as "there was nothing to do", which is indistinguishable from a mandate
|
||||
that was fully evaluated against nothing — the omitted-row silence ``ApproachOutcome``'s
|
||||
``not_evaluated`` status exists to remove.
|
||||
"""
|
||||
with pytest.raises(MandateRoutingError):
|
||||
route_by_bundle(_mandate(Approach(id="a", label="A")), ())
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# 3. Assignment: what the exploration puts in the field, and what it refuses to put there.
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
_CONTRACT = ExplorationContract(
|
||||
max_rounds=6,
|
||||
max_tokens=100_000,
|
||||
max_stall_count=2,
|
||||
max_reset_count=1,
|
||||
max_plan_revisions=0,
|
||||
enable_plan_review=False,
|
||||
)
|
||||
|
||||
_PROMPT = "Find the cheapest saving available."
|
||||
|
||||
|
||||
def _ledger_json(*, satisfied: bool, speaker: str) -> str:
|
||||
return json.dumps(
|
||||
{
|
||||
"is_request_satisfied": {"reason": "r", "answer": satisfied},
|
||||
"is_in_loop": {"reason": "r", "answer": False},
|
||||
"is_progress_being_made": {"reason": "r", "answer": True},
|
||||
"next_speaker": {"reason": "r", "answer": speaker},
|
||||
"instruction_or_question": {"reason": "r", "answer": "Shape one hypothesis."},
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _manager_script(ledgers: list[str]) -> Callable[[str, str], str]:
|
||||
"""Route a manager prompt blob to its scripted reply. The ORDER of these branches is
|
||||
load-bearing and was measured in økt 56 (§ F, A6): the selector receives the CONCATENATION of
|
||||
every message in the call, so a later-stage prompt still carries the earlier stage's text."""
|
||||
|
||||
def _select(blob: str, _role: str) -> str:
|
||||
if "provide the final answer" in blob:
|
||||
return "FINAL: exploration done."
|
||||
if "pure JSON format" in blob:
|
||||
return ledgers.pop(0) if ledgers else _ledger_json(satisfied=True, speaker="navigator")
|
||||
if "went wrong on this last run" in blob:
|
||||
return "PLAN-UPDATE: revised plan."
|
||||
if "rewrite the following fact sheet" in blob:
|
||||
return "FACTS-UPDATE: revised facts."
|
||||
if "bullet-point plan" in blob:
|
||||
return "PLAN: - ask the hypothesiser"
|
||||
if "pre-survey" in blob:
|
||||
return "FACTS: the bundle is anchored."
|
||||
return "{}"
|
||||
|
||||
return _select
|
||||
|
||||
|
||||
def _factory(
|
||||
*, ledgers: list[str], hypothesiser: list[str], sink: list[str] | None = None
|
||||
) -> Callable[[str], BaseChatClient]:
|
||||
"""One fresh ``ScriptedChatClient`` per role. ``sink`` records every prompt that reached a
|
||||
client, which is how "refused BEFORE the first model call" becomes measurable rather than
|
||||
merely asserted at the exit code."""
|
||||
|
||||
def factory(role: str) -> BaseChatClient:
|
||||
if role == explore.MANAGER_ROLE:
|
||||
return ScriptedChatClient(reply_selector=_manager_script(ledgers), role=role, sink=sink)
|
||||
if role == explore.HYPOTHESISER_ROLE:
|
||||
replies = list(hypothesiser)
|
||||
|
||||
def _hyp(_blob: str, _role: str) -> str:
|
||||
return replies.pop(0) if replies else "nothing further."
|
||||
|
||||
return ScriptedChatClient(reply_selector=_hyp, role=role, sink=sink)
|
||||
return ScriptedChatClient("NAVIGATOR: index read.", role=role, sink=sink)
|
||||
|
||||
return factory
|
||||
|
||||
|
||||
def _hypothesis_line(label: str, rationale: str, bundle_id: str | None = None) -> str:
|
||||
payload: dict[str, Any] = {"label": label, "rationale": rationale}
|
||||
if bundle_id is not None:
|
||||
payload["bundle_id"] = bundle_id
|
||||
return f"{explore.HYPOTHESIS_MARKER} " + json.dumps(payload)
|
||||
|
||||
|
||||
def _two_round_ledgers() -> list[str]:
|
||||
return [
|
||||
_ledger_json(satisfied=False, speaker=explore.HYPOTHESISER_ROLE),
|
||||
_ledger_json(satisfied=True, speaker=explore.HYPOTHESISER_ROLE),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_marked_hypothesis_may_name_its_base_and_the_mandate_carries_it() -> None:
|
||||
"""T8: the marker's ``bundle_id`` reaches ``Approach.bundle_id``.
|
||||
|
||||
Asserted DIRECTLY on the returned mandate rather than through anything downstream: with a
|
||||
single base configured a router that dropped the field would route identically, so a
|
||||
behavioural assertion here could not tell an implementation that assigns from one that does
|
||||
not. Two bases are configured for the same reason.
|
||||
"""
|
||||
result = await explore.explore(
|
||||
_PROMPT,
|
||||
contract=_CONTRACT,
|
||||
bundle_dirs=(str(_TUNNEL), str(_VEGLYS)),
|
||||
client_factory=_factory(
|
||||
ledgers=_two_round_ledgers(),
|
||||
hypothesiser=[_hypothesis_line("LED retrofit", "old fixtures", "veglys-fv-soer")],
|
||||
),
|
||||
)
|
||||
|
||||
assert [a.bundle_id for a in result.mandate.approaches] == ["veglys-fv-soer"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_with_one_base_a_marker_that_names_none_still_yields_an_assigned_approach() -> None:
|
||||
"""T9: one base configured, marker silent — the minted approach carries that base's id.
|
||||
|
||||
Again asserted on the FIELD. This is the arm that would go quietly green under an
|
||||
implementation that never assigns anything, which is precisely why T8 configures two bases and
|
||||
why neither test reads the field through the router.
|
||||
"""
|
||||
result = await explore.explore(
|
||||
_PROMPT,
|
||||
contract=_CONTRACT,
|
||||
bundle_dirs=(str(_TUNNEL),),
|
||||
client_factory=_factory(
|
||||
ledgers=_two_round_ledgers(),
|
||||
hypothesiser=[_hypothesis_line("LED retrofit", "old fixtures")],
|
||||
),
|
||||
)
|
||||
|
||||
assert [a.bundle_id for a in result.mandate.approaches] == ["tunnel-hauglia"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_with_several_bases_a_marker_that_names_none_is_refused() -> None:
|
||||
"""T10: the discriminator for T9 — two bases and a silent marker is a hard error.
|
||||
|
||||
A marked line is a claim the loop committed to; one that cannot be routed is a claim it could
|
||||
not finish making. That is the ``write_concept_file`` rule (validation, never repair) and NOT
|
||||
the tolerant RAW-inbox rule, because this is the product of the run.
|
||||
"""
|
||||
with pytest.raises(HypothesisParseError):
|
||||
await explore.explore(
|
||||
_PROMPT,
|
||||
contract=_CONTRACT,
|
||||
bundle_dirs=(str(_TUNNEL), str(_VEGLYS)),
|
||||
client_factory=_factory(
|
||||
ledgers=_two_round_ledgers(),
|
||||
hypothesiser=[_hypothesis_line("LED retrofit", "old fixtures")],
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_marker_naming_an_unconfigured_base_is_refused() -> None:
|
||||
"""T11: a ``bundle_id`` matching no configured base refuses — ``_resolve_bundle``'s own rule.
|
||||
|
||||
The navigator's tools already refuse an unknown base id rather than resolving it by order; a
|
||||
hypothesis that names one must not be treated more leniently than a read of one.
|
||||
"""
|
||||
with pytest.raises(ExplorationError):
|
||||
await explore.explore(
|
||||
_PROMPT,
|
||||
contract=_CONTRACT,
|
||||
bundle_dirs=(str(_TUNNEL), str(_VEGLYS)),
|
||||
client_factory=_factory(
|
||||
ledgers=_two_round_ledgers(),
|
||||
hypothesiser=[_hypothesis_line("LED", "old fixtures", "no-such-base")],
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_seed_naming_an_unknown_base_is_refused_before_the_first_model_call() -> None:
|
||||
"""T12: an unroutable SEED refuses with ZERO model calls made.
|
||||
|
||||
The assertion is on the sink, not on the exception, and that is the økt-57 outbox-hoist
|
||||
precedent: at the exception alone, a refusal AFTER the exploration has spent its whole budget
|
||||
looks identical to one before. The expert's mandate cannot be dispatched either way — what is
|
||||
at stake is whether they pay to find out.
|
||||
"""
|
||||
sink: list[str] = []
|
||||
with pytest.raises(ExplorationError):
|
||||
await explore.explore(
|
||||
_PROMPT,
|
||||
contract=_CONTRACT,
|
||||
bundle_dirs=(str(_TUNNEL), str(_VEGLYS)),
|
||||
seed_approaches=(Approach(id="s1", label="Seed", bundle_id="no-such-base"),),
|
||||
client_factory=_factory(ledgers=_two_round_ledgers(), hypothesiser=[], sink=sink),
|
||||
)
|
||||
|
||||
assert sink == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_an_unassigned_seed_with_several_bases_is_refused_before_the_first_call() -> None:
|
||||
"""T13: a seed naming no base, with more than one configured, refuses — also before spending.
|
||||
|
||||
Same defect as T10 one door earlier (§ C.6 door 1 rather than the loop's own findings). The
|
||||
expert configured several bases; which one their hypothesis belongs to is a thing only they
|
||||
know, and inventing it would put their name on a direction they did not commission.
|
||||
"""
|
||||
sink: list[str] = []
|
||||
with pytest.raises(ExplorationError):
|
||||
await explore.explore(
|
||||
_PROMPT,
|
||||
contract=_CONTRACT,
|
||||
bundle_dirs=(str(_TUNNEL), str(_VEGLYS)),
|
||||
seed_approaches=(Approach(id="s1", label="Seed"),),
|
||||
client_factory=_factory(ledgers=_two_round_ledgers(), hypothesiser=[], sink=sink),
|
||||
)
|
||||
|
||||
assert sink == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_seed_is_never_rewritten_only_validated() -> None:
|
||||
"""T14: with one base, an unassigned seed comes back EXACTLY as it was written.
|
||||
|
||||
§ C.6 door 1 is a preservation rule: the seed is the expert's own words, and the mandate hands
|
||||
``description`` to the proposer verbatim. Filling in ``bundle_id`` on their behalf would be
|
||||
repair of an input, which this repo refuses even when the repair is obviously right — the
|
||||
single-base default belongs to the ROUTER, at consumption, where it is unambiguous by
|
||||
construction.
|
||||
"""
|
||||
seed = Approach(id="s1", label="Night setback", description="the expert's own words")
|
||||
result = await explore.explore(
|
||||
_PROMPT,
|
||||
contract=_CONTRACT,
|
||||
bundle_dirs=(str(_TUNNEL),),
|
||||
seed_approaches=(seed,),
|
||||
client_factory=_factory(ledgers=_two_round_ledgers(), hypothesiser=[]),
|
||||
)
|
||||
|
||||
assert result.mandate.approaches[0] == seed
|
||||
assert result.mandate.approaches[0].bundle_id == ""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# 4. The dispatch. One run_project per base — the existing function, composed, never widened.
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _recorder(
|
||||
calls: list[dict[str, Any]], *, meter: PortfolioMeter | None = None, charge: int = 0
|
||||
) -> Callable[..., Any]:
|
||||
"""A stand-in for ``run_project`` that records exactly the kwargs it was called with.
|
||||
|
||||
Deliberately paired with ``test_the_dispatch_composes_with_the_real_run_project`` below: a
|
||||
recorder that swallows ``**kwargs`` proves the dispatch's ROUTING but would happily accept an
|
||||
argument ``run_project`` does not take — the Fase 4e defect, one layer up.
|
||||
|
||||
``charge`` credits the global ledger the way a real run does (its ``TokenMeter`` is bound to the
|
||||
same ``PortfolioMeter``). Without it the remainder never moves, and an admission check made once
|
||||
per base could not be told apart from one made once per pass.
|
||||
"""
|
||||
|
||||
async def _fake(project_id: str, profile: Any = "local", **kwargs: Any) -> Any:
|
||||
calls.append({"project_id": project_id, "profile": profile, **kwargs})
|
||||
if meter is not None and charge:
|
||||
meter.record(charge)
|
||||
return _stub_result()
|
||||
|
||||
return _fake
|
||||
|
||||
|
||||
def _stub_result() -> Any:
|
||||
class _Stub:
|
||||
coverage: tuple[Any, ...] = ()
|
||||
provenance = None
|
||||
|
||||
return _Stub()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_dispatch_runs_one_pipeline_per_base_with_that_bases_approaches(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""T15: two bases -> two ``run_project`` calls, each bound to ITS base and ITS approaches.
|
||||
|
||||
This is § C.7's "pipelinen kjøres per bundle" made falsifiable. A dispatch that collapsed the
|
||||
partition would still return a result — one run, one outcome — and nothing else in the suite
|
||||
would notice.
|
||||
"""
|
||||
calls: list[dict[str, Any]] = []
|
||||
monkeypatch.setattr(run_module, "run_project", _recorder(calls))
|
||||
|
||||
mandate = _mandate(
|
||||
Approach(id="a", label="A", bundle_id="tunnel-hauglia"),
|
||||
Approach(id="b", label="B", bundle_id="veglys-fv-soer"),
|
||||
)
|
||||
await run_mandate_across_bundles(
|
||||
mandate,
|
||||
(str(_TUNNEL), str(_VEGLYS)),
|
||||
verdict_input=_VERDICT_INPUT,
|
||||
store=VerdictStore(verdicts=[]),
|
||||
)
|
||||
|
||||
assert len(calls) == 2
|
||||
assert [c["bundle_dir"] for c in calls] == [str(_TUNNEL), str(_VEGLYS)]
|
||||
assert [[ap.id for ap in c["mandate"].approaches] for c in calls] == [["a"], ["b"]]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_each_bases_project_id_comes_from_that_base_not_from_the_caller(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""T16: the ``project_id`` per call is read from THAT base's own IR projection.
|
||||
|
||||
The dispatch takes no ``project_id`` argument at all, and that is the design point rather than
|
||||
an omission: ``_project_from_bundle`` already fail-fasts when a bundle's ``project_id`` is not
|
||||
the requested one, so a caller-supplied constant could only ever be right for one of N bases.
|
||||
Turning that existing fail-fast into the routing key removes the guess entirely.
|
||||
"""
|
||||
calls: list[dict[str, Any]] = []
|
||||
monkeypatch.setattr(run_module, "run_project", _recorder(calls))
|
||||
|
||||
mandate = _mandate(
|
||||
Approach(id="a", label="A", bundle_id="tunnel-hauglia"),
|
||||
Approach(id="b", label="B", bundle_id="veglys-fv-soer"),
|
||||
)
|
||||
await run_mandate_across_bundles(
|
||||
mandate,
|
||||
(str(_TUNNEL), str(_VEGLYS)),
|
||||
verdict_input=_VERDICT_INPUT,
|
||||
store=VerdictStore(verdicts=[]),
|
||||
)
|
||||
|
||||
assert [c["project_id"] for c in calls] == ["TUNNEL-HAUGLIA", "VEGLYS-FV-SOER"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_one_base_is_dispatched_exactly_as_a_single_run(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""T17: the control — a single base makes exactly ONE call, carrying the whole mandate.
|
||||
|
||||
Without this arm T15 could pass on an implementation that fanned out over configured bases
|
||||
regardless of what the mandate said, spending a run on every base a caller merely made
|
||||
available.
|
||||
"""
|
||||
calls: list[dict[str, Any]] = []
|
||||
monkeypatch.setattr(run_module, "run_project", _recorder(calls))
|
||||
|
||||
await run_mandate_across_bundles(
|
||||
_mandate(Approach(id="a", label="A"), Approach(id="b", label="B")),
|
||||
(str(_TUNNEL),),
|
||||
verdict_input=_VERDICT_INPUT,
|
||||
store=VerdictStore(verdicts=[]),
|
||||
)
|
||||
|
||||
assert len(calls) == 1
|
||||
assert [ap.id for ap in calls[0]["mandate"].approaches] == ["a", "b"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_base_that_cannot_be_funded_is_never_started_and_its_approaches_are_reported(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""T18: the S3.4 admission tooth — an unfundable base is NEVER STARTED, and it SAYS SO.
|
||||
|
||||
Two halves, and both are load-bearing. Never-started is the point: a base that is merely
|
||||
aborted mid-run has already cost real model calls. And reporting is the ``not_evaluated``
|
||||
rule — an approach the pass never reached must appear as unreached, because an omitted row is
|
||||
indistinguishable from an approach nobody commissioned.
|
||||
"""
|
||||
calls: list[dict[str, Any]] = []
|
||||
meter = PortfolioMeter(PortfolioBudget(max_total_tokens=1_000, max_tokens_per_run=500))
|
||||
monkeypatch.setattr(run_module, "run_project", _recorder(calls, meter=meter, charge=600))
|
||||
|
||||
result = await run_mandate_across_bundles(
|
||||
_mandate(
|
||||
Approach(id="a", label="A", bundle_id="tunnel-hauglia"),
|
||||
Approach(id="b", label="B", bundle_id="veglys-fv-soer"),
|
||||
),
|
||||
(str(_TUNNEL), str(_VEGLYS)),
|
||||
verdict_input=_VERDICT_INPUT,
|
||||
store=VerdictStore(verdicts=[]),
|
||||
portfolio_meter=meter,
|
||||
)
|
||||
|
||||
# Base 1 is funded (1000 left, 500 required) and spends 600; base 2 then has 400 left against a
|
||||
# 500 reserve. The SECOND base is the one that must never start.
|
||||
assert [c["bundle_dir"] for c in calls] == [str(_TUNNEL)]
|
||||
assert result.stopped_early is True
|
||||
assert result.budget_stop is not None
|
||||
assert {row.id for row in result.unreached} == {"b"}
|
||||
assert all(row.status == "not_evaluated" for row in result.unreached)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_pass_that_can_fund_nothing_at_all_is_refused_at_startup() -> None:
|
||||
"""T19: a global remainder below one run's reserve raises ``BudgetRefused`` before anything
|
||||
loads — ``run_portfolio``'s startup refusal, same primitive, same reason: a pass that has room
|
||||
for zero runs is a caller error, not a result.
|
||||
"""
|
||||
meter = PortfolioMeter(
|
||||
PortfolioBudget(max_total_tokens=1_000, max_tokens_per_run=500), spent=1_000
|
||||
)
|
||||
with pytest.raises(BudgetRefused):
|
||||
await run_mandate_across_bundles(
|
||||
_mandate(Approach(id="a", label="A")),
|
||||
(str(_TUNNEL),),
|
||||
verdict_input=_VERDICT_INPUT,
|
||||
store=VerdictStore(verdicts=[]),
|
||||
portfolio_meter=meter,
|
||||
)
|
||||
|
||||
|
||||
def _reply_for(code: str, quantity: float, unit_cost: float, claimed: int) -> str:
|
||||
return json.dumps(
|
||||
{
|
||||
"measure": "Redusert omfang",
|
||||
"affected_items": [{"code": code, "quantity": quantity, "unit_cost": unit_cost}],
|
||||
"claimed_saving_nok": claimed,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_dispatch_composes_with_the_real_run_project(tmp_path: Path) -> None:
|
||||
"""T20: every kwarg the dispatch passes is a REAL ``run_project`` parameter.
|
||||
|
||||
The Fase 4e proof, one layer up: every routing test above uses a stand-in that swallows
|
||||
``**kwargs``, so the dispatch could name an argument ``run_project`` does not take — or pass
|
||||
one twice — and not a single one of them would notice, while a live call raised ``TypeError``.
|
||||
Driven against two real bases end to end, offline, through the scripted client seam.
|
||||
"""
|
||||
bases = []
|
||||
for src in (_BYGG, _TUNNEL):
|
||||
dst = tmp_path / src.name
|
||||
shutil.copytree(src, dst)
|
||||
bases.append(str(dst))
|
||||
|
||||
def factory(role: str) -> BaseChatClient:
|
||||
return ScriptedChatClient(
|
||||
reply_selector=lambda _blob, _role: _reply_for("ENERGI-TOTAL-EL", 300000, 1.0, 30_000),
|
||||
role=role,
|
||||
)
|
||||
|
||||
result = await run_mandate_across_bundles(
|
||||
_mandate(
|
||||
Approach(id="a", label="A", bundle_id="bygg-energi-mikro"),
|
||||
Approach(id="b", label="B", bundle_id="tunnel-hauglia"),
|
||||
),
|
||||
tuple(bases),
|
||||
"local",
|
||||
verdict_input=_VERDICT_INPUT,
|
||||
store=VerdictStore(verdicts=[]),
|
||||
client_factory=factory,
|
||||
max_rounds=1,
|
||||
)
|
||||
|
||||
assert [r.bundle_id for r in result.runs] == ["bygg-energi-mikro", "tunnel-hauglia"]
|
||||
assert [r.project_id for r in result.runs] == ["BYGG-KONTOR-NORD", "TUNNEL-HAUGLIA"]
|
||||
assert all(isinstance(r.result, RunResult) for r in result.runs)
|
||||
# Each run answered for ITS OWN commissioned approach, and for nobody else's.
|
||||
assert [{row.id for row in r.result.coverage} for r in result.runs] == [
|
||||
{"a", "own-proposal"},
|
||||
{"b", "own-proposal"},
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_one_store_is_threaded_across_every_base(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""T21: the SAME ``VerdictStore`` instance reaches every base's run.
|
||||
|
||||
The cross-base learning claim, and the same one ``run_portfolio`` makes across projects: a
|
||||
verdict captured while evaluating base k must be able to reach base k+1's hypothesis. A fresh
|
||||
store per base would leave the loop looking wired while carrying nothing between the runs.
|
||||
|
||||
**Asserted on IDENTITY, and that is a correction the mutation forced** (this repo's vacuous-gate
|
||||
class, eighth occurrence). The first version compared with ``==``, and ``VerdictStore`` is a
|
||||
pydantic model with VALUE equality — so a mutation handing every base its own
|
||||
``VerdictStore(verdicts=[])`` left the whole suite green: three distinct empty stores are all
|
||||
equal to one another. Sharing an instance is the actual claim, so ``is`` is the actual test.
|
||||
"""
|
||||
calls: list[dict[str, Any]] = []
|
||||
monkeypatch.setattr(run_module, "run_project", _recorder(calls))
|
||||
|
||||
store = VerdictStore(verdicts=[])
|
||||
await run_mandate_across_bundles(
|
||||
_mandate(
|
||||
Approach(id="a", label="A", bundle_id="tunnel-hauglia"),
|
||||
Approach(id="b", label="B", bundle_id="veglys-fv-soer"),
|
||||
),
|
||||
(str(_TUNNEL), str(_VEGLYS)),
|
||||
verdict_input=_VERDICT_INPUT,
|
||||
store=store,
|
||||
)
|
||||
|
||||
assert len(calls) == 2
|
||||
assert all(c["store"] is store for c in calls)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_exploration_and_dispatch_close_the_loop_over_two_bases(tmp_path: Path) -> None:
|
||||
"""T22: the end-to-end witness for § C.7 — prompt + TWO bases -> mandate -> two pipelines.
|
||||
|
||||
Every test above pins one seam. This one is the only place the three meet: the hypothesiser
|
||||
shapes two directions and names a DIFFERENT base for each, ``route_by_bundle`` partitions them,
|
||||
and each base's own ``run_project`` answers for its own hypothesis and for nobody else's. Every
|
||||
reply is scripted, so what is shown is that the plumbing closes — never that a live model would
|
||||
shape either direction well (the demo's §1 honesty limit, unchanged).
|
||||
|
||||
Assertion is on the COVERAGE rows rather than on call arguments, because that is the report an
|
||||
expert actually reads: an approach that reached the wrong base would still appear evaluated.
|
||||
"""
|
||||
bases = []
|
||||
for src in (_BYGG, _TUNNEL):
|
||||
dst = tmp_path / src.name
|
||||
shutil.copytree(src, dst)
|
||||
bases.append(str(dst))
|
||||
|
||||
exploration = await explore.explore(
|
||||
_PROMPT,
|
||||
contract=_CONTRACT,
|
||||
bundle_dirs=tuple(bases),
|
||||
client_factory=_factory(
|
||||
ledgers=[
|
||||
_ledger_json(satisfied=False, speaker=explore.HYPOTHESISER_ROLE),
|
||||
_ledger_json(satisfied=True, speaker=explore.HYPOTHESISER_ROLE),
|
||||
],
|
||||
hypothesiser=[
|
||||
_hypothesis_line("Behovsstyrt lys", "fixtures are 1990s", "bygg-energi-mikro")
|
||||
+ "\n"
|
||||
+ _hypothesis_line("Nattsenking", "the tunnel runs lit all night", "tunnel-hauglia")
|
||||
],
|
||||
),
|
||||
)
|
||||
|
||||
assert [a.bundle_id for a in exploration.mandate.approaches] == [
|
||||
"bygg-energi-mikro",
|
||||
"tunnel-hauglia",
|
||||
]
|
||||
|
||||
def factory(role: str) -> BaseChatClient:
|
||||
return ScriptedChatClient(
|
||||
reply_selector=lambda _blob, _role: _reply_for("ENERGI-TOTAL-EL", 300000, 1.0, 30_000),
|
||||
role=role,
|
||||
)
|
||||
|
||||
result = await run_mandate_across_bundles(
|
||||
exploration.mandate,
|
||||
tuple(bases),
|
||||
"local",
|
||||
verdict_input=_VERDICT_INPUT,
|
||||
store=VerdictStore(verdicts=[]),
|
||||
client_factory=factory,
|
||||
max_rounds=1,
|
||||
)
|
||||
|
||||
assert [r.bundle_id for r in result.runs] == ["bygg-energi-mikro", "tunnel-hauglia"]
|
||||
assert [{row.id for row in r.result.coverage} for r in result.runs] == [
|
||||
{"hypothesis-1", "own-proposal"},
|
||||
{"hypothesis-2", "own-proposal"},
|
||||
]
|
||||
328
tests/test_navigation_visibility_loadbearing.py
Normal file
328
tests/test_navigation_visibility_loadbearing.py
Normal file
|
|
@ -0,0 +1,328 @@
|
|||
"""A knowledge base that was only HALF read SAYS so — a structured trace, and one line when it fired.
|
||||
|
||||
``okf._walk`` tolerates a cross-link it cannot follow, exactly as OKF SPEC §4 requires: it skips and
|
||||
never raises. **That tolerance is correct and is not what this file changes.** What it changes is
|
||||
that the skip left NO TRACE. ``navigate_bundle`` returned only the files it FOUND, so a base whose
|
||||
other half was never reached is indistinguishable from a base where those documents were never
|
||||
written — and ``--live-dry-run`` exited 0 in silence over both.
|
||||
|
||||
Measured (session 51, 2026-08-21, against ``b639722``): ``okf.py:182`` was
|
||||
``continue # broken link: tolerated, never raised (OKF §4)``, ``Bundle`` carried ``dir`` and
|
||||
``files`` alone, and ``grep -rn "Bundle(" src tests`` returned exactly ONE construction site
|
||||
(``okf.py:203``, inside ``navigate_bundle``) — so the trace has a single producer and cannot be
|
||||
forged by a second constructor.
|
||||
|
||||
Same defect class and same SHAPE as order ``20260821T092039Z`` (visible un-anchoring, delivered in
|
||||
``156312c``): a tolerance that is right, plus an absence that is not.
|
||||
|
||||
Two teeth:
|
||||
|
||||
1. ``okf.SkippedLink`` + ``Bundle.skipped`` — a STRUCTURED trace, never a string, because "which
|
||||
document is missing" and "why" are two different operative questions (kø-(y)). ``_walk`` has TWO
|
||||
distinct skip reasons and they mean different things: ``outside-bundle`` (the target resolves
|
||||
outside the bundle root — often a deliberate link to a neighbouring base) and ``missing`` (it
|
||||
resolves INSIDE and no readable file is there — almost always a typo in the link). The third
|
||||
branch, ``canonical in seen``, is DE-DUPLICATION: correct behaviour, also what terminates cycles,
|
||||
and never a skip — arm (d) exists to keep it out of the trace.
|
||||
2. ``run.skipped_links_notice`` — ONE renderer, taking the already-resolved value, returning ``None``
|
||||
when nothing was skipped (omission, never an empty row — ``mandate.announce``'s rule, reused by
|
||||
``cost_baseline_notice``).
|
||||
|
||||
**The default DIFFERS from the previous order's, and that difference is the insight.**
|
||||
``ProvenanceStamp.cost_baseline_anchored`` is REQUIRED with no default because both defaults lie:
|
||||
``True`` claims an anchoring that may never have happened, ``False`` under-claims a real one. Here
|
||||
the honest reading is the opposite: an EMPTY tuple is a positive statement — "every cross-link was
|
||||
followed" — in the same class as ``ProvenanceStamp.external_calls`` ("nothing outside this process
|
||||
was contacted"). A caller that constructs a ``Bundle`` without a trace is not withholding a fact; it
|
||||
is stating one. So ``skipped`` defaults to ``()``, and the road path (which navigates nothing) is
|
||||
honestly empty rather than dishonestly required to invent a value.
|
||||
|
||||
Arms:
|
||||
(a) a MISSING target yields exactly one entry, carrying its own reason;
|
||||
(b) an ESCAPING target yields exactly one entry, carrying the OTHER reason — asserted on the
|
||||
structured ``reason`` field, never on shared prose (the 08-09 class), and the two reasons are
|
||||
asserted to DIFFER so a single collapsed reason cannot pass both;
|
||||
(c) an intact base yields an EMPTY tuple and NO line (the control — without it (a) passes on a
|
||||
constant), with the navigation proved to have happened first;
|
||||
(d) the dedup branch (a repeated link, and a cycle) yields NO entry at all;
|
||||
(e) both CLI surfaces carry it — ``--live-dry-run`` and the full run — plus the typed carriers.
|
||||
|
||||
The commons-owned ``nav-golden-escape`` fasit is a free independent witness that the SEMANTICS did
|
||||
not move: every link but one escapes there, and it must still render byte-identically.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from conftest import SyntheticUsageChatClient
|
||||
|
||||
from portfolio_optimiser import okf, run
|
||||
from portfolio_optimiser.okf import SkippedLink
|
||||
from portfolio_optimiser.run import DryRunReport, RunResult, run_project, skipped_links_notice
|
||||
|
||||
_DATA = Path(__file__).resolve().parents[1] / "src" / "portfolio_optimiser" / "data" / "bundles"
|
||||
INTACT_BUNDLE = _DATA / "bygg-energi-mikro-a"
|
||||
|
||||
_VERDICT_INPUT = {"decision": "approved", "rationale": "expert reviewed (sim)"}
|
||||
|
||||
#: A reply the pre-amendment fixture's un-anchored gate accepts far enough to produce an outcome.
|
||||
_REPLY = json.dumps(
|
||||
{
|
||||
"measure": "LED-retrofit",
|
||||
"affected_items": [{"code": "ENERGI-TOTAL-EL", "quantity": 180000, "unit_cost": 1.0}],
|
||||
"claimed_saving_nok": 30000,
|
||||
}
|
||||
)
|
||||
|
||||
#: The phrase the notice carries and an intact run cannot: an intact run prints NO line at all.
|
||||
_SENTINEL = "NOT followed"
|
||||
|
||||
#: The link an arm appends to a COPY of the fixture index — a name nothing in the bundle provides.
|
||||
_DANGLING = "fantes-aldri.md"
|
||||
|
||||
|
||||
def _factory(reply: str = _REPLY):
|
||||
def factory(role: str):
|
||||
return SyntheticUsageChatClient(default_reply=reply)
|
||||
|
||||
return factory
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _isolate_model_env(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Hermetic env: the operator's Foundry overrides must not reach the CLI arms."""
|
||||
monkeypatch.delenv("PORTFOLIO_MODEL_MAP", raising=False)
|
||||
monkeypatch.delenv("PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT", raising=False)
|
||||
|
||||
|
||||
def _write(path: Path, body: str) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(body, encoding="utf-8")
|
||||
|
||||
|
||||
def _bundle_copy(tmp_path: Path, *, dangling: bool) -> Path:
|
||||
"""A throwaway copy of the repo-local fixture — the shipped bundle is never touched. With
|
||||
``dangling`` the root index gains ONE link to a file that does not exist."""
|
||||
dest = tmp_path / "bundle"
|
||||
shutil.copytree(INTACT_BUNDLE, dest)
|
||||
if dangling:
|
||||
index = dest / "index.md"
|
||||
index.write_text(
|
||||
index.read_text(encoding="utf-8") + f"\n- [Borte]({_DANGLING})\n", encoding="utf-8"
|
||||
)
|
||||
return dest
|
||||
|
||||
|
||||
# --- Arm (a): a MISSING target is recorded, with its own reason -----------------------------------
|
||||
|
||||
|
||||
def test_missing_target_is_recorded_as_one_entry(tmp_path) -> None:
|
||||
"""RED: the base still navigates and still raises nothing (§4 tolerance UNTOUCHED) — but the
|
||||
link it could not follow is now on the bundle, naming the file it was written in, the link text
|
||||
VERBATIM, and why. Detach the recording and the walk goes silent again."""
|
||||
_write(tmp_path / "index.md", "---\ntype: index\n---\n\n- [Borte](fantes-aldri.md)\n")
|
||||
bundle = okf.navigate_bundle(str(tmp_path))
|
||||
assert [f.name for f in bundle.files] == ["index.md"] # tolerated, not raised
|
||||
assert bundle.skipped == (
|
||||
SkippedLink(from_file="index.md", target="fantes-aldri.md", reason="missing"),
|
||||
)
|
||||
|
||||
|
||||
# --- Arm (b): an ESCAPING target is recorded, with the OTHER reason -------------------------------
|
||||
|
||||
|
||||
def test_escaping_target_is_recorded_with_a_different_reason(tmp_path) -> None:
|
||||
"""The two skip branches mean different things and must not be collapsed: this target EXISTS,
|
||||
one level up, and is refused because it is outside the bundle root — not because it is absent.
|
||||
Asserted on the structured ``reason``, never on prose the two branches could share."""
|
||||
_write(tmp_path / "outside.md", "---\ntype: reference\n---\n\nSHOULD NOT BE READ\n")
|
||||
root = tmp_path / "bundle"
|
||||
_write(root / "index.md", "---\ntype: index\n---\n\n- [Ute](../outside.md)\n")
|
||||
bundle = okf.navigate_bundle(str(root))
|
||||
assert [f.name for f in bundle.files] == ["index.md"]
|
||||
assert bundle.skipped == (
|
||||
SkippedLink(from_file="index.md", target="../outside.md", reason="outside-bundle"),
|
||||
)
|
||||
|
||||
|
||||
def test_the_two_reasons_are_distinct_values(tmp_path) -> None:
|
||||
"""A single collapsed reason would pass BOTH arms above if they were read in isolation. This
|
||||
pins the discrimination itself: same shape of bundle, two skips, two different values."""
|
||||
_write(tmp_path / "outside.md", "---\ntype: reference\n---\n\nx\n")
|
||||
root = tmp_path / "bundle"
|
||||
_write(
|
||||
root / "index.md",
|
||||
"---\ntype: index\n---\n\n- [Borte](fantes-aldri.md)\n- [Ute](../outside.md)\n",
|
||||
)
|
||||
reasons = [s.reason for s in okf.navigate_bundle(str(root)).skipped]
|
||||
assert len(reasons) == 2
|
||||
assert reasons[0] != reasons[1]
|
||||
|
||||
|
||||
# --- Arm (c): the control — an intact base records nothing and prints nothing ---------------------
|
||||
|
||||
|
||||
def test_intact_bundle_records_nothing(tmp_path) -> None:
|
||||
"""Causality control. Without it arm (a) would pass on an implementation that records a constant
|
||||
entry for every base. The navigation is proved to have HAPPENED first (two files reached), so
|
||||
the empty trace is a measured absence rather than a base that was never walked."""
|
||||
_write(tmp_path / "index.md", "---\ntype: index\n---\n\n- [A](a.md)\n")
|
||||
_write(tmp_path / "a.md", "---\ntype: project\n---\n\nA body\n")
|
||||
bundle = okf.navigate_bundle(str(tmp_path))
|
||||
assert [f.name for f in bundle.files] == ["index.md", "a.md"] # the walk really ran
|
||||
assert bundle.skipped == ()
|
||||
assert skipped_links_notice(bundle.skipped) is None
|
||||
|
||||
|
||||
# --- Arm (d): de-duplication is NOT a skip --------------------------------------------------------
|
||||
|
||||
|
||||
def test_dedup_and_cycles_produce_no_entry(tmp_path) -> None:
|
||||
"""``canonical in seen`` is correct behaviour, not a failure: it is what makes a repeated link
|
||||
one entry and what terminates a cycle. An implementation that recorded every ``continue`` would
|
||||
report a healthy base as half-unread. Both forms are exercised: ``a.md`` is linked twice from
|
||||
the index (once as ``./a.md``, deduped on the RESOLVED path) and links back to the index."""
|
||||
_write(tmp_path / "index.md", "---\ntype: index\n---\n\n- [A](a.md)\n- [A again](./a.md)\n")
|
||||
_write(tmp_path / "a.md", "---\ntype: project\n---\n\nA body\n\n- [Back](index.md)\n")
|
||||
bundle = okf.navigate_bundle(str(tmp_path))
|
||||
assert [f.name for f in bundle.files] == ["index.md", "a.md"] # deduped + cycle terminated
|
||||
assert bundle.skipped == ()
|
||||
|
||||
|
||||
# --- Arm (e): the renderer and both CLI surfaces --------------------------------------------------
|
||||
|
||||
|
||||
def test_notice_is_rendered_only_when_something_was_skipped() -> None:
|
||||
"""One renderer, two branches sharing NO wording: a non-empty trace returns a line carrying the
|
||||
sentinel AND the operative facts; an empty trace returns ``None`` (omitted, never an empty row).
|
||||
The reason token printed is the STRUCTURED value itself, so there is no second display
|
||||
vocabulary free to drift from the field (kø-(p))."""
|
||||
rendered = skipped_links_notice(
|
||||
(SkippedLink(from_file="index.md", target="fantes-aldri.md", reason="missing"),)
|
||||
)
|
||||
assert rendered is not None
|
||||
assert _SENTINEL in rendered
|
||||
assert "index.md" in rendered
|
||||
assert "fantes-aldri.md" in rendered
|
||||
assert "missing" in rendered
|
||||
assert skipped_links_notice(()) is None
|
||||
|
||||
|
||||
async def test_dry_run_report_carries_the_trace(tmp_path, fresh_store) -> None:
|
||||
"""The dry-run type is the carrier for the surface the order measured: a run that stops before
|
||||
the first model call already knows what it could not read."""
|
||||
assert "skipped_links" in DryRunReport.__dataclass_fields__
|
||||
bundle = _bundle_copy(tmp_path, dangling=True)
|
||||
report = await run_project(
|
||||
"BYGG-ENERGI-MIKRO-A",
|
||||
"local",
|
||||
docs_dir=str(bundle),
|
||||
bundle_dir=str(bundle),
|
||||
verdict_input=_VERDICT_INPUT,
|
||||
client_factory=_factory(),
|
||||
store=fresh_store,
|
||||
live_dry_run=True,
|
||||
)
|
||||
assert isinstance(report, DryRunReport)
|
||||
assert [s.target for s in report.skipped_links] == [_DANGLING]
|
||||
|
||||
|
||||
async def test_run_result_carries_the_trace(tmp_path, fresh_store) -> None:
|
||||
"""The full run too: navigation happens ONCE per run, before any proposal exists, so the trace
|
||||
is a RUN-level fact carried on ``RunResult`` — not on the per-proposal ``ProvenanceStamp``,
|
||||
which describes the gate that judged one candidate."""
|
||||
assert "skipped_links" in RunResult.__dataclass_fields__
|
||||
bundle = _bundle_copy(tmp_path, dangling=True)
|
||||
result = await run_project(
|
||||
"BYGG-ENERGI-MIKRO-A",
|
||||
"local",
|
||||
docs_dir=str(bundle),
|
||||
bundle_dir=str(bundle),
|
||||
verdict_input=_VERDICT_INPUT,
|
||||
client_factory=_factory(),
|
||||
store=fresh_store,
|
||||
)
|
||||
assert [s.target for s in result.skipped_links] == [_DANGLING]
|
||||
|
||||
|
||||
async def test_road_path_has_an_empty_trace(docs_dir, fresh_store) -> None:
|
||||
"""The road path navigates no bundle, so "nothing was skipped" is literally true there — which
|
||||
is exactly why the empty tuple is an honest DEFAULT rather than a withheld fact."""
|
||||
result = await run_project(
|
||||
"FV42-GSV-E1",
|
||||
"local",
|
||||
docs_dir=docs_dir,
|
||||
verdict_input=_VERDICT_INPUT,
|
||||
client_factory=_factory(
|
||||
json.dumps(
|
||||
{
|
||||
"measure": "Reduce scope",
|
||||
"affected_items": [{"code": "05.2", "quantity": 4300.0, "unit_cost": 215.0}],
|
||||
"claimed_saving_nok": 200000.0,
|
||||
}
|
||||
)
|
||||
),
|
||||
store=fresh_store,
|
||||
)
|
||||
assert result.skipped_links == ()
|
||||
|
||||
|
||||
def _dry_run_argv(bundle: Path) -> list[str]:
|
||||
return [
|
||||
"BYGG-ENERGI-MIKRO-A",
|
||||
"--docs-dir",
|
||||
str(bundle),
|
||||
"--bundle-dir",
|
||||
str(bundle),
|
||||
"--live-dry-run",
|
||||
]
|
||||
|
||||
|
||||
def test_cli_dry_run_announces_the_skipped_link(tmp_path, capsys) -> None:
|
||||
"""RED (the measured defect, verbatim): a dry run over a base with an unfollowable cross-link
|
||||
exited 0 with nothing said. It now names the document it never reached."""
|
||||
rc = run.main(_dry_run_argv(_bundle_copy(tmp_path, dangling=True)))
|
||||
assert rc == 0
|
||||
out = capsys.readouterr().out
|
||||
assert _SENTINEL in out
|
||||
assert _DANGLING in out
|
||||
|
||||
|
||||
def test_cli_dry_run_says_nothing_when_every_link_was_followed(tmp_path, capsys) -> None:
|
||||
"""Control: the same base with its links intact prints NO navigation line at all. A line for
|
||||
something the run does not have is omitted, never rendered blank."""
|
||||
rc = run.main(_dry_run_argv(_bundle_copy(tmp_path, dangling=False)))
|
||||
assert rc == 0
|
||||
out = capsys.readouterr().out
|
||||
assert _SENTINEL not in out
|
||||
assert "Knowledge base:" not in out
|
||||
|
||||
|
||||
def test_cli_full_run_announces_the_skipped_link(tmp_path, capsys) -> None:
|
||||
"""The full-run surface too, through the offline scripted door — so the notice is a property of
|
||||
a RUN, not of the dry-run branch alone. A run that PRODUCED a proposal from a half-read base is
|
||||
the case where the silence cost the most."""
|
||||
bundle = _bundle_copy(tmp_path, dangling=True)
|
||||
replies = tmp_path / "replies.json"
|
||||
replies.write_text(
|
||||
json.dumps({"proposer": _REPLY, "checker": "Holder. VERDICT: APPROVE"}), encoding="utf-8"
|
||||
)
|
||||
rc = run.main(
|
||||
[
|
||||
"BYGG-ENERGI-MIKRO-A",
|
||||
"--docs-dir",
|
||||
str(bundle),
|
||||
"--bundle-dir",
|
||||
str(bundle),
|
||||
"--scripted-replies",
|
||||
str(replies),
|
||||
]
|
||||
)
|
||||
assert rc == 0
|
||||
out = capsys.readouterr().out
|
||||
assert _SENTINEL in out
|
||||
assert _DANGLING in out
|
||||
|
|
@ -62,6 +62,9 @@ _PROVENANCE = ProvenanceStamp(
|
|||
role="proposer",
|
||||
validator_decision="validated",
|
||||
token_usage=8,
|
||||
# These fixtures stand in for an ordinary complete run; the road path is anchored by
|
||||
# construction, so ``True`` is the honest value here. The un-anchored case has its own file.
|
||||
cost_baseline_anchored=True,
|
||||
)
|
||||
_VALIDATED = ValidatedProposal(
|
||||
proposal=_PROPOSAL, p10=100.0, p50=150.0, p90=200.0, nominal_feasible=180.0
|
||||
|
|
|
|||
284
tests/test_parse_failure_capture_loadbearing.py
Normal file
284
tests/test_parse_failure_capture_loadbearing.py
Normal file
|
|
@ -0,0 +1,284 @@
|
|||
"""Load-bearing: the model's RAW reply must survive a parse failure (Fase 1b, funn 1).
|
||||
|
||||
The gap, measured on the project's first live run (``docs/2026-08-14-fase1b-forste-levende-kjoring.md``):
|
||||
``generate._fetch_parsed`` caught ``(ValidationError, ValueError, TypeError)`` and ``continue``-d.
|
||||
The text the model actually produced — the thing that was PAID for and the only evidence of WHY it
|
||||
did not parse — was dropped on the floor. The live run burned all twelve rounds on parse failures
|
||||
and died with ``BudgetExceeded``, and no artefact anywhere held a single character of what the model
|
||||
said. Every further paid run would have been a guess.
|
||||
|
||||
**WHERE the text surfaces is decided by a measurement, not by symmetry with Step 5.**
|
||||
``BudgetExceeded`` is raised by ``meter.tick_round()`` INSIDE ``_fetch_parsed``, so it propagates out
|
||||
of ``generate_via_llm`` before any value is returned, and (without a mandate) nothing catches it —
|
||||
``run.py``'s only ``except BudgetExceeded`` is the mandate arm. So:
|
||||
|
||||
* a typed RETURN VALUE (``GenerationResult.parse_failures``, mirroring Step 5's ``refinements``) is
|
||||
blind to exactly the one path the capture exists for: the returning path is the path where the run
|
||||
already succeeded in parsing something;
|
||||
* an outbox artefact written AFTER the run (``write_outbox``, run.py step 7) is blind for the same
|
||||
reason — ``run_project`` never reaches it.
|
||||
|
||||
The seam is therefore a CALLER-OWNED SINK plus an artefact written in a ``finally``:
|
||||
``run_project`` owns the ``list[ParseFailure]``, hands it to ``generate_via_llm`` beside ``meter``,
|
||||
and writes ``{run_id}-parse-failures.json`` however the evaluation ended. The sink mirrors ``meter``
|
||||
exactly — a caller-owned accumulator the loop mutates, whose contents the caller still holds after an
|
||||
exception. Step 5's "a returned value cannot be silently lost by a caller that forgets to pass a
|
||||
collector" governs a value that REACHES the caller; here it does not, and copying that rule blindly
|
||||
would have rebuilt the defect one layer up.
|
||||
|
||||
Five tests, load-bearing as a set. Each detach point is RED on its own:
|
||||
|
||||
* T1 — the budget-exhausted run (the live 1b shape) still writes the artefact
|
||||
(RED when the capture is detached, when the write is not in a ``finally``, or when run.py does not
|
||||
hand the sink over);
|
||||
* T2 — the sink receives the reply VERBATIM at the ``generate_via_llm`` level
|
||||
(RED when the capture is detached OR when the text is truncated/summarised on the way in);
|
||||
* T3 — the CONTROL: a run whose replies all parse writes NO artefact
|
||||
(RED on a "write it always" implementation, which would make the artefact's presence meaningless);
|
||||
* T4 — a run that COMPLETES writes the artefact too
|
||||
(RED on an implementation that only writes from the exception arm — the mirror of T1);
|
||||
* T5 — the marker gate: the sentinel is ABSENT from the bundle, so a positive assert on it cannot be
|
||||
satisfied by bundle context leaking into a prompt (repo rule for bundle-driven tests).
|
||||
|
||||
They drive the CANONICAL ``ScriptedChatClient`` through its ``reply_selector`` seam (S2.5
|
||||
consolidation) and key on the generation prompt's own instruction line, so the debate's replies —
|
||||
which are never parsed — cannot be mistaken for a capture.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from portfolio_optimiser.budget import Budget, BudgetExceeded, TokenMeter
|
||||
from portfolio_optimiser.generate import ParseFailure, generate_via_llm
|
||||
from portfolio_optimiser.reference_domain import load_reference_projects
|
||||
from portfolio_optimiser.run import RunResult, run_project
|
||||
from portfolio_optimiser.simulation import ScriptedChatClient
|
||||
from portfolio_optimiser.verdicts import VerdictStore
|
||||
|
||||
BUNDLE_DIR = Path(__file__).resolve().parents[1] / "shared" / "examples" / "bygg-energi-mikro"
|
||||
_VERDICT_INPUT = {"decision": "approved", "rationale": "expert reviewed (sim)"}
|
||||
_PROJECT_ID = "BYGG-KONTOR-NORD"
|
||||
|
||||
#: The instruction line ``generate._build_messages`` puts in EVERY generation prompt and nowhere
|
||||
#: else — the one identifier that separates a generation call from a debate turn.
|
||||
_GENERATION_MARK = "Respond with ONLY a JSON object"
|
||||
|
||||
#: A reply shaped like what a chatty model actually returns: prose, a fenced block, newlines and
|
||||
#: non-ASCII — NOT JSON. The hex sentinel is what the positive asserts key on; it is gated absent
|
||||
#: from the bundle by T5, so bundle context cannot satisfy them. Deliberately >200 characters, so a
|
||||
#: truncating capture (``text[:80]``) is measurable rather than merely suspected.
|
||||
_MALFORMED = (
|
||||
"Certainly! Here is my analysis of the project.\n\n"
|
||||
"PARSE-FAIL-SENTINEL-7f3a9c: the måling below is prose, not the requested object.\n\n"
|
||||
"```\n"
|
||||
"measure: Behovsstyrt belysning\n"
|
||||
"claimed_saving_nok: about 30 000 NOK (approx.)\n"
|
||||
"```\n\n"
|
||||
"Let me know if you would like me to format this as JSON instead!"
|
||||
)
|
||||
_SENTINEL = "PARSE-FAIL-SENTINEL-7f3a9c"
|
||||
|
||||
#: BYGG-KONTOR-NORD: affected total 300000 x 1.0 -> degenerate Monte Carlo P90 = 90000, so a claim
|
||||
#: of 30000 validates (same arithmetic as test_a5_per_approach_artifacts_loadbearing).
|
||||
_VALID_REPLY = (
|
||||
'{"measure":"Behovsstyrt belysning i fellesarealer","affected_items":'
|
||||
'[{"code":"ENERGI-TOTAL-EL","quantity":300000,"unit_cost":1.0}],'
|
||||
'"claimed_saving_nok":30000}'
|
||||
)
|
||||
|
||||
|
||||
def _factory(
|
||||
select: Callable[[str, str], str],
|
||||
) -> Callable[[str], ScriptedChatClient]:
|
||||
def factory(role: str) -> ScriptedChatClient:
|
||||
return ScriptedChatClient(role=role, reply_selector=select, default_reply="ok")
|
||||
|
||||
return factory
|
||||
|
||||
|
||||
def _always_malformed(blob: str, _role: str) -> str:
|
||||
return _MALFORMED if _GENERATION_MARK in blob else "ok"
|
||||
|
||||
|
||||
def _malformed_then_valid(failures: int) -> Callable[[str, str], str]:
|
||||
"""Fail to parse ``failures`` times, then answer with a proposal that validates. The counter is
|
||||
keyed on the GENERATION prompt only, so debate turns do not consume it."""
|
||||
seen = {"n": 0}
|
||||
|
||||
def _select(blob: str, _role: str) -> str:
|
||||
if _GENERATION_MARK not in blob:
|
||||
return "ok"
|
||||
seen["n"] += 1
|
||||
return _MALFORMED if seen["n"] <= failures else _VALID_REPLY
|
||||
|
||||
return _select
|
||||
|
||||
|
||||
async def _run(
|
||||
select: Callable[[str, str], str],
|
||||
outbox_dir: Path,
|
||||
*,
|
||||
run_id: str,
|
||||
max_rounds: int,
|
||||
) -> RunResult:
|
||||
result = await run_project(
|
||||
_PROJECT_ID,
|
||||
"local",
|
||||
docs_dir=str(BUNDLE_DIR),
|
||||
bundle_dir=str(BUNDLE_DIR),
|
||||
verdict_input=_VERDICT_INPUT,
|
||||
store=VerdictStore(verdicts=[]),
|
||||
client_factory=_factory(select),
|
||||
outbox_dir=str(outbox_dir),
|
||||
run_id=run_id,
|
||||
# The ROUND cap is the bound under test; tokens are left effectively unbounded so a failure
|
||||
# here can only be the round ledger (the live 1b shape: "rounds limit=12 observed=13").
|
||||
meter=TokenMeter(Budget(max_tokens=10**9, max_rounds=max_rounds)),
|
||||
)
|
||||
assert isinstance(result, RunResult)
|
||||
return result
|
||||
|
||||
|
||||
def _artefact(outbox_dir: Path, run_id: str) -> Path:
|
||||
return outbox_dir / f"{run_id}-parse-failures.json"
|
||||
|
||||
|
||||
def _failures(outbox_dir: Path, run_id: str) -> list[dict[str, str]]:
|
||||
payload = json.loads(_artefact(outbox_dir, run_id).read_text(encoding="utf-8"))
|
||||
assert payload["run_id"] == run_id
|
||||
failures = payload["parse_failures"]
|
||||
assert isinstance(failures, list)
|
||||
return failures
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------------------------
|
||||
# T1 — the live 1b shape: the run dies inside the generation loop, and the text survives anyway.
|
||||
# --------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_budget_exhausted_run_still_writes_the_raw_text(tmp_path: Path) -> None:
|
||||
"""The measured 1b failure, reproduced offline: every reply is unparseable, the round ledger is
|
||||
exhausted inside ``_fetch_parsed``, and ``BudgetExceeded`` leaves ``run_project`` as an
|
||||
exception. The artefact must exist ANYWAY — this is the path a return value cannot reach.
|
||||
|
||||
RED when the capture is detached, when the write is moved out of the ``finally``, or when
|
||||
``run.py`` stops handing the sink to ``generate_via_llm``."""
|
||||
outbox_dir = tmp_path / "outbox"
|
||||
run_id = "run-parse-exhausted"
|
||||
|
||||
with pytest.raises(BudgetExceeded) as excinfo:
|
||||
await _run(_always_malformed, outbox_dir, run_id=run_id, max_rounds=3)
|
||||
# Prove the event this test claims actually happened, and that it is the ROUND ledger — an
|
||||
# assert on the artefact alone could not tell an exhausted run from a completed one.
|
||||
assert excinfo.value.kind == "rounds"
|
||||
|
||||
failures = _failures(outbox_dir, run_id)
|
||||
# max_rounds=3 -> ticks 1..3 each fetch a reply and fail to parse; tick 4 raises. So exactly
|
||||
# three replies were paid for, and exactly three must be recoverable.
|
||||
assert len(failures) == 3
|
||||
assert all(_SENTINEL in f["text"] for f in failures)
|
||||
assert all(f["error"] for f in failures), (
|
||||
"the parse error itself must be recorded, not just why"
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------------------------
|
||||
# T2 — the capture itself, at the generate level: VERBATIM, not summarised.
|
||||
# --------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_the_sink_receives_the_reply_verbatim(tmp_path: Path) -> None:
|
||||
"""``generate_via_llm`` appends to the caller's sink at the moment the parse fails, and the text
|
||||
is the model's reply BYTE-FOR-BYTE. A capture that truncates, strips or summarises would leave
|
||||
the operator reading a paraphrase of the evidence.
|
||||
|
||||
RED when the capture is detached, and RED when the text is shortened on the way in."""
|
||||
# A reference project rather than a hand-built one: this test never reaches the validator (no
|
||||
# reply ever parses), so the project only has to be a real one the prompt can name.
|
||||
project = load_reference_projects()[0]
|
||||
client = ScriptedChatClient(reply_selector=_always_malformed, role="proposer")
|
||||
sink: list[ParseFailure] = []
|
||||
|
||||
with pytest.raises(BudgetExceeded):
|
||||
await generate_via_llm(
|
||||
client,
|
||||
project,
|
||||
"",
|
||||
TokenMeter(Budget(max_tokens=10**9, max_rounds=2)),
|
||||
max_attempts=3,
|
||||
parse_failures=sink,
|
||||
)
|
||||
|
||||
assert len(sink) == 2
|
||||
assert [f.text for f in sink] == [_MALFORMED, _MALFORMED], "the reply must arrive verbatim"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------------------------
|
||||
# T3 — the CONTROL: no parse failure, no artefact.
|
||||
# --------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_a_run_whose_replies_parse_writes_no_artefact(tmp_path: Path) -> None:
|
||||
"""A clean run must leave the outbox exactly as it was before this seam existed. Without this
|
||||
control an implementation that writes the file unconditionally passes T1 and T4, and the
|
||||
artefact's PRESENCE would then say nothing about whether anything failed to parse.
|
||||
|
||||
RED when the ``if parse_failures`` guard is dropped."""
|
||||
outbox_dir = tmp_path / "outbox"
|
||||
run_id = "run-parse-clean"
|
||||
|
||||
result = await _run(_malformed_then_valid(0), outbox_dir, run_id=run_id, max_rounds=8)
|
||||
assert result.outcome is not None
|
||||
|
||||
assert not _artefact(outbox_dir, run_id).exists()
|
||||
# The pre-existing artefacts are untouched — the addition is inert on the clean path.
|
||||
written = sorted(p.name for p in outbox_dir.iterdir())
|
||||
assert written == [
|
||||
f"{run_id}-outcome.json",
|
||||
f"{run_id}-proposal.json",
|
||||
f"{run_id}-runconfig.json",
|
||||
]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------------------------
|
||||
# T4 — the mirror of T1: a run that COMPLETES writes the artefact too.
|
||||
# --------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_completed_run_writes_the_artefact_too(tmp_path: Path) -> None:
|
||||
"""One unparseable reply, then a proposal that validates: the run completes normally and the
|
||||
discarded first reply is still recoverable. Without this, an implementation that writes only
|
||||
from the exception arm would pass T1 and silently lose every parse failure on runs that
|
||||
eventually succeeded — the common case once prompting improves.
|
||||
|
||||
RED when the write happens only on the exception path."""
|
||||
outbox_dir = tmp_path / "outbox"
|
||||
run_id = "run-parse-recovered"
|
||||
|
||||
result = await _run(_malformed_then_valid(1), outbox_dir, run_id=run_id, max_rounds=8)
|
||||
assert result.outcome is not None
|
||||
|
||||
failures = _failures(outbox_dir, run_id)
|
||||
assert len(failures) == 1
|
||||
assert _SENTINEL in failures[0]["text"]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------------------------
|
||||
# T5 — the marker gate.
|
||||
# --------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_the_sentinel_is_absent_from_the_bundle() -> None:
|
||||
"""Repo rule for bundle-driven tests: a marker asserted as PRESENT must be absent from the
|
||||
bundle, or bundle context reaching the prompt could satisfy the assert on its own."""
|
||||
hits = [
|
||||
path.name
|
||||
for path in BUNDLE_DIR.rglob("*")
|
||||
if path.is_file() and _SENTINEL in path.read_text(encoding="utf-8", errors="ignore")
|
||||
]
|
||||
assert hits == []
|
||||
405
tests/test_plan_review_cli_door_loadbearing.py
Normal file
405
tests/test_plan_review_cli_door_loadbearing.py
Normal file
|
|
@ -0,0 +1,405 @@
|
|||
"""F4 (docs/2026-08-25-fable-misjonsreview.md) — "be om svar, bruke svarene" must be reachable
|
||||
from an OPERATOR surface, not only from the library API.
|
||||
|
||||
Before this, ``explore(..., plan_reviewer=...)`` (``explore.py:849``) was the sole door onto the
|
||||
synchronous plan review, and BOTH operator surfaces refused ``enable_plan_review`` outright
|
||||
(``run.py:1975-1988``, ``hosting.py:167-172``) — measured against the source, not taken from the
|
||||
review's prose. Målbilde's "still spørsmål, be om svar, bruke svarene" was therefore unreachable
|
||||
by anyone who was not importing the package.
|
||||
|
||||
**The gate is the SECOND half of that phrase.** A door that prints the plan, reads a line and
|
||||
throws it away passes "the operator was asked" and fails the målbilde — this repo's vacuous-gate
|
||||
class, eight times over. Every test here is therefore built so an always-approve reviewer is RED:
|
||||
the discriminator is that a ``revise`` reaches the manager, the manager replans, and the operator
|
||||
is asked AGAIN about the NEW plan (the T15 shape from ``test_explore_loadbearing.py``, lifted to
|
||||
the CLI), with the feedback recorded VERBATIM.
|
||||
|
||||
The witness is ``{run_id}-exploration.json`` rather than scraped stdout: ``trace_payload``
|
||||
(``explore.py:285-293``) already carries the decisions, the feedback and the order, and it is
|
||||
written from a ``finally`` — so it survives the one run that most needs it, the one a cap or an
|
||||
unanswered review cut short.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
from portfolio_optimiser import explore as ex
|
||||
from portfolio_optimiser import hosting, run
|
||||
|
||||
_REPO = Path(__file__).resolve().parents[1]
|
||||
_BUNDLE_DIR = _REPO / "shared" / "examples" / "bygg-energi-mikro"
|
||||
_PID = "BYGG-KONTOR-NORD"
|
||||
_RUN_ID = "plan-review-door"
|
||||
|
||||
_PROPOSER_REPLY = json.dumps(
|
||||
{
|
||||
"measure": "LED-retrofit",
|
||||
"affected_items": [{"code": "ENERGI-TOTAL-EL", "quantity": 300000, "unit_cost": 1.0}],
|
||||
"claimed_saving_nok": 30000,
|
||||
}
|
||||
)
|
||||
_MANAGER_REPLY = json.dumps(
|
||||
{
|
||||
"is_request_satisfied": {"reason": "r", "answer": True},
|
||||
"is_in_loop": {"reason": "r", "answer": False},
|
||||
"is_progress_being_made": {"reason": "r", "answer": True},
|
||||
"next_speaker": {"reason": "r", "answer": "hypothesiser"},
|
||||
"instruction_or_question": {"reason": "r", "answer": "go"},
|
||||
}
|
||||
)
|
||||
_REPLIES = {
|
||||
"proposer": _PROPOSER_REPLY,
|
||||
"checker": "VERDICT: APPROVE",
|
||||
"manager": _MANAGER_REPLY,
|
||||
"navigator": "NAVIGATOR: read the index.",
|
||||
"hypothesiser": "HYPOTHESIS: " + json.dumps({"label": "Night setback", "rationale": "y"}),
|
||||
}
|
||||
|
||||
_FEEDBACK = "Also test night setback on the ventilation."
|
||||
|
||||
|
||||
def _config_file(tmp_path: Path, **overrides: Any) -> str:
|
||||
path = tmp_path / "exploration.json"
|
||||
path.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"max_rounds": 4,
|
||||
"max_tokens": 200_000,
|
||||
"max_stall_count": 2,
|
||||
"max_reset_count": 1,
|
||||
"max_plan_revisions": 2,
|
||||
"enable_plan_review": True,
|
||||
**overrides,
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
return str(path)
|
||||
|
||||
|
||||
def _replies_file(tmp_path: Path) -> str:
|
||||
path = tmp_path / "replies.json"
|
||||
path.write_text(json.dumps(_REPLIES), encoding="utf-8")
|
||||
return str(path)
|
||||
|
||||
|
||||
def _argv(tmp_path: Path, *, plan_review: bool = True, **config: Any) -> list[str]:
|
||||
argv = [
|
||||
_PID,
|
||||
"--docs-dir",
|
||||
str(_BUNDLE_DIR),
|
||||
"--bundle-dir",
|
||||
str(_BUNDLE_DIR),
|
||||
"--explore",
|
||||
"Find the cheapest saving.",
|
||||
"--explore-config",
|
||||
_config_file(tmp_path, **config),
|
||||
"--scripted-replies",
|
||||
_replies_file(tmp_path),
|
||||
"--outbox-dir",
|
||||
str(tmp_path / "outbox"),
|
||||
"--run-id",
|
||||
_RUN_ID,
|
||||
]
|
||||
if plan_review:
|
||||
argv.append("--plan-review")
|
||||
return argv
|
||||
|
||||
|
||||
def _artefact(tmp_path: Path) -> dict[str, Any]:
|
||||
path = tmp_path / "outbox" / f"{_RUN_ID}-exploration.json"
|
||||
assert path.exists(), "the exploration artefact must be written even when the run failed"
|
||||
return json.loads(path.read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
def _stdin(monkeypatch: pytest.MonkeyPatch, text: str) -> None:
|
||||
monkeypatch.setattr(sys, "stdin", io.StringIO(text))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# 1. THE GOAL — asked, answered, and the answer USED
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_an_operator_answer_typed_at_the_cli_reaches_the_manager_and_is_asked_again(
|
||||
tmp_path, capsys, monkeypatch
|
||||
) -> None:
|
||||
"""T1: the whole point. ``revise`` typed at the CLI reaches the manager, the manager replans,
|
||||
and the operator is asked to sign off on the NEW plan.
|
||||
|
||||
RED against an always-approve reviewer (the vacuous door): that yields ONE review and
|
||||
``["approve"]``, so both the count and the order fail. RED against a reviewer that reads the
|
||||
line and discards it: the feedback assertion fails and the manager is never asked twice.
|
||||
|
||||
Detach point: drop the ``plan_reviewer=`` wiring in ``run.py`` → the CLI refuses instead
|
||||
(there is no reviewer), so this never runs at all.
|
||||
"""
|
||||
_stdin(monkeypatch, f"revise {_FEEDBACK}\napprove\n")
|
||||
|
||||
rc = run.main(_argv(tmp_path))
|
||||
|
||||
err = capsys.readouterr().err
|
||||
assert "Traceback" not in err, err
|
||||
assert rc in (0, 1), f"expected a clean exit, got rc={rc} stderr={err!r}"
|
||||
|
||||
reviews = _artefact(tmp_path)["plan_reviews"]
|
||||
assert [r["decision"] for r in reviews] == ["revise", "approve"], (
|
||||
"a revision must produce a SECOND review, not resume silently — an always-approve door "
|
||||
"gives ['approve']"
|
||||
)
|
||||
assert reviews[0]["feedback"] == _FEEDBACK, (
|
||||
"what a human told the loop is worth nothing paraphrased"
|
||||
)
|
||||
assert reviews[1]["plan"] != "", "the second review must show the replanned plan"
|
||||
|
||||
|
||||
def test_the_operator_is_shown_the_plan_and_the_answer_vocabulary(
|
||||
tmp_path, capsys, monkeypatch
|
||||
) -> None:
|
||||
"""T2: a review nobody can read is a review nobody can answer. The prompt carries the plan,
|
||||
the progress and the two words that answer it.
|
||||
|
||||
Detach point: print only "plan review?" → RED. The control for T1: T1 proves the answer is
|
||||
used, this proves the question was askable.
|
||||
"""
|
||||
_stdin(monkeypatch, "approve\napprove\napprove\n")
|
||||
|
||||
run.main(_argv(tmp_path))
|
||||
|
||||
out = capsys.readouterr().out
|
||||
reviews = _artefact(tmp_path)["plan_reviews"]
|
||||
assert reviews, "the run must actually have reached a plan review"
|
||||
assert reviews[0]["plan"][:40] in out, (
|
||||
"the operator must be shown the plan they are signing off"
|
||||
)
|
||||
assert "approve" in out and "revise" in out, "the prompt must name the vocabulary it accepts"
|
||||
|
||||
|
||||
def test_an_unrecognised_answer_is_asked_again_never_taken_as_a_sign_off(
|
||||
tmp_path, capsys, monkeypatch
|
||||
) -> None:
|
||||
"""T3: fail-closed on the operator's own input. Anything outside the vocabulary is re-asked;
|
||||
it is never read as approval, and never as a revision either.
|
||||
|
||||
Detach point: treat any non-``revise`` line as approve → the first line ("yes please") would
|
||||
sign the plan off and the recorded decision would still be ``approve``, so the count of
|
||||
prompts is what discriminates: RED here, green there.
|
||||
"""
|
||||
_stdin(monkeypatch, f"yes please\n\nrevise {_FEEDBACK}\napprove\n")
|
||||
|
||||
run.main(_argv(tmp_path))
|
||||
|
||||
out = capsys.readouterr().out
|
||||
reviews = _artefact(tmp_path)["plan_reviews"]
|
||||
assert [r["decision"] for r in reviews] == ["revise", "approve"], (
|
||||
"the junk line and the blank line must both be re-asked, not consumed as decisions"
|
||||
)
|
||||
assert reviews[0]["feedback"] == _FEEDBACK
|
||||
assert out.count("PLAN REVIEW") == 2, (
|
||||
"two REVIEWS were answered; a third prompt would mean a junk line was consumed as one"
|
||||
)
|
||||
|
||||
|
||||
def test_end_of_input_never_becomes_an_approval_and_the_evidence_still_lands(
|
||||
tmp_path, monkeypatch
|
||||
) -> None:
|
||||
"""T4: the silence that must not be read as a yes.
|
||||
|
||||
A pipe that ends — or an operator who walks away — leaves the review unanswered. Reading that
|
||||
as approval would let an autonomous loop run on a plan no human signed, which is the exact
|
||||
thing the door exists to prevent, and it would do so invisibly. It raises instead.
|
||||
|
||||
The artefact is asserted TOO, and that is the load-bearing half: the write is in a ``finally``
|
||||
(``run.py``), so the run that failed still leaves a record of what the operator was asked and
|
||||
what they had answered so far. ``completed: false`` is what says the run never finished
|
||||
(``trace_payload``'s required field — an absent ``stop`` cannot say it).
|
||||
|
||||
Detach point: return ``PlanReviewDecision.approve()`` at EOF → no exception, ``completed``
|
||||
true, and the loop runs on an unsigned plan (RED on all three).
|
||||
"""
|
||||
_stdin(monkeypatch, "")
|
||||
|
||||
with pytest.raises(ex.PlanReviewInputError):
|
||||
run.main(_argv(tmp_path))
|
||||
|
||||
artefact = _artefact(tmp_path)
|
||||
assert artefact["completed"] is False
|
||||
assert artefact["plan_reviews"] == [], "nothing was decided, so nothing may be recorded"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# 2. The refusals — every combination that would silently drop the flag
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_plan_review_without_an_exploration_is_refused_by_name(tmp_path, capsys) -> None:
|
||||
"""T5: there is no plan to review without an exploration. Refused rather than loaded and
|
||||
dropped — the ``--explore-config`` precedent, verbatim.
|
||||
|
||||
Detach point: accept it silently → RED (rc 0, flag ignored).
|
||||
"""
|
||||
rc = run.main([_PID, "--docs-dir", str(_BUNDLE_DIR), "--plan-review", "--live-dry-run"])
|
||||
|
||||
assert rc == 1
|
||||
assert "--plan-review" in capsys.readouterr().err
|
||||
|
||||
|
||||
def test_plan_review_against_a_config_that_asks_for_no_review_is_refused(tmp_path, capsys) -> None:
|
||||
"""T6: a reviewer nobody will ever call. ``explore()`` refuses this too, but as an
|
||||
``ExplorationError`` — a ``RuntimeError``, outside ``main()``'s refusal tuple — so it would
|
||||
leave as a traceback instead of the rc-1 line. Hoisted here for that reason alone.
|
||||
|
||||
The assertion names wording UNIQUE to this branch: after F4 the CLI has two refusals
|
||||
containing ``enable_plan_review``, and asserting on the shared token is this repo's
|
||||
"assert never on wording two branches share" defect (measured in økt 57).
|
||||
|
||||
Detach point: leave it to ``explore()`` → RED (traceback, not rc 1).
|
||||
"""
|
||||
rc = run.main(_argv(tmp_path, enable_plan_review=False, max_plan_revisions=0))
|
||||
|
||||
assert rc == 1
|
||||
assert "no review is ever requested" in capsys.readouterr().err
|
||||
|
||||
|
||||
def test_a_review_with_no_reviewer_still_refuses_and_now_names_the_door(tmp_path, capsys) -> None:
|
||||
"""T7: the opposite half — the config asks for a review and no ``--plan-review`` was given.
|
||||
|
||||
The refusal SURVIVES F4 (a run must never stop at a review nobody can answer), but its wording
|
||||
was a claim the surface made about itself: "the synchronous door is the library API" stopped
|
||||
being true the moment this CLI grew one. It now names the flag.
|
||||
|
||||
Detach point: leave the old wording → RED. Same class as the Fase 3 credential claim.
|
||||
"""
|
||||
rc = run.main(_argv(tmp_path, plan_review=False))
|
||||
|
||||
assert rc == 1
|
||||
err = capsys.readouterr().err
|
||||
assert "no reviewer was offered" in err
|
||||
assert "--plan-review" in err, "the refusal must name the door that answers it"
|
||||
|
||||
|
||||
def test_plan_review_belongs_to_single_project_mode(tmp_path, capsys) -> None:
|
||||
"""T8: the documented partition. ``--explore`` is single-project-only and ``--plan-review``
|
||||
answers its review, so a ``--portfolio --plan-review`` argv has to hear which flag is wrong.
|
||||
|
||||
The assertion names ``--portfolio`` rather than ``--plan-review``: the refusal below it
|
||||
(``--plan-review requires --explore``) names ``--plan-review`` too, and asserting on the shared
|
||||
token would pass against no partition entry at all — the økt-57 mutation, verbatim.
|
||||
|
||||
Detach point: leave it out of ``single_only`` → RED (the message names --explore only, or the
|
||||
run falls through to the requires-refusal).
|
||||
"""
|
||||
rc = run.main(["--portfolio", "--plan-review"])
|
||||
|
||||
assert rc == 1
|
||||
assert "--portfolio" in capsys.readouterr().err
|
||||
|
||||
|
||||
def test_plan_review_is_refused_in_report_mode(tmp_path, capsys) -> None:
|
||||
"""T9: ``--report`` returns BEFORE every exploration refusal, so a flag missing from
|
||||
``report_forbidden`` is silently dropped rather than refused — the reason that list enumerates
|
||||
every distinguishable flag in the first place.
|
||||
|
||||
Detach point: leave it out of ``report_forbidden`` → rc 0 and a printed report, the flag gone
|
||||
without a word (RED).
|
||||
"""
|
||||
ledger = tmp_path / "ledger.json"
|
||||
ledger.write_text(json.dumps({"entries": []}), encoding="utf-8")
|
||||
|
||||
rc = run.main(["--report", "--ledger", str(ledger), "--plan-review"])
|
||||
|
||||
assert rc == 1
|
||||
assert "mode-exclusive" in capsys.readouterr().err
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_hosted_surface_still_refuses_and_names_the_cli_door() -> None:
|
||||
"""T10: hosting keeps its refusal — a synchronous review would block the HTTP request on a
|
||||
reviewer that does not exist, and it would block the event loop that answers ``/readiness``
|
||||
while doing it. What changes is the honesty of the message: there is now an operator door,
|
||||
and the refusal says where.
|
||||
|
||||
Detach point: leave the message pointing only at the library API → RED (the same claim-drift
|
||||
class as the Fase 3 credential line).
|
||||
"""
|
||||
with pytest.raises(ValueError) as excinfo:
|
||||
await hosting.invoke(
|
||||
{
|
||||
"project_id": _PID,
|
||||
"docs_dir": str(_BUNDLE_DIR),
|
||||
"verdict_input": {"decision": "approved", "rationale": "expert reviewed"},
|
||||
"profile": "local",
|
||||
"bundle_dir": str(_BUNDLE_DIR),
|
||||
"explore_prompt": "p",
|
||||
"explore_contract": {
|
||||
"max_rounds": 4,
|
||||
"max_tokens": 200_000,
|
||||
"max_stall_count": 2,
|
||||
"max_reset_count": 1,
|
||||
"max_plan_revisions": 2,
|
||||
"enable_plan_review": True,
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
assert "--plan-review" in str(excinfo.value)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# 3. A real argv, in a real process (the P4 precedent)
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_the_flag_answers_a_review_from_a_real_argv(tmp_path) -> None:
|
||||
"""T11: in-process tests patch ``sys.stdin`` and call ``main()`` directly, so neither proves
|
||||
the flag exists on the parsed command line or that a real pipe reaches the reviewer. A child
|
||||
process settles both — the same reason the demo's stderr and the hosting shim are measured in
|
||||
a subprocess rather than with ``capsys``.
|
||||
|
||||
Detach point: never add the argparse flag → the child exits 2 with an argparse usage error
|
||||
(RED).
|
||||
"""
|
||||
argv = _argv(tmp_path)
|
||||
proc = subprocess.run(
|
||||
[sys.executable, "-m", "portfolio_optimiser.run", *argv],
|
||||
input=f"revise {_FEEDBACK}\napprove\n",
|
||||
capture_output=True,
|
||||
text=True,
|
||||
cwd=_REPO,
|
||||
)
|
||||
|
||||
assert "unrecognized arguments" not in proc.stderr, proc.stderr
|
||||
assert "Traceback" not in proc.stderr, proc.stderr
|
||||
reviews = _artefact(tmp_path)["plan_reviews"]
|
||||
assert [r["decision"] for r in reviews] == ["revise", "approve"]
|
||||
assert reviews[0]["feedback"] == _FEEDBACK
|
||||
|
||||
|
||||
def test_the_reviewer_reads_the_stream_that_exists_when_it_is_asked(monkeypatch) -> None:
|
||||
"""T12: the streams are resolved at CALL time, not when the reviewer is built.
|
||||
|
||||
A factory that captured ``sys.stdin`` at construction would answer from whatever stream
|
||||
happened to be installed when ``run.py`` built the reviewer — before the loop, before anything
|
||||
was asked. Nothing above catches that (every other test here installs its stream first), so the
|
||||
claim would be prose. This asks the question the other way round: build FIRST, swap AFTER.
|
||||
|
||||
Detach point: resolve the streams in ``terminal_plan_reviewer``'s body instead of inside
|
||||
``review`` → RED (the reviewer reads the stream that is gone).
|
||||
"""
|
||||
reviewer = ex.terminal_plan_reviewer()
|
||||
monkeypatch.setattr(sys, "stdin", io.StringIO(f"revise {_FEEDBACK}\n"))
|
||||
monkeypatch.setattr(sys, "stdout", io.StringIO())
|
||||
|
||||
decision = reviewer(
|
||||
ex.PlanReviewRequest(index=0, plan="a plan", current_progress="", is_stalled=False)
|
||||
)
|
||||
|
||||
assert decision.feedback == _FEEDBACK
|
||||
|
|
@ -2,8 +2,16 @@
|
|||
|
||||
NOT default CI: skips cleanly without a configured Foundry endpoint + deployment. When set,
|
||||
``run_portfolio`` fans out over the reference portfolio on the REAL ``azure`` profile (one
|
||||
project, hard token-capped per D6). Mirrors ``test_foundry_profile_live.py:14-24``. SC9 is
|
||||
unchanged — this arm stays skipped offline.
|
||||
project, hard token-capped per D6). SC9 is unchanged — this arm stays skipped offline.
|
||||
|
||||
**``PORTFOLIO_LIVE_FULL_RUN`` added 14.08 (økt 40), and the reason is this file, not the new one.**
|
||||
This test passes NO ``client_factory``, so it drives the real Azure backend: it is a PAID run, and
|
||||
until now it fired on the same two variables as ``test_foundry_profile_live.py``'s one-word probe.
|
||||
That is exactly the collapse the new full-run gate was designed to prevent — an operator exporting
|
||||
the pair to run the CHEAP rung would have paid for this fan-out too, from a bare ``uv run pytest``.
|
||||
The rule the sibling file states must hold for EVERY paid arm, or it is not a rule; leaving this one
|
||||
on the two-var pair would have made the CLAUDE.md invariant half-true the day it was written.
|
||||
Truthiness, not presence (Fase 4b): an exported-but-empty value must not arm a paid run.
|
||||
"""
|
||||
|
||||
import os
|
||||
|
|
@ -14,12 +22,16 @@ from portfolio_optimiser.run import PortfolioResult, run_portfolio
|
|||
|
||||
_ENDPOINT = os.environ.get("PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT")
|
||||
_DEPLOYMENT = os.environ.get("PORTFOLIO_FOUNDRY_DEPLOYMENT")
|
||||
_NO_FOUNDRY = not (_ENDPOINT and _DEPLOYMENT)
|
||||
_OPTED_IN = bool(os.environ.get("PORTFOLIO_LIVE_FULL_RUN"))
|
||||
_NO_FOUNDRY = not (_ENDPOINT and _DEPLOYMENT and _OPTED_IN)
|
||||
|
||||
|
||||
@pytest.mark.skipif(
|
||||
_NO_FOUNDRY,
|
||||
reason="Foundry not configured (set PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT + PORTFOLIO_FOUNDRY_DEPLOYMENT)",
|
||||
reason=(
|
||||
"paid fan-out not armed (set PORTFOLIO_LIVE_FULL_RUN=1 alongside "
|
||||
"PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT + PORTFOLIO_FOUNDRY_DEPLOYMENT)"
|
||||
),
|
||||
)
|
||||
async def test_portfolio_live_azure_fanout() -> None:
|
||||
# No client_factory -> the real AZURE backend is used; hard-capped per D6.
|
||||
|
|
|
|||
|
|
@ -155,3 +155,58 @@ def test_auth_recipe_doc_exists_and_names_facts() -> None:
|
|||
assert "Foundry User" in text
|
||||
assert "services.ai.azure.com" in text
|
||||
assert "necessary-but-not-sufficient" in text
|
||||
|
||||
|
||||
# --- Fase 5: preflight and the run path must know the SAME endpoint variables -----------------
|
||||
# Before this, preflight read ONLY our own name while backends.py accepted the platform-injected
|
||||
# one as a fallback. Inside a hosted Foundry container — where the platform injects
|
||||
# FOUNDRY_PROJECT_ENDPOINT and nothing else — the gate therefore refused a configuration the run
|
||||
# path would have accepted. That is the repo's own "checker and run path validate differently"
|
||||
# defect class, and it is friction the receiver of the handover package pays.
|
||||
|
||||
_INJECTED_ENV = "FOUNDRY_PROJECT_ENDPOINT"
|
||||
_OURS_ENV = "PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT"
|
||||
|
||||
|
||||
def test_injected_endpoint_alone_is_accepted(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||
) -> None:
|
||||
"""A hosted container sets only the platform name. Detach the fallback → RED (refuses)."""
|
||||
monkeypatch.delenv(_OURS_ENV, raising=False)
|
||||
monkeypatch.setenv(_INJECTED_ENV, _VALID_ENDPOINT)
|
||||
monkeypatch.setenv("PORTFOLIO_MODEL_MAP", str(_write_map(tmp_path, _VALID_MAP)))
|
||||
assert isinstance(preflight.check_azure_preflight("azure"), preflight.PreflightOK)
|
||||
|
||||
|
||||
def test_our_name_wins_over_the_injected_one(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||
) -> None:
|
||||
"""Precedence mirrors backends.py: ours first. The injected value here is a WRONG host, so a
|
||||
green result can only mean ours was read — the two arms are distinguishable."""
|
||||
monkeypatch.setenv(_OURS_ENV, _VALID_ENDPOINT)
|
||||
monkeypatch.setenv(_INJECTED_ENV, "https://wrong.openai.azure.com/")
|
||||
monkeypatch.setenv("PORTFOLIO_MODEL_MAP", str(_write_map(tmp_path, _VALID_MAP)))
|
||||
assert isinstance(preflight.check_azure_preflight("azure"), preflight.PreflightOK)
|
||||
|
||||
|
||||
def test_precedence_is_over_values_not_declarations(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||
) -> None:
|
||||
"""An exported-but-EMPTY own name must fall through to a real injected one rather than shadow
|
||||
it into a refusal (the 4b rule, same seam)."""
|
||||
monkeypatch.setenv(_OURS_ENV, "")
|
||||
monkeypatch.setenv(_INJECTED_ENV, _VALID_ENDPOINT)
|
||||
monkeypatch.setenv("PORTFOLIO_MODEL_MAP", str(_write_map(tmp_path, _VALID_MAP)))
|
||||
assert isinstance(preflight.check_azure_preflight("azure"), preflight.PreflightOK)
|
||||
|
||||
|
||||
def test_missing_endpoint_refusal_names_both_variables(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Line-anchored, not substring: PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT CONTAINS
|
||||
FOUNDRY_PROJECT_ENDPOINT, so a message naming only ours satisfies a naive assert (the repo's
|
||||
08-09 defect class). Strip our name before looking for the injected one."""
|
||||
monkeypatch.delenv(_OURS_ENV, raising=False)
|
||||
monkeypatch.delenv(_INJECTED_ENV, raising=False)
|
||||
result = preflight.check_azure_preflight("azure")
|
||||
assert isinstance(result, preflight.PreflightRefusal)
|
||||
assert _OURS_ENV in result.reason
|
||||
assert _INJECTED_ENV in result.reason.replace(_OURS_ENV, "")
|
||||
|
|
|
|||
|
|
@ -19,6 +19,7 @@ def _stamp() -> ProvenanceStamp:
|
|||
role="proposer",
|
||||
validator_decision="validated",
|
||||
token_usage=42,
|
||||
cost_baseline_anchored=True,
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -30,6 +31,7 @@ def test_zero_citations_is_rejected() -> None:
|
|||
role="proposer",
|
||||
validator_decision="validated",
|
||||
token_usage=1,
|
||||
cost_baseline_anchored=True,
|
||||
)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
"""Load-bearing gates on two claims the PUBLISHED surface makes about itself.
|
||||
"""Load-bearing gates on claims the PUBLISHED surface makes about itself.
|
||||
|
||||
AAA+ criterion A5 is that no claim on the public surface is untrue. Two of this repo's claims are
|
||||
made in prose that no test could see, and both drift silently:
|
||||
AAA+ criterion A5 is that no claim on the public surface is untrue. Four of this repo's claims are
|
||||
made in prose that no test could see, and all four drift silently:
|
||||
|
||||
1. ``env.template`` tells the reader which credential the AZURE profile resolves. It said
|
||||
``DefaultAzureCredential`` while :mod:`portfolio_optimiser.backends` has never constructed one —
|
||||
|
|
@ -10,12 +10,22 @@ made in prose that no test could see, and both drift silently:
|
|||
2. ``README.md`` publishes a wheel-install command that spells the wheel's FILENAME, and a wheel
|
||||
filename carries the version. A version bump moves the file the build produces without touching
|
||||
the README, leaving a stranger with an install command for a file that does not exist.
|
||||
3. ``README.md``'s status paragraph said "no live-model run yet". Measured false as of the F1
|
||||
review (2026-08-25): a live run against ``gpt-4.1-mini`` completed 2026-08-14 (see
|
||||
``docs/2026-08-14-fase1b-forste-levende-kjoring.md``) and ended in a correct ``rejected``
|
||||
outcome.
|
||||
4. ``README.md``'s status paragraph said the Claude Agents SDK sibling is "built in parallel".
|
||||
Measured false the same day: ``STATE.md`` records it as parked.
|
||||
|
||||
Both gates read the source artefacts as RAW TEXT, because that is the only thing that can see prose.
|
||||
Both are LINE-ANCHORED rather than substring-matched: ``backends.py`` NAMES ``DefaultAzureCredential``
|
||||
four times in the comments that explain why it is not used, so a whole-file substring check would be
|
||||
red on exactly the prose it protects (this repo's 08-09 defect class, and the reason the 4e
|
||||
``azure.yaml`` gate is line-anchored too).
|
||||
Most gates read the source artefacts as RAW TEXT, because that is the only thing that can see
|
||||
prose. Gates 1 and 2 are LINE-ANCHORED rather than substring-matched: ``backends.py`` NAMES
|
||||
``DefaultAzureCredential`` four times in the comments that explain why it is not used, so a
|
||||
whole-file substring check would be red on exactly the prose it protects (this repo's 08-09 defect
|
||||
class, and the reason the handover package's python-only gate matches archive member NAMES rather
|
||||
than document prose). Gates 3 and 4 are literal regression guards on the specific wording measured
|
||||
false — ``STATE.md`` is local-only (gitignored, not shipped to a fresh clone) and cannot serve as a
|
||||
portable ground-truth source for gate 4, so that gate cannot verify the CURRENT sibling status,
|
||||
only refuse the specific false claim already made once.
|
||||
|
||||
Each positive assertion is paired with a CONTROL that the thing being searched for is actually
|
||||
present. An extractor that silently finds nothing makes a gate that can only ever be green, which
|
||||
|
|
@ -34,6 +44,7 @@ _ENV_TEMPLATE = _REPO_ROOT / "env.template"
|
|||
_README = _REPO_ROOT / "README.md"
|
||||
_PYPROJECT = _REPO_ROOT / "pyproject.toml"
|
||||
_BACKENDS = _REPO_ROOT / "src" / "portfolio_optimiser" / "backends.py"
|
||||
_LIVE_RUN_EVIDENCE = _REPO_ROOT / "docs" / "2026-08-14-fase1b-forste-levende-kjoring.md"
|
||||
|
||||
# The credential is chosen on ONE assignment statement. Reading the credential names off that line —
|
||||
# rather than off the whole module — is what keeps the explanatory comments out of the measurement.
|
||||
|
|
@ -43,6 +54,9 @@ _CREDENTIAL_CALL = re.compile(r"(\w*Credential)\s*\(")
|
|||
# A wheel filename spells the distribution, the version and the tags. The version is the drifting part.
|
||||
_WHEEL_FILENAME = re.compile(r"portfolio_optimiser-(?P<version>[0-9][^-\s]*)-py3-none-any\.whl")
|
||||
|
||||
# The completed-live-run record on the evidence doc, tolerant of the exact column width used there.
|
||||
_LIVE_RUN_OUTCOME = re.compile(r"outcome_type:\s+rejected")
|
||||
|
||||
# `[project]`'s own version line: the value hatchling stamps into the wheel filename.
|
||||
_PROJECT_VERSION = re.compile(r'^version\s*=\s*"(?P<version>[^"]+)"', re.MULTILINE)
|
||||
|
||||
|
|
@ -133,7 +147,56 @@ def test_readme_wheel_command_cites_the_built_version() -> None:
|
|||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("artefact", [_ENV_TEMPLATE, _README, _PYPROJECT, _BACKENDS])
|
||||
# --------------------------------------------------------------------------------------------
|
||||
# F1 (2026-08-25) — the README status paragraph does not repeat two claims measured false
|
||||
# --------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_live_run_evidence_records_a_completed_run() -> None:
|
||||
"""CONTROL. The negative assertion below is only meaningful while this run is on record."""
|
||||
text = _LIVE_RUN_EVIDENCE.read_text(encoding="utf-8")
|
||||
assert _LIVE_RUN_OUTCOME.search(text), (
|
||||
f"{_LIVE_RUN_EVIDENCE} no longer records a completed live run with outcome_type: "
|
||||
"rejected — the claim the status paragraph must not contradict has nothing to guard it"
|
||||
)
|
||||
|
||||
|
||||
def _readme_prose() -> str:
|
||||
"""``README.md``'s status paragraph is a markdown blockquote wrapped across several lines,
|
||||
each continued with ``"> "``. Rendered markdown collapses those soft line breaks into one
|
||||
paragraph, so a raw substring search must normalise the same way first — otherwise a claim
|
||||
that happens to wrap across a line boundary is invisible to the gate (measured: "no
|
||||
live-model run yet" wraps as "no live-model\\n> run yet" in the raw file).
|
||||
"""
|
||||
text = _README.read_text(encoding="utf-8")
|
||||
return re.sub(r"\n>\s*", " ", text)
|
||||
|
||||
|
||||
def test_readme_status_does_not_claim_no_live_run() -> None:
|
||||
"""A live run against a real endpoint completed 2026-08-14 (see ``_LIVE_RUN_EVIDENCE``); the
|
||||
status paragraph may not say otherwise, even though no run has yet produced a *validated*
|
||||
proposal against a live model.
|
||||
"""
|
||||
assert "no live-model run yet" not in _readme_prose(), (
|
||||
"README.md claims no live-model run has happened, but one completed on 2026-08-14 "
|
||||
f"(see {_LIVE_RUN_EVIDENCE.name})"
|
||||
)
|
||||
|
||||
|
||||
def test_readme_status_does_not_claim_sibling_built_in_parallel() -> None:
|
||||
"""The Claude Agents SDK sibling is parked (``STATE.md``: 'po-claude PARKERT'); the status
|
||||
paragraph may not claim active parallel development. ``STATE.md`` is local-only and is not
|
||||
read here — this is a literal regression guard on the specific claim measured false on
|
||||
2026-08-25, not a check against the sibling's current status.
|
||||
"""
|
||||
assert "built in parallel" not in _readme_prose(), (
|
||||
"README.md claims the Claude Agents SDK sibling is built in parallel, but it is parked"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"artefact", [_ENV_TEMPLATE, _README, _PYPROJECT, _BACKENDS, _LIVE_RUN_EVIDENCE]
|
||||
)
|
||||
def test_guarded_artefacts_exist(artefact: Path) -> None:
|
||||
"""CONTROL. A missing artefact must fail here rather than turn a gate into a no-op."""
|
||||
assert artefact.is_file(), f"{artefact} is missing; the gates above would read nothing"
|
||||
|
|
|
|||
|
|
@ -86,9 +86,12 @@ def test_scripted_door_runs_the_whole_loop_offline(bundle, replies_file, capsys)
|
|||
rc = run.main(_argv(bundle, replies_file))
|
||||
out = capsys.readouterr().out
|
||||
assert rc == 0, out
|
||||
# A full run reports its outcome type + the minted verdict id; a dry-run never gets this far.
|
||||
# A full run reports its outcome type + the candidate's verdict key; a dry-run never gets this
|
||||
# far. Since F2 this argv records NO expert verdict (no --decision/--rationale), so the line
|
||||
# says so and quotes the key one would arrive under — asserting "verdict id=" here would be
|
||||
# asserting that a run nobody reviewed minted an approval.
|
||||
assert "BYGG-KONTOR-NORD:" in out
|
||||
assert "verdict id=" in out
|
||||
assert "no expert verdict given; verdict key=" in out
|
||||
assert "LIVE-DRY-RUN" not in out
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -39,6 +39,11 @@ _DELEGATING_OVERRIDES = [
|
|||
# It cannot live in the reply-selector seam either — that seam returns a reply STRING, and a
|
||||
# response that is not text is precisely this double's subject.
|
||||
"tests/test_b4_mcp_call_trace_loadbearing.py",
|
||||
# Fase 1b structured-output probe: records the ``options`` mapping of every generation call,
|
||||
# then delegates. It cannot live in the reply-selector seam either — that seam is handed
|
||||
# ``(prompt_blob, role)`` and returns a reply string, and ``options`` (this double's whole
|
||||
# subject) never reaches it.
|
||||
"tests/test_structured_output_loadbearing.py",
|
||||
]
|
||||
|
||||
# Doubles in a DIFFERENT lineage (``spikes._harness.FakeChatClient``). There is no canonical
|
||||
|
|
|
|||
147
tests/test_scripted_explore_door_loadbearing.py
Normal file
147
tests/test_scripted_explore_door_loadbearing.py
Normal file
|
|
@ -0,0 +1,147 @@
|
|||
"""MAJOR-2 (docs/2026-08-25-syretest-vei-ab.md) — ``--explore --scripted-replies`` must not
|
||||
crash with a raw ``KeyError: 'navigator'``.
|
||||
|
||||
``_SCRIPTED_ROLES = ("proposer", "checker")`` (``run.py:1531``) is the debate's two roles.
|
||||
``explore()`` asks the SAME ``client_factory`` for three more: ``manager``, ``navigator``,
|
||||
``hypothesiser`` (``explore.py:576``). ``_load_scripted_replies`` was fail-fast for the two roles
|
||||
it knew about — measured (docs/2026-08-25-syretest-vei-ab.md § MAJOR-2) to let the three it did not
|
||||
know about surface exactly the ``KeyError`` deep inside ``scripted_factory``'s lookup its own
|
||||
docstring warns against, mid-run, after the banner had already printed.
|
||||
|
||||
The fix widens the required-role set to include the exploration's three roles WHEN ``--explore``
|
||||
is in play, so a missing role is refused BY NAME before any model/agent work starts — the same
|
||||
door, never a second one.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from portfolio_optimiser import run
|
||||
|
||||
_BUNDLE_DIR = Path(__file__).resolve().parents[1] / "shared" / "examples" / "bygg-energi-mikro"
|
||||
_PID = "BYGG-KONTOR-NORD"
|
||||
|
||||
_PROPOSER_REPLY = json.dumps(
|
||||
{
|
||||
"measure": "LED-retrofit",
|
||||
"affected_items": [{"code": "ENERGI-TOTAL-EL", "quantity": 300000, "unit_cost": 1.0}],
|
||||
"claimed_saving_nok": 30000,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _config_file(tmp_path: Path, **overrides: Any) -> str:
|
||||
path = tmp_path / "exploration.json"
|
||||
path.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"max_rounds": 4,
|
||||
"max_tokens": 100_000,
|
||||
"max_stall_count": 2,
|
||||
"max_reset_count": 1,
|
||||
"max_plan_revisions": 0,
|
||||
"enable_plan_review": False,
|
||||
**overrides,
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
return str(path)
|
||||
|
||||
|
||||
def _replies_file(tmp_path: Path, roles: dict[str, str]) -> str:
|
||||
path = tmp_path / "replies.json"
|
||||
path.write_text(json.dumps(roles), encoding="utf-8")
|
||||
return str(path)
|
||||
|
||||
|
||||
def _base_argv(tmp_path: Path, replies_path: str) -> list[str]:
|
||||
return [
|
||||
_PID,
|
||||
"--docs-dir",
|
||||
str(_BUNDLE_DIR),
|
||||
"--bundle-dir",
|
||||
str(_BUNDLE_DIR),
|
||||
"--explore",
|
||||
"Find the cheapest saving.",
|
||||
"--explore-config",
|
||||
_config_file(tmp_path),
|
||||
"--scripted-replies",
|
||||
replies_path,
|
||||
]
|
||||
|
||||
|
||||
def test_explore_with_debate_only_scripted_replies_is_refused_by_name(tmp_path, capsys) -> None:
|
||||
"""A ``--scripted-replies`` file that only answers the debate (proposer/checker) — exactly the
|
||||
file MAJOR-2 was measured against — is refused BY NAME, never left to crash mid-run.
|
||||
|
||||
Detach point: revert ``_SCRIPTED_ROLES`` to the fixed debate-only tuple used for ``--explore``
|
||||
too → this raises an unhandled ``KeyError`` instead of returning 1 (RED, reproduces MAJOR-2).
|
||||
"""
|
||||
replies = _replies_file(tmp_path, {"proposer": _PROPOSER_REPLY, "checker": "VERDICT: APPROVE"})
|
||||
|
||||
rc = run.main(_base_argv(tmp_path, replies))
|
||||
|
||||
assert rc == 1, "a role the exploration needs is missing — this must be a clean refusal"
|
||||
err = capsys.readouterr().err
|
||||
assert "run refused" in err
|
||||
for role in ("manager", "navigator", "hypothesiser"):
|
||||
assert role in err, f"the refusal must name the missing role {role!r}"
|
||||
|
||||
|
||||
def test_explore_with_debate_only_scripted_replies_never_reaches_a_model_call(
|
||||
tmp_path, capsys, monkeypatch
|
||||
) -> None:
|
||||
"""The refusal fires at the DOOR — before ``explore()`` is even entered. A spy on ``explore``
|
||||
proves zero exploration work happened, the same way econ 57's outbox/run-id hoist was proved.
|
||||
|
||||
Detach point: let the flag through and catch the ``KeyError`` further in → this spy would still
|
||||
record a call (RED).
|
||||
"""
|
||||
calls: list[object] = []
|
||||
monkeypatch.setattr(
|
||||
"portfolio_optimiser.run.explore",
|
||||
lambda *a, **kw: (
|
||||
calls.append((a, kw)) or (_ for _ in ()).throw(AssertionError("unreached"))
|
||||
),
|
||||
)
|
||||
replies = _replies_file(tmp_path, {"proposer": _PROPOSER_REPLY, "checker": "VERDICT: APPROVE"})
|
||||
|
||||
rc = run.main(_base_argv(tmp_path, replies))
|
||||
|
||||
assert rc == 1
|
||||
assert calls == [], "explore() must never be entered when a required role is missing"
|
||||
|
||||
|
||||
def test_explore_with_the_full_five_role_scripted_replies_does_not_crash(tmp_path, capsys) -> None:
|
||||
"""With every role ``explore()`` can ask for supplied as a constant string, the CLI door must
|
||||
run the loop to completion (or a typed budget/refusal outcome) — never a raw traceback.
|
||||
|
||||
A constant per-role reply cannot answer every stage-specific shape the magentic manager can be
|
||||
prompted with (facts / plan / progress-ledger JSON / final answer all differ) — so this does
|
||||
not assert the exploration finds anything, only that the documented crash is gone.
|
||||
"""
|
||||
replies = _replies_file(
|
||||
tmp_path,
|
||||
{
|
||||
"proposer": _PROPOSER_REPLY,
|
||||
"checker": "VERDICT: APPROVE",
|
||||
"manager": '{"is_request_satisfied": {"reason": "r", "answer": true}, '
|
||||
'"is_in_loop": {"reason": "r", "answer": false}, '
|
||||
'"is_progress_being_made": {"reason": "r", "answer": true}, '
|
||||
'"next_speaker": {"reason": "r", "answer": "hypothesiser"}, '
|
||||
'"instruction_or_question": {"reason": "r", "answer": "go"}}',
|
||||
"navigator": "NAVIGATOR: read the index.",
|
||||
"hypothesiser": "HYPOTHESIS: " + json.dumps({"label": "x", "rationale": "y"}),
|
||||
},
|
||||
)
|
||||
|
||||
rc = run.main(_base_argv(tmp_path, replies))
|
||||
|
||||
err = capsys.readouterr().err
|
||||
assert "KeyError" not in err
|
||||
assert "Traceback" not in err
|
||||
assert rc in (0, 1), f"expected a clean exit, got rc={rc} stderr={err!r}"
|
||||
338
tests/test_structured_output_loadbearing.py
Normal file
338
tests/test_structured_output_loadbearing.py
Normal file
|
|
@ -0,0 +1,338 @@
|
|||
"""Load-bearing: the proposer call must carry a STRICT structured-output schema (Fase 1b, funn 1b).
|
||||
|
||||
The gap, measured on the project's first live run
|
||||
(``docs/2026-08-14-fase1b-forste-levende-kjoring.md``): ``generate_via_llm`` asked for the typed IR
|
||||
in PROSE ("Respond with ONLY a JSON object ...") and nothing else. Twelve rounds burned on replies
|
||||
that did not parse, the run died with ``BudgetExceeded``, and the validator was never reached with a
|
||||
parseable candidate. ``e371890`` made the raw text visible; this seam removes the cause: the model is
|
||||
handed a GRAMMAR, not a request.
|
||||
|
||||
**The wire form is decided by measurement, not from memory** (MAF 1.9.0, agent-framework-openai
|
||||
1.8.2, agent-framework-foundry 1.8.2):
|
||||
|
||||
* ``ChatOptions`` carries ``response_format: type[BaseModel] | Mapping[str, Any] | None``, and BOTH
|
||||
wired profiles honour it. LOCAL (``OpenAIChatCompletionClient``) passes a Mapping through verbatim
|
||||
to Chat Completions; AZURE (``FoundryChatClient`` -> ``RawFoundryChatClient`` ->
|
||||
``RawOpenAIChatClient``) converts the SAME mapping into the Responses API's ``text.format``. One
|
||||
mapping, both profiles.
|
||||
* A ``type[BaseModel]`` would be the shorter spelling, and it is REJECTED here on evidence. The
|
||||
client then converts it with ``openai.lib._parsing._completions.type_to_response_format_param``,
|
||||
which (measured) emits ``minimum`` / ``exclusiveMinimum`` / ``minItems`` / ``prefixItems`` and an
|
||||
``assumptions`` node whose ``additionalProperties`` is a SCHEMA. Azure's documented structured-
|
||||
output subset lists every one of those as unsupported and requires ``additionalProperties: false``
|
||||
in every object
|
||||
(https://learn.microsoft.com/azure/foundry/openai/how-to/structured-outputs). Passing our own
|
||||
mapping is the only way to control what actually goes on the wire.
|
||||
|
||||
**Stripping the constraints loses nothing**, and that is the point of the split: the schema's job is
|
||||
SHAPE, the deterministic validator's job is VALUES. ``minItems``/``gt=0`` are re-applied by pydantic
|
||||
in ``_parse_ir`` and by ``validate_proposal`` — which the module docstring already names as the
|
||||
reliability mechanism.
|
||||
|
||||
**``assumptions`` may NOT simply be dropped, and that is a measurement too.** It is the one IR field
|
||||
inexpressible in the strict subset (a free-form map of 2-tuples). Dropping it looks harmless because
|
||||
the field is optional — but ``validator._monte_carlo`` samples ``item.unit_cost`` unchanged when a
|
||||
code has no band, so with no bands at all the 512 samples are IDENTICAL and P10 == P50 == P90. The
|
||||
stochastic falsifier would go inert while still reporting percentiles: a gate that can only be green,
|
||||
which is this repo's cardinal defect class. So the schema carries the bands in a strict-legal
|
||||
ARRAY-of-entries form and ``_parse_ir`` normalises them back to the IR's map — the IR itself is
|
||||
untouched.
|
||||
|
||||
Five tests, load-bearing as a set. Each detach point is RED on its own:
|
||||
|
||||
* T1 — the WIRING: the response_format reaches the client's ``options`` on the generation call
|
||||
(RED when ``options=`` is dropped, or when a bare pydantic class is passed instead of the mapping);
|
||||
* T2 — the schema is inside the DOCUMENTED subset, recursively, with a paired CONTROL proving the
|
||||
same walker finds those keywords in the RAW pydantic schema (RED when the sanitiser is detached —
|
||||
and non-vacuous, because the control proves the walker reaches nested ``$defs`` at all);
|
||||
* T3 — the Monte Carlo falsifier SURVIVES: a wire reply with bands yields P90 > P10, with a control
|
||||
proving a band-less reply collapses to P90 == P10 (RED when ``assumptions`` is dropped from the
|
||||
schema or the bands do not reach the IR);
|
||||
* T4 — FAIL-CLOSED: a model carrying an inexpressible node with no declared override RAISES rather
|
||||
than emitting an illegal schema (RED on a sanitiser that silently skips what it cannot express);
|
||||
* T5 — the round trip is VERBATIM and ADDITIVE: array-form bands reach the IR with their exact
|
||||
values, and the IR's own map form is still accepted unchanged.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections.abc import Iterator, Mapping, Sequence
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from agent_framework import ChatResponse, ChatResponseUpdate, Message
|
||||
from agent_framework._types import ResponseStream
|
||||
from pydantic import BaseModel
|
||||
|
||||
from portfolio_optimiser.budget import Budget, TokenMeter
|
||||
from portfolio_optimiser.generate import (
|
||||
UNSUPPORTED_SCHEMA_KEYWORDS,
|
||||
StructuredOutputUnsupported,
|
||||
generate_via_llm,
|
||||
proposal_response_format,
|
||||
strict_json_schema,
|
||||
)
|
||||
from portfolio_optimiser.ir import SavingsProposal
|
||||
from portfolio_optimiser.reference_domain import load_reference_projects
|
||||
from portfolio_optimiser.simulation import ScriptedChatClient
|
||||
from portfolio_optimiser.validator import ValidatedProposal
|
||||
|
||||
#: The instruction line ``generate._build_messages`` puts in EVERY generation prompt and nowhere
|
||||
#: else — the one identifier that separates a generation call from a debate turn.
|
||||
_GENERATION_MARK = "Respond with ONLY a JSON object"
|
||||
|
||||
#: One affected line: 100 x 10 = 1000 total -> nominal feasible = 0.30 * 1000 = 300.
|
||||
#: A band of (8, 12) encloses the unit_cost (the IR's own model_validator requires that) and makes
|
||||
#: the sampled totals range over 800..1200, i.e. feasible 240..360. A claim of 200 therefore clears
|
||||
#: BOTH the P90 stage and the nominal stage whether or not bands are present — so the ONLY thing
|
||||
#: that differs between T3's positive and its control is the band itself.
|
||||
_CODE = "STRUCTURED-OUT-LINE"
|
||||
_QUANTITY = 100.0
|
||||
_UNIT_COST = 10.0
|
||||
_BAND_LOW = 8.0
|
||||
_BAND_HIGH = 12.0
|
||||
_CLAIM = 200.0
|
||||
#: NOT ``energy_efficiency`` — that measure would additionally hit the method cap (0.15 * 1000 = 150)
|
||||
#: and reject a claim of 200 for a reason that has nothing to do with this seam.
|
||||
_MEASURE = "behovsstyrt_drift"
|
||||
|
||||
|
||||
def _wire_reply(*, with_band: bool) -> str:
|
||||
"""A reply in exactly the shape the strict schema commissions: ``assumptions`` is an ARRAY of
|
||||
entries, never the IR's map. Both arms are byte-identical apart from that array."""
|
||||
bands = (
|
||||
[{"code": _CODE, "low_unit_cost": _BAND_LOW, "high_unit_cost": _BAND_HIGH}]
|
||||
if with_band
|
||||
else []
|
||||
)
|
||||
return json.dumps(
|
||||
{
|
||||
"project_id": "FV42-GSV-E1",
|
||||
"measure": _MEASURE,
|
||||
"affected_items": [{"code": _CODE, "quantity": _QUANTITY, "unit_cost": _UNIT_COST}],
|
||||
"claimed_saving_nok": _CLAIM,
|
||||
"assumptions": bands,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _meter() -> TokenMeter:
|
||||
# Caps well above what one attempt needs, so max_attempts -- not the budget -- is the bound.
|
||||
return TokenMeter(Budget(max_tokens=10**9, max_rounds=20))
|
||||
|
||||
|
||||
class _OptionsRecordingChatClient(ScriptedChatClient):
|
||||
"""Records the ``options`` mapping of every call, then DELEGATES to the canonical scripted body.
|
||||
|
||||
This double cannot live in the ``reply_selector`` seam: that seam is handed
|
||||
``(prompt_blob, role)`` and returns a reply string, and ``options`` — this double's entire
|
||||
subject — never reaches it. Registered in ``test_scripted_client_consolidation``'s
|
||||
``_DELEGATING_OVERRIDES`` for exactly that reason.
|
||||
"""
|
||||
|
||||
def __init__(self, reply: str) -> None:
|
||||
super().__init__(reply)
|
||||
self.seen_options: list[Mapping[str, Any]] = []
|
||||
|
||||
def _inner_get_response(
|
||||
self,
|
||||
*,
|
||||
messages: Sequence[Message],
|
||||
options: Mapping[str, Any],
|
||||
stream: bool = False,
|
||||
**kwargs: Any,
|
||||
) -> Any | ResponseStream[ChatResponseUpdate, ChatResponse]:
|
||||
blob = " ".join(getattr(m, "text", "") or "" for m in messages)
|
||||
if _GENERATION_MARK in blob:
|
||||
self.seen_options.append(dict(options))
|
||||
return super()._inner_get_response(
|
||||
messages=messages, options=options, stream=stream, **kwargs
|
||||
)
|
||||
|
||||
|
||||
def _sub_schemas(node: Any) -> Iterator[Mapping[str, Any]]:
|
||||
"""Every mapping in the schema tree, including those under ``$defs`` — the walker both T2 and
|
||||
its control run, so the control genuinely proves the walker's reach."""
|
||||
if isinstance(node, Mapping):
|
||||
yield node
|
||||
for value in node.values():
|
||||
yield from _sub_schemas(value)
|
||||
elif isinstance(node, list):
|
||||
for value in node:
|
||||
yield from _sub_schemas(value)
|
||||
|
||||
|
||||
def _keywords_found(schema: Any) -> set[str]:
|
||||
return {
|
||||
key for node in _sub_schemas(schema) for key in node if key in UNSUPPORTED_SCHEMA_KEYWORDS
|
||||
}
|
||||
|
||||
|
||||
async def test_generation_call_carries_the_strict_schema() -> None:
|
||||
"""T1 LOAD-BEARING: the strict response_format reaches the client on the generation call.
|
||||
|
||||
RED when ``generate_via_llm`` stops passing ``options=``, and RED when it passes a bare
|
||||
``type[BaseModel]`` instead of the sanitised mapping (the assert is on the mapping's identity,
|
||||
not merely on the key's presence)."""
|
||||
project = load_reference_projects()[0]
|
||||
client = _OptionsRecordingChatClient(_wire_reply(with_band=True))
|
||||
|
||||
await generate_via_llm(client, project, "", _meter(), max_attempts=1)
|
||||
|
||||
# Control FIRST: a positive assert over an empty list would pass vacuously.
|
||||
assert client.seen_options, "no generation call was observed — the assert below proves nothing"
|
||||
assert client.seen_options[0].get("response_format") == proposal_response_format(), (
|
||||
"the generation call did not carry the strict structured-output schema"
|
||||
)
|
||||
|
||||
|
||||
def test_schema_stays_inside_the_documented_strict_subset() -> None:
|
||||
"""T2 LOAD-BEARING: the emitted schema uses only what Azure's structured-output subset allows.
|
||||
|
||||
RED when the sanitiser is detached (the raw pydantic schema's keywords come straight through).
|
||||
The CONTROL is what makes it non-vacuous: the SAME walker must FIND those keywords in the raw
|
||||
schema — including inside ``$defs``, where ``AffectedItem``'s constraints live. Without it, a
|
||||
walker that silently visits nothing would make every assert below green."""
|
||||
raw = SavingsProposal.model_json_schema()
|
||||
raw_found = _keywords_found(raw)
|
||||
assert raw_found, (
|
||||
"control failed: the walker found NO unsupported keyword in the raw pydantic schema, so "
|
||||
"the positive assert below cannot distinguish a working sanitiser from a dead walker"
|
||||
)
|
||||
assert "$defs" in raw and _keywords_found(raw["$defs"]), (
|
||||
"control failed: the walker does not reach nested $defs, where AffectedItem's constraints "
|
||||
"live — a sanitiser that skipped $defs would still pass"
|
||||
)
|
||||
|
||||
schema = proposal_response_format()["json_schema"]["schema"]
|
||||
assert _keywords_found(schema) == set(), (
|
||||
f"emitted schema carries keywords outside the documented strict subset: "
|
||||
f"{sorted(_keywords_found(schema))}"
|
||||
)
|
||||
for node in _sub_schemas(schema):
|
||||
if "properties" not in node:
|
||||
continue
|
||||
assert node.get("additionalProperties") is False, (
|
||||
f"object node without ``additionalProperties: false``: {sorted(node.get('properties'))}"
|
||||
)
|
||||
assert set(node.get("required", [])) == set(node["properties"]), (
|
||||
f"strict mode requires EVERY property listed as required; node "
|
||||
f"{sorted(node['properties'])} lists {sorted(node.get('required', []))}"
|
||||
)
|
||||
|
||||
|
||||
async def test_assumption_bands_keep_the_monte_carlo_falsifier_alive() -> None:
|
||||
"""T3 LOAD-BEARING: the commissioned schema can still carry uncertainty bands, and they reach
|
||||
the Monte Carlo.
|
||||
|
||||
Two halves, because a scripted reply cannot be constrained by the schema the way a live model
|
||||
is — so the schema half must be asserted DIRECTLY or this test could not tell a schema that
|
||||
commissions bands from one that does not:
|
||||
|
||||
* the schema still COMMISSIONS the bands (RED when ``assumptions`` is dropped from it — which
|
||||
would leave a live model unable to supply a band at all);
|
||||
* a band that does arrive REACHES the Monte Carlo (RED when the array form is not folded into
|
||||
the IR).
|
||||
|
||||
The CONTROL is the band-less arm — it proves the spread asserted below is caused by the band
|
||||
and not by the Monte Carlo being noisy in general."""
|
||||
project = load_reference_projects()[0]
|
||||
|
||||
schema = proposal_response_format()["json_schema"]["schema"]
|
||||
assert "assumptions" in schema["properties"], (
|
||||
"the schema no longer commissions uncertainty bands — a live model constrained by it could "
|
||||
"not supply one, and the Monte Carlo would be degenerate on every generated proposal"
|
||||
)
|
||||
assert "assumptions" in schema["required"]
|
||||
|
||||
banded = await generate_via_llm(
|
||||
_OptionsRecordingChatClient(_wire_reply(with_band=True)),
|
||||
project,
|
||||
"",
|
||||
_meter(),
|
||||
max_attempts=1,
|
||||
)
|
||||
bandless = await generate_via_llm(
|
||||
_OptionsRecordingChatClient(_wire_reply(with_band=False)),
|
||||
project,
|
||||
"",
|
||||
_meter(),
|
||||
max_attempts=1,
|
||||
)
|
||||
|
||||
assert isinstance(banded.outcome, ValidatedProposal), f"fixture: {banded.outcome}"
|
||||
assert isinstance(bandless.outcome, ValidatedProposal), f"fixture: {bandless.outcome}"
|
||||
|
||||
# The control: with no band the falsifier is degenerate — this is the state the seam must NOT
|
||||
# silently ship.
|
||||
assert bandless.outcome.p90 == bandless.outcome.p10, (
|
||||
"control failed: a band-less proposal already spreads, so the spread asserted below would "
|
||||
"not prove the band arrived"
|
||||
)
|
||||
assert banded.outcome.p90 > banded.outcome.p10, (
|
||||
"the uncertainty band did not reach the Monte Carlo — the stochastic falsifier is inert"
|
||||
)
|
||||
|
||||
|
||||
def test_sanitiser_is_fail_closed_on_an_inexpressible_node() -> None:
|
||||
"""T4 LOAD-BEARING: a node the strict subset cannot express RAISES rather than being emitted or
|
||||
silently skipped.
|
||||
|
||||
Without this, a field added to the IR later would quietly produce an illegal schema and turn
|
||||
every live generation call into a 400 — or, worse, be dropped so the model is never asked for
|
||||
it. Validation, never repair (mirrors ``write_concept_file`` / ``promote_verdict``)."""
|
||||
|
||||
class _FreeFormMap(BaseModel):
|
||||
label: str
|
||||
bands: dict[str, float] # free-form map: additionalProperties is a SCHEMA, not ``false``
|
||||
|
||||
with pytest.raises(StructuredOutputUnsupported):
|
||||
strict_json_schema(_FreeFormMap)
|
||||
|
||||
class _Expressible(BaseModel):
|
||||
label: str
|
||||
count: int
|
||||
|
||||
# Control: the raise above is caused by the inexpressible node, not by the sanitiser rejecting
|
||||
# everything it is handed.
|
||||
assert strict_json_schema(_Expressible)["properties"].keys() == {"label", "count"}
|
||||
|
||||
|
||||
async def test_band_round_trip_is_verbatim_and_the_ir_map_form_still_parses() -> None:
|
||||
"""T5 LOAD-BEARING: the array form reaches the IR with its EXACT values, and the change is
|
||||
ADDITIVE — the IR's own map form is still accepted.
|
||||
|
||||
RED when the normalisation is detached (the array never becomes a map), and RED when it is
|
||||
written as a REPLACEMENT rather than an addition (the map form would stop parsing, which would
|
||||
break every existing scripted reply in the suite)."""
|
||||
project = load_reference_projects()[0]
|
||||
|
||||
from_array = await generate_via_llm(
|
||||
_OptionsRecordingChatClient(_wire_reply(with_band=True)),
|
||||
project,
|
||||
"",
|
||||
_meter(),
|
||||
max_attempts=1,
|
||||
)
|
||||
assert isinstance(from_array.outcome, ValidatedProposal)
|
||||
assert from_array.outcome.proposal.assumptions == {_CODE: (_BAND_LOW, _BAND_HIGH)}, (
|
||||
"the array-form band did not reach the IR verbatim"
|
||||
)
|
||||
|
||||
map_form = json.dumps(
|
||||
{
|
||||
"project_id": "FV42-GSV-E1",
|
||||
"measure": _MEASURE,
|
||||
"affected_items": [{"code": _CODE, "quantity": _QUANTITY, "unit_cost": _UNIT_COST}],
|
||||
"claimed_saving_nok": _CLAIM,
|
||||
"assumptions": {_CODE: [_BAND_LOW, _BAND_HIGH]},
|
||||
}
|
||||
)
|
||||
from_map = await generate_via_llm(
|
||||
_OptionsRecordingChatClient(map_form), project, "", _meter(), max_attempts=1
|
||||
)
|
||||
assert isinstance(from_map.outcome, ValidatedProposal), (
|
||||
"the IR's own map form stopped parsing — the normalisation replaced rather than extended"
|
||||
)
|
||||
assert from_map.outcome.proposal.assumptions == {_CODE: (_BAND_LOW, _BAND_HIGH)}
|
||||
486
tests/test_tracing_loadbearing.py
Normal file
486
tests/test_tracing_loadbearing.py
Normal file
|
|
@ -0,0 +1,486 @@
|
|||
"""U14 (økt 55) — the tracing seam: opt-in OpenTelemetry, and no silent egress.
|
||||
|
||||
**Why the seam exists.** The exploration loop this program is building hands a manager the freedom
|
||||
to decide what to look at next. The programme's load-bearing ordering rule is that *nothing which
|
||||
gives the manager more freedom lands before we can see what it did with it* — a loop without a
|
||||
trace is a black box, and a black box is not something an organisation can be asked to trust. The
|
||||
measured starting point (spike S6, økt 54) is that MAF's own Magentic code emits **zero** spans of
|
||||
its own, while ``ENABLE_INSTRUMENTATION`` defaults to ``True``, so spans are already being *made*
|
||||
today and thrown away for want of a provider. This seam is the provider — nothing more.
|
||||
|
||||
**What is measured here, and what is deliberately not.**
|
||||
|
||||
* Measured: that tracing is OFF unless asked for (and then MAF is never called at all), that
|
||||
console mode writes to **stderr** so the pinned demo transcript stays byte-identical, that a
|
||||
console run cannot silently also ship spans over the wire, and that the OTLP mode refuses to
|
||||
pretend when no endpoint was declared.
|
||||
* Not built: the ``PLAN_CREATED`` / ``REPLANNED`` / ``PROGRESS_LEDGER_UPDATED`` events the plan
|
||||
names. They belong to a loop that does not exist yet (U4, økt 56-57). An emitter with no call
|
||||
site is a shape guessed instead of measured, and this repo has paid for that guess before.
|
||||
|
||||
**The measurement that shaped the code, not a preference.** ``configure_otel_providers`` composes
|
||||
its exporter list in a fixed order (``observability.py:849``): (1) exporters derived from the
|
||||
standard ``OTEL_EXPORTER_OTLP_*`` environment variables, **unconditionally**, (2) exporters passed
|
||||
in, (3) a ``ConsoleSpanExporter()`` — whose default sink is **stdout** — if
|
||||
``enable_console_exporters`` is true, from the argument *or* from ``ENABLE_CONSOLE_EXPORTERS`` in
|
||||
the environment. Two consequences drive two of the tests below:
|
||||
|
||||
1. ``enable_console_exporters`` must be passed **explicitly False**, in both modes. Left to the
|
||||
environment, an operator with ``ENABLE_CONSOLE_EXPORTERS=true`` exported gets a stdout span
|
||||
dump — which is precisely the thing S6 measured as destroying the golden transcript.
|
||||
2. Console mode must **refuse** when an OTLP endpoint variable is set, because step (1) would add
|
||||
a network exporter that the word "console" promises is not there. Validation, NEVER repair: we
|
||||
do not unset the operator's environment behind their back (the ``write_concept_file`` rule).
|
||||
|
||||
**The subprocess arm is the measurement; the spy arms only prove wiring.** Every unit test here
|
||||
injects a recorder in place of ``configure_otel_providers``, and a suite made only of those would
|
||||
prove that we call *something* named right — the exact vacuity 4b's credential tests were rewritten
|
||||
to escape. So the real arm runs the actual demo as a subprocess under ``PORTFOLIO_OTEL=console``
|
||||
and reads what came out: at least one ``workflow.run`` span on stderr, and stdout byte-identical to
|
||||
``tests/golden/demo-transcript.stdout``. Its control runs the same demo with the variable absent
|
||||
and requires **no** span — without that control, "spans appeared" could not be attributed to the
|
||||
variable rather than to something the demo does anyway.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
from portfolio_optimiser import tracing
|
||||
|
||||
_GOLDEN_STDOUT = Path(__file__).resolve().parent / "golden" / "demo-transcript.stdout"
|
||||
|
||||
|
||||
class _ConfigureRecorder:
|
||||
"""Stand-in for ``agent_framework.observability.configure_otel_providers``.
|
||||
|
||||
Records every call verbatim. The point of recording rather than counting is that the two
|
||||
keyword arguments carry the whole safety property: which exporters were handed over, and
|
||||
whether the console (stdout) exporters were left to the environment to decide.
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.calls: list[dict[str, Any]] = []
|
||||
|
||||
def __call__(self, **kwargs: Any) -> None:
|
||||
self.calls.append(kwargs)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# OFF by default — and "off" means MAF is never called, not called-with-nothing
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_absent_variable_configures_nothing_at_all() -> None:
|
||||
"""T1: no ``PORTFOLIO_OTEL`` → mode ``None`` and ``configure_otel_providers`` is NEVER called.
|
||||
|
||||
The distinction matters: a call with an empty exporter list would still install global
|
||||
providers and re-read every ``OTEL_EXPORTER_OTLP_*`` variable in the ambient environment. Not
|
||||
calling is the only shape under which "tracing is off" also means "nothing can leave".
|
||||
"""
|
||||
spy = _ConfigureRecorder()
|
||||
setup = tracing.configure_tracing(env={}, configure=spy)
|
||||
|
||||
assert setup.mode is None
|
||||
assert setup.endpoints == ()
|
||||
assert spy.calls == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize("raw", ["", " ", "\t\n"])
|
||||
def test_variable_is_read_on_truthiness_not_presence(raw: str) -> None:
|
||||
"""T2: an exported-but-empty ``PORTFOLIO_OTEL`` is a shell accident, not a request.
|
||||
|
||||
Same rule as ``FOUNDRY_HOSTING_ENVIRONMENT`` (Fase 4b) and ``PORT`` (Fase 4d). Reading this on
|
||||
presence would turn ``export PORTFOLIO_OTEL=`` into a fail-fast on an unknown mode.
|
||||
"""
|
||||
spy = _ConfigureRecorder()
|
||||
setup = tracing.configure_tracing(env={tracing.TRACING_ENV: raw}, configure=spy)
|
||||
|
||||
assert setup.mode is None
|
||||
assert spy.calls == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# console mode — spans to stderr, stdout untouched
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_console_mode_exports_to_the_given_stream_and_never_to_stdout() -> None:
|
||||
"""T3: console mode hands MAF exactly one exporter, writing to the stream we chose.
|
||||
|
||||
``ConsoleSpanExporter``'s default sink is ``sys.stdout``; S6 measured that letting it take that
|
||||
default destroys the pinned transcript. So the exporter is constructed explicitly against the
|
||||
stream, and ``enable_console_exporters`` is pinned False so that the environment cannot add a
|
||||
second, stdout-bound one behind it.
|
||||
"""
|
||||
from opentelemetry.sdk.trace.export import ConsoleSpanExporter
|
||||
|
||||
stream = io.StringIO()
|
||||
spy = _ConfigureRecorder()
|
||||
setup = tracing.configure_tracing(
|
||||
env={tracing.TRACING_ENV: "console"}, stream=stream, configure=spy
|
||||
)
|
||||
|
||||
assert setup.mode == tracing.MODE_CONSOLE
|
||||
assert len(spy.calls) == 1
|
||||
call = spy.calls[0]
|
||||
assert call["enable_console_exporters"] is False
|
||||
exporters = call["exporters"]
|
||||
assert len(exporters) == 1
|
||||
exporter = exporters[0]
|
||||
assert isinstance(exporter, ConsoleSpanExporter)
|
||||
assert exporter.out is stream
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"endpoint_var",
|
||||
[
|
||||
"OTEL_EXPORTER_OTLP_ENDPOINT",
|
||||
"OTEL_EXPORTER_OTLP_TRACES_ENDPOINT",
|
||||
"OTEL_EXPORTER_OTLP_METRICS_ENDPOINT",
|
||||
"OTEL_EXPORTER_OTLP_LOGS_ENDPOINT",
|
||||
],
|
||||
)
|
||||
def test_console_mode_refuses_when_the_environment_declares_a_network_exporter(
|
||||
endpoint_var: str,
|
||||
) -> None:
|
||||
"""T4: ``console`` + any OTLP endpoint variable → ``TracingConfigError``, and MAF is not called.
|
||||
|
||||
This is the no-silent-egress test. ``_configure`` adds env-derived exporters unconditionally and
|
||||
BEFORE ours, so the word "console" would have been a lie about where the spans went. Refusal
|
||||
names the offending variable, because the operator has to be able to find it; it does NOT unset
|
||||
it (validation, never repair).
|
||||
"""
|
||||
spy = _ConfigureRecorder()
|
||||
env = {tracing.TRACING_ENV: "console", endpoint_var: "http://collector.example:4317"}
|
||||
|
||||
with pytest.raises(tracing.TracingConfigError) as excinfo:
|
||||
tracing.configure_tracing(env=env, configure=spy)
|
||||
|
||||
assert endpoint_var in str(excinfo.value)
|
||||
assert spy.calls == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# otlp mode — egress, and only when it was asked for by name
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_otlp_mode_without_a_declared_endpoint_is_refused() -> None:
|
||||
"""T5: ``otlp`` with no endpoint variable → refusal, not a provider that exports nowhere.
|
||||
|
||||
``configure_otel_providers()`` with nothing to export to succeeds silently and installs
|
||||
providers whose spans go into the void. An operator who typed ``PORTFOLIO_OTEL=otlp`` asked to
|
||||
see the run somewhere; answering with a working-looking no-op is the failure mode this repo
|
||||
calls "a gate that can only be green".
|
||||
"""
|
||||
spy = _ConfigureRecorder()
|
||||
|
||||
with pytest.raises(tracing.TracingConfigError) as excinfo:
|
||||
tracing.configure_tracing(env={tracing.TRACING_ENV: "otlp"}, configure=spy)
|
||||
|
||||
assert "OTEL_EXPORTER_OTLP_ENDPOINT" in str(excinfo.value)
|
||||
assert spy.calls == []
|
||||
|
||||
|
||||
def test_otlp_mode_leaves_the_exporter_to_maf_and_still_pins_the_console_flag() -> None:
|
||||
"""T6: ``otlp`` + a declared endpoint → MAF builds the network exporter from the environment.
|
||||
|
||||
We pass no exporters of our own: MAF's step (1) already reads the standard variables, and a
|
||||
second, hand-rolled OTLP exporter would be the duplicate free to drift from the spec. What we
|
||||
still pin is ``enable_console_exporters=False`` — otherwise an operator running OTLP with
|
||||
``ENABLE_CONSOLE_EXPORTERS`` exported also gets a stdout dump.
|
||||
"""
|
||||
spy = _ConfigureRecorder()
|
||||
env = {
|
||||
tracing.TRACING_ENV: "otlp",
|
||||
"OTEL_EXPORTER_OTLP_ENDPOINT": "http://collector.example:4317",
|
||||
}
|
||||
|
||||
setup = tracing.configure_tracing(env=env, configure=spy)
|
||||
|
||||
assert setup.mode == tracing.MODE_OTLP
|
||||
assert setup.endpoints == (("OTEL_EXPORTER_OTLP_ENDPOINT", "http://collector.example:4317"),)
|
||||
assert len(spy.calls) == 1
|
||||
assert spy.calls[0]["enable_console_exporters"] is False
|
||||
assert spy.calls[0]["exporters"] is None
|
||||
|
||||
|
||||
def test_an_unknown_mode_is_refused_by_name() -> None:
|
||||
"""T7: a typo is a refusal that lists the closed set, never a silent fallback to off.
|
||||
|
||||
Falling back to off would mean an operator who asked for tracing and mistyped gets a run with
|
||||
no trace and no complaint — the black box the seam exists to remove, arrived at by accident.
|
||||
"""
|
||||
spy = _ConfigureRecorder()
|
||||
|
||||
with pytest.raises(tracing.TracingConfigError) as excinfo:
|
||||
tracing.configure_tracing(env={tracing.TRACING_ENV: "jaeger"}, configure=spy)
|
||||
|
||||
message = str(excinfo.value)
|
||||
assert "jaeger" in message
|
||||
assert tracing.MODE_CONSOLE in message and tracing.MODE_OTLP in message
|
||||
assert spy.calls == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# The announcement — one renderer, and omission rather than an empty row
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_notice_is_omitted_when_tracing_is_off() -> None:
|
||||
"""T8a: no tracing → no line at all (``mandate.announce``'s rule).
|
||||
|
||||
The same rule ``cost_baseline_notice`` and ``skipped_links_notice`` follow, and it is
|
||||
load-bearing beyond style here: the pinned demo stderr is four lines, and a "tracing: off" row
|
||||
would have made it five.
|
||||
"""
|
||||
assert tracing.tracing_notice(tracing.TracingSetup(mode=None, endpoints=())) is None
|
||||
|
||||
|
||||
def test_notice_names_the_sink_for_console_and_the_endpoint_for_otlp() -> None:
|
||||
"""T8b: the announcement names where spans go, BEFORE the first one is emitted.
|
||||
|
||||
Same discipline as the MCP announcement (``mcp_tools``): every destination is named up front,
|
||||
so there is no undeclared egress. The OTLP line names the *variable* as well as the value,
|
||||
because that is what the operator edits.
|
||||
"""
|
||||
console = tracing.tracing_notice(tracing.TracingSetup(mode=tracing.MODE_CONSOLE, endpoints=()))
|
||||
assert console is not None and "stderr" in console
|
||||
|
||||
otlp = tracing.tracing_notice(
|
||||
tracing.TracingSetup(
|
||||
mode=tracing.MODE_OTLP,
|
||||
endpoints=(("OTEL_EXPORTER_OTLP_ENDPOINT", "http://collector.example:4317"),),
|
||||
)
|
||||
)
|
||||
assert otlp is not None
|
||||
assert "http://collector.example:4317" in otlp
|
||||
assert "OTEL_EXPORTER_OTLP_ENDPOINT" in otlp
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# The real arm: the actual demo, in a subprocess, with nothing patched
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
_SPAN_MARKER = re.compile(r'"name": "workflow\.run"')
|
||||
|
||||
|
||||
def _run_demo(extra_env: dict[str, str]) -> subprocess.CompletedProcess[str]:
|
||||
"""Run the real demo the same way the golden-transcript module does.
|
||||
|
||||
The ``-m`` form (no PATH assumption) and a pinned ``PYTHONIOENCODING`` — without the latter the
|
||||
stdout comparison would be measuring the operator's locale rather than the program.
|
||||
"""
|
||||
proc = subprocess.run(
|
||||
[sys.executable, "-m", "portfolio_optimiser.simulation"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
encoding="utf-8",
|
||||
env={**os.environ, "PYTHONIOENCODING": "utf-8", **extra_env},
|
||||
check=False,
|
||||
)
|
||||
assert proc.returncode == 0, proc.stderr
|
||||
return proc
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def traced_demo() -> subprocess.CompletedProcess[str]:
|
||||
return _run_demo({tracing.TRACING_ENV: "console"})
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def untraced_demo() -> subprocess.CompletedProcess[str]:
|
||||
return _run_demo({tracing.TRACING_ENV: ""})
|
||||
|
||||
|
||||
def test_traced_demo_emits_workflow_run_spans_on_stderr(
|
||||
traced_demo: subprocess.CompletedProcess[str],
|
||||
) -> None:
|
||||
"""T9a: the plan's U14 criterion, half one — ``PORTFOLIO_OTEL=console`` produces real spans.
|
||||
|
||||
``workflow.run`` is asserted specifically rather than "some span": it is the span that says a
|
||||
workflow was executed, which is the thing a reader of the trace is looking for. S6 measured two
|
||||
of them for this demo; the assertion is ``>= 1`` because the count is a property of the demo's
|
||||
scripted script, not of the seam.
|
||||
"""
|
||||
assert len(_SPAN_MARKER.findall(traced_demo.stderr)) >= 1
|
||||
|
||||
|
||||
def test_traced_demo_leaves_stdout_byte_identical_to_the_golden_transcript(
|
||||
traced_demo: subprocess.CompletedProcess[str],
|
||||
) -> None:
|
||||
"""T9b: the plan's U14 criterion, half two — turning tracing ON does not move one byte of stdout.
|
||||
|
||||
This is what makes the seam safe to ship: the operator can trace a live run on stage without
|
||||
the transcript they rehearsed against changing under them. It is also the test that fails if
|
||||
the exporter is ever allowed to take its stdout default.
|
||||
"""
|
||||
assert traced_demo.stdout == _GOLDEN_STDOUT.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def test_untraced_demo_emits_no_spans_at_all(
|
||||
untraced_demo: subprocess.CompletedProcess[str],
|
||||
) -> None:
|
||||
"""T9c (control): without the variable there is NO span — so T9a measured the variable.
|
||||
|
||||
Without this, "spans on stderr" would be consistent with a demo that emits them regardless, and
|
||||
the seam would be unproven while looking proven.
|
||||
"""
|
||||
assert _SPAN_MARKER.findall(untraced_demo.stderr) == []
|
||||
|
||||
|
||||
def test_traced_demo_announces_where_the_spans_go(
|
||||
traced_demo: subprocess.CompletedProcess[str],
|
||||
) -> None:
|
||||
"""T9d: the announcement reaches the operator, not just the renderer's return value.
|
||||
|
||||
The wiring half of T8: a renderer nobody prints is the silent-success shape this repo keeps
|
||||
finding. Only a subprocess run can catch a detached ``print`` in ``main()`` (the P4 precedent).
|
||||
"""
|
||||
assert "PORTFOLIO_OTEL" in traced_demo.stderr
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# The other two process entries — the CLI and the hosted service
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
#
|
||||
# The demo is a scripted proof, not the product. Wiring the seam only there would leave the two
|
||||
# entries an organisation actually runs — `portfolio-optimiser` on a terminal and `python main.py`
|
||||
# in a container — untraceable, which is the state U14 exists to end. Both are exercised as
|
||||
# SUBPROCESSES for the P4 reason: a detached call inside a `main()` is invisible to every in-process
|
||||
# test, because no in-process test calls `main()`.
|
||||
|
||||
|
||||
def _run_cli(argv: list[str], extra_env: dict[str, str]) -> subprocess.CompletedProcess[str]:
|
||||
"""Drive the CLI's refusal path — the cheapest argv that reaches past ``parse_args``.
|
||||
|
||||
``--json`` without ``--report`` is an existing, documented rc-1 refusal. It is used here as a
|
||||
carrier, not as the thing under test: it proves that the tracing seam is resolved BEFORE any
|
||||
branch of the CLI can return, which is what "call once at startup" requires. No model is
|
||||
contacted and no workflow is built, so the arm stays free.
|
||||
"""
|
||||
return subprocess.run(
|
||||
[sys.executable, "-m", "portfolio_optimiser.run", *argv],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
encoding="utf-8",
|
||||
env={**os.environ, "PYTHONIOENCODING": "utf-8", **extra_env},
|
||||
check=False,
|
||||
)
|
||||
|
||||
|
||||
def test_cli_announces_tracing_before_it_refuses_anything() -> None:
|
||||
"""T10: ``run.main`` resolves and announces the seam, ahead of every other branch.
|
||||
|
||||
RED if the call is detached from ``main()`` or moved below a ``return``.
|
||||
"""
|
||||
proc = _run_cli(["--json"], {tracing.TRACING_ENV: "console"})
|
||||
|
||||
assert proc.returncode == 1, proc.stderr
|
||||
assert "PORTFOLIO_OTEL=console" in proc.stderr
|
||||
assert "run refused: --json requires --report" in proc.stderr
|
||||
|
||||
|
||||
def test_cli_says_nothing_about_tracing_when_it_is_off() -> None:
|
||||
"""T10 control: the same argv without the variable prints the refusal and NOTHING else.
|
||||
|
||||
Without this, T10 could not distinguish an announcement caused by the variable from a banner
|
||||
the CLI prints unconditionally — and an unconditional banner would have changed every existing
|
||||
stderr expectation in the suite.
|
||||
"""
|
||||
proc = _run_cli(["--json"], {tracing.TRACING_ENV: ""})
|
||||
|
||||
assert proc.returncode == 1
|
||||
assert "PORTFOLIO_OTEL" not in proc.stderr
|
||||
|
||||
|
||||
def test_cli_refuses_an_unusable_tracing_request_as_a_run_refusal() -> None:
|
||||
"""T11: a malformed ``PORTFOLIO_OTEL`` exits through the CLI's own rc-1 refusal surface.
|
||||
|
||||
Not a traceback: this repo's CLI contract is that a refusal is a printed line and rc 1, and a
|
||||
configuration the operator can fix belongs there. The measured content matters too — the line
|
||||
must name the variable, since the operator is looking for something they exported, not for a
|
||||
flag they typed.
|
||||
"""
|
||||
proc = _run_cli(["--json"], {tracing.TRACING_ENV: "jaeger"})
|
||||
|
||||
assert proc.returncode == 1
|
||||
assert "run refused" in proc.stderr
|
||||
assert "PORTFOLIO_OTEL" in proc.stderr
|
||||
assert proc.stderr.count("Traceback") == 0
|
||||
|
||||
|
||||
def test_hosted_entrypoint_announces_tracing_on_stderr() -> None:
|
||||
"""T12: ``python main.py`` — the ONE start command DEPLOY.md prints — carries the seam too.
|
||||
|
||||
The hosted flate is where "an enterprise must be able to see what the run did" is actually
|
||||
cashed: a container's stderr is its log. Served, then stopped with SIGTERM exactly as the
|
||||
existing entrypoint test does, so this measures the real process rather than an import.
|
||||
"""
|
||||
import socket
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
with socket.socket() as probe:
|
||||
probe.bind(("127.0.0.1", 0))
|
||||
port = probe.getsockname()[1]
|
||||
repo_root = Path(__file__).resolve().parents[1]
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, str(repo_root / "main.py")],
|
||||
env={**os.environ, "PORT": str(port), tracing.TRACING_ENV: "console"},
|
||||
cwd=repo_root,
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.PIPE,
|
||||
text=True,
|
||||
encoding="utf-8",
|
||||
)
|
||||
try:
|
||||
deadline = time.monotonic() + 60
|
||||
up = False
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
with urllib.request.urlopen(
|
||||
f"http://127.0.0.1:{port}/readiness", timeout=5
|
||||
) as resp:
|
||||
if resp.status == 200:
|
||||
up = True
|
||||
break
|
||||
except (urllib.error.URLError, OSError):
|
||||
time.sleep(0.2)
|
||||
assert up, "main.py never served /readiness"
|
||||
import signal as _signal
|
||||
|
||||
proc.send_signal(_signal.SIGTERM)
|
||||
stderr = proc.communicate(timeout=15)[1]
|
||||
assert proc.returncode == 0
|
||||
finally:
|
||||
if proc.poll() is None:
|
||||
proc.kill()
|
||||
proc.wait()
|
||||
|
||||
assert "PORTFOLIO_OTEL=console" in stderr
|
||||
|
||||
|
||||
def test_console_mode_survives_an_operator_who_exported_enable_console_exporters() -> None:
|
||||
"""T13: ``ENABLE_CONSOLE_EXPORTERS=true`` in the environment does NOT reach stdout.
|
||||
|
||||
The behavioural half of ``enable_console_exporters=False``. Asserting only the keyword argument
|
||||
(T3, T6) proves what we passed, never what it prevents — and what it prevents is MAF's step (3)
|
||||
adding a second ``ConsoleSpanExporter()`` whose default sink is stdout, which is the exact shape
|
||||
S6 measured as destroying the pinned transcript. Left as ``None``, the flag falls back to this
|
||||
variable, so this run is the one that would break.
|
||||
"""
|
||||
proc = _run_demo({tracing.TRACING_ENV: "console", "ENABLE_CONSOLE_EXPORTERS": "true"})
|
||||
|
||||
assert proc.stdout == _GOLDEN_STDOUT.read_text(encoding="utf-8")
|
||||
assert len(_SPAN_MARKER.findall(proc.stderr)) >= 1
|
||||
420
tests/test_ungiven_verdict_loadbearing.py
Normal file
420
tests/test_ungiven_verdict_loadbearing.py
Normal file
|
|
@ -0,0 +1,420 @@
|
|||
"""F2 (misjonsreview ``docs/2026-08-25-fable-misjonsreview.md``, non-goal 3): an expert verdict
|
||||
must not be able to ARISE without an expert having given one, and a verdict nobody gave must not
|
||||
PROPAGATE into the next project's hypothesis prompt as if it were one.
|
||||
|
||||
Before this seam, ``run_project`` REQUIRED ``verdict_input`` and unconditionally ran
|
||||
``capture_verdict(features, verdict_input["decision"], ...)``. The CLI defaulted that to
|
||||
``{"decision": "approved", "rationale": "reviewed by expert"}``, so every flagless run minted an
|
||||
approval nobody spoke; the hosted surface listed the field as REQUIRED, so an external caller had
|
||||
to invent one to get a run at all; and in ``run_portfolio`` the minted verdict entered the shared
|
||||
store and reached the next project's ExpeL few-shot.
|
||||
|
||||
The tests here fasten the GOAL, not the mechanism: absence of a verdict must be REPRESENTABLE and
|
||||
must be what silence produces. Every negative assert is paired with a control that proves the
|
||||
event it denies can actually happen — a test that can only be green proves nothing (this repo's
|
||||
recurring vacuous-gate class).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import shutil
|
||||
from collections.abc import Callable
|
||||
from importlib.resources import files
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from agent_framework import BaseChatClient
|
||||
from pydantic import ValidationError
|
||||
from conftest import SyntheticUsageChatClient
|
||||
|
||||
from portfolio_optimiser import hosting, run
|
||||
from portfolio_optimiser.ledger import LedgerEntry, SavingsLedger
|
||||
from portfolio_optimiser.reference_domain import Project
|
||||
from portfolio_optimiser.run import run_portfolio, run_project
|
||||
|
||||
BUNDLE_DIR = Path(__file__).resolve().parents[1] / "shared" / "examples" / "bygg-energi-mikro"
|
||||
_PID = "BYGG-KONTOR-NORD"
|
||||
_MINI_BUNDLE = str(files("portfolio_optimiser").joinpath("data/bundles/bygg-energi-mikro-a"))
|
||||
|
||||
_VALID_PROPOSER_REPLY = (
|
||||
'{"measure":"LED-retrofit av kontorbelysning","affected_items":'
|
||||
'[{"code":"ENERGI-TOTAL-EL","quantity":300000,"unit_cost":1.0}],"claimed_saving_nok":30000}'
|
||||
)
|
||||
_CHECKER_APPROVE = "The retrofit is supported by the cited documents. VERDICT: APPROVE"
|
||||
_GIVEN = {"decision": "approved", "rationale": "expert reviewed (sim)"}
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _isolate_model_env(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.delenv("PORTFOLIO_MODEL_MAP", raising=False)
|
||||
monkeypatch.delenv("PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT", raising=False)
|
||||
|
||||
|
||||
def _role_factory(proposer_reply: str, checker_reply: str) -> Callable[[str], BaseChatClient]:
|
||||
def factory(role: str) -> BaseChatClient:
|
||||
return SyntheticUsageChatClient(
|
||||
default_reply=checker_reply if role == "checker" else proposer_reply
|
||||
)
|
||||
|
||||
return factory
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def bundle(tmp_path: Path) -> Path:
|
||||
"""A throwaway COPY — the shared fixture is commons-owned and is never mutated by a test."""
|
||||
dst = tmp_path / "bundle"
|
||||
shutil.copytree(BUNDLE_DIR, dst)
|
||||
return dst
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def replies_file(tmp_path: Path) -> Path:
|
||||
path = tmp_path / "replies.json"
|
||||
path.write_text(
|
||||
json.dumps({"proposer": _VALID_PROPOSER_REPLY, "checker": _CHECKER_APPROVE}),
|
||||
encoding="utf-8",
|
||||
)
|
||||
return path
|
||||
|
||||
|
||||
def _argv(bundle: Path, replies_file: Path) -> list[str]:
|
||||
return [
|
||||
_PID,
|
||||
"--docs-dir",
|
||||
str(bundle),
|
||||
"--bundle-dir",
|
||||
str(bundle),
|
||||
"--scripted-replies",
|
||||
str(replies_file),
|
||||
]
|
||||
|
||||
|
||||
# --- T1: a verdict cannot ARISE from silence (library seam) ---------------------------------
|
||||
|
||||
|
||||
async def test_run_without_verdict_input_captures_no_verdict(tmp_path) -> None:
|
||||
"""T1 GOAL: a run nobody reviewed produces NO verdict — the field is ``None`` and the store
|
||||
stays empty. RED before the seam: ``verdict_input`` was a REQUIRED keyword and step 8 minted
|
||||
an approval unconditionally."""
|
||||
result = await run_project(
|
||||
_PID,
|
||||
"local",
|
||||
docs_dir=str(BUNDLE_DIR),
|
||||
bundle_dir=str(BUNDLE_DIR),
|
||||
client_factory=_role_factory(_VALID_PROPOSER_REPLY, _CHECKER_APPROVE),
|
||||
max_rounds=2,
|
||||
)
|
||||
assert result.verdict is None, "a verdict nobody gave was minted anyway"
|
||||
assert result.store.verdicts == [], "an ungiven verdict entered the learning store"
|
||||
|
||||
|
||||
async def test_run_with_verdict_input_still_captures_it(tmp_path) -> None:
|
||||
"""T1 CONTROL: when an expert DOES speak, the verdict is captured and stored exactly as
|
||||
before. Without this arm T1 would pass against an implementation that never captures at all."""
|
||||
result = await run_project(
|
||||
_PID,
|
||||
"local",
|
||||
docs_dir=str(BUNDLE_DIR),
|
||||
bundle_dir=str(BUNDLE_DIR),
|
||||
verdict_input=_GIVEN,
|
||||
client_factory=_role_factory(_VALID_PROPOSER_REPLY, _CHECKER_APPROVE),
|
||||
max_rounds=2,
|
||||
)
|
||||
assert result.verdict is not None
|
||||
assert result.verdict.decision == "approved"
|
||||
assert result.verdict.rationale == "expert reviewed (sim)"
|
||||
assert [v.id for v in result.store.verdicts] == [result.verdict.id]
|
||||
|
||||
|
||||
async def test_verdict_key_is_available_even_with_no_verdict(tmp_path) -> None:
|
||||
"""T1b: the KEY an expert verdict on this candidate will arrive under is always available —
|
||||
it is derived from the candidate, not from a decision. This is what keeps the outbox artefact
|
||||
and the hosted response judgeable on a run nobody has reviewed yet (``verdicts.verdict_key``'s
|
||||
own documented purpose), so 'no verdict' costs no traceability."""
|
||||
result = await run_project(
|
||||
_PID,
|
||||
"local",
|
||||
docs_dir=str(BUNDLE_DIR),
|
||||
bundle_dir=str(BUNDLE_DIR),
|
||||
client_factory=_role_factory(_VALID_PROPOSER_REPLY, _CHECKER_APPROVE),
|
||||
max_rounds=2,
|
||||
)
|
||||
assert result.verdict is None
|
||||
assert result.verdict_key, "the run must still name the key a verdict on it would arrive under"
|
||||
|
||||
given = await run_project(
|
||||
_PID,
|
||||
"local",
|
||||
docs_dir=str(BUNDLE_DIR),
|
||||
bundle_dir=str(BUNDLE_DIR),
|
||||
verdict_input=_GIVEN,
|
||||
client_factory=_role_factory(_VALID_PROPOSER_REPLY, _CHECKER_APPROVE),
|
||||
max_rounds=2,
|
||||
)
|
||||
# Same candidate -> same key, and a captured verdict keys under exactly it.
|
||||
assert given.verdict is not None
|
||||
assert given.verdict.id == given.verdict_key == result.verdict_key
|
||||
|
||||
|
||||
# --- T2: the CLI's silence is silence, not an approval --------------------------------------
|
||||
|
||||
|
||||
def test_cli_without_decision_flags_reports_no_verdict(bundle, replies_file, capsys) -> None:
|
||||
"""T2 GOAL: a flagless CLI run says a verdict was NOT given. RED before the seam: argparse
|
||||
defaulted to ``approved``/``reviewed by expert`` and the line read ``verdict id=…,
|
||||
decision=approved`` for a run nobody reviewed."""
|
||||
rc = run.main(_argv(bundle, replies_file))
|
||||
out = capsys.readouterr().out
|
||||
assert rc == 0, out
|
||||
assert "verdict id=" not in out, out
|
||||
assert "no expert verdict" in out, out
|
||||
|
||||
|
||||
def test_cli_with_decision_flags_reports_the_verdict(bundle, replies_file, capsys) -> None:
|
||||
"""T2 CONTROL: an operator who DOES record a verdict gets the unchanged line. Proves T2's
|
||||
negative is caused by the absent flags, not by the reporting having been removed."""
|
||||
rc = run.main(
|
||||
[*_argv(bundle, replies_file), "--decision", "approved", "--rationale", "I reviewed it"]
|
||||
)
|
||||
out = capsys.readouterr().out
|
||||
assert rc == 0, out
|
||||
assert "verdict id=" in out, out
|
||||
assert "decision=approved" in out, out
|
||||
|
||||
|
||||
def test_cli_outbox_artefact_still_carries_a_verdict_key(
|
||||
bundle, replies_file, tmp_path, capsys
|
||||
) -> None:
|
||||
"""T2b: 'no verdict' must not cost the artefact its identity — the outcome file still carries
|
||||
the key a later expert verdict on this candidate will arrive under, which is how the honest
|
||||
inbox channel (Step 7) joins back to this run."""
|
||||
outbox = tmp_path / "outbox"
|
||||
rc = run.main([*_argv(bundle, replies_file), "--outbox-dir", str(outbox), "--run-id", "r1"])
|
||||
assert rc == 0, capsys.readouterr().out
|
||||
payload = json.loads((outbox / "r1-outcome.json").read_text(encoding="utf-8"))
|
||||
assert payload["verdict_id"], "the outbox lost the candidate's verdict key"
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("flags", "missing"),
|
||||
[
|
||||
(["--decision", "approved"], "--rationale"),
|
||||
(["--rationale", "I reviewed it"], "--decision"),
|
||||
],
|
||||
)
|
||||
def test_cli_half_a_verdict_is_refused_by_name(
|
||||
bundle, replies_file, capsys, flags: list[str], missing: str
|
||||
) -> None:
|
||||
"""T3: a verdict is a decision AND its reasoning. Half of one is refused BY NAME before any
|
||||
model call — validation, never repair (the alternative is filling the other half in on the
|
||||
expert's behalf, which is the very defect F2 closes). RED before the seam: the missing half
|
||||
silently took its argparse default."""
|
||||
rc = run.main([*_argv(bundle, replies_file), *flags])
|
||||
captured = capsys.readouterr()
|
||||
assert rc == 1, captured.out
|
||||
assert missing in captured.err, captured.err
|
||||
assert "verdict id=" not in captured.out, captured.out
|
||||
|
||||
|
||||
# --- T4/T5: an ungiven verdict cannot PROPAGATE to the next project --------------------------
|
||||
|
||||
_SENTINEL = "SENTINEL-F2-3d71ac realiseringskorreksjon fra prosjekt k"
|
||||
_ALIGNED_REPLY = (
|
||||
'{"measure":"LED-retrofit av kontorbelysning","affected_items":'
|
||||
'[{"code":"ENERGI-TOTAL-EL","quantity":180000,"unit_cost":1.0}],"claimed_saving_nok":18000}'
|
||||
)
|
||||
|
||||
|
||||
def _generation_prompts(sink: list[str]) -> list[str]:
|
||||
return [p for p in sink if "SavingsProposal" in p]
|
||||
|
||||
|
||||
def _make_docs(tmp_path: Path, name: str) -> str:
|
||||
d = tmp_path / name
|
||||
d.mkdir()
|
||||
(d / "cost.txt").write_text(
|
||||
"Asphalt Ab11 unit rate renegotiation reduced the paving cost on the school stretch.",
|
||||
encoding="utf-8",
|
||||
)
|
||||
return str(d)
|
||||
|
||||
|
||||
def _road_k(tmp_path: Path, *, verdict_input: dict[str, str] | None) -> Project:
|
||||
return Project(
|
||||
id="ROAD-K",
|
||||
name="Road k",
|
||||
description="road-backed project k",
|
||||
currency="NOK",
|
||||
cost_items=(),
|
||||
docs_dir=_make_docs(tmp_path, "k-docs"),
|
||||
verdict_input=verdict_input,
|
||||
bundle_dir=None,
|
||||
verdict_dir=None,
|
||||
)
|
||||
|
||||
|
||||
def _bundle_kplus1(tmp_path: Path) -> Project:
|
||||
return Project(
|
||||
id="BYGG-ENERGI-MIKRO-A",
|
||||
name="Bundle k+1",
|
||||
description="bundle-backed project k+1",
|
||||
currency="NOK",
|
||||
cost_items=(),
|
||||
docs_dir=_make_docs(tmp_path, "kplus1-docs"),
|
||||
verdict_input={"decision": "approved", "rationale": "k+1 reviewed (sim)"},
|
||||
bundle_dir=_MINI_BUNDLE,
|
||||
verdict_dir=None,
|
||||
)
|
||||
|
||||
|
||||
async def test_a_given_verdict_on_k_does_reach_kplus1(
|
||||
tmp_path, monkeypatch, make_recording_client_factory
|
||||
) -> None:
|
||||
"""T4 CONTROL (mirrors ``test_portfolio_learning_loadbearing``): when an expert DID review
|
||||
project *k*, that verdict reaches *k+1*'s hypothesis prompt. This is the event T5 denies —
|
||||
proving first that it can happen is what makes T5's negative assert mean anything."""
|
||||
k = _road_k(tmp_path, verdict_input={"decision": "approved", "rationale": _SENTINEL})
|
||||
kplus1 = _bundle_kplus1(tmp_path)
|
||||
monkeypatch.setattr("portfolio_optimiser.run.load_reference_projects", lambda: (k, kplus1))
|
||||
factory, recorded = make_recording_client_factory(_ALIGNED_REPLY)
|
||||
|
||||
result = await run_portfolio(profile="local", client_factory=factory)
|
||||
|
||||
assert result.runs[0].verdict is not None
|
||||
prompts = _generation_prompts(recorded)
|
||||
assert any(_SENTINEL in p for p in prompts)
|
||||
assert any(result.runs[0].verdict.id in p for p in prompts)
|
||||
|
||||
|
||||
async def test_an_ungiven_verdict_on_k_never_reaches_kplus1(
|
||||
tmp_path, monkeypatch, make_recording_client_factory
|
||||
) -> None:
|
||||
"""T5 GOAL: project *k* that nobody reviewed contributes NOTHING to *k+1*'s hypothesis
|
||||
prompt — no verdict is minted, none enters the shared store, and *k*'s candidate key never
|
||||
shows up as a prior judgement. RED before the seam: *k* minted an ``approved`` verdict from a
|
||||
``verdict_input`` no expert supplied, and the threaded store carried it forward."""
|
||||
k = _road_k(tmp_path, verdict_input=None)
|
||||
kplus1 = _bundle_kplus1(tmp_path)
|
||||
monkeypatch.setattr("portfolio_optimiser.run.load_reference_projects", lambda: (k, kplus1))
|
||||
factory, recorded = make_recording_client_factory(_ALIGNED_REPLY)
|
||||
|
||||
result = await run_portfolio(profile="local", client_factory=factory)
|
||||
|
||||
assert result.runs[0].verdict is None, "k minted a verdict nobody gave"
|
||||
k_key = result.runs[0].verdict_key
|
||||
prompts = _generation_prompts(recorded)
|
||||
assert prompts, "the generation calls must have happened"
|
||||
assert not any(k_key in p for p in prompts), (
|
||||
"k's ungiven verdict propagated into k+1's hypothesis prompt as a prior judgement"
|
||||
)
|
||||
assert [v.id for v in result.store.verdicts] == [result.runs[1].verdict.id], (
|
||||
"the shared store holds a verdict for a project nobody reviewed"
|
||||
)
|
||||
|
||||
|
||||
# --- T6: the hosted surface no longer FORCES a caller to invent a verdict ---------------------
|
||||
|
||||
|
||||
async def test_hosted_invocation_without_verdict_input_is_accepted(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""T6 GOAL: ``verdict_input`` is no longer a REQUIRED field, so an external caller who has no
|
||||
expert verdict can run at all. A RELAXATION — a caller that still sends the field is
|
||||
unaffected (the control below). RED before the seam: 400, naming ``verdict_input``."""
|
||||
calls: list[dict[str, Any]] = []
|
||||
|
||||
async def _runner(*args: Any, **kwargs: Any):
|
||||
calls.append(kwargs)
|
||||
raise AssertionError("stop after the whitelist") # pragma: no cover
|
||||
|
||||
monkeypatch.setattr(hosting, "run_project", _runner)
|
||||
payload = {"project_id": "P1", "docs_dir": "docs"}
|
||||
with pytest.raises(AssertionError, match="stop after the whitelist"):
|
||||
await hosting.invoke(payload)
|
||||
assert calls, "the whitelist refused a payload that omitted verdict_input"
|
||||
assert calls[0].get("verdict_input") is None
|
||||
|
||||
|
||||
async def test_hosted_invocation_with_verdict_input_still_forwards_it(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""T6 CONTROL: the outward-facing contract is only WIDENED — a caller that sends the field
|
||||
still has it forwarded verbatim, so no call that exists out there breaks."""
|
||||
calls: list[dict[str, Any]] = []
|
||||
|
||||
async def _runner(*args: Any, **kwargs: Any):
|
||||
calls.append(kwargs)
|
||||
raise AssertionError("stop after the whitelist") # pragma: no cover
|
||||
|
||||
monkeypatch.setattr(hosting, "run_project", _runner)
|
||||
payload = {"project_id": "P1", "docs_dir": "docs", "verdict_input": _GIVEN}
|
||||
with pytest.raises(AssertionError, match="stop after the whitelist"):
|
||||
await hosting.invoke(payload)
|
||||
assert calls[0]["verdict_input"] == _GIVEN
|
||||
|
||||
|
||||
# --- T7: the library refuses half a verdict too, and the partitions no longer drop a real one ---
|
||||
|
||||
|
||||
async def test_library_half_a_verdict_is_refused_by_field_name(tmp_path) -> None:
|
||||
"""T7: the CLI is not the only door. A caller that hands ``run_project`` half a verdict is
|
||||
refused at step 1 by ``FeedbackContract`` — the ONE place the shape is validated — rather than
|
||||
having the missing half filled in for the expert. RED if that contract is made tolerant."""
|
||||
with pytest.raises(ValidationError):
|
||||
await run_project(
|
||||
_PID,
|
||||
"local",
|
||||
docs_dir=str(BUNDLE_DIR),
|
||||
bundle_dir=str(BUNDLE_DIR),
|
||||
verdict_input={"decision": "approved"},
|
||||
client_factory=_role_factory(_VALID_PROPOSER_REPLY, _CHECKER_APPROVE),
|
||||
max_rounds=2,
|
||||
)
|
||||
|
||||
|
||||
def test_portfolio_mode_refuses_a_recorded_verdict_by_name(bundle, replies_file, capsys) -> None:
|
||||
"""T8a: before F2 the argparse defaults made ``--decision`` indistinguishable from its default,
|
||||
so the partition could not refuse it and an operator's real expert verdict was silently
|
||||
dropped in portfolio mode (a pass takes each project's verdict from its own row). It is
|
||||
distinguishable now, so it is refused BY NAME — 'refused, never ignored' is the partition's own
|
||||
rule. RED when the rows are dropped from ``single_only``."""
|
||||
rc = run.main(["--portfolio", "--decision", "approved", "--rationale", "I reviewed it"])
|
||||
captured = capsys.readouterr()
|
||||
assert rc == 1, captured.out
|
||||
assert "--decision" in captured.err, captured.err
|
||||
assert "--rationale" in captured.err, captured.err
|
||||
|
||||
|
||||
def test_report_mode_refuses_a_recorded_verdict(tmp_path, capsys) -> None:
|
||||
"""T8b: same rule on the OTHER partition. Report mode's refusal is generic by construction (it
|
||||
names the allowlist, not the offender), so the discriminator has to be the OUTCOME — and the
|
||||
argv must be one that report mode would otherwise ACCEPT. Measured while mutating: with a bare
|
||||
``--report`` this test was VACUOUS (green with the rows dropped), because ``--report`` without
|
||||
``--ledger`` refuses with rc 1 for a completely different reason. A valid ``--ledger`` makes
|
||||
rc 0 the mutant's outcome, so rc 1 here means the partition refused."""
|
||||
led = SavingsLedger()
|
||||
led.add_realized(
|
||||
LedgerEntry(
|
||||
project_id="P1",
|
||||
dimension="energi",
|
||||
candidate_identity="c-a",
|
||||
amount_ore=100000,
|
||||
verdict_id="v1",
|
||||
provenance="p1",
|
||||
)
|
||||
)
|
||||
ledger_path = tmp_path / "ledger.json"
|
||||
led.save(str(ledger_path))
|
||||
|
||||
# CONTROL: the same argv WITHOUT the verdict flags is accepted and prints a report.
|
||||
assert run.main(["--report", "--ledger", str(ledger_path)]) == 0
|
||||
capsys.readouterr()
|
||||
|
||||
rc = run.main(
|
||||
["--report", "--ledger", str(ledger_path), "--decision", "approved", "--rationale", "x"]
|
||||
)
|
||||
captured = capsys.readouterr()
|
||||
assert rc == 1, captured.out
|
||||
assert "--report" in captured.err, captured.err
|
||||
41
uv.lock
generated
41
uv.lock
generated
|
|
@ -59,14 +59,14 @@ wheels = [
|
|||
|
||||
[[package]]
|
||||
name = "agent-framework-orchestrations"
|
||||
version = "1.0.0"
|
||||
version = "1.0.1"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "agent-framework-core" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/7a/98/1f943e3a383bf37af24f1cc3ab3812fbe2041b65bc2e2f82626352907190/agent_framework_orchestrations-1.0.0.tar.gz", hash = "sha256:7f4d26de1ac0b2add3742964836927df44138e79d16bbf084d6e866f1d4b5aff", size = 60805, upload-time = "2026-06-18T09:43:02.667Z" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/ca/e7/e6a123b74995bba5eaa9e4491e2db3cf94ebd8508df0abefffdec3f86468/agent_framework_orchestrations-1.0.1.tar.gz", hash = "sha256:8441488edb1512d50e0ca70e0bee3ae87a023636cbe860b57b80103b4bac50ec", size = 61003, upload-time = "2026-07-21T19:50:42.112Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/48/cc/64448d22e95db33bb7701d3a2bb8888d39f2fa8f4e410bd24232417c9ea3/agent_framework_orchestrations-1.0.0-py3-none-any.whl", hash = "sha256:1f0885c3de69c919bb631f7e4e24727d94733b135a1529ab88ffa47bf18a2878", size = 66828, upload-time = "2026-06-18T09:42:59.363Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/47/5d/a662a69014881e5ee593b39f64b97c0d34fc036c333eab73e9bc10019308/agent_framework_orchestrations-1.0.1-py3-none-any.whl", hash = "sha256:eb10454d462728ab83db14ba42d2c870f58d536966b9b60b525e5e504aa7e8c9", size = 67069, upload-time = "2026-07-21T19:50:13.408Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
|
@ -1448,14 +1448,41 @@ wheels = [
|
|||
|
||||
[[package]]
|
||||
name = "opentelemetry-api"
|
||||
version = "1.42.1"
|
||||
version = "1.44.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "typing-extensions" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/b4/1c/125e1c936c0873796771b7f04f6c93b9f1bf5d424cea90fda94a99f61da8/opentelemetry_api-1.42.1.tar.gz", hash = "sha256:56c63bea9f77b62856be8c47600474acad853b2924b99b1687c4cb6297166716", size = 72296, upload-time = "2026-05-21T16:32:49.335Z" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/ee/8b/aa9e2d8b8dfa7c946f7dec5d1f8f6ba8eca062f43509a06bdb5ce93d26c0/opentelemetry_api-1.44.0.tar.gz", hash = "sha256:67647e5e9566edcf421166fdf022b3537f818635daa852b289e34604dc6fb33a", size = 72406, upload-time = "2026-07-16T15:25:32.678Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/a3/ca/9520cc1f3dfbbd03ac5903bbf55833e257bc64b1cf30fa8b0d6df374d821/opentelemetry_api-1.42.1-py3-none-any.whl", hash = "sha256:51a69edacadbc03a8950ace1c4c21099cacc538820ac2c9e36277e78cebba714", size = 61311, upload-time = "2026-05-21T16:32:28.822Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/ca/6f/a04e900f465ff3221ccc395522503e2d10e79fa21f2723c8e177aae1e0d1/opentelemetry_api-1.44.0-py3-none-any.whl", hash = "sha256:94b98c893a91b88657eaac1e3ba89618cdb85be6918196705354f34728b2cdef", size = 60018, upload-time = "2026-07-16T15:25:11.657Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "opentelemetry-sdk"
|
||||
version = "1.44.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "opentelemetry-api" },
|
||||
{ name = "opentelemetry-semantic-conventions" },
|
||||
{ name = "typing-extensions" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/5d/77/a6592cbc7c8d9bcc9d6757a9df45e04a7c585e3e6e7a13456da522b21109/opentelemetry_sdk-1.44.0.tar.gz", hash = "sha256:cebe7f65dc12f26ead75c6064de12fd2a9052e5060c0272d402cfa203aae123b", size = 208624, upload-time = "2026-07-16T15:25:46.078Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/e7/23/ff077e61886ee020a17ce9c8b6fa11c601c8d8345b09ea24f605445df62a/opentelemetry_sdk-1.44.0-py3-none-any.whl", hash = "sha256:df081c4c6bcfdb1211e3e86140376792643128a25f8d72d1d27675936e7e96ad", size = 137221, upload-time = "2026-07-16T15:25:29.534Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "opentelemetry-semantic-conventions"
|
||||
version = "0.65b0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "opentelemetry-api" },
|
||||
{ name = "typing-extensions" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/8f/73/0cbdebcb4cf545fdd328da14f5137e37d0770c3f26185e478b0d15d94f50/opentelemetry_semantic_conventions-0.65b0.tar.gz", hash = "sha256:f9b2b81e9d5b64f11bc952075e7e9c7fb0aab075c7fd1c46d597f1b919852d60", size = 148774, upload-time = "2026-07-16T15:25:46.902Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/a6/0e/49df70d9b81fb5cbae4bbf2a49d865b09bcbcbc4eb53f5851b1027738d78/opentelemetry_semantic_conventions-0.65b0-py3-none-any.whl", hash = "sha256:1cacde7b0ad306f84c5ef08c3dbe1bbaf20165bba6f8bff43b670e555a086bcb", size = 204645, upload-time = "2026-07-16T15:25:30.688Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
|
@ -1500,6 +1527,7 @@ dependencies = [
|
|||
{ name = "llm-ingestion-okf" },
|
||||
{ name = "mcp" },
|
||||
{ name = "numpy" },
|
||||
{ name = "opentelemetry-sdk" },
|
||||
{ name = "pulp" },
|
||||
{ name = "pydantic" },
|
||||
]
|
||||
|
|
@ -1524,6 +1552,7 @@ requires-dist = [
|
|||
{ name = "llm-ingestion-okf", git = "https://git.fromaitochitta.com/open/llm-ingestion-okf.git?rev=v0.3.2" },
|
||||
{ name = "mcp", specifier = ">=1.28.0" },
|
||||
{ name = "numpy", specifier = ">=2.0,<2.3" },
|
||||
{ name = "opentelemetry-sdk", specifier = ">=1.42,<2" },
|
||||
{ name = "pulp", specifier = ">=2.8" },
|
||||
{ name = "pydantic", specifier = ">=2.11,<3" },
|
||||
]
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue