open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 3 Activity

test(spikes): S0-S6 maalt for Magentic-utforskningssloeyfa (ORDRE 20260823T162224Z) [skip-docs]

Browse source 
Maaling foer bygging. Alle aatte antakelsene i planens § F flyttet fra «umaalt» til
et maalt utfall; ingenting bygget, ingenting i src/.

S0 (versjon): 1.0.1 loeser paa core 1.9.0. Diffen mot 1.0.0 er upstream #4371 -
`StandardMagenticManager.__init__` mistet sin persistente `AgentSession`, og hvert
manager-kall mynter naa en engangs-sesjon. **E2 OG E4 er dermed BORTE (4/4 -> 0/5).**
E1 (single-use) og E7 (orphan `_agent_thread`, :1369) staar. Ordrens «felles hvis»
(E7 staar -> revert) hviler paa at 1.0.1 ikke kjoeper noe; den kjoepte noe stoerre
enn det som ble haapet, saa laasen staar paa 1.0.1 I PAAVENTE AV OPERATOEREN.

S1 (B7): E1-E4 + E7 i repoets form. Versjons-sensitiviteten testes mot en STRUKTURELL
sonde (holder manageren en persistent sesjon?), aldri en versjonsstreng - den sier
AARSAKEN og overlever en versjon planen ikke har sett.
S2 (budsjett): A1+A2 GROENNE. `BudgetMiddleware` fyrer paa manager-stien
(`meter.tokens == 8`), og `BudgetExceeded` forlater `workflow.run` som repoets EGEN
type med `kind`/`limit`/`observed` intakt - ikke pakket i en ExceptionGroup.
S3 (plan review): rundturen virker; en revise koster 2 manager-kall, 0 ledger-kall,
0 runder, og SPOER PAA NYTT -> `max_plan_revisions` maa inn i kontrakten.
S3b: doer 3 staar. To rundturer per menneskesvar; `from_strings` gjenopptar IKKE
manageren, kun `approve` gjoer det. Pris: `AgentApprovalExecutor` er ikke re-eksportert.
S4 (resume i NY prosess): GROENN. Pris: `FileCheckpointStorage` nekter aa deserialisere
plan-review-typene uten `allowed_checkpoint_types` - uten det feiler resume som et
FRAVAER (tom listing), ikke som en feil.
S5: median `validate_proposal` 13,6 ms - fritt kallbart i loekka.
S6 (scratch-venv, ingenting lagt til pyproject): 2 `workflow.run`-spans, men
`enable_console_exporters` skriver til STDOUT og ville oedelagt golden-transkriptet;
`ConsoleSpanExporter(out=sys.stderr)` gir spanene paa stderr OG byte-identisk stdout.

Klienten er repoets `ScriptedChatClient` og budsjett-typene er PRODUKSJONENS - en bar
`BaseChatClient` no-op-er middleware, og `spikes/_harness.py`s egen kopi er nettopp
grunnen til at koe-(y) fantes.

Load-bearing MAALT mot HELE suiten, groenn kontroll 920/5: konstant persistent-sesjon
(2 roede) · aldri fest middleware paa manageren (3 roede, detach-armen groenn) · detach
markoer-registreringen (1 roed) · flipp `_route`-rekkefoelgen (4 roede) · resume uten
`checkpoint_id` (1 roed) · builder uten `with_checkpointing` (1 roed) · tom
`_ALLOWED_CHECKPOINT_TYPES` (1 roed). Og EN falsifisert: resume uten
`checkpoint_storage=` gir 0 roede - planens E-tabell navngir feil detach-punkt, og
det er skrevet inn i § F i stedet for aa staa som en gate som ikke kan bli roed.

Planens V1-sti ble portabel i 2eb4622 (pakke-gaten var roed paa HEAD siden 2e33905).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UTEa7uw2JMxgijx8k8XgxG
This commit is contained in:
Kjell Tore Guttormsen 2026-08-23 19:36:00 +02:00
commit 9f0843ab6d
5 changed files with 997 additions and 13 deletions
Download patch file Download diff file Expand all files Collapse all files

59
docs/plan/2026-08-23-magentic-utforskningssloeyfe.md
View file

@ -443,21 +443,60 @@ grønn, ellers synkron-først og U12 etter planen, sammen med core-bumpen.
## F. Nøkkelantakelser, risiko og åpne beslutninger
**Alle åtte radene er MÅLT i økt 54** (ordre `20260823T162224Z`, spikes S0–S6). Måleapparatet er
`spikes/e_magentic.py` + `tests/spikes/test_e_magentic.py` (16 tester), kjørt mot orchestrations
**1.0.1** på core 1.9.0; hele suiten 920 passed / 5 skipped, golden-transkriptet uendret
(`ea8c534773acdbe41ae68f2c55724d69aaf8be4f`).
| # | Antakelse | Test | Status |
|---|---|---|---|
| A1 | Agent-nivå `ChatMiddleware` fyrer på managerens kall | S2 | **umålt — RISIKO** (planens budsjettgaranti hviler her) |
| A2 | `BudgetExceeded` reist inni en Magentic-deltaker propagerer ut av `workflow.run` | S2 | målt for GroupChat (live-kjøringen 14.08), umålt for Magentic |
| A3 | `request_info`-rundtur på 1.0.0 | E6 | **MÅLT grønn** |
| A4 | Resume med pending plan review i NY prosess | S4 | umålt — kode-lest (`_runner_context.py:414-426`) |
| A5 | `AgentApprovalExecutor` som deltaker gir svar midt i kjøringen | S3b | umålt |
| A6 | Persona-manuset kan drive en Magentic-manager (ledger-JSON) offline | E1–E6 brukte en `FakeClient` med gyldig ledger-JSON | **MÅLT mulig**; portering til `ScriptedChatClient` er S1/S3 |
| A7 | Utforskning som opt-in holder demo-transkriptet byte-uendret | `shasum` i E-tabellen | bygges slik; verifiseres per økt |
| A8 | 1.0.1 er API-identisk med 1.0.0 | S0 | umålt |
| A1 | Agent-nivå `ChatMiddleware` fyrer på managerens kall | S2 | **MÅLT GRØNN.** `BudgetMiddleware` på manager-agenten krediterer meteret (`meter.tokens == 8` etter ett kall) og stopper kjøringen. Detach-kontroll: uten middleware fullfører SAMME 1-token-budsjett. Planens budsjettgaranti står. |
| A2 | `BudgetExceeded` reist inni en Magentic-deltaker propagerer ut av `workflow.run` | S2 | **MÅLT GRØNN, og som repoets EGEN type** — ikke pakket i en `ExceptionGroup`: `isinstance(exc, BudgetExceeded)`, `kind="tokens"`, `limit=1`, `observed=8`. Trippelen kø-(y) leser er intakt, så 429-kanalen kan brukes uendret. |
| A3 | `request_info`-rundtur | S3 | **MÅLT GRØNN på 1.0.1.** Review stopper kjøringen uten output (manageren har da kun kalt `facts`+`plan`); `revise` koster nøyaktig 2 manager-kall (`facts_update`, `plan_update`), **null ledger-kall og null runder**, og **spør på nytt**; `approve` kjører løkka til sluttsvar. Bekrefter at `max_plan_revisions` MÅ inn i kontrakten — ellers er en alltid-reviderende ekspert et ubundet forbruk. |
| A4 | Resume med pending plan review i NY prosess | S4 | **MÅLT GRØNN.** Foreldreprosessen stopper på review og etterlater checkpoints; en subprosess (`spikes/e_magentic_resume.py`) som aldri så kjøringen svarer fra checkpointen alene og driver workflowen til sluttsvar. **Pris, ikke forutsett:** `FileCheckpointStorage` NEKTER å deserialisere `MagenticPlanReviewRequest`/`…Response` uten at begge navngis i `allowed_checkpoint_types` — uten det er checkpoint-fila uleselig og listingen TOM, altså en resume som feiler som et FRAVÆR. Begge prosesser må deklarere dem. |
| A5 | `AgentApprovalExecutor` som deltaker gir svar midt i kjøringen | S3b | **MÅLT GRØNN — dør 3 i C.6 står.** Ekspertens ord når både liaisonen og en senere manager-prompt. **To rundturer per menneskesvar:** `from_strings([svar])` mater svaret tilbake INN i liaisonen og gjenopptar IKKE manageren (målt: null manager-kall mellom de to forespørslene); først `approve()` sender liaisonens output videre. **Kostnad:** `AgentApprovalExecutor` er IKKE re-eksportert fra `agent_framework.orchestrations` (svartypen ER det) — døra koster i dag en privat-API-import. |
| A6 | Persona-manuset kan drive en Magentic-manager (ledger-JSON) offline | S1/S3 | **MÅLT GRØNN med repoets `ScriptedChatClient`** (ikke lenger en ad-hoc `FakeClient`). Ett forbehold funnet: selectoren får den SAMMENSLÅTTE prompten, så ett av fem manager-kall bærer to markører — rutingen må teste senere-stadium-markøren først. Og en `next_speaker` som ikke matcher en deltaker gir **stille sluttsvar uten at noen ble spurt** (`_magentic.py:1128-1131`), målt da liaison-spiken først het `worker`. |
| A7 | Utforskning som opt-in holder demo-transkriptet byte-uendret | `shasum` | **MÅLT uendret** gjennom hele økten (`ea8c534…`). |
| A8 | 1.0.1 er API-identisk med 1.0.0 | S0 | **MÅLT: API-identisk, ATFERD ikke.** Diffen er upstream-regresjonsfiksen #4371: `StandardMagenticManager.__init__` mistet `self._session = agent.create_session()`, og hvert manager-kall mynter nå en engangs-sesjon. **Konsekvens: E2 OG E4 er BORTE** (4/4 → **0/5** begge). E1 (single-use `RuntimeError`, null kall) og E7 (`MagenticResetSignal` skriver til orphan-attributtet `_agent_thread`, `:1369`, sesjonsidentitet uendret) står. Manager-sesjonen fjernet også fra checkpoint-state, konsistent med at manageren nå er tilstandsløs per kall. |
**Målt korreksjon til E-tabellen (U12/S4):** kriteriet «detach `checkpoint_storage` → rød» er
FEIL — å fjerne `checkpoint_storage=` fra `run()` lar HELE suiten stå grønn (920 passed), fordi
`.with_checkpointing(...)` på builderen allerede ga workflowen lageret. De to bærende punktene er
`checkpoint_id=` (fjernes → rød) og builderens `.with_checkpointing(...)` (fjernes → rød).
**Load-bearing MÅLT** (mot HELE suiten, grønn kontroll 920/5): `manager_keeps_persistent_session`
konstant `True` (2 røde — E2+E4 alene) · aldri fest `BudgetMiddleware` på manageren (3 røde, mens
detach-armen forblir grønn) · detach markør-registreringen (1 rød — S3b-positiven alene, kontrollen
grønn) · flipp `_route`-rekkefølgen så `pre-survey` testes først (4 røde) · resume uten
`checkpoint_id=` (1 rød) · builder uten `.with_checkpointing()` (1 rød) · tom
`_ALLOWED_CHECKPOINT_TYPES` (1 rød) · og den falsifiserte: resume uten `checkpoint_storage=` (**0
røde** — funnet over).
**S5 — `quick_validate`-latens: median 13,6 ms** over 20 kall på `bygg-energi-baseline-mikro`
(forankret baseline + assumption-bånd, så stage 0 + CBC + 512-sample Monte Carlo er alle med).
Langt under 2 s-terskelen: verktøyet kan kalles fritt i løkka, og kontrakten trenger ingen
egen latens-post. Båndet er med med vilje — uten det faller `_monte_carlo` tilbake på
`item.unit_cost`, alle draw blir identiske, og tallet ville underrapportert den ekte kostnaden.
**S6 — OTEL er gratis, men IKKE via `enable_console_exporters`.** I et scratch-venv pinnet til
samme stack (core 1.9.0 / orch 1.0.1 / `opentelemetry-sdk` 1.44.0; **ingenting lagt til
`pyproject.toml`**) gir `configure_otel_providers(enable_console_exporters=True)` **2
`workflow.run`-spans** — pluss `workflow.build`, `executor.process`, `edge_group.process`,
`message.send`, `invoke_agent`, `chat synthetic` — men de skrives til **stdout**, som ville
ødelagt golden-transkriptet. Med `exporters=[ConsoleSpanExporter(out=sys.stderr)]` kommer de 2
`workflow.run`-spanene på **stderr** og demoens stdout er **byte-identisk med fasiten** (samme
shasum). U14 er altså én økts arbeid — forutsatt at exporteren konstrueres eksplisitt mot stderr,
aldri via flagget.
**Beslutninger som trenger operatøren (speiles i avslutningsblokken):**
1. Versjon: bli på 1.0.x (anbefalt) vs. core-bump til 1.15 nå.
1. **Versjon — premisset har flyttet seg.** Ordrens «felles hvis» sa: står E7 i 1.0.1, revert til
1.0.0. E7 STÅR — men 1.0.1 viste seg å fikse noe større enn det som ble håpet: E2/E4, altså
«en plausibel fasit produsert av null arbeid». Låsen står derfor på **1.0.1** i påvente av
operatøren. Alternativene: bli på 1.0.1 (anbefalt) · revert til 1.0.0 etter ordrens bokstav ·
core-bump til 1.15 nå (egen økt).
2. U14 som deklarert avhengighet (`opentelemetry-sdk` + exporter) i wheel/handover — ja/nei.
3. HITL-leveranse: synkron-først (anbefalt, uansett S4) vs. asynkron som krav i samme plan.
3. HITL-leveranse: synkron-først (anbefalt) vs. asynkron i samme plan — **S4 grønn, så asynkron
er nå teknisk mulig**; valget er ren rekkefølge, ikke lenger risiko.
4. Commons-amendment «Step 0 — Explore» — sende forslag nå (anbefalt: ja, via coord, uten å vente).
---

646
spikes/e_magentic.py Normal file
View file

@ -0,0 +1,646 @@
"""Spike E — the Magentic exploration loop, measured BEFORE it is built
(order 20260823T162224Z; plan ``docs/plan/2026-08-23-magentic-utforskningssloeyfe.md`` § D.1).
Nothing here is production code and nothing here is wired into ``src/``. Each function is one
measurement whose outcome moves a row of the plan's § F assumption table from "umålt" to a fact.
**The client is the repo's own ``ScriptedChatClient``, not an ad-hoc fake.** The scratch scripts
this spike ports used a bare ``BaseChatClient``, on which ``BudgetMiddleware`` is silently a
no-op (measured, ``simulation.py:373-375``) — so a budget claim proved with one would have proved
nothing. ``ScriptedChatClient`` subclasses the LAYERED ``OpenAIChatCompletionClient``, which is
what makes S2 a real measurement.
**The budget types are the PRODUCTION ones** (``portfolio_optimiser.budget``), deliberately NOT
``spikes/_harness.py``'s private copy. The harness copy is exactly why ``tick_round``'s
``observed`` went four raise-sites without coverage (kø-(y)); S2's whole question is whether the
SHIPPED middleware reaches the manager, and only the shipped object can answer it.
**Routing is on the joined prompt blob, in a fixed priority order, and that order is
load-bearing.** ``ScriptedChatClient``'s selector receives the concatenation of every message in
the call, so one manager call carries two markers (the plan prompt is built on a history that
still holds the pre-survey text — measured: 1 ambiguous call in 5). Checking the later-stage
marker first resolves it; the five-kind call shape each experiment asserts is what proves the
routing stayed correct.
"""
from __future__ import annotations
import json
import statistics
import subprocess
import sys
import time
from collections.abc import Callable, Sequence
from dataclasses import dataclass
from pathlib import Path
from typing import Any
from agent_framework import Agent, FileCheckpointStorage
from agent_framework.orchestrations import (
AgentRequestInfoResponse,
MagenticAgentExecutor,
MagenticBuilder,
MagenticPlanReviewResponse,
MagenticResetSignal,
StandardMagenticManager,
)
# ``AgentApprovalExecutor`` is the ONE name S3b needs that the package does not re-export
# (measured: ``hasattr(agent_framework.orchestrations, "AgentApprovalExecutor")`` is False while
# its response type IS public). Reaching into the private module is therefore part of the S3b
# FINDING, not an oversight: door 3 of § C.6 currently costs a private-API dependency, and that
# is a fact the operator's decision needs to carry.
from agent_framework_orchestrations._orchestration_request_info import AgentApprovalExecutor
import portfolio_optimiser
from portfolio_optimiser.budget import Budget, BudgetExceeded, BudgetMiddleware, TokenMeter
from portfolio_optimiser.ir import SavingsProposal
from portfolio_optimiser.okf import load_ir_projection, load_optional_cost_baseline
from portfolio_optimiser.simulation import ScriptedChatClient
from portfolio_optimiser.validator import validate_proposal
TASK_ALPHA = "TASK-ALPHA: find the saving in the alpha project."
TASK_BETA = "TASK-BETA: find the saving in the beta project."
WORKER_SENTINEL = "WORKER-SAW"
def _ledger(*, satisfied: bool, speaker: str) -> str:
"""A progress ledger naming ``speaker`` as the next talker.
The name is a PARAMETER because an unknown ``next_speaker`` is a silent footgun: the
orchestrator does not error on it, it quietly produces a final answer with zero participant
work (``_magentic.py:1128-1131``). Hard-coding "worker" here made the S3b run — whose
participant is the ``expert_liaison`` — finish without ever asking anyone (measured).
"""
return json.dumps(
{
"is_request_satisfied": {
"reason": "the participant replied" if satisfied else "no one has spoken yet",
"answer": satisfied,
},
"is_in_loop": {"reason": "no", "answer": False},
"is_progress_being_made": {"reason": "yes", "answer": True},
"next_speaker": {"reason": "it does the work", "answer": speaker},
"instruction_or_question": {
"reason": "done" if satisfied else "kick off",
"answer": "none" if satisfied else "Do the work now.",
},
}
)
@dataclass(frozen=True)
class ExplorationCallRecord:
"""One manager call, reduced to the facts every verdict here reads.
``sees_marker`` is the HITL half (S3b): whether a caller-supplied sentinel — an expert's
answer injected mid-run — had reached this manager prompt. It defaults to ``False`` so the
contamination experiments, which supply no marker, are unchanged.
"""
kind: str
messages: int
sees_alpha: bool
sees_beta: bool
sees_marker: bool = False
def _route(blob: str, speaker: str) -> tuple[str, str]:
"""Map a manager prompt blob to ``(kind, reply)``.
Order matters: the later-stage marker is tested FIRST because an earlier stage's text is
still present in the joined blob. Reversing two of these silently changes which prompt a
kind is attributed to, which is why every experiment asserts the resulting call shape.
"""
if "provide the final answer" in blob:
return "final", "FINAL: the worker did it."
if "pure JSON format" in blob:
if WORKER_SENTINEL in blob:
return "ledger_SAT", _ledger(satisfied=True, speaker=speaker)
return "ledger_UNSAT", _ledger(satisfied=False, speaker=speaker)
if "went wrong on this last run" in blob:
return "plan_update", "PLAN-UPDATE: ask the worker again."
if "rewrite the following fact sheet" in blob:
return "facts_update", "FACTS-UPDATE: still nothing."
if "bullet-point plan" in blob:
return "plan", "PLAN: - ask the worker"
if "pre-survey" in blob:
return "facts", "FACTS: nothing given."
return "unknown", "{}"
def _manager_client(
records: list[ExplorationCallRecord],
*,
marker: str | None = None,
speaker: str = "worker",
) -> ScriptedChatClient:
"""A manager-shaped scripted client appending one record per call to ``records``.
``records`` is CALLER-owned (the parse-failure-capture precedent): the evidence must survive
however the run ended, including a run cut short by ``BudgetExceeded`` mid-way.
"""
def _select(blob: str, _role: str) -> str:
kind, reply = _route(blob, speaker)
records.append(
ExplorationCallRecord(
kind=kind,
messages=blob.count("\n") + 1,
sees_alpha="TASK-ALPHA" in blob,
sees_beta="TASK-BETA" in blob,
sees_marker=marker is not None and marker in blob,
)
)
return reply
return ScriptedChatClient(reply_selector=_select, role="manager")
def _worker_client(seen: list[str]) -> ScriptedChatClient:
def _select(blob: str, _role: str) -> str:
which = "ALPHA" if "TASK-ALPHA" in blob else ("BETA" if "TASK-BETA" in blob else "NOTHING")
seen.append(which)
return f"{WORKER_SENTINEL}-{which}"
return ScriptedChatClient(reply_selector=_select, role="worker")
def _manager_agent(client: ScriptedChatClient, *, middleware: Sequence[Any] | None = None) -> Agent:
return Agent(
client,
"You are the Magentic manager.",
name="manager",
description="plans the work",
middleware=middleware,
)
def _worker_agent(client: ScriptedChatClient) -> Agent:
return Agent(client, "You are the worker.", name="worker", description="does the work")
async def _run(workflow: Any, task: str) -> dict[str, Any]:
try:
result = await workflow.run(task)
outputs = [str(o) for o in (result.get_outputs() or [])]
return {"ok": True, "outputs": outputs}
except Exception as exc: # noqa: BLE001 - the failure mode IS the measurement
return {"ok": False, "error": type(exc).__name__, "message": str(exc)}
def manager_keeps_persistent_session() -> bool:
"""Does the INSTALLED ``StandardMagenticManager`` hold one ``AgentSession`` for its whole
life, or mint a throwaway one per call?
This is the single structural property that decides E2 and E4. Orchestrations 1.0.0 assigns
``self._session = self._agent.create_session()`` in ``__init__``; 1.0.1 removed that line and
creates the session inside the call instead (upstream regression fix #4371). Probing the
attribute rather than the version string states the CAUSE, and keeps the spike honest across
a version the plan has not seen.
"""
manager = StandardMagenticManager(agent=_manager_agent(_manager_client([])))
return hasattr(manager, "_session")
async def single_use_second_run() -> dict[str, Any]:
"""E1: build one Magentic workflow, run it twice. Measures the second run's outcome and
what it COST — a refusal that still made model calls would be a different finding."""
records: list[ExplorationCallRecord] = []
worker_seen: list[str] = []
workflow = MagenticBuilder(
participants=[_worker_agent(_worker_client(worker_seen))],
manager_agent=_manager_agent(_manager_client(records)),
max_round_count=6,
).build()
first = await _run(workflow, TASK_ALPHA)
manager_before, worker_before = len(records), len(worker_seen)
second = await _run(workflow, TASK_BETA)
return {
"first_ok": first["ok"],
"second_error": second.get("error"),
"second_message": second.get("message", ""),
"manager_calls_added": len(records) - manager_before,
"worker_calls_added": len(worker_seen) - worker_before,
}
async def _two_runs(
build_second: Callable[[list[ExplorationCallRecord]], Any],
*,
shared_records: bool,
) -> tuple[int, int, list[str]]:
"""Run ALPHA, then BETA, and report how many of run 2's manager calls still see ALPHA.
``shared_records`` says whether run 2's manager is the same object as run 1's — when it is,
run 2's records are the tail of one list; when it is not, they are their own list. Both
shapes reduce to the same verdict triple so the callers stay comparable.
"""
records: list[ExplorationCallRecord] = []
first_worker: list[str] = []
first = MagenticBuilder(
participants=[_worker_agent(_worker_client(first_worker))],
manager_agent=_manager_agent(_manager_client(records)),
max_round_count=6,
).build()
await _run(first, TASK_ALPHA)
split = len(records)
second_records = records if shared_records else []
second = build_second(second_records)
await _run(second, TASK_BETA)
tail = records[split:] if shared_records else second_records
bled = sum(1 for record in tail if record.sees_alpha)
return bled, len(tail), [record.kind for record in tail]
async def shared_manager_contamination() -> tuple[int, int, list[str]]:
"""E2: two builders sharing ONE ``StandardMagenticManager`` instance.
Built directly (not via ``_two_runs``) because run 1 must go through the SAME manager object,
which ``manager=`` accepts and ``manager_agent=`` does not.
"""
records: list[ExplorationCallRecord] = []
shared = StandardMagenticManager(
agent=_manager_agent(_manager_client(records)), max_round_count=6
)
first = MagenticBuilder(
participants=[_worker_agent(_worker_client([]))], manager=shared
).build()
await _run(first, TASK_ALPHA)
split = len(records)
second = MagenticBuilder(
participants=[_worker_agent(_worker_client([]))], manager=shared
).build()
await _run(second, TASK_BETA)
tail = records[split:]
return sum(1 for r in tail if r.sees_alpha), len(tail), [r.kind for r in tail]
async def shared_builder_contamination() -> tuple[int, int, list[str]]:
"""E4: ONE ``MagenticBuilder``, ``.build()`` twice. ``manager_agent=`` constructs the manager
eagerly and hands the same instance to every build — the accidental route into E2."""
records: list[ExplorationCallRecord] = []
builder = MagenticBuilder(
participants=[_worker_agent(_worker_client([]))],
manager_agent=_manager_agent(_manager_client(records)),
max_round_count=6,
)
await _run(builder.build(), TASK_ALPHA)
split = len(records)
await _run(builder.build(), TASK_BETA)
tail = records[split:]
return sum(1 for r in tail if r.sees_alpha), len(tail), [r.kind for r in tail]
async def fresh_manager_contamination() -> tuple[int, int, list[str]]:
"""E3, the control and the mitigation: a fresh builder, agent and client per exploration."""
def _second(records: list[ExplorationCallRecord]) -> Any:
return MagenticBuilder(
participants=[_worker_agent(_worker_client([]))],
manager_agent=_manager_agent(_manager_client(records)),
max_round_count=6,
).build()
return await _two_runs(_second, shared_records=False)
async def reset_signal_resets_participant_session() -> dict[str, Any]:
"""E7: does ``MagenticResetSignal`` actually give the participant a clean session?
Measured no: the fresh session is written to ``_agent_thread``, which nothing reads, while
the live ``_session`` keeps its identity. The stall-replan path therefore hands the manager a
clean ledger and the participants their old memory.
"""
executor = MagenticAgentExecutor(_worker_agent(_worker_client([])))
session_before = id(executor._session)
executor._cache.append("sentinel") # type: ignore[arg-type]
executor._full_conversation.append("sentinel") # type: ignore[arg-type]
await executor.handle_magentic_reset(MagenticResetSignal(), None) # type: ignore[arg-type]
return {
"cache_cleared": len(executor._cache) == 0,
"conversation_cleared": len(executor._full_conversation) == 0,
"session_identity_changed": id(executor._session) != session_before,
"orphan_attribute_written": hasattr(executor, "_agent_thread"),
}
async def manager_budget_enforced(
*, max_tokens: int, attach: bool, return_exception: bool = False
) -> dict[str, Any]:
"""S2: put the SHIPPED ``BudgetMiddleware`` on the manager agent and see whether the typed
refusal leaves ``workflow.run``.
``attach=False`` is the detach control: the same one-token budget with no middleware must
stop nothing. Without that arm the positive test would pass on any implementation in which
something, anything, raised.
"""
meter = TokenMeter(Budget(max_tokens=max_tokens, max_rounds=8))
middleware = [BudgetMiddleware(meter)] if attach else None
workflow = MagenticBuilder(
participants=[_worker_agent(_worker_client([]))],
manager_agent=_manager_agent(_manager_client([]), middleware=middleware),
max_round_count=6,
).build()
raised: str | None = None
kind: str | None = None
exception: BaseException | None = None
completed = False
try:
result = await workflow.run(TASK_ALPHA)
completed = bool(result.get_outputs())
except BudgetExceeded as exc:
raised, kind, exception = type(exc).__name__, exc.kind, exc
except Exception as exc: # noqa: BLE001 - a DIFFERENT exception type is itself the finding
raised, exception = type(exc).__name__, exc
payload: dict[str, Any] = {
"raised": raised,
"kind": kind,
"completed": completed,
"meter_tokens": meter.tokens,
}
if return_exception:
payload["exception"] = exception
return payload
# ---------------------------------------------------------------------------
# S3 / S3b — the two HITL doors (plan § C.5, § C.6)
# ---------------------------------------------------------------------------
def _pending_requests(result: Any) -> list[Any]:
return [event for event in result if event.type == "request_info"]
async def plan_review_round_trip() -> dict[str, Any]:
"""S3: ``enable_plan_review=True`` → the run stops with a ``MagenticPlanReviewRequest`` and no
output → ``revise(...)`` replans and asks AGAIN → ``approve()`` lets the loop run.
The manager-call count PER revise is the number the contract needs: a revise costs model
calls but is not counted as a round by the orchestration (measured: no ledger call), so an
uncapped reviser is an unbounded spend the plan's ``max_plan_revisions`` has to bound.
"""
records: list[ExplorationCallRecord] = []
workflow = MagenticBuilder(
participants=[_worker_agent(_worker_client([]))],
manager_agent=_manager_agent(_manager_client(records)),
max_round_count=6,
enable_plan_review=True,
).build()
first = await workflow.run(TASK_ALPHA)
pending = _pending_requests(first)
stopped_without_output = not first.get_outputs()
review_kinds = [r.kind for r in records]
before_revise = len(records)
revised = await workflow.run(
responses={pending[0].request_id: MagenticPlanReviewResponse.revise("Test the LED case.")}
)
revise_calls = [r.kind for r in records[before_revise:]]
pending_after_revise = _pending_requests(revised)
approve_id = (
pending_after_revise[0].request_id if pending_after_revise else pending[0].request_id
)
approved = await workflow.run(responses={approve_id: MagenticPlanReviewResponse.approve()})
return {
"pending_before_review": len(pending),
"request_type": type(pending[0].data).__name__ if pending else None,
"is_stalled": bool(pending[0].data.is_stalled) if pending else None,
"stopped_without_output": stopped_without_output,
"kinds_before_review": review_kinds,
"revise_manager_calls": revise_calls,
"pending_after_revise": len(pending_after_revise),
"outputs_after_approve": [str(o) for o in (approved.get_outputs() or [])],
}
async def expert_liaison_answer_round_trip(
*, answer: str, probe: str | None = None
) -> dict[str, Any]:
"""S3b: an ``AgentApprovalExecutor`` standing in as the ``expert_liaison`` PARTICIPANT — the
third door of § C.6, the one that lets the manager ask a question MID-run.
Measured shape, two round-trips per human turn: the manager picks the liaison → its output
becomes a ``request_info`` → ``from_strings([answer])`` feeds the human's words back INTO the
liaison, which runs again → a second ``request_info`` → ``approve()`` forwards the liaison's
(now informed) output to the manager, which resumes. ``from_strings`` alone does NOT resume
the manager: measured zero manager calls between the two requests.
``probe`` is the CONTROL knob: the manager's prompts are scanned for IT instead of for
``answer``, so a sentinel the expert never sent must come back absent. Without that arm,
"the manager saw the answer" could equally mean "the scanner matches anything".
"""
records: list[ExplorationCallRecord] = []
liaison_prompts: list[str] = []
def _liaison_reply(blob: str, _role: str) -> str:
liaison_prompts.append(blob)
if answer in blob:
return f"{WORKER_SENTINEL}-LIAISON heard: {answer}"
return f"{WORKER_SENTINEL}-LIAISON has no expert input yet."
liaison = AgentApprovalExecutor(
Agent(
ScriptedChatClient(reply_selector=_liaison_reply, role="expert_liaison"),
"You relay the expert's answers.",
name="expert_liaison",
description="asks the human expert",
)
)
workflow = MagenticBuilder(
participants=[liaison],
manager_agent=_manager_agent(
_manager_client(records, marker=probe or answer, speaker="expert_liaison")
),
max_round_count=6,
).build()
first = await workflow.run(TASK_ALPHA)
asked = _pending_requests(first)
before_answer = len(records)
answered = await workflow.run(
responses={asked[0].request_id: AgentRequestInfoResponse.from_strings([answer])}
)
manager_calls_on_answer = [r.kind for r in records[before_answer:]]
second = _pending_requests(answered)
approved = (
await workflow.run(responses={second[0].request_id: AgentRequestInfoResponse.approve()})
if second
else answered
)
return {
"reachable": bool(asked),
"request_type": type(asked[0].data).__name__ if asked else None,
"manager_calls_between_requests": manager_calls_on_answer,
"second_request": len(second),
"liaison_saw_answer": any(answer in prompt for prompt in liaison_prompts),
"manager_saw_answer": any(r.sees_marker for r in records),
"outputs": [str(o) for o in (approved.get_outputs() or [])],
}
# ---------------------------------------------------------------------------
# S4 — resume a pending plan review in a NEW PROCESS (plan U12)
# ---------------------------------------------------------------------------
async def checkpoint_until_plan_review(storage_dir: str) -> dict[str, Any]:
"""Run until the plan review stops the workflow, leaving checkpoints on disk.
This is the FIRST half of S4 and runs in the parent process; the resume half must run in a
separate interpreter (``spikes.e_magentic_resume``), because a resume that quietly rode on
live in-process objects would prove nothing about the asynchronous file inbox U12 needs.
"""
records: list[ExplorationCallRecord] = []
workflow = _plan_review_workflow(records, storage_dir)
result = await workflow.run(TASK_ALPHA)
pending = _pending_requests(result)
checkpoints = await _checkpoint_storage(storage_dir).list_checkpoints(
workflow_name=workflow.name
)
return {
"request_id": pending[0].request_id if pending else None,
"checkpoint_ids": [c.checkpoint_id for c in checkpoints],
"outputs": [str(o) for o in (result.get_outputs() or [])],
}
# Measured, and a real cost of the asynchronous HITL door: ``FileCheckpointStorage`` refuses to
# unpickle a plan-review request unless its type is declared. Without this the checkpoint file is
# written but comes back UNREADABLE ("Checkpoint deserialization blocked for type ..."), and the
# listing is empty — a resume that fails as an absence rather than as an error, which is exactly
# the shape the fourth face of the verification law warns about. Both processes must declare it.
_ALLOWED_CHECKPOINT_TYPES = [
"agent_framework_orchestrations._magentic:MagenticPlanReviewRequest",
"agent_framework_orchestrations._magentic:MagenticPlanReviewResponse",
]
def _checkpoint_storage(storage_dir: str) -> FileCheckpointStorage:
return FileCheckpointStorage(storage_dir, allowed_checkpoint_types=_ALLOWED_CHECKPOINT_TYPES)
def _plan_review_workflow(records: list[ExplorationCallRecord], storage_dir: str) -> Any:
"""The workflow BOTH processes build — identical construction, so the only thing carried
across the process boundary is the checkpoint on disk."""
return (
MagenticBuilder(
participants=[_worker_agent(_worker_client([]))],
manager_agent=_manager_agent(_manager_client(records)),
max_round_count=6,
enable_plan_review=True,
)
.with_checkpointing(_checkpoint_storage(storage_dir))
.build()
)
async def resume_from_checkpoint(
storage_dir: str, *, request_id: str, checkpoint_id: str
) -> dict[str, Any]:
"""The SECOND half of S4, called by ``spikes.e_magentic_resume`` in a fresh interpreter.
**Measured, and it contradicts the plan's E-table:** ``checkpoint_storage=`` on ``run()`` is
NOT the load-bearing seam here — removing it leaves the whole suite green (920 passed),
because ``.with_checkpointing(...)`` on the builder already gave this workflow its storage.
The two arguments that ARE load-bearing are ``checkpoint_id=`` (drop it → red) and the
builder's ``.with_checkpointing(...)`` (drop it → red). It is passed anyway, explicitly,
because an exploration layer that builds its workflow WITHOUT checkpointing and resumes by
handing storage in at call time is a legitimate second shape — but a criterion that names it
as the detach point would be a gate that cannot go red.
"""
records: list[ExplorationCallRecord] = []
workflow = _plan_review_workflow(records, storage_dir)
result = await workflow.run(
responses={request_id: MagenticPlanReviewResponse.approve()},
checkpoint_id=checkpoint_id,
checkpoint_storage=_checkpoint_storage(storage_dir),
)
return {
"manager_kinds": [r.kind for r in records],
"pending_after_resume": len(_pending_requests(result)),
"outputs": [str(o) for o in (result.get_outputs() or [])],
}
def run_resume_subprocess(
storage_dir: str, *, request_id: str, checkpoint_id: str
) -> dict[str, Any]:
"""Launch ``spikes.e_magentic_resume`` in a FRESH interpreter and parse its one JSON line.
``sys.executable`` is this venv's Python, and the repo root is the working directory, so the
child imports the same tree the parent did without any path juggling. A non-zero exit is
surfaced with the child's stderr attached: a resume that failed must read as a failed resume,
never as an empty result.
"""
completed = subprocess.run(
[sys.executable, "-m", "spikes.e_magentic_resume", storage_dir, request_id, checkpoint_id],
capture_output=True,
text=True,
cwd=str(Path(__file__).resolve().parents[1]),
)
if completed.returncode != 0:
raise RuntimeError(
f"resume subprocess exited {completed.returncode}: {completed.stderr.strip()}"
)
return dict(json.loads(completed.stdout.strip().splitlines()[-1]))
def micro_bundle_dir() -> str:
"""The repo's own anchored micro bundle — the one bundle that ships BOTH a cost baseline and
an IR projection, so a latency number measured here is measured through the WHOLE gate
(stage 0 reconciliation + CBC solve + 512-sample Monte Carlo), not a subset of it."""
return str(
Path(portfolio_optimiser.__file__).parent
/ "data"
/ "bundles"
/ "bygg-energi-baseline-mikro"
)
def micro_proposal() -> SavingsProposal:
"""The bundle's IR projection as a validated ``SavingsProposal``, carrying an assumption band.
The band is not decoration: without one, ``validator._monte_carlo`` falls back to each item's
own ``unit_cost`` and every draw is identical — a cheaper computation than any real hypothesis
would trigger, so a latency measured without it would understate the in-loop cost.
"""
projection = dict(load_ir_projection(micro_bundle_dir()))
projection.pop("_note", None)
item = projection["affected_items"][0]
unit_cost = float(item["unit_cost"])
projection["assumptions"] = {item["code"]: (unit_cost * 0.9, unit_cost * 1.1)}
return SavingsProposal.model_validate(projection)
def validator_latency_seconds(*, runs: int) -> tuple[float, int]:
"""S5: median wall-clock of one ``validate_proposal`` against the micro reference bundle —
the cost the hypothesiser's ``quick_validate`` tool would pay per call, per hypothesis."""
baseline = load_optional_cost_baseline(micro_bundle_dir())
proposal = micro_proposal()
timings: list[float] = []
for _ in range(runs):
start = time.perf_counter()
validate_proposal(proposal, baseline=baseline)
timings.append(time.perf_counter() - start)
return statistics.median(timings), len(timings)

39
spikes/e_magentic_resume.py Normal file
View file

@ -0,0 +1,39 @@
"""S4's second half: resume a pending Magentic plan review in a **fresh interpreter**.
A resume proved inside the parent process would prove nothing about U12. The whole point of the
asynchronous HITL time-scale is that the expert answers days later, in a process that never saw
the run — so the only thing allowed to cross the boundary is the checkpoint on disk. Running the
resume as its own module, launched by ``subprocess``, is what makes that boundary real
(``tests/test_hosting_loadbearing.py`` set this precedent for the hosted shim).
Usage (the test supplies the arguments; nothing here is wired into ``src/``)::
python -m spikes.e_magentic_resume <storage_dir> <request_id> <checkpoint_id>
Writes ONE line of JSON to stdout: the resumed run's manager calls, remaining requests and
outputs. Anything else on stdout would make the parent's parse the fragile part of the spike.
"""
from __future__ import annotations
import asyncio
import json
import sys
from spikes.e_magentic import resume_from_checkpoint
def main(argv: list[str]) -> int:
if len(argv) != 4:
print(f"usage: {argv[0]} <storage_dir> <request_id> <checkpoint_id>", file=sys.stderr)
return 2
storage_dir, request_id, checkpoint_id = argv[1], argv[2], argv[3]
result = asyncio.run(
resume_from_checkpoint(storage_dir, request_id=request_id, checkpoint_id=checkpoint_id)
)
print(json.dumps(result, sort_keys=True))
return 0
if __name__ == "__main__":
raise SystemExit(main(sys.argv))

260
tests/spikes/test_e_magentic.py Normal file
View file

@ -0,0 +1,260 @@
"""Spike E tests — Magentic exploration loop, MEASURED before anything is built
(order 20260823T162224Z, plan § D.1 spikes S0–S6).
Every test here pins ONE binary outcome the plan's § F assumption table needs. Where an
outcome is version-dependent (the manager's session lifetime changed in orchestrations
1.0.1, upstream regression fix #4371), the test asserts against the STRUCTURAL probe of the
installed manager — never a version string — and carries a non-vacuity control so a zero
can never be the absence of a run.
Pattern: tests/spikes/test_b_footguns.py.
"""
from __future__ import annotations
from pathlib import Path
import pytest
from portfolio_optimiser.budget import BudgetExceeded
from spikes.e_magentic import (
ExplorationCallRecord,
checkpoint_until_plan_review,
expert_liaison_answer_round_trip,
fresh_manager_contamination,
manager_budget_enforced,
manager_keeps_persistent_session,
plan_review_round_trip,
reset_signal_resets_participant_session,
run_resume_subprocess,
shared_builder_contamination,
shared_manager_contamination,
single_use_second_run,
validator_latency_seconds,
)
# ---------------------------------------------------------------------------
# S1 — B7 state isolation (E1–E4, E7) in the repo's own form
# ---------------------------------------------------------------------------
async def test_e1_second_run_of_one_built_workflow_is_refused() -> None:
"""E1: a built Magentic workflow is SINGLE-USE. The second ``.run()`` raises and makes
ZERO model calls — stronger than GroupChat 1.9.0's silent empty [2, 0, 0] re-run, because
a workflow that cannot run cannot fabricate an answer."""
result = await single_use_second_run()
assert result["first_ok"] is True
assert result["second_error"] == "RuntimeError"
assert "already been completed" in result["second_message"]
# The refusal is FREE: nothing was spent proving it.
assert result["manager_calls_added"] == 0
assert result["worker_calls_added"] == 0
async def test_e3_fresh_manager_per_build_never_contaminates() -> None:
"""E3 — the CONTROL, and the invariant the plan's C.4 rule rests on: a fresh manager
(fresh builder, fresh agent, fresh client) per exploration leaks nothing across runs, on
EVERY measured version. This is what ``fresh_exploration_workflow`` will implement."""
bled, total, kinds = await fresh_manager_contamination()
assert total >= 4, f"only {total} manager calls — the control would be reporting on nothing"
assert bled == 0
# The five-call shape (facts, plan, ledger_UNSAT, ledger_SAT, final) is the discriminator:
# a contaminated run answers satisfied on its FIRST ledger and never calls the worker.
assert kinds == ["facts", "plan", "ledger_UNSAT", "ledger_SAT", "final"]
async def test_e2_shared_manager_bleed_tracks_the_persistent_session() -> None:
"""E2: two workflows sharing ONE ``StandardMagenticManager``. Whether run 2's manager still
sees run 1's task is decided by ONE structural property of the installed manager — whether
it holds a persistent ``AgentSession`` built in ``__init__`` (1.0.0) or creates a throwaway
one per call (1.0.1, #4371). The probe reads that property, never a version string."""
bled, total, kinds = await shared_manager_contamination()
assert total >= 4, f"only {total} manager calls — a zero here would be vacuous"
if manager_keeps_persistent_session():
assert bled == total, "a persistent manager session must bleed EVERY run-2 call"
# The fabricated-answer signature: satisfied on the first ledger, worker never called.
assert "ledger_UNSAT" not in kinds
else:
assert bled == 0, "a per-call manager session must leak nothing"
assert kinds == ["facts", "plan", "ledger_UNSAT", "ledger_SAT", "final"]
async def test_e4_shared_builder_bleed_tracks_the_persistent_session() -> None:
"""E4: ONE ``MagenticBuilder``, two ``.build()`` calls. ``manager_agent=`` constructs the
manager eagerly and hands the SAME instance to every build, so E4 is E2 reached by the
route a caller is most likely to take by accident. Same discriminator."""
bled, total, kinds = await shared_builder_contamination()
assert total >= 4, f"only {total} manager calls — a zero here would be vacuous"
if manager_keeps_persistent_session():
assert bled == total
assert "ledger_UNSAT" not in kinds
else:
assert bled == 0
assert kinds == ["facts", "plan", "ledger_UNSAT", "ledger_SAT", "final"]
async def test_e7_reset_signal_does_not_reset_the_participant_session() -> None:
"""E7: ``MagenticResetSignal`` clears the cache and the conversation but writes the fresh
session to ``_agent_thread`` — an attribute the executor never reads. Stall-replan therefore
gives a fresh manager ledger and STALE participants. Measured on 1.0.0 and still true on
1.0.1: never rely on the reset to empty participant memory."""
result = await reset_signal_resets_participant_session()
assert result["cache_cleared"] is True
assert result["conversation_cleared"] is True
assert result["session_identity_changed"] is False # the whole finding
assert result["orphan_attribute_written"] is True
# ---------------------------------------------------------------------------
# S2 — does BudgetMiddleware fire on the MANAGER's calls? (plan § F / A1, A2)
# ---------------------------------------------------------------------------
async def test_s2_budget_middleware_fires_on_the_manager_path() -> None:
"""A1 + A2: the manager is the most talkative participant, and the plan's hard token cap is
a lie if agent-level ``ChatMiddleware`` does not reach it. Two halves, both required: the
typed ``BudgetExceeded`` must LEAVE ``workflow.run`` (A2), and the meter must have been
CHARGED (A1) — a refusal with a zero meter would prove only that something raised."""
result = await manager_budget_enforced(max_tokens=1, attach=True)
assert result["raised"] == "BudgetExceeded"
assert result["kind"] == "tokens"
assert result["meter_tokens"] > 0, "the middleware never charged — it did not run"
assert result["completed"] is False
async def test_s2_control_detaching_the_manager_middleware_lets_the_run_finish() -> None:
"""The detach control the order requires: with no middleware on the manager the SAME
one-token budget stops nothing and the run completes. Without this the test above could
pass on an implementation where anything at all raised."""
result = await manager_budget_enforced(max_tokens=1, attach=False)
assert result["raised"] is None
assert result["completed"] is True
assert result["meter_tokens"] == 0
async def test_s2_budget_exceeded_is_the_repo_type_not_a_look_alike() -> None:
"""The exception that leaves ``workflow.run`` must be the repo's own ``BudgetExceeded``
carrying ``kind``/``limit``/``observed`` — the triple the 429 channel reads (kø-(y)). An
orchestration layer that wrapped it in an ``ExceptionGroup`` would make the exploration
layer's error mapping a fiction, so the object itself is asserted, not just its name."""
result = await manager_budget_enforced(max_tokens=1, attach=True, return_exception=True)
exc = result["exception"]
assert isinstance(exc, BudgetExceeded)
assert exc.kind == "tokens"
assert exc.limit == 1
assert exc.observed > exc.limit # limit and observed must not be the same number
# ---------------------------------------------------------------------------
# S5 — quick_validate latency (plan C.0 level 1: the tool the hypothesiser calls)
# ---------------------------------------------------------------------------
def test_s5_validator_latency_is_affordable_as_an_in_loop_tool() -> None:
"""S5: ``validate_proposal`` is what ``quick_validate`` wraps. If a single call costs
seconds, every hypothesis the manager tests costs wall-clock the contract has to budget."""
median, samples = validator_latency_seconds(runs=20)
assert samples == 20
assert median > 0.0, "a zero median means the clock never moved — nothing was measured"
assert median < 2.0, f"median {median:.3f}s per validate_proposal — budget it in the contract"
def test_call_record_is_a_plain_readable_row() -> None:
"""The record type carries the four facts every contamination verdict is computed from;
a spike whose evidence cannot be printed is a spike nobody can re-check."""
record = ExplorationCallRecord(kind="facts", messages=2, sees_alpha=True, sees_beta=False)
assert record.kind == "facts"
assert record.sees_alpha is True
@pytest.mark.parametrize("attach", [True, False])
async def test_s2_arms_disagree(attach: bool) -> None:
"""Both arms of S2 run under one parametrisation too, so a future refactor that made the
two arms identical shows up as a shared outcome rather than as two green tests."""
result = await manager_budget_enforced(max_tokens=1, attach=attach)
assert result["completed"] is not attach
# ---------------------------------------------------------------------------
# S3 / S3b — the two HITL doors (plan § C.5 / § C.6, assumptions A3 and A5)
# ---------------------------------------------------------------------------
async def test_s3_plan_review_round_trip_revises_then_approves() -> None:
"""A3: the request/response round-trip the plan's synchronous HITL rests on.
Four facts in one run: the review STOPS the workflow before any output; a ``revise`` replans
and asks AGAIN (so an always-revising expert is an unbounded loop unless the contract caps
it); the revise costs manager calls but NO ledger call (it is not a round); and an
``approve`` lets the loop finish."""
result = await plan_review_round_trip()
assert result["pending_before_review"] == 1
assert result["request_type"] == "MagenticPlanReviewRequest"
assert result["is_stalled"] is False
assert result["stopped_without_output"] is True
# Before the review the manager has only surveyed and planned -- it never reached a ledger.
assert result["kinds_before_review"] == ["facts", "plan"]
# The measured cost of one revise: two manager calls, zero ledger calls, zero rounds.
assert result["revise_manager_calls"] == ["facts_update", "plan_update"]
assert not any(k.startswith("ledger") for k in result["revise_manager_calls"])
# ... and it asks again -- this is why max_plan_revisions must exist in the contract.
assert result["pending_after_revise"] == 1
assert result["outputs_after_approve"] == ["FINAL: the worker did it."]
async def test_s3b_expert_liaison_answer_reaches_the_manager() -> None:
"""A5: ``AgentApprovalExecutor`` as the ``expert_liaison`` PARTICIPANT — door 3 of § C.6.
Reachable, and the expert's words do become context: the liaison sees them, and the manager
sees them in a later prompt. The cost is two round-trips per human turn -- ``from_strings``
feeds the answer back into the liaison and does NOT resume the manager (measured: zero
manager calls between the two requests); only ``approve`` forwards the output."""
answer = "EXPERT-SAYS-TEST-THE-LED-RETROFIT"
result = await expert_liaison_answer_round_trip(answer=answer)
assert result["reachable"] is True
assert result["manager_calls_between_requests"] == [] # from_strings alone resumes nothing
assert result["second_request"] == 1 # ... it asks again, with the answer folded in
assert result["liaison_saw_answer"] is True
assert result["manager_saw_answer"] is True # the point: it becomes context for the next round
assert result["outputs"] == ["FINAL: the worker did it."]
async def test_s3b_control_an_unsent_sentinel_never_reaches_the_manager() -> None:
"""The control that makes the assertion above non-vacuous. Same run, same expert answer --
but the manager's prompts are scanned for a sentinel the expert NEVER sent. It must come
back absent. Without this arm, ``manager_saw_answer is True`` would be equally consistent
with a scanner that matches anything, and door 3 would look proven when it was not."""
result = await expert_liaison_answer_round_trip(
answer="EXPERT-SAYS-TEST-THE-LED-RETROFIT", probe="NEVER-SENT-SENTINEL-XYZ"
)
assert result["reachable"] is True
assert result["liaison_saw_answer"] is True # the run really happened
assert result["manager_saw_answer"] is False # ... and the unsent sentinel is nowhere in it
# ---------------------------------------------------------------------------
# S4 — resume a pending plan review in a NEW PROCESS (plan U12, assumption A4)
# ---------------------------------------------------------------------------
async def test_s4_pending_plan_review_resumes_in_a_fresh_process(tmp_path: Path) -> None:
"""A4: the asynchronous HITL time-scale. The parent runs until the plan review stops it and
leaves checkpoints on disk; a SEPARATE interpreter -- which never saw the run -- answers the
pending request from the checkpoint alone and drives the workflow to its final answer.
In-process resume would prove nothing here: U12's whole claim is that the expert can answer
days later from a file inbox."""
storage = tmp_path / "checkpoints"
storage.mkdir()
first = await checkpoint_until_plan_review(str(storage))
assert first["request_id"], "no plan review was raised -- nothing to resume"
assert first["outputs"] == []
assert first["checkpoint_ids"], "no checkpoint was written -- the resume would be vacuous"
resumed = run_resume_subprocess(
str(storage), request_id=first["request_id"], checkpoint_id=first["checkpoint_ids"][-1]
)
assert resumed["outputs"] == ["FINAL: the worker did it."]
assert resumed["pending_after_resume"] == 0
# The fresh process really did the remaining work -- not a replay of a cached answer.
assert any(k.startswith("ledger") for k in resumed["manager_kinds"])

6
uv.lock generated
View file

@ -59,14 +59,14 @@ wheels = [
[[package]]
name = "agent-framework-orchestrations"
version = "1.0.0"
version = "1.0.1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "agent-framework-core" },
]
sdist = { url = "https://files.pythonhosted.org/packages/7a/98/1f943e3a383bf37af24f1cc3ab3812fbe2041b65bc2e2f82626352907190/agent_framework_orchestrations-1.0.0.tar.gz", hash = "sha256:7f4d26de1ac0b2add3742964836927df44138e79d16bbf084d6e866f1d4b5aff", size = 60805, upload-time = "2026-06-18T09:43:02.667Z" }
sdist = { url = "https://files.pythonhosted.org/packages/ca/e7/e6a123b74995bba5eaa9e4491e2db3cf94ebd8508df0abefffdec3f86468/agent_framework_orchestrations-1.0.1.tar.gz", hash = "sha256:8441488edb1512d50e0ca70e0bee3ae87a023636cbe860b57b80103b4bac50ec", size = 61003, upload-time = "2026-07-21T19:50:42.112Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/48/cc/64448d22e95db33bb7701d3a2bb8888d39f2fa8f4e410bd24232417c9ea3/agent_framework_orchestrations-1.0.0-py3-none-any.whl", hash = "sha256:1f0885c3de69c919bb631f7e4e24727d94733b135a1529ab88ffa47bf18a2878", size = 66828, upload-time = "2026-06-18T09:42:59.363Z" },
{ url = "https://files.pythonhosted.org/packages/47/5d/a662a69014881e5ee593b39f64b97c0d34fc036c333eab73e9bc10019308/agent_framework_orchestrations-1.0.1-py3-none-any.whl", hash = "sha256:eb10454d462728ab83db14ba42d2c870f58d536966b9b60b525e5e504aa7e8c9", size = 67069, upload-time = "2026-07-21T19:50:13.408Z" },
]
[[package]]