open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 3 Activity

feat(explore): hvilken base som faktisk ble AAPNET forlater kjoeringen (MAJOR-1 a, ORDRE 20260902T151931Z)

Browse source 
`{run_id}-exploration.json` bar syv noekler og ingen rad for verktoey: en
utforskning som kalte NULL verktoey var uskillbar fra en som navigerte hele
korpuset, baade offline og etter en BETALT kjoering. `ExplorationToolRecorder`
er en `FunctionMiddleware` paa utforskningsagentene som registrerer navnet og
`bundle_id`-argumentet, i kall-rekkefoelge, paa den kaller-eide
`ExplorationTrace` — og `trace_payload` skriver det ved siden av
`quick_validations`.

SOESKEN av `mcp_tools.ToolCallRecorder`, ALDRI en gjenbruk: invariantene er
motsatte. Den filtrerer til KONFIGURERTE eksterne verktoey, dedupliserer per
`(server, tool)` og returnerer SORTERT, fordi dens rad er en egress-paastand i
et byte-deterministisk artefakt. Denne beholder hvert kall i REKKEFOELGE uten
dedup, fordi spoersmaalet er det motsatte: aapnet navigatoeren noe, og i hvilken
sekvens. Et sortert dedupet sett kan ikke skille en generalproeve som LESTE en
base fra en som bare listet dem. Aa generalisere den ene til aa tjene begge
ville brutt den andre.

RESULTATET registreres ALDRI — det er basens innhold, altsaa nettopp det som er
for stort til aa ri med (MAJOR-3 maalte 73-93 % av alle prompt-tokens), og et
spor som bar det ville vaert en andre kopi av konteksten. Middlewaren observerer
kun; `call_next` ventes alltid.

MAALT FOERST (Iron Law): fire tester roede mot HEAD foer sommen fantes.
`FunctionInvocationContext.arguments` er `BaseModel | Mapping[str, Any]` (maalt
mot den installerte signaturen), saa BEGGE former leses; et verktoey uten
`bundle_id` gir `""`, aldri en oppdiktet etikett. Wiret i BEGGE
workflow-byggene — `explore()` OG `resume_exploration()` — ellers ville et
gjenopptatt leg registrert null verktoeykall, samme stille gap som
`plan_reviews.extend` i oekt 64.

Suite 2 failed (F15-diffen, KJENT) / 1082 passed / 5 skipped (1089 samlet, +4).
Golden `demo-transcript.stdout` BYTE-UENDRET
(ea8c534773acdbe41ae68f2c55724d69aaf8be4f).

Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-02 17:49:45 +02:00
commit 4aa4f9c429
2 changed files with 218 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

86
src/portfolio_optimiser/explore.py
View file

@ -25,12 +25,20 @@ from __future__ import annotations
import json
import sys
from collections.abc import Callable, Mapping, Sequence
from collections.abc import Awaitable, Callable, Mapping, Sequence
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any, Final, Literal, TextIO
from agent_framework import Agent, BaseChatClient, FileCheckpointStorage, FunctionTool, tool
from agent_framework import (
Agent,
BaseChatClient,
FileCheckpointStorage,
FunctionInvocationContext,
FunctionMiddleware,
FunctionTool,
tool,
)
from agent_framework.orchestrations import (
MagenticBuilder,
MagenticOrchestratorEventType,
@ -253,6 +261,67 @@ class QuickValidation:
verdict: Mapping[str, Any]
@dataclass(frozen=True)
class ToolCall:
"""One exploration tool invocation: what was asked for, never what came back.
``bundle_id`` is the base the call named, and is ``""`` for a tool that takes none
(``list_bundles``). The RESULT is deliberately absent: it is the base's content, which is the
very thing that is too big to ride along (MAJOR-3 measured it at 73-93 % of all prompt tokens),
and a trace carrying it would be a second copy of the context rather than a record of the run.
"""
name: str
bundle_id: str
def _bundle_argument(arguments: Any) -> str:
"""The ``bundle_id`` a call named, from either shape ``FunctionInvocationContext`` allows.
``arguments`` is typed ``BaseModel | Mapping[str, Any]`` (measured against the installed
signature), so both are read rather than one being assumed. A tool without the parameter — or
a value that is not a string — yields ``""``: the recorder describes the call, and inventing a
label for a base that was never named would be the false-attribution that ``ToolCallRecorder``
refuses for unconfigured tools.
"""
if isinstance(arguments, Mapping):
value: Any = arguments.get("bundle_id")
else:
value = getattr(arguments, "bundle_id", None)
return value if isinstance(value, str) else ""
class ExplorationToolRecorder(FunctionMiddleware):
"""Records WHICH exploration tool an agent actually called, in the order it called them.
**A sibling of ``mcp_tools.ToolCallRecorder``, never a reuse of it** — the same shape, one
layer over: that one observes the debate's EXTERNAL tool calls, this one the exploration's
IN-PROCESS ones. Their invariants are opposites and generalising one to serve both would break
the other. ``ToolCallRecorder`` filters to configured servers, de-duplicates per
``(server, tool)`` and returns SORTED, because its record is an egress claim stamped into a
byte-deterministic artefact. This one keeps every call in INVOCATION ORDER without dedup,
because the question it answers is the opposite one: did the navigator open anything, and in
what sequence. A sorted, de-duplicated set cannot tell a rehearsal that read a base from one
that only listed them, which is the whole of MAJOR-1.
It observes only — ``call_next`` is always awaited, and nothing here can block, alter or
short-circuit an invocation. A trace that changed the run it traces would not be a trace.
"""
def __init__(self, sink: list[ToolCall]) -> None:
self._sink = sink
async def process(
self, context: FunctionInvocationContext, call_next: Callable[[], Awaitable[None]]
) -> None:
name = getattr(getattr(context, "function", None), "name", None)
if isinstance(name, str):
self._sink.append(
ToolCall(name=name, bundle_id=_bundle_argument(getattr(context, "arguments", None)))
)
await call_next()
@dataclass
class ExplorationTrace:
"""The CALLER-owned accumulator for everything one exploration produced along the way.
@ -274,6 +343,12 @@ class ExplorationTrace:
ledger: list[LedgerEntry] = field(default_factory=list)
plan_reviews: list[PlanReview] = field(default_factory=list)
quick_validations: list[QuickValidation] = field(default_factory=list)
#: Every exploration tool call, in order (MAJOR-1). It sits BESIDE ``quick_validations``
#: rather than inside it: that list is the level-1 VERDICTS the hypothesiser saw, this is
#: whether any base was opened at all. An offline rehearsal that called nothing looks exactly
#: like a successful one on every other field, which is what made the dress rehearsal vacuous
#: by construction and unreadable after the fact.
tool_calls: list[ToolCall] = field(default_factory=list)
#: Tokens spent so far, refreshed as the loop turns rather than written once at the end. The
#: meter is internal to ``explore``, so this is the only way the artefact can report a spend —
#: and updating it per iteration is what makes it readable for a run a cap cut short, which is
@ -325,6 +400,9 @@ def trace_payload(trace: ExplorationTrace, *, stop: str | None, completed: bool)
}
for call in trace.quick_validations
],
"tool_calls": [
{"name": call.name, "bundle_id": call.bundle_id} for call in trace.tool_calls
],
}
@ -1285,7 +1363,7 @@ async def explore(
client_factory,
contract=contract,
bundle_dirs=bundle_dirs,
middleware=[BudgetMiddleware(meter)],
middleware=[BudgetMiddleware(meter), ExplorationToolRecorder(trace.tool_calls)],
quick_validate_sink=trace.quick_validations,
checkpoint_dir=checkpoint_dir,
)
@ -1565,7 +1643,7 @@ async def resume_exploration(
client_factory,
contract=parked.contract,
bundle_dirs=parked.bundle_dirs,
middleware=[BudgetMiddleware(meter)],
middleware=[BudgetMiddleware(meter), ExplorationToolRecorder(trace.tool_calls)],
quick_validate_sink=trace.quick_validations,
checkpoint_dir=checkpoint_dir,
)

136
tests/test_scripted_explore_door_loadbearing.py
View file

@ -11,6 +11,30 @@ docstring warns against, mid-run, after the banner had already printed.
The fix widens the required-role set to include the exploration's three roles WHEN ``--explore``
is in play, so a missing role is refused BY NAME before any model/agent work starts — the same
door, never a second one.
MAJOR-1 (``docs/2026-09-02-misjonsreview-v2.md`` § 7) — and then the door that no longer crashed
turned out to be VACUOUS BY CONSTRUCTION, with an artefact that could not see it.
One constant string per role means no scripted role can ever emit a ``function_call``: measured,
0 tool calls / 0 approaches / 1 round on 4/4 bases, while every other field of
``{run_id}-exploration.json`` looked like a run that had worked. The third test above says as much
in its own docstring and asserts only the absence of a traceback — which was honest, and is
exactly the ceiling this section raises. An operator following this repo's measurement ladder
("prove as much as possible for free before the paid step") could not prove the navigator opens
anything, and after a PAID run nobody could read whether it had.
Two halves, each asserted so the OTHER cannot carry it:
(a) ``ExplorationToolRecorder`` — a ``FunctionMiddleware`` on the exploration agents recording the
tool NAME and the ``bundle_id`` it was asked for (never the result), in invocation order, onto
the caller-owned ``ExplorationTrace``; ``trace_payload`` writes it beside ``quick_validations``.
(b) ``_load_scripted_replies`` accepts, for the exploration roles, a LIST of steps where a step may
be ``{"call": "<tool>", "args": {...}}``, and ``scripted_factory`` then builds a client that
emits a ``function_call``.
Without (a) the record is empty though the tools ran; without (b) the tools never run though the
recorder is wired. The end-to-end test is RED against either detached, and the constant-string
control proves the recorder does not invent entries.
"""
from __future__ import annotations
@ -19,6 +43,7 @@ import json
from pathlib import Path
from typing import Any
from portfolio_optimiser import explore as ex
from portfolio_optimiser import run
_BUNDLE_DIR = Path(__file__).resolve().parents[1] / "shared" / "examples" / "bygg-energi-mikro"
@ -145,3 +170,114 @@ def test_explore_with_the_full_five_role_scripted_replies_does_not_crash(tmp_pat
assert "KeyError" not in err
assert "Traceback" not in err
assert rc in (0, 1), f"expected a clean exit, got rc={rc} stderr={err!r}"
# ---------------------------------------------------------------------------------------------
# MAJOR-1 (a) — THE RECORDER: name + the base asked for, in order, never the result
# ---------------------------------------------------------------------------------------------
class _FakeTool:
def __init__(self, name: str) -> None:
self.name = name
class _FakeContext:
"""The two attributes the middleware reads, and nothing else.
Invoked DIRECTLY rather than through a driven run, for the reason
``test_explore_loadbearing``'s tool-freedom test calls the tool bodies directly: until half
(b) exists no scripted client emits a tool call, so a middleware test routed through one would
exercise nothing — the vacuity this section is about, rebuilt inside its own gate.
"""
def __init__(self, name: str, arguments: Any) -> None:
self.function = _FakeTool(name)
self.arguments = arguments
async def _invoke(recorder: "ex.ExplorationToolRecorder", context: Any) -> bool:
called = False
async def _next() -> None:
nonlocal called
called = True
await recorder.process(context, _next)
return called
def test_the_recorder_keeps_every_call_in_order_with_the_base_it_named() -> None:
"""Order and repetition ARE the signal. ``mcp_tools.ToolCallRecorder``'s sorted, de-duplicated
set answers "what was contacted" because its record is an egress claim; this one must answer
"what was opened, and in what sequence" — a set cannot tell a rehearsal that read a base from
one that only listed them, which is the whole of MAJOR-1.
Detach point: sort or de-duplicate the sink → RED on both the order and the repeat.
"""
import asyncio
sink: list[ex.ToolCall] = []
recorder = ex.ExplorationToolRecorder(sink)
asyncio.run(_invoke(recorder, _FakeContext("read_bundle", {"bundle_id": "b2"})))
asyncio.run(_invoke(recorder, _FakeContext("list_bundles", {})))
asyncio.run(_invoke(recorder, _FakeContext("read_bundle", {"bundle_id": "b2"})))
assert [(c.name, c.bundle_id) for c in sink] == [
("read_bundle", "b2"),
("list_bundles", ""),
("read_bundle", "b2"),
], (
"invocation order and repeats are the record; a sorted, de-duplicated one answers something else"
)
def test_a_call_is_recorded_but_never_altered_or_blocked() -> None:
"""It observes only — ``call_next`` is always awaited. A trace that changed the run it traces
would not be a trace (the ``ToolCallRecorder`` rule, restated one layer up)."""
import asyncio
sink: list[ex.ToolCall] = []
assert (
asyncio.run(_invoke(ex.ExplorationToolRecorder(sink), _FakeContext("read_file", {})))
is True
)
assert len(sink) == 1
def test_the_base_argument_is_read_from_both_shapes_the_context_allows() -> None:
"""``FunctionInvocationContext.arguments`` is ``BaseModel | Mapping[str, Any]`` (measured
against the installed signature), so BOTH are read rather than one assumed. A tool that names
no base yields ``""`` — a label invented for a base nobody named is the false attribution
``ToolCallRecorder`` refuses for unconfigured tools.
"""
import asyncio
from types import SimpleNamespace
sink: list[ex.ToolCall] = []
recorder = ex.ExplorationToolRecorder(sink)
asyncio.run(
_invoke(recorder, _FakeContext("read_bundle", SimpleNamespace(bundle_id="modelled")))
)
asyncio.run(_invoke(recorder, _FakeContext("list_bundles", None)))
asyncio.run(_invoke(recorder, _FakeContext("read_bundle", {"bundle_id": 7})))
assert [c.bundle_id for c in sink] == ["modelled", "", ""]
def test_the_record_leaves_the_run_in_the_artefact_beside_the_advisory_verdicts() -> None:
"""A field no artefact carries is a field nobody can read after the run — MAJOR-1's second
half. ``trace_payload`` is the ONE rendering of a trace, so it is asserted there.
Detach point: drop the ``tool_calls`` key from ``trace_payload`` → RED here and end-to-end.
"""
trace = ex.ExplorationTrace()
trace.tool_calls.append(ex.ToolCall(name="read_bundle", bundle_id="bygg-energi-mikro"))
payload = ex.trace_payload(trace, stop=None, completed=True)
assert payload["tool_calls"] == [{"name": "read_bundle", "bundle_id": "bygg-energi-mikro"}], (
"the artefact must say which bases were opened, or a paid run leaves no record that any were"
)
assert json.dumps(payload), "the payload must stay plain data — the RAW layer is MAF-free"