open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 3 Activity

feat(explore): U4+U13 synkron - utforskningssloeyfa som mandat-former (ORDRE 20260823T185602Z) [skip-docs]

Browse source 
Magentic legges OVER den normative sloeyfa, aldri inni Steg 3: prompt +
kunnskapsbaser -> Mandate -> run_project(mandate=...) UENDRET. Manageren velger
VEI; det som forlater friheten er et Mandate, aldri et forslag. explore() skriver
ingenting - niva 3 (skriverettigheter) tilhoerer pipelinen alene.

Levert i denne oekten (kjernen; kallstedene staar til oekt 57):
- ExplorationContract: seks paakrevde felt uten default. max_reset_count=0 nektes
  paa en MAALING - reset_count >= max_reset_count mot en teller som starter paa 0
  terminerer kjoeringen FOER foerste runde med null ledger-events, altsaa en
  utforskning som utforsket ingenting, forkledd som en stall som aldri skjedde.
- explore() + fresh_exploration_workflow(): fersk builder per utforskning,
  BudgetMiddleware paa HVER agent inkl. manageren, synkron plan review via
  request_info, og max_plan_revisions som binder den ubundne revise-loekka.
- Tre kanaler: tokens OG runder raiser BudgetExceeded (rundene oversatt av vaart
  lag som kind="exploration_rounds", fordi orkestreringen maalt ikke raiser ved
  sitt eget rundetak); alt semantisk er en VERDI i stop.
- quick_validate (niva 1, raadgivende) + navigator-verktoey over safe_resolve.
- U14s tre utsatte events landet som span-events paa EN exploration-span.

Load-bearing MAALT mot HELE suiten, groenn kontroll 975/5, golden ea8c534
uendret: tolv mutasjoner alle roede. TO av dem falsifiserte testen foerst -
skrivefrihets-testen naadde aldri en verktoeykropp (ScriptedChatClient emitterer
ingen verktoeykall), og stdout-testens capsys er blind for ConsoleSpanExporter,
hvis out-default bindes ved modulimport. Begge er rettet; stdout-armen er naa en
subprosess, som er P4-presedensen.

[skip-docs] fordi flaten ikke er naabar for en bruker enna: --explore, det
whitelistede hosting-feltet og sim-scenarioet bygges i oekt 57, og en
README-oppfoering naa ville vaert en paastand om en inngang som ikke finnes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-08-23 22:39:00 +02:00
commit f0c54cc8dc
4 changed files with 1892 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

804
src/portfolio_optimiser/explore.py Normal file
View file

@ -0,0 +1,804 @@
"""U4 + U13-synchronous — the Magentic exploration loop, sitting OVER the normative pipeline.
**One sentence of design, and it is not to be reopened without a measurement.** Magentic is laid
over the eight-step loop, never inside Step 3: ``prompt + knowledge bases -> Mandate ->
run_project(mandate=...)``, with that last call byte-for-byte the one it already was. The manager
is free to choose which base to open and which hypothesis to shape next; what leaves that freedom
is a ``mandate.Mandate`` — approaches worth *testing*. The exploration's own final answer is RAW,
never a proposal, and ``explore()`` writes to neither the outbox nor the wiki. Every number that
survives is gated by ``validate_proposal`` inside ``run_project``, in the same blocking gate as
today, and ``shared/method-spec.md`` §3's maker-checker debate is untouched (it is commons-owned
and normative; replacing it would need an amendment, not a module).
**Three levels of guarantee, stated so nobody reads more into the loop than is there.** The
``quick_validate`` tool the hypothesiser calls in-loop is level 1: the SAME ``validate_proposal``,
against the SAME baseline, but its verdict is advisory — it never becomes provenance. Level 2 is
the pipeline, where each approach is validated and stamped. Level 3 is write access, which only
the pipeline has.
This module imports ``agent_framework.orchestrations`` and therefore may never be imported from
``okf.py`` / ``mandate.py`` / ``hitl.py``, which are held framework-neutral by
``tests/test_okf.py::test_okf_is_maf_free``.
"""
from __future__ import annotations
import json
from collections.abc import Callable, Mapping, Sequence
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Final, Literal
from agent_framework import Agent, BaseChatClient, FunctionTool, tool
from agent_framework.orchestrations import (
MagenticBuilder,
MagenticOrchestratorEventType,
MagenticPlanReviewResponse,
)
from pydantic import BaseModel, Field, ValidationError, model_validator
from portfolio_optimiser import okf
from portfolio_optimiser.backends import Profile
from portfolio_optimiser.budget import Budget, BudgetExceeded, BudgetMiddleware, TokenMeter
from portfolio_optimiser.ir import SavingsProposal
from portfolio_optimiser.mandate import OWN_PROPOSAL_ID, Approach, Mandate
from portfolio_optimiser.retrieval import safe_resolve
from portfolio_optimiser.tracing import exploration_tracer
from portfolio_optimiser.validator import Rejection, validate_proposal
class ExplorationContract(BaseModel):
"""The stated bounds of ONE exploration. Every field is required; none has a default.
Mirrors ``contracts.TerminationContract`` in spirit and goes further in one respect: there is
nothing to inherit. ``MagenticBuilder`` defaults ``max_round_count`` to ``None`` (unbounded)
and ``max_reset_count`` to ``None`` (unlimited), so a field left out here would not fall back
to something conservative — it would fall back to the one shape ``shared/method-spec.md`` §8
forbids outright, an unbounded loop. A default would also assert an intent nobody stated,
which is the ground ``ProvenanceStamp.cost_baseline_anchored`` is required on.
``max_plan_revisions`` earns its place by measurement, not symmetry (§ F, A3): a plan-review
``revise`` costs two manager calls, emits **no** progress ledger and consumes **no** round,
then asks again. Under the round cap alone an always-revising expert is an unbounded spend
that the round counter never sees. ``0`` is meaningful and allowed: the plan must be approved
as first written or the exploration stops.
"""
#: Hard cap on orchestration rounds (``MagenticBuilder(max_round_count=...)``).
max_rounds: int = Field(gt=0)
#: Hard cap on tokens, enforced by ``BudgetMiddleware`` on EVERY agent, manager included.
max_tokens: int = Field(gt=0)
#: Consecutive no-progress rounds tolerated before the manager resets and replans. ``0`` is
#: allowed and means "reset on the first round that reports no progress": the orchestrator's
#: check is STRICT (``stall_count > max_stall_count``, ``_magentic.py:1118``) and the counter is
#: incremented ahead of it, so zero is strictness rather than self-defeat.
max_stall_count: int = Field(ge=0)
#: Resets tolerated before the exploration is declared stalled. Must be POSITIVE, and that is a
#: MEASUREMENT: the limit check is ``reset_count >= max_reset_count`` (``:1243``) against a
#: counter starting at zero, so a cap of zero is already met before the first round. Measured
#: against the installed stack, ``max_reset_count=0`` produces only the ``facts`` and ``plan``
#: manager calls, **zero** progress-ledger events, and the canonical "maximum reset count"
#: termination — an exploration that explored nothing, dressed as a stall that never happened.
max_reset_count: int = Field(gt=0)
#: Plan revisions the reviewer may ask for before the exploration stops. See the class note.
max_plan_revisions: int = Field(ge=0)
#: Whether a human/persona signs the plan off before the loop runs (the U13 synchronous door).
enable_plan_review: bool
@model_validator(mode="after")
def _a_revision_cap_needs_a_review_to_cap(self) -> "ExplorationContract":
"""A revision cap with the review switched off bounds an event that cannot occur.
Refused rather than dropped: a caller who wrote ``max_plan_revisions=3`` believes they
bounded something, and silently ignoring it is the failure mode this repo's flag contract
exists to prevent (``--embedder-config requires --semantic-retrieval``, same rule). The
coherent way to say "no reviews" is ``enable_plan_review=False, max_plan_revisions=0``.
"""
if not self.enable_plan_review and self.max_plan_revisions:
raise ValueError(
f"max_plan_revisions={self.max_plan_revisions} bounds plan revisions, but "
"enable_plan_review is false so no plan review is ever requested and no revision "
"can occur; set max_plan_revisions=0 or enable the review"
)
return self
#: The manager's role name. Not a participant: the manager plans, picks the next speaker and
#: keeps the progress ledger, and ``BudgetMiddleware`` rides on it like on everyone else (A1,
#: measured green — without that, the most talkative agent in the loop would be the one outside
#: the token cap).
MANAGER_ROLE: Final = "manager"
#: Reads the knowledge bases PROGRESSIVELY (§3 Steg 1) — index, then whatever the index links.
NAVIGATOR_ROLE: Final = "navigator"
#: Shapes ONE candidate direction at a time and may have its numbers advisory-checked in-loop.
HYPOTHESISER_ROLE: Final = "hypothesiser"
#: The participants, in build order. The manager's ``next_speaker`` is validated against exactly
#: these names: a ledger naming anyone else makes the orchestrator emit a final answer having
#: asked NOBODY (measured, ``_magentic.py:1128-1131``), which is a plausible answer produced by
#: zero work — the hazard class this repo retired ``E2`` for.
PARTICIPANT_ROLES: Final = (NAVIGATOR_ROLE, HYPOTHESISER_ROLE)
#: The span every exploration records its decisions under. One span per exploration, with the
#: manager's decisions as span EVENTS on it, because those are moments INSIDE one activity rather
#: than activities of their own — and because a reader wants them in order, on one timeline.
EXPLORATION_SPAN: Final = "exploration"
#: The line prefix that turns a hypothesiser turn into a candidate direction. A marker rather than
#: "parse every turn as JSON" because most turns legitimately are not hypotheses — the agent also
#: reasons out loud. That distinction is what lets an UNPARSEABLE marked line be a hard error
#: instead of a silence: with no marker there is nothing to be silent about.
HYPOTHESIS_MARKER: Final = "HYPOTHESIS:"
_INSTRUCTIONS: Final = {
NAVIGATOR_ROLE: (
"You read the project's knowledge bases. Use list_bundles to see what exists, then "
"read_bundle to open ONE at a time and read_file to follow a specific document. Quote "
"what you found; never guess at content you have not read."
),
HYPOTHESISER_ROLE: (
"You shape ONE candidate cost-saving direction at a time from what the navigator found. "
"You may call quick_validate to sanity-check a candidate's numbers; its verdict is "
"ADVISORY and is not the project's decision. When you commit to a direction, end your "
f'turn with a line of the form: {HYPOTHESIS_MARKER} {{"label": "<short name>", '
'"rationale": "<why this project, in your own words>"}'
),
}
class ExplorationError(RuntimeError):
"""The exploration cannot be honoured as configured, or produced something unreadable."""
@dataclass(frozen=True)
class LedgerEntry:
"""One progress-ledger round, reduced to the five fields the manager steers on (C.2).
``speaker_known`` is not part of the ledger MAF produces — it is this layer's reading of it,
and the reason the whole entry is recorded rather than counted. A ``next_speaker`` matching no
participant does not raise: the orchestrator logs a warning and jumps straight to a final
answer, so the run returns a plausible result that nobody worked for. Recording the fact is
what lets ``explore()`` refuse to hand that result onward as if it had been explored.
"""
round_index: int
is_request_satisfied: bool
is_in_loop: bool
is_progress_being_made: bool
next_speaker: str
instruction_or_question: str
speaker_known: bool
@dataclass(frozen=True)
class PlanReview:
"""One synchronous plan-review round trip (U13, door 2 of § C.6).
``is_stalled`` is carried because the same request type serves two very different moments:
the initial sign-off, and a re-review after the manager reset and replanned. A reviewer that
cannot tell them apart cannot answer the second one usefully.
"""
index: int
plan: str
is_stalled: bool
decision: Literal["approve", "revise"]
feedback: str = ""
#: Why an exploration ended without a satisfied request. ``None`` means it concluded normally.
#: Resource exhaustion is NOT here: tokens and rounds raise ``BudgetExceeded`` (the 429 channel),
#: because "we ran out" and "we finished, unsatisfied" are the two things S3.4 split apart and a
#: single field would fuse back together.
ExplorationStop = Literal["stalled", "plan_revisions_exhausted", "unknown_speaker"]
@dataclass(frozen=True)
class ExplorationResult:
"""What one exploration produced. The mandate is the product; the rest is the evidence.
``mandate`` is ALWAYS present, including on a stop: an exploration that was cut short still
carries the expert's seed approaches forward, because door 1 of § C.6 is a preservation rule
and not a reward for finishing. ``stop`` is what says the mandate is smaller than it might
have been, and why.
"""
mandate: Mandate
ledger_log: tuple[LedgerEntry, ...]
stop: ExplorationStop | None
plan_reviews: tuple[PlanReview, ...]
@dataclass(frozen=True)
class PlanReviewRequest:
"""What the reviewer is shown before the loop is allowed to run."""
index: int
plan: str
current_progress: str
is_stalled: bool
@dataclass(frozen=True)
class PlanReviewDecision:
"""The reviewer's answer. ``feedback is None`` means approve.
Two named constructors mirroring ``MagenticPlanReviewResponse.approve()/.revise()``, so a
persona or operator answering a review never has to import ``agent_framework`` — the adapter
is one-directional and lives in exactly one place.
"""
feedback: str | None
@staticmethod
def approve() -> PlanReviewDecision:
return PlanReviewDecision(feedback=None)
@staticmethod
def revise(feedback: str) -> PlanReviewDecision:
if not feedback.strip():
raise ValueError("a revision must say what to revise; use approve() to sign off")
return PlanReviewDecision(feedback=feedback)
#: The synchronous HITL seam (U13). Given the request, answer it. Called in-process, so the
#: exploration blocks on it exactly as a human at a terminal would block the loop.
PlanReviewer = Callable[[PlanReviewRequest], PlanReviewDecision]
# ---------------------------------------------------------------------------------------------
# The tools. Level 1 of the three-guarantee table: real computation, ADVISORY verdicts.
# ---------------------------------------------------------------------------------------------
def _bundle_index(bundle_dirs: Sequence[str]) -> dict[str, str]:
"""Map each knowledge base's id to its directory. The id is the directory's BASENAME.
A duplicate basename is REFUSED rather than resolved by order: the id is what the manager
names a base by, and two bases answering to one name would let it read A while believing it
read B — the S3.2 key-collision class, one layer up.
"""
index: dict[str, str] = {}
for raw in bundle_dirs:
bundle_id = Path(raw).name
if bundle_id in index:
raise ExplorationError(
f"two knowledge bases share the id {bundle_id!r} ({index[bundle_id]!r} and "
f"{raw!r}); the manager names a base by that id, so it must be unique"
)
index[bundle_id] = raw
return index
def _resolve_bundle(index: Mapping[str, str], bundle_id: str) -> str:
if bundle_id not in index:
known = ", ".join(sorted(index)) or "(none configured)"
raise ExplorationError(f"unknown knowledge base {bundle_id!r}; configured: {known}")
return index[bundle_id]
def navigator_tools(bundle_dirs: Sequence[str]) -> list[FunctionTool]:
"""The navigator's three tools: survey the catalogue, open one base, read one document.
Progressive disclosure, not stuffing (målbilde §2/§4): ``list_bundles`` never returns content,
only what each base IS — and crucially whether it ships a ``cost-baseline.json``, because a base
without one cannot have its hypotheses reconciled against the project's real cost lines, and a
manager that does not know which bases are anchored cannot plan around it (§ C.7).
``read_bundle`` returns ``okf.bundle_context``, which EXCLUDES the ``type: verdict`` layer by
construction — prior verdicts reach a hypothesis only through the gated ExpeL fold inside
``run_project``, never by being read as context here.
"""
index = _bundle_index(bundle_dirs)
@tool(
name="list_bundles",
description=(
"List the knowledge bases available to this exploration: id, what the index says the "
"base is about, how many prior expert verdicts it holds, and whether it ships a cost "
"baseline (without one, numbers cannot be reconciled against the project's own)."
),
)
def list_bundles() -> list[dict[str, Any]]:
catalogue: list[dict[str, Any]] = []
for bundle_id, bundle_dir in index.items():
bundle = okf.navigate_bundle(bundle_dir)
catalogue.append(
{
"id": bundle_id,
"index_summary": bundle.index_summary,
"verdict_count": len(bundle.verdicts),
# Tolerant on CONTENT, fail-fast on the PATH: an operator's bad directory is
# refused by navigate_bundle above, while a navigable base that simply has no
# baseline is legitimate (load_optional_cost_baseline's own contract).
"cost_baseline": okf.load_optional_cost_baseline(bundle_dir) is not None,
"skipped_links": [
{"from_file": s.from_file, "target": s.target, "reason": s.reason}
for s in bundle.skipped
],
}
)
return catalogue
@tool(
name="read_bundle",
description="Open ONE knowledge base by id and read its navigated context.",
)
def read_bundle(bundle_id: str) -> str:
return okf.bundle_context(okf.navigate_bundle(_resolve_bundle(index, bundle_id)))
@tool(
name="read_file",
description="Read ONE document inside a knowledge base, by base id and relative path.",
)
def read_file(bundle_id: str, path: str) -> str:
bundle_dir = _resolve_bundle(index, bundle_id)
# safe_resolve is the ONE in-/out-of-bundle test in this repo, and it is fail-closed. A
# model-chosen path is untrusted input by definition, so it goes through the same gate the
# navigation walk uses rather than a second, laxer check.
return Path(safe_resolve(bundle_dir, path)).read_text(encoding="utf-8")
return [list_bundles, read_bundle, read_file]
def quick_validate_tool(bundle_dirs: Sequence[str]) -> FunctionTool:
"""The hypothesiser's in-loop deterministic check — level 1, and advisory by construction.
It is the SAME ``validate_proposal`` against the SAME baseline the pipeline will use, so the
numbers it reports are real; what it never becomes is provenance. ``ProvenanceStamp`` is
written by ``run_project`` alone, and nothing here reaches it.
Measured cheap (S5: 13.6 ms median through stage 0 + the CBC solve + a 512-sample Monte Carlo,
on an anchored base with assumption bands), so it may be called freely in the loop and the
contract carries no latency budget.
``anchored`` is reported alongside the verdict for the same reason
``ProvenanceStamp.cost_baseline_anchored`` is a required field: a verdict reached without the
project's own cost lines is a weaker claim, and one that does not say so is the silence S4.0's
visibility work closed.
"""
index = _bundle_index(bundle_dirs)
@tool(
name="quick_validate",
description=(
"Run the deterministic validator over a candidate, as an ADVISORY check. Pass the "
"knowledge base id and the candidate as IR JSON. The verdict is not the project's "
"decision — the pipeline re-validates and stamps."
),
)
def quick_validate(bundle_id: str, proposal_json: str) -> dict[str, Any]:
bundle_dir = _resolve_bundle(index, bundle_id)
baseline = okf.load_optional_cost_baseline(bundle_dir)
try:
proposal = SavingsProposal.model_validate_json(proposal_json)
except ValidationError as exc:
return {"decision": "unparseable", "reason": str(exc), "anchored": baseline is not None}
outcome = validate_proposal(proposal, baseline=baseline)
if isinstance(outcome, Rejection):
return {
"decision": "rejected",
"reason": outcome.reason,
"anchored": baseline is not None,
}
return {
"decision": "validated",
"reason": "",
"anchored": baseline is not None,
"p10": outcome.p10,
"p50": outcome.p50,
"p90": outcome.p90,
}
return quick_validate
# ---------------------------------------------------------------------------------------------
# The workflow. One builder, one build, ONE run per exploration (B7 / § C.4).
# ---------------------------------------------------------------------------------------------
def fresh_exploration_workflow(
client_factory: Callable[[str], BaseChatClient],
*,
contract: ExplorationContract,
bundle_dirs: Sequence[str] = (),
middleware: Sequence[Any] | None = None,
) -> Any:
"""Build a FRESH Magentic workflow with fresh agents and fresh clients (mirrors
``workflow.fresh_workflow``).
**Fresh builder per exploration, never a reused one.** A built Magentic workflow is
single-use: a second ``run()`` on it raises ``RuntimeError`` having made zero model calls
(measured, E1) — which is a safe failure, unlike GroupChat 1.9.0's silently empty second run.
Reuse is prevented here rather than relied upon to fail.
``manager_agent_factory=`` rather than ``manager_agent=``: the eager form constructs the
manager once and hands the same instance to every ``build()`` (``:1683``, ``:1729-1730``),
which on orchestrations 1.0.0 leaked one exploration's task into the next. Orchestrations
1.0.1 removed the manager's persistent session, so that leak is GONE and **no test here can
tell the two forms apart today** (measured, § F A8: 4/4 → 0/5). The factory form is used
anyway because it costs nothing and does not depend on an upstream regression fix staying
fixed — but it is stated plainly rather than gated, because a gate that cannot go red proves
nothing.
Every agent carries the SAME ``middleware`` list, manager included. That is the whole of the
token guarantee: agent-level ``ChatMiddleware`` does fire on the manager's own calls (measured
A1), and the manager is the most talkative participant in the loop.
"""
hypothesiser_tools: list[Any] = [quick_validate_tool(bundle_dirs)]
tools_by_role: dict[str, list[Any]] = {
NAVIGATOR_ROLE: list(navigator_tools(bundle_dirs)),
HYPOTHESISER_ROLE: hypothesiser_tools,
}
participants = [
Agent(
client_factory(role),
_INSTRUCTIONS[role],
name=role,
description=_INSTRUCTIONS[role],
tools=tools_by_role[role],
middleware=middleware,
)
for role in PARTICIPANT_ROLES
]
def _manager_agent() -> Agent:
return Agent(
client_factory(MANAGER_ROLE),
"You plan and coordinate an exploration of a project's knowledge bases to find "
"cost-saving directions worth testing. You never state a saving figure yourself.",
name=MANAGER_ROLE,
description="plans the exploration",
middleware=middleware,
)
return MagenticBuilder(
participants=participants,
manager_agent_factory=_manager_agent,
max_round_count=contract.max_rounds,
max_stall_count=contract.max_stall_count,
max_reset_count=contract.max_reset_count,
enable_plan_review=contract.enable_plan_review,
).build()
def _truthy(answer: Any) -> bool:
"""Read a ledger answer exactly as the orchestrator does.
``MagenticProgressLedgerItem.answer`` is ``str | bool`` and is NOT normalised per field
(``:292``); the orchestrator then tests it with plain Python truthiness (``:1106``, ``:1112``).
A smarter coercion here — treating the string ``"false"`` as false, say — would describe a run
the orchestrator did not have. This layer reports what the loop DID, so it copies the loop's
own rule rather than improving on it.
"""
return bool(answer)
def _plan_text(content: Any) -> str:
"""The task-ledger plan as text. ``PLAN_CREATED``/``REPLANNED`` carry a ``Message``."""
return str(getattr(content, "text", "") or "")
def _absorb(
result: Any,
*,
ledger_log: list[LedgerEntry],
hypotheses: list[str],
seen: set[int],
span: Any,
) -> int:
"""Fold one ``run()``'s events into the running log and onto the trace. Returns this call's
replans.
``seen`` de-duplicates by object identity because MAF may surface the same payload under more
than one event type; the alternative — pinning one event-type name — would silently under- or
over-count if that changed. Participant output is matched on ``executor_id``, which is the
agent's name, exactly as ``run._authored_texts`` matches the debate's ``author_name``.
**The three span events land here** (U14's deferred half, now that it has a call site). They
carry the manager's DECISIONS — which plan, who was asked, whether the request was satisfied —
as separate typed attributes rather than a rendered sentence, so a collector can query them.
Stated honesty limit: they are recorded as the event stream is folded, i.e. after ``run()``
returns, so their ORDER is faithful and their timestamps are not the moments the manager acted.
Emitting live would mean driving the loop through ``run_stream``, which is a different
measurement from the one the plan-review round trip was proved against.
"""
replans = 0
for event in result:
data = getattr(event, "data", None)
if data is None or id(data) in seen:
continue
event_type = getattr(data, "event_type", None)
if event_type is not None:
seen.add(id(data))
if event_type == MagenticOrchestratorEventType.PLAN_CREATED:
span.add_event("plan_created", {"plan": _plan_text(data.content)})
elif event_type == MagenticOrchestratorEventType.REPLANNED:
replans += 1
span.add_event("replanned", {"plan": _plan_text(data.content)})
elif event_type == MagenticOrchestratorEventType.PROGRESS_LEDGER_UPDATED:
ledger = data.content
speaker = str(ledger.next_speaker.answer)
entry = LedgerEntry(
round_index=len(ledger_log) + 1,
is_request_satisfied=_truthy(ledger.is_request_satisfied.answer),
is_in_loop=_truthy(ledger.is_in_loop.answer),
is_progress_being_made=_truthy(ledger.is_progress_being_made.answer),
next_speaker=speaker,
instruction_or_question=str(ledger.instruction_or_question.answer),
speaker_known=speaker in PARTICIPANT_ROLES,
)
ledger_log.append(entry)
span.add_event(
"progress_ledger_updated",
{
"round_index": entry.round_index,
"is_request_satisfied": entry.is_request_satisfied,
"is_in_loop": entry.is_in_loop,
"is_progress_being_made": entry.is_progress_being_made,
"next_speaker": entry.next_speaker,
"speaker_known": entry.speaker_known,
"instruction_or_question": entry.instruction_or_question,
},
)
continue
if getattr(data, "executor_id", None) == HYPOTHESISER_ROLE:
response = getattr(data, "agent_response", None)
text = getattr(response, "text", None)
if text:
seen.add(id(data))
hypotheses.append(text)
return replans
class HypothesisParseError(ExplorationError):
"""A hypothesiser turn carried the marker and then something unreadable.
Fail-closed, and the marker is what makes that affordable: most turns legitimately are not
hypotheses, so an unmarked turn is not a failure and there is nothing to be silent about. A
MARKED line that will not parse is a claim the loop tried to make and could not — raising it
is the ``write_concept_file`` rule (validation, never repair) rather than the tolerant RAW
inbox rule, because this is the product of the run, not a folder anyone may drop things in.
"""
def _parse_hypotheses(texts: Sequence[str]) -> list[tuple[str, str]]:
"""Every marked ``(label, rationale)`` pair the hypothesiser committed to, in turn order."""
found: list[tuple[str, str]] = []
for text in texts:
for line in text.splitlines():
stripped = line.strip()
if not stripped.startswith(HYPOTHESIS_MARKER):
continue
payload = stripped[len(HYPOTHESIS_MARKER) :].strip()
try:
data = json.loads(payload)
except json.JSONDecodeError as exc:
raise HypothesisParseError(
f"hypothesiser marked a line as a hypothesis and it is not JSON: {exc}; "
f"line was: {stripped}"
) from exc
if not isinstance(data, dict) or not data.get("label") or not data.get("rationale"):
raise HypothesisParseError(
"a marked hypothesis needs a non-empty 'label' and 'rationale'; got: "
f"{stripped}"
)
found.append((str(data["label"]), str(data["rationale"])))
return found
def _mint_approaches(
seeds: Sequence[Approach], discovered: Sequence[tuple[str, str]]
) -> tuple[Approach, ...]:
"""Seeds FIRST, untouched, then one approach per discovered direction.
Seeds lead because the pipeline evaluates approaches in order under one shared budget: a
direction the domain expert asked for must be reached before the loop's own findings spend
what is left. The minted ids skip anything a seed already claims, so a seed literally named
``hypothesis-1`` cannot collide — ``Mandate`` refuses duplicate ids at construction, and
turning an expert's naming choice into a hard failure would be a refusal that teaches nothing.
"""
taken = {approach.id for approach in seeds}
minted: list[Approach] = list(seeds)
counter = 0
for label, rationale in discovered:
counter += 1
candidate = f"hypothesis-{counter}"
while candidate in taken or candidate == OWN_PROPOSAL_ID:
counter += 1
candidate = f"hypothesis-{counter}"
taken.add(candidate)
# description is the hypothesiser's own words, VERBATIM: ``generate._build_messages`` feeds
# an Approach.description to the proposer unchanged, and the reason a direction is worth
# trying is exactly the half a model cannot re-derive from the cost table.
minted.append(Approach(id=candidate, label=label, description=rationale))
return tuple(minted)
def _pending_plan_reviews(result: Any) -> list[Any]:
return [event for event in result if event.type == "request_info"]
async def explore(
prompt: str,
*,
contract: ExplorationContract,
bundle_dirs: Sequence[str] = (),
profile: Profile | str = Profile.LOCAL,
client_factory: Callable[[str], BaseChatClient] | None = None,
seed_approaches: Sequence[Approach] = (),
plan_reviewer: PlanReviewer | None = None,
meter: TokenMeter | None = None,
success_criteria: str = "",
) -> ExplorationResult:
"""Explore the knowledge bases and return the ``Mandate`` the pipeline should evaluate.
Writes NOTHING. No outbox artefact, no wiki promotion, no verdict — level 3 of the guarantee
table belongs to ``run_project`` alone, and an exploration that could write would be a route
around the gate that makes an answer checkable.
**Three stops, three channels, deliberately not one.** Tokens raise ``BudgetExceeded`` exactly
as the debate does, so the hosted surface's 429 arm needs no new case. Rounds raise it too,
with ``kind="exploration_rounds"`` — the orchestration itself raises nothing at its round cap
(measured, § F): it returns a canonical assistant message that is indistinguishable from
success at the transport, so this layer produces the typed stop. Everything semantic —
stalling out, an exhausted revision cap, a ledger naming nobody — is a VALUE in ``stop``,
because those are outcomes of the exploration rather than exhaustion of a resource, and S3.4
split those two apart for a reason.
``seed_approaches`` are the domain expert's own hypotheses (door 1 of § C.6). They are in the
returned mandate whatever the loop found, including when the loop found nothing and including
when it stopped early.
Honesty limits, stated rather than implied. (1) A run is bound to ONE mandate; multi-base
dispatch (``Approach.bundle_id``, § C.7) waits for ``run_project`` to accept more than one
``bundle_dir``, and shipping the field before its consumer would be a shape guessed instead of
measured. (2) The ``quick_validate`` verdicts the hypothesiser saw are not in
``ExplorationResult``: they are level-1 advisory, and their home is the
``{run_id}-exploration.json`` artefact the CLI wiring writes. (3) The exploration roles resolve
through ``resolve_model``'s ``default`` fallback unless an operator maps them explicitly.
"""
if contract.enable_plan_review and plan_reviewer is None:
raise ExplorationError(
"enable_plan_review is set but no plan_reviewer was given: the exploration would stop "
"at a review nobody can answer, which is a hang rather than a result"
)
if plan_reviewer is not None and not contract.enable_plan_review:
raise ExplorationError(
"a plan_reviewer was given but enable_plan_review is false, so no review is ever "
"requested and the reviewer would never be called (refused, never silently ignored)"
)
if meter is None:
meter = TokenMeter(Budget(max_tokens=contract.max_tokens, max_rounds=contract.max_rounds))
if client_factory is None:
# Imported HERE, not at module scope: ``run`` imports this module for its ``--explore``
# door, so a top-level import would be circular. One copy of the backend factory, never a
# second (the (p) rule) — the same move ``run.main`` makes for ``scripted_factory``.
from portfolio_optimiser.run import _default_factory
client_factory = _default_factory(profile)
workflow = fresh_exploration_workflow(
client_factory,
contract=contract,
bundle_dirs=bundle_dirs,
middleware=[BudgetMiddleware(meter)],
)
ledger_log: list[LedgerEntry] = []
hypothesis_texts: list[str] = []
plan_reviews: list[PlanReview] = []
seen: set[int] = set()
replans = 0
stop: ExplorationStop | None = None
# ONE span for the whole exploration, opened before the first model call and closed however
# the loop ends — including on a BudgetExceeded, which the span records rather than swallows.
# With no provider installed this is a no-op tracer and every event is discarded, which is
# exactly what "tracing is off" has meant since U14.
with exploration_tracer().start_as_current_span(EXPLORATION_SPAN) as span:
result = await workflow.run(prompt)
while True:
replans += _absorb(
result,
ledger_log=ledger_log,
hypotheses=hypothesis_texts,
seen=seen,
span=span,
)
pending = _pending_plan_reviews(result)
if not pending:
break
assert plan_reviewer is not None # guarded above; the review implies a reviewer
request = pending[0]
review = request.data
decision = plan_reviewer(
PlanReviewRequest(
index=len(plan_reviews),
plan=str(review.plan),
current_progress=str(review.current_progress),
is_stalled=_truthy(review.is_stalled),
)
)
if decision.feedback is None:
plan_reviews.append(
PlanReview(
index=len(plan_reviews),
plan=str(review.plan),
is_stalled=_truthy(review.is_stalled),
decision="approve",
)
)
response = MagenticPlanReviewResponse.approve()
else:
# The revision is recorded whether or not it is APPLIED: ``plan_reviews`` is the
# record of what the reviewer DECIDED, and ``stop`` is what says the last one was
# refused.
applied = sum(1 for entry in plan_reviews if entry.decision == "revise")
plan_reviews.append(
PlanReview(
index=len(plan_reviews),
plan=str(review.plan),
is_stalled=_truthy(review.is_stalled),
decision="revise",
feedback=decision.feedback,
)
)
if applied >= contract.max_plan_revisions:
# Measured (§ F, A3): a revise costs two manager calls, emits no progress
# ledger and consumes no round, then asks AGAIN. Under the round cap alone this
# loop never terminates. Stopping is the honest move — forcing an approve the
# reviewer did not give would be repair, and repair of a human's decision most
# of all.
stop = "plan_revisions_exhausted"
break
response = MagenticPlanReviewResponse.revise(decision.feedback)
result = await workflow.run(responses={request.request_id: response})
if stop is None:
stop = _classify_stop(ledger_log, replans=replans, contract=contract)
discovered = _parse_hypotheses(hypothesis_texts) if stop != "unknown_speaker" else []
return ExplorationResult(
mandate=Mandate(
objective=prompt,
approaches=_mint_approaches(seed_approaches, discovered),
allow_own_proposals=True,
success_criteria=success_criteria,
),
ledger_log=tuple(ledger_log),
stop=stop,
plan_reviews=tuple(plan_reviews),
)
def _classify_stop(
ledger_log: Sequence[LedgerEntry], *, replans: int, contract: ExplorationContract
) -> ExplorationStop | None:
"""Decide what a finished ``run()`` actually was. Structural, never prose-matched.
The orchestration ends four ways and reports three of them with an ordinary-looking result:
a satisfied request, the round cap, the reset cap, and a ``next_speaker`` matching nobody.
Only the first is success. The discriminators used here are counts and ledger flags — the
termination MESSAGE is deliberately not matched, because it is an f-string built inline
(``:1253``) with no constant to pin against, and a model's own final answer could contain the
same words.
A ledger naming nobody is checked FIRST and outranks everything else. It is the only ending
in which the orchestrator produced an answer having asked no participant at all (``:1128``),
so the run has a result that no work stands behind — the same class as the retired E2 finding,
and not something a later, softer verdict should be allowed to paper over.
"""
if any(not entry.speaker_known for entry in ledger_log):
return "unknown_speaker"
if ledger_log and ledger_log[-1].is_request_satisfied:
return None
if len(ledger_log) >= contract.max_rounds:
raise BudgetExceeded("exploration_rounds", contract.max_rounds, len(ledger_log))
# Whatever is left ended without satisfying the request and without exhausting the round cap:
# the manager reset until it ran out of resets, or terminated with nothing to show. Both are
# the same fact for a caller — the exploration gave up — so they share one token rather than
# inventing a distinction the ledger cannot support.
return "stalled"

38
src/portfolio_optimiser/tracing.py
View file

@ -37,10 +37,14 @@ cause.
(``opentelemetry-exporter-otlp-proto-grpc`` / ``-http``) are not declared dependencies. They are
egress, they drag grpc and protobuf into a published wheel for a mode that is off by default, and
MAF already raises an ``ImportError`` that names the package to install. Stated honesty limit:
``PORTFOLIO_OTEL=otlp`` works only after the operator installs one of them. Equally absent are the
``PLAN_CREATED`` / ``REPLANNED`` / ``PROGRESS_LEDGER_UPDATED`` events the plan names — they belong
to the exploration loop (U4), which does not exist yet, and an emitter written before its call site
is a shape guessed rather than measured.
``PORTFOLIO_OTEL=otlp`` works only after the operator installs one of them.
**The ``PLAN_CREATED`` / ``REPLANNED`` / ``PROGRESS_LEDGER_UPDATED`` events now exist** (U4, økt
56). They were held back here in økt 55 on the ground that an emitter written before its call site
is a shape guessed rather than measured; the call site is ``explore._absorb``, and the events are
recorded on the exploration span this module's ``exploration_tracer`` hands out. Nothing about the
contract above changed: with tracing off there is no provider, so those events are discarded like
every other span this process makes.
MAF-touching by construction, so this module never enters the framework-neutral context layer
(``okf.py``); the ``test_okf_is_maf_free`` guard keeps that boundary.
@ -233,3 +237,29 @@ def tracing_notice(setup: TracingSetup) -> str | None:
f" Tracing: {TRACING_ENV}={MODE_OTLP} — OpenTelemetry spans are EXPORTED OVER THE NETWORK "
f"to the endpoints declared below\n{rows}"
)
#: The instrumentation scope every exploration span is created under. One name, so a collector
#: can select this framework's own spans apart from MAF's (``invoke_agent``, ``workflow.run``)
#: without matching on span names that MAF owns and may rename.
EXPLORATION_TRACER_NAME: Final = "portfolio_optimiser.explore"
def exploration_tracer() -> Any:
"""The tracer the exploration loop records its decisions on.
``get_tracer`` is safe to call whether or not a provider was installed: with none, OpenTelemetry
hands back a no-op tracer and every span and event is discarded. That is the SAME shape MAF's
own instrumentation already has (``ENABLE_INSTRUMENTATION`` defaults to True and its spans are
thrown away for want of a provider), and it is what lets the exploration emit unconditionally.
Gating emission on ``PORTFOLIO_OTEL`` would be a second resolution of a rule this module owns,
free to disagree with the providers actually installed.
A FUNCTION rather than a module-level tracer, and the reason is ordering: ``configure_tracing``
runs at process startup, and a tracer bound at import time would have been taken from the
global provider that existed BEFORE it — a no-op one, permanently. It is also the seam a test
substitutes a local provider through, without installing anything globally.
"""
from opentelemetry import trace
return trace.get_tracer(EXPLORATION_TRACER_NAME)