feat(explore): U4 kallsted 1 - --explore former mandatet, og sporet overlever taket (ORDRE 20260823T204216Z) [skip-docs]
Kallsted (1) + artefaktet (4) av oerkt 57s fire.
--explore "<prompt>" + --explore-config FILE i run.py: utforskningen kjoerer FOER
pipelinen og mandatet den former gaar rett inn i run_project(mandate=...). Flagget
er opt-in, og hver ting det ikke kan aere NEKTES ved navn:
- --explore-config uten --explore (--embedder-config-presedensen, ordrett)
- --explore uten --explore-config: CLI-en oppfinner ALDRI grenser, fordi
MagenticBuilders egen fallback er "ubegrenset"
- --explore + --mandate: TO KILDER TIL ETT MANDAT. Nektet, aldri slaatt sammen -
explore() tar objective fra prompten og hardkoder allow_own_proposals=True, saa
en sammenslaaing ville stille overskrevet tre felt operatoeren skrev selv.
Nekten NAVNGIR biblioteksdoera (seed_approaches), fordi C.6 doer 1 er et ekte
behov denne flaten ikke betjener
- --explore + --live-dry-run (motstrid), --explore uten --bundle-dir (leser
ingenting), --portfolio --explore (partisjonen)
- enable_plan_review=true nektes HER, ikke i explore(): ExplorationError er en
RuntimeError og ligger UTENFOR main()s (ValueError, FileNotFoundError,
ValidationError)-tuppel, saa den ville forlatt som traceback i stedet for rc 1
ExplorationTrace er en KALLER-EID akkumulator (funn-1-sinken, ett lag opp):
explore() raiser BudgetExceeded paa rundetaket og tokentaket fyrer fra middleware
midt i loepet - paa begge stier finnes ingen ExplorationResult, og C.2 krever at
artefaktet er lesbart uansett hvilken vakt som fyrte. ExplorationResult.ledger_log
BYGGES FRA akkumulatoren, aldri ved siden av (kø-(p)).
{run_id}-exploration.json skrives fra en finally (write_parse_failures-presedensen)
med rundene, plan-reviewene og quick_validate-dommene - de siste bor bevisst ikke i
ExplorationResult. `completed` er et eget felt: en stop: null som betyr BAADE
"avsluttet normalt" og "vi fikk aldri vite" er stillheten cost_baseline_anchored
ble paakrevd for aa lukke.
Load-bearing MAALT (tests/test_explore_callsites_loadbearing.py, 15 tester), ti
mutasjoner alle roede mot HELE suiten + groenn kontroll 990/5: detach sink-appenden
(1 roed) - andre liste for rundene (4 roede) - detach --mandate-nekten (1) - detach
--explore-config-nekten (1) - skriv artefaktet kun naar kjoeringen fullfoerte (1) -
detach mandate= inn i run_project (1) - slipp enable_plan_review gjennom (1) -
detach --bundle-dir-kravet (1) - detach --live-dry-run-nekten (1) - fjern --explore
fra portefoelje-partisjonen (1).
EN MUTASJON FALSIFISERTE TESTEN FOERST (repoets vakuoes-gate-klasse, syvende gang):
portefoelje-testen asserterte kun at meldingen nevnte --explore, og var groenn UTEN
partisjonen - kjoeringen falt da gjennom til "--explore requires --bundle-dir", som
nevner --explore ogsaa. To nekter som deler en delstreng; testen navngir naa
--portfolio.
Golden-transkriptet byte-uendret (ea8c534773acdbe41ae68f2c55724d69aaf8be4f).
mypy + ruff rene.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YRZhBJcxqTcqWyMW6hBttx
This commit is contained in:
parent
ddc33eed8b
commit
8ba824c96f
4 changed files with 897 additions and 21 deletions
|
|
@ -25,7 +25,7 @@ from __future__ import annotations
|
|||
|
||||
import json
|
||||
from collections.abc import Callable, Mapping, Sequence
|
||||
from dataclasses import dataclass
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any, Final, Literal
|
||||
|
||||
|
|
@ -103,6 +103,40 @@ class ExplorationContract(BaseModel):
|
|||
return self
|
||||
|
||||
|
||||
def exploration_notice(result: "ExplorationResult") -> str:
|
||||
"""The ONE line an exploration says about itself before the pipeline takes over.
|
||||
|
||||
Always a line, never an omission — unlike ``cost_baseline_notice`` and
|
||||
``skipped_links_notice``, which qualify a run that happened anyway. This one is printed only on
|
||||
a surface where ``--explore`` was asked for, so there is no run it could speak for silently.
|
||||
|
||||
``stop`` is named when there is one, because a mandate shaped by a loop that gave up is a
|
||||
smaller mandate than the same loop finishing would have produced, and nothing else on stdout
|
||||
would say so.
|
||||
"""
|
||||
ended = "concluded" if result.stop is None else f"stopped ({result.stop})"
|
||||
return (
|
||||
f"Exploration: {ended} after {len(result.ledger_log)} round(s); "
|
||||
f"{len(result.mandate.approaches)} approach(es) to evaluate"
|
||||
)
|
||||
|
||||
|
||||
def load_exploration_contract(path: str | Path) -> ExplorationContract:
|
||||
"""Fail-fast standalone loader for an exploration's bounds (mirrors ``mandate.load_mandate``).
|
||||
|
||||
The bounds are authoritative startup input, and the ONE thing that keeps a Magentic loop from
|
||||
being unbounded — so a missing or malformed file refuses the run rather than degrading to
|
||||
defaults, which is the shape ``ExplorationContract`` refuses to have in the first place.
|
||||
|
||||
:raises FileNotFoundError: ``path`` does not point at an existing file.
|
||||
:raises pydantic.ValidationError: the content is not JSON, or violates the contract.
|
||||
"""
|
||||
p = Path(path)
|
||||
if not p.is_file():
|
||||
raise FileNotFoundError(f"exploration config not found: {str(path)!r}")
|
||||
return ExplorationContract.model_validate_json(p.read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
#: The manager's role name. Not a participant: the manager plans, picks the next speaker and
|
||||
#: keeps the progress ledger, and ``BudgetMiddleware`` rides on it like on everyone else (A1,
|
||||
#: measured green — without that, the most talkative agent in the loop would be the one outside
|
||||
|
|
@ -186,6 +220,89 @@ class PlanReview:
|
|||
feedback: str = ""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class QuickValidation:
|
||||
"""One advisory (level 1) verdict, as the tool answered it.
|
||||
|
||||
``proposal_json`` is kept VERBATIM for the reason the parse-failure capture keeps the raw reply
|
||||
verbatim: the operative question after a run is what the hypothesiser actually asked about, and
|
||||
a re-serialised form would answer a different one.
|
||||
"""
|
||||
|
||||
bundle_id: str
|
||||
proposal_json: str
|
||||
verdict: Mapping[str, Any]
|
||||
|
||||
|
||||
@dataclass
|
||||
class ExplorationTrace:
|
||||
"""The CALLER-owned accumulator for everything one exploration produced along the way.
|
||||
|
||||
**Why a caller-owned sink and not a return value** — the same measurement that shaped the
|
||||
parse-failure capture (Fase 1b, funn 1), one layer up. ``explore()`` raises ``BudgetExceeded``
|
||||
at its round cap, and a token cap fires from inside ``BudgetMiddleware`` mid-run; on both paths
|
||||
no ``ExplorationResult`` is ever constructed. § C.2 requires the exploration artefact to be
|
||||
readable "uansett hvilken vakt som fyrte", so the evidence has to live somewhere that survives
|
||||
the ending — which a return value, by definition, does not. Step 5's opposite rule ("a returned
|
||||
value, never an out-parameter") governs a value that DOES reach the caller; copying it blindly
|
||||
here would rebuild the very defect it was written against.
|
||||
|
||||
``ExplorationResult.ledger_log`` and ``.plan_reviews`` are built FROM these lists rather than
|
||||
accumulated beside them: two containers holding one fact drift (kø-(p)), and a drifted pair
|
||||
would let the returned result and the written artefact describe different runs.
|
||||
"""
|
||||
|
||||
ledger: list[LedgerEntry] = field(default_factory=list)
|
||||
plan_reviews: list[PlanReview] = field(default_factory=list)
|
||||
quick_validations: list[QuickValidation] = field(default_factory=list)
|
||||
|
||||
|
||||
def trace_payload(trace: ExplorationTrace, *, stop: str | None, completed: bool) -> dict[str, Any]:
|
||||
"""The ONE rendering of a trace into plain data for ``outbox.write_exploration``.
|
||||
|
||||
Plain mappings only, so the RAW output layer stays MAF-free (the ``write_parse_failures``
|
||||
precedent — ``outbox.py`` may not import this module).
|
||||
|
||||
``completed`` is a required field rather than an inference from ``stop``. With no result there
|
||||
is no stop, and a ``stop: null`` meaning BOTH "concluded normally" and "we never found out"
|
||||
is exactly the silence ``ProvenanceStamp.cost_baseline_anchored`` was made required to close.
|
||||
"""
|
||||
return {
|
||||
"completed": completed,
|
||||
"stop": stop,
|
||||
"rounds": [
|
||||
{
|
||||
"round_index": entry.round_index,
|
||||
"is_request_satisfied": entry.is_request_satisfied,
|
||||
"is_in_loop": entry.is_in_loop,
|
||||
"is_progress_being_made": entry.is_progress_being_made,
|
||||
"next_speaker": entry.next_speaker,
|
||||
"instruction_or_question": entry.instruction_or_question,
|
||||
"speaker_known": entry.speaker_known,
|
||||
}
|
||||
for entry in trace.ledger
|
||||
],
|
||||
"plan_reviews": [
|
||||
{
|
||||
"index": review.index,
|
||||
"plan": review.plan,
|
||||
"is_stalled": review.is_stalled,
|
||||
"decision": review.decision,
|
||||
"feedback": review.feedback,
|
||||
}
|
||||
for review in trace.plan_reviews
|
||||
],
|
||||
"quick_validations": [
|
||||
{
|
||||
"bundle_id": call.bundle_id,
|
||||
"proposal_json": call.proposal_json,
|
||||
"verdict": dict(call.verdict),
|
||||
}
|
||||
for call in trace.quick_validations
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
#: Why an exploration ended without a satisfied request. ``None`` means it concluded normally.
|
||||
#: Resource exhaustion is NOT here: tokens and rounds raise ``BudgetExceeded`` (the 429 channel),
|
||||
#: because "we ran out" and "we finished, unsatisfied" are the two things S3.4 split apart and a
|
||||
|
|
@ -341,7 +458,9 @@ def navigator_tools(bundle_dirs: Sequence[str]) -> list[FunctionTool]:
|
|||
return [list_bundles, read_bundle, read_file]
|
||||
|
||||
|
||||
def quick_validate_tool(bundle_dirs: Sequence[str]) -> FunctionTool:
|
||||
def quick_validate_tool(
|
||||
bundle_dirs: Sequence[str], *, sink: list[QuickValidation] | None = None
|
||||
) -> FunctionTool:
|
||||
"""The hypothesiser's in-loop deterministic check — level 1, and advisory by construction.
|
||||
|
||||
It is the SAME ``validate_proposal`` against the SAME baseline the pipeline will use, so the
|
||||
|
|
@ -356,6 +475,11 @@ def quick_validate_tool(bundle_dirs: Sequence[str]) -> FunctionTool:
|
|||
``ProvenanceStamp.cost_baseline_anchored`` is a required field: a verdict reached without the
|
||||
project's own cost lines is a weaker claim, and one that does not say so is the silence S4.0's
|
||||
visibility work closed.
|
||||
|
||||
``sink`` is the caller's ``ExplorationTrace.quick_validations``. These verdicts are the one
|
||||
thing ``ExplorationResult`` deliberately does not carry — they are level-1 advisory, and their
|
||||
home is the ``{run_id}-exploration.json`` artefact — so without a sink they would leave no
|
||||
trace of having been asked for at all.
|
||||
"""
|
||||
index = _bundle_index(bundle_dirs)
|
||||
|
||||
|
|
@ -370,25 +494,40 @@ def quick_validate_tool(bundle_dirs: Sequence[str]) -> FunctionTool:
|
|||
def quick_validate(bundle_id: str, proposal_json: str) -> dict[str, Any]:
|
||||
bundle_dir = _resolve_bundle(index, bundle_id)
|
||||
baseline = okf.load_optional_cost_baseline(bundle_dir)
|
||||
verdict: dict[str, Any]
|
||||
try:
|
||||
proposal = SavingsProposal.model_validate_json(proposal_json)
|
||||
except ValidationError as exc:
|
||||
return {"decision": "unparseable", "reason": str(exc), "anchored": baseline is not None}
|
||||
outcome = validate_proposal(proposal, baseline=baseline)
|
||||
if isinstance(outcome, Rejection):
|
||||
return {
|
||||
"decision": "rejected",
|
||||
"reason": outcome.reason,
|
||||
verdict = {
|
||||
"decision": "unparseable",
|
||||
"reason": str(exc),
|
||||
"anchored": baseline is not None,
|
||||
}
|
||||
return {
|
||||
"decision": "validated",
|
||||
"reason": "",
|
||||
"anchored": baseline is not None,
|
||||
"p10": outcome.p10,
|
||||
"p50": outcome.p50,
|
||||
"p90": outcome.p90,
|
||||
}
|
||||
else:
|
||||
outcome = validate_proposal(proposal, baseline=baseline)
|
||||
if isinstance(outcome, Rejection):
|
||||
verdict = {
|
||||
"decision": "rejected",
|
||||
"reason": outcome.reason,
|
||||
"anchored": baseline is not None,
|
||||
}
|
||||
else:
|
||||
verdict = {
|
||||
"decision": "validated",
|
||||
"reason": "",
|
||||
"anchored": baseline is not None,
|
||||
"p10": outcome.p10,
|
||||
"p50": outcome.p50,
|
||||
"p90": outcome.p90,
|
||||
}
|
||||
# Recorded AFTER the verdict is decided and on EVERY branch — an unparseable candidate is
|
||||
# as much a thing the hypothesiser asked about as a validated one. A refused bundle id
|
||||
# raises above and is deliberately not recorded: nothing was validated.
|
||||
if sink is not None:
|
||||
sink.append(
|
||||
QuickValidation(bundle_id=bundle_id, proposal_json=proposal_json, verdict=verdict)
|
||||
)
|
||||
return verdict
|
||||
|
||||
return quick_validate
|
||||
|
||||
|
|
@ -404,6 +543,7 @@ def fresh_exploration_workflow(
|
|||
contract: ExplorationContract,
|
||||
bundle_dirs: Sequence[str] = (),
|
||||
middleware: Sequence[Any] | None = None,
|
||||
quick_validate_sink: list[QuickValidation] | None = None,
|
||||
) -> Any:
|
||||
"""Build a FRESH Magentic workflow with fresh agents and fresh clients (mirrors
|
||||
``workflow.fresh_workflow``).
|
||||
|
|
@ -426,7 +566,7 @@ def fresh_exploration_workflow(
|
|||
token guarantee: agent-level ``ChatMiddleware`` does fire on the manager's own calls (measured
|
||||
A1), and the manager is the most talkative participant in the loop.
|
||||
"""
|
||||
hypothesiser_tools: list[Any] = [quick_validate_tool(bundle_dirs)]
|
||||
hypothesiser_tools: list[Any] = [quick_validate_tool(bundle_dirs, sink=quick_validate_sink)]
|
||||
tools_by_role: dict[str, list[Any]] = {
|
||||
NAVIGATOR_ROLE: list(navigator_tools(bundle_dirs)),
|
||||
HYPOTHESISER_ROLE: hypothesiser_tools,
|
||||
|
|
@ -631,6 +771,7 @@ async def explore(
|
|||
plan_reviewer: PlanReviewer | None = None,
|
||||
meter: TokenMeter | None = None,
|
||||
success_criteria: str = "",
|
||||
trace: ExplorationTrace | None = None,
|
||||
) -> ExplorationResult:
|
||||
"""Explore the knowledge bases and return the ``Mandate`` the pipeline should evaluate.
|
||||
|
||||
|
|
@ -656,8 +797,13 @@ async def explore(
|
|||
``bundle_dir``, and shipping the field before its consumer would be a shape guessed instead of
|
||||
measured. (2) The ``quick_validate`` verdicts the hypothesiser saw are not in
|
||||
``ExplorationResult``: they are level-1 advisory, and their home is the
|
||||
``{run_id}-exploration.json`` artefact the CLI wiring writes. (3) The exploration roles resolve
|
||||
through ``resolve_model``'s ``default`` fallback unless an operator maps them explicitly.
|
||||
``{run_id}-exploration.json`` artefact the CLI wiring writes — pass an ``ExplorationTrace`` to
|
||||
collect them. (3) The exploration roles resolve through ``resolve_model``'s ``default``
|
||||
fallback unless an operator maps them explicitly.
|
||||
|
||||
``trace`` is the caller's accumulator and is the ONLY way to see what a run that RAISED
|
||||
produced: both budget channels destroy the ``ExplorationResult`` before it exists. When it is
|
||||
omitted a private one is used, so the returned result is unchanged for every existing caller.
|
||||
"""
|
||||
if contract.enable_plan_review and plan_reviewer is None:
|
||||
raise ExplorationError(
|
||||
|
|
@ -680,16 +826,22 @@ async def explore(
|
|||
|
||||
client_factory = _default_factory(profile)
|
||||
|
||||
if trace is None:
|
||||
trace = ExplorationTrace()
|
||||
|
||||
workflow = fresh_exploration_workflow(
|
||||
client_factory,
|
||||
contract=contract,
|
||||
bundle_dirs=bundle_dirs,
|
||||
middleware=[BudgetMiddleware(meter)],
|
||||
quick_validate_sink=trace.quick_validations,
|
||||
)
|
||||
|
||||
ledger_log: list[LedgerEntry] = []
|
||||
# ONE accumulator per fact, held by the caller (see ``ExplorationTrace``). The local names are
|
||||
# aliases, never copies — a second list here is the kø-(p) drift this shape exists to prevent.
|
||||
ledger_log = trace.ledger
|
||||
plan_reviews = trace.plan_reviews
|
||||
hypothesis_texts: list[str] = []
|
||||
plan_reviews: list[PlanReview] = []
|
||||
seen: set[int] = set()
|
||||
replans = 0
|
||||
stop: ExplorationStop | None = None
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue