open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 4 Activity

feat(p19): the trace says HOW, and a run says what it spent and why it stopped

Browse source 
DEL C. P18 gave read_dir a window (filter/offset/limit) and then measured its
own paid round without being able to see it used: five of 31 documents read
lay outside the default window, so the window HAD been widened and the trace
could not say with which knob. ToolCall now carries the three arguments,
always present and empty/zero when not passed -- an absent key and "not
narrowed" must not read the same -- and the judge counts filter_calls and
paged_calls. _number_argument is a SIBLING of _string_argument, not a widening
of it: a model may send limit as 10 or as "10", and a reader that knew one
shape would report a paged call as unpaged.

DEL D. P18's finding 4 was WRONG AS WRITTEN. provenance.token_usage has been
stamped on every proposal artefact since S3.4 and stands in every one of round
2's; what was missing is a READER. The judge reads it now (round 2 measured:
289 054 tokens against round 1's 2 679 305, -89 %), and the P18 report gets a
dated correction UNDER its original paragraph rather than instead of it.

What was genuinely absent is {run_id}-coverage.json. settle prints the
coverage report and ApproachOutcome has carried not_evaluated since Trekk A3,
but neither ever reached a file, so a judge could see an approach had no
artefact and could not tell a budget stop from an approach nobody ordered.
Written from the finally IFF a mandate was given. stop_reason comes from a
CALLER-OWNED sink rather than from in_flight, and that is a measurement:
_evaluate_mandate SWALLOWS BudgetExceeded once something has been produced, so
run_project's own in_flight never sees it.

Load-bearing measured (10 arms), four mutations all red against the whole
suite, green control 1744/5 and the golden byte-unchanged. D-i stood GREEN
first -- the vacuous-gate class, 25th time: the arm called write_coverage
itself and therefore chose the reason it then asserted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-15 03:05:52 +02:00
commit 4c6084e5df
12 changed files with 684 additions and 11 deletions
Download patch file Download diff file Expand all files Collapse all files

51
src/portfolio_optimiser/run.py
View file

@ -509,6 +509,7 @@ def _select_outcome(
async def _evaluate_mandate(
mandate: Mandate,
evaluate: Callable[[Approach | None], Awaitable[ValidatedProposal | Rejection]],
budget_stops: list[str] | None = None,
) -> tuple[
ValidatedProposal | Rejection,
tuple[ApproachOutcome, ...],
@ -541,9 +542,16 @@ async def _evaluate_mandate(
for index, (row_id, label, approach) in enumerate(plan):
try:
outcome = await evaluate(approach)
except BudgetExceeded:
except BudgetExceeded as stop:
if not produced:
raise
# P19 D2: WHICH cap bound, recorded on a caller-owned sink before the rows are built.
# The exception is SWALLOWED here (the approaches that were reached are a real result),
# so ``run_project``'s own ``in_flight`` never sees it — and a coverage artefact that
# said "nothing stopped this run" about a commission cut in half would be the silence
# the artefact exists to remove.
if budget_stops is not None:
budget_stops.append(stop.kind)
rows.extend(
ApproachOutcome(
id=rid,
@ -1452,6 +1460,10 @@ async def run_project(
# on the very attempt a revise bought, and on that path ``generate_via_llm`` returns nothing —
# so the run whose record matters most is exactly the one a return value cannot reach.
expert_reviews: list[ProposalReview] = []
# P19 D2: which cap, if any, cut the commission short. Caller-owned for the reason every other
# sink here is: ``_evaluate_mandate`` SWALLOWS the stop once something has been produced, so a
# return value would not reach the ``finally`` that writes the artefact.
budget_stops: list[str] = []
async def _evaluate(approach: Approach | None) -> ValidatedProposal | Rejection:
# Which candidate the expert is being asked about. With a mandate every entry is keyed —
@ -1496,7 +1508,9 @@ async def run_project(
if mandate is None:
validator_outcome = await _evaluate(None)
else:
validator_outcome, coverage, evaluated = await _evaluate_mandate(mandate, _evaluate)
validator_outcome, coverage, evaluated = await _evaluate_mandate(
mandate, _evaluate, budget_stops
)
except BaseException as stop:
# Recorded and re-raised UNTOUCHED. This arm decides nothing about the exception itself —
# only what the two writers in the ``finally`` are allowed to do to it (``_write_or_report``).
@ -1518,6 +1532,39 @@ async def run_project(
what=f"{run_id}-parse-failures.json",
in_flight=in_flight,
)
# Same ``finally``, a THIRD write rule: IFF a mandate was given, including when the run
# stopped before a single approach was evaluated (P19 D2). Coverage is the MANDATE's
# report by construction — without one there are no approaches and the file would describe
# nothing — so a mandate-less run leaves the outbox byte-identical, which two existing
# tests pin as an exact listing. The stop reason comes from the in-flight exception rather
# than being inferred: a ``BudgetExceeded`` carries ``kind`` as a field precisely so that
# "which cap bound" is readable by machine (kø-(y)), and a run that finished says so with
# an empty string rather than with a missing key.
if outbox_dir is not None and mandate is not None:
assert run_id is not None # narrowed by the step-0 guard (no wall-clock default)
_write_or_report(
lambda: outbox.write_coverage(
outbox_dir,
run_id,
rows=[
{
"id": row.id,
"label": row.label,
"status": row.status,
"detail": row.detail,
"saving_nok": row.saving_nok,
}
for row in coverage
],
stop_reason=(
budget_stops[0]
if budget_stops
else (in_flight.kind if isinstance(in_flight, BudgetExceeded) else "")
),
),
what=f"{run_id}-coverage.json",
in_flight=in_flight,
)
# Same ``finally``, different write rule: IFF a reviewer was given, including when the
# list is empty (D4). A reviewer-less run must leave the outbox byte-identical, while a
# reviewer that was offered and never consulted is a fact the artefact must be able to