open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 4 Activity

feat(p19): the trace says HOW, and a run says what it spent and why it stopped

Browse source 
DEL C. P18 gave read_dir a window (filter/offset/limit) and then measured its
own paid round without being able to see it used: five of 31 documents read
lay outside the default window, so the window HAD been widened and the trace
could not say with which knob. ToolCall now carries the three arguments,
always present and empty/zero when not passed -- an absent key and "not
narrowed" must not read the same -- and the judge counts filter_calls and
paged_calls. _number_argument is a SIBLING of _string_argument, not a widening
of it: a model may send limit as 10 or as "10", and a reader that knew one
shape would report a paged call as unpaged.

DEL D. P18's finding 4 was WRONG AS WRITTEN. provenance.token_usage has been
stamped on every proposal artefact since S3.4 and stands in every one of round
2's; what was missing is a READER. The judge reads it now (round 2 measured:
289 054 tokens against round 1's 2 679 305, -89 %), and the P18 report gets a
dated correction UNDER its original paragraph rather than instead of it.

What was genuinely absent is {run_id}-coverage.json. settle prints the
coverage report and ApproachOutcome has carried not_evaluated since Trekk A3,
but neither ever reached a file, so a judge could see an approach had no
artefact and could not tell a budget stop from an approach nobody ordered.
Written from the finally IFF a mandate was given. stop_reason comes from a
CALLER-OWNED sink rather than from in_flight, and that is a measurement:
_evaluate_mandate SWALLOWS BudgetExceeded once something has been produced, so
run_project's own in_flight never sees it.

Load-bearing measured (10 arms), four mutations all red against the whole
suite, green control 1744/5 and the golden byte-unchanged. D-i stood GREEN
first -- the vacuous-gate class, 25th time: the arm called write_coverage
itself and therefore chose the reason it then asserted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-15 03:05:52 +02:00
commit 4c6084e5df
12 changed files with 684 additions and 11 deletions
Download patch file Download diff file Expand all files Collapse all files

36
src/portfolio_optimiser/stress.py
View file

@ -125,6 +125,12 @@ class ApproachVerdict:
#: wrote one, and RE-DERIVED with the same classifier when it did not, so rounds 1 and 2 -
#: written before the field existed - can be re-judged with the same instrument.
prose_codes: tuple[str, ...]
#: P19 D2 - WHY this row was not evaluated: ``rounds`` / ``tokens`` when a cap cut the run
#: short, ``absent`` when the artefact is simply missing and no coverage file says otherwise,
#: and ``""`` for a row that WAS evaluated. Before this, "no artefact" could not be told from
#: "an approach nobody ordered" -- the silence ``ApproachOutcome`` exists to remove, one layer
#: out, and it reached no file until ``{run_id}-coverage.json``.
not_evaluated_reason: str
ferdig: bool
@ -153,6 +159,19 @@ class ContextSetVerdict:
#: none" is the measurement, and a missing field would be indistinguishable from a judge that
#: did not look.
requirements_declared: tuple[str, ...]
#: P19 DEL C - how the run asked for its listings. ``filter_calls`` is how many calls narrowed
#: a level by word, ``paged_calls`` how many asked for a window other than the default. P18 § 1
#: could only infer that the window HAD been widened (five of 31 documents read lay outside the
#: default) and never with which knob; these two make it readable directly.
filter_calls: int
paged_calls: int
#: P19 D1 - what the run SPENT, read off ``provenance.token_usage``, which has been stamped on
#: every proposal artefact since S3.4 and which P18's report wrongly said could not be given.
#: ``0`` when no artefact carried one.
token_usage: int
#: P19 D2 - ``BudgetExceeded.kind`` when a cap cut the run short, ``""`` when nothing did, and
#: ``"absent"`` when the run wrote no coverage file at all (every run before today).
stop_reason: str
tool_calls_seen: int
citations_seen: int
approach_rows_seen: int
@ -269,10 +288,18 @@ def score_context_set(
hallucinated_reads.append(raw)
reads_clean = not hallucinated_reads
# ---- P19 D1/D2: what the run spent, and why it stopped -----------------------------------
coverage_path = outbox / f"{run_id}-coverage.json"
stop_reason = (
str(_read_json(coverage_path).get("stop_reason", "")) if coverage_path.is_file() else ""
)
coverage_seen = coverage_path.is_file()
# ---- per approach ------------------------------------------------------------------------
rows: list[ApproachVerdict] = []
citations_seen = 0
rows_seen = 0
token_usage = 0
validated_codes: set[str] = set()
validated_ids: set[str] = set()
@ -299,6 +326,7 @@ def score_context_set(
requirement_source=_attributable(approach, declared_paths)[1],
requirement_hit=bool(set(_attributable(approach, declared_paths)[0]) & wanted),
prose_codes=(),
not_evaluated_reason=stop_reason or "absent",
ferdig=False,
)
)
@ -306,6 +334,7 @@ def score_context_set(
rows_seen += 1
payload = _read_json(proposal_path)
token_usage = max(token_usage, int(payload.get("provenance", {}).get("token_usage", 0)))
proposal = payload.get("proposal", {})
citations = payload.get("provenance", {}).get("citations", [])
citations_seen += len(citations)
@ -375,6 +404,7 @@ def score_context_set(
requirement_source=requirement_source,
requirement_hit=requirement_hit,
prose_codes=prose_codes,
not_evaluated_reason="",
ferdig=(
grounded
and (named_in_measure or named_in_snippet)
@ -421,6 +451,12 @@ def score_context_set(
must_refuse=tuple(refusals),
hallucinated_reads=tuple(hallucinated_reads),
requirements_declared=tuple(declared_paths),
token_usage=token_usage,
stop_reason=stop_reason if coverage_seen else "absent",
filter_calls=sum(1 for c in tool_calls if str(c.get("filter", ""))),
paged_calls=sum(
1 for c in tool_calls if int(c.get("offset", 0) or 0) or int(c.get("limit", 0) or 0)
),
tool_calls_seen=len(tool_calls),
citations_seen=citations_seen,
approach_rows_seen=rows_seen,