feat(p19): the trace says HOW, and a run says what it spent and why it stopped

DEL C. P18 gave read_dir a window (filter/offset/limit) and then measured its
own paid round without being able to see it used: five of 31 documents read
lay outside the default window, so the window HAD been widened and the trace
could not say with which knob. ToolCall now carries the three arguments,
always present and empty/zero when not passed -- an absent key and "not
narrowed" must not read the same -- and the judge counts filter_calls and
paged_calls. _number_argument is a SIBLING of _string_argument, not a widening
of it: a model may send limit as 10 or as "10", and a reader that knew one
shape would report a paged call as unpaged.

DEL D. P18's finding 4 was WRONG AS WRITTEN. provenance.token_usage has been
stamped on every proposal artefact since S3.4 and stands in every one of round
2's; what was missing is a READER. The judge reads it now (round 2 measured:
289 054 tokens against round 1's 2 679 305, -89 %), and the P18 report gets a
dated correction UNDER its original paragraph rather than instead of it.

What was genuinely absent is {run_id}-coverage.json. settle prints the
coverage report and ApproachOutcome has carried not_evaluated since Trekk A3,
but neither ever reached a file, so a judge could see an approach had no
artefact and could not tell a budget stop from an approach nobody ordered.
Written from the finally IFF a mandate was given. stop_reason comes from a
CALLER-OWNED sink rather than from in_flight, and that is a measurement:
_evaluate_mandate SWALLOWS BudgetExceeded once something has been produced, so
run_project's own in_flight never sees it.

Load-bearing measured (10 arms), four mutations all red against the whole
suite, green control 1744/5 and the golden byte-unchanged. D-i stood GREEN
first -- the vacuous-gate class, 25th time: the arm called write_coverage
itself and therefore chose the reason it then asserted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-15 03:05:52 +02:00
commit 4c6084e5df
12 changed files with 684 additions and 11 deletions

View file

@ -2690,6 +2690,44 @@ Python ≥3.10. MAF (`agent-framework-core` 1.16.0, `-orchestrations` 1.1.1 —
er en kode som telles som prosa — feilretningen er nekt, ikke aksept, og dét er hva er en kode som telles som prosa — feilretningen er nekt, ikke aksept, og dét er hva
generalitetsvernet og baseline-unntaket finnes for); og `assumptions`-nøkler er fortsatt utenfor generalitetsvernet og baseline-unntaket finnes for); og `assumptions`-nøkler er fortsatt utenfor
(økt 109s grense). (økt 109s grense).
- **Sporet sier HVORDAN nivået ble spurt om, og en kjøring sier hva den brukte og hvorfor den
stoppet (P19 DEL C + DEL D, 15.09):** P18 ga `read_dir` et VINDU (`filter`/`offset`/`limit`) og
målte så sin egen betalte runde uten å kunne se det brukt — fem av 31 leste dokumenter lå UTENFOR
default-vinduet, altså var vinduet utvidet, og sporet kunne ikke si med hvilken knapp. `ToolCall`
bærer nå de tre argumentene, `tool_call_payload` renderer dem (alltid til stede, tomme/null når
de ikke ble sendt — `write_debate_tools`-regelen: en fraværende nøkkel og «ikke innsnevret» må
ikke leses likt), og dommeren teller `filter_calls`/`paged_calls`. **`_number_argument` er en
SØSKEN av `_string_argument`, ikke en utvidelse av den:** en modell kan sende `limit` som `10`
eller `"10"`, og en leser som bare kjente den ene formen ville rapportert et paginert kall som
upaginert; `bool` er eksplisitt ekskludert, fordi et flagg ikke er et vindu.
**P18s FUNN 4 VAR FEIL SOM FORMULERT, og rettelsen er en TILFØYELSE:** `provenance.token_usage`
er stemplet på HVERT `-proposal.json` siden S3.4, også ved rc 0 — det som manglet var en LESER.
`stress.py` leser det nå (målt runde 2: 35 406 · 45 642 · 44 468 · 73 627 · 89 911 = **289 054**
mot runde 1s 2 679 305, **89 %**), og P18-rapportens § 7 har fått en datert rettelse UNDER det
opprinnelige avsnittet, aldri i stedet for det. **Det som GENUINT manglet er `{run_id}-coverage
.json`:** `settle` printer coverage-rapporten og `ApproachOutcome` har båret `not_evaluated`
siden Trekk A3, men ingen av dem nådde en fil, så en dommer kunne se at en approach manglet
artefakt og ikke skille et budsjettstopp fra en approach ingen bestilte — nøyaktig stillheten
`ApproachOutcome` finnes for, ett lag ut. Skrives fra `finally` **IFF et mandat ble gitt**
(coverage ER mandatets rapport; uten et er det ingen approaches, og en mandatløs kjøring lar
utboksen stå byte-identisk — to eldre tester pinner en eksakt listing). **`stop_reason` kommer
fra en KALLER-EID SINK, ikke fra `in_flight`, og dét er en MÅLING:** `_evaluate_mandate` SVELGER
`BudgetExceeded` så snart noe er produsert (de nådde approachene er et ekte resultat), så
`run_project`s egen `in_flight` ser den aldri — et coverage-artefakt som sa «ingenting stoppet
denne kjøringen» om en kommisjon kappet på midten ville vært stillheten artefaktet finnes for.
Dommerens `not_evaluated_reason` er `rounds`/`tokens` fra fila, `absent` når ingen fil finnes
(hver kjøring før i dag), og `""` for en rad som FAKTISK ble evaluert. Load-bearing MÅLT
(`tests/test_trace_and_coverage_loadbearing.py`, 10 armer), **fire mutasjoner alle røde mot HELE
suiten** + grønn kontroll **1744/5** (fra 1734/5, supersett, 0 fjernet) og golden BYTE-UENDRET
(`shasum -a 1` av INNHOLDET = `ea8c534773acdbe41ae68f2c55724d69aaf8be4f`): C-i vindus-argumentene
registreres ikke (2) · D-i coverage skrives med tom grunn (1) · D-ii kjøringen skriver den aldri
(2) · D-iii dommeren slutter å lese forbruket (1). **ÉN MUTASJON STO GRØNN FØRST (repoets
vakuøs-gate-klasse, TJUEFEMTE gang):** D-i lot HELE suiten stå grønn, fordi armen kalte
`write_coverage` selv og dermed VALGTE grunnen den så asserterte på; bare en kjøring et tak
faktisk kappet kan skille de to, og den armen driver nå `run_project` med `max_rounds=1` (én
runde betaler første approach, den andre er den taket kutter). **Ærlighets-grense, uttalt:**
`token_usage` er kjøringens ENE teller — den skiller ikke debatt fra generering, og en
per-fase-fordeling ville krevd en andre måler.
- **STATE.md er local-only** (gitignored). Voyage session-state er efemert; STATE.md er kanonisk kontinuitet. - **STATE.md er local-only** (gitignored). Voyage session-state er efemert; STATE.md er kanonisk kontinuitet.
- Prosess: Voyage-plugin (`/trekbrief → /trekplan → /trekexecute → /trekreview`) per større fase. - Prosess: Voyage-plugin (`/trekbrief → /trekplan → /trekexecute → /trekreview`) per større fase.

View file

@ -718,6 +718,16 @@ when the seam is detached, so the loop cannot silently degrade into theater.
run also writes `{run_id}-exploration.json`: the per-round ledger, the plan reviews and the run also writes `{run_id}-exploration.json`: the per-round ledger, the plan reviews and the
in-loop advisory verdicts, written even when a cap cut the exploration short. in-loop advisory verdicts, written even when a cap cut the exploration short.
Every tool call recorded in `{run_id}-exploration.json` and `{run_id}-debate.json` says HOW the
level was asked for — `filter`, `offset` and `limit` beside the tool name, the base and the path
— so "did the model narrow the level, or page through it" is readable from the artefact rather
than inferred from which documents happened to fall outside a default window.
A run given a mandate also writes `{run_id}-coverage.json`: one row per commissioned approach
with its status and detail, including the ones the run never reached, plus the `stop_reason`
(`tokens` / `rounds`) when a cap cut the commission short. A run without a mandate writes no
such file — coverage is the mandate's report.
The hosted surface takes the same door as `explore_prompt` + `explore_contract` on The hosted surface takes the same door as `explore_prompt` + `explore_contract` on
`POST /invocations`. `POST /invocations`.

View file

@ -260,6 +260,34 @@ brukte 2 679 305 tokens på sju kjøringer ≈ NOK 15 (anslag fra listepris). Ru
med kortere veggtid og billigere listinger; **et anslag på under NOK 10 er en antakelse, ikke en med kortere veggtid og billigere listinger; **et anslag på under NOK 10 er en antakelse, ikke en
måling**, og oppgis bare fordi ordren ber om et anslag med uttalt antakelse. måling**, og oppgis bare fordi ordren ber om et anslag med uttalt antakelse.
### Rettelse 15.09 (P19 D1) — feltet FANTES, og tallene står her
**Setningen over er feil som skrevet, og funn 4 var feil som formulert.**
`provenance.token_usage` (`provenance.py:60`, satt i `run.py` fra `meter.tokens`) er stemplet på
HVERT `-proposal.json` siden S3.4, også ved rc 0, og står i hver eneste av runde 2 sine. Det som
manglet var ikke feltet, men en LESER: ingenting i treet leste det. P19 D1 gir `stress.py` den
lesningen, og `{run_id}-verdict.json` bærer nå `token_usage`.
Målt 15.09 mot artefaktene runde 2 etterlot:
| kjøring | token_usage |
| --- | --- |
| gate-nordvik-2027-02 | 35 406 |
| tunnel-hauglia-2027-02 | 45 642 |
| fv412-dekkefornyelse-2027-02 | 44 468 |
| kontrakt-sorasen-2027-02 | 73 627 |
| gate-nordvik-2027-03 | 89 911 |
| **sum** | **289 054** |
Mot runde 1 sine **2 679 305** (§ 3 i P16-rapporten) er det **89 %**, og det er den største målte
effekten av P18s navigasjonsarbeid. Anslaget «under NOK 10» står som anslag; det som ikke lenger er
en antakelse er forbruket. **Tilføyelse, ikke omskriving:** avsnittet over står som det ble
skrevet, fordi en rapport som retter seg selv i stillhet ikke er en rapport.
Det som IKKE fantes og nå er bygget (P19 D2) er den andre halvdelen: `settle` printer
coverage-rapporten og `ApproachOutcome` har båret `not_evaluated` siden Trekk A3, men ingen av dem
nådde en FIL. `{run_id}-coverage.json` bærer dem nå, med `stop_reason` fra `BudgetExceeded.kind`.
--- ---
## 8. Verifisering ## 8. Verifisering

View file

@ -343,6 +343,15 @@ class ToolCall:
name: str name: str
bundle_id: str bundle_id: str
path: str path: str
#: P19 DEL C — HOW the listing was asked for, not just WHICH one. P18 gave ``read_dir`` a
#: window (``filter``/``offset``/``limit``) and the trace could not say whether a model used
#: it: five of 31 documents read in round 2 lay outside the default window, so the window HAD
#: been widened, and nothing said with which knob. Empty/zero mean "not passed" — the
#: ``bundle_id``/``path`` rule one field over, and unambiguous here because ``limit`` is
#: clamped to at least one wherever it is given.
filter: str = ""
offset: int = 0
limit: int = 0
@dataclass(frozen=True) @dataclass(frozen=True)
@ -393,6 +402,28 @@ def _string_argument(arguments: Any, key: str) -> str:
return value if isinstance(value, str) else "" return value if isinstance(value, str) else ""
def _number_argument(arguments: Any, key: str) -> int:
"""One NUMERIC argument of a call — ``_string_argument``'s sibling, and deliberately separate.
A model may send ``limit`` as ``10`` or as ``"10"`` (both reach a tool through the same wire),
so a recorder that read only the first shape would say a paging call was unpaged. Anything that
is neither is ``0``: the recorder describes the call, and inventing a number for an argument
nobody passed would be the false attribution ``_string_argument`` refuses for its own field.
``bool`` is excluded explicitly because it is an ``int`` in Python and a flag is not a window.
"""
if isinstance(arguments, Mapping):
value: Any = arguments.get(key)
else:
value = getattr(arguments, key, None)
if isinstance(value, bool):
return 0
if isinstance(value, int):
return value
if isinstance(value, str) and value.strip().lstrip("-").isdigit():
return int(value.strip())
return 0
class ExplorationToolRecorder(FunctionMiddleware): class ExplorationToolRecorder(FunctionMiddleware):
"""Records WHICH exploration tool an agent actually called, in the order it called them. """Records WHICH exploration tool an agent actually called, in the order it called them.
@ -424,6 +455,9 @@ class ExplorationToolRecorder(FunctionMiddleware):
name=name, name=name,
bundle_id=_string_argument(arguments, "bundle_id"), bundle_id=_string_argument(arguments, "bundle_id"),
path=_string_argument(arguments, "path"), path=_string_argument(arguments, "path"),
filter=_string_argument(arguments, "filter"),
offset=_number_argument(arguments, "offset"),
limit=_number_argument(arguments, "limit"),
) )
) )
await call_next() await call_next()
@ -479,7 +513,20 @@ def tool_call_payload(calls: Sequence[ToolCall]) -> list[dict[str, Any]]:
Plain mappings only, so the RAW output layer stays MAF-free (``outbox.py`` may not import Plain mappings only, so the RAW output layer stays MAF-free (``outbox.py`` may not import
this module). this module).
""" """
return [{"name": call.name, "bundle_id": call.bundle_id, "path": call.path} for call in calls] return [
{
"name": call.name,
"bundle_id": call.bundle_id,
"path": call.path,
# P19 DEL C: HOW the level was asked for. Always present, zero/empty when not passed —
# the ``write_debate_tools`` rule one field down: an absent key and "not narrowed" must
# not be the same reading.
"filter": call.filter,
"offset": call.offset,
"limit": call.limit,
}
for call in calls
]
def trace_payload( def trace_payload(

View file

@ -312,6 +312,43 @@ def write_proposal_reviews(
return path return path
def write_coverage(
outbox_dir: str,
run_id: str,
*,
rows: Sequence[Mapping[str, Any]],
stop_reason: str,
) -> Path:
"""Write ``{run_id}-coverage.json`` — WHY each commissioned approach ended as it did (P19 D2).
``settle`` prints the coverage report and ``ApproachOutcome`` has carried ``not_evaluated``
since Trekk A3, but neither ever reached a FILE: measured 14.09, a judge reading an outbox could
see that an approach had no artefact and could not tell a budget stop from an approach nobody
ordered. That is the very silence ``ApproachOutcome`` exists to remove, one layer out.
``stop_reason`` is ``BudgetExceeded.kind`` when a cap cut the run short (``tokens`` /
``rounds`` / the portfolio's own kinds) and ``""`` when nothing did. A REQUIRED argument rather
than an inferred one, for ``cost_baseline_anchored``'s reason: "the run finished" and "we never
found out" must not be the same value.
Byte-deterministic and wall-clock-free, mirroring ``write_run_config``; plain data only, so the
RAW output layer stays MAF-free."""
directory = Path(outbox_dir)
directory.mkdir(parents=True, exist_ok=True)
path = directory / f"{run_id}-coverage.json"
path.write_text(
_dump(
{
"run_id": run_id,
"stop_reason": stop_reason,
"rows": [dict(row) for row in rows],
}
),
encoding="utf-8",
)
return path
def write_run_config( def write_run_config(
config_dir: str, config_dir: str,
run_id: str, run_id: str,

View file

@ -509,6 +509,7 @@ def _select_outcome(
async def _evaluate_mandate( async def _evaluate_mandate(
mandate: Mandate, mandate: Mandate,
evaluate: Callable[[Approach | None], Awaitable[ValidatedProposal | Rejection]], evaluate: Callable[[Approach | None], Awaitable[ValidatedProposal | Rejection]],
budget_stops: list[str] | None = None,
) -> tuple[ ) -> tuple[
ValidatedProposal | Rejection, ValidatedProposal | Rejection,
tuple[ApproachOutcome, ...], tuple[ApproachOutcome, ...],
@ -541,9 +542,16 @@ async def _evaluate_mandate(
for index, (row_id, label, approach) in enumerate(plan): for index, (row_id, label, approach) in enumerate(plan):
try: try:
outcome = await evaluate(approach) outcome = await evaluate(approach)
except BudgetExceeded: except BudgetExceeded as stop:
if not produced: if not produced:
raise raise
# P19 D2: WHICH cap bound, recorded on a caller-owned sink before the rows are built.
# The exception is SWALLOWED here (the approaches that were reached are a real result),
# so ``run_project``'s own ``in_flight`` never sees it — and a coverage artefact that
# said "nothing stopped this run" about a commission cut in half would be the silence
# the artefact exists to remove.
if budget_stops is not None:
budget_stops.append(stop.kind)
rows.extend( rows.extend(
ApproachOutcome( ApproachOutcome(
id=rid, id=rid,
@ -1452,6 +1460,10 @@ async def run_project(
# on the very attempt a revise bought, and on that path ``generate_via_llm`` returns nothing — # on the very attempt a revise bought, and on that path ``generate_via_llm`` returns nothing —
# so the run whose record matters most is exactly the one a return value cannot reach. # so the run whose record matters most is exactly the one a return value cannot reach.
expert_reviews: list[ProposalReview] = [] expert_reviews: list[ProposalReview] = []
# P19 D2: which cap, if any, cut the commission short. Caller-owned for the reason every other
# sink here is: ``_evaluate_mandate`` SWALLOWS the stop once something has been produced, so a
# return value would not reach the ``finally`` that writes the artefact.
budget_stops: list[str] = []
async def _evaluate(approach: Approach | None) -> ValidatedProposal | Rejection: async def _evaluate(approach: Approach | None) -> ValidatedProposal | Rejection:
# Which candidate the expert is being asked about. With a mandate every entry is keyed — # Which candidate the expert is being asked about. With a mandate every entry is keyed —
@ -1496,7 +1508,9 @@ async def run_project(
if mandate is None: if mandate is None:
validator_outcome = await _evaluate(None) validator_outcome = await _evaluate(None)
else: else:
validator_outcome, coverage, evaluated = await _evaluate_mandate(mandate, _evaluate) validator_outcome, coverage, evaluated = await _evaluate_mandate(
mandate, _evaluate, budget_stops
)
except BaseException as stop: except BaseException as stop:
# Recorded and re-raised UNTOUCHED. This arm decides nothing about the exception itself — # Recorded and re-raised UNTOUCHED. This arm decides nothing about the exception itself —
# only what the two writers in the ``finally`` are allowed to do to it (``_write_or_report``). # only what the two writers in the ``finally`` are allowed to do to it (``_write_or_report``).
@ -1518,6 +1532,39 @@ async def run_project(
what=f"{run_id}-parse-failures.json", what=f"{run_id}-parse-failures.json",
in_flight=in_flight, in_flight=in_flight,
) )
# Same ``finally``, a THIRD write rule: IFF a mandate was given, including when the run
# stopped before a single approach was evaluated (P19 D2). Coverage is the MANDATE's
# report by construction — without one there are no approaches and the file would describe
# nothing — so a mandate-less run leaves the outbox byte-identical, which two existing
# tests pin as an exact listing. The stop reason comes from the in-flight exception rather
# than being inferred: a ``BudgetExceeded`` carries ``kind`` as a field precisely so that
# "which cap bound" is readable by machine (kø-(y)), and a run that finished says so with
# an empty string rather than with a missing key.
if outbox_dir is not None and mandate is not None:
assert run_id is not None # narrowed by the step-0 guard (no wall-clock default)
_write_or_report(
lambda: outbox.write_coverage(
outbox_dir,
run_id,
rows=[
{
"id": row.id,
"label": row.label,
"status": row.status,
"detail": row.detail,
"saving_nok": row.saving_nok,
}
for row in coverage
],
stop_reason=(
budget_stops[0]
if budget_stops
else (in_flight.kind if isinstance(in_flight, BudgetExceeded) else "")
),
),
what=f"{run_id}-coverage.json",
in_flight=in_flight,
)
# Same ``finally``, different write rule: IFF a reviewer was given, including when the # Same ``finally``, different write rule: IFF a reviewer was given, including when the
# list is empty (D4). A reviewer-less run must leave the outbox byte-identical, while a # list is empty (D4). A reviewer-less run must leave the outbox byte-identical, while a
# reviewer that was offered and never consulted is a fact the artefact must be able to # reviewer that was offered and never consulted is a fact the artefact must be able to

View file

@ -125,6 +125,12 @@ class ApproachVerdict:
#: wrote one, and RE-DERIVED with the same classifier when it did not, so rounds 1 and 2 - #: wrote one, and RE-DERIVED with the same classifier when it did not, so rounds 1 and 2 -
#: written before the field existed - can be re-judged with the same instrument. #: written before the field existed - can be re-judged with the same instrument.
prose_codes: tuple[str, ...] prose_codes: tuple[str, ...]
#: P19 D2 - WHY this row was not evaluated: ``rounds`` / ``tokens`` when a cap cut the run
#: short, ``absent`` when the artefact is simply missing and no coverage file says otherwise,
#: and ``""`` for a row that WAS evaluated. Before this, "no artefact" could not be told from
#: "an approach nobody ordered" -- the silence ``ApproachOutcome`` exists to remove, one layer
#: out, and it reached no file until ``{run_id}-coverage.json``.
not_evaluated_reason: str
ferdig: bool ferdig: bool
@ -153,6 +159,19 @@ class ContextSetVerdict:
#: none" is the measurement, and a missing field would be indistinguishable from a judge that #: none" is the measurement, and a missing field would be indistinguishable from a judge that
#: did not look. #: did not look.
requirements_declared: tuple[str, ...] requirements_declared: tuple[str, ...]
#: P19 DEL C - how the run asked for its listings. ``filter_calls`` is how many calls narrowed
#: a level by word, ``paged_calls`` how many asked for a window other than the default. P18 § 1
#: could only infer that the window HAD been widened (five of 31 documents read lay outside the
#: default) and never with which knob; these two make it readable directly.
filter_calls: int
paged_calls: int
#: P19 D1 - what the run SPENT, read off ``provenance.token_usage``, which has been stamped on
#: every proposal artefact since S3.4 and which P18's report wrongly said could not be given.
#: ``0`` when no artefact carried one.
token_usage: int
#: P19 D2 - ``BudgetExceeded.kind`` when a cap cut the run short, ``""`` when nothing did, and
#: ``"absent"`` when the run wrote no coverage file at all (every run before today).
stop_reason: str
tool_calls_seen: int tool_calls_seen: int
citations_seen: int citations_seen: int
approach_rows_seen: int approach_rows_seen: int
@ -269,10 +288,18 @@ def score_context_set(
hallucinated_reads.append(raw) hallucinated_reads.append(raw)
reads_clean = not hallucinated_reads reads_clean = not hallucinated_reads
# ---- P19 D1/D2: what the run spent, and why it stopped -----------------------------------
coverage_path = outbox / f"{run_id}-coverage.json"
stop_reason = (
str(_read_json(coverage_path).get("stop_reason", "")) if coverage_path.is_file() else ""
)
coverage_seen = coverage_path.is_file()
# ---- per approach ------------------------------------------------------------------------ # ---- per approach ------------------------------------------------------------------------
rows: list[ApproachVerdict] = [] rows: list[ApproachVerdict] = []
citations_seen = 0 citations_seen = 0
rows_seen = 0 rows_seen = 0
token_usage = 0
validated_codes: set[str] = set() validated_codes: set[str] = set()
validated_ids: set[str] = set() validated_ids: set[str] = set()
@ -299,6 +326,7 @@ def score_context_set(
requirement_source=_attributable(approach, declared_paths)[1], requirement_source=_attributable(approach, declared_paths)[1],
requirement_hit=bool(set(_attributable(approach, declared_paths)[0]) & wanted), requirement_hit=bool(set(_attributable(approach, declared_paths)[0]) & wanted),
prose_codes=(), prose_codes=(),
not_evaluated_reason=stop_reason or "absent",
ferdig=False, ferdig=False,
) )
) )
@ -306,6 +334,7 @@ def score_context_set(
rows_seen += 1 rows_seen += 1
payload = _read_json(proposal_path) payload = _read_json(proposal_path)
token_usage = max(token_usage, int(payload.get("provenance", {}).get("token_usage", 0)))
proposal = payload.get("proposal", {}) proposal = payload.get("proposal", {})
citations = payload.get("provenance", {}).get("citations", []) citations = payload.get("provenance", {}).get("citations", [])
citations_seen += len(citations) citations_seen += len(citations)
@ -375,6 +404,7 @@ def score_context_set(
requirement_source=requirement_source, requirement_source=requirement_source,
requirement_hit=requirement_hit, requirement_hit=requirement_hit,
prose_codes=prose_codes, prose_codes=prose_codes,
not_evaluated_reason="",
ferdig=( ferdig=(
grounded grounded
and (named_in_measure or named_in_snippet) and (named_in_measure or named_in_snippet)
@ -421,6 +451,12 @@ def score_context_set(
must_refuse=tuple(refusals), must_refuse=tuple(refusals),
hallucinated_reads=tuple(hallucinated_reads), hallucinated_reads=tuple(hallucinated_reads),
requirements_declared=tuple(declared_paths), requirements_declared=tuple(declared_paths),
token_usage=token_usage,
stop_reason=stop_reason if coverage_seen else "absent",
filter_calls=sum(1 for c in tool_calls if str(c.get("filter", ""))),
paged_calls=sum(
1 for c in tool_calls if int(c.get("offset", 0) or 0) or int(c.get("limit", 0) or 0)
),
tool_calls_seen=len(tool_calls), tool_calls_seen=len(tool_calls),
citations_seen=citations_seen, citations_seen=citations_seen,
approach_rows_seen=rows_seen, approach_rows_seen=rows_seen,

View file

@ -262,7 +262,10 @@ async def test_the_debate_tool_trace_reaches_the_result_and_the_outbox(tmp_path)
], f"the debate's tool trace is {observed} — the call sequence is not recorded as it happened" ], f"the debate's tool trace is {observed} — the call sequence is not recorded as it happened"
payload = json.loads((outbox_dir / "s2c-debate.json").read_text(encoding="utf-8")) payload = json.loads((outbox_dir / "s2c-debate.json").read_text(encoding="utf-8"))
assert payload["tool_calls"][:2] == [ # P19 DEL C added HOW the level was asked for (``filter``/``offset``/``limit``); the
# identity of the call is still these three, so the arm compares on them.
identity = [{k: c[k] for k in ("name", "bundle_id", "path")} for c in payload["tool_calls"]]
assert identity[:2] == [
{"name": "read_bundle", "bundle_id": bundle_id, "path": ""}, {"name": "read_bundle", "bundle_id": bundle_id, "path": ""},
{"name": "read_file", "bundle_id": bundle_id, "path": "metode-ipmvp-a.md"}, {"name": "read_file", "bundle_id": bundle_id, "path": "metode-ipmvp-a.md"},
], f"the artefact does not carry the call sequence: {payload['tool_calls']}" ], f"the artefact does not carry the call sequence: {payload['tool_calls']}"

View file

@ -282,7 +282,10 @@ def test_the_record_leaves_the_run_in_the_artefact_beside_the_advisory_verdicts(
# ``path`` joined the record in S7a-3 pkt. 3 and is ``""`` here because ``read_bundle`` takes # ``path`` joined the record in S7a-3 pkt. 3 and is ``""`` here because ``read_bundle`` takes
# none; its own gate is ``tests/test_tool_call_path_loadbearing.py``. # none; its own gate is ``tests/test_tool_call_path_loadbearing.py``.
assert payload["tool_calls"] == [ # P19 DEL C added HOW the level was asked for (``filter``/``offset``/``limit``); the
# identity of the call is still these three, so the arm compares on them.
identity = [{k: c[k] for k in ("name", "bundle_id", "path")} for c in payload["tool_calls"]]
assert identity == [
{"name": "read_bundle", "bundle_id": "bygg-energi-mikro", "path": ""} {"name": "read_bundle", "bundle_id": "bygg-energi-mikro", "path": ""}
], "the artefact must say which bases were opened, or a paid run leaves no record that any were" ], "the artefact must say which bases were opened, or a paid run leaves no record that any were"
assert json.dumps(payload), "the payload must stay plain data — the RAW layer is MAF-free" assert json.dumps(payload), "the payload must stay plain data — the RAW layer is MAF-free"
@ -479,7 +482,8 @@ def test_a_step_list_is_accepted_for_a_debate_role(tmp_path) -> None:
payload = json.loads( payload = json.loads(
(tmp_path / "outbox" / "scripted-debate-list-debate.json").read_text(encoding="utf-8") (tmp_path / "outbox" / "scripted-debate-list-debate.json").read_text(encoding="utf-8")
) )
assert payload["tool_calls"][0] == { first = payload["tool_calls"][0]
assert {k: first[k] for k in ("name", "bundle_id", "path")} == {
"name": "read_bundle", "name": "read_bundle",
"bundle_id": "bygg-energi-mikro", "bundle_id": "bygg-energi-mikro",
"path": "", "path": "",

View file

@ -99,7 +99,10 @@ def test_the_result_is_still_never_recorded() -> None:
sink = _record(("read_file", {"bundle_id": "k2", "path": "a.md"})) sink = _record(("read_file", {"bundle_id": "k2", "path": "a.md"}))
recorded = json.dumps([vars(c) for c in sink]) recorded = json.dumps([vars(c) for c in sink])
assert set(vars(sink[0])) == {"name", "bundle_id", "path"} # P19 DEL C added HOW the listing was asked for (``filter``/``offset``/``limit``). The claim
# this arm makes is unchanged and is the one that matters: the fields are all ARGUMENTS, and
# none of them is the result.
assert set(vars(sink[0])) == {"name", "bundle_id", "path", "filter", "offset", "limit"}
assert "result" not in recorded assert "result" not in recorded
@ -115,7 +118,16 @@ def test_the_artefact_carries_the_path_beside_the_name_and_the_base() -> None:
payload = ex.trace_payload(trace, stop=None, completed=True, mandate=None) payload = ex.trace_payload(trace, stop=None, completed=True, mandate=None)
assert payload["tool_calls"] == [ assert payload["tool_calls"] == [
{"name": "read_file", "bundle_id": "k2", "path": "prisskjema.md"} {
"name": "read_file",
"bundle_id": "k2",
"path": "prisskjema.md",
# P19 DEL C: always present, zero/empty for a call that did not pass them — an absent
# key and "not narrowed" must not read the same.
"filter": "",
"offset": 0,
"limit": 0,
}
] ]
assert json.dumps(payload), "the payload must stay plain data - the RAW layer is MAF-free" assert json.dumps(payload), "the payload must stay plain data - the RAW layer is MAF-free"

View file

@ -0,0 +1,410 @@
"""P19 DEL C + DEL D — the trace says HOW, and a run says what it spent and why it stopped.
**DEL C, the measured silence.** P18 gave ``read_dir`` a window ``filter`` / ``offset`` /
``limit`` and then measured its own paid round without being able to see it used: five of 31
documents read lay OUTSIDE the default window, so the window had been widened, and the trace could
not say with which knob (``docs/2026-09-14-p18-stressrunde-2.md`` § 1, finding 1). The recorder
kept ``name`` / ``bundle_id`` / ``path`` and nothing else. "Did the model narrow the level, or page
through it?" is the operative question about a corpus of 2 756 documents, and it was unanswerable
from the artefact the run leaves behind.
**DEL D, and one of the two findings it fixes was WRONG AS WRITTEN.** P18's finding 4 said a
successful run does not say what it used. Measured 14.09: ``provenance.token_usage`` has been
stamped on every proposal artefact since S3.4 and stands in every one of round 2's. What was
missing is a JUDGE that reads it so the report's sentence is corrected in its own § 7 addendum
rather than the field being invented a second time. What was genuinely absent is the OTHER half:
``settle`` prints the coverage report, ``ApproachOutcome`` has carried ``not_evaluated`` since
Trekk A3, and neither ever reached a file. A judge reading an outbox could see that an approach had
no artefact and could not tell a budget stop from an approach nobody ordered the silence
``ApproachOutcome`` exists to remove, one layer out.
What each arm pins:
(a) C the three window arguments are RECORDED, in the shape the wire actually carries;
(b) C a call that passes none of them yields empty/zero fields, never a refusal and never an
absent key: "not narrowed" and "we did not look" must not read the same;
(c) C a numeric argument sent as a STRING is coerced. A model may send either, and a recorder
that read only one shape would report a paged call as unpaged;
(d) C the judge counts them, so "did the model use the filter" is readable directly rather than
inferred from which documents happened to lie outside a default window;
(e) D2 ``{run_id}-coverage.json`` is written IFF a mandate was given, INCLUDING when a cap cut
the run short before a single approach was evaluated. A mandate-less run leaves the outbox
byte-identical, which two older tests pin as an exact listing;
(f) D2 the stop reason is ``BudgetExceeded.kind``, carried as a field rather than inferred, and
the judge turns it into each unevaluated row's ``not_evaluated_reason``;
(g) D1 the judge reads ``provenance.token_usage`` and reports it, with ``absent`` distinguishing
a run that wrote no coverage file (every run before today) from one that finished cleanly.
"""
from __future__ import annotations
import json
from pathlib import Path
from typing import Any
import pytest
from agent_framework import BaseChatClient
from conftest import SyntheticUsageChatClient
from portfolio_optimiser import explore as ex
from portfolio_optimiser import outbox, stress
from portfolio_optimiser.budget import Budget, TokenMeter
from portfolio_optimiser.mandate import Approach, Mandate
from portfolio_optimiser.run import run_project
_BUNDLE_DIR = Path(__file__).resolve().parents[1] / "shared" / "examples" / "bygg-energi-mikro"
_PID = "BYGG-KONTOR-NORD"
_PROPOSER = (
'{"measure":"LED-retrofit av kontorbelysning","affected_items":'
'[{"code":"ENERGI-TOTAL-EL","quantity":300000,"unit_cost":1.0}],"claimed_saving_nok":30000}'
)
_CHECKER = "Supported by the cited documents. VERDICT: APPROVE"
@pytest.fixture(autouse=True)
def _isolate_model_env(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.delenv("PORTFOLIO_MODEL_MAP", raising=False)
monkeypatch.delenv("PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT", raising=False)
def _factory(role: str) -> BaseChatClient:
return SyntheticUsageChatClient(default_reply=_CHECKER if role == "checker" else _PROPOSER)
class _Fn:
def __init__(self, name: str) -> None:
self.name = name
class _Ctx:
def __init__(self, name: str, arguments: Any) -> None:
self.function = _Fn(name)
self.arguments = arguments
def _record(*calls: tuple[str, Any]) -> list[ex.ToolCall]:
import asyncio
sink: list[ex.ToolCall] = []
recorder = ex.ExplorationToolRecorder(sink)
async def _noop() -> None:
return None
for name, args in calls:
asyncio.run(recorder.process(_Ctx(name, args), _noop)) # type: ignore[arg-type]
return sink
# ---------------------------------------------------------------------------------------------
# (a)-(d) DEL C
# ---------------------------------------------------------------------------------------------
def test_the_window_arguments_are_recorded() -> None:
"""(a) HOW the level was asked for, not only which one."""
(call,) = _record(
("read_dir", {"bundle_id": "k2", "path": "krav/N100", "filter": "rundkjoring", "limit": 25})
)
assert (call.filter, call.offset, call.limit) == ("rundkjoring", 0, 25)
def test_a_call_that_passes_none_of_them_yields_empty_fields() -> None:
"""(b) Never a refusal, never an absent key — the ``path``/``bundle_id`` rule one field over."""
(call,) = _record(("list_bundles", {}))
assert (call.filter, call.offset, call.limit) == ("", 0, 0)
payload = ex.tool_call_payload([call])
assert payload[0]["filter"] == "" and payload[0]["offset"] == 0 and payload[0]["limit"] == 0
def test_a_numeric_argument_sent_as_a_string_is_coerced() -> None:
"""(c) A model may send either shape; a recorder that read one would misreport the other."""
(call,) = _record(("read_dir", {"bundle_id": "k2", "path": "p", "offset": "30", "limit": "10"}))
assert (call.offset, call.limit) == (30, 10)
# …and a value that is neither is 0, never an invented number.
(other,) = _record(("read_dir", {"bundle_id": "k2", "path": "p", "limit": True}))
assert other.limit == 0
# ---------------------------------------------------------------------------------------------
# a synthetic base + outbox, self-contained (nothing here touches a real bundle or a model)
# ---------------------------------------------------------------------------------------------
_GOOD = "krav/N1/id-good.md"
def _minibase(root: Path) -> Path:
base = root / "minibase"
(base / "krav" / "N1").mkdir(parents=True)
(base / "index.md").write_text(
"---\nbundle_id: minibase\n---\n\n- [good](krav/N1/id-good.md)\n", encoding="utf-8"
)
(base / _GOOD).write_text(
'---\ntype: concept\ntitle: "T"\nreq_number: "Krav 1.2.3-4"\n---\n\nBody.\n',
encoding="utf-8",
)
return base
def _context_dir(root: Path) -> Path:
ctx = root / "ctx"
(ctx / "docs").mkdir(parents=True)
(ctx / "bundle.txt").write_text("name: minibase\nbundle_id: minibase\n", encoding="utf-8")
(ctx / "mandate.json").write_text(
json.dumps(
{
"objective": "o",
"success_criteria": "s",
"approaches": [
{"id": "a1", "label": "L", "affected_codes": ["CODE-1"]},
{"id": "a2", "label": "M", "affected_codes": ["CODE-2"]},
],
}
),
encoding="utf-8",
)
(ctx / "fasit.json").write_text(
json.dumps(
{
"project_id": "proj",
"bundle": "minibase",
"bundle_id": "minibase",
"must_cite": [
{
"approach_id": "a1",
"rationale": "why",
"concepts": [{"path": _GOOD, "title": "T", "ref": "Krav 1.2.3-4"}],
}
],
"must_refuse": [],
"honesty": "synthetic",
}
),
encoding="utf-8",
)
return ctx
def _outbox(root: Path, *, tokens: int, tool_calls: list[dict[str, Any]]) -> Path:
out = root / "out"
out.mkdir(parents=True, exist_ok=True)
(out / "r1-a1-proposal.json").write_text(
json.dumps(
{
"proposal": {
"project_id": "proj",
"measure": "m",
"affected_items": [{"code": "CODE-1", "quantity": 1.0, "unit_cost": 2.0}],
"claimed_saving_nok": 1.0,
"assumptions": {},
},
"provenance": {"citations": [], "token_usage": tokens},
}
),
encoding="utf-8",
)
(out / "r1-a1-outcome.json").write_text(
json.dumps({"outcome_type": "rejected", "reason": "no"}), encoding="utf-8"
)
(out / "r1-debate.json").write_text(
json.dumps({"run_id": "r1", "tool_calls": tool_calls}), encoding="utf-8"
)
return out
def test_the_judge_counts_filtered_and_paged_calls(tmp_path: Path) -> None:
"""(d) Readable directly, instead of inferred from which documents fell outside a default."""
calls = [
{
"name": "read_dir",
"bundle_id": "b",
"path": "p",
"filter": "rund",
"offset": 0,
"limit": 0,
},
{"name": "read_dir", "bundle_id": "b", "path": "p", "filter": "", "offset": 30, "limit": 0},
{
"name": "read_file",
"bundle_id": "b",
"path": _GOOD,
"filter": "",
"offset": 0,
"limit": 0,
},
]
verdict = stress.score_context_set(
_context_dir(tmp_path),
_outbox(tmp_path, tokens=1234, tool_calls=calls),
"r1",
_minibase(tmp_path),
)
assert (verdict.filter_calls, verdict.paged_calls) == (1, 1)
assert verdict.tool_calls_seen == 3
# ---------------------------------------------------------------------------------------------
# (e)-(g) DEL D
# ---------------------------------------------------------------------------------------------
def test_the_coverage_artefact_carries_the_rows_and_the_stop_reason(tmp_path: Path) -> None:
"""(e)/(f) A required stop reason, never an inferred one: "finished" and "we never found out"
must not be the same value."""
outbox.write_coverage(
str(tmp_path),
"r1",
rows=[
{"id": "a1", "label": "L", "status": "validated", "detail": "", "saving_nok": 10.0},
{
"id": "a2",
"label": "M",
"status": "not_evaluated",
"detail": "budget",
"saving_nok": None,
},
],
stop_reason="tokens",
)
payload = json.loads((tmp_path / "r1-coverage.json").read_text(encoding="utf-8"))
assert payload["stop_reason"] == "tokens"
assert [r["status"] for r in payload["rows"]] == ["validated", "not_evaluated"]
# Byte-deterministic: the same inputs twice produce the same file (write_run_config's rule).
first = (tmp_path / "r1-coverage.json").read_bytes()
outbox.write_coverage(
str(tmp_path),
"r1",
rows=[
{"id": "a1", "label": "L", "status": "validated", "detail": "", "saving_nok": 10.0},
{
"id": "a2",
"label": "M",
"status": "not_evaluated",
"detail": "budget",
"saving_nok": None,
},
],
stop_reason="tokens",
)
assert (tmp_path / "r1-coverage.json").read_bytes() == first
def test_the_judge_reads_the_stop_reason_onto_every_unevaluated_row(tmp_path: Path) -> None:
"""(f) The second half: a row with no artefact says WHY, instead of leaving a reader to guess."""
ctx = _context_dir(tmp_path)
out = _outbox(tmp_path, tokens=45_642, tool_calls=[])
base = _minibase(tmp_path)
# Before any coverage file exists — every run written before today.
blind = stress.score_context_set(ctx, out, "r1", base)
assert blind.stop_reason == "absent"
assert [r.not_evaluated_reason for r in blind.approaches] == ["", "absent"]
outbox.write_coverage(
str(out),
"r1",
rows=[
{"id": "a2", "label": "M", "status": "not_evaluated", "detail": "b", "saving_nok": None}
],
stop_reason="rounds",
)
informed = stress.score_context_set(ctx, out, "r1", base)
assert informed.stop_reason == "rounds"
assert [r.not_evaluated_reason for r in informed.approaches] == ["", "rounds"]
def test_the_judge_reports_what_the_run_spent(tmp_path: Path) -> None:
"""(g) D1. P18's finding 4 was wrong as written: the field was always there, the READER was not.
The figure is the one measured on round 2's own artefacts (tunnel-hauglia-2027-02).
"""
verdict = stress.score_context_set(
_context_dir(tmp_path),
_outbox(tmp_path, tokens=45_642, tool_calls=[]),
"r1",
_minibase(tmp_path),
)
assert verdict.token_usage == 45_642
# ---------------------------------------------------------------------------------------------
# (e) the run itself writes it -- without this arm the wiring has no witness
# ---------------------------------------------------------------------------------------------
async def test_a_commissioned_run_writes_the_coverage_artefact(tmp_path: Path) -> None:
"""(e) END-TO-END. The arm above drives ``write_coverage`` directly and would stay green with
the ``run_project`` wiring detached -- the vacuous-gate shape this repo has now met two dozen
times. This one runs the real pipeline."""
out = tmp_path / "with-mandate"
await run_project(
_PID,
"local",
docs_dir=str(_BUNDLE_DIR),
bundle_dir=str(_BUNDLE_DIR),
client_factory=_factory,
max_rounds=2,
outbox_dir=str(out),
run_id="r1",
mandate=Mandate(
objective="o",
approaches=(Approach(id="a1", label="LED", description="swap the fittings"),),
allow_own_proposals=False,
),
)
payload = json.loads((out / "r1-coverage.json").read_text(encoding="utf-8"))
assert payload["stop_reason"] == "", "nothing cut this run short"
assert [r["id"] for r in payload["rows"]] == ["a1"]
assert payload["rows"][0]["status"] in {"validated", "rejected"}
async def test_a_mandateless_run_leaves_the_outbox_byte_identical(tmp_path: Path) -> None:
"""(e), the other half. Coverage is the MANDATE's report; a run without one would be described
by a file saying nothing, and two older tests pin an exact listing on such a run."""
out = tmp_path / "no-mandate"
await run_project(
_PID,
"local",
docs_dir=str(_BUNDLE_DIR),
bundle_dir=str(_BUNDLE_DIR),
client_factory=_factory,
max_rounds=2,
outbox_dir=str(out),
run_id="r1",
)
assert not (out / "r1-coverage.json").exists()
async def test_a_cap_that_cuts_the_run_short_is_named_in_the_artefact(tmp_path: Path) -> None:
"""(f) END-TO-END, and this arm exists because the direct one was VACUOUS.
MEASURED: mutation D-i ``stop_reason=""`` whatever happened left the WHOLE suite green
(1743/5), because the arm above calls ``write_coverage`` itself and therefore chooses the
reason it then asserts. Only a run that a cap actually cut short can tell the two apart, and
it is exactly the run for which the artefact matters: a commission that stopped halfway.
"""
out = tmp_path / "capped"
await run_project(
_PID,
"local",
docs_dir=str(_BUNDLE_DIR),
bundle_dir=str(_BUNDLE_DIR),
client_factory=_factory,
max_rounds=2,
outbox_dir=str(out),
run_id="r1",
# tick_round fires once per generation ATTEMPT: one round pays for the first
# approach, the second is the one the cap cuts.
meter=TokenMeter(Budget(max_tokens=10_000_000, max_rounds=1)),
mandate=Mandate(
objective="o",
approaches=(
Approach(id="a1", label="LED", description="one"),
Approach(id="a2", label="Sensors", description="two"),
),
allow_own_proposals=False,
),
)
payload = json.loads((out / "r1-coverage.json").read_text(encoding="utf-8"))
assert payload["stop_reason"] == "rounds", payload
assert [r["status"] for r in payload["rows"]][-1] == "not_evaluated", payload

View file

@ -304,9 +304,10 @@ async def test_a_refused_read_is_recorded_and_leaks_nothing(tmp_path) -> None:
"nobody can audit" "nobody can audit"
) )
payload = json.loads((outbox_dir / "verdict-gate-debate.json").read_text(encoding="utf-8")) payload = json.loads((outbox_dir / "verdict-gate-debate.json").read_text(encoding="utf-8"))
assert {"name": "read_file", "bundle_id": bundle_id, "path": _VERDICT_FILE} in payload[ identity = [{k: c[k] for k in ("name", "bundle_id", "path")} for c in payload["tool_calls"]]
"tool_calls" assert {"name": "read_file", "bundle_id": bundle_id, "path": _VERDICT_FILE} in identity, (
], f"the artefact does not carry the refused call: {payload['tool_calls']}" f"the artefact does not carry the refused call: {payload['tool_calls']}"
)
# ------------------------------------------------------ (5) the document, not the walk, decides # ------------------------------------------------------ (5) the document, not the walk, decides