feat(p19): the trace says HOW, and a run says what it spent and why it stopped
DEL C. P18 gave read_dir a window (filter/offset/limit) and then measured its
own paid round without being able to see it used: five of 31 documents read
lay outside the default window, so the window HAD been widened and the trace
could not say with which knob. ToolCall now carries the three arguments,
always present and empty/zero when not passed -- an absent key and "not
narrowed" must not read the same -- and the judge counts filter_calls and
paged_calls. _number_argument is a SIBLING of _string_argument, not a widening
of it: a model may send limit as 10 or as "10", and a reader that knew one
shape would report a paged call as unpaged.
DEL D. P18's finding 4 was WRONG AS WRITTEN. provenance.token_usage has been
stamped on every proposal artefact since S3.4 and stands in every one of round
2's; what was missing is a READER. The judge reads it now (round 2 measured:
289 054 tokens against round 1's 2 679 305, -89 %), and the P18 report gets a
dated correction UNDER its original paragraph rather than instead of it.
What was genuinely absent is {run_id}-coverage.json. settle prints the
coverage report and ApproachOutcome has carried not_evaluated since Trekk A3,
but neither ever reached a file, so a judge could see an approach had no
artefact and could not tell a budget stop from an approach nobody ordered.
Written from the finally IFF a mandate was given. stop_reason comes from a
CALLER-OWNED sink rather than from in_flight, and that is a measurement:
_evaluate_mandate SWALLOWS BudgetExceeded once something has been produced, so
run_project's own in_flight never sees it.
Load-bearing measured (10 arms), four mutations all red against the whole
suite, green control 1744/5 and the golden byte-unchanged. D-i stood GREEN
first -- the vacuous-gate class, 25th time: the arm called write_coverage
itself and therefore chose the reason it then asserted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
d74f32dc1c
commit
4c6084e5df
12 changed files with 684 additions and 11 deletions
38
CLAUDE.md
38
CLAUDE.md
|
|
@ -2690,6 +2690,44 @@ Python ≥3.10. MAF (`agent-framework-core` 1.16.0, `-orchestrations` 1.1.1 —
|
|||
er en kode som telles som prosa — feilretningen er nekt, ikke aksept, og dét er hva
|
||||
generalitetsvernet og baseline-unntaket finnes for); og `assumptions`-nøkler er fortsatt utenfor
|
||||
(økt 109s grense).
|
||||
- **Sporet sier HVORDAN nivået ble spurt om, og en kjøring sier hva den brukte og hvorfor den
|
||||
stoppet (P19 DEL C + DEL D, 15.09):** P18 ga `read_dir` et VINDU (`filter`/`offset`/`limit`) og
|
||||
målte så sin egen betalte runde uten å kunne se det brukt — fem av 31 leste dokumenter lå UTENFOR
|
||||
default-vinduet, altså var vinduet utvidet, og sporet kunne ikke si med hvilken knapp. `ToolCall`
|
||||
bærer nå de tre argumentene, `tool_call_payload` renderer dem (alltid til stede, tomme/null når
|
||||
de ikke ble sendt — `write_debate_tools`-regelen: en fraværende nøkkel og «ikke innsnevret» må
|
||||
ikke leses likt), og dommeren teller `filter_calls`/`paged_calls`. **`_number_argument` er en
|
||||
SØSKEN av `_string_argument`, ikke en utvidelse av den:** en modell kan sende `limit` som `10`
|
||||
eller `"10"`, og en leser som bare kjente den ene formen ville rapportert et paginert kall som
|
||||
upaginert; `bool` er eksplisitt ekskludert, fordi et flagg ikke er et vindu.
|
||||
**P18s FUNN 4 VAR FEIL SOM FORMULERT, og rettelsen er en TILFØYELSE:** `provenance.token_usage`
|
||||
er stemplet på HVERT `-proposal.json` siden S3.4, også ved rc 0 — det som manglet var en LESER.
|
||||
`stress.py` leser det nå (målt runde 2: 35 406 · 45 642 · 44 468 · 73 627 · 89 911 = **289 054**
|
||||
mot runde 1s 2 679 305, **−89 %**), og P18-rapportens § 7 har fått en datert rettelse UNDER det
|
||||
opprinnelige avsnittet, aldri i stedet for det. **Det som GENUINT manglet er `{run_id}-coverage
|
||||
.json`:** `settle` printer coverage-rapporten og `ApproachOutcome` har båret `not_evaluated`
|
||||
siden Trekk A3, men ingen av dem nådde en fil, så en dommer kunne se at en approach manglet
|
||||
artefakt og ikke skille et budsjettstopp fra en approach ingen bestilte — nøyaktig stillheten
|
||||
`ApproachOutcome` finnes for, ett lag ut. Skrives fra `finally` **IFF et mandat ble gitt**
|
||||
(coverage ER mandatets rapport; uten et er det ingen approaches, og en mandatløs kjøring lar
|
||||
utboksen stå byte-identisk — to eldre tester pinner en eksakt listing). **`stop_reason` kommer
|
||||
fra en KALLER-EID SINK, ikke fra `in_flight`, og dét er en MÅLING:** `_evaluate_mandate` SVELGER
|
||||
`BudgetExceeded` så snart noe er produsert (de nådde approachene er et ekte resultat), så
|
||||
`run_project`s egen `in_flight` ser den aldri — et coverage-artefakt som sa «ingenting stoppet
|
||||
denne kjøringen» om en kommisjon kappet på midten ville vært stillheten artefaktet finnes for.
|
||||
Dommerens `not_evaluated_reason` er `rounds`/`tokens` fra fila, `absent` når ingen fil finnes
|
||||
(hver kjøring før i dag), og `""` for en rad som FAKTISK ble evaluert. Load-bearing MÅLT
|
||||
(`tests/test_trace_and_coverage_loadbearing.py`, 10 armer), **fire mutasjoner alle røde mot HELE
|
||||
suiten** + grønn kontroll **1744/5** (fra 1734/5, supersett, 0 fjernet) og golden BYTE-UENDRET
|
||||
(`shasum -a 1` av INNHOLDET = `ea8c534773acdbe41ae68f2c55724d69aaf8be4f`): C-i vindus-argumentene
|
||||
registreres ikke (2) · D-i coverage skrives med tom grunn (1) · D-ii kjøringen skriver den aldri
|
||||
(2) · D-iii dommeren slutter å lese forbruket (1). **ÉN MUTASJON STO GRØNN FØRST (repoets
|
||||
vakuøs-gate-klasse, TJUEFEMTE gang):** D-i lot HELE suiten stå grønn, fordi armen kalte
|
||||
`write_coverage` selv og dermed VALGTE grunnen den så asserterte på; bare en kjøring et tak
|
||||
faktisk kappet kan skille de to, og den armen driver nå `run_project` med `max_rounds=1` (én
|
||||
runde betaler første approach, den andre er den taket kutter). **Ærlighets-grense, uttalt:**
|
||||
`token_usage` er kjøringens ENE teller — den skiller ikke debatt fra generering, og en
|
||||
per-fase-fordeling ville krevd en andre måler.
|
||||
- **STATE.md er local-only** (gitignored). Voyage session-state er efemert; STATE.md er kanonisk kontinuitet.
|
||||
- Prosess: Voyage-plugin (`/trekbrief → /trekplan → /trekexecute → /trekreview`) per større fase.
|
||||
|
||||
|
|
|
|||
10
README.md
10
README.md
|
|
@ -718,6 +718,16 @@ when the seam is detached, so the loop cannot silently degrade into theater.
|
|||
run also writes `{run_id}-exploration.json`: the per-round ledger, the plan reviews and the
|
||||
in-loop advisory verdicts, written even when a cap cut the exploration short.
|
||||
|
||||
Every tool call recorded in `{run_id}-exploration.json` and `{run_id}-debate.json` says HOW the
|
||||
level was asked for — `filter`, `offset` and `limit` beside the tool name, the base and the path
|
||||
— so "did the model narrow the level, or page through it" is readable from the artefact rather
|
||||
than inferred from which documents happened to fall outside a default window.
|
||||
|
||||
A run given a mandate also writes `{run_id}-coverage.json`: one row per commissioned approach
|
||||
with its status and detail, including the ones the run never reached, plus the `stop_reason`
|
||||
(`tokens` / `rounds`) when a cap cut the commission short. A run without a mandate writes no
|
||||
such file — coverage is the mandate's report.
|
||||
|
||||
The hosted surface takes the same door as `explore_prompt` + `explore_contract` on
|
||||
`POST /invocations`.
|
||||
|
||||
|
|
|
|||
|
|
@ -260,6 +260,34 @@ brukte 2 679 305 tokens på sju kjøringer ≈ NOK 15 (anslag fra listepris). Ru
|
|||
med kortere veggtid og billigere listinger; **et anslag på under NOK 10 er en antakelse, ikke en
|
||||
måling**, og oppgis bare fordi ordren ber om et anslag med uttalt antakelse.
|
||||
|
||||
### Rettelse 15.09 (P19 D1) — feltet FANTES, og tallene står her
|
||||
|
||||
**Setningen over er feil som skrevet, og funn 4 var feil som formulert.**
|
||||
`provenance.token_usage` (`provenance.py:60`, satt i `run.py` fra `meter.tokens`) er stemplet på
|
||||
HVERT `-proposal.json` siden S3.4, også ved rc 0, og står i hver eneste av runde 2 sine. Det som
|
||||
manglet var ikke feltet, men en LESER: ingenting i treet leste det. P19 D1 gir `stress.py` den
|
||||
lesningen, og `{run_id}-verdict.json` bærer nå `token_usage`.
|
||||
|
||||
Målt 15.09 mot artefaktene runde 2 etterlot:
|
||||
|
||||
| kjøring | token_usage |
|
||||
| --- | --- |
|
||||
| gate-nordvik-2027-02 | 35 406 |
|
||||
| tunnel-hauglia-2027-02 | 45 642 |
|
||||
| fv412-dekkefornyelse-2027-02 | 44 468 |
|
||||
| kontrakt-sorasen-2027-02 | 73 627 |
|
||||
| gate-nordvik-2027-03 | 89 911 |
|
||||
| **sum** | **289 054** |
|
||||
|
||||
Mot runde 1 sine **2 679 305** (§ 3 i P16-rapporten) er det **−89 %**, og det er den største målte
|
||||
effekten av P18s navigasjonsarbeid. Anslaget «under NOK 10» står som anslag; det som ikke lenger er
|
||||
en antakelse er forbruket. **Tilføyelse, ikke omskriving:** avsnittet over står som det ble
|
||||
skrevet, fordi en rapport som retter seg selv i stillhet ikke er en rapport.
|
||||
|
||||
Det som IKKE fantes og nå er bygget (P19 D2) er den andre halvdelen: `settle` printer
|
||||
coverage-rapporten og `ApproachOutcome` har båret `not_evaluated` siden Trekk A3, men ingen av dem
|
||||
nådde en FIL. `{run_id}-coverage.json` bærer dem nå, med `stop_reason` fra `BudgetExceeded.kind`.
|
||||
|
||||
---
|
||||
|
||||
## 8. Verifisering
|
||||
|
|
|
|||
|
|
@ -343,6 +343,15 @@ class ToolCall:
|
|||
name: str
|
||||
bundle_id: str
|
||||
path: str
|
||||
#: P19 DEL C — HOW the listing was asked for, not just WHICH one. P18 gave ``read_dir`` a
|
||||
#: window (``filter``/``offset``/``limit``) and the trace could not say whether a model used
|
||||
#: it: five of 31 documents read in round 2 lay outside the default window, so the window HAD
|
||||
#: been widened, and nothing said with which knob. Empty/zero mean "not passed" — the
|
||||
#: ``bundle_id``/``path`` rule one field over, and unambiguous here because ``limit`` is
|
||||
#: clamped to at least one wherever it is given.
|
||||
filter: str = ""
|
||||
offset: int = 0
|
||||
limit: int = 0
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
|
|
@ -393,6 +402,28 @@ def _string_argument(arguments: Any, key: str) -> str:
|
|||
return value if isinstance(value, str) else ""
|
||||
|
||||
|
||||
def _number_argument(arguments: Any, key: str) -> int:
|
||||
"""One NUMERIC argument of a call — ``_string_argument``'s sibling, and deliberately separate.
|
||||
|
||||
A model may send ``limit`` as ``10`` or as ``"10"`` (both reach a tool through the same wire),
|
||||
so a recorder that read only the first shape would say a paging call was unpaged. Anything that
|
||||
is neither is ``0``: the recorder describes the call, and inventing a number for an argument
|
||||
nobody passed would be the false attribution ``_string_argument`` refuses for its own field.
|
||||
``bool`` is excluded explicitly because it is an ``int`` in Python and a flag is not a window.
|
||||
"""
|
||||
if isinstance(arguments, Mapping):
|
||||
value: Any = arguments.get(key)
|
||||
else:
|
||||
value = getattr(arguments, key, None)
|
||||
if isinstance(value, bool):
|
||||
return 0
|
||||
if isinstance(value, int):
|
||||
return value
|
||||
if isinstance(value, str) and value.strip().lstrip("-").isdigit():
|
||||
return int(value.strip())
|
||||
return 0
|
||||
|
||||
|
||||
class ExplorationToolRecorder(FunctionMiddleware):
|
||||
"""Records WHICH exploration tool an agent actually called, in the order it called them.
|
||||
|
||||
|
|
@ -424,6 +455,9 @@ class ExplorationToolRecorder(FunctionMiddleware):
|
|||
name=name,
|
||||
bundle_id=_string_argument(arguments, "bundle_id"),
|
||||
path=_string_argument(arguments, "path"),
|
||||
filter=_string_argument(arguments, "filter"),
|
||||
offset=_number_argument(arguments, "offset"),
|
||||
limit=_number_argument(arguments, "limit"),
|
||||
)
|
||||
)
|
||||
await call_next()
|
||||
|
|
@ -479,7 +513,20 @@ def tool_call_payload(calls: Sequence[ToolCall]) -> list[dict[str, Any]]:
|
|||
Plain mappings only, so the RAW output layer stays MAF-free (``outbox.py`` may not import
|
||||
this module).
|
||||
"""
|
||||
return [{"name": call.name, "bundle_id": call.bundle_id, "path": call.path} for call in calls]
|
||||
return [
|
||||
{
|
||||
"name": call.name,
|
||||
"bundle_id": call.bundle_id,
|
||||
"path": call.path,
|
||||
# P19 DEL C: HOW the level was asked for. Always present, zero/empty when not passed —
|
||||
# the ``write_debate_tools`` rule one field down: an absent key and "not narrowed" must
|
||||
# not be the same reading.
|
||||
"filter": call.filter,
|
||||
"offset": call.offset,
|
||||
"limit": call.limit,
|
||||
}
|
||||
for call in calls
|
||||
]
|
||||
|
||||
|
||||
def trace_payload(
|
||||
|
|
|
|||
|
|
@ -312,6 +312,43 @@ def write_proposal_reviews(
|
|||
return path
|
||||
|
||||
|
||||
def write_coverage(
|
||||
outbox_dir: str,
|
||||
run_id: str,
|
||||
*,
|
||||
rows: Sequence[Mapping[str, Any]],
|
||||
stop_reason: str,
|
||||
) -> Path:
|
||||
"""Write ``{run_id}-coverage.json`` — WHY each commissioned approach ended as it did (P19 D2).
|
||||
|
||||
``settle`` prints the coverage report and ``ApproachOutcome`` has carried ``not_evaluated``
|
||||
since Trekk A3, but neither ever reached a FILE: measured 14.09, a judge reading an outbox could
|
||||
see that an approach had no artefact and could not tell a budget stop from an approach nobody
|
||||
ordered. That is the very silence ``ApproachOutcome`` exists to remove, one layer out.
|
||||
|
||||
``stop_reason`` is ``BudgetExceeded.kind`` when a cap cut the run short (``tokens`` /
|
||||
``rounds`` / the portfolio's own kinds) and ``""`` when nothing did. A REQUIRED argument rather
|
||||
than an inferred one, for ``cost_baseline_anchored``'s reason: "the run finished" and "we never
|
||||
found out" must not be the same value.
|
||||
|
||||
Byte-deterministic and wall-clock-free, mirroring ``write_run_config``; plain data only, so the
|
||||
RAW output layer stays MAF-free."""
|
||||
directory = Path(outbox_dir)
|
||||
directory.mkdir(parents=True, exist_ok=True)
|
||||
path = directory / f"{run_id}-coverage.json"
|
||||
path.write_text(
|
||||
_dump(
|
||||
{
|
||||
"run_id": run_id,
|
||||
"stop_reason": stop_reason,
|
||||
"rows": [dict(row) for row in rows],
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
return path
|
||||
|
||||
|
||||
def write_run_config(
|
||||
config_dir: str,
|
||||
run_id: str,
|
||||
|
|
|
|||
|
|
@ -509,6 +509,7 @@ def _select_outcome(
|
|||
async def _evaluate_mandate(
|
||||
mandate: Mandate,
|
||||
evaluate: Callable[[Approach | None], Awaitable[ValidatedProposal | Rejection]],
|
||||
budget_stops: list[str] | None = None,
|
||||
) -> tuple[
|
||||
ValidatedProposal | Rejection,
|
||||
tuple[ApproachOutcome, ...],
|
||||
|
|
@ -541,9 +542,16 @@ async def _evaluate_mandate(
|
|||
for index, (row_id, label, approach) in enumerate(plan):
|
||||
try:
|
||||
outcome = await evaluate(approach)
|
||||
except BudgetExceeded:
|
||||
except BudgetExceeded as stop:
|
||||
if not produced:
|
||||
raise
|
||||
# P19 D2: WHICH cap bound, recorded on a caller-owned sink before the rows are built.
|
||||
# The exception is SWALLOWED here (the approaches that were reached are a real result),
|
||||
# so ``run_project``'s own ``in_flight`` never sees it — and a coverage artefact that
|
||||
# said "nothing stopped this run" about a commission cut in half would be the silence
|
||||
# the artefact exists to remove.
|
||||
if budget_stops is not None:
|
||||
budget_stops.append(stop.kind)
|
||||
rows.extend(
|
||||
ApproachOutcome(
|
||||
id=rid,
|
||||
|
|
@ -1452,6 +1460,10 @@ async def run_project(
|
|||
# on the very attempt a revise bought, and on that path ``generate_via_llm`` returns nothing —
|
||||
# so the run whose record matters most is exactly the one a return value cannot reach.
|
||||
expert_reviews: list[ProposalReview] = []
|
||||
# P19 D2: which cap, if any, cut the commission short. Caller-owned for the reason every other
|
||||
# sink here is: ``_evaluate_mandate`` SWALLOWS the stop once something has been produced, so a
|
||||
# return value would not reach the ``finally`` that writes the artefact.
|
||||
budget_stops: list[str] = []
|
||||
|
||||
async def _evaluate(approach: Approach | None) -> ValidatedProposal | Rejection:
|
||||
# Which candidate the expert is being asked about. With a mandate every entry is keyed —
|
||||
|
|
@ -1496,7 +1508,9 @@ async def run_project(
|
|||
if mandate is None:
|
||||
validator_outcome = await _evaluate(None)
|
||||
else:
|
||||
validator_outcome, coverage, evaluated = await _evaluate_mandate(mandate, _evaluate)
|
||||
validator_outcome, coverage, evaluated = await _evaluate_mandate(
|
||||
mandate, _evaluate, budget_stops
|
||||
)
|
||||
except BaseException as stop:
|
||||
# Recorded and re-raised UNTOUCHED. This arm decides nothing about the exception itself —
|
||||
# only what the two writers in the ``finally`` are allowed to do to it (``_write_or_report``).
|
||||
|
|
@ -1518,6 +1532,39 @@ async def run_project(
|
|||
what=f"{run_id}-parse-failures.json",
|
||||
in_flight=in_flight,
|
||||
)
|
||||
# Same ``finally``, a THIRD write rule: IFF a mandate was given, including when the run
|
||||
# stopped before a single approach was evaluated (P19 D2). Coverage is the MANDATE's
|
||||
# report by construction — without one there are no approaches and the file would describe
|
||||
# nothing — so a mandate-less run leaves the outbox byte-identical, which two existing
|
||||
# tests pin as an exact listing. The stop reason comes from the in-flight exception rather
|
||||
# than being inferred: a ``BudgetExceeded`` carries ``kind`` as a field precisely so that
|
||||
# "which cap bound" is readable by machine (kø-(y)), and a run that finished says so with
|
||||
# an empty string rather than with a missing key.
|
||||
if outbox_dir is not None and mandate is not None:
|
||||
assert run_id is not None # narrowed by the step-0 guard (no wall-clock default)
|
||||
_write_or_report(
|
||||
lambda: outbox.write_coverage(
|
||||
outbox_dir,
|
||||
run_id,
|
||||
rows=[
|
||||
{
|
||||
"id": row.id,
|
||||
"label": row.label,
|
||||
"status": row.status,
|
||||
"detail": row.detail,
|
||||
"saving_nok": row.saving_nok,
|
||||
}
|
||||
for row in coverage
|
||||
],
|
||||
stop_reason=(
|
||||
budget_stops[0]
|
||||
if budget_stops
|
||||
else (in_flight.kind if isinstance(in_flight, BudgetExceeded) else "")
|
||||
),
|
||||
),
|
||||
what=f"{run_id}-coverage.json",
|
||||
in_flight=in_flight,
|
||||
)
|
||||
# Same ``finally``, different write rule: IFF a reviewer was given, including when the
|
||||
# list is empty (D4). A reviewer-less run must leave the outbox byte-identical, while a
|
||||
# reviewer that was offered and never consulted is a fact the artefact must be able to
|
||||
|
|
|
|||
|
|
@ -125,6 +125,12 @@ class ApproachVerdict:
|
|||
#: wrote one, and RE-DERIVED with the same classifier when it did not, so rounds 1 and 2 -
|
||||
#: written before the field existed - can be re-judged with the same instrument.
|
||||
prose_codes: tuple[str, ...]
|
||||
#: P19 D2 - WHY this row was not evaluated: ``rounds`` / ``tokens`` when a cap cut the run
|
||||
#: short, ``absent`` when the artefact is simply missing and no coverage file says otherwise,
|
||||
#: and ``""`` for a row that WAS evaluated. Before this, "no artefact" could not be told from
|
||||
#: "an approach nobody ordered" -- the silence ``ApproachOutcome`` exists to remove, one layer
|
||||
#: out, and it reached no file until ``{run_id}-coverage.json``.
|
||||
not_evaluated_reason: str
|
||||
ferdig: bool
|
||||
|
||||
|
||||
|
|
@ -153,6 +159,19 @@ class ContextSetVerdict:
|
|||
#: none" is the measurement, and a missing field would be indistinguishable from a judge that
|
||||
#: did not look.
|
||||
requirements_declared: tuple[str, ...]
|
||||
#: P19 DEL C - how the run asked for its listings. ``filter_calls`` is how many calls narrowed
|
||||
#: a level by word, ``paged_calls`` how many asked for a window other than the default. P18 § 1
|
||||
#: could only infer that the window HAD been widened (five of 31 documents read lay outside the
|
||||
#: default) and never with which knob; these two make it readable directly.
|
||||
filter_calls: int
|
||||
paged_calls: int
|
||||
#: P19 D1 - what the run SPENT, read off ``provenance.token_usage``, which has been stamped on
|
||||
#: every proposal artefact since S3.4 and which P18's report wrongly said could not be given.
|
||||
#: ``0`` when no artefact carried one.
|
||||
token_usage: int
|
||||
#: P19 D2 - ``BudgetExceeded.kind`` when a cap cut the run short, ``""`` when nothing did, and
|
||||
#: ``"absent"`` when the run wrote no coverage file at all (every run before today).
|
||||
stop_reason: str
|
||||
tool_calls_seen: int
|
||||
citations_seen: int
|
||||
approach_rows_seen: int
|
||||
|
|
@ -269,10 +288,18 @@ def score_context_set(
|
|||
hallucinated_reads.append(raw)
|
||||
reads_clean = not hallucinated_reads
|
||||
|
||||
# ---- P19 D1/D2: what the run spent, and why it stopped -----------------------------------
|
||||
coverage_path = outbox / f"{run_id}-coverage.json"
|
||||
stop_reason = (
|
||||
str(_read_json(coverage_path).get("stop_reason", "")) if coverage_path.is_file() else ""
|
||||
)
|
||||
coverage_seen = coverage_path.is_file()
|
||||
|
||||
# ---- per approach ------------------------------------------------------------------------
|
||||
rows: list[ApproachVerdict] = []
|
||||
citations_seen = 0
|
||||
rows_seen = 0
|
||||
token_usage = 0
|
||||
validated_codes: set[str] = set()
|
||||
validated_ids: set[str] = set()
|
||||
|
||||
|
|
@ -299,6 +326,7 @@ def score_context_set(
|
|||
requirement_source=_attributable(approach, declared_paths)[1],
|
||||
requirement_hit=bool(set(_attributable(approach, declared_paths)[0]) & wanted),
|
||||
prose_codes=(),
|
||||
not_evaluated_reason=stop_reason or "absent",
|
||||
ferdig=False,
|
||||
)
|
||||
)
|
||||
|
|
@ -306,6 +334,7 @@ def score_context_set(
|
|||
|
||||
rows_seen += 1
|
||||
payload = _read_json(proposal_path)
|
||||
token_usage = max(token_usage, int(payload.get("provenance", {}).get("token_usage", 0)))
|
||||
proposal = payload.get("proposal", {})
|
||||
citations = payload.get("provenance", {}).get("citations", [])
|
||||
citations_seen += len(citations)
|
||||
|
|
@ -375,6 +404,7 @@ def score_context_set(
|
|||
requirement_source=requirement_source,
|
||||
requirement_hit=requirement_hit,
|
||||
prose_codes=prose_codes,
|
||||
not_evaluated_reason="",
|
||||
ferdig=(
|
||||
grounded
|
||||
and (named_in_measure or named_in_snippet)
|
||||
|
|
@ -421,6 +451,12 @@ def score_context_set(
|
|||
must_refuse=tuple(refusals),
|
||||
hallucinated_reads=tuple(hallucinated_reads),
|
||||
requirements_declared=tuple(declared_paths),
|
||||
token_usage=token_usage,
|
||||
stop_reason=stop_reason if coverage_seen else "absent",
|
||||
filter_calls=sum(1 for c in tool_calls if str(c.get("filter", ""))),
|
||||
paged_calls=sum(
|
||||
1 for c in tool_calls if int(c.get("offset", 0) or 0) or int(c.get("limit", 0) or 0)
|
||||
),
|
||||
tool_calls_seen=len(tool_calls),
|
||||
citations_seen=citations_seen,
|
||||
approach_rows_seen=rows_seen,
|
||||
|
|
|
|||
|
|
@ -262,7 +262,10 @@ async def test_the_debate_tool_trace_reaches_the_result_and_the_outbox(tmp_path)
|
|||
], f"the debate's tool trace is {observed} — the call sequence is not recorded as it happened"
|
||||
|
||||
payload = json.loads((outbox_dir / "s2c-debate.json").read_text(encoding="utf-8"))
|
||||
assert payload["tool_calls"][:2] == [
|
||||
# P19 DEL C added HOW the level was asked for (``filter``/``offset``/``limit``); the
|
||||
# identity of the call is still these three, so the arm compares on them.
|
||||
identity = [{k: c[k] for k in ("name", "bundle_id", "path")} for c in payload["tool_calls"]]
|
||||
assert identity[:2] == [
|
||||
{"name": "read_bundle", "bundle_id": bundle_id, "path": ""},
|
||||
{"name": "read_file", "bundle_id": bundle_id, "path": "metode-ipmvp-a.md"},
|
||||
], f"the artefact does not carry the call sequence: {payload['tool_calls']}"
|
||||
|
|
|
|||
|
|
@ -282,7 +282,10 @@ def test_the_record_leaves_the_run_in_the_artefact_beside_the_advisory_verdicts(
|
|||
|
||||
# ``path`` joined the record in S7a-3 pkt. 3 and is ``""`` here because ``read_bundle`` takes
|
||||
# none; its own gate is ``tests/test_tool_call_path_loadbearing.py``.
|
||||
assert payload["tool_calls"] == [
|
||||
# P19 DEL C added HOW the level was asked for (``filter``/``offset``/``limit``); the
|
||||
# identity of the call is still these three, so the arm compares on them.
|
||||
identity = [{k: c[k] for k in ("name", "bundle_id", "path")} for c in payload["tool_calls"]]
|
||||
assert identity == [
|
||||
{"name": "read_bundle", "bundle_id": "bygg-energi-mikro", "path": ""}
|
||||
], "the artefact must say which bases were opened, or a paid run leaves no record that any were"
|
||||
assert json.dumps(payload), "the payload must stay plain data — the RAW layer is MAF-free"
|
||||
|
|
@ -479,7 +482,8 @@ def test_a_step_list_is_accepted_for_a_debate_role(tmp_path) -> None:
|
|||
payload = json.loads(
|
||||
(tmp_path / "outbox" / "scripted-debate-list-debate.json").read_text(encoding="utf-8")
|
||||
)
|
||||
assert payload["tool_calls"][0] == {
|
||||
first = payload["tool_calls"][0]
|
||||
assert {k: first[k] for k in ("name", "bundle_id", "path")} == {
|
||||
"name": "read_bundle",
|
||||
"bundle_id": "bygg-energi-mikro",
|
||||
"path": "",
|
||||
|
|
|
|||
|
|
@ -99,7 +99,10 @@ def test_the_result_is_still_never_recorded() -> None:
|
|||
sink = _record(("read_file", {"bundle_id": "k2", "path": "a.md"}))
|
||||
|
||||
recorded = json.dumps([vars(c) for c in sink])
|
||||
assert set(vars(sink[0])) == {"name", "bundle_id", "path"}
|
||||
# P19 DEL C added HOW the listing was asked for (``filter``/``offset``/``limit``). The claim
|
||||
# this arm makes is unchanged and is the one that matters: the fields are all ARGUMENTS, and
|
||||
# none of them is the result.
|
||||
assert set(vars(sink[0])) == {"name", "bundle_id", "path", "filter", "offset", "limit"}
|
||||
assert "result" not in recorded
|
||||
|
||||
|
||||
|
|
@ -115,7 +118,16 @@ def test_the_artefact_carries_the_path_beside_the_name_and_the_base() -> None:
|
|||
payload = ex.trace_payload(trace, stop=None, completed=True, mandate=None)
|
||||
|
||||
assert payload["tool_calls"] == [
|
||||
{"name": "read_file", "bundle_id": "k2", "path": "prisskjema.md"}
|
||||
{
|
||||
"name": "read_file",
|
||||
"bundle_id": "k2",
|
||||
"path": "prisskjema.md",
|
||||
# P19 DEL C: always present, zero/empty for a call that did not pass them — an absent
|
||||
# key and "not narrowed" must not read the same.
|
||||
"filter": "",
|
||||
"offset": 0,
|
||||
"limit": 0,
|
||||
}
|
||||
]
|
||||
assert json.dumps(payload), "the payload must stay plain data - the RAW layer is MAF-free"
|
||||
|
||||
|
|
|
|||
410
tests/test_trace_and_coverage_loadbearing.py
Normal file
410
tests/test_trace_and_coverage_loadbearing.py
Normal file
|
|
@ -0,0 +1,410 @@
|
|||
"""P19 DEL C + DEL D — the trace says HOW, and a run says what it spent and why it stopped.
|
||||
|
||||
**DEL C, the measured silence.** P18 gave ``read_dir`` a window — ``filter`` / ``offset`` /
|
||||
``limit`` — and then measured its own paid round without being able to see it used: five of 31
|
||||
documents read lay OUTSIDE the default window, so the window had been widened, and the trace could
|
||||
not say with which knob (``docs/2026-09-14-p18-stressrunde-2.md`` § 1, finding 1). The recorder
|
||||
kept ``name`` / ``bundle_id`` / ``path`` and nothing else. "Did the model narrow the level, or page
|
||||
through it?" is the operative question about a corpus of 2 756 documents, and it was unanswerable
|
||||
from the artefact the run leaves behind.
|
||||
|
||||
**DEL D, and one of the two findings it fixes was WRONG AS WRITTEN.** P18's finding 4 said a
|
||||
successful run does not say what it used. Measured 14.09: ``provenance.token_usage`` has been
|
||||
stamped on every proposal artefact since S3.4 and stands in every one of round 2's. What was
|
||||
missing is a JUDGE that reads it — so the report's sentence is corrected in its own § 7 addendum
|
||||
rather than the field being invented a second time. What was genuinely absent is the OTHER half:
|
||||
``settle`` prints the coverage report, ``ApproachOutcome`` has carried ``not_evaluated`` since
|
||||
Trekk A3, and neither ever reached a file. A judge reading an outbox could see that an approach had
|
||||
no artefact and could not tell a budget stop from an approach nobody ordered — the silence
|
||||
``ApproachOutcome`` exists to remove, one layer out.
|
||||
|
||||
What each arm pins:
|
||||
|
||||
(a) C — the three window arguments are RECORDED, in the shape the wire actually carries;
|
||||
(b) C — a call that passes none of them yields empty/zero fields, never a refusal and never an
|
||||
absent key: "not narrowed" and "we did not look" must not read the same;
|
||||
(c) C — a numeric argument sent as a STRING is coerced. A model may send either, and a recorder
|
||||
that read only one shape would report a paged call as unpaged;
|
||||
(d) C — the judge counts them, so "did the model use the filter" is readable directly rather than
|
||||
inferred from which documents happened to lie outside a default window;
|
||||
(e) D2 — ``{run_id}-coverage.json`` is written IFF a mandate was given, INCLUDING when a cap cut
|
||||
the run short before a single approach was evaluated. A mandate-less run leaves the outbox
|
||||
byte-identical, which two older tests pin as an exact listing;
|
||||
(f) D2 — the stop reason is ``BudgetExceeded.kind``, carried as a field rather than inferred, and
|
||||
the judge turns it into each unevaluated row's ``not_evaluated_reason``;
|
||||
(g) D1 — the judge reads ``provenance.token_usage`` and reports it, with ``absent`` distinguishing
|
||||
a run that wrote no coverage file (every run before today) from one that finished cleanly.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from agent_framework import BaseChatClient
|
||||
from conftest import SyntheticUsageChatClient
|
||||
|
||||
from portfolio_optimiser import explore as ex
|
||||
from portfolio_optimiser import outbox, stress
|
||||
from portfolio_optimiser.budget import Budget, TokenMeter
|
||||
from portfolio_optimiser.mandate import Approach, Mandate
|
||||
from portfolio_optimiser.run import run_project
|
||||
|
||||
_BUNDLE_DIR = Path(__file__).resolve().parents[1] / "shared" / "examples" / "bygg-energi-mikro"
|
||||
_PID = "BYGG-KONTOR-NORD"
|
||||
_PROPOSER = (
|
||||
'{"measure":"LED-retrofit av kontorbelysning","affected_items":'
|
||||
'[{"code":"ENERGI-TOTAL-EL","quantity":300000,"unit_cost":1.0}],"claimed_saving_nok":30000}'
|
||||
)
|
||||
_CHECKER = "Supported by the cited documents. VERDICT: APPROVE"
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _isolate_model_env(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.delenv("PORTFOLIO_MODEL_MAP", raising=False)
|
||||
monkeypatch.delenv("PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT", raising=False)
|
||||
|
||||
|
||||
def _factory(role: str) -> BaseChatClient:
|
||||
return SyntheticUsageChatClient(default_reply=_CHECKER if role == "checker" else _PROPOSER)
|
||||
|
||||
|
||||
class _Fn:
|
||||
def __init__(self, name: str) -> None:
|
||||
self.name = name
|
||||
|
||||
|
||||
class _Ctx:
|
||||
def __init__(self, name: str, arguments: Any) -> None:
|
||||
self.function = _Fn(name)
|
||||
self.arguments = arguments
|
||||
|
||||
|
||||
def _record(*calls: tuple[str, Any]) -> list[ex.ToolCall]:
|
||||
import asyncio
|
||||
|
||||
sink: list[ex.ToolCall] = []
|
||||
recorder = ex.ExplorationToolRecorder(sink)
|
||||
|
||||
async def _noop() -> None:
|
||||
return None
|
||||
|
||||
for name, args in calls:
|
||||
asyncio.run(recorder.process(_Ctx(name, args), _noop)) # type: ignore[arg-type]
|
||||
return sink
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# (a)-(d) DEL C
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_the_window_arguments_are_recorded() -> None:
|
||||
"""(a) HOW the level was asked for, not only which one."""
|
||||
(call,) = _record(
|
||||
("read_dir", {"bundle_id": "k2", "path": "krav/N100", "filter": "rundkjoring", "limit": 25})
|
||||
)
|
||||
assert (call.filter, call.offset, call.limit) == ("rundkjoring", 0, 25)
|
||||
|
||||
|
||||
def test_a_call_that_passes_none_of_them_yields_empty_fields() -> None:
|
||||
"""(b) Never a refusal, never an absent key — the ``path``/``bundle_id`` rule one field over."""
|
||||
(call,) = _record(("list_bundles", {}))
|
||||
assert (call.filter, call.offset, call.limit) == ("", 0, 0)
|
||||
payload = ex.tool_call_payload([call])
|
||||
assert payload[0]["filter"] == "" and payload[0]["offset"] == 0 and payload[0]["limit"] == 0
|
||||
|
||||
|
||||
def test_a_numeric_argument_sent_as_a_string_is_coerced() -> None:
|
||||
"""(c) A model may send either shape; a recorder that read one would misreport the other."""
|
||||
(call,) = _record(("read_dir", {"bundle_id": "k2", "path": "p", "offset": "30", "limit": "10"}))
|
||||
assert (call.offset, call.limit) == (30, 10)
|
||||
# …and a value that is neither is 0, never an invented number.
|
||||
(other,) = _record(("read_dir", {"bundle_id": "k2", "path": "p", "limit": True}))
|
||||
assert other.limit == 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# a synthetic base + outbox, self-contained (nothing here touches a real bundle or a model)
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
_GOOD = "krav/N1/id-good.md"
|
||||
|
||||
|
||||
def _minibase(root: Path) -> Path:
|
||||
base = root / "minibase"
|
||||
(base / "krav" / "N1").mkdir(parents=True)
|
||||
(base / "index.md").write_text(
|
||||
"---\nbundle_id: minibase\n---\n\n- [good](krav/N1/id-good.md)\n", encoding="utf-8"
|
||||
)
|
||||
(base / _GOOD).write_text(
|
||||
'---\ntype: concept\ntitle: "T"\nreq_number: "Krav 1.2.3-4"\n---\n\nBody.\n',
|
||||
encoding="utf-8",
|
||||
)
|
||||
return base
|
||||
|
||||
|
||||
def _context_dir(root: Path) -> Path:
|
||||
ctx = root / "ctx"
|
||||
(ctx / "docs").mkdir(parents=True)
|
||||
(ctx / "bundle.txt").write_text("name: minibase\nbundle_id: minibase\n", encoding="utf-8")
|
||||
(ctx / "mandate.json").write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"objective": "o",
|
||||
"success_criteria": "s",
|
||||
"approaches": [
|
||||
{"id": "a1", "label": "L", "affected_codes": ["CODE-1"]},
|
||||
{"id": "a2", "label": "M", "affected_codes": ["CODE-2"]},
|
||||
],
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
(ctx / "fasit.json").write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"project_id": "proj",
|
||||
"bundle": "minibase",
|
||||
"bundle_id": "minibase",
|
||||
"must_cite": [
|
||||
{
|
||||
"approach_id": "a1",
|
||||
"rationale": "why",
|
||||
"concepts": [{"path": _GOOD, "title": "T", "ref": "Krav 1.2.3-4"}],
|
||||
}
|
||||
],
|
||||
"must_refuse": [],
|
||||
"honesty": "synthetic",
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
return ctx
|
||||
|
||||
|
||||
def _outbox(root: Path, *, tokens: int, tool_calls: list[dict[str, Any]]) -> Path:
|
||||
out = root / "out"
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
(out / "r1-a1-proposal.json").write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"proposal": {
|
||||
"project_id": "proj",
|
||||
"measure": "m",
|
||||
"affected_items": [{"code": "CODE-1", "quantity": 1.0, "unit_cost": 2.0}],
|
||||
"claimed_saving_nok": 1.0,
|
||||
"assumptions": {},
|
||||
},
|
||||
"provenance": {"citations": [], "token_usage": tokens},
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
(out / "r1-a1-outcome.json").write_text(
|
||||
json.dumps({"outcome_type": "rejected", "reason": "no"}), encoding="utf-8"
|
||||
)
|
||||
(out / "r1-debate.json").write_text(
|
||||
json.dumps({"run_id": "r1", "tool_calls": tool_calls}), encoding="utf-8"
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
def test_the_judge_counts_filtered_and_paged_calls(tmp_path: Path) -> None:
|
||||
"""(d) Readable directly, instead of inferred from which documents fell outside a default."""
|
||||
calls = [
|
||||
{
|
||||
"name": "read_dir",
|
||||
"bundle_id": "b",
|
||||
"path": "p",
|
||||
"filter": "rund",
|
||||
"offset": 0,
|
||||
"limit": 0,
|
||||
},
|
||||
{"name": "read_dir", "bundle_id": "b", "path": "p", "filter": "", "offset": 30, "limit": 0},
|
||||
{
|
||||
"name": "read_file",
|
||||
"bundle_id": "b",
|
||||
"path": _GOOD,
|
||||
"filter": "",
|
||||
"offset": 0,
|
||||
"limit": 0,
|
||||
},
|
||||
]
|
||||
verdict = stress.score_context_set(
|
||||
_context_dir(tmp_path),
|
||||
_outbox(tmp_path, tokens=1234, tool_calls=calls),
|
||||
"r1",
|
||||
_minibase(tmp_path),
|
||||
)
|
||||
assert (verdict.filter_calls, verdict.paged_calls) == (1, 1)
|
||||
assert verdict.tool_calls_seen == 3
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# (e)-(g) DEL D
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_the_coverage_artefact_carries_the_rows_and_the_stop_reason(tmp_path: Path) -> None:
|
||||
"""(e)/(f) A required stop reason, never an inferred one: "finished" and "we never found out"
|
||||
must not be the same value."""
|
||||
outbox.write_coverage(
|
||||
str(tmp_path),
|
||||
"r1",
|
||||
rows=[
|
||||
{"id": "a1", "label": "L", "status": "validated", "detail": "", "saving_nok": 10.0},
|
||||
{
|
||||
"id": "a2",
|
||||
"label": "M",
|
||||
"status": "not_evaluated",
|
||||
"detail": "budget",
|
||||
"saving_nok": None,
|
||||
},
|
||||
],
|
||||
stop_reason="tokens",
|
||||
)
|
||||
payload = json.loads((tmp_path / "r1-coverage.json").read_text(encoding="utf-8"))
|
||||
assert payload["stop_reason"] == "tokens"
|
||||
assert [r["status"] for r in payload["rows"]] == ["validated", "not_evaluated"]
|
||||
# Byte-deterministic: the same inputs twice produce the same file (write_run_config's rule).
|
||||
first = (tmp_path / "r1-coverage.json").read_bytes()
|
||||
outbox.write_coverage(
|
||||
str(tmp_path),
|
||||
"r1",
|
||||
rows=[
|
||||
{"id": "a1", "label": "L", "status": "validated", "detail": "", "saving_nok": 10.0},
|
||||
{
|
||||
"id": "a2",
|
||||
"label": "M",
|
||||
"status": "not_evaluated",
|
||||
"detail": "budget",
|
||||
"saving_nok": None,
|
||||
},
|
||||
],
|
||||
stop_reason="tokens",
|
||||
)
|
||||
assert (tmp_path / "r1-coverage.json").read_bytes() == first
|
||||
|
||||
|
||||
def test_the_judge_reads_the_stop_reason_onto_every_unevaluated_row(tmp_path: Path) -> None:
|
||||
"""(f) The second half: a row with no artefact says WHY, instead of leaving a reader to guess."""
|
||||
ctx = _context_dir(tmp_path)
|
||||
out = _outbox(tmp_path, tokens=45_642, tool_calls=[])
|
||||
base = _minibase(tmp_path)
|
||||
|
||||
# Before any coverage file exists — every run written before today.
|
||||
blind = stress.score_context_set(ctx, out, "r1", base)
|
||||
assert blind.stop_reason == "absent"
|
||||
assert [r.not_evaluated_reason for r in blind.approaches] == ["", "absent"]
|
||||
|
||||
outbox.write_coverage(
|
||||
str(out),
|
||||
"r1",
|
||||
rows=[
|
||||
{"id": "a2", "label": "M", "status": "not_evaluated", "detail": "b", "saving_nok": None}
|
||||
],
|
||||
stop_reason="rounds",
|
||||
)
|
||||
informed = stress.score_context_set(ctx, out, "r1", base)
|
||||
assert informed.stop_reason == "rounds"
|
||||
assert [r.not_evaluated_reason for r in informed.approaches] == ["", "rounds"]
|
||||
|
||||
|
||||
def test_the_judge_reports_what_the_run_spent(tmp_path: Path) -> None:
|
||||
"""(g) D1. P18's finding 4 was wrong as written: the field was always there, the READER was not.
|
||||
|
||||
The figure is the one measured on round 2's own artefacts (tunnel-hauglia-2027-02).
|
||||
"""
|
||||
verdict = stress.score_context_set(
|
||||
_context_dir(tmp_path),
|
||||
_outbox(tmp_path, tokens=45_642, tool_calls=[]),
|
||||
"r1",
|
||||
_minibase(tmp_path),
|
||||
)
|
||||
assert verdict.token_usage == 45_642
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# (e) the run itself writes it -- without this arm the wiring has no witness
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_a_commissioned_run_writes_the_coverage_artefact(tmp_path: Path) -> None:
|
||||
"""(e) END-TO-END. The arm above drives ``write_coverage`` directly and would stay green with
|
||||
the ``run_project`` wiring detached -- the vacuous-gate shape this repo has now met two dozen
|
||||
times. This one runs the real pipeline."""
|
||||
out = tmp_path / "with-mandate"
|
||||
await run_project(
|
||||
_PID,
|
||||
"local",
|
||||
docs_dir=str(_BUNDLE_DIR),
|
||||
bundle_dir=str(_BUNDLE_DIR),
|
||||
client_factory=_factory,
|
||||
max_rounds=2,
|
||||
outbox_dir=str(out),
|
||||
run_id="r1",
|
||||
mandate=Mandate(
|
||||
objective="o",
|
||||
approaches=(Approach(id="a1", label="LED", description="swap the fittings"),),
|
||||
allow_own_proposals=False,
|
||||
),
|
||||
)
|
||||
payload = json.loads((out / "r1-coverage.json").read_text(encoding="utf-8"))
|
||||
assert payload["stop_reason"] == "", "nothing cut this run short"
|
||||
assert [r["id"] for r in payload["rows"]] == ["a1"]
|
||||
assert payload["rows"][0]["status"] in {"validated", "rejected"}
|
||||
|
||||
|
||||
async def test_a_mandateless_run_leaves_the_outbox_byte_identical(tmp_path: Path) -> None:
|
||||
"""(e), the other half. Coverage is the MANDATE's report; a run without one would be described
|
||||
by a file saying nothing, and two older tests pin an exact listing on such a run."""
|
||||
out = tmp_path / "no-mandate"
|
||||
await run_project(
|
||||
_PID,
|
||||
"local",
|
||||
docs_dir=str(_BUNDLE_DIR),
|
||||
bundle_dir=str(_BUNDLE_DIR),
|
||||
client_factory=_factory,
|
||||
max_rounds=2,
|
||||
outbox_dir=str(out),
|
||||
run_id="r1",
|
||||
)
|
||||
assert not (out / "r1-coverage.json").exists()
|
||||
|
||||
|
||||
async def test_a_cap_that_cuts_the_run_short_is_named_in_the_artefact(tmp_path: Path) -> None:
|
||||
"""(f) END-TO-END, and this arm exists because the direct one was VACUOUS.
|
||||
|
||||
MEASURED: mutation D-i — ``stop_reason=""`` whatever happened — left the WHOLE suite green
|
||||
(1743/5), because the arm above calls ``write_coverage`` itself and therefore chooses the
|
||||
reason it then asserts. Only a run that a cap actually cut short can tell the two apart, and
|
||||
it is exactly the run for which the artefact matters: a commission that stopped halfway.
|
||||
"""
|
||||
out = tmp_path / "capped"
|
||||
await run_project(
|
||||
_PID,
|
||||
"local",
|
||||
docs_dir=str(_BUNDLE_DIR),
|
||||
bundle_dir=str(_BUNDLE_DIR),
|
||||
client_factory=_factory,
|
||||
max_rounds=2,
|
||||
outbox_dir=str(out),
|
||||
run_id="r1",
|
||||
# tick_round fires once per generation ATTEMPT: one round pays for the first
|
||||
# approach, the second is the one the cap cuts.
|
||||
meter=TokenMeter(Budget(max_tokens=10_000_000, max_rounds=1)),
|
||||
mandate=Mandate(
|
||||
objective="o",
|
||||
approaches=(
|
||||
Approach(id="a1", label="LED", description="one"),
|
||||
Approach(id="a2", label="Sensors", description="two"),
|
||||
),
|
||||
allow_own_proposals=False,
|
||||
),
|
||||
)
|
||||
payload = json.loads((out / "r1-coverage.json").read_text(encoding="utf-8"))
|
||||
assert payload["stop_reason"] == "rounds", payload
|
||||
assert [r["status"] for r in payload["rows"]][-1] == "not_evaluated", payload
|
||||
|
|
@ -304,9 +304,10 @@ async def test_a_refused_read_is_recorded_and_leaks_nothing(tmp_path) -> None:
|
|||
"nobody can audit"
|
||||
)
|
||||
payload = json.loads((outbox_dir / "verdict-gate-debate.json").read_text(encoding="utf-8"))
|
||||
assert {"name": "read_file", "bundle_id": bundle_id, "path": _VERDICT_FILE} in payload[
|
||||
"tool_calls"
|
||||
], f"the artefact does not carry the refused call: {payload['tool_calls']}"
|
||||
identity = [{k: c[k] for k in ("name", "bundle_id", "path")} for c in payload["tool_calls"]]
|
||||
assert {"name": "read_file", "bundle_id": bundle_id, "path": _VERDICT_FILE} in identity, (
|
||||
f"the artefact does not carry the refused call: {payload['tool_calls']}"
|
||||
)
|
||||
|
||||
|
||||
# ------------------------------------------------------ (5) the document, not the walk, decides
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue