feat(visibility): en uforankret kjoering sier det - felt + linje (ORDRE 20260821T092039Z)
Maalt oekt 48: en bundle-kjoering uten cost-baseline.json gikk rc 0 uten et ord, og `grep baseline provenance.py outbox.py` ga null treff - hverken stdout, stempelet eller utboksen bar at validatorens steg 0 ble hoppet over. To tenner, begge smaa, og begge fra kjoeringens ENE oppslag av baselinen (koe-(p)): 1. ProvenanceStamp.cost_baseline_anchored - PAAKREVD bool uten default. Begge defaults lyver: True lar en glemsom konstruktoer paastaa en ankring som ikke skjedde, False underrapporterer en ekte. Naar utboksen gratis (write_proposal dumper hele stempelet). DryRunReport baerer det samme - en dry-run stopper foer noe stempel finnes. 2. run.cost_baseline_notice(anchored) - ENESTE renderer, tar den alt opploeste booleanen, returnerer None naar kjoeringen ER forankret (omisjon, aldri en tom rad). Printes paa tre flater: --live-dry-run, full enkeltkjoering, og per prosjekt i portefoeljemodus. IKKE foldet inn i mandate.announce, og det er en MAALING: den fyrer kun med --mandate, saa nettopp de bare bundle-dry-runsene defekten ble maalt paa ville fortsatt sagt ingenting - og den renderes foer run_project, altsaa foer noen har opploest baselinen. Ankeringen forblir VALGFRI (en pre-amendment-base kjoerer uendret) - dette er synlighet, ikke en ny nekt. Golden-transkriptet er byte-uendret: demoen kjoerer en base som HAR fila. Load-bearing MAALT (tests/test_baseline_visibility_loadbearing.py, 11 tester), seks mutasjoner alle roede mot HELE suiten + groenn kontroll 885/5: konstant stamp-wiring (3 roede) - konstant dry-run-wiring (1) - detach dry-run-printen (1) - renderer returnerer alltid linja (2, inkl. den forankrede kontrollen) - detach full-run-printen (1) - detach portefoelje-printen (1). Portefoelje-armen er DEFENSIV og uttalt (ingen referanse-prosjekt setter bundle_dir; budget_stop-presedensen, crafted PortfolioResult). Det paakrevde feltet tvang fem eksisterende test-konstruktoerer til aa ta stilling. Dokumentene som beskrev den gamle stillheten er rettet: kunnskapsbase-for-en-kjoring.md S4.1 (tabellraden re-maalt live), S6 og S7; README «How it is set up»; CLAUDE.md S4.0-raden. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DA6HAF8HFQxGYC2h6ypRQe
This commit is contained in:
parent
0bc8dae5c0
commit
156312c32e
11 changed files with 481 additions and 17 deletions
20
CLAUDE.md
20
CLAUDE.md
|
|
@ -374,6 +374,26 @@ Python ≥3.10. MAF (`agent-framework-core` 1.9.0). Pakkehåndtering: `uv`. To b
|
|||
S3.2); **D7-speiling ÅPEN.** Load-bearing MÅLT (`tests/test_s40_cost_baseline_loadbearing.py`), seks
|
||||
mutasjoner alle røde: detach avstemmings-stagen · detach magnitude-toleransen · detach road-wiringen ·
|
||||
detach bundle-wiringen · ignorer det injiserte cap-registeret · gjør den valgfrie loaderen tolerant.
|
||||
**En UFORANKRET kjøring sier det nå — og BEGGE utsagn stammer fra kjøringens ENE oppslag, aldri
|
||||
en andre lesing av bundelen (21.08):** `ProvenanceStamp.cost_baseline_anchored` er PÅKREVD uten
|
||||
default (begge defaults lyver: `True` lar en glemsom konstruktør påstå en ankring som ikke skjedde,
|
||||
`False` underrapporterer en ekte — en binær kjensgjerning om en falsifiserer har ingen ærlig
|
||||
default), og `DryRunReport` bærer det samme fordi en dry-run stopper før noe stempel finnes.
|
||||
`run.cost_baseline_notice(anchored)` er ENESTE renderer, tar den alt oppløste BOOLEANEN, og
|
||||
returnerer `None` når kjøringen ER forankret — omisjon, aldri en tom rad (`announce`-regelen).
|
||||
**IKKE foldet inn i `mandate.announce`, og det er en MÅLING:** den fyrer kun med `--mandate`, så
|
||||
nettopp de bare bundle-dry-runsene defekten ble målt på ville fortsatt sagt ingenting — og den
|
||||
renderes FØR `run_project`, altså før noen har oppløst baselinen. Utboksen trengte ingen endring
|
||||
(`write_proposal` dumper hele stempelet). Ankeringen forblir VALGFRI: dette er synlighet, ikke en
|
||||
ny nekt, og golden-transkriptet er byte-uendret fordi demoen kjører en base som HAR fila.
|
||||
Portefølje-armen er DEFENSIV og uttalt (ingen referanse-prosjekt setter `bundle_dir`, så den er
|
||||
unåbar i dag — `budget_stop`-presedensen; testen driver en crafted `PortfolioResult`). Load-bearing
|
||||
MÅLT (`tests/test_baseline_visibility_loadbearing.py`), seks mutasjoner alle røde mot HELE suiten +
|
||||
grønn kontroll 885/5: konstant stamp-wiring (3 røde) · konstant dry-run-wiring (1 rød) · detach
|
||||
dry-run-printen (1 rød) · renderer returnerer alltid linja (2 røde — inkl. den forankrede
|
||||
kontrollen, altså er omisjonen selv gatet) · detach full-run-printen (1 rød) · detach
|
||||
portefølje-printen (1 rød). Det PÅKREVDE feltet tvang fem eksisterende test-konstruktører til å ta
|
||||
stilling — det er egenskapen, ikke friksjonen.
|
||||
- **Globalt token-tak håndheves FØR kall, aldri bare etterpå (S3.4, F10):** `PortfolioBudget` +
|
||||
`PortfolioMeter` er ÉN ledger over hele porteføljepasset (og — seedet av `read_spend` — på tvers
|
||||
av pass), mens per-run `Budget`/`TokenMeter` er uendret. Taket har tre tenner, med hver sin jobb:
|
||||
|
|
|
|||
|
|
@ -436,6 +436,13 @@ when the seam is detached, so the loop cannot silently degrade into theater.
|
|||
reference-domain (non-bundle) path the project's own cost items are the baseline, so those runs
|
||||
are always anchored.
|
||||
|
||||
**An un-anchored run says so.** Every run stamps `provenance.cost_baseline_anchored` (a required
|
||||
boolean, so no stamp can omit it), which reaches the outbox in `{run_id}-proposal.json`; and when
|
||||
a run is un-anchored the CLI prints one line naming the skipped stage — on `--live-dry-run`, on a
|
||||
full single run, and per project in portfolio mode. An anchored run prints no such line at all:
|
||||
a line for something the run does not have is omitted rather than rendered empty. Anchoring stays
|
||||
optional; this is visibility, not a new refusal.
|
||||
|
||||
The **prior-verdict fold — the learning step — happens only on the `--bundle-dir` path**; a
|
||||
plain `--docs-dir`-only run is single-shot (no fold). `--decision`/`--rationale` apply to the
|
||||
single-project path only and are inert in portfolio mode. **`--outbox-dir` must differ from
|
||||
|
|
|
|||
|
|
@ -297,17 +297,31 @@ Men fila er **valgfri** på bundle-stien, og fraværet er stille. Målt 2026-08-
|
|||
|---|---|---|---|
|
||||
| intakt | `LIVE-DRY-RUN OK` | 0 | — |
|
||||
| uten `validator-input.json` | nektet | 1 | `live-dry-run refused: IR projection not found in bundle: 'validator-input.json'` |
|
||||
| **uten `cost-baseline.json`** | **`LIVE-DRY-RUN OK`** | **0** | **ingen** |
|
||||
| **uten `cost-baseline.json`** | **`LIVE-DRY-RUN OK`** | **0** | **`Cost baseline: NONE in the bundle — this run is un-anchored: …`** |
|
||||
| korrupt `cost-baseline.json` (`quantity: -1`, `unit_cost` mangler) | nektet | 1 | `live-dry-run refused: 2 validation errors for CostBaseline …` |
|
||||
|
||||
Kjeden som gir det utfallet: `okf.load_optional_cost_baseline` returnerer `None` når fila mangler
|
||||
(VERIFISERT: `okf.py:323-335`), `run.py:516` tar imot `None`, og `validate_proposal` hopper over
|
||||
steg 0 når baselinen er `None` (VERIFISERT: `validator.py:213`). Det er med vilje — en base skrevet
|
||||
før fila fantes skal fortsatt kjøre byte-identisk (VERIFISERT: `CLAUDE.md`, S4.0-raden). Men
|
||||
ingenting i kjørestien *sier* at det skjedde: verken stdout, `provenance` eller utboksen bærer et
|
||||
felt for forankret/uforankret (VERIFISERT: `grep baseline` over `provenance.py` og `outbox.py`
|
||||
gir null treff). Den eneste flaten som skriver det høyt er demoen, som printer
|
||||
«KUNNSKAPSBASE: … — uten kostbaseline» når fila mangler (VERIFISERT: `simulation.py:791-802`).
|
||||
før fila fantes skal fortsatt kjøre byte-identisk (VERIFISERT: `CLAUDE.md`, S4.0-raden).
|
||||
|
||||
**Fraværet er ikke lenger stille (2026-08-21).** Kjøringen sier det nå på to måter, og begge
|
||||
stammer fra den *samme* oppslagsverdien inne i kjøringen — ikke fra en ny lesing av basen:
|
||||
|
||||
- **Et strukturert felt:** `provenance.cost_baseline_anchored` er `true`/`false`, og følger med
|
||||
inn i utboksens `{run_id}-proposal.json`, som dumper hele stempelet (VERIFISERT:
|
||||
`tests/test_baseline_visibility_loadbearing.py`, armene (a) og (d)).
|
||||
- **Én linje på stdout:** `Cost baseline: NONE in the bundle — this run is un-anchored: …`,
|
||||
skrevet av tørrkjøringen, av den fulle enkeltkjøringen og per prosjekt i porteføljemodus. Er
|
||||
basen forankret, skrives **ingen linje i det hele tatt** — en linje for noe kjøringen ikke har
|
||||
utelates, samme regel som resten av kunngjøringen følger (VERIFISERT: kjørt 2026-08-21 mot to
|
||||
kopier av veglys-basen; intakt kopi er byte-uendret, kopi uten fila bærer linja).
|
||||
|
||||
Ankeringen er fortsatt **valgfri** — en base skrevet før fila fantes kjører uendret. Dette er
|
||||
synlighet, ikke en ny nekt. Demoen har sin egen, norske formulering
|
||||
(«KUNNSKAPSBASE: … — uten kostbaseline», `simulation.py:791-802`): to formuleringer med vilje, fordi
|
||||
demoen beskriver hva *basen erklærer* før kjøringen, mens linja over rapporterer hva *kjøringen
|
||||
gjorde*.
|
||||
|
||||
**Konsekvensen for planleggingen:** spørsmål 2 i [§3](#3-avgjørelsesdelen-spørsmålene-du-stiller-fagpersonen)
|
||||
er det eneste spørsmålet der et «vet ikke» ikke stopper noe — og derfor det eneste der svaret må
|
||||
|
|
@ -524,12 +538,13 @@ så dokumentet ikke lover mer enn det som kan leveres.
|
|||
tilnærming i bestillingen (VERIFISERT: `run.py:634-638`; én `expel_query` for hele
|
||||
kjøringen). En bestilling med to tilnærminger mot to ulike kostlinjer får tidligere dommer
|
||||
hentet for den kandidaten `validator-input.json` beskriver.
|
||||
- **Det finnes ingen «er basen klar?»-kommando.** `--live-dry-run` beviser at basen *åpner*;
|
||||
den sier ingenting om at `cost-baseline.json` mangler, og ingenting om brutte lenker.
|
||||
`preflight` sjekker Azure-konfigurasjon, ikke basen (VERIFISERT: `preflight.py` docstring).
|
||||
- **En uforankret kjøring etterlater ingen spor.** Verken stdout, `provenance` eller utboksen
|
||||
sier om validatoren hadde en kostbaseline (VERIFISERT: §4.1). Dette dokumentet er i dag den
|
||||
eneste beskrivelsen av den egenskapen utenfor koden og `CLAUDE.md`.
|
||||
- **Det finnes ingen «er basen klar?»-kommando.** `--live-dry-run` beviser at basen *åpner*, og
|
||||
sier nå fra hvis `cost-baseline.json` mangler — men ingenting om brutte lenker, og ingenting om
|
||||
innholdets kvalitet. `preflight` sjekker Azure-konfigurasjon, ikke basen (VERIFISERT:
|
||||
`preflight.py` docstring).
|
||||
- **Uforankret kjøring er synlig, men ikke summert.** Feltet og linja finnes per kjøring
|
||||
([§4.1](#41-den-skarpeste-mangelen-cost-baselinejson)); det finnes ingen rapport som teller opp
|
||||
hvor mange kjøringer i et porteføljepass som gikk uforankret.
|
||||
- **Eksempelbasene er ikke ekte prosjekter.** Prosjektlaget er fiktivt; realiseringsgraden i alle
|
||||
tre frø-dommene er lånt fra utenlandsk programlitteratur fordi ingen norsk ex-post-måling
|
||||
finnes (VERIFISERT: `provenance`-feltet i de tre dom-filene).
|
||||
|
|
@ -542,12 +557,17 @@ så dokumentet ikke lover mer enn det som kan leveres.
|
|||
- **1–2 uker.** Oppskriften sier det, og ingenting i dette dokumentet korter det ned. Det som
|
||||
står her er hva ukene skal brukes til.
|
||||
|
||||
## 7. Hvis utforskningen skulle konkludere med kode
|
||||
## 7. Punktet der utforskningen konkluderte med kode
|
||||
|
||||
Den gjør det på ett punkt, og det er uttalt her i stedet for bygget: en kjøring som går uten
|
||||
`cost-baseline.json` burde si det — i kunngjøringen, i `provenance`, eller begge. Demoen gjør det
|
||||
allerede; kjørestien gjør det ikke. Det er en beslutning for eieren av repoet, ikke for dette
|
||||
dokumentet.
|
||||
Dokumentet pekte opprinnelig på ett punkt der det ikke holdt å beskrive: en kjøring som går uten
|
||||
`cost-baseline.json` burde si det — i kunngjøringen, i `provenance`, eller begge. Demoen gjorde det
|
||||
allerede; kjørestien gjorde det ikke.
|
||||
|
||||
**Det er nå bygget (2026-08-21), med begge deler:** et strukturert `cost_baseline_anchored`-felt på
|
||||
provenance-stempelet, og én linje på stdout når kjøringen er uforankret — se
|
||||
[§4.1](#41-den-skarpeste-mangelen-cost-baselinejson). Ankeringen forble valgfri, og
|
||||
golden-transkriptet er byte-uendret, fordi demoen kjører en base som *har* fila og linja derfor
|
||||
ikke fyrer der.
|
||||
|
||||
## 8. Verifiseringslogg
|
||||
|
||||
|
|
|
|||
|
|
@ -57,6 +57,20 @@ class ProvenanceStamp(BaseModel):
|
|||
role: str
|
||||
validator_decision: Literal["validated", "rejected"]
|
||||
token_usage: int
|
||||
#: Was the deterministic gate ANCHORED to the project's own cost lines? ``True`` means
|
||||
#: ``validate_proposal`` ran its stage-0 reconciliation (every ``affected_item`` checked against
|
||||
#: a real ``CostBaseline`` line, within tolerance, BEFORE the solver); ``False`` means the
|
||||
#: bundle shipped no ``cost-baseline.json``, so that stage was SKIPPED and the gate reasoned
|
||||
#: only about numbers the proposal itself supplied. Anchoring stays OPTIONAL (a pre-amendment
|
||||
#: bundle is legitimately un-anchored) — this field does not gate anything, it makes the skip
|
||||
#: legible. A STRUCTURED field rather than prose, for the reason ``BudgetExceeded`` carries
|
||||
#: ``kind``/``limit``/``observed`` as fields (kø-(y)): "was the falsifier anchored" is an
|
||||
#: operative question that must be readable by machine.
|
||||
#:
|
||||
#: REQUIRED, with no default, because both defaults lie: ``True`` would let a constructor that
|
||||
#: forgot claim an anchoring that never happened, and ``False`` would under-claim a real one.
|
||||
#: A binary fact about a falsifier has no honest default.
|
||||
cost_baseline_anchored: bool
|
||||
#: External service calls the run made (B4). EMPTY is a positive statement — "nothing outside
|
||||
#: this process was contacted" — not an absent field, which is why it is always serialized.
|
||||
external_calls: list[ExternalCall] = Field(default_factory=list)
|
||||
|
|
|
|||
|
|
@ -172,6 +172,11 @@ class DryRunReport:
|
|||
max_rounds: int
|
||||
max_tokens: int
|
||||
top_k: int
|
||||
#: Whether a REAL run of this configuration would have its deterministic gate anchored to the
|
||||
#: project's own cost lines (see ``ProvenanceStamp.cost_baseline_anchored``). Carried here too
|
||||
#: because a dry run stops before any proposal exists, so there is no stamp to read it off —
|
||||
#: and this surface is precisely where the un-anchored case was measured to be silent.
|
||||
cost_baseline_anchored: bool
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
|
|
@ -423,6 +428,38 @@ def _default_factory(profile: Profile | str) -> Callable[[str], BaseChatClient]:
|
|||
return factory
|
||||
|
||||
|
||||
#: The one line a run prints about its own anchoring. Rendered ONLY when the run is un-anchored:
|
||||
#: an anchored run has nothing to warn about, and ``mandate.announce``'s rule is that a line for
|
||||
#: something the run does not have is OMITTED rather than rendered empty.
|
||||
_UNANCHORED_NOTICE = (
|
||||
" Cost baseline: NONE in the bundle — this run is un-anchored: the validator's stage 0 "
|
||||
"(reconciling each proposed cost line against the project's own) is SKIPPED"
|
||||
)
|
||||
|
||||
|
||||
def cost_baseline_notice(anchored: bool) -> str | None:
|
||||
"""Render the un-anchored notice, or ``None`` when the run IS anchored.
|
||||
|
||||
ONE renderer with N callsites, never N copies of the wording (kø-(p)) — and it takes the
|
||||
already-resolved BOOLEAN rather than a bundle path, so the printed line and the machine-readable
|
||||
``ProvenanceStamp.cost_baseline_anchored`` can never disagree: both descend from the single
|
||||
``okf.load_optional_cost_baseline`` call inside ``run_project``. A renderer that re-read the
|
||||
bundle would be a second resolution of the same rule, free to drift from the run it describes.
|
||||
|
||||
**Not folded into ``mandate.announce``, and that is a measurement rather than a preference:**
|
||||
``announce`` is printed only when ``--mandate`` is given, so the runs this notice exists for —
|
||||
the bare bundle dry-runs that exit 0 in silence — would still say nothing. It also renders
|
||||
BEFORE ``run_project`` is called, i.e. before anyone has resolved the baseline; putting the line
|
||||
there would have required ``main`` to open the bundle itself.
|
||||
|
||||
English, like every other line this CLI prints; the Norwegian explanation of what an un-anchored
|
||||
run means belongs in ``docs/kunnskapsbase-for-en-kjoring.md``, next to the domain expert.
|
||||
|
||||
Tense-neutral on purpose ("is SKIPPED"): the same string serves ``--live-dry-run`` (where the
|
||||
run has not happened) and a completed run (where it has)."""
|
||||
return None if anchored else _UNANCHORED_NOTICE
|
||||
|
||||
|
||||
async def run_project(
|
||||
project_id: str,
|
||||
profile: Profile | str = Profile.LOCAL,
|
||||
|
|
@ -585,6 +622,7 @@ async def run_project(
|
|||
max_rounds=max_rounds,
|
||||
max_tokens=max_tokens,
|
||||
top_k=top_k,
|
||||
cost_baseline_anchored=baseline is not None,
|
||||
)
|
||||
# The MCP lifecycle (Trekk B2): entered HERE, after the dry-run cut above, so a dry run never
|
||||
# opens a connection — its promise to stop before the first call covers egress too. Constructed
|
||||
|
|
@ -710,6 +748,11 @@ async def run_project(
|
|||
"validated" if isinstance(validator_outcome, ValidatedProposal) else "rejected"
|
||||
),
|
||||
token_usage=meter.tokens,
|
||||
# Whether stage 0 of the deterministic gate had a baseline to reconcile against. Read off
|
||||
# the SAME ``baseline`` the validator was handed, so the record cannot describe a different
|
||||
# anchoring than the one that ran. The road path is anchored by construction (the reference
|
||||
# project's own cost_items ARE the baseline); a bundle is anchored iff it ships the file.
|
||||
cost_baseline_anchored=baseline is not None,
|
||||
# B4: which external service the debate actually called. Read AFTER the debate, so it is a
|
||||
# record rather than an intention. The honesty limit lives on ``ExternalCall`` itself: this
|
||||
# is the call and its source, not a verified rendering of the service's answer.
|
||||
|
|
@ -1684,6 +1727,14 @@ def main(argv: list[str] | None = None) -> int:
|
|||
return 1
|
||||
for r in portfolio_result.runs:
|
||||
print(f"{type(r.outcome).__name__}: verdict id={r.verdict.id}")
|
||||
# Per project, because anchoring is a per-project fact. DEFENSIVE and currently
|
||||
# unreachable from this branch — measured, and said out loud for the same reason the
|
||||
# ``budget_stop`` arm below is: no reference project sets ``bundle_dir``, so every
|
||||
# portfolio run today takes the road path and is anchored by construction. The test
|
||||
# that covers it drives a crafted ``PortfolioResult``, and says so.
|
||||
run_notice = cost_baseline_notice(r.provenance.cost_baseline_anchored)
|
||||
if run_notice is not None:
|
||||
print(run_notice)
|
||||
# One settlement per project: the mandate applies to each project in the pass, so
|
||||
# each project answers for it separately. Empty without a mandate.
|
||||
project_settlement = settle(r.coverage)
|
||||
|
|
@ -1776,6 +1827,12 @@ def main(argv: list[str] | None = None) -> int:
|
|||
f"max_tokens={report.max_tokens}, top_k={report.top_k}) — "
|
||||
"ingen modellkall gjort (stoppet før første debate.run)"
|
||||
)
|
||||
# The measured silence this closes: a bundle without ``cost-baseline.json`` used to dry-run
|
||||
# to rc 0 with nothing said about the gate's stage 0 being skipped. Printed AFTER the line
|
||||
# it qualifies, and only when there is something to say.
|
||||
notice = cost_baseline_notice(report.cost_baseline_anchored)
|
||||
if notice is not None:
|
||||
print(notice)
|
||||
return 0
|
||||
|
||||
try:
|
||||
|
|
@ -1813,6 +1870,11 @@ def main(argv: list[str] | None = None) -> int:
|
|||
return 1
|
||||
kind = type(result.outcome).__name__
|
||||
print(f"{args.project_id}: {kind} (verdict id={result.verdict.id}, decision={args.decision})")
|
||||
# Same notice, same renderer, read off the run's OWN stamp — so stdout and the outbox artefact
|
||||
# cannot disagree about whether the gate was anchored.
|
||||
notice = cost_baseline_notice(result.provenance.cost_baseline_anchored)
|
||||
if notice is not None:
|
||||
print(notice)
|
||||
# The settlement against the commission (Trekk A4). Empty without a mandate, so an
|
||||
# un-commissioned run prints exactly what it printed before.
|
||||
settlement = settle(result.coverage)
|
||||
|
|
|
|||
327
tests/test_baseline_visibility_loadbearing.py
Normal file
327
tests/test_baseline_visibility_loadbearing.py
Normal file
|
|
@ -0,0 +1,327 @@
|
|||
"""An UN-ANCHORED run says so — in a machine-readable field AND in one line on stdout.
|
||||
|
||||
S4.0 made the deterministic gate anchorable: when a bundle ships ``cost-baseline.json`` the
|
||||
validator's stage 0 reconciles every ``affected_item`` against the project's own cost lines BEFORE
|
||||
the solver, and when it does not, that stage is simply skipped (``None`` = pre-amendment behaviour,
|
||||
which is what keeps every commons-owned golden bundle running). The anchoring stayed OPTIONAL on
|
||||
purpose — and that is not what this file changes.
|
||||
|
||||
What it changes is that the skip was INVISIBLE. Measured (session 48, ``9d149b3``): four
|
||||
``--live-dry-run``s over copies of the veglys bundle — intact rc 0 · without ``validator-input.json``
|
||||
rc 1 · **without ``cost-baseline.json`` rc 0 with no message at all** · corrupt baseline rc 1. And
|
||||
``grep baseline provenance.py outbox.py`` returned 0 hits, so neither the stamp nor the outbox
|
||||
artefacts carried it either. An operator could therefore run the whole gate un-anchored, read a
|
||||
clean rc 0, and have nothing anywhere to tell them the fabrication stage never ran.
|
||||
|
||||
Two teeth, both small:
|
||||
|
||||
1. ``ProvenanceStamp.cost_baseline_anchored`` — a REQUIRED bool, no default. "Was the gate
|
||||
anchored" is a binary fact about a falsifier, in the same class as ``BudgetExceeded``'s
|
||||
``kind``/``limit``/``observed`` (kø-(y)): it must be readable by machine, not inferred from
|
||||
prose. It carries no default because BOTH defaults lie — ``True`` would let a forgetful
|
||||
constructor claim an anchoring that never happened, ``False`` would under-claim a real one — and
|
||||
a binary fact with no honest default is exactly what a required field is for. It reaches the
|
||||
outbox for free: ``outbox.write_proposal`` dumps the whole stamp.
|
||||
2. ``run.cost_baseline_notice`` — ONE renderer, rendering ONE line when the run is un-anchored and
|
||||
``None`` when it is anchored. Omission, not an empty row, mirrors ``mandate.announce``'s rule
|
||||
that a line for something the run does not have is left out rather than rendered blank.
|
||||
|
||||
**The line is rendered from the run's OWN resolution, never from a second read of the bundle**
|
||||
(kø-(p)). ``run_project`` is the single place that calls ``okf.load_optional_cost_baseline`` on the
|
||||
run path; the fact leaves the run as a typed field on ``DryRunReport`` and on ``ProvenanceStamp``,
|
||||
and ``main`` prints from that. Rendering it inside ``mandate.announce`` was MEASURED and rejected:
|
||||
``announce`` fires only when ``--mandate`` is given, so the very runs this file exists for — the
|
||||
four bare dry-runs above, none of which had a mandate — would still have printed nothing.
|
||||
|
||||
Arms:
|
||||
(a) the provenance field is ``False`` on an un-anchored bundle run and ``True`` on an anchored one,
|
||||
end-to-end through ``run_project`` (+ the road path, which is anchored by construction);
|
||||
(b) the notice EXISTS un-anchored and is ABSENT anchored — asserted on a sentinel that the anchored
|
||||
branch cannot contain, because it prints no line at all (never a substring both branches share:
|
||||
the 08-09 class);
|
||||
(c) both CLI surfaces carry it — ``--live-dry-run`` and the full run;
|
||||
(d) the outbox artefact carries the field.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from conftest import SyntheticUsageChatClient
|
||||
|
||||
from portfolio_optimiser import run
|
||||
from portfolio_optimiser.ir import AffectedItem, SavingsProposal
|
||||
from portfolio_optimiser.provenance import Citation, ProvenanceStamp
|
||||
from portfolio_optimiser.retrieval import TextSpan
|
||||
from portfolio_optimiser.run import (
|
||||
DryRunReport,
|
||||
PortfolioResult,
|
||||
RunResult,
|
||||
cost_baseline_notice,
|
||||
run_project,
|
||||
)
|
||||
from portfolio_optimiser.validator import Rejection
|
||||
from portfolio_optimiser.verdicts import ProposalFeatures, Verdict, VerdictStore
|
||||
|
||||
_DATA = Path(__file__).resolve().parents[1] / "src" / "portfolio_optimiser" / "data" / "bundles"
|
||||
#: The ONLY repo-local bundle shipping a ``cost-baseline.json`` (S4.0 fixture) -> anchored.
|
||||
BASELINE_BUNDLE = _DATA / "bygg-energi-baseline-mikro"
|
||||
#: A bundle written before the amendment -> legitimately un-anchored (this is the case under test).
|
||||
PRE_AMENDMENT_BUNDLE = _DATA / "bygg-energi-mikro-a"
|
||||
|
||||
_VERDICT_INPUT = {"decision": "approved", "rationale": "expert reviewed (sim)"}
|
||||
|
||||
#: A reply that reconciles against the S4.0 fixture's own line, so the ANCHORED control run reaches
|
||||
#: a proposal rather than being rejected by stage 0 — the arms below are about visibility, and a
|
||||
#: control that died in the gate would not exercise the stamp.
|
||||
_REPLY = json.dumps(
|
||||
{
|
||||
"measure": "LED-retrofit",
|
||||
"affected_items": [{"code": "ENERGI-TOTAL-EL", "quantity": 180000, "unit_cost": 1.0}],
|
||||
"claimed_saving_nok": 30000,
|
||||
}
|
||||
)
|
||||
|
||||
#: The word the un-anchored line carries and the anchored branch cannot: it prints NO line at all.
|
||||
_SENTINEL = "un-anchored"
|
||||
|
||||
|
||||
def _factory(reply: str = _REPLY):
|
||||
def factory(role: str):
|
||||
return SyntheticUsageChatClient(default_reply=reply)
|
||||
|
||||
return factory
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _isolate_model_env(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Hermetic env (mirrors ``test_scripted_cli_door_loadbearing``): the operator's Foundry
|
||||
overrides must not reach the CLI arms."""
|
||||
monkeypatch.delenv("PORTFOLIO_MODEL_MAP", raising=False)
|
||||
monkeypatch.delenv("PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT", raising=False)
|
||||
|
||||
|
||||
# --- Arm (a): the structured field ----------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_provenance_records_an_unanchored_bundle_run(fresh_store) -> None:
|
||||
"""RED: a bundle with no ``cost-baseline.json`` stamps ``cost_baseline_anchored=False``. Detach
|
||||
the wiring (stamp a constant, or drop the field) and the run again records nothing about the
|
||||
skipped stage."""
|
||||
result = await run_project(
|
||||
"BYGG-ENERGI-MIKRO-A",
|
||||
"local",
|
||||
docs_dir=str(PRE_AMENDMENT_BUNDLE),
|
||||
bundle_dir=str(PRE_AMENDMENT_BUNDLE),
|
||||
verdict_input=_VERDICT_INPUT,
|
||||
client_factory=_factory(),
|
||||
store=fresh_store,
|
||||
)
|
||||
assert result.provenance.cost_baseline_anchored is False
|
||||
|
||||
|
||||
async def test_provenance_records_an_anchored_bundle_run(fresh_store) -> None:
|
||||
"""Causality control: the SAME code path over a bundle that DOES ship a baseline stamps
|
||||
``True``. Without this the arm above would pass on a constant ``False``."""
|
||||
result = await run_project(
|
||||
"BYGG-ENERGI-BASELINE-MIKRO",
|
||||
"local",
|
||||
docs_dir=str(BASELINE_BUNDLE),
|
||||
bundle_dir=str(BASELINE_BUNDLE),
|
||||
verdict_input=_VERDICT_INPUT,
|
||||
client_factory=_factory(),
|
||||
store=fresh_store,
|
||||
)
|
||||
assert result.provenance.cost_baseline_anchored is True
|
||||
|
||||
|
||||
async def test_road_path_is_anchored_by_construction(docs_dir, fresh_store) -> None:
|
||||
"""The road path derives its baseline from the reference project's own ``cost_items``, so it is
|
||||
ALWAYS anchored — the stamp says so rather than leaving the reader to know it."""
|
||||
result = await run_project(
|
||||
"FV42-GSV-E1",
|
||||
"local",
|
||||
docs_dir=docs_dir,
|
||||
verdict_input=_VERDICT_INPUT,
|
||||
client_factory=_factory(
|
||||
json.dumps(
|
||||
{
|
||||
"measure": "Reduce scope",
|
||||
"affected_items": [{"code": "05.2", "quantity": 4300.0, "unit_cost": 215.0}],
|
||||
"claimed_saving_nok": 200000.0,
|
||||
}
|
||||
)
|
||||
),
|
||||
store=fresh_store,
|
||||
)
|
||||
assert result.provenance.cost_baseline_anchored is True
|
||||
|
||||
|
||||
def test_the_field_has_no_default() -> None:
|
||||
"""A binary fact about a falsifier gets no default: both defaults lie (see the module docstring),
|
||||
so a stamp that forgot to say must not construct at all."""
|
||||
with pytest.raises(Exception):
|
||||
ProvenanceStamp( # type: ignore[call-arg]
|
||||
citations=[],
|
||||
model="m",
|
||||
role="proposer",
|
||||
validator_decision="validated",
|
||||
token_usage=0,
|
||||
)
|
||||
|
||||
|
||||
# --- Arm (b): the renderer ------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_notice_is_rendered_only_when_unanchored() -> None:
|
||||
"""One renderer, two branches that share NO wording: un-anchored returns a line carrying the
|
||||
sentinel, anchored returns ``None`` (omitted, never an empty row — ``announce``'s rule)."""
|
||||
unanchored = cost_baseline_notice(False)
|
||||
assert unanchored is not None
|
||||
assert _SENTINEL in unanchored
|
||||
assert cost_baseline_notice(True) is None
|
||||
|
||||
|
||||
def test_dry_run_report_carries_the_anchoring(fresh_store) -> None:
|
||||
"""The dry-run type is the carrier for the surface the order measured: a run that stops before
|
||||
the first model call still knows whether the gate would have been anchored."""
|
||||
assert "cost_baseline_anchored" in DryRunReport.__dataclass_fields__
|
||||
|
||||
|
||||
# --- Arm (c): both CLI surfaces -------------------------------------------------------------------
|
||||
|
||||
|
||||
def _dry_run(bundle: Path, project_id: str) -> list[str]:
|
||||
return [
|
||||
project_id,
|
||||
"--docs-dir",
|
||||
str(bundle),
|
||||
"--bundle-dir",
|
||||
str(bundle),
|
||||
"--live-dry-run",
|
||||
]
|
||||
|
||||
|
||||
def test_cli_dry_run_announces_an_unanchored_bundle(capsys) -> None:
|
||||
"""RED (the measured defect, verbatim): ``--live-dry-run`` over a bundle without
|
||||
``cost-baseline.json`` exits 0 — and now SAYS the gate is un-anchored instead of exiting
|
||||
silently."""
|
||||
rc = run.main(_dry_run(PRE_AMENDMENT_BUNDLE, "BYGG-ENERGI-MIKRO-A"))
|
||||
assert rc == 0
|
||||
assert _SENTINEL in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_cli_dry_run_says_nothing_when_the_bundle_is_anchored(capsys) -> None:
|
||||
"""Control: the anchored bundle prints NO baseline line at all. Lines for what a run does not
|
||||
have are omitted (``announce``); a run that IS anchored has nothing to warn about."""
|
||||
rc = run.main(_dry_run(BASELINE_BUNDLE, "BYGG-ENERGI-BASELINE-MIKRO"))
|
||||
assert rc == 0
|
||||
out = capsys.readouterr().out
|
||||
assert _SENTINEL not in out
|
||||
assert "Cost baseline" not in out
|
||||
|
||||
|
||||
def test_cli_full_run_announces_an_unanchored_bundle(tmp_path, capsys) -> None:
|
||||
"""The full-run surface too, through the offline scripted door — so the notice is a property of
|
||||
a RUN, not of the dry-run branch alone."""
|
||||
replies = tmp_path / "replies.json"
|
||||
replies.write_text(
|
||||
json.dumps({"proposer": _REPLY, "checker": "Holder. VERDICT: APPROVE"}), encoding="utf-8"
|
||||
)
|
||||
rc = run.main(
|
||||
[
|
||||
"BYGG-ENERGI-MIKRO-A",
|
||||
"--docs-dir",
|
||||
str(PRE_AMENDMENT_BUNDLE),
|
||||
"--bundle-dir",
|
||||
str(PRE_AMENDMENT_BUNDLE),
|
||||
"--scripted-replies",
|
||||
str(replies),
|
||||
]
|
||||
)
|
||||
assert rc == 0
|
||||
assert _SENTINEL in capsys.readouterr().out
|
||||
|
||||
|
||||
# --- Arm (d): the outbox artefact -----------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_outbox_proposal_carries_the_anchoring(tmp_path, fresh_store) -> None:
|
||||
"""The outbox needed no change of its own: the artefact dumps the whole stamp, so the field
|
||||
lands in ``{run_id}-proposal.json`` the moment it exists on the stamp."""
|
||||
await run_project(
|
||||
"BYGG-ENERGI-MIKRO-A",
|
||||
"local",
|
||||
docs_dir=str(PRE_AMENDMENT_BUNDLE),
|
||||
bundle_dir=str(PRE_AMENDMENT_BUNDLE),
|
||||
verdict_input=_VERDICT_INPUT,
|
||||
client_factory=_factory(),
|
||||
store=fresh_store,
|
||||
outbox_dir=str(tmp_path),
|
||||
run_id="vis-1",
|
||||
)
|
||||
payload = json.loads((tmp_path / "vis-1-proposal.json").read_text(encoding="utf-8"))
|
||||
assert payload["provenance"]["cost_baseline_anchored"] is False
|
||||
|
||||
|
||||
# --- Arm (e): the portfolio surface (DEFENSIVE, and said out loud) --------------------------------
|
||||
|
||||
|
||||
def _unanchored_run() -> RunResult:
|
||||
"""One ``RunResult`` whose stamp says the gate was un-anchored."""
|
||||
proposal = SavingsProposal(
|
||||
project_id="P",
|
||||
measure="m",
|
||||
affected_items=[AffectedItem(code="05.2", quantity=1.0, unit_cost=1.0)],
|
||||
claimed_saving_nok=1.0,
|
||||
assumptions={},
|
||||
)
|
||||
return RunResult(
|
||||
outcome=Rejection(proposal=proposal, reason="r"),
|
||||
provenance=ProvenanceStamp(
|
||||
citations=[Citation(file="f.md", locator=TextSpan(0, 1), snippet="x")],
|
||||
model="synthetic",
|
||||
role="proposer",
|
||||
validator_decision="rejected",
|
||||
token_usage=1,
|
||||
cost_baseline_anchored=False,
|
||||
),
|
||||
verdict=Verdict(
|
||||
id="v1",
|
||||
proposal_features=ProposalFeatures(
|
||||
affected_codes=frozenset({"05.2"}), measure_type="m", claimed_saving_nok=1.0
|
||||
),
|
||||
decision="rejected",
|
||||
rationale="r",
|
||||
),
|
||||
retrieved=[],
|
||||
store=VerdictStore([]),
|
||||
debate_output="",
|
||||
)
|
||||
|
||||
|
||||
def test_portfolio_surface_announces_an_unanchored_run(monkeypatch, capsys) -> None:
|
||||
"""The portfolio branch reports per project, because anchoring is a per-project fact.
|
||||
|
||||
Driven by a CRAFTED ``PortfolioResult`` (the ``budget_stop`` precedent in
|
||||
``test_portfolio_cli_offline_loadbearing``), and for the same measured reason: no reference
|
||||
project sets ``bundle_dir``, so every portfolio run today takes the road path and is anchored by
|
||||
construction. This arm is therefore DEFENSIVE — it guards the surface for the day a bundle-backed
|
||||
project is wired into a pass, rather than covering a path reachable now."""
|
||||
|
||||
async def _fake(*_args, **_kwargs) -> PortfolioResult:
|
||||
return PortfolioResult(
|
||||
runs=(_unanchored_run(),),
|
||||
store=VerdictStore([]),
|
||||
validated_count=0,
|
||||
rejected_count=1,
|
||||
sum_claimed_saving_nok=0.0,
|
||||
sum_token_usage=1,
|
||||
)
|
||||
|
||||
monkeypatch.setattr(run, "run_portfolio", _fake)
|
||||
rc = run.main(["--portfolio"])
|
||||
assert rc == 0
|
||||
assert _SENTINEL in capsys.readouterr().out
|
||||
|
|
@ -33,6 +33,9 @@ _PROVENANCE = ProvenanceStamp(
|
|||
role="proposer",
|
||||
validator_decision="validated",
|
||||
token_usage=8,
|
||||
# These fixtures stand in for an ordinary complete run; the road path is anchored by
|
||||
# construction, so ``True`` is the honest value here. The un-anchored case has its own file.
|
||||
cost_baseline_anchored=True,
|
||||
)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -31,6 +31,9 @@ _PROVENANCE = ProvenanceStamp(
|
|||
role="proposer",
|
||||
validator_decision="validated",
|
||||
token_usage=8,
|
||||
# These fixtures stand in for an ordinary complete run; the road path is anchored by
|
||||
# construction, so ``True`` is the honest value here. The un-anchored case has its own file.
|
||||
cost_baseline_anchored=True,
|
||||
)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -98,6 +98,9 @@ _PROVENANCE = ProvenanceStamp(
|
|||
role="proposer",
|
||||
validator_decision="validated",
|
||||
token_usage=8,
|
||||
# These fixtures stand in for an ordinary complete run; the road path is anchored by
|
||||
# construction, so ``True`` is the honest value here. The un-anchored case has its own file.
|
||||
cost_baseline_anchored=True,
|
||||
)
|
||||
_VALIDATED = ValidatedProposal(
|
||||
proposal=_PROPOSAL, p10=100.0, p50=150.0, p90=200.0, nominal_feasible=180.0
|
||||
|
|
|
|||
|
|
@ -62,6 +62,9 @@ _PROVENANCE = ProvenanceStamp(
|
|||
role="proposer",
|
||||
validator_decision="validated",
|
||||
token_usage=8,
|
||||
# These fixtures stand in for an ordinary complete run; the road path is anchored by
|
||||
# construction, so ``True`` is the honest value here. The un-anchored case has its own file.
|
||||
cost_baseline_anchored=True,
|
||||
)
|
||||
_VALIDATED = ValidatedProposal(
|
||||
proposal=_PROPOSAL, p10=100.0, p50=150.0, p90=200.0, nominal_feasible=180.0
|
||||
|
|
|
|||
|
|
@ -19,6 +19,7 @@ def _stamp() -> ProvenanceStamp:
|
|||
role="proposer",
|
||||
validator_decision="validated",
|
||||
token_usage=42,
|
||||
cost_baseline_anchored=True,
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -30,6 +31,7 @@ def test_zero_citations_is_rejected() -> None:
|
|||
role="proposer",
|
||||
validator_decision="validated",
|
||||
token_usage=1,
|
||||
cost_baseline_anchored=True,
|
||||
)
|
||||
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue