feat(validator,generate,run): an identifier a proposal builds on must be in the input, or the verdict falls [skip-docs]
P6 (økt 108) ended in ValidatedProposal (verdict 5fd6272e3725fe68) on two cost codes -- M-04-01 / M-04-03 -- that appear in NO prompt of that run. Measured here first, verbatim: validate_proposal(p, baseline=None) validates it; the same proposal against any non-empty CostBaseline is rejected naming both codes. So the hole was never "fabrication goes uncaught" -- _reconcile_against_baseline exists and is right -- but that the falsifier is reached only through `if baseline is not None`. The input always exists; the baseline does not. New stage 0b (_ground_against_input), OUTSIDE the baseline branch, after stage 0 so an anchored run's message is byte-identical to before. ONE Rejection, the validator's own type, naming EVERY ungrounded identifier "; "-joined in the proposal's own order (økt 94's completeness reason). The rule has NO pattern -- `code in grounding`, exact substring -- and that is a measurement: over the delivered corpora (K2 1108 files / 2 005 561 chars, the three N payloads 8 excerpts each) the identifier forms are heterogeneous, and a pattern chosen to cover them would be a rule about shapes. Bare numerals are the one inert class (46 394 occurrences / 2 117 distinct in K2); the rule fails OPEN there, never closed. Evidence is three non-model-authored sources: what run_project DELIVERED (the rendered cut/pointer/chunks plus the base's context_files -- never files, which would make the type: verdict layer evidence), the project's own cost lines, and the baseline's codes when anchored. The rendered PROMPT is deliberately NOT evidence, on two measurements: gen_context IS the debate output on the S2c path, and from attempt 2 the prompt carries the previous Rejection.reason verbatim -- which for this stage QUOTES the identifier it just refused. Grounding in the prompt would let the gate's own refusal disarm it on its second round. Prose scanning was chosen against WITH THE NUMBERS: a typed gate catches 2/2 (P6) and 2/2 (S7c) -- 100% of what reached a verdict. What stays uncaught, said plainly: an ungrounded identifier that lives only in agent/debate prose and never becomes an affected_item code (2 of 4 P6, 2 of 4 S7c, 1 of 2 P4). Iron Law: 9 red / 2 green before the rule existed. Ten mutations all red against the whole suite, green control 1558 passed / 5 skipped (from 1543/5, superset, 0 removed), golden demo-transcript.stdout BYTE-UNCHANGED (shasum -a 1 of the CONTENT = ea8c534773acdbe41ae68f2c55724d69aaf8be4f). Three existing fixtures changed, no gate weakened -- most of all test_pre_amendment_bundle_runs_unchanged, which sent the SAME FABRICATED code and asserted it validated: the økt-108 hole written down as an expectation. No paid run. Order 20260909T113641Z-38938691-from-.claude. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
999846a485
commit
277bb95777
12 changed files with 831 additions and 13 deletions
58
CLAUDE.md
58
CLAUDE.md
|
|
@ -2258,6 +2258,64 @@ Python ≥3.10. MAF (`agent-framework-core` 1.16.0, `-orchestrations` 1.1.1 —
|
||||||
hver sin uavhengige gate. **Én betalt N100-arm med `--require-cost-baseline`: rc 1, 0 modellkall,
|
hver sin uavhengige gate. **Én betalt N100-arm med `--require-cost-baseline`: rc 1, 0 modellkall,
|
||||||
NOK 0,00** — (b′) forblir STRUKTURELT umålbar under flagget (F4: en vegnormal bærer ingen
|
NOK 0,00** — (b′) forblir STRUKTURELT umålbar under flagget (F4: en vegnormal bærer ingen
|
||||||
kostlinjer, uendret av denne fiksen). Måling: `docs/2026-09-08-n-bundlene-hypoteseform.md` § 10.
|
kostlinjer, uendret av denne fiksen). Måling: `docs/2026-09-08-n-bundlene-hypoteseform.md` § 10.
|
||||||
|
- **En identifikator forslaget bygger på skal finnes ORDRETT i inputen, ellers faller dommen — og
|
||||||
|
hullet var ALDRI at fabrikasjon var ufanget (P7, 09.09):** `_reconcile_against_baseline`
|
||||||
|
(`validator.py`) bærer allerede setningen «the cost code is absent from the baseline — a
|
||||||
|
fabricated line», men den nås KUN gjennom `if baseline is not None`, så falsifisereren var koblet
|
||||||
|
til om det tilfeldigvis fantes en kostnadsbaseline. MÅLT (økt 108, dom `5fd6272e3725fe68`): en
|
||||||
|
uforankret K2-kjøring endte i `ValidatedProposal` på to kostkoder — `M-04-01`/`M-04-03` — som
|
||||||
|
står i INGEN prompt av den kjøringen, og S7cs `PRD-001`/`PRD-002` falt på MAGNITUDE (stage 4),
|
||||||
|
ikke på fabrikasjon. **Inputen finnes ALLTID; baselinen gjør ikke det.** `_ground_against_input`
|
||||||
|
er derfor et EGET stadium (0b) som ikke ser på `baseline`, plassert ETTER stage 0 så en forankret
|
||||||
|
kjøring der begge ville fyrt får byte-identisk samme melding (baselinens setning navngir
|
||||||
|
prosjektet og kodetallet, og Steg 5 mater nettopp den tilbake). ÉN `Rejection`, validatorens egen
|
||||||
|
type, som navngir HVER ugrunnet identifikator `"; "`-joinet i FORSLAGETS rekkefølge (økt 94s
|
||||||
|
fullstendighets-grunn). **REGELEN HAR INTET MØNSTER, og det er en MÅLING:** sjekken er
|
||||||
|
`code in grounding` — eksakt delstreng — fordi de leverte korpusene bærer heterogene former
|
||||||
|
(K2: 499 `UPPER-num`-treff / 23 unike, 25 enkeltbokstav-koder à `B-20-00-00`, **0** `Krav`-numre i
|
||||||
|
kroppen; N-payloadene: `Krav X.Y.Z—N` i `req_number`/`title` 8/8 i hver, 71 UUID-er i N200), så et
|
||||||
|
mønster valgt for å dekke dem ville vært en regel om FASONGER. **Rene tall er den ene inerte
|
||||||
|
klassen** (K2: 46 394 forekomster / 2 117 distinkte), og feilretningen er ÅPEN — regelen kan ikke
|
||||||
|
felle en ekte kode. **Beviset er TRE ikke-modell-forfattede kilder** (`_grounding_text`):
|
||||||
|
`delivered` fra `run_project` (den leverte rendringen + basens `context_files`, **aldri `files`** —
|
||||||
|
den property-en dropper `type: verdict`-laget, og å grunne et forslag i en tidligere DOM ville
|
||||||
|
rutet ExpeL-foldens materiale rundt sin egen gate), prosjektets EGNE `cost_items` (MÅLT:
|
||||||
|
`_project_from_bundle` bygger `cost_items=()`, så kilden bidrar med null på bundle-stien) og
|
||||||
|
baselinens koder når kjøringen er forankret (stage 0 har alt dømt dem EKTE; det svakere stadiet
|
||||||
|
skal ikke overprøve det sterkere). **Den rendrede PROMPTEN er BEVISST IKKE bevis, og det er to
|
||||||
|
målinger:** på S2c-stien er `gen_context` DEBATT-OUTPUTEN (en proposer som navngir en kode i en
|
||||||
|
debatt-tur grunner så sitt eget forslag i den), og fra forsøk 2 bærer prompten forrige
|
||||||
|
`Rejection.reason` ORDRETT (Steg 5) — som for dette stadiet SITERER identifikatoren den nettopp
|
||||||
|
nektet, altså **en falsifiserer som avvæpner seg selv på sin andre runde**. **Prosa-skanningen er
|
||||||
|
VALGT BORT med tall:** en typet gate fanger 2/2 (P6) og 2/2 (S7c) — 100 % av det som nådde en DOM
|
||||||
|
— mens en prosa-gate legger til fire tokens som aldri nådde noen dom og krever nøyaktig det
|
||||||
|
mønsteret regelen unngår. **Ufanget forblir, uttalt:** en ugrunnet identifikator som kun står i
|
||||||
|
agent-/debatt-prosa og aldri blir en `affected_item`-kode (2 av 4 på P6, 2 av 4 på S7c, 1 av 2 på
|
||||||
|
P4). Testene leser SPOREDE fixturer (`tests/fixtures/p7-grounding/`, de tre genererings-promptene
|
||||||
|
ORDRETT) — opptakene ligger i `scratchpad/`, som `git archive HEAD` ikke bærer, så en test som
|
||||||
|
leste dem ville felt handover-gaten. Kjent-positiven `Krav 3.3.1—13` (EM-DASH; bindestrek-varianten
|
||||||
|
scorer 0/6) flagges IKKE, med KONTROLLEN `CRS-01` på SAMME opptak. Load-bearing MÅLT
|
||||||
|
(`tests/test_identifier_grounding_loadbearing.py`, 15 armer, **9 røde / 2 grønne FØR regelen**),
|
||||||
|
**ti mutasjoner alle røde mot HELE suiten** + grønn kontroll **1558/5** (fra 1543/5, supersett,
|
||||||
|
0 fjernet) og golden `demo-transcript.stdout` BYTE-UENDRET (`shasum -a 1` av INNHOLDET =
|
||||||
|
`ea8c534773acdbe41ae68f2c55724d69aaf8be4f`): M1 regelen finner aldri noe (10) · M2 regelen flagger
|
||||||
|
alt (72, inkl. kjent-positiven OG golden-transkriptet) · M3 grunnen navngir bare første (3) ·
|
||||||
|
M4 stadiet bak `baseline is not None` (9) · M5 detach `generate.py`-wiringen (3) · M6 `run_project`
|
||||||
|
erklærer ingen levert input (33, de fleste eldre) · M7 bygg fra `bundle.files` (1, den armen
|
||||||
|
alene) · M8 dropp baselinens koder (1) · M9 dropp prosjektets kostlinjer (15, alle i eldre tester
|
||||||
|
— kilden har ingen egen arm, og det ER dens vitne) · M10 prompten blir bevis igjen (2).
|
||||||
|
**Tre eksisterende fixturer endret, ingen gate svekket** — mest av alt
|
||||||
|
`test_s40_cost_baseline_loadbearing::test_pre_amendment_bundle_runs_unchanged`, som sendte den
|
||||||
|
SAMME FABRIKKERTE koden og påsto at den validerte: økt-108-hullet skrevet ned som en FORVENTNING.
|
||||||
|
Den bærer nå en kode basen navner. **Ærlighets-grenser, uttalt:** ingen betalt kjøring bekrefter
|
||||||
|
at regelen endrer utfallet levende (tre opptak, ÉN modell, ETT deployment — structured-output-
|
||||||
|
grensens klasse); `assumptions`-nøkler sjekkes ikke (en nøkkel uten `affected_item` samples aldri
|
||||||
|
av Monte Carlo, så den kan ikke flytte dommen); de tre andre `validate_proposal`-kallstedene er
|
||||||
|
BEVISST urørt (`self_repair` er en ren hjelper, `run.py:450` bygger kandidaten FRA baselinen og
|
||||||
|
kan strukturelt ikke fabrikkere — økt 82s M7-grense — og `quick_validate` er nivå 1, rådgivende);
|
||||||
|
den hostede flaten er urørt (feltet er i ingen av hostings tre sett); og
|
||||||
|
`--require-cost-baseline` er IKKE gjort til default (F4/D-3). Måling:
|
||||||
|
`docs/2026-09-09-p7-forankrede-identifikatorer.md`.
|
||||||
- **STATE.md er local-only** (gitignored). Voyage session-state er efemert; STATE.md er kanonisk kontinuitet.
|
- **STATE.md er local-only** (gitignored). Voyage session-state er efemert; STATE.md er kanonisk kontinuitet.
|
||||||
- Prosess: Voyage-plugin (`/trekbrief → /trekplan → /trekexecute → /trekreview`) per større fase.
|
- Prosess: Voyage-plugin (`/trekbrief → /trekplan → /trekexecute → /trekreview`) per større fase.
|
||||||
|
|
||||||
|
|
|
||||||
223
docs/2026-09-09-p7-forankrede-identifikatorer.md
Normal file
223
docs/2026-09-09-p7-forankrede-identifikatorer.md
Normal file
|
|
@ -0,0 +1,223 @@
|
||||||
|
# P7 — en identifikator forslaget bygger på skal finnes ORDRETT i inputen, ellers faller dommen
|
||||||
|
|
||||||
|
**Ordre:** `20260909T113641Z-38938691-from-.claude` · **Økt 109** · 2026-09-09
|
||||||
|
**Kontroll før arbeidet:** 1543 passed / 5 skipped · `ruff check src tests` og `uv run mypy src` rene
|
||||||
|
**Kontroll etter:** **1558 passed / 5 skipped** (+15 node-ider, 0 fjernet)
|
||||||
|
**Golden `demo-transcript.stdout`:** `shasum -a 1` av INNHOLDET = `ea8c534773acdbe41ae68f2c55724d69aaf8be4f`, UENDRET
|
||||||
|
**Ingen betalt kjøring.** Alt i dette dokumentet er målt på eksisterende, gratis opptak. **NOK 0,00.**
|
||||||
|
|
||||||
|
## 0. Hva som ER målt, og hva som IKKE er det
|
||||||
|
|
||||||
|
**MÅLT:** at hullet finnes nøyaktig der PM sa (`validate_proposal(p, baseline=None)` på P6-forslaget
|
||||||
|
gir `ValidatedProposal`; samme forslag med en ikke-tom `CostBaseline` gir `Rejection` som navngir
|
||||||
|
BEGGE koder); hvilke identifikator-former de leverte korpusene faktisk bærer, med nevnere; hva en
|
||||||
|
typet gate og en prosa-gate ville fanget på hvert av de tre opptakene, med nevnere; og at den nye
|
||||||
|
regelen feller de fire fabrikkerte kodene mens kjent-positiven `Krav 3.3.1—13` står.
|
||||||
|
|
||||||
|
**IKKE MÅLT:** at regelen endrer utfallet i en LEVENDE kjøring. Ingen betalt kjøring er gjort, og
|
||||||
|
ingen er bestilt. De tre opptakene kommer fra ÉN modell og ETT deployment.
|
||||||
|
|
||||||
|
**DOMMEN:** hullet var ikke «fang fabrikasjon» — den koden finnes og er riktig — men at
|
||||||
|
forankringen var **koblet til om det tilfeldigvis fantes en kostnadsbaseline**. Inputen finnes
|
||||||
|
alltid; baselinen gjør ikke det. Regelen er derfor et EGET stadium (0b) som ikke ser på `baseline`
|
||||||
|
i det hele tatt.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Premissene, hver verifisert selv
|
||||||
|
|
||||||
|
| Premiss (PM målte 09.09) | Verifisert her med | Utfall |
|
||||||
|
|---|---|---|
|
||||||
|
| (i) HEAD `999846a`, `git ls-remote origin main` = `eb41374` → UPUSHET = 1 | `git log --oneline -1`, `git ls-remote origin main` | **BEKREFTET** |
|
||||||
|
| (ii) To utrackede: presentasjons-HTML (parallell økt) + `scratchpad/` | `git status --short` | **BEKREFTET.** Ingen tredje. Begge forblir utrackede |
|
||||||
|
| (iii) 1543 passed / 5 skipped, ruff + mypy rene, golden `ea8c534…` | `uv run pytest -q`, `ruff`, `mypy`, `shasum -a 1` | **BEKREFTET, alle fire** |
|
||||||
|
| (iv) `_reconcile_against_baseline` bak `if baseline is not None` | Kjørt PMs egen kontroll på P6-forslaget | **BEKREFTET ORDRETT.** `baseline=None` → `ValidatedProposal`; ikke-tom baseline → `Rejection: unknown cost code 'M-04-01' … 'M-04-03'` |
|
||||||
|
| (v) P6 `validated`/`5fd6272e3725fe68`/`approve`; S7c `rejected` på MAGNITUDE | Lest begge `*-outcome.json` + begge `*-proposal.json` | **BEKREFTET.** S7cs grunn er «claimed saving 1400000 exceeds P90 feasible 879107» — stage 4, ikke fabrikasjon |
|
||||||
|
| (vi) ugrunnede identifikatorer per opptak med PMs grove mønster | Re-målt, se § 3 | **BEKREFTET** (2/2 · 4/4 · 1/2), og mønsteret er erstattet — se § 2 |
|
||||||
|
| (vii) `Krav 3.3.1—13` i 6/6 prompter OG 6/6 svar, EM-DASH | Re-målt på `p4-n100-free-records.json` | **BEKREFTET.** Bindestrek-varianten: 0 i begge |
|
||||||
|
| (viii) sømmen er `generate.py` (`_fetch_parsed` → `validate_proposal`); fem kallsteder | Lest alle: `validator.py:292` (`self_repair`), `run.py:450` (mandat, ingen modell), `explore.py:1266` (`quick_validate`, rådgivende), `generate.py` | **BEKREFTET.** Regelen wires i `generate.py`; de tre andre er uendret, med grunn i § 6 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Mønsteret, målt fra korpuset — og hvorfor regelen ikke har noe mønster
|
||||||
|
|
||||||
|
Målt over det som faktisk ble levert: K2-basen (`scratchpad/s7c/k2-bundle-s7c/`, **1 108 `.md`,
|
||||||
|
2 005 561 tegn**) og de tre N-payloadene (`scratchpad/nbundler-p2/payload-n{100,200,500}.json`,
|
||||||
|
**8 leverte utdrag hver**).
|
||||||
|
|
||||||
|
| Form | K2 (treff / unike) | N100 | N200 | N500 | Tas inn i regelen? |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| `Krav X.Y.Z—N` (em-dash) i KROPPEN | 0 / 0 | 0 | 0 | 0 | — se raden under |
|
||||||
|
| `Krav X.Y.Z—N` i `req_number` / `title` | — | 8/8 | 8/8 | 8/8 | **JA** (den bor i frontmatter, ikke i kroppen — derfor leser grunnlaget også `frontmatter`) |
|
||||||
|
| `UPPER-num` (`FOR-2005-04`, `HKFK-22`) | 499 / 23 | 0 | 0 | 0 | **JA** |
|
||||||
|
| Enkelt-bokstav-kode (`B-20-00-00`, `M-04-01`-formen) | 25 / 14 | 0 | 0 | 0 | **JA** |
|
||||||
|
| UUID | 0 / 0 | 0 | 71 / 71 | 0 | **JA** (høy entropi; ingen ekstra risiko) |
|
||||||
|
| `concept_id`-slugger | 1 108 stier | 8/8 | 8/8 | 8/8 | **JA** |
|
||||||
|
| Rene tall, 2–6 sifre | **46 394 / 2 117** | 153 / 37 | 132 / 80 | 19 / 14 | **JA i formen, men regelen er INERT der** — se under |
|
||||||
|
| Punktnummer (`3.3.4`) | 7 392 / 1 755 | 9 / 2 | 25 / 8 | 1 | samme |
|
||||||
|
|
||||||
|
**Konklusjonen målingen tvang fram: regelen har ikke noe mønster i det hele tatt.** Sjekken er
|
||||||
|
`code in grounding` — eksakt delstreng. Formene over er heterogene nok til at et mønster valgt for
|
||||||
|
å dekke dem ville vært en regel om FASONGER, ikke om forankring, og hver form jeg ikke kunne telle
|
||||||
|
ville blitt en gjetning. En mønsterløs regel kan ikke ta feil om en form korpuset bærer.
|
||||||
|
|
||||||
|
**Rene tall er den ene svakheten, og den er MÅLT, ikke antatt:** K2 bærer 46 394 forekomster av
|
||||||
|
tall over 2 117 distinkte verdier, så en ren-numerisk kode er nesten alltid «grunnet» ved et
|
||||||
|
sammentreff (P6 § 4(c) så nøyaktig dette med `1000`). Regelen feiler altså **ÅPENT** der, aldri
|
||||||
|
lukket — den kan ikke felle en ekte kode, den kan bare la en oppdiktet numerisk kode passere. Det er
|
||||||
|
den ærlige feilretningen for en gate som uttaler seg om en modells oppfinnelse, og den står som en
|
||||||
|
grense i § 6, ikke som en påstand om dekning.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Hva de to variantene ville fanget, per opptak, med nevnere
|
||||||
|
|
||||||
|
«Ugrunnet» = finnes i et SVAR, finnes ikke i NOEN prompt i samme opptak.
|
||||||
|
|
||||||
|
| Opptak | Variant A (typede `affected_items`) | Variant B (identifikator-formede tokens i svarets PROSA) |
|
||||||
|
|---|---|---|
|
||||||
|
| P6 `scratchpad/s7c/p6-Aopen-records.json` (5 records) | **2 av 2 ugrunnet** — `M-04-01`, `M-04-03` | **4 av 4** — `MER-001`, `MER-002`, `M-04-01`, `M-04-03` |
|
||||||
|
| S7c `scratchpad/s7c/Aopen-records.json` (11 records) | **2 av 2** — `PRD-001`, `PRD-002` | **4 av 4** — `MEETINGS-05`, `LOGGING-02`, `PRD-001`, `PRD-002` |
|
||||||
|
| P4 `scratchpad/nbundler-p2/p4-n100-free-records.json` (6 records) | **nevner 0** — opptaket har intet lagret forslag (fri kjøring) | **1 av 2** — `CRS-01` ugrunnet, **`Krav 3.3.1—13` GRUNNET** (6/6 prompter) |
|
||||||
|
|
||||||
|
**VALGT: variant A. Prosa-skanningen er IKKE bygget.** Begrunnelsen er tallene over, ikke smak:
|
||||||
|
|
||||||
|
* variant A fanger **100 % av de identifikatorene som faktisk nådde en DOM** (2/2 og 2/2). Ordrens
|
||||||
|
tittel er «or the verdict falls», og de fire tokenene variant B legger til nådde aldri en dom:
|
||||||
|
`MER-001`/`MER-002` sto i en debatt-tur, og en debatt har ingen dom å felle;
|
||||||
|
* variant A trenger **intet mønster** (§ 2). Variant B kan ikke bygges uten ett, og hvert mønster
|
||||||
|
er en ny kilde til både falske positive og falske negative på et korpus jeg har tre opptak av;
|
||||||
|
* variant A er allerede den formen validatoren tar (en typet `SavingsProposal`), så den blir et
|
||||||
|
stadium ved siden av de andre i stedet for en ny inngang.
|
||||||
|
|
||||||
|
**HVA SOM DA FORBLIR UFANGET, ordrett:** en ugrunnet identifikator som KUN står i agent-/debatt-prosa
|
||||||
|
og aldri blir en `affected_item`-kode. Målt: **2 av 4 på P6** (`MER-001`, `MER-002`), **2 av 4 på
|
||||||
|
S7c** (`LOGGING-02`, `MEETINGS-05`), og **1 av 2 på P4** (`CRS-01`, i en fri kjøring uten forslag i
|
||||||
|
det hele tatt). Det samme står i STATEs NESTE.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Regelen
|
||||||
|
|
||||||
|
`validator._ground_against_input(proposal, grounding)` — **stadium 0b**, ETTER stage 0 og FØR
|
||||||
|
løseren, og **utenfor** `if baseline is not None`. Returnerer ÉN `Rejection` (validatorens egen
|
||||||
|
type, aldri en ny domstype) som navngir **HVER** ugrunnet identifikator, `"; "`-joinet, i
|
||||||
|
FORSLAGETS egen rekkefølge — samme form og samme målte grunn som `_reconcile_against_baseline`
|
||||||
|
(økt 94: en melding som navngir bare den første leses som en instruks om å fikse den ene).
|
||||||
|
|
||||||
|
`_reconcile_against_baseline` er **URØRT**. 0b er plassert ETTER stage 0, så en forankret kjøring
|
||||||
|
der begge ville fyrt får byte-identisk samme melding som før: baselinens setning er den mer
|
||||||
|
handlingsbare (den navngir prosjektet og hvor mange koder det kjenner), og Steg 5 mater nettopp den
|
||||||
|
tilbake.
|
||||||
|
|
||||||
|
### 4.1 Beviset er tre kilder, og PROMPTEN er BEVISST IKKE en av dem
|
||||||
|
|
||||||
|
`generate._grounding_text(project, baseline, delivered)` komponerer på ÉTT sted:
|
||||||
|
|
||||||
|
1. **`delivered`** — hva KALLEREN kan bevise at kjøringen fikk. `run_project` fyller den fra den
|
||||||
|
leverte rendringen (pre-pass-kuttet / bundle-pekeren / vegstiens hentede chunks) PLUSS den
|
||||||
|
navigerte basens `context_files` — **aldri `files`**, som er egenskapen som dropper
|
||||||
|
`type: verdict`-laget på hvert nivå (MAJOR-3 N2 / S7a-3). Å grunne et forslag i en tidligere DOM
|
||||||
|
ville rutet ExpeL-foldens eget materiale rundt sin egen gate;
|
||||||
|
2. **prosjektets EGNE kostlinjer.** Vegstiens estimat ER prosjektet. **MÅLT:**
|
||||||
|
`_project_from_bundle` bygger `cost_items=()`, så denne kilden bidrar med INGENTING på
|
||||||
|
bundle-stien og gaten forblir like skarp der fabrikasjonen ble målt;
|
||||||
|
3. **baselinens koder når kjøringen er forankret.** Stage 0 har allerede dømt hver kode som når 0b
|
||||||
|
som en EKTE linje; det svakere stadiet skal ikke overprøve det sterkere fordi et prosa-sammendrag
|
||||||
|
ikke gjentok koden.
|
||||||
|
|
||||||
|
**Den rendrede PROMPTEN er ikke bevis, og det er en MÅLING — ikke smak.** To av promptens deler er
|
||||||
|
modellens egne ord matet tilbake:
|
||||||
|
|
||||||
|
* på S2c-bundle-stien er `gen_context` **debatt-outputen**. Målt: en skriptet proposer som navngir
|
||||||
|
en kode i en debatt-tur grunner så sitt eget forslag i den turen (armen
|
||||||
|
`test_g_run_project_grounds_a_candidate_in_the_base_it_delivered` var RØD på nøyaktig dette før
|
||||||
|
prompten ble tatt ut av beviset);
|
||||||
|
* fra forsøk 2 bærer prompten forrige `Rejection.reason` ORDRETT (Steg 5) — og dette stadiets grunn
|
||||||
|
**siterer identifikatoren den nettopp nektet**. Med prompten som bevis ville gatens egen nekt
|
||||||
|
grunnet neste forsøk: **en falsifiserer som avvæpner seg selv på sin andre runde.** Gatet av
|
||||||
|
`test_h_the_refusal_does_not_ground_the_next_attempt_that_repeats_the_code`.
|
||||||
|
|
||||||
|
### 4.2 Sporet, og hva som ikke er rørt
|
||||||
|
|
||||||
|
`grounding` er VALGFRI på `validate_proposal` av samme grunn som `baseline` er det (`None` = pre-P7,
|
||||||
|
så hver eksisterende kaller, hver golden og demo-transkriptet er uendret), men den lukker et ANNET
|
||||||
|
hull: stage 0 fyrer bare når en baseline finnes, mens inputen finnes alltid.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Testene, kjent-positiven og mutasjonene
|
||||||
|
|
||||||
|
**Iron Law:** testfila ble skrevet FØRST og var **9 røde / 2 grønne** (de to grønne var kontrollene,
|
||||||
|
som korrekt skal passere pre-P7) før én linje av regelen fantes.
|
||||||
|
|
||||||
|
**Fixturene er SPORET, ikke lest fra `scratchpad/`.** Opptakene ligger utracket og finnes ikke i
|
||||||
|
`git archive HEAD`, så en test som leste dem ville passert her og felt handover-gaten.
|
||||||
|
`tests/fixtures/p7-grounding/` bærer de tre genererings-promptene ORDRETT (1 788 / 1 397 / 985
|
||||||
|
tegn) — hele inputen proposeren så på forsøket som produserte kandidaten. Ingen test skipper.
|
||||||
|
|
||||||
|
**Kjent-positiven, bindende:** `Krav 3.3.1—13` (EM-DASH, U+2014) står i N100-genererings-prompten og
|
||||||
|
flagges IKKE. **KONTROLLEN på samme opptak:** `CRS-01` — det ene identifikator-formede tokenet den
|
||||||
|
kjøringen produserte som ingen prompt bærer — FELLES. Uten den kontrollen ville en grønn
|
||||||
|
kjent-positiv ikke bevist noe.
|
||||||
|
|
||||||
|
### Mutasjonstabell — ti mutasjoner, ALLE RØDE mot HELE suiten
|
||||||
|
|
||||||
|
Grønn kontroll **1558 passed / 5 skipped**, golden BYTE-UENDRET, restaurert fra
|
||||||
|
`scratchpad/p7/mut-backup/` + `shasum -c` etter hver (aldri `git checkout`), ÉN per kjøring.
|
||||||
|
|
||||||
|
| # | Mutasjon | Røde |
|
||||||
|
|---|---|---|
|
||||||
|
| M1 | regelen finner aldri en violation | **10** |
|
||||||
|
| M2 | regelen flagger ALT (grunnet eller ei) | **72** — inkl. kjent-positiven og golden-transkriptet |
|
||||||
|
| M3 | grunnen navngir bare den FØRSTE violationen | **3** |
|
||||||
|
| M4 | stadiet legges bak `baseline is not None` (økt-108-hullet gjeninnført) | **9** |
|
||||||
|
| M5 | detach `grounding=` i `generate.py` | **3** |
|
||||||
|
| M6 | `run_project` erklærer ingen levert input | **33** — hvorav de fleste i tester eldre enn dette arbeidet |
|
||||||
|
| M7 | grunnlaget bygges fra `bundle.files` (verdict-laget blir bevis) | **1** — den armen ALENE |
|
||||||
|
| M8 | komponisten dropper baselinens koder | **1** |
|
||||||
|
| M9 | komponisten dropper prosjektets egne kostlinjer | **15** — alle i eldre tester; kilden har ingen egen arm i den nye fila, og det er dens vitne |
|
||||||
|
| M10 | den rendrede prompten blir bevis igjen (selv-avvæpningen) | **2** |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Honesty limits
|
||||||
|
|
||||||
|
* **Ingen betalt kjøring bekrefter at regelen endrer utfallet levende.** Alt her er målt offline på
|
||||||
|
tre gratis opptak fra ÉN modell og ETT deployment. En levende bekreftelse er en senere, separat
|
||||||
|
ordre.
|
||||||
|
* **Prosa-skanningen er valgt bort med tallene i § 3, ikke bygget.** Ufanget forblir: en ugrunnet
|
||||||
|
identifikator som kun står i agent-/debatt-prosa og aldri blir en `affected_item`-kode — målt
|
||||||
|
2 av 4 (P6), 2 av 4 (S7c), 1 av 2 (P4).
|
||||||
|
* **Rene og svært korte identifikatorer er regelen INERT mot** (§ 2, 46 394 forekomster / 2 117
|
||||||
|
distinkte i K2). Feilretningen er åpen, aldri en falsk nekt.
|
||||||
|
* **`assumptions`-nøkler sjekkes ikke.** En nøkkel som ikke navngir noe `affected_item` samples
|
||||||
|
aldri av Monte Carlo (`SavingsProposal._assumption_bands_enclose_unit_cost` sier det i de samme
|
||||||
|
ordene), så den kan ikke flytte dommen; en sjekk på den ville vært en gren intet opptak øver.
|
||||||
|
* **De tre andre `validate_proposal`-kallstedene er BEVISST urørt**, hvert med sin grunn:
|
||||||
|
`self_repair` er en ren hjelper uten input-tekst; `run.py:450` (`evaluate_mandate_candidates`)
|
||||||
|
bygger kandidaten FRA baselinen, så den kan strukturelt ikke fabrikkere (økt 82s M7-grense); og
|
||||||
|
`explore.py:1266` (`quick_validate`) er nivå 1, rådgivende, og når aldri provenance.
|
||||||
|
* **Den hostede flaten er urørt** — `grounding` er ikke i noen av hostings tre sett, så den
|
||||||
|
generiske 400-en svarer og Fase 4es to halvdeler står (MAJOR-4/S7bs eget valg gjentatt).
|
||||||
|
* **`--require-cost-baseline` er IKKE gjort til default** (F4 valgte den opt-in, D-3 låste bruken på
|
||||||
|
N-kjøringer). P7 gjør den mindre nødvendig, ikke overflødig.
|
||||||
|
* **Tre eksisterende fixturer ble endret, ingen gate svekket:** `test_structured_output_loadbearing`
|
||||||
|
gir sin syntetiske kontekst linja forslaget gjenforteller; `test_dimension_loadbearing`s
|
||||||
|
fremmed-dimensjon-arm bytter `SENTINEL-FOREIGN` mot en kode basen NAVNER (armen ville ellers ridd
|
||||||
|
på forankringen i stedet for på `admits` — den er nå SKARPERE); og
|
||||||
|
`test_s40_cost_baseline_loadbearing::test_pre_amendment_bundle_runs_unchanged` sendte den SAMME
|
||||||
|
FABRIKKERTE koden og påsto at den validerte — altså økt-108-hullet skrevet ned som en
|
||||||
|
FORVENTNING. Den bærer nå en kode basen navner og beviser dermed det den påstår (en uforankret
|
||||||
|
kjøring når fortsatt en dom) uten også å love at fabrikasjon klarerer.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Reproduksjon (gratis)
|
||||||
|
|
||||||
|
```
|
||||||
|
uv run pytest -q tests/test_identifier_grounding_loadbearing.py
|
||||||
|
```
|
||||||
|
|
||||||
|
Mønster- og variant-målingene i § 2 og § 3 kjøres med `scratchpad/p7/measure_forms.py` og
|
||||||
|
`scratchpad/p7/measure_variants.py` (utracket; de leser opptakene under `scratchpad/`).
|
||||||
|
|
@ -420,6 +420,42 @@ def generate_with_validation(
|
||||||
return self_repair(_attempt, max_attempts=max_attempts)
|
return self_repair(_attempt, max_attempts=max_attempts)
|
||||||
|
|
||||||
|
|
||||||
|
def _grounding_text(project: Project, baseline: CostBaseline | None, delivered: str) -> str:
|
||||||
|
"""P7: compose the ONE text a candidate's identifiers must be grounded in — the run's
|
||||||
|
non-model-authored input, and nothing else.
|
||||||
|
|
||||||
|
Three sources, each of which the run can point at without asking the model:
|
||||||
|
|
||||||
|
* ``delivered`` — what the CALLER can prove this run was GIVEN. ``run_project`` fills it from
|
||||||
|
the delivered rendered context (the pre-pass cut, the bundle pointer, or the road path's
|
||||||
|
retrieved chunks) PLUS the navigated base's ``context_files`` — never ``files``, because that
|
||||||
|
is the property which drops the ``type: verdict`` layer at every level, and grounding a
|
||||||
|
proposal in a prior verdict would route the ExpeL fold's own material around its gate;
|
||||||
|
* the project's OWN cost lines. The road path's estimate IS the project, so a code it carries
|
||||||
|
is real whether or not any text restated it. MEASURED: ``_project_from_bundle`` builds
|
||||||
|
``cost_items=()``, so this source contributes nothing on the bundle path and the gate stays
|
||||||
|
exactly as sharp where fabrication was measured (K2, økt 108);
|
||||||
|
* the baseline's codes when a run is anchored. Stage 0 has already ruled every code that
|
||||||
|
reaches stage 0b a REAL line of this project; the weaker stage must not overrule the stronger
|
||||||
|
falsifier because a prose summary happened not to repeat the code.
|
||||||
|
|
||||||
|
**The rendered PROMPT is deliberately NOT a source, and that is a measurement, not taste.** Two
|
||||||
|
of the prompt's parts are the model's own words fed back to it: on the S2c bundle path
|
||||||
|
``gen_context`` is the DEBATE OUTPUT (measured — a scripted proposer that names a code in a
|
||||||
|
debate turn then grounds its own proposal in that turn), and from attempt 2 onward the prompt
|
||||||
|
carries the previous ``Rejection.reason`` VERBATIM (Step 5) — which, for this stage, quotes the
|
||||||
|
very identifier it just refused. Grounding in the prompt would therefore let the gate's own
|
||||||
|
refusal ground the next attempt: a falsifier that disarms itself on its second round.
|
||||||
|
"""
|
||||||
|
return "\n".join(
|
||||||
|
[
|
||||||
|
delivered,
|
||||||
|
*(item.code for item in project.cost_items),
|
||||||
|
*(() if baseline is None else baseline.items),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
async def generate_via_llm(
|
async def generate_via_llm(
|
||||||
chat_client: BaseChatClient,
|
chat_client: BaseChatClient,
|
||||||
project: Project,
|
project: Project,
|
||||||
|
|
@ -434,6 +470,7 @@ async def generate_via_llm(
|
||||||
reviews: list[ProposalReview] | None = None,
|
reviews: list[ProposalReview] | None = None,
|
||||||
review_key: tuple[str | None, str | None] = (None, None),
|
review_key: tuple[str | None, str | None] = (None, None),
|
||||||
checker_verdict: str = "absent",
|
checker_verdict: str = "absent",
|
||||||
|
grounding: str | None = None,
|
||||||
) -> GenerationResult:
|
) -> GenerationResult:
|
||||||
"""Async LLM path: non-streaming chat -> parse -> validate, with TWO bounded retry kinds,
|
"""Async LLM path: non-streaming chat -> parse -> validate, with TWO bounded retry kinds,
|
||||||
the meter checked in this loop:
|
the meter checked in this loop:
|
||||||
|
|
@ -576,7 +613,25 @@ async def generate_via_llm(
|
||||||
if pending_revise is not None and reviews is not None:
|
if pending_revise is not None and reviews is not None:
|
||||||
reviews[pending_revise] = replace(reviews[pending_revise], honoured=True)
|
reviews[pending_revise] = replace(reviews[pending_revise], honoured=True)
|
||||||
pending_revise = None
|
pending_revise = None
|
||||||
result = validate_proposal(candidate, baseline=baseline)
|
# P7: ground the candidate in the EXACT text this attempt sent. ``messages`` is the
|
||||||
|
# proposer's entire input for this call -- on the S2c bundle path the debate navigates and
|
||||||
|
# generation sees only the debate output, so an identifier the candidate carries but the
|
||||||
|
# messages do not came from the model's weights, not from this run. Rendered from the ONE
|
||||||
|
# ``_build_messages`` result rather than recomposed here: a second rendering of the prompt
|
||||||
|
# would be free to disagree with the one that was actually sent.
|
||||||
|
result = validate_proposal(
|
||||||
|
candidate,
|
||||||
|
baseline=baseline,
|
||||||
|
grounding=_grounding_text(
|
||||||
|
project,
|
||||||
|
baseline,
|
||||||
|
# ``None`` -> the retrieval ``context`` this caller handed in, which for a
|
||||||
|
# caller that declared nothing else IS the input it declared. ``run_project``
|
||||||
|
# always passes it EXPLICITLY, because on the debate path ``context`` has been
|
||||||
|
# replaced by the model's OWN summary of what it read.
|
||||||
|
context if grounding is None else grounding,
|
||||||
|
),
|
||||||
|
)
|
||||||
last_ruling = result
|
last_ruling = result
|
||||||
if isinstance(result, Rejection):
|
if isinstance(result, Rejection):
|
||||||
last = result
|
last = result
|
||||||
|
|
|
||||||
|
|
@ -1053,8 +1053,16 @@ async def run_project(
|
||||||
"a pre-pass payload declares a cut of a knowledge base, so it needs the bundle it "
|
"a pre-pass payload declares a cut of a knowledge base, so it needs the bundle it "
|
||||||
"was cut from; this run was given no bundle_dir"
|
"was cut from; this run was given no bundle_dir"
|
||||||
)
|
)
|
||||||
|
# P7: the delivered base is the run's own evidence for what identifiers EXIST. Built from
|
||||||
|
# ``context_files`` (MAJOR-3/S7a-3's rule), so the ``type: verdict`` layer stays out — a
|
||||||
|
# proposal grounded in a prior verdict would reach the ExpeL fold's material around its gate.
|
||||||
|
bundle_grounding = ""
|
||||||
if bundle_dir is not None:
|
if bundle_dir is not None:
|
||||||
bundle = okf.navigate_bundle(bundle_dir)
|
bundle = okf.navigate_bundle(bundle_dir)
|
||||||
|
bundle_grounding = "\n".join(
|
||||||
|
"\n".join([f.name, *f.frontmatter.values(), f.body])
|
||||||
|
for f in bundle.context_files
|
||||||
|
)
|
||||||
# ONE bundle-id rule (Step 10, slackened S7a-3 pkt. 1): the DECLARED id is the identity and
|
# ONE bundle-id rule (Step 10, slackened S7a-3 pkt. 1): the DECLARED id is the identity and
|
||||||
# the mount is carried alongside, so a base delivered under a directory name of its own is
|
# the mount is carried alongside, so a base delivered under a directory name of its own is
|
||||||
# opened rather than refused. What is still refused, before a single model call: a base
|
# opened rather than refused. What is still refused, before a single model call: a base
|
||||||
|
|
@ -1366,6 +1374,11 @@ async def run_project(
|
||||||
# an attempt is helped by knowing it; it never enters the record, because the two
|
# an attempt is helped by knowing it; it never enters the record, because the two
|
||||||
# falsifiers are never blended.
|
# falsifiers are never blended.
|
||||||
checker_verdict=checker_decision,
|
checker_verdict=checker_decision,
|
||||||
|
# P7: what this run was GIVEN, as opposed to what the debate said about it.
|
||||||
|
# ``context`` is the DELIVERED rendering (pre-pass cut / bundle pointer / retrieved
|
||||||
|
# chunks) — never ``gen_context``, which on the debate path is the model's own
|
||||||
|
# summary and would let a code the debate invented ground the proposal repeating it.
|
||||||
|
grounding="\n".join([context, bundle_grounding]),
|
||||||
)
|
)
|
||||||
refinements.extend(generated.refinements)
|
refinements.extend(generated.refinements)
|
||||||
return generated.outcome
|
return generated.outcome
|
||||||
|
|
|
||||||
|
|
@ -208,10 +208,58 @@ def _reconcile_against_baseline(
|
||||||
return Rejection(proposal=proposal, reason="; ".join(violations))
|
return Rejection(proposal=proposal, reason="; ".join(violations))
|
||||||
|
|
||||||
|
|
||||||
|
def _ground_against_input(proposal: SavingsProposal, grounding: str) -> Rejection | None:
|
||||||
|
"""P7: every identifier the proposal builds on must appear VERBATIM in the input it was built
|
||||||
|
from, or the verdict falls.
|
||||||
|
|
||||||
|
**Why this is a SEPARATE stage and not a widening of stage 0.** ``_reconcile_against_baseline``
|
||||||
|
already carries the sentence "the cost code is absent from the baseline — a fabricated line",
|
||||||
|
and it is right. But it is reached only through ``if baseline is not None``, so the falsifier
|
||||||
|
is tied to whether a cost baseline happens to exist — and MEASURED (økt 108, verdict
|
||||||
|
``5fd6272e3725fe68``), an unanchored K2 run ended in ``ValidatedProposal`` on two cost codes
|
||||||
|
(``M-04-01`` / ``M-04-03``) that appear in NO prompt of that run. **The input always exists;
|
||||||
|
the baseline does not.** This stage therefore runs on its own, whatever ``baseline`` is.
|
||||||
|
|
||||||
|
**Exact substring, no pattern.** The check is ``code in grounding``: nothing here needs to know
|
||||||
|
what an identifier LOOKS like, so nothing here can be wrong about a form the corpus carries.
|
||||||
|
That is deliberate — measured over the delivered corpora (K2: 1 108 concept files / 2 005 561
|
||||||
|
chars; the three N payloads: 8 delivered excerpts each), the identifier forms are heterogeneous
|
||||||
|
(499 ``UPPER-num`` occurrences / 23 unique and 25 single-letter ``B-20-00-00``-style codes in
|
||||||
|
K2; requirement numbers such as ``Krav 3.3.1—13`` that live in ``req_number``/``title`` and
|
||||||
|
never in an excerpt body; 71 UUIDs in one payload), and a pattern chosen to cover them would
|
||||||
|
be a rule about shapes rather than about grounding.
|
||||||
|
|
||||||
|
It fails OPEN, never closed, on a coincidental match: measured, K2 carries 46 394 bare-number
|
||||||
|
occurrences over 2 117 distinct values, so a numeric-only code is almost always "grounded" by
|
||||||
|
accident. That is a known weakness of this stage, never a false rejection — the honest failure
|
||||||
|
direction for a gate that speaks about a model's invention.
|
||||||
|
|
||||||
|
Returns ONE ``Rejection`` (the validator's own type — never a new gate) naming EVERY ungrounded
|
||||||
|
identifier, ``"; "``-joined, in the PROPOSAL's own order, exactly as ``_reconcile_against_
|
||||||
|
baseline`` does and for the same measured reason (økt 94): Step 5 feeds this reason verbatim
|
||||||
|
into the next attempt's prompt, and a message naming only the first violation reads as an
|
||||||
|
instruction to fix that one and leave the rest.
|
||||||
|
|
||||||
|
Only ``affected_items`` codes are checked. An ``assumptions`` key naming no affected item is
|
||||||
|
deliberately out of scope: the Monte Carlo never samples such a band (``SavingsProposal.
|
||||||
|
_assumption_bands_enclose_unit_cost`` says so in the same words), so it cannot move the verdict,
|
||||||
|
and a check on it would be a branch no recording exercises."""
|
||||||
|
violations = [
|
||||||
|
f"ungrounded identifier {item.code!r}: it appears nowhere in the input this proposal "
|
||||||
|
f"was built from ({len(grounding)} chars)"
|
||||||
|
for item in proposal.affected_items
|
||||||
|
if item.code not in grounding
|
||||||
|
]
|
||||||
|
if not violations:
|
||||||
|
return None
|
||||||
|
return Rejection(proposal=proposal, reason="; ".join(violations))
|
||||||
|
|
||||||
|
|
||||||
def validate_proposal(
|
def validate_proposal(
|
||||||
proposal: SavingsProposal,
|
proposal: SavingsProposal,
|
||||||
*,
|
*,
|
||||||
baseline: CostBaseline | None = None,
|
baseline: CostBaseline | None = None,
|
||||||
|
grounding: str | None = None,
|
||||||
tolerance: float = BASELINE_TOLERANCE_DEFAULT,
|
tolerance: float = BASELINE_TOLERANCE_DEFAULT,
|
||||||
method_caps: Mapping[str, float] | None = None,
|
method_caps: Mapping[str, float] | None = None,
|
||||||
) -> ValidatedProposal | Rejection:
|
) -> ValidatedProposal | Rejection:
|
||||||
|
|
@ -223,7 +271,12 @@ def validate_proposal(
|
||||||
hallucination clears the gate. It is OPTIONAL — ``None`` is exactly the pre-S4.0 behaviour, so a
|
hallucination clears the gate. It is OPTIONAL — ``None`` is exactly the pre-S4.0 behaviour, so a
|
||||||
caller with no baseline (a bundle authored before the amendment) is unchanged — but both run
|
caller with no baseline (a bundle authored before the amendment) is unchanged — but both run
|
||||||
paths SET it. ``tolerance`` is the reconciliation's config knob; ``method_caps`` overrides the
|
paths SET it. ``tolerance`` is the reconciliation's config knob; ``method_caps`` overrides the
|
||||||
built-in method-cap registry (F8)."""
|
built-in method-cap registry (F8).
|
||||||
|
|
||||||
|
``grounding`` (P7) is the input text the proposal must be grounded in — the rendered prompt the
|
||||||
|
model actually received. It is OPTIONAL for the same reason ``baseline`` is (``None`` = the
|
||||||
|
pre-P7 gate, so every caller and every golden is unchanged), but it closes a DIFFERENT hole:
|
||||||
|
stage 0 only fires when a baseline exists, while the input exists always."""
|
||||||
# Stage 0 (S4.0): reconcile against the cost baseline BEFORE the solver. It is the cheapest
|
# Stage 0 (S4.0): reconcile against the cost baseline BEFORE the solver. It is the cheapest
|
||||||
# stage and the only one that can tell a fabricated line from a real one — spending a CBC solve
|
# stage and the only one that can tell a fabricated line from a real one — spending a CBC solve
|
||||||
# on numbers that do not belong to the project is work on a claim that cannot be validated.
|
# on numbers that do not belong to the project is work on a claim that cannot be validated.
|
||||||
|
|
@ -231,6 +284,16 @@ def validate_proposal(
|
||||||
blocked = _reconcile_against_baseline(proposal, baseline, tolerance)
|
blocked = _reconcile_against_baseline(proposal, baseline, tolerance)
|
||||||
if blocked is not None:
|
if blocked is not None:
|
||||||
return blocked
|
return blocked
|
||||||
|
# Stage 0b (P7): every identifier the proposal builds on must appear verbatim in the input. It
|
||||||
|
# sits OUTSIDE the baseline branch above -- that is the whole point: stage 0 is the stronger
|
||||||
|
# check but only an anchored run has it, while an unanchored run had NO falsifier for a
|
||||||
|
# fabricated code at all. Placed AFTER stage 0 so an anchored run's message is byte-identical
|
||||||
|
# to before: where both would fire, the baseline's sentence is the more actionable one (it
|
||||||
|
# names the project and how many codes it knows), and Step 5 feeds that sentence back.
|
||||||
|
if grounding is not None:
|
||||||
|
adrift = _ground_against_input(proposal, grounding)
|
||||||
|
if adrift is not None:
|
||||||
|
return adrift
|
||||||
# Stage 1 (Pydantic) already ran at construction. Stage 2: real CBC solve.
|
# Stage 1 (Pydantic) already ran at construction. Stage 2: real CBC solve.
|
||||||
nominal = _solve_max_feasible(proposal.affected_items, MAX_SAVING_FRACTION)
|
nominal = _solve_max_feasible(proposal.affected_items, MAX_SAVING_FRACTION)
|
||||||
# Stage 3: Monte Carlo percentiles of the feasible saving.
|
# Stage 3: Monte Carlo percentiles of the feasible saving.
|
||||||
|
|
|
||||||
7
tests/fixtures/p7-grounding/p4-n100-generation-prompt.txt
vendored
Normal file
7
tests/fixtures/p7-grounding/p4-n100-generation-prompt.txt
vendored
Normal file
|
|
@ -0,0 +1,7 @@
|
||||||
|
Propose ONE concrete cost-saving measure for this project.
|
||||||
|
Project: N100 - N100
|
||||||
|
Context (prior verdicts / cited cost docs):
|
||||||
|
A concrete cost-saving measure based on the N100 requirements is to avoid grade-separated (planskilt) crossings between pedestrian/cycle paths and roads when the road has an ÅDT of 4,000 or less. According to Krav 3.3.1—13, planskilt crossings are required only if ÅDT > 4,000, so using at-grade crossings under this threshold reduces construction costs while still complying with N100.
|
||||||
|
|
||||||
|
Respond with ONLY a JSON object for a SavingsProposal with keys: project_id, measure, affected_items (list of {code, quantity, unit_cost}), claimed_saving_nok, and optional assumptions.
|
||||||
|
Each entry in affected_items must restate a cost line as the project's price schedule already carries it: quantity and unit_cost are the unchanged baseline figures, not the reduced quantity or unit cost your measure would produce. The effect of the measure belongs in claimed_saving_nok.
|
||||||
11
tests/fixtures/p7-grounding/p6-k2-generation-prompt.txt
vendored
Normal file
11
tests/fixtures/p7-grounding/p6-k2-generation-prompt.txt
vendored
Normal file
|
|
@ -0,0 +1,11 @@
|
||||||
|
Propose ONE concrete cost-saving measure for this project.
|
||||||
|
Project: K2 - K2
|
||||||
|
Context (prior verdicts / cited cost docs):
|
||||||
|
A concrete cost-saving measure for K2 in the Stange skole extension project is to optimize the marking (merking) process for technical systems by reducing the extent and complexity of permanent engraved marking where possible, and instead use durable pre-printed adhesive labels or supplier-marked components. Since the general technical requirements specify extensive engraving and permanent marking for all components, including those above and below ceilings, and full coordination with documentation, this can be costly in labor and materials.
|
||||||
|
|
||||||
|
By standardizing label types, using supplier pre-marked equipment, minimizing marking in hard-to-access or low-maintenance areas, and avoiding duplicated markings, the project can reduce marking time and material costs without compromising operational functionality or compliance. Close coordination early in the project between RITB, system integrators, and TE can finalize an efficient marking plan that meets requirements with lower cost.
|
||||||
|
|
||||||
|
This measure addresses a specific and detailed area described in the technical requirements and has potential to save substantial costs related to labor time, materials, and rework during installation.
|
||||||
|
|
||||||
|
Respond with ONLY a JSON object for a SavingsProposal with keys: project_id, measure, affected_items (list of {code, quantity, unit_cost}), claimed_saving_nok, and optional assumptions.
|
||||||
|
Each entry in affected_items must restate a cost line as the project's price schedule already carries it: quantity and unit_cost are the unchanged baseline figures, not the reduced quantity or unit cost your measure would produce. The effect of the measure belongs in claimed_saving_nok.
|
||||||
9
tests/fixtures/p7-grounding/s7c-k2-generation-prompt.txt
vendored
Normal file
9
tests/fixtures/p7-grounding/s7c-k2-generation-prompt.txt
vendored
Normal file
|
|
@ -0,0 +1,9 @@
|
||||||
|
Propose ONE concrete cost-saving measure for this project.
|
||||||
|
Project: K2 - K2
|
||||||
|
Context (prior verdicts / cited cost docs):
|
||||||
|
A concrete cost-saving measure for K2 is to focus on streamlining the extensive prøvedrift (trial operation) phase described in the general technical requirements. This phase involves many integrated tests, monthly coordination meetings, continuous updating of logs, and the risk of automatic extensions if systems are not fully functional.
|
||||||
|
|
||||||
|
By enhancing upfront coordination, ensuring all systems meet acceptance criteria before starting prøvedrift, and reducing repeated tests through better planning and quality control, the project can minimize costly delays and reduce resource consumption for testing and defect resolution. This approach cuts down on contractor hours and administrative overhead, resulting in tangible cost savings without compromising the project's technical quality or safety.
|
||||||
|
|
||||||
|
Respond with ONLY a JSON object for a SavingsProposal with keys: project_id, measure, affected_items (list of {code, quantity, unit_cost}), claimed_saving_nok, and optional assumptions.
|
||||||
|
Each entry in affected_items must restate a cost line as the project's price schedule already carries it: quantity and unit_cost are the unchanged baseline figures, not the reduced quantity or unit cost your measure would produce. The effect of the measure belongs in claimed_saving_nok.
|
||||||
|
|
@ -73,9 +73,14 @@ def _role_factory(proposer_reply: str, checker_reply: str) -> Callable[[str], Ba
|
||||||
|
|
||||||
async def test_foreign_dimension_candidate_rejected_when_dimension_set() -> None:
|
async def test_foreign_dimension_candidate_rejected_when_dimension_set() -> None:
|
||||||
"""LOAD-BEARING: a foreign-dimension candidate that validates on the numbers is rejected by the
|
"""LOAD-BEARING: a foreign-dimension candidate that validates on the numbers is rejected by the
|
||||||
scope gate. RED if ``admits`` is detached (the foreign candidate slips through as validated)."""
|
scope gate. RED if ``admits`` is detached (the foreign candidate slips through as validated).
|
||||||
|
|
||||||
|
The cost code is one the delivered base NAMES (P7). It used to be ``SENTINEL-FOREIGN``, which
|
||||||
|
the base does not carry — so once identifier grounding existed the candidate fell for a reason
|
||||||
|
that has nothing to do with the dimension, and this arm would have ridden on that instead of on
|
||||||
|
``admits``. Only the MEASURE is foreign now, which is what §4.1b is about."""
|
||||||
factory = _role_factory(
|
factory = _role_factory(
|
||||||
_valid_reply("paving_renegotiation", "SENTINEL-FOREIGN"), "VERDICT: APPROVE"
|
_valid_reply("paving_renegotiation", "ENERGI-TOTAL-EL"), "VERDICT: APPROVE"
|
||||||
)
|
)
|
||||||
|
|
||||||
result = await run_project(
|
result = await run_project(
|
||||||
|
|
|
||||||
363
tests/test_identifier_grounding_loadbearing.py
Normal file
363
tests/test_identifier_grounding_loadbearing.py
Normal file
|
|
@ -0,0 +1,363 @@
|
||||||
|
"""P7 — an identifier the proposal builds on must appear VERBATIM in the input, or the verdict
|
||||||
|
falls.
|
||||||
|
|
||||||
|
The hole this closes was MEASURED (økt 108/109). ``_reconcile_against_baseline`` already carries
|
||||||
|
the sentence "the cost code is absent from the baseline — a fabricated line", but it is reached
|
||||||
|
only through ``if baseline is not None`` (``validator.py``), and an unanchored run has no baseline.
|
||||||
|
So the paid K2 run of økt 108 ended in ``ValidatedProposal`` (verdict ``5fd6272e3725fe68``) on two
|
||||||
|
cost codes — ``M-04-01`` / ``M-04-03`` — that appear in NO prompt of that run. The falsifier was
|
||||||
|
tied to whether a cost baseline happened to exist. **The input always exists; the baseline does
|
||||||
|
not.**
|
||||||
|
|
||||||
|
Every fixture under ``tests/fixtures/p7-grounding/`` is the VERBATIM generation prompt of a free
|
||||||
|
recording — the entire input the proposer saw on the attempt that produced its candidate. They are
|
||||||
|
tracked test data on purpose: the recordings live in ``scratchpad/``, which ``git archive HEAD``
|
||||||
|
does not carry, so a test reading them would pass here and fail the handover gate.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from portfolio_optimiser.ir import (
|
||||||
|
AffectedItem,
|
||||||
|
CostBaseline,
|
||||||
|
CostBaselineLine,
|
||||||
|
SavingsProposal,
|
||||||
|
)
|
||||||
|
from portfolio_optimiser.validator import Rejection, ValidatedProposal, validate_proposal
|
||||||
|
|
||||||
|
FIXTURES = Path(__file__).parent / "fixtures" / "p7-grounding"
|
||||||
|
|
||||||
|
#: The known positive (premiss (vii), `docs/2026-09-08-n-bundlene-hypoteseform.md` § 11.4): the
|
||||||
|
#: model quoted this requirement number VERBATIM in 6 of 6 replies, and it stands in 6 of 6
|
||||||
|
#: prompts. Note the EM-DASH (U+2014): the hyphen variant scores 0 of 6 in both.
|
||||||
|
KNOWN_POSITIVE = "Krav 3.3.1—13"
|
||||||
|
|
||||||
|
|
||||||
|
def _prompt(name: str) -> str:
|
||||||
|
return (FIXTURES / name).read_text(encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
def _proposal(*codes: str, unit_cost: float = 100.0) -> SavingsProposal:
|
||||||
|
"""A minimal proposal carrying exactly ``codes``, cheap enough that every LATER stage passes —
|
||||||
|
so a rejection can only have come from the grounding stage."""
|
||||||
|
return SavingsProposal(
|
||||||
|
project_id="K2",
|
||||||
|
measure="reduce marking scope",
|
||||||
|
affected_items=[
|
||||||
|
AffectedItem(code=c, quantity=10.0, unit_cost=unit_cost) for c in codes
|
||||||
|
],
|
||||||
|
claimed_saving_nok=1.0,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _project():
|
||||||
|
"""The reference project, used only as the prompt's header — the grounding text is the rendered
|
||||||
|
prompt, so the project's own cost items play no part here."""
|
||||||
|
from portfolio_optimiser.reference_domain import load_reference_projects
|
||||||
|
|
||||||
|
return load_reference_projects()[0]
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------------------------
|
||||||
|
# (a)/(b) the two recordings that reached a verdict on fabricated codes
|
||||||
|
# --------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("fixture", "codes"),
|
||||||
|
[
|
||||||
|
("p6-k2-generation-prompt.txt", ("M-04-01", "M-04-03")),
|
||||||
|
("s7c-k2-generation-prompt.txt", ("PRD-001", "PRD-002")),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_a_fabricated_code_from_a_real_recording_falls(fixture: str, codes: tuple[str, ...]) -> None:
|
||||||
|
"""(a)+(b). Both recordings' codes are absent from the prompt that produced them (denominator
|
||||||
|
2 of 2 in each), and the verdict must now fall — with ``baseline=None``, which is exactly the
|
||||||
|
configuration under which økt 108 said ``validated``."""
|
||||||
|
text = _prompt(fixture)
|
||||||
|
for code in codes:
|
||||||
|
assert code not in text, f"fixture drifted: {code!r} is IN the prompt"
|
||||||
|
ruling = validate_proposal(_proposal(*codes), baseline=None, grounding=text)
|
||||||
|
assert isinstance(ruling, Rejection), f"{codes} cleared the gate on an input that names neither"
|
||||||
|
for code in codes:
|
||||||
|
assert repr(code) in ruling.reason
|
||||||
|
|
||||||
|
|
||||||
|
def test_b_the_reason_names_every_ungrounded_identifier_in_the_proposal_s_own_order() -> None:
|
||||||
|
"""Completeness, for ``_reconcile_against_baseline``'s own reason (økt 94, measured live): a
|
||||||
|
message naming only the FIRST violation reads as an instruction to fix that one field, and the
|
||||||
|
proposer fixes one and rebreaks the other. Same ``"; "`` joiner, same PROPOSAL order."""
|
||||||
|
text = _prompt("p6-k2-generation-prompt.txt")
|
||||||
|
ruling = validate_proposal(_proposal("M-04-03", "M-04-01"), baseline=None, grounding=text)
|
||||||
|
assert isinstance(ruling, Rejection)
|
||||||
|
parts = ruling.reason.split("; ")
|
||||||
|
assert len(parts) == 2, f"expected one sentence per violation, got {ruling.reason!r}"
|
||||||
|
assert "'M-04-03'" in parts[0] and "'M-04-01'" in parts[1], ruling.reason
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------------------------
|
||||||
|
# (c) THE KNOWN POSITIVE — with its control on the SAME input
|
||||||
|
# --------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_c_the_known_positive_is_not_flagged() -> None:
|
||||||
|
"""``Krav 3.3.1—13`` stands VERBATIM in the N100 generation prompt, so a proposal that cites it
|
||||||
|
is grounded and the grounding stage must stay silent. A rule that flags everything code-shaped
|
||||||
|
is red here."""
|
||||||
|
text = _prompt("p4-n100-generation-prompt.txt")
|
||||||
|
assert KNOWN_POSITIVE in text, "fixture drifted: the known positive is not in the prompt"
|
||||||
|
ruling = validate_proposal(_proposal(KNOWN_POSITIVE), baseline=None, grounding=text)
|
||||||
|
assert isinstance(ruling, ValidatedProposal), getattr(ruling, "reason", "")
|
||||||
|
|
||||||
|
|
||||||
|
def test_c_control_the_rule_can_still_flag_on_that_same_recording() -> None:
|
||||||
|
"""THE CONTROL that keeps the arm above from proving nothing. ``CRS-01`` is the one identifier
|
||||||
|
the model produced on the N100 recording that no prompt carries (1 of 2 ungrounded). Same
|
||||||
|
fixture, same call — only the identifier differs, and this one must fall."""
|
||||||
|
text = _prompt("p4-n100-generation-prompt.txt")
|
||||||
|
assert "CRS-01" not in text
|
||||||
|
ruling = validate_proposal(_proposal("CRS-01"), baseline=None, grounding=text)
|
||||||
|
assert isinstance(ruling, Rejection)
|
||||||
|
assert "'CRS-01'" in ruling.reason
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------------------------
|
||||||
|
# (d) a code that IS in the input is never flagged, however fabricated it looks
|
||||||
|
# --------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_d_a_code_quoted_verbatim_from_the_input_is_not_flagged() -> None:
|
||||||
|
"""The discriminator between this rule and "flag anything that looks like a code". The token is
|
||||||
|
deliberately shaped like the fabrications above; the ONLY difference is that the input says it."""
|
||||||
|
text = "Context:\nPrice schedule line ZZZ-999-01 covers technical marking.\n"
|
||||||
|
ruling = validate_proposal(_proposal("ZZZ-999-01"), baseline=None, grounding=text)
|
||||||
|
assert isinstance(ruling, ValidatedProposal), getattr(ruling, "reason", "")
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------------------------
|
||||||
|
# (e) THE HOLE ITSELF: the stage does not depend on a baseline
|
||||||
|
# --------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_e_the_stage_fires_with_no_baseline_at_all() -> None:
|
||||||
|
"""Premiss (iv). The pre-P7 tree returns ``ValidatedProposal`` here — measured on the real P6
|
||||||
|
proposal — because stage 0 sits behind ``if baseline is not None``."""
|
||||||
|
text = _prompt("p6-k2-generation-prompt.txt")
|
||||||
|
assert isinstance(
|
||||||
|
validate_proposal(_proposal("M-04-01"), baseline=None, grounding=text), Rejection
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_e_the_stage_is_not_gated_on_the_absence_of_a_baseline() -> None:
|
||||||
|
"""Independence in the other direction, and a statement about the STAGE rather than about the
|
||||||
|
product: an anchored call whose baseline carries the code still reaches stage 0b, and stage 0b
|
||||||
|
judges only the text it was handed. What counts as "the input" is the CALLER's decision — and
|
||||||
|
the caller (``generate._grounding_text``) folds the baseline's own codes in, so no real run
|
||||||
|
rejects a line its baseline proves. The pair is deliberate: putting the baseline inside this
|
||||||
|
stage would make the stage untestable in isolation, and putting it nowhere would turn an
|
||||||
|
anchored run into a wall (see ``test_g_...`` below for the product-level half)."""
|
||||||
|
text = _prompt("p6-k2-generation-prompt.txt")
|
||||||
|
baseline = CostBaseline(
|
||||||
|
project_id="K2", items={"M-04-01": CostBaselineLine(quantity=10.0, unit_cost=100.0)}
|
||||||
|
)
|
||||||
|
assert isinstance(
|
||||||
|
validate_proposal(_proposal("M-04-01"), baseline=baseline, grounding=text), Rejection
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_e_control_no_grounding_argument_is_the_pre_P7_behaviour() -> None:
|
||||||
|
"""``grounding=None`` must be byte-for-byte the old gate — that is what keeps every existing
|
||||||
|
caller, every golden and the demo transcript unchanged."""
|
||||||
|
ruling = validate_proposal(_proposal("M-04-01", "M-04-03"), baseline=None)
|
||||||
|
assert isinstance(ruling, ValidatedProposal)
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------------------------
|
||||||
|
# (f) THE WIRING — the seam where a model-authored proposal meets its own input
|
||||||
|
# --------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
async def test_f_generate_via_llm_grounds_the_candidate_in_the_prompt_it_sent() -> None:
|
||||||
|
"""LOAD-BEARING WIRING, behavioural. ``generate_via_llm`` is the ONE place a model-authored
|
||||||
|
proposal and the exact text the model was given exist at the same moment (premiss (viii)).
|
||||||
|
The rendered ``messages`` ARE the proposer's entire input for that call, so an identifier the
|
||||||
|
proposal carries but the messages do not came from the model's weights, not from this run.
|
||||||
|
|
||||||
|
Driven through the canonical ``ScriptedChatClient`` (S2.5) so the seam is exercised, not
|
||||||
|
inspected. RED when ``grounding=`` is detached at the call site: the outcome flips to
|
||||||
|
``ValidatedProposal`` — which is precisely the økt-108 defect."""
|
||||||
|
from portfolio_optimiser.budget import Budget, TokenMeter
|
||||||
|
from portfolio_optimiser.generate import generate_via_llm
|
||||||
|
from portfolio_optimiser.simulation import ScriptedChatClient
|
||||||
|
|
||||||
|
project = _project()
|
||||||
|
reply = _proposal("GHOST-77-01").model_dump_json()
|
||||||
|
meter = TokenMeter(Budget(max_tokens=10**9, max_rounds=20))
|
||||||
|
result = await generate_via_llm(
|
||||||
|
ScriptedChatClient(reply=reply, tokens_per_reply=8),
|
||||||
|
project,
|
||||||
|
"The price schedule carries line REAL-11-02 for technical marking.",
|
||||||
|
meter,
|
||||||
|
max_attempts=1,
|
||||||
|
)
|
||||||
|
assert isinstance(result.outcome, Rejection), "an identifier no prompt carried was validated"
|
||||||
|
assert "'GHOST-77-01'" in result.outcome.reason
|
||||||
|
|
||||||
|
|
||||||
|
async def test_f_control_a_code_the_context_names_survives_the_same_call() -> None:
|
||||||
|
"""THE CONTROL. Same client, same call, same meter — only the identifier changes, and this one
|
||||||
|
the context states verbatim. Without it the arm above would also pass on a rule that rejects
|
||||||
|
every proposal ``generate_via_llm`` ever produces."""
|
||||||
|
from portfolio_optimiser.budget import Budget, TokenMeter
|
||||||
|
from portfolio_optimiser.generate import generate_via_llm
|
||||||
|
from portfolio_optimiser.simulation import ScriptedChatClient
|
||||||
|
|
||||||
|
project = _project()
|
||||||
|
reply = _proposal("REAL-11-02").model_dump_json()
|
||||||
|
meter = TokenMeter(Budget(max_tokens=10**9, max_rounds=20))
|
||||||
|
result = await generate_via_llm(
|
||||||
|
ScriptedChatClient(reply=reply, tokens_per_reply=8),
|
||||||
|
project,
|
||||||
|
"The price schedule carries line REAL-11-02 for technical marking.",
|
||||||
|
meter,
|
||||||
|
max_attempts=1,
|
||||||
|
)
|
||||||
|
assert isinstance(result.outcome, ValidatedProposal), getattr(result.outcome, "reason", "")
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------------------------
|
||||||
|
# (g) THE SEAM'S OTHER TWO SOURCES — what only the caller can prove the run was given
|
||||||
|
# --------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_g_the_seam_grounds_a_code_the_baseline_proves_even_when_no_prompt_repeats_it() -> None:
|
||||||
|
"""The product-level half of the arm above. Stage 0 has ALREADY ruled this code a real line of
|
||||||
|
the project; the weaker stage must not overrule the stronger falsifier just because the debate's
|
||||||
|
prose summary did not repeat it. RED on a composer that drops the baseline: every anchored S2c
|
||||||
|
run would then reject its own real codes — a wall, not a gate."""
|
||||||
|
from portfolio_optimiser.generate import _grounding_text
|
||||||
|
|
||||||
|
project = _project()
|
||||||
|
baseline = CostBaseline(
|
||||||
|
project_id="K2", items={"M-04-01": CostBaselineLine(quantity=10.0, unit_cost=100.0)}
|
||||||
|
)
|
||||||
|
text = _grounding_text(project, baseline, "the debate summarised this in prose, naming no code")
|
||||||
|
assert isinstance(
|
||||||
|
validate_proposal(_proposal("M-04-01"), baseline=baseline, grounding=text),
|
||||||
|
ValidatedProposal,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_g_the_seam_grounds_a_code_the_delivered_base_carries() -> None:
|
||||||
|
"""``delivered`` is the source only ``run_project`` can supply: on the S2c bundle path the
|
||||||
|
debate NAVIGATES, so the base's own text never reaches the generation prompt. RED when the
|
||||||
|
threading is detached — the run would then reject every real code of every unanchored base."""
|
||||||
|
from portfolio_optimiser.generate import _grounding_text
|
||||||
|
|
||||||
|
project = _project()
|
||||||
|
text = _grounding_text(project, None, "the price schedule line BASE-77-01 is real")
|
||||||
|
assert isinstance(
|
||||||
|
validate_proposal(_proposal("BASE-77-01"), baseline=None, grounding=text),
|
||||||
|
ValidatedProposal,
|
||||||
|
)
|
||||||
|
assert isinstance(
|
||||||
|
validate_proposal(_proposal("BASE-77-02"), baseline=None, grounding=text), Rejection
|
||||||
|
), "CONTROL: the delivered base must not ground a code it does not carry"
|
||||||
|
|
||||||
|
|
||||||
|
async def test_g_run_project_grounds_a_candidate_in_the_base_it_delivered(tmp_path) -> None:
|
||||||
|
"""BEHAVIOURAL, over the real ``run_project`` bundle arm — the ONE place the delivered base's
|
||||||
|
text is composed. Two arms over the same crafted copy of the fixture base:
|
||||||
|
|
||||||
|
* a code whose ONLY occurrence is inside a ``type: verdict`` document must NOT ground the
|
||||||
|
proposal. That is MAJOR-3 N2 / S7a-3's ``context_files``-not-``files`` rule applied to
|
||||||
|
grounding: the verdict layer reaches a hypothesis through the GATED ExpeL fold alone, and
|
||||||
|
grounding a proposal in a prior verdict routes that material around its own gate. RED when
|
||||||
|
the grounding text is built from ``bundle.files``;
|
||||||
|
* the CONTROL, an ordinary concept-file code on the same base, must survive — otherwise the arm
|
||||||
|
above would also pass on a run that rejects everything, and RED when the threading from
|
||||||
|
``run_project`` is detached at all.
|
||||||
|
"""
|
||||||
|
import shutil
|
||||||
|
|
||||||
|
from portfolio_optimiser.run import run_project
|
||||||
|
from portfolio_optimiser.simulation import ScriptedChatClient
|
||||||
|
from portfolio_optimiser.validator import ValidatedProposal as VP
|
||||||
|
|
||||||
|
base = tmp_path / "base"
|
||||||
|
shutil.copytree(
|
||||||
|
Path(__file__).resolve().parents[1] / "shared" / "examples" / "bygg-energi-mikro", base
|
||||||
|
)
|
||||||
|
(base / "dom-tidligere.md").write_text(
|
||||||
|
"---\ntype: verdict\ntitle: Prior judgement\ndecision: approved\n---\n\n"
|
||||||
|
"The measure on cost line VERDICT-ONLY-9 was approved.\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
index = base / "index.md"
|
||||||
|
index.write_text(
|
||||||
|
index.read_text(encoding="utf-8") + "\n- [Prior judgement](dom-tidligere.md)\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
|
||||||
|
async def _outcome(code: str):
|
||||||
|
reply = (
|
||||||
|
f'{{"measure":"LED-retrofit","affected_items":'
|
||||||
|
f'[{{"code":"{code}","quantity":300000,"unit_cost":1.0}}],'
|
||||||
|
f'"claimed_saving_nok":30000}}'
|
||||||
|
)
|
||||||
|
result = await run_project(
|
||||||
|
"BYGG-KONTOR-NORD",
|
||||||
|
"local",
|
||||||
|
docs_dir=str(base),
|
||||||
|
bundle_dir=str(base),
|
||||||
|
client_factory=lambda role: ScriptedChatClient(
|
||||||
|
"Reasoning holds.\nVERDICT: APPROVE" if role == "checker" else reply, role=role
|
||||||
|
),
|
||||||
|
)
|
||||||
|
return result.outcome
|
||||||
|
|
||||||
|
control = await _outcome("ENERGI-TOTAL-EL")
|
||||||
|
assert isinstance(control, VP), (
|
||||||
|
f"CONTROL: a code the delivered base carries was not grounded — {control}"
|
||||||
|
)
|
||||||
|
leaked = await _outcome("VERDICT-ONLY-9")
|
||||||
|
assert isinstance(leaked, Rejection), "a verdict-layer code grounded a proposal"
|
||||||
|
assert "'VERDICT-ONLY-9'" in leaked.reason
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------------------------
|
||||||
|
# (h) THE GATE MUST NOT DISARM ITSELF ON ITS SECOND ROUND
|
||||||
|
# --------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
async def test_h_the_refusal_does_not_ground_the_next_attempt_that_repeats_the_code() -> None:
|
||||||
|
"""MEASURED while building this rule, and the reason the rendered PROMPT is not evidence.
|
||||||
|
|
||||||
|
Step 5 feeds the previous ``Rejection.reason`` VERBATIM into the next attempt's prompt — and
|
||||||
|
this stage's reason QUOTES the identifier it just refused. An implementation that grounded a
|
||||||
|
candidate in its own prompt would therefore accept on attempt 2 exactly what it refused on
|
||||||
|
attempt 1: a falsifier that disarms itself. Both attempts here repeat the same ungrounded code,
|
||||||
|
and both must fall.
|
||||||
|
"""
|
||||||
|
from portfolio_optimiser.budget import Budget, TokenMeter
|
||||||
|
from portfolio_optimiser.generate import generate_via_llm
|
||||||
|
from portfolio_optimiser.simulation import ScriptedChatClient
|
||||||
|
|
||||||
|
reply = _proposal("GHOST-77-01").model_dump_json()
|
||||||
|
client = ScriptedChatClient(reply=reply, tokens_per_reply=8)
|
||||||
|
result = await generate_via_llm(
|
||||||
|
client,
|
||||||
|
_project(),
|
||||||
|
"The price schedule carries line REAL-11-02 for technical marking.",
|
||||||
|
TokenMeter(Budget(max_tokens=10**9, max_rounds=20)),
|
||||||
|
max_attempts=2,
|
||||||
|
)
|
||||||
|
assert client.call_count == 2, "the informed-refinement loop did not reach a second attempt"
|
||||||
|
assert isinstance(result.outcome, Rejection), (
|
||||||
|
"attempt 2 was accepted after attempt 1 refused the same identifier — the refusal grounded it"
|
||||||
|
)
|
||||||
|
assert "'GHOST-77-01'" in result.outcome.reason
|
||||||
|
|
@ -250,16 +250,23 @@ async def test_bundle_path_anchors_when_the_bundle_declares_a_baseline(fresh_sto
|
||||||
|
|
||||||
|
|
||||||
async def test_pre_amendment_bundle_runs_unchanged(fresh_store) -> None:
|
async def test_pre_amendment_bundle_runs_unchanged(fresh_store) -> None:
|
||||||
"""Control + backward compatibility: the SAME fabricated reply validates against a bundle with
|
"""Control + backward compatibility: an unanchored bundle still runs to a VALIDATED outcome, so
|
||||||
no ``cost-baseline.json``. Anchoring is opt-in per bundle, so every pre-S4.0 bundle (including
|
anchoring stays opt-in per bundle and every pre-S4.0 bundle (including the commons-owned
|
||||||
the commons-owned goldens) runs byte-identically to before."""
|
goldens) is unaffected.
|
||||||
|
|
||||||
|
**NARROWED BY P7, deliberately.** This arm used to send the SAME FABRICATED code and assert
|
||||||
|
that it validated — which was, stated plainly, the økt-108 hole written down as an expectation:
|
||||||
|
with no baseline there was no falsifier for an invented cost line at all. The reply now carries
|
||||||
|
a code the delivered base actually names, so the arm proves what it claims (an unanchored run
|
||||||
|
still reaches a verdict) without also promising that fabrication clears it. The fabricated half
|
||||||
|
lives in ``test_identifier_grounding_loadbearing``."""
|
||||||
result = await run_project(
|
result = await run_project(
|
||||||
"BYGG-ENERGI-MIKRO-A",
|
"BYGG-ENERGI-MIKRO-A",
|
||||||
"local",
|
"local",
|
||||||
docs_dir=str(PRE_AMENDMENT_BUNDLE),
|
docs_dir=str(PRE_AMENDMENT_BUNDLE),
|
||||||
bundle_dir=str(PRE_AMENDMENT_BUNDLE),
|
bundle_dir=str(PRE_AMENDMENT_BUNDLE),
|
||||||
verdict_input=_VERDICT_INPUT,
|
verdict_input=_VERDICT_INPUT,
|
||||||
client_factory=_factory(_reply(_FAKE_CODE, 1.0, 300000.0, 90000.0)),
|
client_factory=_factory(_reply("ENERGI-TOTAL-EL", 1.0, 300000.0, 90000.0)),
|
||||||
store=fresh_store,
|
store=fresh_store,
|
||||||
)
|
)
|
||||||
assert isinstance(result.outcome, ValidatedProposal)
|
assert isinstance(result.outcome, ValidatedProposal)
|
||||||
|
|
|
||||||
|
|
@ -96,6 +96,10 @@ _CLAIM = 200.0
|
||||||
#: NOT ``energy_efficiency`` — that measure would additionally hit the method cap (0.15 * 1000 = 150)
|
#: NOT ``energy_efficiency`` — that measure would additionally hit the method cap (0.15 * 1000 = 150)
|
||||||
#: and reject a claim of 200 for a reason that has nothing to do with this seam.
|
#: and reject a claim of 200 for a reason that has nothing to do with this seam.
|
||||||
_MEASURE = "behovsstyrt_drift"
|
_MEASURE = "behovsstyrt_drift"
|
||||||
|
#: P7: the synthetic context must NAME the line the synthetic proposal restates, exactly as a
|
||||||
|
#: real run's input does. Without it the grounding stage rejects a sentinel code no input ever
|
||||||
|
#: mentioned — correctly, and for a reason that has nothing to do with this seam.
|
||||||
|
_CONTEXT = f"Price schedule line {_CODE} covers demand-driven operation."
|
||||||
|
|
||||||
|
|
||||||
def _wire_reply(*, with_band: bool) -> str:
|
def _wire_reply(*, with_band: bool) -> str:
|
||||||
|
|
@ -178,7 +182,7 @@ async def test_generation_call_carries_the_strict_schema() -> None:
|
||||||
project = load_reference_projects()[0]
|
project = load_reference_projects()[0]
|
||||||
client = _OptionsRecordingChatClient(_wire_reply(with_band=True))
|
client = _OptionsRecordingChatClient(_wire_reply(with_band=True))
|
||||||
|
|
||||||
await generate_via_llm(client, project, "", _meter(), max_attempts=1)
|
await generate_via_llm(client, project, _CONTEXT, _meter(), max_attempts=1)
|
||||||
|
|
||||||
# Control FIRST: a positive assert over an empty list would pass vacuously.
|
# Control FIRST: a positive assert over an empty list would pass vacuously.
|
||||||
assert client.seen_options, "no generation call was observed — the assert below proves nothing"
|
assert client.seen_options, "no generation call was observed — the assert below proves nothing"
|
||||||
|
|
@ -249,14 +253,14 @@ async def test_assumption_bands_keep_the_monte_carlo_falsifier_alive() -> None:
|
||||||
banded = await generate_via_llm(
|
banded = await generate_via_llm(
|
||||||
_OptionsRecordingChatClient(_wire_reply(with_band=True)),
|
_OptionsRecordingChatClient(_wire_reply(with_band=True)),
|
||||||
project,
|
project,
|
||||||
"",
|
_CONTEXT,
|
||||||
_meter(),
|
_meter(),
|
||||||
max_attempts=1,
|
max_attempts=1,
|
||||||
)
|
)
|
||||||
bandless = await generate_via_llm(
|
bandless = await generate_via_llm(
|
||||||
_OptionsRecordingChatClient(_wire_reply(with_band=False)),
|
_OptionsRecordingChatClient(_wire_reply(with_band=False)),
|
||||||
project,
|
project,
|
||||||
"",
|
_CONTEXT,
|
||||||
_meter(),
|
_meter(),
|
||||||
max_attempts=1,
|
max_attempts=1,
|
||||||
)
|
)
|
||||||
|
|
@ -311,7 +315,7 @@ async def test_band_round_trip_is_verbatim_and_the_ir_map_form_still_parses() ->
|
||||||
from_array = await generate_via_llm(
|
from_array = await generate_via_llm(
|
||||||
_OptionsRecordingChatClient(_wire_reply(with_band=True)),
|
_OptionsRecordingChatClient(_wire_reply(with_band=True)),
|
||||||
project,
|
project,
|
||||||
"",
|
_CONTEXT,
|
||||||
_meter(),
|
_meter(),
|
||||||
max_attempts=1,
|
max_attempts=1,
|
||||||
)
|
)
|
||||||
|
|
@ -330,7 +334,7 @@ async def test_band_round_trip_is_verbatim_and_the_ir_map_form_still_parses() ->
|
||||||
}
|
}
|
||||||
)
|
)
|
||||||
from_map = await generate_via_llm(
|
from_map = await generate_via_llm(
|
||||||
_OptionsRecordingChatClient(map_form), project, "", _meter(), max_attempts=1
|
_OptionsRecordingChatClient(map_form), project, _CONTEXT, _meter(), max_attempts=1
|
||||||
)
|
)
|
||||||
assert isinstance(from_map.outcome, ValidatedProposal), (
|
assert isinstance(from_map.outcome, ValidatedProposal), (
|
||||||
"the IR's own map form stopped parsing — the normalisation replaced rather than extended"
|
"the IR's own map form stopped parsing — the normalisation replaced rather than extended"
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue