open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 3 Activity

feat(validator,generate,run): an identifier a proposal builds on must be in the input, or the verdict falls [skip-docs]

Browse source 
P6 (økt 108) ended in ValidatedProposal (verdict 5fd6272e3725fe68) on two cost
codes -- M-04-01 / M-04-03 -- that appear in NO prompt of that run. Measured
here first, verbatim: validate_proposal(p, baseline=None) validates it; the same
proposal against any non-empty CostBaseline is rejected naming both codes.

So the hole was never "fabrication goes uncaught" -- _reconcile_against_baseline
exists and is right -- but that the falsifier is reached only through
`if baseline is not None`. The input always exists; the baseline does not.

New stage 0b (_ground_against_input), OUTSIDE the baseline branch, after stage 0
so an anchored run's message is byte-identical to before. ONE Rejection, the
validator's own type, naming EVERY ungrounded identifier "; "-joined in the
proposal's own order (økt 94's completeness reason).

The rule has NO pattern -- `code in grounding`, exact substring -- and that is a
measurement: over the delivered corpora (K2 1108 files / 2 005 561 chars, the
three N payloads 8 excerpts each) the identifier forms are heterogeneous, and a
pattern chosen to cover them would be a rule about shapes. Bare numerals are the
one inert class (46 394 occurrences / 2 117 distinct in K2); the rule fails OPEN
there, never closed.

Evidence is three non-model-authored sources: what run_project DELIVERED (the
rendered cut/pointer/chunks plus the base's context_files -- never files, which
would make the type: verdict layer evidence), the project's own cost lines, and
the baseline's codes when anchored. The rendered PROMPT is deliberately NOT
evidence, on two measurements: gen_context IS the debate output on the S2c path,
and from attempt 2 the prompt carries the previous Rejection.reason verbatim --
which for this stage QUOTES the identifier it just refused. Grounding in the
prompt would let the gate's own refusal disarm it on its second round.

Prose scanning was chosen against WITH THE NUMBERS: a typed gate catches 2/2
(P6) and 2/2 (S7c) -- 100% of what reached a verdict. What stays uncaught, said
plainly: an ungrounded identifier that lives only in agent/debate prose and never
becomes an affected_item code (2 of 4 P6, 2 of 4 S7c, 1 of 2 P4).

Iron Law: 9 red / 2 green before the rule existed. Ten mutations all red against
the whole suite, green control 1558 passed / 5 skipped (from 1543/5, superset,
0 removed), golden demo-transcript.stdout BYTE-UNCHANGED (shasum -a 1 of the
CONTENT = ea8c534773acdbe41ae68f2c55724d69aaf8be4f).

Three existing fixtures changed, no gate weakened -- most of all
test_pre_amendment_bundle_runs_unchanged, which sent the SAME FABRICATED code and
asserted it validated: the økt-108 hole written down as an expectation.

No paid run. Order 20260909T113641Z-38938691-from-.claude.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-09 16:04:01 +02:00
commit 277bb95777
12 changed files with 831 additions and 13 deletions
Download patch file Download diff file Expand all files Collapse all files

57
src/portfolio_optimiser/generate.py
View file

@ -420,6 +420,42 @@ def generate_with_validation(
return self_repair(_attempt, max_attempts=max_attempts)
def _grounding_text(project: Project, baseline: CostBaseline | None, delivered: str) -> str:
"""P7: compose the ONE text a candidate's identifiers must be grounded in — the run's
non-model-authored input, and nothing else.
Three sources, each of which the run can point at without asking the model:
* ``delivered`` — what the CALLER can prove this run was GIVEN. ``run_project`` fills it from
the delivered rendered context (the pre-pass cut, the bundle pointer, or the road path's
retrieved chunks) PLUS the navigated base's ``context_files`` — never ``files``, because that
is the property which drops the ``type: verdict`` layer at every level, and grounding a
proposal in a prior verdict would route the ExpeL fold's own material around its gate;
* the project's OWN cost lines. The road path's estimate IS the project, so a code it carries
is real whether or not any text restated it. MEASURED: ``_project_from_bundle`` builds
``cost_items=()``, so this source contributes nothing on the bundle path and the gate stays
exactly as sharp where fabrication was measured (K2, økt 108);
* the baseline's codes when a run is anchored. Stage 0 has already ruled every code that
reaches stage 0b a REAL line of this project; the weaker stage must not overrule the stronger
falsifier because a prose summary happened not to repeat the code.
**The rendered PROMPT is deliberately NOT a source, and that is a measurement, not taste.** Two
of the prompt's parts are the model's own words fed back to it: on the S2c bundle path
``gen_context`` is the DEBATE OUTPUT (measured — a scripted proposer that names a code in a
debate turn then grounds its own proposal in that turn), and from attempt 2 onward the prompt
carries the previous ``Rejection.reason`` VERBATIM (Step 5) — which, for this stage, quotes the
very identifier it just refused. Grounding in the prompt would therefore let the gate's own
refusal ground the next attempt: a falsifier that disarms itself on its second round.
"""
return "\n".join(
[
delivered,
*(item.code for item in project.cost_items),
*(() if baseline is None else baseline.items),
]
)
async def generate_via_llm(
chat_client: BaseChatClient,
project: Project,
@ -434,6 +470,7 @@ async def generate_via_llm(
reviews: list[ProposalReview] | None = None,
review_key: tuple[str | None, str | None] = (None, None),
checker_verdict: str = "absent",
grounding: str | None = None,
) -> GenerationResult:
"""Async LLM path: non-streaming chat -> parse -> validate, with TWO bounded retry kinds,
the meter checked in this loop:
@ -576,7 +613,25 @@ async def generate_via_llm(
if pending_revise is not None and reviews is not None:
reviews[pending_revise] = replace(reviews[pending_revise], honoured=True)
pending_revise = None
result = validate_proposal(candidate, baseline=baseline)
# P7: ground the candidate in the EXACT text this attempt sent. ``messages`` is the
# proposer's entire input for this call -- on the S2c bundle path the debate navigates and
# generation sees only the debate output, so an identifier the candidate carries but the
# messages do not came from the model's weights, not from this run. Rendered from the ONE
# ``_build_messages`` result rather than recomposed here: a second rendering of the prompt
# would be free to disagree with the one that was actually sent.
result = validate_proposal(
candidate,
baseline=baseline,
grounding=_grounding_text(
project,
baseline,
# ``None`` -> the retrieval ``context`` this caller handed in, which for a
# caller that declared nothing else IS the input it declared. ``run_project``
# always passes it EXPLICITLY, because on the debate path ``context`` has been
# replaced by the model's OWN summary of what it read.
context if grounding is None else grounding,
),
)
last_ruling = result
if isinstance(result, Rejection):
last = result

13
src/portfolio_optimiser/run.py
View file

@ -1053,8 +1053,16 @@ async def run_project(
"a pre-pass payload declares a cut of a knowledge base, so it needs the bundle it "
"was cut from; this run was given no bundle_dir"
)
# P7: the delivered base is the run's own evidence for what identifiers EXIST. Built from
# ``context_files`` (MAJOR-3/S7a-3's rule), so the ``type: verdict`` layer stays out — a
# proposal grounded in a prior verdict would reach the ExpeL fold's material around its gate.
bundle_grounding = ""
if bundle_dir is not None:
bundle = okf.navigate_bundle(bundle_dir)
bundle_grounding = "\n".join(
"\n".join([f.name, *f.frontmatter.values(), f.body])
for f in bundle.context_files
)
# ONE bundle-id rule (Step 10, slackened S7a-3 pkt. 1): the DECLARED id is the identity and
# the mount is carried alongside, so a base delivered under a directory name of its own is
# opened rather than refused. What is still refused, before a single model call: a base
@ -1366,6 +1374,11 @@ async def run_project(
# an attempt is helped by knowing it; it never enters the record, because the two
# falsifiers are never blended.
checker_verdict=checker_decision,
# P7: what this run was GIVEN, as opposed to what the debate said about it.
# ``context`` is the DELIVERED rendering (pre-pass cut / bundle pointer / retrieved
# chunks) — never ``gen_context``, which on the debate path is the model's own
# summary and would let a code the debate invented ground the proposal repeating it.
grounding="\n".join([context, bundle_grounding]),
)
refinements.extend(generated.refinements)
return generated.outcome

65
src/portfolio_optimiser/validator.py
View file

@ -208,10 +208,58 @@ def _reconcile_against_baseline(
return Rejection(proposal=proposal, reason="; ".join(violations))
def _ground_against_input(proposal: SavingsProposal, grounding: str) -> Rejection | None:
"""P7: every identifier the proposal builds on must appear VERBATIM in the input it was built
from, or the verdict falls.
**Why this is a SEPARATE stage and not a widening of stage 0.** ``_reconcile_against_baseline``
already carries the sentence "the cost code is absent from the baseline — a fabricated line",
and it is right. But it is reached only through ``if baseline is not None``, so the falsifier
is tied to whether a cost baseline happens to exist — and MEASURED (økt 108, verdict
``5fd6272e3725fe68``), an unanchored K2 run ended in ``ValidatedProposal`` on two cost codes
(``M-04-01`` / ``M-04-03``) that appear in NO prompt of that run. **The input always exists;
the baseline does not.** This stage therefore runs on its own, whatever ``baseline`` is.
**Exact substring, no pattern.** The check is ``code in grounding``: nothing here needs to know
what an identifier LOOKS like, so nothing here can be wrong about a form the corpus carries.
That is deliberate — measured over the delivered corpora (K2: 1 108 concept files / 2 005 561
chars; the three N payloads: 8 delivered excerpts each), the identifier forms are heterogeneous
(499 ``UPPER-num`` occurrences / 23 unique and 25 single-letter ``B-20-00-00``-style codes in
K2; requirement numbers such as ``Krav 3.3.1—13`` that live in ``req_number``/``title`` and
never in an excerpt body; 71 UUIDs in one payload), and a pattern chosen to cover them would
be a rule about shapes rather than about grounding.
It fails OPEN, never closed, on a coincidental match: measured, K2 carries 46 394 bare-number
occurrences over 2 117 distinct values, so a numeric-only code is almost always "grounded" by
accident. That is a known weakness of this stage, never a false rejection — the honest failure
direction for a gate that speaks about a model's invention.
Returns ONE ``Rejection`` (the validator's own type — never a new gate) naming EVERY ungrounded
identifier, ``"; "``-joined, in the PROPOSAL's own order, exactly as ``_reconcile_against_
baseline`` does and for the same measured reason (økt 94): Step 5 feeds this reason verbatim
into the next attempt's prompt, and a message naming only the first violation reads as an
instruction to fix that one and leave the rest.
Only ``affected_items`` codes are checked. An ``assumptions`` key naming no affected item is
deliberately out of scope: the Monte Carlo never samples such a band (``SavingsProposal.
_assumption_bands_enclose_unit_cost`` says so in the same words), so it cannot move the verdict,
and a check on it would be a branch no recording exercises."""
violations = [
f"ungrounded identifier {item.code!r}: it appears nowhere in the input this proposal "
f"was built from ({len(grounding)} chars)"
for item in proposal.affected_items
if item.code not in grounding
]
if not violations:
return None
return Rejection(proposal=proposal, reason="; ".join(violations))
def validate_proposal(
proposal: SavingsProposal,
*,
baseline: CostBaseline | None = None,
grounding: str | None = None,
tolerance: float = BASELINE_TOLERANCE_DEFAULT,
method_caps: Mapping[str, float] | None = None,
) -> ValidatedProposal | Rejection:
@ -223,7 +271,12 @@ def validate_proposal(
hallucination clears the gate. It is OPTIONAL — ``None`` is exactly the pre-S4.0 behaviour, so a
caller with no baseline (a bundle authored before the amendment) is unchanged — but both run
paths SET it. ``tolerance`` is the reconciliation's config knob; ``method_caps`` overrides the
built-in method-cap registry (F8)."""
built-in method-cap registry (F8).
``grounding`` (P7) is the input text the proposal must be grounded in — the rendered prompt the
model actually received. It is OPTIONAL for the same reason ``baseline`` is (``None`` = the
pre-P7 gate, so every caller and every golden is unchanged), but it closes a DIFFERENT hole:
stage 0 only fires when a baseline exists, while the input exists always."""
# Stage 0 (S4.0): reconcile against the cost baseline BEFORE the solver. It is the cheapest
# stage and the only one that can tell a fabricated line from a real one — spending a CBC solve
# on numbers that do not belong to the project is work on a claim that cannot be validated.
@ -231,6 +284,16 @@ def validate_proposal(
blocked = _reconcile_against_baseline(proposal, baseline, tolerance)
if blocked is not None:
return blocked
# Stage 0b (P7): every identifier the proposal builds on must appear verbatim in the input. It
# sits OUTSIDE the baseline branch above -- that is the whole point: stage 0 is the stronger
# check but only an anchored run has it, while an unanchored run had NO falsifier for a
# fabricated code at all. Placed AFTER stage 0 so an anchored run's message is byte-identical
# to before: where both would fire, the baseline's sentence is the more actionable one (it
# names the project and how many codes it knows), and Step 5 feeds that sentence back.
if grounding is not None:
adrift = _ground_against_input(proposal, grounding)
if adrift is not None:
return adrift
# Stage 1 (Pydantic) already ran at construction. Stage 2: real CBC solve.
nominal = _solve_max_feasible(proposal.affected_items, MAX_SAVING_FRACTION)
# Stage 3: Monte Carlo percentiles of the feasible saving.