open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 4 Activity

feat(p18): an identifier that stands everywhere identifies nothing

Browse source 
P18 parts B and C (order 20260914T105139Z).

B1 -- stage 0b. P7 made it `item.code in grounding`: plain containment over
ONE concatenated string. P16 ran it against a delivered corpus and measured
what containment cannot tell apart: the falsification arm a4-indeksregulering
put 250 000 NOK on a single cost line coded R761 -- the knowledge base's OWN
NAME, carried by all 2 756 of its concept documents -- and the whole gate
said validated (stage 0 skipped, un-anchored run; checker approve).

The grounding is now carried as the DOCUMENTS it is made of (validator.
Grounding), not as a blob. A structure and not a second argument beside the
text: the boundaries and the text are one fact, and .text is derived, so the
gate and P8's report measure the same characters. run.py composes one
document per concept file where the base is already walked; generate.
_grounding_text folds each cost line in as a one-line document.

N and A are MEASURED, not chosen (14.09, four mounted vegnormal bases):
- every must_cite ref and mandate affected_code in the four context sets --
  shortest real identifier is FOUR characters (12.1, 52.1), so N = 3 sits one
  below the measurement and cannot refuse anything measured;
- document frequency of every code-shaped token per base -- 1 692 distinct
  and NOT ONE reaches 5 %. Highest anywhere 6/446 (1.35 %), highest a fasit
  names 3/446 (0.67 %), R761 2 756/2 756 (100 %). A = 0.05 therefore sits
  3.7x above the highest real token and 20x below the defect.
Length is NOT what makes the defect inert (R761 is four characters); the
share is. And a share is not a measurement without a denominator big enough
to take one (ansikt 4): one of three is 33 %, so an ABSOLUTE floor of 10
documents gates it. Highest absolute count any real identifier reaches is 6,
and every fixture in the repo is far below 10 -- which is why every pre-P18
gate is UNTOUCHED by this rule rather than exempted from it. Grounding.of
(one document) can never reach the floor by construction.

The refusal NAMES the denominator ("appears in 2756 of the 2756 documents
this run was given"), because Step 5 feeds that reason verbatim into the next
attempt's prompt: a proposer told only "ungrounded" answers with another
token of the same kind.

B2 SPIKE (measured, NOT built) FELLED the order's own alternative: option (b)
"ground in what the run OPENED" was run over P16's 16 code rows -- R761
stands in every OPENED document too, so (b) would NOT have caught the defect,
while B1 makes it inert and still grounds the real process line 65
ASFALTDEKKER (29/2756 = 1.05 %). (b) is not a substitute for B1.

C1 -- --docs-dir is optional once --bundle-dir is given (P16 FUNN 2). On the
bundle path docs_dir is never read: retrieval, the chunk tool and the "no
citable content" check all live in the road branch. Bound ONCE from
--bundle-dir, which is byte-identically what the README already tells an
operator to type by hand. NOT the "--docs-dir omvei": no such path is opened
and the road branch still refuses without a real --docs-dir (own arm).

C2 -- the judge's snippet arm counts only under citation_scope == "narrowed",
as (a) already does (PM decision, P16 s 6.2). P16's reason for (b') being
clean -- snippets are bodies while ref/title live in frontmatter, 0 of 446
n100 bodies -- holds for "Krav 4.1.2-1" but NOT for R761, where a process
number like 12.1 stands in the bodies. Under a whole-base citation list that
mark was "cited" before any model call.

tests: test_inert_identifier_loadbearing.py (7 arms; known positive is P16's
OWN artefact replayed against the base that run was given, known negative is
26 of 26 fasit references still grounding), test_docs_dir_optional_
loadbearing.py (5 arms). test_stress_judge_loadbearing.py's snippet arm split
into narrowed/whole-base -- the pair is the discriminator, same snippet, same
mark, only the scope differs. The grounding tests migrate from str to
Grounding.of (the honest reading of a caller that declared no boundaries).

Verification: uv run pytest -q 1698 passed / 5 skipped (1685 after part A,
strict superset, 0 removed). ruff check + format clean, mypy clean (38
files). Golden demo-transcript.stdout BYTE-UNCHANGED, shasum -a 1 of the
CONTENT = ea8c534773acdbe41ae68f2c55724d69aaf8be4f. No version bump, no push.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-14 22:28:40 +02:00
commit 7a7c988253
11 changed files with 628 additions and 51 deletions
Download patch file Download diff file Expand all files Collapse all files

35
tests/test_identifier_grounding_loadbearing.py
View file

@ -27,7 +27,12 @@ from portfolio_optimiser.ir import (
CostBaselineLine,
SavingsProposal,
)
from portfolio_optimiser.validator import Rejection, ValidatedProposal, validate_proposal
from portfolio_optimiser.validator import (
Grounding,
Rejection,
ValidatedProposal,
validate_proposal,
)
FIXTURES = Path(__file__).parent / "fixtures" / "p7-grounding"
@ -81,7 +86,7 @@ def test_a_fabricated_code_from_a_real_recording_falls(
text = _prompt(fixture)
for code in codes:
assert code not in text, f"fixture drifted: {code!r} is IN the prompt"
ruling = validate_proposal(_proposal(*codes), baseline=None, grounding=text)
ruling = validate_proposal(_proposal(*codes), baseline=None, grounding=Grounding.of(text))
assert isinstance(ruling, Rejection), f"{codes} cleared the gate on an input that names neither"
for code in codes:
assert repr(code) in ruling.reason
@ -92,7 +97,9 @@ def test_b_the_reason_names_every_ungrounded_identifier_in_the_proposal_s_own_or
message naming only the FIRST violation reads as an instruction to fix that one field, and the
proposer fixes one and rebreaks the other. Same ``"; "`` joiner, same PROPOSAL order."""
text = _prompt("p6-k2-generation-prompt.txt")
ruling = validate_proposal(_proposal("M-04-03", "M-04-01"), baseline=None, grounding=text)
ruling = validate_proposal(
_proposal("M-04-03", "M-04-01"), baseline=None, grounding=Grounding.of(text)
)
assert isinstance(ruling, Rejection)
parts = ruling.reason.split("; ")
assert len(parts) == 2, f"expected one sentence per violation, got {ruling.reason!r}"
@ -110,7 +117,9 @@ def test_c_the_known_positive_is_not_flagged() -> None:
is red here."""
text = _prompt("p4-n100-generation-prompt.txt")
assert KNOWN_POSITIVE in text, "fixture drifted: the known positive is not in the prompt"
ruling = validate_proposal(_proposal(KNOWN_POSITIVE), baseline=None, grounding=text)
ruling = validate_proposal(
_proposal(KNOWN_POSITIVE), baseline=None, grounding=Grounding.of(text)
)
assert isinstance(ruling, ValidatedProposal), getattr(ruling, "reason", "")
@ -120,7 +129,7 @@ def test_c_control_the_rule_can_still_flag_on_that_same_recording() -> None:
fixture, same call — only the identifier differs, and this one must fall."""
text = _prompt("p4-n100-generation-prompt.txt")
assert "CRS-01" not in text
ruling = validate_proposal(_proposal("CRS-01"), baseline=None, grounding=text)
ruling = validate_proposal(_proposal("CRS-01"), baseline=None, grounding=Grounding.of(text))
assert isinstance(ruling, Rejection)
assert "'CRS-01'" in ruling.reason
@ -134,7 +143,7 @@ def test_d_a_code_quoted_verbatim_from_the_input_is_not_flagged() -> None:
"""The discriminator between this rule and "flag anything that looks like a code". The token is
deliberately shaped like the fabrications above; the ONLY difference is that the input says it."""
text = "Context:\nPrice schedule line ZZZ-999-01 covers technical marking.\n"
ruling = validate_proposal(_proposal("ZZZ-999-01"), baseline=None, grounding=text)
ruling = validate_proposal(_proposal("ZZZ-999-01"), baseline=None, grounding=Grounding.of(text))
assert isinstance(ruling, ValidatedProposal), getattr(ruling, "reason", "")
@ -148,7 +157,8 @@ def test_e_the_stage_fires_with_no_baseline_at_all() -> None:
proposal — because stage 0 sits behind ``if baseline is not None``."""
text = _prompt("p6-k2-generation-prompt.txt")
assert isinstance(
validate_proposal(_proposal("M-04-01"), baseline=None, grounding=text), Rejection
validate_proposal(_proposal("M-04-01"), baseline=None, grounding=Grounding.of(text)),
Rejection,
)
@ -165,7 +175,8 @@ def test_e_the_stage_is_not_gated_on_the_absence_of_a_baseline() -> None:
project_id="K2", items={"M-04-01": CostBaselineLine(quantity=10.0, unit_cost=100.0)}
)
assert isinstance(
validate_proposal(_proposal("M-04-01"), baseline=baseline, grounding=text), Rejection
validate_proposal(_proposal("M-04-01"), baseline=baseline, grounding=Grounding.of(text)),
Rejection,
)
@ -245,7 +256,9 @@ def test_g_the_seam_grounds_a_code_the_baseline_proves_even_when_no_prompt_repea
baseline = CostBaseline(
project_id="K2", items={"M-04-01": CostBaselineLine(quantity=10.0, unit_cost=100.0)}
)
text = _grounding_text(project, baseline, "the debate summarised this in prose, naming no code")
text = _grounding_text(
project, baseline, Grounding.of("the debate summarised this in prose, naming no code")
)
assert isinstance(
validate_proposal(_proposal("M-04-01"), baseline=baseline, grounding=text),
ValidatedProposal,
@ -259,7 +272,9 @@ def test_g_the_seam_grounds_a_code_the_delivered_base_carries() -> None:
from portfolio_optimiser.generate import _grounding_text
project = _project()
text = _grounding_text(project, None, "the price schedule line BASE-77-01 is real")
text = _grounding_text(
project, None, Grounding.of("the price schedule line BASE-77-01 is real")
)
assert isinstance(
validate_proposal(_proposal("BASE-77-01"), baseline=None, grounding=text),
ValidatedProposal,