open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 4 Activity

feat(p17b): a context set that spans TWO bases, and a judge told which one [skip-docs]

Browse source 
``contexts/dekke-og-kontrakt-lindaas-2027`` is the first set whose approaches
route at more than one knowledge base: a1/a2 at n200-2024 (material requirements)
and a3/a4 at r761-2025 (the rig, and the falsification arm). That is the whole
reason it exists -- P17b measures that ONE commission can be run across several.

``bundle.txt`` grows a block per base; a set naming one base is one block, so the
four pre-P17b files parse byte-identically. The reader now has ONE home
(``stress.read_bundle_declarations``): it used to be a private copy in the P14
gate and a second, looser one inside ``stress.main``, and the multi-base form is
exactly the change that would have let them drift.

Rule U becomes the UNION of every declared base, and that is not a formality.
MEASURED 15.09: ``enhetspris`` is absent from n200-2024 and carried by 70 of
r761-2025's 2 756 concepts, so anchors admitted per base would have admitted a
question the pass as a whole CAN ground. It was dropped from the fifth set's
anchors for that reason.

``score_context_set(bundle_id=...)`` restricts the judgement to the approaches
routed at THIS base. Without it, judging the n200 outbox reports the r761
approach as ``not_evaluated``/``absent`` -- a false finding, because that
approach WAS evaluated, against the other base, under the other run_id. That
defect is pinned by its own arm. The judge's CLI refuses to guess when a set
declares several bases, with an rc-0 control on ``--bundle``.

Arm (d) gained a second half: every DECLARED base must be named by some
approach, because a base no approach names is never run.

The P19/B2 fasit denominator moved 26 -> 32 and is asserted, not dropped: six new
references, two of them bare ``prosessnr`` (12.11, 12.12), so B1's
punctuation-and-digits form is now exercised by a fasit and not only by a
known-positive.

Suite 1774/5, golden byte-unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-15 04:44:37 +02:00
commit da0ccd0489
8 changed files with 496 additions and 57 deletions
Download patch file Download diff file Expand all files Collapse all files

115
src/portfolio_optimiser/stress.py
View file

@ -183,6 +183,41 @@ class ContextSetVerdict:
return asdict(self)
def read_bundle_declarations(path: str | Path) -> tuple[dict[str, str], ...]:
"""Parse a context set's ``bundle.txt`` into ONE declaration per knowledge base.
``key: value`` lines, nothing else; each ``name:`` OPENS a block and each block must close
with its own ``bundle_id:``. A set naming ONE base is one block, so every file written before
P17b parses byte-identically — the multi-base form (P17b DEL 2) is an extension, not a new
format.
**The ONE reader.** Before P17b the rule had two private copies — one in ``stress.main``'s
argument parsing, one in the P14 gate's own test file — and the multi-base form is exactly the
kind of change that would have let them drift into two answers about one set (kø-(p)). Both
now call this.
:raises ValueError: a malformed line, or a block that declares no ``bundle_id``.
"""
blocks: list[dict[str, str]] = []
for line in Path(path).read_text(encoding="utf-8").splitlines():
line = line.strip()
if not line or line.startswith("#"):
continue
if ": " not in line:
raise ValueError(f"malformed bundle.txt line in {path}: {line!r}")
key, value = (part.strip() for part in line.split(": ", 1))
if key == "name" or not blocks:
blocks.append({})
blocks[-1][key] = value
if not blocks:
raise ValueError(f"{path} declares no knowledge base at all")
for block in blocks:
for required in ("name", "bundle_id"):
if required not in block:
raise ValueError(f"{path} declares no {required!r}")
return tuple(blocks)
def _read_json(path: Path) -> dict[str, Any]:
return json.loads(path.read_text(encoding="utf-8")) # type: ignore[no-any-return]
@ -233,9 +268,20 @@ def score_context_set(
outbox_dir: str | Path,
run_id: str,
bundle_dir: str | Path,
bundle_id: str | None = None,
) -> ContextSetVerdict:
"""Judge ONE context set against ONE outbox. ``bundle_dir`` is the MOUNTED base itself (the
CLI resolves it from ``--bundle-root`` plus the set's own ``bundle.txt`` name)."""
CLI resolves it from ``--bundle-root`` plus the set's own ``bundle.txt`` name).
``bundle_id`` RESTRICTS the judgement to the approaches a multi-base set routed at THIS base
(P17b DEL 2). Without it, judging a two-base set's n200 outbox would report the r761 approach
as ``not_evaluated`` with reason ``absent`` — a false finding, because that approach WAS
evaluated, against the other base, under the other ``run_id``. ``None`` keeps every single-base
set judged exactly as before, which is why this is a restriction rather than a new mode: the
order offered a ``--multibase`` summary reader, and MEASURED against the shape the artefacts
actually take, the per-base run already has its own full artefact set and its own run_id — so
what the judge was missing was not a new file to read but the one thing the mandate already
knows, namely which approaches belong here."""
context = Path(context_dir)
outbox = Path(outbox_dir)
base = Path(bundle_dir)
@ -256,6 +302,19 @@ def score_context_set(
must_cite = {row["approach_id"]: row.get("concepts", []) for row in fasit.get("must_cite", [])}
refuse_ids = {row["approach_id"] for row in fasit.get("must_refuse", [])}
# P17b: the approaches THIS base was asked about. Read off the mandate, never off a second
# per-row key in the fasit — the routing already has exactly one home (kø-(p)).
judged_approaches = mandate.approaches
if bundle_id is not None:
judged_approaches = tuple(
a for a in judged_approaches if (a.bundle_id or bundle_id) == bundle_id
)
if not judged_approaches:
raise EmptyMeasurement(
f"no approach in {context} is routed at {bundle_id!r} - a judgement over zero "
"rows has no denominator"
)
# ---- run-level trace ---------------------------------------------------------------------
debate = outbox / f"{run_id}-debate.json"
tool_calls: list[dict[str, Any]] = (
@ -303,7 +362,7 @@ def score_context_set(
validated_codes: set[str] = set()
validated_ids: set[str] = set()
for approach in mandate.approaches:
for approach in judged_approaches:
proposal_path, outcome_path = _artefacts(outbox, run_id, approach.id)
concepts = must_cite.get(approach.id, [])
wanted = {c["path"] for c in concepts}
@ -422,9 +481,14 @@ def score_context_set(
# ---- the falsification arm ---------------------------------------------------------------
refusals: list[RefusalVerdict] = []
judged_ids = {a.id for a in judged_approaches}
for row in fasit.get("must_refuse", []):
rid = row["approach_id"]
commissioned = next((a for a in mandate.approaches if a.id == rid), None)
# P17b: a falsification arm routed at ANOTHER base was neither asked nor answered here,
# and reporting it would put a pass/fail on a run that did not happen in this outbox.
if rid not in judged_ids:
continue
commissioned = next((a for a in judged_approaches if a.id == rid), None)
refuse_codes = set(commissioned.affected_codes) if commissioned is not None else set()
leaked = sorted(refuse_codes & validated_codes)
if rid in validated_ids:
@ -446,7 +510,7 @@ def score_context_set(
return ContextSetVerdict(
context_set=context.name,
run_id=run_id,
bundle_id=str(fasit.get("bundle_id", "")),
bundle_id=bundle_id if bundle_id is not None else str(fasit.get("bundle_id", "")),
approaches=tuple(rows),
must_refuse=tuple(refusals),
hallucinated_reads=tuple(hallucinated_reads),
@ -480,18 +544,47 @@ def main(argv: list[str] | None = None) -> int:
default=os.environ.get("PORTFOLIO_VEGNORMAL_ROOT", _DEFAULT_BUNDLE_ROOT),
help="directory the set's bundle.txt name is mounted under",
)
parser.add_argument(
"--bundle",
default=None,
help="which of the set's declared bases this outbox is for (name or bundle_id). Required "
"when the set declares more than one (P17b): each approach is judged against ITS OWN "
"base, so a judge that guessed would score one base's run against another's fasit rows",
)
args = parser.parse_args(argv)
context = Path(args.context_dir)
declared = dict(
line.split(":", 1) # type: ignore[misc]
for line in (context / "bundle.txt").read_text(encoding="utf-8").splitlines()
if ":" in line
)
base = Path(args.bundle_root).expanduser() / declared["name"].strip()
declared = read_bundle_declarations(context / "bundle.txt")
if args.bundle is None and len(declared) > 1:
print(
f"stress refused: {context} declares {len(declared)} knowledge bases "
f"({', '.join(d['name'] for d in declared)}); name the one this outbox is for with "
"--bundle, because a judge that picked would be scoring one base's run against "
"another base's fasit rows",
file=sys.stderr,
)
return 1
chosen = declared[0] if args.bundle is None else None
for block in declared:
if args.bundle in (block["name"], block["bundle_id"]):
chosen = block
if chosen is None:
print(
f"stress refused: {args.bundle!r} is not one of the knowledge bases {context} "
f"declares ({', '.join(d['name'] for d in declared)})",
file=sys.stderr,
)
return 1
base = Path(args.bundle_root).expanduser() / chosen["name"]
try:
verdict = score_context_set(context, args.outbox_dir, args.run_id, base)
verdict = score_context_set(
context,
args.outbox_dir,
args.run_id,
base,
bundle_id=chosen["bundle_id"] if len(declared) > 1 else None,
)
except EmptyMeasurement as exc:
print(f"stress refused: {exc}", file=sys.stderr)
return 1