feat(row6): a proposal whose approach declared no requirement is unsupported
Stress round 6 validated three falsification arms, and every validated approach rested only on run-level declarations nobody can attribute to one approach. declare_requirement now takes a required approach_id (a mandate id or own-proposal; an unknown id is refused naming the valid ones), and a ValidatedProposal whose approach has neither a mandate requirement nor a declaration under its own id becomes validator.Unsupported - a Rejection subclass carrying the validator's own ruling, reported as `unsupported` in coverage, the outcome artefact, the settlement and the judge, and never counted or summed. The rule is active whenever the debate held the declaration tool, the micro base included; the road and pre-pass paths are untouched. Declaration quality is not judged, so the rule can be satisfied by declaring any document the run read. The v1 gate's row 6 probes pass; its artefact half reads IKKE MÅLT because stress round 6 predates approach-addressed declarations, and IKKE MÅLT is never green - it fails the exit code. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
9847e014e7
commit
938a1ca30e
23 changed files with 718 additions and 115 deletions
19
CLAUDE.md
19
CLAUDE.md
|
|
@ -3112,6 +3112,25 @@ Python ≥3.10. MAF (`agent-framework-core` 1.16.0, `-orchestrations` 1.1.1 —
|
||||||
`r761-2025` uten `index.md`) — `--bundle-root` peker da på en utpakket kopi; bevisene for type 1, 3
|
`r761-2025` uten `index.md`) — `--bundle-root` peker da på en utpakket kopi; bevisene for type 1, 3
|
||||||
og 7 er EKSISTERENDE tester registrert ved node-id, så en omdøping gjør typen rød til registeret
|
og 7 er EKSISTERENDE tester registrert ved node-id, så en omdøping gjør typen rød til registeret
|
||||||
rettes (gatet av en egen arm).
|
rettes (gatet av en egen arm).
|
||||||
|
- **Et forslag uten tilnærmingens EGEN erklæring kan ikke bære `validated` (rad 6, 17.09):** målt
|
||||||
|
på stressrunde 6 hadde alle 10 validerte tilnærmingene bare kjørings-erklæringer, som ingen kan
|
||||||
|
knytte til én tilnærming — og tre falsifiseringsarmer validerte. `declare_requirement` tar derfor
|
||||||
|
et PÅKREVD `approach_id` (mandatets id-er + `own-proposal`; ukjent id → `UnknownApproach`, en
|
||||||
|
returnert nekt som navngir de gyldige), og `DeclaredRequirement`/`requirement_payload` bærer det.
|
||||||
|
I `_evaluate_mandate` blir en `ValidatedProposal` hvis tilnærming verken har `requirement` i
|
||||||
|
mandatet eller en erklæring under sin egen id til `validator.Unsupported` — en `Rejection`-
|
||||||
|
SUBKLASSE med validatorens egen `ValidatedProposal` på seg, så hver `isinstance(...,
|
||||||
|
ValidatedProposal)` sier nei uten å røres, mens de som NAVNGIR statusen (coverage `unsupported`,
|
||||||
|
`outcome_payload` `outcome_type: "unsupported"` med persentilene, `settle` `UNSUPPORTED`,
|
||||||
|
`rejection_stage` `unsupported`, dommeren) sjekker klassen FØRST. `validator_decision` forblir
|
||||||
|
`validated` (den speiler kun validatoren). **Regelen er aktiv nøyaktig når debatten hadde
|
||||||
|
erklæringsverktøyet** — også på mikro-basen, som har 0 kravnumre (PM-rettelse: ingen
|
||||||
|
spesialbehandling); veg-stien og pre-pass-stien har ingen trapp og er urørt. Erklæringens
|
||||||
|
KVALITET dømmes ikke (P22 § 4), så regelen kan spilles ved å erklære et hvilket som helst lest
|
||||||
|
dokument — uttalt svakhet. Dommeren leser en erklæring under tilnærmingens id som `approach`, en
|
||||||
|
uten `approach_id` (eldre artefakter) som `run`; v1-gatens rad 6 sier da «IKKE MÅLT», og IKKE
|
||||||
|
MÅLT feller exit-koden (aldri grønn). Load-bearing MÅLT
|
||||||
|
(`tests/test_row6_declaration_rule_loadbearing.py` + rad 6-probene), ti mutasjoner alle røde.
|
||||||
- **STATE.md er local-only** (gitignored). Voyage session-state er efemert; STATE.md er kanonisk kontinuitet.
|
- **STATE.md er local-only** (gitignored). Voyage session-state er efemert; STATE.md er kanonisk kontinuitet.
|
||||||
- Prosess: Voyage-plugin (`/trekbrief → /trekplan → /trekexecute → /trekreview`) per større fase.
|
- Prosess: Voyage-plugin (`/trekbrief → /trekplan → /trekexecute → /trekreview`) per større fase.
|
||||||
|
|
||||||
|
|
|
||||||
14
README.md
14
README.md
|
|
@ -495,9 +495,17 @@ when the seam is detached, so the loop cannot silently degrade into theater.
|
||||||
**Naming the requirement that binds a direction.** Whoever navigates a knowledge base — the
|
**Naming the requirement that binds a direction.** Whoever navigates a knowledge base — the
|
||||||
exploration's hypothesiser and, since the debate started navigating, the proposer — is asked to
|
exploration's hypothesiser and, since the debate started navigating, the proposer — is asked to
|
||||||
name the ONE requirement that binds the direction it commits to, and to declare it with
|
name the ONE requirement that binds the direction it commits to, and to declare it with
|
||||||
`declare_requirement(bundle_id, path, ref)` after reading it. A declaration naming a document
|
`declare_requirement(bundle_id, path, ref, approach_id)` after reading it. A declaration naming
|
||||||
the run never opened is **refused by name** (`RequirementNotRead`) and comes back as a turn the
|
a document the run never opened is **refused by name** (`RequirementNotRead`) and comes back as a
|
||||||
model can correct by going and reading it; nothing is recorded until it has. A marked hypothesis
|
turn the model can correct by going and reading it; nothing is recorded until it has.
|
||||||
|
`approach_id` names the approach the requirement binds — one of the mandate's ids, or
|
||||||
|
`own-proposal` — and an id the run was not commissioned with is refused with the valid ones named
|
||||||
|
(`UnknownApproach`). **A proposal whose approach declared nothing cannot be `validated`:** when
|
||||||
|
the debate held the declaration tool, a candidate whose numbers pass every stage but whose
|
||||||
|
approach has neither a requirement written into the mandate nor a declaration filed under its
|
||||||
|
own id is reported `unsupported` — not rejected (the numbers held), not validated (nothing was
|
||||||
|
named as its ground), never counted or summed. The declaration's *quality* is not judged; a run
|
||||||
|
can satisfy the rule by declaring any document it read, and that is a known weakness. A marked hypothesis
|
||||||
therefore carries `"requirement": {"path": ..., "ref": ...}`, or an explicit `null` together with
|
therefore carries `"requirement": {"path": ..., "ref": ...}`, or an explicit `null` together with
|
||||||
`"why_none"` — a base that holds no requirement for a direction is a finding worth stating, and
|
`"why_none"` — a base that holds no requirement for a direction is a finding worth stating, and
|
||||||
the field is never simply omitted. Where an approach carries one, the proposer's prompt names it
|
the field is never simply omitted. Where an approach carries one, the proposer's prompt names it
|
||||||
|
|
|
||||||
|
|
@ -106,11 +106,14 @@ Mandate outcome
|
||||||
Validated: 1 of 3 approaches — best 30000 NOK (the outcome this run carries)
|
Validated: 1 of 3 approaches — best 30000 NOK (the outcome this run carries)
|
||||||
```
|
```
|
||||||
|
|
||||||
De tre statusene betyr forskjellige ting, og forskjellen er poenget:
|
De fire statusene betyr forskjellige ting, og forskjellen er poenget:
|
||||||
|
|
||||||
- **VALIDATED** — tallene holdt. Beløpet er det validatoren slapp gjennom.
|
- **VALIDATED** — tallene holdt. Beløpet er det validatoren slapp gjennom.
|
||||||
- **REJECTED** — tilnærmingen ble vurdert, og falt på tallene. Grunnen står der; det er den du kan
|
- **REJECTED** — tilnærmingen ble vurdert, og falt på tallene. Grunnen står der; det er den du kan
|
||||||
gjøre noe med (ofte er den «regnet mot feil grunnlag», ikke «dårlig idé»).
|
gjøre noe med (ofte er den «regnet mot feil grunnlag», ikke «dårlig idé»).
|
||||||
|
- **UNSUPPORTED** — tallene holdt, men ingen krav i kunnskapsbasen ble erklært som bindende for
|
||||||
|
akkurat denne tilnærmingen. Den telles ikke som validert og summeres ikke. Skriv kravet inn i
|
||||||
|
bestillingen (`requirement` på tilnærmingen), eller la kjøringen erklære det.
|
||||||
- **NOT EVALUATED** — kjøringen rakk den aldri. Den er **ikke** avvist, og den er ikke glemt.
|
- **NOT EVALUATED** — kjøringen rakk den aldri. Den er **ikke** avvist, og den er ikke glemt.
|
||||||
|
|
||||||
En tilnærming du bestilte forsvinner aldri i stillhet. Det er hele grunnen til at oppgjøret finnes.
|
En tilnærming du bestilte forsvinner aldri i stillhet. Det er hele grunnen til at oppgjøret finnes.
|
||||||
|
|
|
||||||
|
|
@ -42,6 +42,8 @@ _AI_LINE_MIN = 30
|
||||||
GREEN = "GRØNN"
|
GREEN = "GRØNN"
|
||||||
RED = "RØD"
|
RED = "RØD"
|
||||||
DIAGNOSIS = "DIAGNOSE"
|
DIAGNOSIS = "DIAGNOSE"
|
||||||
|
#: A row whose evidence could not be read. Never green: on a failing row it fails the exit code.
|
||||||
|
NOT_MEASURED = "IKKE MÅLT"
|
||||||
|
|
||||||
ROUNDS_CONTRACT = """\
|
ROUNDS_CONTRACT = """\
|
||||||
Rundekatalogen (--rounds-dir) har fast form:
|
Rundekatalogen (--rounds-dir) har fast form:
|
||||||
|
|
@ -498,6 +500,8 @@ class StressMeasure:
|
||||||
commissioned: int = 0
|
commissioned: int = 0
|
||||||
where: str = ""
|
where: str = ""
|
||||||
missing: str = ""
|
missing: str = ""
|
||||||
|
#: Declarations with no ``approach_id`` — written before the rule; the row cannot be measured.
|
||||||
|
unaddressed: int = 0
|
||||||
undeclared_ids: tuple[str, ...] = field(default=())
|
undeclared_ids: tuple[str, ...] = field(default=())
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -538,6 +542,7 @@ def measure_stress(
|
||||||
approaches = [a for v in verdicts for a in v.approaches]
|
approaches = [a for v in verdicts for a in v.approaches]
|
||||||
validated = [a for a in approaches if a.status == "validated"]
|
validated = [a for a in approaches if a.status == "validated"]
|
||||||
undeclared = [a for a in validated if a.requirement_source != "approach"]
|
undeclared = [a for a in validated if a.requirement_source != "approach"]
|
||||||
|
unaddressed = sum(v.unaddressed_declarations for v in verdicts)
|
||||||
commissioned = sum(
|
commissioned = sum(
|
||||||
len(load_mandate(repo_root / c / "mandate.json").approaches) for c in contexts
|
len(load_mandate(repo_root / c / "mandate.json").approaches) for c in contexts
|
||||||
)
|
)
|
||||||
|
|
@ -549,6 +554,7 @@ def measure_stress(
|
||||||
rows=len(approaches),
|
rows=len(approaches),
|
||||||
commissioned=commissioned,
|
commissioned=commissioned,
|
||||||
where=str(stress_root),
|
where=str(stress_root),
|
||||||
|
unaddressed=unaddressed,
|
||||||
undeclared_ids=tuple(a.approach_id for a in undeclared),
|
undeclared_ids=tuple(a.approach_id for a in undeclared),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
@ -556,29 +562,54 @@ def measure_stress(
|
||||||
def score_undeclared(
|
def score_undeclared(
|
||||||
probes: Sequence[str], outcomes: Mapping[str, str], m: StressMeasure, label: str
|
probes: Sequence[str], outcomes: Mapping[str, str], m: StressMeasure, label: str
|
||||||
) -> Row:
|
) -> Row:
|
||||||
|
"""GREEN only when every probe passes AND the artefacts were measured with k = 0. Evidence that
|
||||||
|
could not be read is IKKE MÅLT — never green, and it fails the exit code like red does."""
|
||||||
failing = [
|
failing = [
|
||||||
f"{n.split('::')[-1]}={outcomes.get(n, 'missing')}"
|
f"{n.split('::')[-1]}={outcomes.get(n, 'missing')}"
|
||||||
for n in probes
|
for n in probes
|
||||||
if outcomes.get(n) != "passed"
|
if outcomes.get(n) != "passed"
|
||||||
]
|
]
|
||||||
|
if not probes:
|
||||||
|
failing.append("ingen probe registrert")
|
||||||
title = "6 validert UTEN erklært krav (tilnærmingens egen)"
|
title = "6 validert UTEN erklært krav (tilnærmingens egen)"
|
||||||
exceptions = [f"probe {x}" for x in failing]
|
exceptions = [f"probe {x}" for x in failing]
|
||||||
|
probe_state = "prober røde" if failing else "prober grønne"
|
||||||
|
k: int | None = None
|
||||||
|
n: int | None = None
|
||||||
|
diagnostics: tuple[str, ...] = ()
|
||||||
if m.missing:
|
if m.missing:
|
||||||
reason = f"{label}: ikke målt, artefakter mangler ({m.missing})"
|
reason = f"{probe_state}; {label}: ikke målt, artefakter mangler ({m.missing})"
|
||||||
k: int | None = None
|
elif m.unaddressed:
|
||||||
n: int | None = None
|
reason = (
|
||||||
|
f"{probe_state}; {label}: ikke målt: artefaktene er eldre enn regelen "
|
||||||
|
f"(approach_id mangler på {m.unaddressed} erklæring(er))"
|
||||||
|
)
|
||||||
|
diagnostics = (
|
||||||
|
f"før regelen: {m.undeclared} av {m.validated} validerte uten tilnærmingens egen "
|
||||||
|
"erklæring — regelen ville gjort dem unsupported, men modellen fikk aldri spørsmålet",
|
||||||
|
)
|
||||||
else:
|
else:
|
||||||
k, n = m.undeclared, m.validated
|
k, n = m.undeclared, m.validated
|
||||||
reason = (
|
reason = (
|
||||||
f"{label} ({m.where}): {k} av {n} validerte uten erklæring fra tilnærmingen; "
|
f"{probe_state}; {label} ({m.where}): {k} av {n} validerte uten erklæring fra "
|
||||||
f"{m.undeclared_anywhere} uten noen erklæring i kjøringen"
|
f"tilnærmingen; {m.undeclared_anywhere} uten noen erklæring i kjøringen"
|
||||||
)
|
)
|
||||||
exceptions += [f"validert uten erklæring: {a}" for a in m.undeclared_ids]
|
exceptions += [f"validert uten erklæring: {a}" for a in m.undeclared_ids]
|
||||||
red = bool(failing) or bool(k)
|
if failing or k:
|
||||||
if not probes:
|
status = RED
|
||||||
red, exceptions = True, [*exceptions, "ingen probe registrert"]
|
elif k is None:
|
||||||
|
status = NOT_MEASURED
|
||||||
|
else:
|
||||||
|
status = GREEN
|
||||||
return Row(
|
return Row(
|
||||||
"undeclared", title, k, n, RED if red else GREEN, reason, exceptions=tuple(exceptions)
|
"undeclared",
|
||||||
|
title,
|
||||||
|
k,
|
||||||
|
n,
|
||||||
|
status,
|
||||||
|
reason,
|
||||||
|
exceptions=tuple(exceptions),
|
||||||
|
diagnostics=diagnostics,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -596,7 +627,7 @@ def score_named(m: StressMeasure, label: str) -> Row:
|
||||||
title,
|
title,
|
||||||
None,
|
None,
|
||||||
None,
|
None,
|
||||||
DIAGNOSIS,
|
NOT_MEASURED,
|
||||||
f"{label}: ikke målt, artefakter mangler ({m.missing})",
|
f"{label}: ikke målt, artefakter mangler ({m.missing})",
|
||||||
failing=False,
|
failing=False,
|
||||||
diagnostics=(NAMED_WARNING,),
|
diagnostics=(NAMED_WARNING,),
|
||||||
|
|
|
||||||
|
|
@ -215,7 +215,8 @@ _INSTRUCTIONS: Final = {
|
||||||
"BINDS it: pass read_dir a 'filter' word taken from the approach's own label — "
|
"BINDS it: pass read_dir a 'filter' word taken from the approach's own label — "
|
||||||
"filter='rundkjoring' finds the level's requirements about roundabouts, and one of them is "
|
"filter='rundkjoring' finds the level's requirements about roundabouts, and one of them is "
|
||||||
"the 'Krav 4.1.2-1' you are looking for — read it with read_file, then call "
|
"the 'Krav 4.1.2-1' you are looking for — read it with read_file, then call "
|
||||||
"declare_requirement with the base id, that path and the requirement's own number. The "
|
"declare_requirement with the base id, that path, the requirement's own number and the "
|
||||||
|
"short label of the direction as approach_id. The "
|
||||||
"reply gives back the document's own title and number: if they are not about your measure, "
|
"reply gives back the document's own title and number: if they are not about your measure, "
|
||||||
"you declared the wrong requirement and should filter again. A direction with no "
|
"you declared the wrong requirement and should filter again. A direction with no "
|
||||||
"requirement behind it is a guess. "
|
"requirement behind it is a guess. "
|
||||||
|
|
@ -371,6 +372,21 @@ class DeclaredRequirement:
|
||||||
bundle_id: str
|
bundle_id: str
|
||||||
path: str
|
path: str
|
||||||
ref: str
|
ref: str
|
||||||
|
#: WHICH approach the declaration is for (row 6). A requirement bound at run level cannot be
|
||||||
|
#: attributed to one approach — the judge labelled such a declaration ``run`` — so the rule that
|
||||||
|
#: a validated proposal must rest on its own approach's declaration needs the address on the
|
||||||
|
#: record itself. In a commissioned run it is one of the mandate's ids or ``own-proposal``; the
|
||||||
|
#: exploration, which mints its directions after declaring, records the label verbatim.
|
||||||
|
approach_id: str
|
||||||
|
|
||||||
|
|
||||||
|
class UnknownApproach(ValueError):
|
||||||
|
"""A declaration named an approach this run was not commissioned with (row 6).
|
||||||
|
|
||||||
|
Returned as a refused TURN, never raised out of the run (the ``RequirementNotRead`` rule): the
|
||||||
|
refusal NAMES the valid ids, and naming them is the correction — a declaration filed under an
|
||||||
|
id no approach carries would be recorded against nothing and could never satisfy the rule.
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
class RequirementNotRead(ValueError):
|
class RequirementNotRead(ValueError):
|
||||||
|
|
@ -1068,6 +1084,7 @@ def _index_excerpt(body: str) -> tuple[str, bool]:
|
||||||
_RETURNABLE_REFUSALS: Final = (
|
_RETURNABLE_REFUSALS: Final = (
|
||||||
ExplorationError,
|
ExplorationError,
|
||||||
RequirementNotRead,
|
RequirementNotRead,
|
||||||
|
UnknownApproach,
|
||||||
okf.BundleIdMismatch,
|
okf.BundleIdMismatch,
|
||||||
okf.BundlePathNotFound,
|
okf.BundlePathNotFound,
|
||||||
okf.DocumentPathRefused,
|
okf.DocumentPathRefused,
|
||||||
|
|
@ -1186,6 +1203,7 @@ def navigator_tools(
|
||||||
opened: list[ToolCall] | None = None,
|
opened: list[ToolCall] | None = None,
|
||||||
requirements: list[DeclaredRequirement] | None = None,
|
requirements: list[DeclaredRequirement] | None = None,
|
||||||
labels: Sequence[str] = (),
|
labels: Sequence[str] = (),
|
||||||
|
approach_ids: Sequence[str] | None = None,
|
||||||
) -> list[FunctionTool]:
|
) -> list[FunctionTool]:
|
||||||
"""The navigator's tools: survey the catalogue, open one base, read one document — and, when
|
"""The navigator's tools: survey the catalogue, open one base, read one document — and, when
|
||||||
the caller offers the two sinks, DECLARE the requirement that binds a direction.
|
the caller offers the two sinks, DECLARE the requirement that binds a direction.
|
||||||
|
|
@ -1241,6 +1259,11 @@ def navigator_tools(
|
||||||
the run's own read trace and a second record of it would be free to disagree with the first.
|
the run's own read trace and a second record of it would be free to disagree with the first.
|
||||||
Passing one without the other is refused at construction: a log that cannot see what was opened
|
Passing one without the other is refused at construction: a log that cannot see what was opened
|
||||||
would accept every declaration, which is the vacuous-gate class.
|
would accept every declaration, which is the vacuous-gate class.
|
||||||
|
|
||||||
|
**Every declaration names the approach it is for** (row 6). ``approach_ids`` is the set a
|
||||||
|
commissioned run can file under — the mandate's ids plus ``own-proposal`` — and an id outside it
|
||||||
|
is refused with the valid ones named. ``None`` (the exploration, which has no ids until it mints
|
||||||
|
them) records any non-empty id verbatim.
|
||||||
"""
|
"""
|
||||||
if (opened is None) != (requirements is None):
|
if (opened is None) != (requirements is None):
|
||||||
raise ExplorationError(
|
raise ExplorationError(
|
||||||
|
|
@ -1462,20 +1485,36 @@ def navigator_tools(
|
||||||
"'Krav 4.1.2-1'), read_file to read it, then declare it. The reply gives back the "
|
"'Krav 4.1.2-1'), read_file to read it, then declare it. The reply gives back the "
|
||||||
"document's own "
|
"document's own "
|
||||||
"title and number, so you can see whether you declared the requirement you meant: a "
|
"title and number, so you can see whether you declared the requirement you meant: a "
|
||||||
"declaration of a requirement that is not about the measure is worth nothing."
|
"declaration of a requirement that is not about the measure is worth nothing. "
|
||||||
|
"approach_id names the approach the requirement binds: declare once for EACH approach "
|
||||||
|
"you propose for (the run's own proposal is 'own-proposal'). A proposal whose approach "
|
||||||
|
"declared no requirement is not validated, however good its numbers are."
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
def declare_requirement(bundle_id: str, path: str, ref: str) -> dict[str, Any]:
|
def declare_requirement(
|
||||||
|
bundle_id: str, path: str, ref: str, approach_id: str
|
||||||
|
) -> dict[str, Any]:
|
||||||
try:
|
try:
|
||||||
return _declare_requirement(bundle_id, path, ref)
|
return _declare_requirement(bundle_id, path, ref, approach_id)
|
||||||
except _RETURNABLE_REFUSALS as exc:
|
except _RETURNABLE_REFUSALS as exc:
|
||||||
return _refused_mapping(exc)
|
return _refused_mapping(exc)
|
||||||
|
|
||||||
def _declare_requirement(bundle_id: str, path: str, ref: str) -> dict[str, Any]:
|
def _declare_requirement(
|
||||||
|
bundle_id: str, path: str, ref: str, approach_id: str
|
||||||
|
) -> dict[str, Any]:
|
||||||
assert opened is not None and requirements is not None # the constructor guard above
|
assert opened is not None and requirements is not None # the constructor guard above
|
||||||
# The base is resolved by the SAME index every read rung uses, so an unknown base is
|
# The base is resolved by the SAME index every read rung uses, so an unknown base is
|
||||||
# refused here exactly as it is there rather than being accepted into the record.
|
# refused here exactly as it is there rather than being accepted into the record.
|
||||||
resolved_dir = _resolve_bundle(index, bundle_id)
|
resolved_dir = _resolve_bundle(index, bundle_id)
|
||||||
|
# Row 6: the address is checked before anything is read, so a declaration filed under no
|
||||||
|
# approach is refused whatever else is right about it.
|
||||||
|
if approach_ids is not None and approach_id not in approach_ids:
|
||||||
|
raise UnknownApproach(
|
||||||
|
f"{approach_id!r}; this run's approaches are {', '.join(map(repr, approach_ids))}. "
|
||||||
|
"Declare the requirement under the id of the approach it binds"
|
||||||
|
)
|
||||||
|
if not approach_id.strip():
|
||||||
|
raise UnknownApproach("an empty approach_id; name the approach this requirement binds")
|
||||||
read_paths = [call.path for call in opened if call.name == "read_file" and call.path]
|
read_paths = [call.path for call in opened if call.name == "read_file" and call.path]
|
||||||
if path not in read_paths:
|
if path not in read_paths:
|
||||||
raise RequirementNotRead(
|
raise RequirementNotRead(
|
||||||
|
|
@ -1518,7 +1557,9 @@ def navigator_tools(
|
||||||
"Use read_dir with a 'filter' word from the approach's own label to find the "
|
"Use read_dir with a 'filter' word from the approach's own label to find the "
|
||||||
"candidates, then read_file the ones that could bind it"
|
"candidates, then read_file the ones that could bind it"
|
||||||
)
|
)
|
||||||
requirements.append(DeclaredRequirement(bundle_id=bundle_id, path=path, ref=ref))
|
requirements.append(
|
||||||
|
DeclaredRequirement(bundle_id=bundle_id, path=path, ref=ref, approach_id=approach_id)
|
||||||
|
)
|
||||||
# P20/A1: give back the DOCUMENT's own title and number, read off the base rather than
|
# P20/A1: give back the DOCUMENT's own title and number, read off the base rather than
|
||||||
# echoed from the arguments. MEASURED (P19 round 3, P17b): 13 declarations over 5 runs and
|
# echoed from the arguments. MEASURED (P19 round 3, P17b): 13 declarations over 5 runs and
|
||||||
# NOT ONE named a fasit concept — the tool answered ``{"declared": true, ...}`` to every
|
# NOT ONE named a fasit concept — the tool answered ``{"declared": true, ...}`` to every
|
||||||
|
|
@ -1532,6 +1573,7 @@ def navigator_tools(
|
||||||
"bundle_id": bundle_id,
|
"bundle_id": bundle_id,
|
||||||
"path": path,
|
"path": path,
|
||||||
"ref": ref,
|
"ref": ref,
|
||||||
|
"approach_id": approach_id,
|
||||||
"title": declared[0],
|
"title": declared[0],
|
||||||
"req_number": declared[1],
|
"req_number": declared[1],
|
||||||
"binds": (
|
"binds": (
|
||||||
|
|
@ -1577,7 +1619,10 @@ def requirement_payload(declared: Sequence[DeclaredRequirement]) -> list[dict[st
|
||||||
copies of "what a declaration looks like" would drift into two answers about one run, which is
|
copies of "what a declaration looks like" would drift into two answers about one run, which is
|
||||||
the kø-(p) defect landing in exactly the files an operator reads after a paid run.
|
the kø-(p) defect landing in exactly the files an operator reads after a paid run.
|
||||||
"""
|
"""
|
||||||
return [{"bundle_id": d.bundle_id, "path": d.path, "ref": d.ref} for d in declared]
|
return [
|
||||||
|
{"bundle_id": d.bundle_id, "path": d.path, "ref": d.ref, "approach_id": d.approach_id}
|
||||||
|
for d in declared
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
def _refused(
|
def _refused(
|
||||||
|
|
|
||||||
|
|
@ -249,7 +249,7 @@ def _budget_payload(exc: BudgetExceeded) -> dict[str, Any]:
|
||||||
"""The exhausted-budget body: the ledger's own triple, plus the human line for the log.
|
"""The exhausted-budget body: the ledger's own triple, plus the human line for the log.
|
||||||
|
|
||||||
The ``budget_exhausted`` key's PRESENCE is the discriminator — it is not folded into
|
The ``budget_exhausted`` key's PRESENCE is the discriminator — it is not folded into
|
||||||
``outcome_type`` (whose values, ``validated``/``rejected``, mean "the run concluded and
|
``outcome_type`` (whose values, ``validated``/``rejected``/``unsupported``, mean "the run concluded and
|
||||||
here is the verdict") for the same reason ``BudgetStop`` was given its own field instead of
|
here is the verdict") for the same reason ``BudgetStop`` was given its own field instead of
|
||||||
widening ``stop_reason``. Nor could it be: ``outcome_payload`` is the ONE copy of that fork
|
widening ``stop_reason``. Nor could it be: ``outcome_payload`` is the ONE copy of that fork
|
||||||
and takes a ``ValidatedProposal | Rejection``, neither of which an exhausted run has."""
|
and takes a ``ValidatedProposal | Rejection``, neither of which an exhausted run has."""
|
||||||
|
|
|
||||||
|
|
@ -334,7 +334,9 @@ class ApproachOutcome:
|
||||||
|
|
||||||
id: str
|
id: str
|
||||||
label: str
|
label: str
|
||||||
status: Literal["validated", "rejected", "not_evaluated"]
|
#: ``unsupported`` (row 6): the numbers held but the approach declared no binding requirement,
|
||||||
|
#: so it is neither a success nor a numeric rejection. Never counted as validated.
|
||||||
|
status: Literal["validated", "rejected", "unsupported", "not_evaluated"]
|
||||||
detail: str = ""
|
detail: str = ""
|
||||||
saving_nok: float | None = None
|
saving_nok: float | None = None
|
||||||
|
|
||||||
|
|
@ -471,6 +473,8 @@ def settle(
|
||||||
lines.append(f" {row.id:<20} VALIDATED {amount:>14} {row.label}")
|
lines.append(f" {row.id:<20} VALIDATED {amount:>14} {row.label}")
|
||||||
elif row.status == "rejected":
|
elif row.status == "rejected":
|
||||||
lines.append(f" {row.id:<20} REJECTED {row.detail}")
|
lines.append(f" {row.id:<20} REJECTED {row.detail}")
|
||||||
|
elif row.status == "unsupported":
|
||||||
|
lines.append(f" {row.id:<20} UNSUPPORTED {row.detail}")
|
||||||
else:
|
else:
|
||||||
lines.append(f" {row.id:<20} NOT EVALUATED {row.detail}")
|
lines.append(f" {row.id:<20} NOT EVALUATED {row.detail}")
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -35,7 +35,7 @@ from collections.abc import Mapping, Sequence
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import TYPE_CHECKING, Any
|
from typing import TYPE_CHECKING, Any
|
||||||
|
|
||||||
from portfolio_optimiser.validator import Rejection, ValidatedProposal
|
from portfolio_optimiser.validator import Rejection, Unsupported, ValidatedProposal
|
||||||
|
|
||||||
if TYPE_CHECKING: # provenance imports agent_framework — keep it out of the runtime import graph
|
if TYPE_CHECKING: # provenance imports agent_framework — keep it out of the runtime import graph
|
||||||
from portfolio_optimiser.provenance import ProvenanceStamp
|
from portfolio_optimiser.provenance import ProvenanceStamp
|
||||||
|
|
@ -113,7 +113,21 @@ def outcome_payload(
|
||||||
validated/rejected branching, shared by ``write_outbox`` and the hosted invocations
|
validated/rejected branching, shared by ``write_outbox`` and the hosted invocations
|
||||||
response (``hosting._response_payload``). Two copies of the branch would drift, and a
|
response (``hosting._response_payload``). Two copies of the branch would drift, and a
|
||||||
drifted copy would let the HTTP surface describe an outcome the outbox never wrote —
|
drifted copy would let the HTTP surface describe an outcome the outbox never wrote —
|
||||||
the ``to_ore`` single-source rule (kø-(p)) applied to a payload shape."""
|
the ``to_ore`` single-source rule (kø-(p)) applied to a payload shape.
|
||||||
|
|
||||||
|
``unsupported`` (row 6) is checked FIRST because it subclasses ``Rejection``: it carries the
|
||||||
|
validator's percentiles (the numbers held) AND the reason, so a reader sees both halves."""
|
||||||
|
if isinstance(outcome, Unsupported):
|
||||||
|
return {
|
||||||
|
"outcome_type": "unsupported",
|
||||||
|
"reason": outcome.reason,
|
||||||
|
"p10": outcome.validated.p10,
|
||||||
|
"p50": outcome.validated.p50,
|
||||||
|
"p90": outcome.validated.p90,
|
||||||
|
"nominal_feasible": outcome.validated.nominal_feasible,
|
||||||
|
"checker_verdict": checker_verdict,
|
||||||
|
"verdict_id": verdict_id,
|
||||||
|
}
|
||||||
if isinstance(outcome, ValidatedProposal):
|
if isinstance(outcome, ValidatedProposal):
|
||||||
return {
|
return {
|
||||||
"outcome_type": "validated",
|
"outcome_type": "validated",
|
||||||
|
|
|
||||||
|
|
@ -123,8 +123,10 @@ from portfolio_optimiser.provenance import ProvenanceStamp
|
||||||
from portfolio_optimiser.reference_domain import Project, load_reference_projects
|
from portfolio_optimiser.reference_domain import Project, load_reference_projects
|
||||||
from portfolio_optimiser.tracing import TracingConfigError, configure_tracing, tracing_notice
|
from portfolio_optimiser.tracing import TracingConfigError, configure_tracing, tracing_notice
|
||||||
from portfolio_optimiser.validator import (
|
from portfolio_optimiser.validator import (
|
||||||
|
UNSUPPORTED_REASON,
|
||||||
Grounding,
|
Grounding,
|
||||||
Rejection,
|
Rejection,
|
||||||
|
Unsupported,
|
||||||
ValidatedProposal,
|
ValidatedProposal,
|
||||||
baseline_from_project,
|
baseline_from_project,
|
||||||
classify_codes,
|
classify_codes,
|
||||||
|
|
@ -410,7 +412,10 @@ def _coverage_row(
|
||||||
) -> ApproachOutcome:
|
) -> ApproachOutcome:
|
||||||
"""One coverage row from one evaluated approach. A rejection carries the validator's reason
|
"""One coverage row from one evaluated approach. A rejection carries the validator's reason
|
||||||
verbatim — a bare status would tell the expert their approach failed without telling them why,
|
verbatim — a bare status would tell the expert their approach failed without telling them why,
|
||||||
which is the part they can actually act on."""
|
which is the part they can actually act on. An ``Unsupported`` outcome (row 6) is checked
|
||||||
|
first: it subclasses ``Rejection``, and its own status is the whole point of the class."""
|
||||||
|
if isinstance(outcome, Unsupported):
|
||||||
|
return ApproachOutcome(id=row_id, label=label, status="unsupported", detail=outcome.reason)
|
||||||
if isinstance(outcome, ValidatedProposal):
|
if isinstance(outcome, ValidatedProposal):
|
||||||
return ApproachOutcome(
|
return ApproachOutcome(
|
||||||
id=row_id,
|
id=row_id,
|
||||||
|
|
@ -490,6 +495,15 @@ def evaluate_mandate_candidates(
|
||||||
return tuple(rows)
|
return tuple(rows)
|
||||||
|
|
||||||
|
|
||||||
|
def _declarable_ids(mandate: Mandate | None) -> list[str]:
|
||||||
|
"""The approach ids a declaration may be filed under (row 6): the mandate's own, plus the run's
|
||||||
|
own proposal whenever the run makes one — always without a mandate."""
|
||||||
|
ids = [a.id for a in mandate.approaches] if mandate is not None else []
|
||||||
|
if mandate is None or mandate.allow_own_proposals:
|
||||||
|
ids.append(OWN_PROPOSAL_ID)
|
||||||
|
return ids
|
||||||
|
|
||||||
|
|
||||||
def _select_outcome(
|
def _select_outcome(
|
||||||
produced: list[tuple[int, ValidatedProposal | Rejection]],
|
produced: list[tuple[int, ValidatedProposal | Rejection]],
|
||||||
) -> ValidatedProposal | Rejection:
|
) -> ValidatedProposal | Rejection:
|
||||||
|
|
@ -511,6 +525,7 @@ async def _evaluate_mandate(
|
||||||
mandate: Mandate,
|
mandate: Mandate,
|
||||||
evaluate: Callable[[Approach | None], Awaitable[ValidatedProposal | Rejection]],
|
evaluate: Callable[[Approach | None], Awaitable[ValidatedProposal | Rejection]],
|
||||||
budget_stops: list[str] | None = None,
|
budget_stops: list[str] | None = None,
|
||||||
|
declared: Sequence[DeclaredRequirement] | None = None,
|
||||||
) -> tuple[
|
) -> tuple[
|
||||||
ValidatedProposal | Rejection,
|
ValidatedProposal | Rejection,
|
||||||
tuple[ApproachOutcome, ...],
|
tuple[ApproachOutcome, ...],
|
||||||
|
|
@ -532,6 +547,14 @@ async def _evaluate_mandate(
|
||||||
become ``not_evaluated`` rows. But if the very first approach exhausts the budget there is
|
become ``not_evaluated`` rows. But if the very first approach exhausts the budget there is
|
||||||
nothing honest to return, so ``BudgetExceeded`` propagates exactly as it did before — a run
|
nothing honest to return, so ``BudgetExceeded`` propagates exactly as it did before — a run
|
||||||
that produced nothing must still fail loudly rather than hand back an empty report.
|
that produced nothing must still fail loudly rather than hand back an empty report.
|
||||||
|
|
||||||
|
**Row 6: a validated outcome must rest on ITS approach's own declaration.** When ``declared`` is
|
||||||
|
given (the rule is active — see ``run_project``), a ``ValidatedProposal`` whose approach carries
|
||||||
|
no ``requirement`` of its own and has no declaration filed under its id becomes
|
||||||
|
``Unsupported``: the numbers held, the ground was never named. A declaration made for another
|
||||||
|
approach does not stand in, and nothing about the declaration's QUALITY is judged — a
|
||||||
|
requirement the model declared is accepted whatever it says (P22 § 4: an overlap gate would
|
||||||
|
refuse legitimate proposals). ``declared`` is read at evaluation time, after the debate.
|
||||||
"""
|
"""
|
||||||
plan: list[tuple[str, str, Approach | None]] = [(a.id, a.label, a) for a in mandate.approaches]
|
plan: list[tuple[str, str, Approach | None]] = [(a.id, a.label, a) for a in mandate.approaches]
|
||||||
if mandate.allow_own_proposals:
|
if mandate.allow_own_proposals:
|
||||||
|
|
@ -563,6 +586,15 @@ async def _evaluate_mandate(
|
||||||
for rid, lbl, _ in plan[index:]
|
for rid, lbl, _ in plan[index:]
|
||||||
)
|
)
|
||||||
break
|
break
|
||||||
|
if (
|
||||||
|
declared is not None
|
||||||
|
and isinstance(outcome, ValidatedProposal)
|
||||||
|
and not (approach is not None and approach.requirement is not None)
|
||||||
|
and row_id not in {d.approach_id for d in declared}
|
||||||
|
):
|
||||||
|
outcome = Unsupported(
|
||||||
|
proposal=outcome.proposal, reason=UNSUPPORTED_REASON, validated=outcome
|
||||||
|
)
|
||||||
produced.append((index, outcome))
|
produced.append((index, outcome))
|
||||||
evaluated.append((row_id, outcome))
|
evaluated.append((row_id, outcome))
|
||||||
rows.append(_coverage_row(row_id, label, outcome))
|
rows.append(_coverage_row(row_id, label, outcome))
|
||||||
|
|
@ -622,8 +654,10 @@ def _bundle_pointer(bundle: okf.Bundle, bundle_id: str, *, dimension: str | None
|
||||||
"requirement number or path contains that word, and reports 'total_matches'.\n"
|
"requirement number or path contains that word, and reports 'total_matches'.\n"
|
||||||
"Before you settle on a measure, name the ONE requirement of this base that BINDS it: "
|
"Before you settle on a measure, name the ONE requirement of this base that BINDS it: "
|
||||||
"find it with a filter, read it with read_file, then call "
|
"find it with a filter, read it with read_file, then call "
|
||||||
f"declare_requirement({bundle_id!r}, path, ref) with the requirement's own number. A "
|
f"declare_requirement({bundle_id!r}, path, ref, approach_id) with the requirement's own "
|
||||||
"declaration naming a document this run never opened is refused; reading it is the fix."
|
"number and the id of the approach it binds (the run's own proposal is 'own-proposal'). "
|
||||||
|
"Declare once per approach: a proposal whose approach declared nothing is not validated. "
|
||||||
|
"A declaration naming a document this run never opened is refused; reading it is the fix."
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -1184,6 +1218,12 @@ async def run_project(
|
||||||
#: documents above, so the gate's vocabulary and the gate's text describe one reading of one
|
#: documents above, so the gate's vocabulary and the gate's text describe one reading of one
|
||||||
#: base. Empty on the road path, which is what keeps the rule unable to fire there.
|
#: base. Empty on the road path, which is what keeps the rule unable to fire there.
|
||||||
bundle_references: tuple[str, ...] = ()
|
bundle_references: tuple[str, ...] = ()
|
||||||
|
#: Row 6: whether the declaration rung was offered to the debate. The rule that a validated
|
||||||
|
#: proposal needs its approach's own declaration is active exactly when it was: a run that
|
||||||
|
#: could declare and did not is the silence the rule exists for. It is NOT relaxed for a base
|
||||||
|
#: without requirement numbers — any document the run read can be declared, and a base that
|
||||||
|
#: holds nothing worth declaring is a finding about the base, not a reason to validate.
|
||||||
|
requirement_rung = False
|
||||||
# S2c: a CALLER-OWNED sink for what the debate opens (the ``parse_failures``/``ExplorationTrace``
|
# S2c: a CALLER-OWNED sink for what the debate opens (the ``parse_failures``/``ExplorationTrace``
|
||||||
# shape). A returned value would be lost on exactly the run that most needs the evidence — a
|
# shape). A returned value would be lost on exactly the run that most needs the evidence — a
|
||||||
# budget stop mid-debate raises out of ``debate.run`` and constructs no ``RunResult`` at all.
|
# budget stop mid-debate raises out of ``debate.run`` and constructs no ``RunResult`` at all.
|
||||||
|
|
@ -1300,8 +1340,11 @@ async def run_project(
|
||||||
# names every direction the run carries rather than picking one it cannot
|
# names every direction the run carries rather than picking one it cannot
|
||||||
# attribute. Without a mandate this is empty and the reply is unchanged.
|
# attribute. Without a mandate this is empty and the reply is unchanged.
|
||||||
labels=[a.label for a in mandate.approaches] if mandate else (),
|
labels=[a.label for a in mandate.approaches] if mandate else (),
|
||||||
|
# Row 6: the ids a declaration may be filed under.
|
||||||
|
approach_ids=_declarable_ids(mandate),
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
|
requirement_rung = True
|
||||||
# What the navigation could NOT reach, taken from the run's ONE walk. The road path below
|
# What the navigation could NOT reach, taken from the run's ONE walk. The road path below
|
||||||
# navigates no bundle at all, so its empty tuple is literally true rather than a stand-in.
|
# navigates no bundle at all, so its empty tuple is literally true rather than a stand-in.
|
||||||
skipped_links: tuple[okf.SkippedLink, ...] = bundle.skipped
|
skipped_links: tuple[okf.SkippedLink, ...] = bundle.skipped
|
||||||
|
|
@ -1588,7 +1631,10 @@ async def run_project(
|
||||||
validator_outcome = await _evaluate(None)
|
validator_outcome = await _evaluate(None)
|
||||||
else:
|
else:
|
||||||
validator_outcome, coverage, evaluated = await _evaluate_mandate(
|
validator_outcome, coverage, evaluated = await _evaluate_mandate(
|
||||||
mandate, _evaluate, budget_stops
|
mandate,
|
||||||
|
_evaluate,
|
||||||
|
budget_stops,
|
||||||
|
declared=debate_requirements if requirement_rung else None,
|
||||||
)
|
)
|
||||||
except BaseException as stop:
|
except BaseException as stop:
|
||||||
# Recorded and re-raised UNTOUCHED. This arm decides nothing about the exception itself —
|
# Recorded and re-raised UNTOUCHED. This arm decides nothing about the exception itself —
|
||||||
|
|
@ -1679,7 +1725,9 @@ async def run_project(
|
||||||
model=model,
|
model=model,
|
||||||
role="proposer",
|
role="proposer",
|
||||||
validator_decision=(
|
validator_decision=(
|
||||||
"validated" if isinstance(validator_outcome, ValidatedProposal) else "rejected"
|
"validated"
|
||||||
|
if isinstance(validator_outcome, (ValidatedProposal, Unsupported))
|
||||||
|
else "rejected"
|
||||||
),
|
),
|
||||||
token_usage=meter.tokens,
|
token_usage=meter.tokens,
|
||||||
# Whether stage 0 of the deterministic gate had a baseline to reconcile against. Read off
|
# Whether stage 0 of the deterministic gate had a baseline to reconcile against. Read off
|
||||||
|
|
@ -1799,7 +1847,7 @@ async def run_project(
|
||||||
update={
|
update={
|
||||||
"validator_decision": (
|
"validator_decision": (
|
||||||
"validated"
|
"validated"
|
||||||
if isinstance(approach_outcome, ValidatedProposal)
|
if isinstance(approach_outcome, (ValidatedProposal, Unsupported))
|
||||||
else "rejected"
|
else "rejected"
|
||||||
),
|
),
|
||||||
# P20: ``code_forms`` must follow ITS OWN approach too, for
|
# P20: ``code_forms`` must follow ITS OWN approach too, for
|
||||||
|
|
|
||||||
|
|
@ -69,7 +69,7 @@ import argparse
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
import sys
|
import sys
|
||||||
from collections.abc import Sequence
|
from collections.abc import Mapping, Sequence
|
||||||
from dataclasses import asdict, dataclass
|
from dataclasses import asdict, dataclass
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
@ -98,7 +98,7 @@ class ApproachVerdict:
|
||||||
|
|
||||||
approach_id: str
|
approach_id: str
|
||||||
label: str
|
label: str
|
||||||
status: str # "validated" | "rejected" | "not_evaluated"
|
status: str # "validated" | "rejected" | "unsupported" | "not_evaluated"
|
||||||
#: (a) - grounded by an OPENED path, or by a citation under a NARROWED list. See the module
|
#: (a) - grounded by an OPENED path, or by a citation under a NARROWED list. See the module
|
||||||
#: docstring: a whole-base citation list is stamped before any model call and grounds nothing.
|
#: docstring: a whole-base citation list is stamped before any model call and grounds nothing.
|
||||||
grounded: bool
|
grounded: bool
|
||||||
|
|
@ -197,6 +197,10 @@ class ContextSetVerdict:
|
||||||
approach_rows_seen: int
|
approach_rows_seen: int
|
||||||
concepts_in_base: int
|
concepts_in_base: int
|
||||||
ferdig: bool
|
ferdig: bool
|
||||||
|
#: Row 6 - declarations with no ``approach_id``: written before declarations carried one. A run
|
||||||
|
#: with any cannot be measured against the rule that a validated proposal needs its own
|
||||||
|
#: approach's declaration, and the v1 gate says so instead of counting.
|
||||||
|
unaddressed_declarations: int = 0
|
||||||
|
|
||||||
def to_payload(self) -> dict[str, Any]:
|
def to_payload(self) -> dict[str, Any]:
|
||||||
"""Byte-stable plain data: the ONE rendering, shared by the CLI's stdout and its file."""
|
"""Byte-stable plain data: the ONE rendering, shared by the CLI's stdout and its file."""
|
||||||
|
|
@ -265,21 +269,29 @@ def _inside(base: Path, raw: str) -> Path | None:
|
||||||
return resolved if resolved == root or root in resolved.parents else None
|
return resolved if resolved == root or root in resolved.parents else None
|
||||||
|
|
||||||
|
|
||||||
def _attributable(approach: Any, declared: Sequence[str]) -> tuple[tuple[str, ...], str]:
|
def _attributable(
|
||||||
|
approach: Any, declared: Sequence[Mapping[str, Any]]
|
||||||
|
) -> tuple[tuple[str, ...], str]:
|
||||||
"""Which declared requirement paths this approach may be judged on, and where they came from.
|
"""Which declared requirement paths this approach may be judged on, and where they came from.
|
||||||
|
|
||||||
The approach's OWN requirement wins when it has one: the mandate carries it per approach, so
|
The approach's OWN requirement wins when it has one: the mandate carries it per approach, so
|
||||||
it is unambiguous by construction. Otherwise the RUN's declarations are attributable - the
|
it is unambiguous by construction. Next, a declaration filed under THIS approach's id (row 6)
|
||||||
debate declares once for the whole run, so the row says ``run`` rather than pretending the
|
is the approach's own and says ``approach`` too. A declaration with no ``approach_id`` at all
|
||||||
declaration was made about it. ``absent`` is the third value and is not the same as "declared
|
was written before declarations carried one; it can only be attributed to the whole run, and
|
||||||
nothing that matched": a run that declared nothing is a different finding from one that
|
the row says ``run`` rather than pretending it was made about this approach. A declaration
|
||||||
declared the wrong document."""
|
filed under ANOTHER approach's id is not this one's, so a run whose declarations all name other
|
||||||
|
approaches reads ``absent`` here — the same value as a run that declared nothing.
|
||||||
|
"""
|
||||||
own = getattr(approach, "requirement", None)
|
own = getattr(approach, "requirement", None)
|
||||||
path = "" if own is None else str(getattr(own, "path", "") or "")
|
path = "" if own is None else str(getattr(own, "path", "") or "")
|
||||||
if path:
|
if path:
|
||||||
return (path,), "approach"
|
return (path,), "approach"
|
||||||
if declared:
|
addressed = tuple(str(r["path"]) for r in declared if r.get("approach_id") == approach.id)
|
||||||
return tuple(declared), "run"
|
if addressed:
|
||||||
|
return addressed, "approach"
|
||||||
|
legacy = tuple(str(r["path"]) for r in declared if "approach_id" not in r)
|
||||||
|
if legacy:
|
||||||
|
return legacy, "run"
|
||||||
return (), "absent"
|
return (), "absent"
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -359,14 +371,13 @@ def score_context_set(
|
||||||
# what every stress round has been) writes no ``{run_id}-exploration.json`` at all - the
|
# what every stress round has been) writes no ``{run_id}-exploration.json`` at all - the
|
||||||
# hypothesiser never runs - so the debate artefact is the only one that can carry them there.
|
# hypothesiser never runs - so the debate artefact is the only one that can carry them there.
|
||||||
# Both are read, because an ``--explore`` run carries them in the other.
|
# Both are read, because an ``--explore`` run carries them in the other.
|
||||||
declared_paths: list[str] = []
|
declared_records: list[dict[str, Any]] = []
|
||||||
for artefact in (debate, outbox / f"{run_id}-exploration.json"):
|
for artefact in (debate, outbox / f"{run_id}-exploration.json"):
|
||||||
if artefact.is_file():
|
if artefact.is_file():
|
||||||
declared_paths += [
|
declared_records += [
|
||||||
str(r.get("path", ""))
|
r for r in _read_json(artefact).get("requirements", []) if r.get("path")
|
||||||
for r in _read_json(artefact).get("requirements", [])
|
|
||||||
if r.get("path")
|
|
||||||
]
|
]
|
||||||
|
declared_paths = [str(r["path"]) for r in declared_records]
|
||||||
hallucinated_reads: list[str] = []
|
hallucinated_reads: list[str] = []
|
||||||
for call in tool_calls:
|
for call in tool_calls:
|
||||||
raw = str(call.get("path", ""))
|
raw = str(call.get("path", ""))
|
||||||
|
|
@ -417,9 +428,11 @@ def score_context_set(
|
||||||
named_in_measure=False,
|
named_in_measure=False,
|
||||||
named_in_snippet=False,
|
named_in_snippet=False,
|
||||||
hallucinations=(),
|
hallucinations=(),
|
||||||
requirement_declared=_attributable(approach, declared_paths)[0],
|
requirement_declared=_attributable(approach, declared_records)[0],
|
||||||
requirement_source=_attributable(approach, declared_paths)[1],
|
requirement_source=_attributable(approach, declared_records)[1],
|
||||||
requirement_hit=bool(set(_attributable(approach, declared_paths)[0]) & wanted),
|
requirement_hit=bool(
|
||||||
|
set(_attributable(approach, declared_records)[0]) & wanted
|
||||||
|
),
|
||||||
prose_codes=(),
|
prose_codes=(),
|
||||||
priced=False,
|
priced=False,
|
||||||
not_evaluated_reason=stop_reason or "absent",
|
not_evaluated_reason=stop_reason or "absent",
|
||||||
|
|
@ -472,7 +485,7 @@ def score_context_set(
|
||||||
forms = payload.get("provenance", {}).get("code_forms") or classify_codes(codes)
|
forms = payload.get("provenance", {}).get("code_forms") or classify_codes(codes)
|
||||||
prose_codes = tuple(sorted(c for c in codes if forms.get(c) == "prose"))
|
prose_codes = tuple(sorted(c for c in codes if forms.get(c) == "prose"))
|
||||||
|
|
||||||
attributable, requirement_source = _attributable(approach, declared_paths)
|
attributable, requirement_source = _attributable(approach, declared_records)
|
||||||
requirement_hit = bool(set(attributable) & wanted)
|
requirement_hit = bool(set(attributable) & wanted)
|
||||||
|
|
||||||
halluc = [f"citation:{f}" for f in sorted(cited_files - concept_names)]
|
halluc = [f"citation:{f}" for f in sorted(cited_files - concept_names)]
|
||||||
|
|
@ -563,6 +576,7 @@ def score_context_set(
|
||||||
must_refuse=tuple(refusals),
|
must_refuse=tuple(refusals),
|
||||||
hallucinated_reads=tuple(hallucinated_reads),
|
hallucinated_reads=tuple(hallucinated_reads),
|
||||||
requirements_declared=tuple(declared_paths),
|
requirements_declared=tuple(declared_paths),
|
||||||
|
unaddressed_declarations=sum(1 for r in declared_records if "approach_id" not in r),
|
||||||
token_usage=token_usage,
|
token_usage=token_usage,
|
||||||
stop_reason=stop_reason if coverage_seen else "absent",
|
stop_reason=stop_reason if coverage_seen else "absent",
|
||||||
anchored=anchored,
|
anchored=anchored,
|
||||||
|
|
|
||||||
|
|
@ -102,6 +102,30 @@ class Rejection:
|
||||||
reason: str
|
reason: str
|
||||||
|
|
||||||
|
|
||||||
|
#: The ONE sentence an unsupported outcome carries. ``rejection_stage`` keys on it, so the judge can
|
||||||
|
#: tell this falsifier from the numeric ones without a second copy of the wording.
|
||||||
|
UNSUPPORTED_REASON: Final = (
|
||||||
|
"no declared requirement for this approach: the numbers held, but no requirement of the "
|
||||||
|
"knowledge base was declared as binding it"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class Unsupported(Rejection):
|
||||||
|
"""A proposal whose NUMBERS held but whose approach declared no binding requirement (row 6).
|
||||||
|
|
||||||
|
Neither ``validated`` (nothing in the knowledge base was said to support the direction) nor an
|
||||||
|
ordinary rejection (every numeric stage passed). It subclasses ``Rejection`` on purpose: every
|
||||||
|
consumer that asks "is this validated?" with ``isinstance(..., ValidatedProposal)`` answers no
|
||||||
|
without being touched, so it is never counted, summed or carried as a success. The consumers
|
||||||
|
that NAME the status — coverage, the outcome artefact, the settlement, the judge — check for
|
||||||
|
this class first. ``validated`` keeps the validator's own ruling, which is what
|
||||||
|
``provenance.validator_decision`` mirrors: the validator said yes, and the record says so.
|
||||||
|
"""
|
||||||
|
|
||||||
|
validated: ValidatedProposal
|
||||||
|
|
||||||
|
|
||||||
def _solve_max_feasible(items: list[AffectedItem], fraction: float) -> float:
|
def _solve_max_feasible(items: list[AffectedItem], fraction: float) -> float:
|
||||||
"""Real CBC solve: maximize total saving subject to a per-item upper bound and a
|
"""Real CBC solve: maximize total saving subject to a per-item upper bound and a
|
||||||
global fraction cap. Raises ``CbcUnavailable`` if CBC is genuinely missing."""
|
global fraction cap. Raises ``CbcUnavailable`` if CBC is genuinely missing."""
|
||||||
|
|
@ -728,6 +752,7 @@ _REJECTION_STAGES: Final = (
|
||||||
("stage4-p90", ("exceeds P90 feasible",)),
|
("stage4-p90", ("exceeds P90 feasible",)),
|
||||||
("stage4b-nominal", ("exceeds the nominal feasible",)),
|
("stage4b-nominal", ("exceeds the nominal feasible",)),
|
||||||
("stage5-method-cap", ("method cap",)),
|
("stage5-method-cap", ("method cap",)),
|
||||||
|
("unsupported", ("no declared requirement for this approach",)),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -38,7 +38,7 @@ from collections.abc import Callable
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
from portfolio_optimiser import hitl
|
from portfolio_optimiser import hitl
|
||||||
from portfolio_optimiser.mandate import OWN_PROPOSAL_ID, Approach, Mandate
|
from portfolio_optimiser.mandate import BindingRequirement, OWN_PROPOSAL_ID, Approach, Mandate
|
||||||
from portfolio_optimiser.run import run_project
|
from portfolio_optimiser.run import run_project
|
||||||
from portfolio_optimiser.simulation import ScriptedChatClient
|
from portfolio_optimiser.simulation import ScriptedChatClient
|
||||||
from portfolio_optimiser.verdicts import (
|
from portfolio_optimiser.verdicts import (
|
||||||
|
|
@ -66,8 +66,11 @@ def _reply(measure: str, claimed: int) -> str:
|
||||||
|
|
||||||
# Labels ABSENT from the bundle's own prose (the "LED-retrofit" trap: it appears in 6 bundle files,
|
# Labels ABSENT from the bundle's own prose (the "LED-retrofit" trap: it appears in 6 bundle files,
|
||||||
# so a client keyed on it would match every prompt through the context and prove nothing).
|
# so a client keyed on it would match every prompt through the context and prove nothing).
|
||||||
_LED = Approach(id="led-retrofit", label="Behovsstyrt belysning i fellesarealer")
|
#: Row 6: a commissioned approach validates only on a requirement of its own. These tests are about
|
||||||
_HVAC = Approach(id="hvac-swap", label="Utskifting av ventilasjonsaggregat")
|
#: selection and per-approach artefacts, not about declarations, so the commission names one.
|
||||||
|
_REQ = BindingRequirement(path="tiltak-led-retrofit.md", ref="Krav 1")
|
||||||
|
_LED = Approach(id="led-retrofit", label="Behovsstyrt belysning i fellesarealer", requirement=_REQ)
|
||||||
|
_HVAC = Approach(id="hvac-swap", label="Utskifting av ventilasjonsaggregat", requirement=_REQ)
|
||||||
|
|
||||||
_LED_MEASURE = "Behovsstyrt belysning i fellesarealer"
|
_LED_MEASURE = "Behovsstyrt belysning i fellesarealer"
|
||||||
_HVAC_MEASURE = "Utskifting av ventilasjonsaggregat"
|
_HVAC_MEASURE = "Utskifting av ventilasjonsaggregat"
|
||||||
|
|
@ -181,7 +184,8 @@ async def test_each_approach_outcome_is_written_with_its_own_status(tmp_path: Pa
|
||||||
assert by_approach["led-retrofit"]["outcome_type"] == "validated"
|
assert by_approach["led-retrofit"]["outcome_type"] == "validated"
|
||||||
assert by_approach["hvac-swap"]["outcome_type"] == "rejected"
|
assert by_approach["hvac-swap"]["outcome_type"] == "rejected"
|
||||||
assert by_approach["hvac-swap"]["reason"], "a rejected approach must carry the reason"
|
assert by_approach["hvac-swap"]["reason"], "a rejected approach must carry the reason"
|
||||||
assert by_approach[OWN_PROPOSAL_ID]["outcome_type"] == "validated"
|
# The run's own proposal names no requirement and none was declared for it (row 6).
|
||||||
|
assert by_approach[OWN_PROPOSAL_ID]["outcome_type"] == "unsupported"
|
||||||
|
|
||||||
|
|
||||||
async def test_each_artefact_stamps_its_own_validator_decision(tmp_path: Path) -> None:
|
async def test_each_artefact_stamps_its_own_validator_decision(tmp_path: Path) -> None:
|
||||||
|
|
|
||||||
|
|
@ -362,7 +362,7 @@ def test_the_requirement_gate_reads_the_second_bases_own_opened_list(tmp_path: P
|
||||||
opened_1.append(explore.ToolCall(name="read_file", bundle_id=bundle_1, path=doc_1))
|
opened_1.append(explore.ToolCall(name="read_file", bundle_id=bundle_1, path=doc_1))
|
||||||
assert not opened_2, "the two bases shared one opened sink"
|
assert not opened_2, "the two bases shared one opened sink"
|
||||||
|
|
||||||
refusal = tools_2["declare_requirement"].func(bundle_2, doc_1, "Krav 1")
|
refusal = tools_2["declare_requirement"].func(bundle_2, doc_1, "Krav 1", "a1")
|
||||||
assert refusal["refusal"] == "RequirementNotRead", refusal
|
assert refusal["refusal"] == "RequirementNotRead", refusal
|
||||||
assert not reqs_2, "base 2 recorded a requirement it never read"
|
assert not reqs_2, "base 2 recorded a requirement it never read"
|
||||||
|
|
||||||
|
|
@ -371,7 +371,7 @@ def test_the_requirement_gate_reads_the_second_bases_own_opened_list(tmp_path: P
|
||||||
for name in [f.name for f in okf.navigate_bundle(second).context_files][:3]:
|
for name in [f.name for f in okf.navigate_bundle(second).context_files][:3]:
|
||||||
tools_2["read_file"].func(bundle_2, name)
|
tools_2["read_file"].func(bundle_2, name)
|
||||||
opened_2.append(explore.ToolCall(name="read_file", bundle_id=bundle_2, path=name))
|
opened_2.append(explore.ToolCall(name="read_file", bundle_id=bundle_2, path=name))
|
||||||
accepted = tools_2["declare_requirement"].func(bundle_2, doc_2, "Krav 1")
|
accepted = tools_2["declare_requirement"].func(bundle_2, doc_2, "Krav 1", "a1")
|
||||||
assert accepted.get("declared") is True, accepted
|
assert accepted.get("declared") is True, accepted
|
||||||
assert [r.path for r in reqs_2] == [doc_2]
|
assert [r.path for r in reqs_2] == [doc_2]
|
||||||
assert not reqs_1, "the two bases shared one requirements sink"
|
assert not reqs_1, "the two bases shared one requirements sink"
|
||||||
|
|
|
||||||
|
|
@ -102,7 +102,7 @@ def test_a_requirement_the_run_never_opened_is_refused_and_recorded_nowhere() ->
|
||||||
"""(a) The declaration's one falsifier is the run's own read trace."""
|
"""(a) The declaration's one falsifier is the run's own read trace."""
|
||||||
tools, _opened, declared = _wired()
|
tools, _opened, declared = _wired()
|
||||||
answer = tools["declare_requirement"].func(
|
answer = tools["declare_requirement"].func(
|
||||||
bundle_id="tunnel-hauglia", path=_a_concept(), ref="12.1"
|
approach_id="a1", bundle_id="tunnel-hauglia", path=_a_concept(), ref="12.1"
|
||||||
)
|
)
|
||||||
assert answer["refusal"] == "RequirementNotRead"
|
assert answer["refusal"] == "RequirementNotRead"
|
||||||
assert "0 document(s)" in answer["refused"]
|
assert "0 document(s)" in answer["refused"]
|
||||||
|
|
@ -130,7 +130,7 @@ def test_the_correction_is_to_read_it_and_then_it_is_accepted() -> None:
|
||||||
opened.append(ToolCall(name="read_file", bundle_id="tunnel-hauglia", path=other))
|
opened.append(ToolCall(name="read_file", bundle_id="tunnel-hauglia", path=other))
|
||||||
opened.append(ToolCall(name="read_file", bundle_id="tunnel-hauglia", path=path))
|
opened.append(ToolCall(name="read_file", bundle_id="tunnel-hauglia", path=path))
|
||||||
answer = tools["declare_requirement"].func(
|
answer = tools["declare_requirement"].func(
|
||||||
bundle_id="tunnel-hauglia", path=path, ref="Krav 12.1"
|
approach_id="a1", bundle_id="tunnel-hauglia", path=path, ref="Krav 12.1"
|
||||||
)
|
)
|
||||||
# P20/A1 widened the reply: the three arguments PLUS the document's own title and number and
|
# P20/A1 widened the reply: the three arguments PLUS the document's own title and number and
|
||||||
# the sentence saying what the declaration binds. Asserted key by key rather than by equality,
|
# the sentence saying what the declaration binds. Asserted key by key rather than by equality,
|
||||||
|
|
@ -142,15 +142,30 @@ def test_the_correction_is_to_read_it_and_then_it_is_accepted() -> None:
|
||||||
path,
|
path,
|
||||||
"Krav 12.1",
|
"Krav 12.1",
|
||||||
)
|
)
|
||||||
assert set(answer) == {"declared", "bundle_id", "path", "ref", "title", "req_number", "binds"}
|
assert set(answer) == {
|
||||||
assert declared == [DeclaredRequirement(bundle_id="tunnel-hauglia", path=path, ref="Krav 12.1")]
|
"declared",
|
||||||
|
"bundle_id",
|
||||||
|
"path",
|
||||||
|
"ref",
|
||||||
|
"approach_id",
|
||||||
|
"title",
|
||||||
|
"req_number",
|
||||||
|
"binds",
|
||||||
|
}
|
||||||
|
assert declared == [
|
||||||
|
DeclaredRequirement(
|
||||||
|
bundle_id="tunnel-hauglia", path=path, ref="Krav 12.1", approach_id="a1"
|
||||||
|
)
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
def test_a_declaration_naming_an_unknown_base_is_refused() -> None:
|
def test_a_declaration_naming_an_unknown_base_is_refused() -> None:
|
||||||
"""(c) The same base index every read rung uses, so no second, laxer resolution exists."""
|
"""(c) The same base index every read rung uses, so no second, laxer resolution exists."""
|
||||||
tools, opened, declared = _wired()
|
tools, opened, declared = _wired()
|
||||||
opened.append(ToolCall(name="read_file", bundle_id="nope", path="a.md"))
|
opened.append(ToolCall(name="read_file", bundle_id="nope", path="a.md"))
|
||||||
answer = tools["declare_requirement"].func(bundle_id="nope", path="a.md", ref="1")
|
answer = tools["declare_requirement"].func(
|
||||||
|
approach_id="a1", bundle_id="nope", path="a.md", ref="1"
|
||||||
|
)
|
||||||
assert answer["refusal"] == "ExplorationError"
|
assert answer["refusal"] == "ExplorationError"
|
||||||
assert declared == []
|
assert declared == []
|
||||||
|
|
||||||
|
|
@ -252,14 +267,15 @@ def test_a_hit_is_counted_against_this_approachs_fasit_never_the_base() -> None:
|
||||||
label="L",
|
label="L",
|
||||||
requirement=BindingRequirement(path="krav/12-1/a.md", ref="12.1"),
|
requirement=BindingRequirement(path="krav/12-1/a.md", ref="12.1"),
|
||||||
)
|
)
|
||||||
paths, source = stress._attributable(approach, ["krav/99-9/elsewhere.md"])
|
paths, source = stress._attributable(approach, [{"path": "krav/99-9/elsewhere.md"}])
|
||||||
assert (paths, source) == (("krav/12-1/a.md",), "approach")
|
assert (paths, source) == (("krav/12-1/a.md",), "approach")
|
||||||
assert set(paths) & wanted
|
assert set(paths) & wanted
|
||||||
|
|
||||||
# A run-level declaration that is NOT one of this approach's fasit concepts is reported and is
|
# A run-level declaration (one written before declarations carried an ``approach_id``) that is
|
||||||
# not a hit — the base holds it, which is exactly what must not count.
|
# NOT one of this approach's fasit concepts is reported and is not a hit — the base holds it,
|
||||||
|
# which is exactly what must not count.
|
||||||
bare = Approach(id="a2", label="M")
|
bare = Approach(id="a2", label="M")
|
||||||
paths, source = stress._attributable(bare, ["krav/99-9/elsewhere.md"])
|
paths, source = stress._attributable(bare, [{"path": "krav/99-9/elsewhere.md"}])
|
||||||
assert (paths, source) == (("krav/99-9/elsewhere.md",), "run")
|
assert (paths, source) == (("krav/99-9/elsewhere.md",), "run")
|
||||||
assert not set(paths) & wanted
|
assert not set(paths) & wanted
|
||||||
|
|
||||||
|
|
@ -408,11 +424,13 @@ def test_the_declaration_reaches_the_debate_artefact(tmp_path: Path) -> None:
|
||||||
[ToolCall(name="read_file", bundle_id="b", path="k/a.md")]
|
[ToolCall(name="read_file", bundle_id="b", path="k/a.md")]
|
||||||
),
|
),
|
||||||
requirements=explore.requirement_payload(
|
requirements=explore.requirement_payload(
|
||||||
[DeclaredRequirement(bundle_id="b", path="k/a.md", ref="12.1")]
|
[DeclaredRequirement(bundle_id="b", path="k/a.md", ref="12.1", approach_id="a1")]
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
payload = json.loads((tmp_path / "r1-debate.json").read_text(encoding="utf-8"))
|
payload = json.loads((tmp_path / "r1-debate.json").read_text(encoding="utf-8"))
|
||||||
assert payload["requirements"] == [{"bundle_id": "b", "path": "k/a.md", "ref": "12.1"}]
|
assert payload["requirements"] == [
|
||||||
|
{"bundle_id": "b", "path": "k/a.md", "ref": "12.1", "approach_id": "a1"}
|
||||||
|
]
|
||||||
# An empty list is the honest positive statement every pre-P19 run makes.
|
# An empty list is the honest positive statement every pre-P19 run makes.
|
||||||
outbox.write_debate_tools(str(tmp_path), "r2", tool_calls=[])
|
outbox.write_debate_tools(str(tmp_path), "r2", tool_calls=[])
|
||||||
assert (
|
assert (
|
||||||
|
|
|
||||||
|
|
@ -70,7 +70,7 @@ def test_c1_a_declaration_after_one_document_is_refused_with_the_denominator() -
|
||||||
opened.append(ToolCall(name="read_file", bundle_id="tunnel-hauglia", path=path))
|
opened.append(ToolCall(name="read_file", bundle_id="tunnel-hauglia", path=path))
|
||||||
|
|
||||||
answer = tools["declare_requirement"].func(
|
answer = tools["declare_requirement"].func(
|
||||||
bundle_id="tunnel-hauglia", path=path, ref="Krav 1.1—1"
|
approach_id="a1", bundle_id="tunnel-hauglia", path=path, ref="Krav 1.1—1"
|
||||||
)
|
)
|
||||||
|
|
||||||
assert answer["refusal"] == "RequirementNotRead"
|
assert answer["refusal"] == "RequirementNotRead"
|
||||||
|
|
@ -93,7 +93,7 @@ def test_c1_the_same_declaration_after_three_documents_is_accepted() -> None:
|
||||||
opened.append(ToolCall(name="read_file", bundle_id="tunnel-hauglia", path=name))
|
opened.append(ToolCall(name="read_file", bundle_id="tunnel-hauglia", path=name))
|
||||||
|
|
||||||
answer = tools["declare_requirement"].func(
|
answer = tools["declare_requirement"].func(
|
||||||
bundle_id="tunnel-hauglia", path=files[0], ref="Krav 1.1—1"
|
approach_id="a1", bundle_id="tunnel-hauglia", path=files[0], ref="Krav 1.1—1"
|
||||||
)
|
)
|
||||||
|
|
||||||
assert answer["declared"] is True
|
assert answer["declared"] is True
|
||||||
|
|
@ -109,7 +109,7 @@ def test_c1_the_same_path_read_three_times_is_still_one_document() -> None:
|
||||||
opened.append(ToolCall(name="read_file", bundle_id="tunnel-hauglia", path=path))
|
opened.append(ToolCall(name="read_file", bundle_id="tunnel-hauglia", path=path))
|
||||||
|
|
||||||
answer = tools["declare_requirement"].func(
|
answer = tools["declare_requirement"].func(
|
||||||
bundle_id="tunnel-hauglia", path=path, ref="Krav 1.1—1"
|
approach_id="a1", bundle_id="tunnel-hauglia", path=path, ref="Krav 1.1—1"
|
||||||
)
|
)
|
||||||
|
|
||||||
assert answer["refusal"] == "RequirementNotRead"
|
assert answer["refusal"] == "RequirementNotRead"
|
||||||
|
|
@ -135,7 +135,9 @@ def test_c1_a_small_base_stays_declarable(tmp_path: Path) -> None:
|
||||||
for name in ("a.md", "b.md"):
|
for name in ("a.md", "b.md"):
|
||||||
opened.append(ToolCall(name="read_file", bundle_id="mini", path=name))
|
opened.append(ToolCall(name="read_file", bundle_id="mini", path=name))
|
||||||
|
|
||||||
answer = tools["declare_requirement"].func(bundle_id="mini", path="a.md", ref="1")
|
answer = tools["declare_requirement"].func(
|
||||||
|
approach_id="a1", bundle_id="mini", path="a.md", ref="1"
|
||||||
|
)
|
||||||
|
|
||||||
assert answer["declared"] is True
|
assert answer["declared"] is True
|
||||||
assert len(declared) == 1
|
assert len(declared) == 1
|
||||||
|
|
@ -151,7 +153,7 @@ def test_c1_the_never_opened_refusal_still_fires_first() -> None:
|
||||||
opened.append(ToolCall(name="read_file", bundle_id="tunnel-hauglia", path=name))
|
opened.append(ToolCall(name="read_file", bundle_id="tunnel-hauglia", path=name))
|
||||||
|
|
||||||
answer = tools["declare_requirement"].func(
|
answer = tools["declare_requirement"].func(
|
||||||
bundle_id="tunnel-hauglia", path=files[4], ref="Krav 1.1—1"
|
approach_id="a1", bundle_id="tunnel-hauglia", path=files[4], ref="Krav 1.1—1"
|
||||||
)
|
)
|
||||||
|
|
||||||
assert answer["refusal"] == "RequirementNotRead"
|
assert answer["refusal"] == "RequirementNotRead"
|
||||||
|
|
|
||||||
|
|
@ -634,7 +634,8 @@ async def test_a_hosted_exploration_shapes_the_mandate_the_run_evaluates(_hosted
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
|
|
||||||
assert body["outcome_type"] in {"validated", "rejected"}
|
# ``unsupported`` (row 6) is an evaluated outcome too: the numbers were judged.
|
||||||
|
assert body["outcome_type"] in {"validated", "rejected", "unsupported"}
|
||||||
assert any(_LABEL in prompt for prompt in _hosted_backend), (
|
assert any(_LABEL in prompt for prompt in _hosted_backend), (
|
||||||
"the shaped approach never reached a prompt — the hosted door does not wire the mandate"
|
"the shaped approach never reached a prompt — the hosted door does not wire the mandate"
|
||||||
)
|
)
|
||||||
|
|
|
||||||
|
|
@ -24,7 +24,7 @@ from pathlib import Path
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
from portfolio_optimiser.mandate import OWN_PROPOSAL_ID, Approach, Mandate
|
from portfolio_optimiser.mandate import BindingRequirement, OWN_PROPOSAL_ID, Approach, Mandate
|
||||||
from portfolio_optimiser.simulation import ScriptedChatClient
|
from portfolio_optimiser.simulation import ScriptedChatClient
|
||||||
from portfolio_optimiser.run import run_project
|
from portfolio_optimiser.run import run_project
|
||||||
from portfolio_optimiser.validator import Rejection, ValidatedProposal
|
from portfolio_optimiser.validator import Rejection, ValidatedProposal
|
||||||
|
|
@ -48,8 +48,11 @@ def _reply(measure: str, claimed: int) -> str:
|
||||||
# Labels are chosen to be ABSENT from the bundle's own prose: "LED-retrofit" appears in 6 of the
|
# Labels are chosen to be ABSENT from the bundle's own prose: "LED-retrofit" appears in 6 of the
|
||||||
# bundle's files, so a client keyed on it would match every prompt through the context and prove
|
# bundle's files, so a client keyed on it would match every prompt through the context and prove
|
||||||
# nothing about which approach was bound to which call.
|
# nothing about which approach was bound to which call.
|
||||||
_LED = Approach(id="led-retrofit", label="Behovsstyrt belysning i fellesarealer")
|
#: Row 6: a commissioned approach validates only on a requirement of its own. These tests are about
|
||||||
_HVAC = Approach(id="hvac-swap", label="Utskifting av ventilasjonsaggregat")
|
#: selection and per-approach artefacts, not about declarations, so the commission names one.
|
||||||
|
_REQ = BindingRequirement(path="tiltak-led-retrofit.md", ref="Krav 1")
|
||||||
|
_LED = Approach(id="led-retrofit", label="Behovsstyrt belysning i fellesarealer", requirement=_REQ)
|
||||||
|
_HVAC = Approach(id="hvac-swap", label="Utskifting av ventilasjonsaggregat", requirement=_REQ)
|
||||||
|
|
||||||
#: LED validates (30k <= cap); HVAC is above the cap -> the validator rejects it.
|
#: LED validates (30k <= cap); HVAC is above the cap -> the validator rejects it.
|
||||||
_REPLY_BY_LABEL = {
|
_REPLY_BY_LABEL = {
|
||||||
|
|
@ -160,8 +163,12 @@ async def test_outcome_is_the_best_validated_candidate_deterministically(big_fir
|
||||||
orderings pinned, ``produced[-1]`` fails one case and ``produced[0]`` fails the other.
|
orderings pinned, ``produced[-1]`` fails one case and ``produced[0]`` fails the other.
|
||||||
"""
|
"""
|
||||||
sink: list[str] = []
|
sink: list[str] = []
|
||||||
big = Approach(id="big", label="Behovsstyrt belysning i fellesarealer") # 30k, validates
|
big = Approach(
|
||||||
small = Approach(id="small", label="Nattsenking av temperatur") # default reply, 20k
|
id="big", label="Behovsstyrt belysning i fellesarealer", requirement=_REQ
|
||||||
|
) # 30k, validates
|
||||||
|
small = Approach(
|
||||||
|
id="small", label="Nattsenking av temperatur", requirement=_REQ
|
||||||
|
) # default reply, 20k
|
||||||
mandate = Mandate(
|
mandate = Mandate(
|
||||||
objective="Cut energy cost without rebuilding.",
|
objective="Cut energy cost without rebuilding.",
|
||||||
approaches=(big, small) if big_first else (small, big),
|
approaches=(big, small) if big_first else (small, big),
|
||||||
|
|
|
||||||
|
|
@ -61,7 +61,9 @@ def _declare(
|
||||||
for name in [f.name for f in okf.navigate_bundle(str(_TUNNEL)).context_files][:3]:
|
for name in [f.name for f in okf.navigate_bundle(str(_TUNNEL)).context_files][:3]:
|
||||||
opened.append(ToolCall(name="read_file", bundle_id=_BASE_ID, path=name))
|
opened.append(ToolCall(name="read_file", bundle_id=_BASE_ID, path=name))
|
||||||
opened.append(ToolCall(name="read_file", bundle_id=_BASE_ID, path=_DOC))
|
opened.append(ToolCall(name="read_file", bundle_id=_BASE_ID, path=_DOC))
|
||||||
answer = tools["declare_requirement"].func(bundle_id=_BASE_ID, path=_DOC, ref=ref)
|
answer = tools["declare_requirement"].func(
|
||||||
|
approach_id="a1", bundle_id=_BASE_ID, path=_DOC, ref=ref
|
||||||
|
)
|
||||||
return answer, declared
|
return answer, declared
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -115,7 +117,8 @@ def test_the_comparison_never_reads_the_callers_own_ref() -> None:
|
||||||
|
|
||||||
def test_without_directions_the_reply_is_the_one_p20_shipped() -> None:
|
def test_without_directions_the_reply_is_the_one_p20_shipped() -> None:
|
||||||
"""The exploration mints its own directions, so at declaration time it HAS none — and every
|
"""The exploration mints its own directions, so at declaration time it HAS none — and every
|
||||||
call site that passes no labels must be byte-identical to before. Absent keys, not empty ones:
|
call site that passes no labels must carry no comparison keys (row 6 added ``approach_id`` to
|
||||||
|
every reply, the address the declaration was filed under). Absent keys, not empty ones:
|
||||||
"there was nothing to compare against" and "we compared and found nothing" are different
|
"there was nothing to compare against" and "we compared and found nothing" are different
|
||||||
facts, and only one of them is true here."""
|
facts, and only one of them is true here."""
|
||||||
answer, _ = _declare(())
|
answer, _ = _declare(())
|
||||||
|
|
@ -124,6 +127,7 @@ def test_without_directions_the_reply_is_the_one_p20_shipped() -> None:
|
||||||
"bundle_id",
|
"bundle_id",
|
||||||
"path",
|
"path",
|
||||||
"ref",
|
"ref",
|
||||||
|
"approach_id",
|
||||||
"title",
|
"title",
|
||||||
"req_number",
|
"req_number",
|
||||||
"binds",
|
"binds",
|
||||||
|
|
@ -180,7 +184,12 @@ async def test_a_commissioned_run_reaches_the_tool_with_its_own_directions() ->
|
||||||
{"call": "read_file", "args": {"bundle_id": _BASE_ID, "path": _DOC}},
|
{"call": "read_file", "args": {"bundle_id": _BASE_ID, "path": _DOC}},
|
||||||
{
|
{
|
||||||
"call": "declare_requirement",
|
"call": "declare_requirement",
|
||||||
"args": {"bundle_id": _BASE_ID, "path": _DOC, "ref": "Krav 1.1-1"},
|
"args": {
|
||||||
|
"bundle_id": _BASE_ID,
|
||||||
|
"path": _DOC,
|
||||||
|
"ref": "Krav 1.1-1",
|
||||||
|
"approach_id": "a1",
|
||||||
|
},
|
||||||
},
|
},
|
||||||
_VALID_REPLY,
|
_VALID_REPLY,
|
||||||
_VALID_REPLY,
|
_VALID_REPLY,
|
||||||
|
|
|
||||||
|
|
@ -78,7 +78,9 @@ def _declare(
|
||||||
for other in also:
|
for other in also:
|
||||||
opened.append(ToolCall(name="read_file", bundle_id=base, path=other))
|
opened.append(ToolCall(name="read_file", bundle_id=base, path=other))
|
||||||
opened.append(ToolCall(name="read_file", bundle_id=base, path=path))
|
opened.append(ToolCall(name="read_file", bundle_id=base, path=path))
|
||||||
return tools["declare_requirement"].func(bundle_id=base, path=path, ref="Krav 4.1.2-1")
|
return tools["declare_requirement"].func(
|
||||||
|
approach_id="a1", bundle_id=base, path=path, ref="Krav 4.1.2-1"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _base_with_a_requirement(root: Path) -> Path:
|
def _base_with_a_requirement(root: Path) -> Path:
|
||||||
|
|
@ -114,7 +116,11 @@ def test_the_reply_carries_the_documents_own_title_and_number(tmp_path: Path) ->
|
||||||
assert answer["title"] == "Krav 4.1.2-1 Rundkjoring"
|
assert answer["title"] == "Krav 4.1.2-1 Rundkjoring"
|
||||||
assert answer["req_number"] == "Krav 4.1.2-1"
|
assert answer["req_number"] == "Krav 4.1.2-1"
|
||||||
assert "is the requirement the proposal rests on" in answer["binds"]
|
assert "is the requirement the proposal rests on" in answer["binds"]
|
||||||
assert declared == [DeclaredRequirement(bundle_id="n-mini", path="krav.md", ref="Krav 4.1.2-1")]
|
assert declared == [
|
||||||
|
DeclaredRequirement(
|
||||||
|
bundle_id="n-mini", path="krav.md", ref="Krav 4.1.2-1", approach_id="a1"
|
||||||
|
)
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
def test_a_path_that_is_no_concept_answers_with_empty_strings(tmp_path: Path) -> None:
|
def test_a_path_that_is_no_concept_answers_with_empty_strings(tmp_path: Path) -> None:
|
||||||
|
|
|
||||||
292
tests/test_row6_declaration_rule_loadbearing.py
Normal file
292
tests/test_row6_declaration_rule_loadbearing.py
Normal file
|
|
@ -0,0 +1,292 @@
|
||||||
|
"""Row 6 — a proposal whose approach declared no requirement cannot carry ``validated``.
|
||||||
|
|
||||||
|
Before this rule every numeric stage could pass and the run would stamp ``validated`` whether or
|
||||||
|
not anything in the knowledge base had been named as binding the direction. Measured on stress
|
||||||
|
round 6: three falsification arms validated, and every one of the ten validated approaches had only
|
||||||
|
run-level declarations — which a judge cannot attribute to any one approach.
|
||||||
|
|
||||||
|
What each arm pins:
|
||||||
|
|
||||||
|
(a) on the shipped micro base, where the debate holds ``declare_requirement``, a silent approach
|
||||||
|
that used to validate is now ``unsupported`` — and so is the run's own proposal;
|
||||||
|
(b) a declaration filed under the approach's id is what lets it validate, and ONLY that approach;
|
||||||
|
(c) a requirement written into the mandate counts as the approach's own declaration;
|
||||||
|
(d) the declaration's quality is not judged: any requirement the run read is accepted;
|
||||||
|
(e) an id no approach carries is refused with the valid ids named, and nothing is recorded;
|
||||||
|
(f) ``unsupported`` is never counted, summed or selected as a success, yet the validator's own
|
||||||
|
ruling is kept on the record (``provenance.validator_decision``) and in the artefact;
|
||||||
|
(g) with the rule inactive (no declaration rung offered) the old ruling stands;
|
||||||
|
(h) the judge reads the addressed declaration as the approach's own, a legacy one as ``run``.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from portfolio_optimiser import okf, outbox, stress
|
||||||
|
from portfolio_optimiser.explore import DeclaredRequirement, navigator_tools, requirement_payload
|
||||||
|
from portfolio_optimiser.ir import AffectedItem, SavingsProposal
|
||||||
|
from portfolio_optimiser.mandate import Approach, BindingRequirement, Mandate, settle
|
||||||
|
from portfolio_optimiser.run import _evaluate_mandate, run_project
|
||||||
|
from portfolio_optimiser.simulation import scripted_factory
|
||||||
|
from portfolio_optimiser.validator import (
|
||||||
|
UNSUPPORTED_REASON,
|
||||||
|
Rejection,
|
||||||
|
Unsupported,
|
||||||
|
ValidatedProposal,
|
||||||
|
rejection_stage,
|
||||||
|
)
|
||||||
|
from portfolio_optimiser.verdicts import VerdictStore
|
||||||
|
|
||||||
|
_BUNDLE = Path(__file__).resolve().parents[1] / "shared" / "examples" / "bygg-energi-mikro"
|
||||||
|
_BASE_ID = "bygg-energi-mikro"
|
||||||
|
_PID = "BYGG-KONTOR-NORD"
|
||||||
|
_REPLY = (
|
||||||
|
'{"measure":"LED-retrofit av kontorbelysning","affected_items":'
|
||||||
|
'[{"code":"ENERGI-TOTAL-EL","quantity":300000,"unit_cost":1.0}],"claimed_saving_nok":30000}'
|
||||||
|
)
|
||||||
|
_CHECKER = "Reasoning holds.\nVERDICT: APPROVE"
|
||||||
|
_CONCEPTS = [f.name for f in okf.navigate_bundle(str(_BUNDLE)).context_files]
|
||||||
|
|
||||||
|
|
||||||
|
def _mandate(**approach: Any) -> Mandate:
|
||||||
|
return Mandate(
|
||||||
|
objective="Kutt energikostnad",
|
||||||
|
approaches=(Approach(id="a1", label="LED-retrofit", description="reason", **approach),),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _script(*declare_for: str, ref: str = "Krav 1.1-1") -> dict[str, Any]:
|
||||||
|
steps: list[Any] = []
|
||||||
|
if declare_for:
|
||||||
|
steps += [
|
||||||
|
{"call": "read_file", "args": {"bundle_id": _BASE_ID, "path": n}} for n in _CONCEPTS[:3]
|
||||||
|
]
|
||||||
|
steps += [
|
||||||
|
{
|
||||||
|
"call": "declare_requirement",
|
||||||
|
"args": {
|
||||||
|
"bundle_id": _BASE_ID,
|
||||||
|
"path": _CONCEPTS[0],
|
||||||
|
"ref": ref,
|
||||||
|
"approach_id": aid,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for aid in declare_for
|
||||||
|
]
|
||||||
|
return {"proposer": [*steps, _REPLY, _REPLY, _REPLY, _REPLY, _REPLY], "checker": _CHECKER}
|
||||||
|
|
||||||
|
|
||||||
|
async def _run(tmp_path: Path, script: dict[str, Any], mandate: Mandate) -> Any:
|
||||||
|
return await run_project(
|
||||||
|
_PID,
|
||||||
|
"local",
|
||||||
|
docs_dir=str(_BUNDLE),
|
||||||
|
bundle_dir=str(_BUNDLE),
|
||||||
|
store=VerdictStore(verdicts=[]),
|
||||||
|
client_factory=scripted_factory(script, []),
|
||||||
|
mandate=mandate,
|
||||||
|
outbox_dir=str(tmp_path),
|
||||||
|
run_id="r6",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _statuses(result: Any) -> dict[str, str]:
|
||||||
|
return {row.id: row.status for row in result.coverage}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
# (a)-(d) the rule on the real run
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_a_silent_approach_on_the_micro_base_is_unsupported(tmp_path: Path) -> None:
|
||||||
|
result = await _run(tmp_path, _script(), _mandate())
|
||||||
|
assert _statuses(result) == {"a1": "unsupported", "own-proposal": "unsupported"}
|
||||||
|
assert {row.detail for row in result.coverage} == {UNSUPPORTED_REASON}
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_only_the_approach_that_declared_is_validated(tmp_path: Path) -> None:
|
||||||
|
result = await _run(tmp_path, _script("a1"), _mandate())
|
||||||
|
assert _statuses(result) == {"a1": "validated", "own-proposal": "unsupported"}
|
||||||
|
debate = json.loads((tmp_path / "r6-debate.json").read_text(encoding="utf-8"))
|
||||||
|
assert [r["approach_id"] for r in debate["requirements"]] == ["a1"]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_a_declaration_for_the_own_proposal_does_not_stand_in(tmp_path: Path) -> None:
|
||||||
|
result = await _run(tmp_path, _script("own-proposal"), _mandate())
|
||||||
|
assert _statuses(result) == {"a1": "unsupported", "own-proposal": "validated"}
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_a_requirement_written_into_the_mandate_counts(tmp_path: Path) -> None:
|
||||||
|
requirement = BindingRequirement(path=_CONCEPTS[0], ref="Krav 1.1-1")
|
||||||
|
result = await _run(tmp_path, _script(), _mandate(requirement=requirement))
|
||||||
|
assert _statuses(result)["a1"] == "validated"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_the_declarations_quality_is_not_judged(tmp_path: Path) -> None:
|
||||||
|
result = await _run(tmp_path, _script("a1", ref="anything at all"), _mandate())
|
||||||
|
assert _statuses(result)["a1"] == "validated"
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
# (e) the address
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _tool(approach_ids: list[str] | None) -> tuple[Any, list[DeclaredRequirement]]:
|
||||||
|
opened: list[Any] = []
|
||||||
|
declared: list[DeclaredRequirement] = []
|
||||||
|
tools = navigator_tools(
|
||||||
|
[str(_BUNDLE)], opened=opened, requirements=declared, approach_ids=approach_ids
|
||||||
|
)
|
||||||
|
from portfolio_optimiser.explore import ToolCall
|
||||||
|
|
||||||
|
opened += [ToolCall(name="read_file", bundle_id=_BASE_ID, path=n) for n in _CONCEPTS[:3]]
|
||||||
|
return {t.name: t for t in tools}["declare_requirement"], declared
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_unknown_approach_id_is_refused_naming_the_valid_ones() -> None:
|
||||||
|
tool, declared = _tool(["a1", "own-proposal"])
|
||||||
|
reply = tool.func(bundle_id=_BASE_ID, path=_CONCEPTS[0], ref="K", approach_id="a9")
|
||||||
|
assert reply["refusal"] == "UnknownApproach"
|
||||||
|
assert "'a1'" in reply["refused"] and "'own-proposal'" in reply["refused"]
|
||||||
|
assert declared == []
|
||||||
|
ok = tool.func(bundle_id=_BASE_ID, path=_CONCEPTS[0], ref="K", approach_id="a1")
|
||||||
|
assert ok["declared"] is True and ok["approach_id"] == "a1"
|
||||||
|
assert declared == [
|
||||||
|
DeclaredRequirement(bundle_id=_BASE_ID, path=_CONCEPTS[0], ref="K", approach_id="a1")
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_without_a_commission_any_label_is_recorded_but_never_an_empty_one() -> None:
|
||||||
|
tool, declared = _tool(None)
|
||||||
|
assert tool.func(bundle_id=_BASE_ID, path=_CONCEPTS[0], ref="K", approach_id=" ")["refusal"]
|
||||||
|
assert declared == []
|
||||||
|
tool.func(bundle_id=_BASE_ID, path=_CONCEPTS[0], ref="K", approach_id="LED-retrofit")
|
||||||
|
assert requirement_payload(declared) == [
|
||||||
|
{"bundle_id": _BASE_ID, "path": _CONCEPTS[0], "ref": "K", "approach_id": "LED-retrofit"}
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
# (f) never counted as a success; the validator's ruling kept
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_unsupported_is_never_a_success_but_keeps_the_validators_ruling(
|
||||||
|
tmp_path: Path,
|
||||||
|
) -> None:
|
||||||
|
result = await _run(tmp_path, _script(), _mandate())
|
||||||
|
assert isinstance(result.outcome, Unsupported)
|
||||||
|
assert not isinstance(result.outcome, ValidatedProposal)
|
||||||
|
assert result.provenance.validator_decision == "validated"
|
||||||
|
text = settle(result.coverage)
|
||||||
|
assert "UNSUPPORTED" in text and "Validated: 0 of 2" in text
|
||||||
|
artefact = json.loads((tmp_path / "r6-a1-outcome.json").read_text(encoding="utf-8"))
|
||||||
|
assert artefact["outcome_type"] == "unsupported"
|
||||||
|
assert artefact["reason"] == UNSUPPORTED_REASON
|
||||||
|
assert artefact["p50"] > 0
|
||||||
|
proposal = json.loads((tmp_path / "r6-a1-proposal.json").read_text(encoding="utf-8"))
|
||||||
|
assert proposal["provenance"]["validator_decision"] == "validated"
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_stage_label_names_the_new_falsifier() -> None:
|
||||||
|
assert rejection_stage(UNSUPPORTED_REASON) == "unsupported"
|
||||||
|
|
||||||
|
|
||||||
|
def _validated() -> ValidatedProposal:
|
||||||
|
proposal = SavingsProposal(
|
||||||
|
project_id=_PID,
|
||||||
|
measure="m",
|
||||||
|
affected_items=[AffectedItem(code="X", quantity=10.0, unit_cost=10.0)],
|
||||||
|
claimed_saving_nok=5.0,
|
||||||
|
)
|
||||||
|
return ValidatedProposal(proposal=proposal, p10=1.0, p50=2.0, p90=3.0, nominal_feasible=4.0)
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_outcome_payload_carries_both_halves() -> None:
|
||||||
|
v = _validated()
|
||||||
|
payload = outbox.outcome_payload(
|
||||||
|
Unsupported(proposal=v.proposal, reason=UNSUPPORTED_REASON, validated=v),
|
||||||
|
checker_verdict="approve",
|
||||||
|
verdict_id="k",
|
||||||
|
)
|
||||||
|
assert (payload["outcome_type"], payload["p90"]) == ("unsupported", 3.0)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
# (g) inactive rule
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("declared", "status"), [(None, "validated"), ([], "unsupported")], ids=["inactive", "active"]
|
||||||
|
)
|
||||||
|
async def test_the_rule_acts_only_when_a_declaration_rung_was_offered(
|
||||||
|
declared: list[DeclaredRequirement] | None, status: str
|
||||||
|
) -> None:
|
||||||
|
v = _validated()
|
||||||
|
|
||||||
|
async def evaluate(_approach: Approach | None) -> ValidatedProposal | Rejection:
|
||||||
|
return v
|
||||||
|
|
||||||
|
_, rows, _ = await _evaluate_mandate(
|
||||||
|
Mandate(objective="o", approaches=(Approach(id="a1", label="l"),)),
|
||||||
|
evaluate,
|
||||||
|
declared=declared,
|
||||||
|
)
|
||||||
|
assert {r.id: r.status for r in rows} == {"a1": status, "own-proposal": status}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
# (h) the judge
|
||||||
|
# ---------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_judge_attributes_addressed_declarations_and_labels_legacy_ones() -> None:
|
||||||
|
a1 = Approach(id="a1", label="l")
|
||||||
|
new = [{"path": "p1", "approach_id": "a1"}, {"path": "p2", "approach_id": "own-proposal"}]
|
||||||
|
assert stress._attributable(a1, new) == (("p1",), "approach")
|
||||||
|
assert stress._attributable(Approach(id="a2", label="l"), new) == ((), "absent")
|
||||||
|
assert stress._attributable(a1, [{"path": "p0"}]) == (("p0",), "run")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_a_run_offered_no_declaration_rung_keeps_the_validators_ruling(
|
||||||
|
tmp_path: Path,
|
||||||
|
) -> None:
|
||||||
|
"""The road path holds no knowledge base, so no ``declare_requirement`` exists there: nothing
|
||||||
|
could have been declared, and the rule stays out of it. Drives the REAL ``run_project`` —
|
||||||
|
the arm above only proves ``_evaluate_mandate`` honours ``declared=None``, not that the run
|
||||||
|
passes it."""
|
||||||
|
from portfolio_optimiser.reference_domain import load_reference_projects
|
||||||
|
from portfolio_optimiser.validator import proposal_for
|
||||||
|
|
||||||
|
project = load_reference_projects()[0]
|
||||||
|
reply = proposal_for(project, ["05.2", "03.1"], claimed_saving_nok=200_000).model_dump_json()
|
||||||
|
docs = tmp_path / "docs"
|
||||||
|
docs.mkdir()
|
||||||
|
(docs / "kilde.md").write_text("Cost saving measure candidates for the project.\n", "utf-8")
|
||||||
|
result = await run_project(
|
||||||
|
project.id,
|
||||||
|
"local",
|
||||||
|
docs_dir=str(docs),
|
||||||
|
store=VerdictStore(verdicts=[]),
|
||||||
|
client_factory=scripted_factory({"proposer": reply, "checker": _CHECKER}, []),
|
||||||
|
mandate=Mandate(
|
||||||
|
objective="o", approaches=(Approach(id="a1", label="l"),), allow_own_proposals=False
|
||||||
|
),
|
||||||
|
)
|
||||||
|
assert _statuses(result) == {"a1": "validated"}
|
||||||
|
|
@ -355,7 +355,7 @@ async def test_a_commissioned_run_writes_the_coverage_artefact(tmp_path: Path) -
|
||||||
payload = json.loads((out / "r1-coverage.json").read_text(encoding="utf-8"))
|
payload = json.loads((out / "r1-coverage.json").read_text(encoding="utf-8"))
|
||||||
assert payload["stop_reason"] == "", "nothing cut this run short"
|
assert payload["stop_reason"] == "", "nothing cut this run short"
|
||||||
assert [r["id"] for r in payload["rows"]] == ["a1"]
|
assert [r["id"] for r in payload["rows"]] == ["a1"]
|
||||||
assert payload["rows"][0]["status"] in {"validated", "rejected"}
|
assert payload["rows"][0]["status"] in {"validated", "rejected", "unsupported"}
|
||||||
|
|
||||||
|
|
||||||
async def test_a_mandateless_run_leaves_the_outbox_byte_identical(tmp_path: Path) -> None:
|
async def test_a_mandateless_run_leaves_the_outbox_byte_identical(tmp_path: Path) -> None:
|
||||||
|
|
|
||||||
|
|
@ -328,13 +328,47 @@ def test_row6_green_needs_both_the_probes_and_zero_undeclared() -> None:
|
||||||
assert gate.score_undeclared(_PROBES, failing, _CLEAN, "s").status == gate.RED
|
assert gate.score_undeclared(_PROBES, failing, _CLEAN, "s").status == gate.RED
|
||||||
|
|
||||||
|
|
||||||
def test_row6_missing_artefacts_are_never_zero() -> None:
|
def test_row6_missing_artefacts_are_never_zero_and_never_green() -> None:
|
||||||
|
"""Probes passing and artefacts absent (clean clone, CI, a base mid-rebuild) is NOT a pass:
|
||||||
|
the row says IKKE MÅLT and fails the exit code."""
|
||||||
row = gate.score_undeclared(
|
row = gate.score_undeclared(
|
||||||
_PROBES, _all_pass(_PROBES), gate.StressMeasure(missing="ut finnes ikke"), "s"
|
_PROBES, _all_pass(_PROBES), gate.StressMeasure(missing="ut finnes ikke"), "s"
|
||||||
)
|
)
|
||||||
assert row.k is None
|
assert row.k is None
|
||||||
assert "ikke målt, artefakter mangler" in row.reason
|
assert "ikke målt" in row.reason
|
||||||
assert "– av –" in row.line()
|
assert "– av –" in row.line()
|
||||||
|
assert (row.status, row.failing) == (gate.NOT_MEASURED, True)
|
||||||
|
assert gate.exit_code([row]) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_row6_artefacts_older_than_the_rule_are_not_measured() -> None:
|
||||||
|
old = gate.StressMeasure(validated=10, undeclared=10, unaddressed=12, where="x")
|
||||||
|
row = gate.score_undeclared(_PROBES, _all_pass(_PROBES), old, "s")
|
||||||
|
assert (row.k, row.status) == (None, gate.NOT_MEASURED)
|
||||||
|
assert "eldre enn regelen" in row.reason and "approach_id mangler" in row.reason
|
||||||
|
|
||||||
|
|
||||||
|
def test_row7_not_measured_is_not_green_either() -> None:
|
||||||
|
row = gate.score_named(gate.StressMeasure(missing="borte"), "s")
|
||||||
|
assert (row.k, row.status, row.failing) == (None, gate.NOT_MEASURED, False)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("missing", ["all", "outcome0", "kept"])
|
||||||
|
def test_rows_1_2_4_with_missing_files_are_red(tmp_path: Path, missing: str) -> None:
|
||||||
|
root = _green_rounds(tmp_path / "r")
|
||||||
|
if missing == "all":
|
||||||
|
root = tmp_path / "absent"
|
||||||
|
elif missing == "outcome0":
|
||||||
|
(root / "0" / "outcome.json").unlink()
|
||||||
|
else:
|
||||||
|
(root / "3" / "report.kept.md").unlink()
|
||||||
|
rows = [
|
||||||
|
gate.score_rounds(root, 3, _AI),
|
||||||
|
gate.score_changes(root, 3, _AI),
|
||||||
|
gate.score_kept(root, 0.8),
|
||||||
|
]
|
||||||
|
assert gate.exit_code(rows) == 1
|
||||||
|
assert gate.GREEN not in {r.status for r in rows} or missing != "all"
|
||||||
|
|
||||||
|
|
||||||
def test_row6_measures_the_stress_outboxes_when_they_exist(tmp_path: Path) -> None:
|
def test_row6_measures_the_stress_outboxes_when_they_exist(tmp_path: Path) -> None:
|
||||||
|
|
@ -353,6 +387,9 @@ def test_row6_measures_the_stress_outboxes_when_they_exist(tmp_path: Path) -> No
|
||||||
assert m.validated == 0
|
assert m.validated == 0
|
||||||
pytest.skip(f"stress artefacts not judgeable right now: {m.missing}")
|
pytest.skip(f"stress artefacts not judgeable right now: {m.missing}")
|
||||||
assert (m.validated, m.undeclared, m.named, m.commissioned) == (10, 10, 1, 20)
|
assert (m.validated, m.undeclared, m.named, m.commissioned) == (10, 10, 1, 20)
|
||||||
|
assert m.unaddressed > 0 # stress round 6 predates approach-addressed declarations
|
||||||
|
row = gate.score_undeclared(_PROBES, _all_pass(_PROBES), m, "s")
|
||||||
|
assert (row.k, row.status) == (None, gate.NOT_MEASURED)
|
||||||
|
|
||||||
|
|
||||||
def test_row7_is_a_diagnosis_and_never_moves_the_exit_code() -> None:
|
def test_row7_is_a_diagnosis_and_never_moves_the_exit_code() -> None:
|
||||||
|
|
@ -427,5 +464,7 @@ def test_the_command_is_red_today_with_every_row_in_its_output(tmp_path: Path) -
|
||||||
assert (rows["types"]["k"], rows["types"]["n"]) == (3, 8)
|
assert (rows["types"]["k"], rows["types"]["n"]) == (3, 8)
|
||||||
assert rows["kept"]["status"] == gate.RED
|
assert rows["kept"]["status"] == gate.RED
|
||||||
assert (rows["maf"]["k"], rows["maf"]["n"], rows["maf"]["status"]) == (3, 8, gate.RED)
|
assert (rows["maf"]["k"], rows["maf"]["n"], rows["maf"]["status"]) == (3, 8, gate.RED)
|
||||||
assert rows["undeclared"]["status"] == gate.RED
|
# Probes green since row 6; stress round 6 predates the rule (or is absent) -> never green.
|
||||||
|
assert rows["undeclared"]["status"] == gate.NOT_MEASURED
|
||||||
|
assert rows["undeclared"]["k"] is None
|
||||||
assert rows["named"]["failing"] is False
|
assert rows["named"]["failing"] is False
|
||||||
|
|
|
||||||
|
|
@ -14,10 +14,9 @@ action is exactly partial. Such a probe goes green only when it is rewritten to
|
||||||
and observe what it does.
|
and observe what it does.
|
||||||
|
|
||||||
**Row 6 (a validated proposal whose approach declared no requirement).** Two probes against the
|
**Row 6 (a validated proposal whose approach declared no requirement).** Two probes against the
|
||||||
real ``run_project``: no declaration anywhere, and a declaration made by the RUN (the debate) but
|
real ``run_project`` on the shipped micro base, where the debate holds ``declare_requirement``:
|
||||||
not by the approach. The second is the reading the gate measures the stress outboxes with: a
|
no declaration anywhere, and a declaration filed for ANOTHER approach (the run's own proposal).
|
||||||
run-level declaration cannot be attributed to one approach (the judge labels it ``run``), so it
|
Neither may leave ``a1`` validated — a declaration counts only under its own approach's id.
|
||||||
does not count as the approach having declared anything.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
@ -110,42 +109,34 @@ def _mandate() -> Mandate:
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
async def _statuses(script: dict[str, Any], tmp_path: Path) -> dict[str, str]:
|
async def _statuses(script: dict[str, Any], tmp_path: Path, base: Path) -> dict[str, str]:
|
||||||
result = await run_project(
|
result = await run_project(
|
||||||
_PID,
|
_PID,
|
||||||
"local",
|
"local",
|
||||||
docs_dir=str(_BUNDLE),
|
docs_dir=str(base),
|
||||||
bundle_dir=str(_BUNDLE),
|
bundle_dir=str(base),
|
||||||
store=VerdictStore(verdicts=[]),
|
store=VerdictStore(verdicts=[]),
|
||||||
client_factory=scripted_factory(script, []),
|
client_factory=scripted_factory(script, []),
|
||||||
mandate=_mandate(),
|
mandate=_mandate(),
|
||||||
outbox_dir=str(tmp_path),
|
outbox_dir=str(tmp_path / "out"),
|
||||||
run_id="v1-row6",
|
run_id="v1-row6",
|
||||||
)
|
)
|
||||||
return {row.id: row.status for row in result.coverage}
|
return {row.id: row.status for row in result.coverage}
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.xfail(strict=True, reason="row 6: no stage refuses a validation with no declaration")
|
def _declaring_script(base: Path, approach_id: str) -> dict[str, Any]:
|
||||||
@pytest.mark.asyncio
|
concepts = [f.name for f in okf.navigate_bundle(str(base)).context_files][:3]
|
||||||
async def test_row6_an_approach_that_declared_nothing_cannot_be_validated(tmp_path: Path) -> None:
|
return {
|
||||||
statuses = await _statuses({"proposer": _VALID_REPLY, "checker": _CHECKER_REPLY}, tmp_path)
|
|
||||||
debate = json.loads((tmp_path / "v1-row6-debate.json").read_text(encoding="utf-8"))
|
|
||||||
assert debate["requirements"] == [] # precondition: nothing was declared anywhere
|
|
||||||
assert statuses["a1"] != "validated", "validated without any declared requirement"
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.xfail(strict=True, reason="row 6: a run-level declaration still stands in")
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_row6_a_run_level_declaration_does_not_stand_in_for_the_approach(
|
|
||||||
tmp_path: Path,
|
|
||||||
) -> None:
|
|
||||||
concepts = [f.name for f in okf.navigate_bundle(str(_BUNDLE)).context_files][:3]
|
|
||||||
script = {
|
|
||||||
"proposer": [
|
"proposer": [
|
||||||
*({"call": "read_file", "args": {"bundle_id": _BASE_ID, "path": n}} for n in concepts),
|
*({"call": "read_file", "args": {"bundle_id": _BASE_ID, "path": n}} for n in concepts),
|
||||||
{
|
{
|
||||||
"call": "declare_requirement",
|
"call": "declare_requirement",
|
||||||
"args": {"bundle_id": _BASE_ID, "path": concepts[0], "ref": "probe"},
|
"args": {
|
||||||
|
"bundle_id": _BASE_ID,
|
||||||
|
"path": concepts[0],
|
||||||
|
"ref": "Krav 1.1-1",
|
||||||
|
"approach_id": approach_id,
|
||||||
|
},
|
||||||
},
|
},
|
||||||
_VALID_REPLY,
|
_VALID_REPLY,
|
||||||
_VALID_REPLY,
|
_VALID_REPLY,
|
||||||
|
|
@ -154,7 +145,30 @@ async def test_row6_a_run_level_declaration_does_not_stand_in_for_the_approach(
|
||||||
],
|
],
|
||||||
"checker": _CHECKER_REPLY,
|
"checker": _CHECKER_REPLY,
|
||||||
}
|
}
|
||||||
statuses = await _statuses(script, tmp_path)
|
|
||||||
debate = json.loads((tmp_path / "v1-row6-debate.json").read_text(encoding="utf-8"))
|
|
||||||
assert [r["path"] for r in debate["requirements"]] == [concepts[0]] # precondition
|
def _declared(tmp_path: Path) -> list[dict[str, Any]]:
|
||||||
|
debate = json.loads((tmp_path / "out" / "v1-row6-debate.json").read_text(encoding="utf-8"))
|
||||||
|
return list(debate["requirements"])
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_row6_an_approach_that_declared_nothing_cannot_be_validated(tmp_path: Path) -> None:
|
||||||
|
base = _BUNDLE
|
||||||
|
statuses = await _statuses(
|
||||||
|
{"proposer": _VALID_REPLY, "checker": _CHECKER_REPLY}, tmp_path, base
|
||||||
|
)
|
||||||
|
assert _declared(tmp_path) == [] # precondition: nothing was declared anywhere
|
||||||
|
assert statuses["a1"] != "validated", "validated without any declared requirement"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_row6_a_run_level_declaration_does_not_stand_in_for_the_approach(
|
||||||
|
tmp_path: Path,
|
||||||
|
) -> None:
|
||||||
|
"""The debate declares a requirement — but for the run's OWN proposal, not for ``a1``."""
|
||||||
|
base = _BUNDLE
|
||||||
|
statuses = await _statuses(_declaring_script(base, "own-proposal"), tmp_path, base)
|
||||||
|
declared = _declared(tmp_path)
|
||||||
|
assert [d.get("approach_id") for d in declared] == ["own-proposal"] # precondition
|
||||||
assert statuses["a1"] != "validated", "validated on a declaration the approach never made"
|
assert statuses["a1"] != "validated", "validated on a declaration the approach never made"
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue