test(major2): hver genereringstelling paret med debattens telling (SC1/SC4)

Funn 5935d942 (MISSING_TEST, tests/...:820). Briefens SC1 gjorde paringen til
vakten mot aa telle en debatt-tur som en genereringsprompt - og ingen arm i
fila asserterte debattens turantall mellom treated og control (grep «debate»
traff bare en kommentar og et filnavn).

`_debate_entries` er KOMPLEMENTET av `_GENERATION_MARK`, ikke en positiv
debatt-markoer: paastanden som gates er nettopp «ingenting som IKKE er
generering flyttet seg», og en positiv markoer ville latt en tur klassifisereren
ikke kjenner drive usett.

- T5-run: de tre genereringstellingene paret med likhet paa debatt-oppfoeringene.
- T8: fikk sink + en control-kjoering (alltid-godkjenn reviewer) - dens
  attempt-indekser [0,1] per kandidat ER en genereringstelling.
- Begge har en VAKUITETSVAKT (debatt-lista maa vaere ikke-tom): to tomme lister
  er like gratis.
- T13: record-indeksene er DOKUMENTERT som SC4s telle-proxy ved den doera -
  barnet kjoerer i egen interpreter og `--scripted-replies` har ingen
  sink-dump, saa prompt-nivaaet maales in-process (T1 for verbatim, T5-run/T8
  for paringen). Reviewens andre alternativ.

MAALT mot HELE suiten, to mutasjoner, begge roede paa NOEYAKTIG de to parede
armene og paa ingen andre: MA klassifisereren returnerer konstant tom liste
(2 roede - vakuitetsvakten) og MB filteret droppet, saa generering telles som
debatt (2 roede). Kontroll 1364 passed / 5 skipped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-05 22:29:27 +02:00
commit d4a0220b68

View file

@ -657,6 +657,19 @@ _VERDICT_INPUT = {"decision": "approved", "rationale": "expert reviewed (sim)"}
#: else — the one identifier that separates a generation call from a debate turn.
_GENERATION_MARK = "Respond with ONLY a JSON object"
def _debate_entries(sink: list[str]) -> list[str]:
"""Everything in the sink that is NOT a generation prompt — i.e. the debate's turns.
The complement of ``_GENERATION_MARK`` rather than a positive debate marker, because the claim
being guarded is exactly "nothing that is not generation moved": a positive marker would let a
turn the classifier does not know about drift unseen. Every "N generation prompts" assertion in
this file is paired with an equality of THIS list between the treated and the control run
(SC1/SC4) without the pairing, an arm that counted a debate turn as a generation prompt, or a
door that perturbed the debate, would read as a correct generation count."""
return [blob for blob in sink if _GENERATION_MARK not in blob]
# BYGG-KONTOR-NORD: affected total 300000 x 1.0 -> degenerate Monte Carlo P90 = 90000. A claim
# <= 90000 validates; above it the deterministic validator rejects.
_A1 = Approach(id="led-retrofit", label="Behovsstyrt belysning i fellesarealer")
@ -755,18 +768,35 @@ async def test_t8_every_candidate_gets_its_own_keyed_answer(tmp_path: Path) -> N
With a commissioned mandate one expert is asked N times at one terminal, so an artefact that
cannot say WHICH candidate a sentence was about is an artefact nobody can act on. The run's
own proposal is keyed ``OWN_PROPOSAL_ID`` recording ``None`` there would make it
indistinguishable from a non-mandate run's entry, which is the property keying exists for."""
indistinguishable from a non-mandate run's entry, which is the property keying exists for.
The per-candidate attempt indices below are a GENERATION count (two attempts each), so SC1
requires them paired with the debate's count: a control run whose reviewer always approves must
leave the non-generation entries identical."""
mandate = Mandate(
objective="Cut energy cost without rebuilding.",
approaches=(_A1, _A2),
allow_own_proposals=True,
)
reviewer = _RunReviewer()
treated_sink: list[str] = []
result = await _run_with(
outbox_dir=tmp_path / "outbox",
reviewer=reviewer,
mandate=Mandate(
objective="Cut energy cost without rebuilding.",
approaches=(_A1, _A2),
allow_own_proposals=True,
),
mandate=mandate,
sink=treated_sink,
)
assert isinstance(result, RunResult)
control_sink: list[str] = []
control = await _run_with(
outbox_dir=tmp_path / "control",
run_id="run-control",
reviewer=_RunReviewer(revise_for=set()),
mandate=mandate,
sink=control_sink,
)
assert isinstance(result, RunResult) and isinstance(control, RunResult)
assert _debate_entries(treated_sink), "no debate turns reached the sink: the pairing is vacuous"
assert _debate_entries(treated_sink) == _debate_entries(control_sink)
revisions = {
(r.approach_id, r.feedback) for r in result.expert_revisions if r.decision == "revise"
}
@ -824,6 +854,10 @@ async def test_t5run_a_revise_on_one_approach_leaves_the_next_untouched(tmp_path
assert len(_gen_prompts(control_sink, _A2.label)) == 1
assert _gen_prompts(treated_sink, _A3.label) == _gen_prompts(control_sink, _A3.label)
assert all(_FEEDBACK_SENTINEL not in b for b in _gen_prompts(treated_sink, _A3.label))
# SC1's pairing: the counts above mean "one more GENERATION prompt" only if the debate did not
# move. The non-empty check is the vacuity guard — two empty lists are equal for free.
assert _debate_entries(treated_sink), "no debate turns reached the sink: the pairing is vacuous"
assert _debate_entries(treated_sink) == _debate_entries(control_sink)
async def test_t9_a_budget_stop_inside_generation_still_leaves_the_record(tmp_path: Path) -> None:
@ -1129,7 +1163,16 @@ def test_t13_the_flag_answers_the_review_from_a_real_argv_and_the_answer_is_used
(The plan's separate T19 — "a bare, unscripted flag parse" — is folded in here: without a
script that argv would call a live model, so the argparse witness is this child's own
``unrecognized arguments`` assertion.)"""
``unrecognized arguments`` assertion.)
**The attempt indices are SC4's count PROXY at this door, and nothing more — stated rather
than left to be read as a prompt-level count.** The child runs in its own interpreter, so the
``ScriptedChatClient`` sink lives in that process and there is no CLI flag that dumps it: the
prompt-level half of SC4 (the feedback VERBATIM in the second generation prompt, and the
debate's turn count unchanged against a control) is measured in-process, where the sink is
reachable T1 for the verbatim injection, T5-run and T8 for the debate pairing. What this arm
settles that none of those can is that a real argv, a real pipe and a real ``run_project``
bought a second attempt whose OUTCOME differs (40 000, entry 2 of the step list)."""
proc = _child(_cli_argv(tmp_path, outbox="outbox"), stdin=f"revise {_CLI_FEEDBACK}\napprove\n")
assert "unrecognized arguments" not in proc.stderr, proc.stderr
@ -1142,7 +1185,9 @@ def test_t13_the_flag_answers_the_review_from_a_real_argv_and_the_answer_is_used
("approve", True),
]
assert payload["reviews"][0]["feedback"] == _CLI_FEEDBACK # VERBATIM
assert [r["attempt"] for r in payload["reviews"]] == [0, 1] # exactly two generation attempts
# SC4's count proxy at this door (see the docstring): two attempt indices, not two counted
# prompts — the child's sink is in the child.
assert [r["attempt"] for r in payload["reviews"]] == [0, 1]
outcome = json.loads(
(tmp_path / "outbox" / f"{_RUN_ID}-proposal.json").read_text(encoding="utf-8")