feat(consume): give every excerpt the name and the address an answer must cite

The pre-pass delivered the right concept and the answer could not name it.
Measured by portfolio-optimiser 2026-09-08 over three paid arms: the gold
concept came back at rank 1 of 8 on 3 of 3 bundles, and the model answered
correctly on 1 of 3, because a delivered excerpt carried `concept_id`, body
text and nothing the document is known by. The previous session measured the
same gap from the other side: the provenance it had just written into every
concept did not reach the payload at all.

`excerpt_for` now carries `title` unconditionally, and `req_number`, the SPEC
5.1 address `sources` and each locator key (`source_pages`, `source_sheet`,
`source_rows`, `source_lines`, `source_offset`) when the concept has them. A key
the producer did not write stays absent: an empty value would assert that they
wrote an empty one, which is the contract's 6.4 failure.

`sources` is read in BOTH YAML forms, on a measurement rather than a taste. K2
writes the flow form on 629 of 629 concepts; the largest N-bundle writes the
block form on 270 of 270 and carries no locator key at all, so a flow-only
reader delivers that bundle with no address whatsoever. Reading the block form
is not a licence to write it - the emission rule is untouched, because the
line-oriented parser still cannot round-trip a block list. A `sources` value
this reader cannot decode is named (`sources_unreadable`), never dropped into
the same silence as an absent one.

Contract 8 gains the requirement and the checker gains its code
(`excerpt_unnamed`, 15 rules now, was 14): an excerpt a reader cannot name is
one an answer cannot cite, whatever its rank. `req_number`, `sources` and the
locators are SHOULD, not MUST - they are conditional on the producer, and a
bundle whose concepts carry no identifier cannot deliver one.

K2 controls, same question and same k, before against a frozen copy of the tool
at b6a8c8b: the RANKING does not move - the same 8 concept ids in the same
order, identical `text_sha256`, identical `withheld`, identical denominators
(629 = 621 + 8). The FIELD is what moved: payload 108 877 -> 111 744 B
(+2.63 %), budget spent 18 606 -> 20 907 (+287.6 B per excerpt), excerpt
members 9 -> 15, 83 changed lines. The contract document's own bytes moved with
8, so the budget instrument's known-positive moves with it: 10 349 -> 12 049
measured, 10 060 -> 11 719 raw, delta 289 -> 330.

New fixture `tests/fixtures/consume-provenance`: the two address forms and a
concept carrying neither address nor identifier. Purpose-built, because the two
real bundles are complementary and neither exercises both forms.

Suite 1347 (1339 before), ruff clean, mypy src clean.

Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 15:01:41 +02:00
commit 17c49fc04b
14 changed files with 454 additions and 18 deletions

View file

@ -0,0 +1,15 @@
---
type: Krav
title: Krav 10.2-2 Beredskap
req_number: Krav 10.2-2
ingested_at: 2026-09-08T00:00:00Z
adjudication: proposed
bundle_id: provenance-fixture
sources:
- resource: https://example.invalid/api/nisosts/1?languageCode=nb
title: Fiktiv normal 2024
---
## Krav
Beredskapsplanen skal oppdateres hvert aar.

View file

@ -0,0 +1,14 @@
---
type: reference
title: Sideoppslag i regelverket
ingested_at: 2026-09-08T00:00:00Z
adjudication: proposed
bundle_id: provenance-fixture
sources: [{ resource: Fiktivt grunnlag.pdf, title: Fiktivt grunnlag.pdf }]
source_pages: [4, 6]
source_offset: [120, 480]
---
## Sideoppslag
Teksten staar paa sidene fire til seks i det fiktive grunnlaget.

View file

@ -0,0 +1,8 @@
---
okf_version: 0.2
bundle_id: provenance-fixture
---
- [Krav 10.2-2 Beredskap](blokkform-krav.md)
- [Sideoppslag i regelverket](flytform-side.md)
- [Notat uten adresse](uten-adresse.md)

View file

@ -0,0 +1,11 @@
---
type: reference
title: Notat uten adresse
ingested_at: 2026-09-08T00:00:00Z
adjudication: proposed
bundle_id: provenance-fixture
---
## Notat
Dette konseptet har verken sources, kravnummer eller lokator.

View file

@ -29,6 +29,7 @@ import socket
import subprocess
import sys
import unicodedata
from collections.abc import Mapping
from pathlib import Path
import pytest
@ -1615,6 +1616,10 @@ def _synthetic(concept_id: str, title: str, body: str) -> okf_consume.Concept:
source_file="synthetic.md",
adjudication="unknown",
adjudication_present=False,
req_number="",
sources=(),
sources_present=False,
locators={},
frontmatter={},
body=body,
)
@ -2090,3 +2095,136 @@ def test_the_lookup_keeps_the_fused_order_among_the_concepts_it_lifts() -> None:
assert [concept.concept_id for concept, _, _ in ranked[len(hits) :]] == sorted(
concept.concept_id for concept in concepts if concept.concept_id not in hits
)
# --- Step 13: the excerpt carries the key the question is asked ON (C1 step 0) -
#: A bundle purpose-built for one measurement: the two SS 5.1 `sources` forms
#: and a concept carrying neither address nor locator. It exists because the
#: two real bundles are complementary and neither exercises both forms --
#: measured 2026-09-08, K2 writes `sources` in FLOW form on 629 of 629 concepts
#: and carries `req_number` on 0, while the N500 bundle writes BLOCK form on 270
#: of 270, carries `req_number` on 269 and carries no locator key at all.
PROVENANCE = PROJECT_ROOT / "tests" / "fixtures" / "consume-provenance"
PROVENANCE_QUESTION = "Hva krever Krav 10.2-2?"
def _provenance_excerpts() -> dict[str, Mapping[str, object]]:
# Every concept, not only the delivered ones: the cut is a different
# question and would make this measurement depend on a ranking.
excerpts: dict[str, Mapping[str, object]] = {}
for concept_id in okf_consume.enumerate_concepts(PROVENANCE):
excerpt = okf_consume.excerpt_for(
okf_consume.read_concept(
PROVENANCE / f"{concept_id}.md",
bundle_root=PROVENANCE,
root_bundle_id="provenance-fixture",
)
)
assert excerpt is not None
excerpts[concept_id] = excerpt
return excerpts
def test_every_delivered_excerpt_names_the_concept_the_answer_would_cite() -> None:
# po measured 2026-09-08 (`e7ffe9e`) that the pre-pass delivers the gold
# concept at rank 1 and the model still cannot name what it is citing: the
# excerpt carried `concept_id` and `text` and no `title`. The ranking found
# the document; the delivery dropped the key.
payload = okf_consume.build_payload(PROVENANCE, question=PROVENANCE_QUESTION)
delivered = payload["excerpts"]
assert isinstance(delivered, list)
assert delivered, "the cut delivered nothing, so the assertion below is vacuous"
for excerpt in delivered:
concept = okf_consume.read_concept(
PROVENANCE / f"{excerpt['concept_id']}.md", # type: ignore[index]
bundle_root=PROVENANCE,
root_bundle_id="provenance-fixture",
)
assert excerpt["title"] == concept.title # type: ignore[index]
assert excerpt["title"] # type: ignore[index]
def test_an_excerpt_carries_the_identifier_the_question_spells_and_never_invents_one() -> None:
excerpts = _provenance_excerpts()
assert excerpts["blokkform-krav"]["req_number"] == "Krav 10.2-2"
# A concept without the key gets no key. Emitting `""` would assert that the
# producer wrote an empty identifier, which is SS 6.4's failure.
assert "req_number" not in excerpts["flytform-side"]
assert "req_number" not in excerpts["uten-adresse"]
def test_a_flow_form_address_and_its_locator_reach_the_excerpt() -> None:
excerpt = _provenance_excerpts()["flytform-side"]
assert excerpt["sources"] == [
{"resource": "Fiktivt grunnlag.pdf", "title": "Fiktivt grunnlag.pdf"}
]
assert excerpt["source_pages"] == "[4, 6]"
assert excerpt["source_offset"] == "[120, 480]"
assert "source_lines" not in excerpt
def test_a_block_form_address_reaches_the_excerpt_rather_than_becoming_an_empty_string() -> None:
# `parse_frontmatter` SKIPS indented lines, so the block form arrives as
# `''` -- present and unreadable. Measured 2026-09-08: 270 of 270 concepts
# in the largest N-bundle write it that way, so a reader that only handles
# the flow form delivers that bundle with no address at all.
excerpt = _provenance_excerpts()["blokkform-krav"]
assert excerpt["sources"] == [
{
"resource": "https://example.invalid/api/nisosts/1?languageCode=nb",
"title": "Fiktiv normal 2024",
}
]
def test_a_concept_with_no_address_and_no_locator_carries_neither_key() -> None:
excerpt = _provenance_excerpts()["uten-adresse"]
for key in ("sources", "sources_unreadable", *okf_consume.LOCATOR_KEYS):
assert key not in excerpt, key
def test_an_unreadable_address_is_named_rather_than_dropped(tmp_path: Path) -> None:
# The third state. A `sources` value this reader cannot decode is neither an
# address nor an absence, and collapsing it into either reports a fact
# nobody measured.
root = tmp_path / "bundle"
_copy_bundle(PROVENANCE, root)
target = root / "uten-adresse.md"
target.write_text(
target.read_text(encoding="utf-8").replace(
"bundle_id: provenance-fixture",
"bundle_id: provenance-fixture\nsources: not-a-list",
),
encoding="utf-8",
)
concept = okf_consume.read_concept(
target, bundle_root=root, root_bundle_id="provenance-fixture"
)
excerpt = okf_consume.excerpt_for(concept)
assert excerpt is not None
assert excerpt["sources_unreadable"] is True
assert "sources" not in excerpt
def test_the_golden_bundle_excerpt_carries_the_address_and_locator_o3_wrote() -> None:
payload = okf_consume.build_payload(GOLDEN, question="Hva sier veiledningen om krav?")
excerpts = payload["excerpts"]
assert isinstance(excerpts, list)
by_id = {entry["concept_id"]: entry for entry in excerpts} # type: ignore[index,union-attr]
first = by_id["krav/1-1/foerste-krav"]
assert first["title"] == "Foerste krav"
assert first["sources"] == [{"resource": "veiledning.md", "title": "veiledning.md"}]
assert first["source_lines"] == "[5, 8]"
assert first["source_offset"] == "[94, 176]"
def test_the_checker_refuses_an_excerpt_that_cannot_be_named() -> None:
# The SS 8 update, with its own code: a payload whose excerpts carry no
# `title` is the defect po measured, and a checker that passes it certifies
# a payload a model cannot cite from.
payload = json.loads((SKILL.parent / "references" / "example-payload.json").read_text("utf-8"))
assert okf_contract_check.check(SKILL.read_text(encoding="utf-8"), payload).findings == ()
del payload["excerpts"][0]["title"]
codes = [f.code for f in okf_contract_check.check(SKILL.read_text("utf-8"), payload).findings]
assert codes == ["excerpt_unnamed"]