feat(consume): give every excerpt the name and the address an answer must cite
The pre-pass delivered the right concept and the answer could not name it.
Measured by portfolio-optimiser 2026-09-08 over three paid arms: the gold
concept came back at rank 1 of 8 on 3 of 3 bundles, and the model answered
correctly on 1 of 3, because a delivered excerpt carried `concept_id`, body
text and nothing the document is known by. The previous session measured the
same gap from the other side: the provenance it had just written into every
concept did not reach the payload at all.
`excerpt_for` now carries `title` unconditionally, and `req_number`, the SPEC
5.1 address `sources` and each locator key (`source_pages`, `source_sheet`,
`source_rows`, `source_lines`, `source_offset`) when the concept has them. A key
the producer did not write stays absent: an empty value would assert that they
wrote an empty one, which is the contract's 6.4 failure.
`sources` is read in BOTH YAML forms, on a measurement rather than a taste. K2
writes the flow form on 629 of 629 concepts; the largest N-bundle writes the
block form on 270 of 270 and carries no locator key at all, so a flow-only
reader delivers that bundle with no address whatsoever. Reading the block form
is not a licence to write it - the emission rule is untouched, because the
line-oriented parser still cannot round-trip a block list. A `sources` value
this reader cannot decode is named (`sources_unreadable`), never dropped into
the same silence as an absent one.
Contract 8 gains the requirement and the checker gains its code
(`excerpt_unnamed`, 15 rules now, was 14): an excerpt a reader cannot name is
one an answer cannot cite, whatever its rank. `req_number`, `sources` and the
locators are SHOULD, not MUST - they are conditional on the producer, and a
bundle whose concepts carry no identifier cannot deliver one.
K2 controls, same question and same k, before against a frozen copy of the tool
at b6a8c8b: the RANKING does not move - the same 8 concept ids in the same
order, identical `text_sha256`, identical `withheld`, identical denominators
(629 = 621 + 8). The FIELD is what moved: payload 108 877 -> 111 744 B
(+2.63 %), budget spent 18 606 -> 20 907 (+287.6 B per excerpt), excerpt
members 9 -> 15, 83 changed lines. The contract document's own bytes moved with
8, so the budget instrument's known-positive moves with it: 10 349 -> 12 049
measured, 10 060 -> 11 719 raw, delta 289 -> 330.
New fixture `tests/fixtures/consume-provenance`: the two address forms and a
concept carrying neither address nor identifier. Purpose-built, because the two
real bundles are complementary and neither exercises both forms.
Suite 1347 (1339 before), ruff clean, mypy src clean.
Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
parent
b6a8c8bd89
commit
17c49fc04b
14 changed files with 454 additions and 18 deletions
15
tests/fixtures/consume-provenance/blokkform-krav.md
vendored
Normal file
15
tests/fixtures/consume-provenance/blokkform-krav.md
vendored
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
---
|
||||
type: Krav
|
||||
title: Krav 10.2-2 Beredskap
|
||||
req_number: Krav 10.2-2
|
||||
ingested_at: 2026-09-08T00:00:00Z
|
||||
adjudication: proposed
|
||||
bundle_id: provenance-fixture
|
||||
sources:
|
||||
- resource: https://example.invalid/api/nisosts/1?languageCode=nb
|
||||
title: Fiktiv normal 2024
|
||||
---
|
||||
|
||||
## Krav
|
||||
|
||||
Beredskapsplanen skal oppdateres hvert aar.
|
||||
14
tests/fixtures/consume-provenance/flytform-side.md
vendored
Normal file
14
tests/fixtures/consume-provenance/flytform-side.md
vendored
Normal file
|
|
@ -0,0 +1,14 @@
|
|||
---
|
||||
type: reference
|
||||
title: Sideoppslag i regelverket
|
||||
ingested_at: 2026-09-08T00:00:00Z
|
||||
adjudication: proposed
|
||||
bundle_id: provenance-fixture
|
||||
sources: [{ resource: Fiktivt grunnlag.pdf, title: Fiktivt grunnlag.pdf }]
|
||||
source_pages: [4, 6]
|
||||
source_offset: [120, 480]
|
||||
---
|
||||
|
||||
## Sideoppslag
|
||||
|
||||
Teksten staar paa sidene fire til seks i det fiktive grunnlaget.
|
||||
8
tests/fixtures/consume-provenance/index.md
vendored
Normal file
8
tests/fixtures/consume-provenance/index.md
vendored
Normal file
|
|
@ -0,0 +1,8 @@
|
|||
---
|
||||
okf_version: 0.2
|
||||
bundle_id: provenance-fixture
|
||||
---
|
||||
|
||||
- [Krav 10.2-2 Beredskap](blokkform-krav.md)
|
||||
- [Sideoppslag i regelverket](flytform-side.md)
|
||||
- [Notat uten adresse](uten-adresse.md)
|
||||
11
tests/fixtures/consume-provenance/uten-adresse.md
vendored
Normal file
11
tests/fixtures/consume-provenance/uten-adresse.md
vendored
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
---
|
||||
type: reference
|
||||
title: Notat uten adresse
|
||||
ingested_at: 2026-09-08T00:00:00Z
|
||||
adjudication: proposed
|
||||
bundle_id: provenance-fixture
|
||||
---
|
||||
|
||||
## Notat
|
||||
|
||||
Dette konseptet har verken sources, kravnummer eller lokator.
|
||||
|
|
@ -29,6 +29,7 @@ import socket
|
|||
import subprocess
|
||||
import sys
|
||||
import unicodedata
|
||||
from collections.abc import Mapping
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
|
@ -1615,6 +1616,10 @@ def _synthetic(concept_id: str, title: str, body: str) -> okf_consume.Concept:
|
|||
source_file="synthetic.md",
|
||||
adjudication="unknown",
|
||||
adjudication_present=False,
|
||||
req_number="",
|
||||
sources=(),
|
||||
sources_present=False,
|
||||
locators={},
|
||||
frontmatter={},
|
||||
body=body,
|
||||
)
|
||||
|
|
@ -2090,3 +2095,136 @@ def test_the_lookup_keeps_the_fused_order_among_the_concepts_it_lifts() -> None:
|
|||
assert [concept.concept_id for concept, _, _ in ranked[len(hits) :]] == sorted(
|
||||
concept.concept_id for concept in concepts if concept.concept_id not in hits
|
||||
)
|
||||
|
||||
|
||||
# --- Step 13: the excerpt carries the key the question is asked ON (C1 step 0) -
|
||||
|
||||
#: A bundle purpose-built for one measurement: the two SS 5.1 `sources` forms
|
||||
#: and a concept carrying neither address nor locator. It exists because the
|
||||
#: two real bundles are complementary and neither exercises both forms --
|
||||
#: measured 2026-09-08, K2 writes `sources` in FLOW form on 629 of 629 concepts
|
||||
#: and carries `req_number` on 0, while the N500 bundle writes BLOCK form on 270
|
||||
#: of 270, carries `req_number` on 269 and carries no locator key at all.
|
||||
PROVENANCE = PROJECT_ROOT / "tests" / "fixtures" / "consume-provenance"
|
||||
PROVENANCE_QUESTION = "Hva krever Krav 10.2-2?"
|
||||
|
||||
|
||||
def _provenance_excerpts() -> dict[str, Mapping[str, object]]:
|
||||
# Every concept, not only the delivered ones: the cut is a different
|
||||
# question and would make this measurement depend on a ranking.
|
||||
excerpts: dict[str, Mapping[str, object]] = {}
|
||||
for concept_id in okf_consume.enumerate_concepts(PROVENANCE):
|
||||
excerpt = okf_consume.excerpt_for(
|
||||
okf_consume.read_concept(
|
||||
PROVENANCE / f"{concept_id}.md",
|
||||
bundle_root=PROVENANCE,
|
||||
root_bundle_id="provenance-fixture",
|
||||
)
|
||||
)
|
||||
assert excerpt is not None
|
||||
excerpts[concept_id] = excerpt
|
||||
return excerpts
|
||||
|
||||
|
||||
def test_every_delivered_excerpt_names_the_concept_the_answer_would_cite() -> None:
|
||||
# po measured 2026-09-08 (`e7ffe9e`) that the pre-pass delivers the gold
|
||||
# concept at rank 1 and the model still cannot name what it is citing: the
|
||||
# excerpt carried `concept_id` and `text` and no `title`. The ranking found
|
||||
# the document; the delivery dropped the key.
|
||||
payload = okf_consume.build_payload(PROVENANCE, question=PROVENANCE_QUESTION)
|
||||
delivered = payload["excerpts"]
|
||||
assert isinstance(delivered, list)
|
||||
assert delivered, "the cut delivered nothing, so the assertion below is vacuous"
|
||||
for excerpt in delivered:
|
||||
concept = okf_consume.read_concept(
|
||||
PROVENANCE / f"{excerpt['concept_id']}.md", # type: ignore[index]
|
||||
bundle_root=PROVENANCE,
|
||||
root_bundle_id="provenance-fixture",
|
||||
)
|
||||
assert excerpt["title"] == concept.title # type: ignore[index]
|
||||
assert excerpt["title"] # type: ignore[index]
|
||||
|
||||
|
||||
def test_an_excerpt_carries_the_identifier_the_question_spells_and_never_invents_one() -> None:
|
||||
excerpts = _provenance_excerpts()
|
||||
assert excerpts["blokkform-krav"]["req_number"] == "Krav 10.2-2"
|
||||
# A concept without the key gets no key. Emitting `""` would assert that the
|
||||
# producer wrote an empty identifier, which is SS 6.4's failure.
|
||||
assert "req_number" not in excerpts["flytform-side"]
|
||||
assert "req_number" not in excerpts["uten-adresse"]
|
||||
|
||||
|
||||
def test_a_flow_form_address_and_its_locator_reach_the_excerpt() -> None:
|
||||
excerpt = _provenance_excerpts()["flytform-side"]
|
||||
assert excerpt["sources"] == [
|
||||
{"resource": "Fiktivt grunnlag.pdf", "title": "Fiktivt grunnlag.pdf"}
|
||||
]
|
||||
assert excerpt["source_pages"] == "[4, 6]"
|
||||
assert excerpt["source_offset"] == "[120, 480]"
|
||||
assert "source_lines" not in excerpt
|
||||
|
||||
|
||||
def test_a_block_form_address_reaches_the_excerpt_rather_than_becoming_an_empty_string() -> None:
|
||||
# `parse_frontmatter` SKIPS indented lines, so the block form arrives as
|
||||
# `''` -- present and unreadable. Measured 2026-09-08: 270 of 270 concepts
|
||||
# in the largest N-bundle write it that way, so a reader that only handles
|
||||
# the flow form delivers that bundle with no address at all.
|
||||
excerpt = _provenance_excerpts()["blokkform-krav"]
|
||||
assert excerpt["sources"] == [
|
||||
{
|
||||
"resource": "https://example.invalid/api/nisosts/1?languageCode=nb",
|
||||
"title": "Fiktiv normal 2024",
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
def test_a_concept_with_no_address_and_no_locator_carries_neither_key() -> None:
|
||||
excerpt = _provenance_excerpts()["uten-adresse"]
|
||||
for key in ("sources", "sources_unreadable", *okf_consume.LOCATOR_KEYS):
|
||||
assert key not in excerpt, key
|
||||
|
||||
|
||||
def test_an_unreadable_address_is_named_rather_than_dropped(tmp_path: Path) -> None:
|
||||
# The third state. A `sources` value this reader cannot decode is neither an
|
||||
# address nor an absence, and collapsing it into either reports a fact
|
||||
# nobody measured.
|
||||
root = tmp_path / "bundle"
|
||||
_copy_bundle(PROVENANCE, root)
|
||||
target = root / "uten-adresse.md"
|
||||
target.write_text(
|
||||
target.read_text(encoding="utf-8").replace(
|
||||
"bundle_id: provenance-fixture",
|
||||
"bundle_id: provenance-fixture\nsources: not-a-list",
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
concept = okf_consume.read_concept(
|
||||
target, bundle_root=root, root_bundle_id="provenance-fixture"
|
||||
)
|
||||
excerpt = okf_consume.excerpt_for(concept)
|
||||
assert excerpt is not None
|
||||
assert excerpt["sources_unreadable"] is True
|
||||
assert "sources" not in excerpt
|
||||
|
||||
|
||||
def test_the_golden_bundle_excerpt_carries_the_address_and_locator_o3_wrote() -> None:
|
||||
payload = okf_consume.build_payload(GOLDEN, question="Hva sier veiledningen om krav?")
|
||||
excerpts = payload["excerpts"]
|
||||
assert isinstance(excerpts, list)
|
||||
by_id = {entry["concept_id"]: entry for entry in excerpts} # type: ignore[index,union-attr]
|
||||
first = by_id["krav/1-1/foerste-krav"]
|
||||
assert first["title"] == "Foerste krav"
|
||||
assert first["sources"] == [{"resource": "veiledning.md", "title": "veiledning.md"}]
|
||||
assert first["source_lines"] == "[5, 8]"
|
||||
assert first["source_offset"] == "[94, 176]"
|
||||
|
||||
|
||||
def test_the_checker_refuses_an_excerpt_that_cannot_be_named() -> None:
|
||||
# The SS 8 update, with its own code: a payload whose excerpts carry no
|
||||
# `title` is the defect po measured, and a checker that passes it certifies
|
||||
# a payload a model cannot cite from.
|
||||
payload = json.loads((SKILL.parent / "references" / "example-payload.json").read_text("utf-8"))
|
||||
assert okf_contract_check.check(SKILL.read_text(encoding="utf-8"), payload).findings == ()
|
||||
del payload["excerpts"][0]["title"]
|
||||
codes = [f.code for f in okf_contract_check.check(SKILL.read_text("utf-8"), payload).findings]
|
||||
assert codes == ["excerpt_unnamed"]
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue