feat(consume): give every excerpt the name and the address an answer must cite

The pre-pass delivered the right concept and the answer could not name it.
Measured by portfolio-optimiser 2026-09-08 over three paid arms: the gold
concept came back at rank 1 of 8 on 3 of 3 bundles, and the model answered
correctly on 1 of 3, because a delivered excerpt carried `concept_id`, body
text and nothing the document is known by. The previous session measured the
same gap from the other side: the provenance it had just written into every
concept did not reach the payload at all.

`excerpt_for` now carries `title` unconditionally, and `req_number`, the SPEC
5.1 address `sources` and each locator key (`source_pages`, `source_sheet`,
`source_rows`, `source_lines`, `source_offset`) when the concept has them. A key
the producer did not write stays absent: an empty value would assert that they
wrote an empty one, which is the contract's 6.4 failure.

`sources` is read in BOTH YAML forms, on a measurement rather than a taste. K2
writes the flow form on 629 of 629 concepts; the largest N-bundle writes the
block form on 270 of 270 and carries no locator key at all, so a flow-only
reader delivers that bundle with no address whatsoever. Reading the block form
is not a licence to write it - the emission rule is untouched, because the
line-oriented parser still cannot round-trip a block list. A `sources` value
this reader cannot decode is named (`sources_unreadable`), never dropped into
the same silence as an absent one.

Contract 8 gains the requirement and the checker gains its code
(`excerpt_unnamed`, 15 rules now, was 14): an excerpt a reader cannot name is
one an answer cannot cite, whatever its rank. `req_number`, `sources` and the
locators are SHOULD, not MUST - they are conditional on the producer, and a
bundle whose concepts carry no identifier cannot deliver one.

K2 controls, same question and same k, before against a frozen copy of the tool
at b6a8c8b: the RANKING does not move - the same 8 concept ids in the same
order, identical `text_sha256`, identical `withheld`, identical denominators
(629 = 621 + 8). The FIELD is what moved: payload 108 877 -> 111 744 B
(+2.63 %), budget spent 18 606 -> 20 907 (+287.6 B per excerpt), excerpt
members 9 -> 15, 83 changed lines. The contract document's own bytes moved with
8, so the budget instrument's known-positive moves with it: 10 349 -> 12 049
measured, 10 060 -> 11 719 raw, delta 289 -> 330.

New fixture `tests/fixtures/consume-provenance`: the two address forms and a
concept carrying neither address nor identifier. Purpose-built, because the two
real bundles are complementary and neither exercises both forms.

Suite 1347 (1339 before), ruff clean, mypy src clean.

Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 15:01:41 +02:00
commit 17c49fc04b
14 changed files with 454 additions and 18 deletions

View file

@ -157,7 +157,11 @@ are permitted and are not read by the checker.
"concept_id": "<bundle-local id>",
"sha256": "<hex digest of the concept file>",
"adjudication": "proposed",
"trust_tier": "machine-confirmed"
"trust_tier": "machine-confirmed",
"title": "<the concept's own title>",
"req_number": "<the identifier the producer wrote, when there is one>",
"sources": [ { "resource": "<what the concept came from>" } ],
"source_pages": "<a locator into that resource, when the producer wrote one>"
}
],
"withheld": [ { "concept_id": "<bundle-local id>", "rule": "<why it was cut>" } ]
@ -168,6 +172,25 @@ are permitted and are not read by the checker.
equal `denominators.withheld`. The counts and the lists are two statements of
the same fact, and a payload where they disagree is refused.
2. `contract` MUST be present so a reader can tell which revision it is holding.
3. Every excerpt MUST carry a non-empty `title`. An excerpt a reader cannot name
is one an answer cannot cite, whatever its rank. This is a measurement, not a
preference: in three paid arms run 2026-09-08 the pre-pass delivered the gold
concept at rank 1 of 8 on 3 of 3 bundles and the model answered correctly on
1 of 3, because the delivered unit carried `concept_id` and body text and
nothing the document is known by. § 3.1's tuple is an identity for a
*machine*; a title is what a citation is made of.
4. An excerpt SHOULD carry the identifier the producer wrote (`req_number`), the
§ 5.1 address of what it came from (`sources`), and the locator into that
address the producer wrote. These are SHOULD and not MUST because they are
conditional on the producer: a bundle whose concepts carry no identifier
cannot deliver one, and § 6.4 forbids reading their absence as a claim that
the source has none. A payload MUST NOT write an empty value in place of an
absent one, and MUST distinguish an address it could not read from one that
is not there.
5. The locator keys are the *profile's*, not this contract's: SPEC § 5.1 has no
field for a place within a resource, so where a page number, a sheet row or a
line range is named — and under which key — is stated by the profile and read
from the concept, never invented here.
## 9. Prohibitions