open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(consume): give every excerpt the name and the address an answer must cite

Browse source 
The pre-pass delivered the right concept and the answer could not name it.
Measured by portfolio-optimiser 2026-09-08 over three paid arms: the gold
concept came back at rank 1 of 8 on 3 of 3 bundles, and the model answered
correctly on 1 of 3, because a delivered excerpt carried `concept_id`, body
text and nothing the document is known by. The previous session measured the
same gap from the other side: the provenance it had just written into every
concept did not reach the payload at all.

`excerpt_for` now carries `title` unconditionally, and `req_number`, the SPEC
5.1 address `sources` and each locator key (`source_pages`, `source_sheet`,
`source_rows`, `source_lines`, `source_offset`) when the concept has them. A key
the producer did not write stays absent: an empty value would assert that they
wrote an empty one, which is the contract's 6.4 failure.

`sources` is read in BOTH YAML forms, on a measurement rather than a taste. K2
writes the flow form on 629 of 629 concepts; the largest N-bundle writes the
block form on 270 of 270 and carries no locator key at all, so a flow-only
reader delivers that bundle with no address whatsoever. Reading the block form
is not a licence to write it - the emission rule is untouched, because the
line-oriented parser still cannot round-trip a block list. A `sources` value
this reader cannot decode is named (`sources_unreadable`), never dropped into
the same silence as an absent one.

Contract 8 gains the requirement and the checker gains its code
(`excerpt_unnamed`, 15 rules now, was 14): an excerpt a reader cannot name is
one an answer cannot cite, whatever its rank. `req_number`, `sources` and the
locators are SHOULD, not MUST - they are conditional on the producer, and a
bundle whose concepts carry no identifier cannot deliver one.

K2 controls, same question and same k, before against a frozen copy of the tool
at b6a8c8b: the RANKING does not move - the same 8 concept ids in the same
order, identical `text_sha256`, identical `withheld`, identical denominators
(629 = 621 + 8). The FIELD is what moved: payload 108 877 -> 111 744 B
(+2.63 %), budget spent 18 606 -> 20 907 (+287.6 B per excerpt), excerpt
members 9 -> 15, 83 changed lines. The contract document's own bytes moved with
8, so the budget instrument's known-positive moves with it: 10 349 -> 12 049
measured, 10 060 -> 11 719 raw, delta 289 -> 330.

New fixture `tests/fixtures/consume-provenance`: the two address forms and a
concept carrying neither address nor identifier. Purpose-built, because the two
real bundles are complementary and neither exercises both forms.

Suite 1347 (1339 before), ruff clean, mypy src clean.

Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 15:01:41 +02:00
commit 17c49fc04b
14 changed files with 454 additions and 18 deletions
Download patch file Download diff file Expand all files Collapse all files

138
tools/okf_consume.py
View file

@ -262,6 +262,19 @@ class Concept:
#: `False` when the key was absent. `adjudication == "unknown"` already says
#: so, but a separate flag keeps the two facts from being one inference.
adjudication_present: bool
#: The identifier the producer wrote, or `""` when there is no key. The
#: number a lookup question is asked ON, and the one thing a reader needs to
#: name the concept the answer rests on.
req_number: str
#: The SS 5.1 address entries, in either YAML form.
sources: tuple[Mapping[str, str], ...]
#: `True` when a `sources` key was there, whatever this reader made of it.
#: With `sources == ()` that is the third state: present and unreadable.
sources_present: bool
#: The `LOCATOR_KEYS` this concept carries, values as written. Absent keys
#: are absent, never `""`: SS 6.4 forbids reading the absence of a
#: conditionally-written field as the negation of what it asserts.
locators: Mapping[str, str]
frontmatter: Mapping[str, str]
body: str
@ -301,6 +314,7 @@ def read_concept(path: Path, *, bundle_root: Path, root_bundle_id: str) -> Conce
code="adjudication_unknown_value",
)
declared = frontmatter.get("bundle_id")
entries, sources_present = read_sources(_frontmatter_lines(path))
return Concept(
path=path,
concept_id=concept_id,
@ -312,6 +326,12 @@ def read_concept(path: Path, *, bundle_root: Path, root_bundle_id: str) -> Conce
source_file=frontmatter.get("source_file", ""),
adjudication=adjudication,
adjudication_present=raw is not None,
req_number=frontmatter.get("req_number", ""),
sources=entries,
sources_present=sources_present,
locators={
key: frontmatter[key] for key in LOCATOR_KEYS if frontmatter.get(key, "").strip()
},
frontmatter=frontmatter,
body=_body(path),
)
@ -428,6 +448,93 @@ def _split_top_level(body: str, opener: str, closer: str) -> list[str]:
return [part for part in parts if part.strip()]
#: The locator keys O3 (`b6a8c8b`) writes on every SEGMENTED concept, in the
#: order they are emitted so a reader comparing an excerpt against the concept
#: file reads one sequence. The ADDRESS is SPEC SS 5.1's `sources`; these are
#: this library's OWN top-level keys, because SS 5.1 has no field for a place
#: within a resource and the guard rejects every route to putting one inside a
#: `sources` entry. Their VALUES pass through as the frontmatter's own strings:
#: `source_pages: [2, 27]` reaches the payload as `"[2, 27]"`, which is what
#: makes an excerpt greppable against the file it came from.
LOCATOR_KEYS = (
"source_pages",
"source_sheet",
"source_rows",
"source_lines",
"source_offset",
)
def _frontmatter_lines(path: Path) -> list[str]:
"""The raw lines between the two `---` fences, indentation intact.
`parse_frontmatter` SKIPS indented lines on purpose -- a nested `title:`
arriving later would SUBSTITUTE for the document's. That refusal is right
for a flat mapping and it is why the block form has to be read from the
raw lines instead.
"""
lines = path.read_text(encoding="utf-8").splitlines()
if not lines or lines[0].strip() != "---":
return []
block: list[str] = []
for line in lines[1:]:
if line.strip() == "---":
break
block.append(line)
return block
def read_sources(lines: Sequence[str]) -> tuple[tuple[Mapping[str, str], ...], bool]:
"""The SS 5.1 address entries, and whether the key was there at all.
Three states, kept apart because collapsing any two reports something
nobody measured: `((), False)` the concept has no `sources` key; `((), True)`
it has one this reader cannot decode; a non-empty tuple, the entries.
BOTH YAML forms are read, and that is a measurement rather than a
preference. Measured 2026-09-08: K2 writes the flow form on 629 of 629
concepts, the N500 bundle writes the block form on 270 of 270. A reader
handling one form delivers the other bundle with no address at all -- and
for N500 there is nothing else, because it carries zero locator keys.
Reading the block form is not a licence to WRITE it: this library's
line-oriented parser still cannot round-trip block lists, so the emission
rule (flow only) is untouched.
"""
for position, line in enumerate(lines):
if line[:1] in (" ", "\t") or not line.startswith("sources:"):
continue
value = line.partition(":")[2].strip()
if value:
flow = _parse_flow_mappings(value)
if flow is None:
return (), True
return tuple(flow), True
entries: list[dict[str, str]] = []
for nested in lines[position + 1 :]:
if not nested.strip():
continue
if nested[:1] not in (" ", "\t"):
break
item = nested.strip()
if item.startswith("- "):
entries.append({})
item = item[2:].strip()
elif not entries:
# An indented line before any `- ` opens no entry. Refused
# rather than folded into one, which would invent an entry the
# document does not have.
return (), True
key, separator, raw = item.partition(":")
if not separator:
return (), True
entries[-1][key.strip()] = raw.strip()
if not entries:
return (), True
return tuple(entries), True
return (), False
# --- The budget instrument (SS 7) --------------------------------------------
#: SS 7.5 fixes no unit deliberately -- "a token is one encoder family's unit
@ -464,14 +571,14 @@ KNOWN_POSITIVE_CASE = "docs/consumption-contract.md, encoded as a JSON string"
#: `measure()`'s own answer for that file. Vacuous ALONE -- which is why the
#: delta below exists.
KNOWN_POSITIVE_EXPECTED = 10_349
KNOWN_POSITIVE_EXPECTED = 12_049
#: The second, independent route. `wc -c` reports 10 060 raw bytes for the same
#: The second, independent route. `wc -c` reports 11 719 raw bytes for the same
#: file; the difference is this file's JSON quoting and escaping overhead. A
#: reader can derive it without running `measure()` at all, and it moves the
#: moment `measure()` changes what it counts -- which is what stops
#: `expected == measured` from proving nothing.
KNOWN_POSITIVE_ENCODING_DELTA = 289
KNOWN_POSITIVE_ENCODING_DELTA = 330
_KNOWN_POSITIVE_PATH = Path(__file__).resolve().parents[1] / "docs" / "consumption-contract.md"
@ -1027,6 +1134,14 @@ def excerpt_for(concept: Concept) -> dict[str, object] | None:
- **`bundle_id_inherited`** -- whether the first half of the SS 3.1 identity
tuple came from the concept or from the root index.
And, since C1 step 0, the keys that let a reader NAME what it is citing.
`title` is unconditional (SS 8, this revision); `req_number`, `sources` and
each locator are written only when the concept carries them, because a key
with an empty value asserts that the producer wrote one. po measured
2026-09-08 (`e7ffe9e`) that the pre-pass delivered the gold concept at rank
1 on three bundles while the model could not name it: the ranking found the
document and the delivery dropped the key.
Trailing whitespace is stripped per line: a spreadsheet render is padded to
hundreds of trailing spaces per line, and unstripped, most of a budget goes
on padding.
@ -1037,16 +1152,27 @@ def excerpt_for(concept: Concept) -> dict[str, object] | None:
text = "\n".join(
line.rstrip() for line in unicodedata.normalize("NFC", concept.body).split("\n")
)
return {
excerpt: dict[str, object] = {
"bundle_id": concept.bundle_id,
"concept_id": concept.concept_id,
"sha256": concept.sha256,
"adjudication": concept.adjudication,
"trust_tier": tier,
"bundle_id_inherited": concept.bundle_id_inherited,
"text_sha256": hashlib.sha256(text.encode("utf-8")).hexdigest(),
"text": text,
"title": concept.title,
}
if concept.req_number:
excerpt["req_number"] = concept.req_number
if concept.sources:
excerpt["sources"] = [dict(entry) for entry in concept.sources]
elif concept.sources_present:
# The third state, written rather than silently dropped: the concept
# HAS an address and this reader could not decode it.
excerpt["sources_unreadable"] = True
excerpt.update(concept.locators)
excerpt["text_sha256"] = hashlib.sha256(text.encode("utf-8")).hexdigest()
excerpt["text"] = text
return excerpt
def excerpt_weight(excerpt: Mapping[str, object]) -> int:

24
tools/okf_contract_check.py
View file

@ -187,6 +187,29 @@ def rule_excerpt_source_marking(ctx: Context) -> list[Finding]:
return findings
def rule_excerpt_named(ctx: Context) -> list[Finding]:
"""SS 8: every excerpt carries a `title`.
Added 2026-09-08 on a measurement, not a preference: `portfolio-optimiser`
ran three paid arms in which the pre-pass delivered the gold concept at rank
1 of 8 on 3 of 3 bundles and the model answered correctly on 1 of 3, because
the excerpt carried `concept_id` and `text` and nothing a reader could name
the document by. A payload no answer can cite from is not conformant; the
identity fields are what SS 3.1's tuple is FOR.
"""
if not ctx.payload_is_mapping:
return []
return [
Finding(
"excerpt_unnamed",
f"excerpt {position} carries no 'title'; an excerpt a reader cannot "
"name is one an answer cannot cite, whatever its rank (SS 8)",
)
for position, raw in enumerate(_sequence(ctx.payload.get("excerpts")))
if not _text(_mapping(raw).get("title"))
]
def rule_excerpt_states(ctx: Context) -> list[Finding]:
if not ctx.payload_is_mapping:
return []
@ -407,6 +430,7 @@ RULES: tuple[Callable[[Context], list[Finding]], ...] = (
rule_contract_version,
rule_bundle_ref,
rule_excerpt_source_marking,
rule_excerpt_named,
rule_excerpt_states,
rule_denominator_identity,
rule_denominator_lists,