open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(check): a skill and a payload naming different bundles is a finding

Browse source 
`okf check` had fifteen rules and none asked whether the skill and the payload
were talking about the same bundle. Reproduced on this HEAD before any code
moved: three pairs reported `conformant: 15 rules over 8 excerpts and 438
withheld entries, 0 findings` -- a skill generated from one corpus against
another corpus's payload, the unfilled template against that payload, and a
payload sharing the skill's `bundle_id` at a foreign `ref`. All three now exit 1
with one `bundle_mismatch` finding over 16 rules.

BOTH halves are compared and the `ref` half is load-bearing: three distinct
builds on this machine carry one `bundle_id`, so an id comparison would pass a
stale skill. SS 3.3: "a version is the producer's assertion; a ref is a fact
about bytes". An identity the rule cannot read is a finding, never a silent
pass -- that is what refuses the unfilled template.

No new field: the identity was already in the generated skill's prose, now
factored into `skill.identity_line` and read back by
`contract_check.skill_identity`. Generated skill bytes unchanged, measured on
both tracked bundles on one interpreter.

The rule's first real find is this repository's own hand-made
`skills/okf-consume/SKILL.md`, which predates `okf skill` and declares no
identity a reader can act on: 1 of 1. Nine tests that asserted the old, false
conformance now pair a skill with its own bundle's payload.

Measured, nothing else moved: `~/okf-test/dokumenter` `diff -r` empty old
source vs new on one interpreter (52 files, 26 concepts), `okf project` still
byte-equal to `okf build`, K2 pin unmodified and green (453 concepts, ranks
1,1,1,1,1,5), known-negative `{}` unchanged at 9 findings.

Report: docs/2026-09-10-k3-runde15-bundle-mismatch.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-10 23:42:06 +02:00
commit 7cca9e079e
9 changed files with 468 additions and 56 deletions
Download patch file Download diff file Expand all files Collapse all files

86
src/llm_ingestion_okf/contract_check.py
View file

@ -36,6 +36,7 @@ from __future__ import annotations
import argparse
import json
import re
from collections.abc import Callable, Mapping, Sequence
from dataclasses import dataclass
from pathlib import Path
@ -62,6 +63,23 @@ ADJUDICATION_STATES = ("proposed", "adjudicated", "unknown")
#: SS 6.2, from SPEC SS 5.3: derived from `verified`, lowest to highest.
TRUST_TIERS = ("unverified", "machine-confirmed", "human-reviewed")
#: SS 3.1 and SS 3.3, read back out of the skill's own prose. `okf skill` writes
#: the bundle it was generated for in one authored sentence, and this is the
#: pattern that reads it: there is no structured skill model, only its text.
#: Held to the generator by a test, because the two live in different files.
SKILL_IDENTITY = re.compile(r"for one bundle: `([^`<>]+)` at ref\s+`([^`<>]+)`")
def skill_identity(skill_text: str) -> tuple[str, str] | None:
"""The `(bundle_id, ref)` the skill declares, or `None` when it declares
none a reader could act on. `None` is a finding, never a silent pass: the
unfilled template's `<CORPUS>` and `<REF>` are placeholders, and the
template's own rule is that a copy leaving one unfilled is not configured,
it is unfinished."""
match = SKILL_IDENTITY.search(skill_text)
return (match.group(1), match.group(2)) if match else None
#: The level-2 headings a conformant skill carries. Checked by name because the
#: alternative -- searching the prose for the obligation -- passes on any
#: document that happens to use the words.
@ -174,6 +192,73 @@ def rule_bundle_ref(ctx: Context) -> list[Finding]:
return findings
def rule_bundle_identity(ctx: Context) -> list[Finding]:
"""SS 3.1 and SS 3.3: the skill and the payload must name one bundle.
Added 2026-09-10 on a measurement this checker had published about itself
since 2026-09-08 and not closed: it reported `conformant, 15 rules, 0
findings` for a skill generated from one corpus against a payload assembled
from another, for the unfilled template against that payload, and for a
payload sharing the skill's `bundle_id` at a foreign `ref`.
**Both halves are compared, and the `ref` half is the load-bearing one.**
Three distinct builds on one machine were measured carrying the same
`bundle_id`, so an id comparison would pass a stale skill -- exactly the
case the generated skill warns about in its own words. SS 3.3: "a version
is the producer's assertion; a ref is a fact about bytes".
**It compares a DECLARED identity against a DECLARED identity** and never
opens the bundle, so a payload misreporting its own `ref` passes here.
Proving a ref against bytes is `okf consume --ref`'s job and needs a bundle
path this command deliberately does not take.
A payload that declares no identity at all is `rule_bundle_ref`'s defect,
not this one's: restating it would report one hole twice.
"""
declared = skill_identity(ctx.skill)
if declared is None:
return [
Finding(
"bundle_mismatch",
"the skill declares no readable bundle identity, so no payload "
"can be shown to belong to it; a `<PLACEHOLDER>` left unfilled "
"is not an identity, and neither is its absence (SS 3.1, SS 3.3)",
)
]
skill_id, skill_ref = declared
bundle = _mapping(ctx.payload.get("bundle"))
payload_id, payload_ref = _text(bundle.get("bundle_id")), _text(bundle.get("ref"))
disagreements = [
f"{key} (skill {mine!r}, payload {theirs!r})"
for key, mine, theirs in (
("bundle_id", skill_id, payload_id),
("ref", skill_ref, payload_ref),
)
if theirs and theirs != mine
]
if disagreements:
return [
Finding(
"bundle_mismatch",
"the skill was generated for a bundle the payload does not "
f"describe: {'; '.join(disagreements)}. Every number in the "
"skill was measured against its own bundle's bytes (SS 3.1, "
"SS 3.3)",
)
]
return [
Finding(
"bundle_mismatch",
f"excerpt {position} names bundle {found!r}, which is not the "
f"payload's {payload_id!r}; identity across bundles is the "
"(bundle_id, concept_id) tuple, so an excerpt naming another "
"bundle is another bundle's excerpt (SS 3.1)",
)
for position, raw in enumerate(_sequence(ctx.payload.get("excerpts")))
if payload_id and (found := _text(_mapping(raw).get("bundle_id"))) and found != payload_id
]
def rule_excerpt_source_marking(ctx: Context) -> list[Finding]:
if not ctx.payload_is_mapping:
return []
@ -435,6 +520,7 @@ RULES: tuple[Callable[[Context], list[Finding]], ...] = (
rule_payload_shape,
rule_contract_version,
rule_bundle_ref,
rule_bundle_identity,
rule_excerpt_source_marking,
rule_excerpt_named,
rule_excerpt_states,

37
src/llm_ingestion_okf/skill.py
View file

@ -6,11 +6,20 @@ Filling it by hand is what produced `skills/okf-consume/` for one corpus. This
command does the same thing for any bundle, from values it measures rather than
values someone remembered.
**Why a generator rather than one generic skill.** Measured 2026-09-08: the
contract checker passes the UNFILLED template against a real payload (exit 0, 15
rules, 0 findings), and passes a skill built for a different bundle against this
one's payload. So the checker cannot tell the two forms apart, and the choice
could not be made on conformance. It was made on what the skill has to state:
**Why a generator rather than one generic skill.** The measurement this
paragraph used to rest on is CLOSED 2026-09-10. It read: the contract checker
passes the UNFILLED template against a real payload (exit 0, 15 rules, 0
findings), and passes a skill built for a different bundle against this one's
payload. `contract_check.rule_bundle_identity` now compares the identity a
skill declares with the identity its payload declares, so all three measured
pairs are refused at exit 1 with one `bundle_mismatch` finding over 16 rules:
a skill against another bundle's payload, the unfilled template against a real
payload, and -- the arm an id comparison would miss -- a payload sharing the
skill's `bundle_id` at a foreign `ref`. The right pair is untouched at exit 0
with 0 findings.
**The argument for a generator never rested on conformance, and still does
not.** It was made on what the skill has to state:
§ 5's denominators, § 7.6's breaking point and § 6.4's conditional-field list
are all per-bundle numbers. A generic skill can either leave them as holes -- the
template's own definition of unfinished -- or carry another corpus's numbers,
@ -383,6 +392,18 @@ def render(
return header + text, payload
def identity_line(bundle_id: str, ref: str) -> str:
"""The one sentence that says which bundle a generated skill belongs to.
Authored here because the generator writes it, and read back by
`contract_check.skill_identity`, whose `bundle_mismatch` rule is the reason
it has to be findable rather than merely present. Two copies of this
sentence would drift, and the copy nobody reads is the one that goes wrong,
so the coupling has its own test.
"""
return f"generated by `okf skill` for one bundle: `{bundle_id}` at ref\n`{ref}`"
def _description(bundle_id: str, total: int, ref: str) -> str:
return (
f"Answer one question about the OKF bundle `{bundle_id}` ({total} concepts, "
@ -417,9 +438,9 @@ def _rewrite(
(
TEMPLATE_HEADER,
"**This file is an instantiated copy of "
"`skills/okf-consume-template/SKILL.md`,** generated by "
f"`okf skill` for one bundle: `{bundle_id}` at ref\n"
f"`{ref}`. Every value below was measured against those bytes. If the\n"
"`skills/okf-consume-template/SKILL.md`,** "
f"{identity_line(bundle_id, ref)}. Every value below was measured "
"against those bytes. If the\n"
"bundle moves, the ref moves with it and this file is stale — regenerate\n"
"it rather than editing a number here. The section headings are fixed:\n"
"the contract checker reads them by name.",