open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(check): a skill and a payload naming different bundles is a finding

Browse source 
`okf check` had fifteen rules and none asked whether the skill and the payload
were talking about the same bundle. Reproduced on this HEAD before any code
moved: three pairs reported `conformant: 15 rules over 8 excerpts and 438
withheld entries, 0 findings` -- a skill generated from one corpus against
another corpus's payload, the unfilled template against that payload, and a
payload sharing the skill's `bundle_id` at a foreign `ref`. All three now exit 1
with one `bundle_mismatch` finding over 16 rules.

BOTH halves are compared and the `ref` half is load-bearing: three distinct
builds on this machine carry one `bundle_id`, so an id comparison would pass a
stale skill. SS 3.3: "a version is the producer's assertion; a ref is a fact
about bytes". An identity the rule cannot read is a finding, never a silent
pass -- that is what refuses the unfilled template.

No new field: the identity was already in the generated skill's prose, now
factored into `skill.identity_line` and read back by
`contract_check.skill_identity`. Generated skill bytes unchanged, measured on
both tracked bundles on one interpreter.

The rule's first real find is this repository's own hand-made
`skills/okf-consume/SKILL.md`, which predates `okf skill` and declares no
identity a reader can act on: 1 of 1. Nine tests that asserted the old, false
conformance now pair a skill with its own bundle's payload.

Measured, nothing else moved: `~/okf-test/dokumenter` `diff -r` empty old
source vs new on one interpreter (52 files, 26 concepts), `okf project` still
byte-equal to `okf build`, K2 pin unmodified and green (453 concepts, ranks
1,1,1,1,1,5), known-negative `{}` unchanged at 9 findings.

Report: docs/2026-09-10-k3-runde15-bundle-mismatch.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-10 23:42:06 +02:00
commit 7cca9e079e
9 changed files with 468 additions and 56 deletions
Download patch file Download diff file Expand all files Collapse all files

42
CHANGELOG.md
View file

@ -7,6 +7,48 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added
- **`okf check` refuses a skill and a payload that name different bundles
(`bundle_mismatch`).** The checker had published this hole about itself since
2026-09-08 and not closed it: reproduced on this repository's HEAD, it
reported `conformant: 15 rules over 8 excerpts and 438 withheld entries, 0
findings` for a skill generated from one corpus against a payload assembled
from another; the same line for the UNFILLED template against that payload;
and the same line again for a payload sharing the skill's `bundle_id` at a
foreign `ref`. All three now exit **1** with one finding. The right pair is
untouched at exit 0 with 0 findings, and a payload declaring no identity at
all stays `ref_missing`'s defect at 9 findings -- no rule restates another.
- **BOTH halves are compared, and the `ref` half is the load-bearing one.**
Three distinct builds on one machine were measured carrying the same
`bundle_id`, so an id comparison alone would pass a stale skill -- the case
the generated skill warns about in its own words ("if the bundle moves, the
ref moves with it and this file is stale"). SS 3.3: "a version is the
producer's assertion; a ref is a fact about bytes".
- **An identity the rule cannot read is a finding, never a silent pass.**
That is what refuses the unfilled template, whose `<CORPUS>` and `<REF>`
are not an identity. It also refuses the repository's own hand-made
`skills/okf-consume/SKILL.md`, which predates `okf skill` and declares no
bundle identity a reader can act on -- **1 of 1** shipped hand-made skill,
a real find and not a fixture.
- **The rule compares a DECLARED identity against a DECLARED identity and
never opens the bundle**, so a payload misreporting its own `ref` still
passes. Proving a ref against bytes is `okf consume --ref`'s job and needs
a bundle path this command deliberately does not take.
### Changed
- **The checker's rule count is 16, not 15**, and `Report.rules_evaluated` is
the denominator every report line quotes -- so `15 rules` becomes `16 rules`
in every published line. A consumer citing the old number is citing a number
that has changed. No payload bytes move: this is the checker, not the
pre-pass.
- `skill.identity_line` is now the single authored copy of the sentence a
generated skill declares its bundle in, read back by
`contract_check.skill_identity` and held to it by a test. Generated skill
bytes are unchanged -- measured, both tracked bundles byte-identical before
and after on the same interpreter.
## [0.8.1] — 2026-09-10
### Added

44
CLAUDE.md
View file

@ -687,13 +687,49 @@ and fixtures, never code.
names on PATH. The template and `docs/consumption-contract.md` (the § 7.4
known-positive) are force-included into the wheel from the file they are
authored in — one authored copy, no committed duplicate.
The form was chosen on a measurement: the contract checker passes the
UNFILLED template and passes a skill built for another bundle, so it cannot
The form was chosen on a measurement: the contract checker passed the
UNFILLED template and passed a skill built for another bundle, so it could not
tell the two apart — the choice rests on § 5/§ 6.4/§ 7.6 being per-bundle
numbers a generic skill can only leave as holes or state falsely.
numbers a generic skill can only leave as holes or state falsely, and that
argument NEVER rested on conformance, so closing the measurement leaves it
standing. **That half is CLOSED 2026-09-10 by `bundle_mismatch`, the checker's
SIXTEENTH rule** (`RULES` is a tuple and `Report.rules_evaluated` is
`len(RULES)`, so every published «15 rules» line is now «16 rules» — a
contract change for anyone quoting it). It compares the identity a skill
DECLARES against the identity its payload declares, **both `bundle_id` AND
`ref`**: three distinct builds on this machine carry the one `bundle_id`
`k2-trinn1-20260903` at three refs, so an id-only rule would pass a stale
skill, which is the case the generated skill warns about in its own words.
SS 3.3 is the ground — «a version is the producer's assertion; a ref is a fact
about bytes» — and SS 3.1 for the same question one level down, an excerpt
naming a bundle its payload does not. Reproduced on `113b3f8` before any code
moved, three pairs at `conformant: 15 rules over 8 excerpts and 438 withheld
entries, 0 findings` and rc 0; after, all three at rc 1 with **one** finding
over 16 rules — a foreign corpus's payload, the unfilled template, and (the
arm that separates a whole rule from half of one) a payload sharing the id at
a foreign ref. **An identity the rule cannot READ is a finding, never a silent
pass**, or the template passes again. The right pair is untouched at rc 0 / 0
findings, and `{}` is unchanged at **9** findings because a payload declaring
no identity stays `ref_missing`'s defect — no rule restates another. NO new
field was needed: the identity already lived in the generated skill's prose,
now one authored copy in `skill.identity_line` read back by
`contract_check.skill_identity`, generated skill bytes byte-identical before
and after on both tracked bundles. **The rule compares DECLARED against
DECLARED and never opens the bundle**, so a payload lying about its own `ref`
still passes — that is `okf consume --ref`'s job and needs a bundle path this
command deliberately does not take; «closed» means the three measured forms
now fell, not that no fourth exists. Report:
`docs/2026-09-10-k3-runde15-bundle-mismatch.md`.
The first instantiated consumption skill is `skills/okf-consume/`; the
measurement behind it, including the control that FAILED, is
`docs/2026-09-07-okf-konsumskill-maaling.md`. **The ranking is this
`docs/2026-09-07-okf-konsumskill-maaling.md`. **That skill is the new rule's
first REAL find**: hand-filled before `okf skill` existed, it declares no
bundle identity a reader can act on, so `okf check` now refuses it against its
own shipped example payload (rc 1, 1 finding) — **1 of 1** shipped hand-made
instantiated skill. Behind it sits a second fact: its prose states K2 numbers
while the payload beside it is regenerated from the tracked golden bundle,
because K2 cannot ship. What that artefact should BE is an operator question,
not a patch. **The ranking is this
repository's own choice** — the contract binds a payload, not a retrieval
algorithm (§ 10) — and it has FOUR optional widenings. Three are **off by
default** and keep the default payload byte-identical; the fourth

5
README.md
View file

@ -531,8 +531,9 @@ puts the project somewhere other than the current directory.
Measured 2026-09-09 from a fresh `uv tool install` with this repository nowhere
on the path: 5 documents in, **26** concepts out, a skill carrying **0** paths
into any checkout, and `okf check` conformant on its own payload (15 rules, 0
findings). The 2026-09-08 run of the same measurement reported 15 concepts, and
into any checkout, and `okf check` conformant on its own payload (16 rules, 0
findings; the count was 15 until `bundle_mismatch` landed 2026-09-10, and the
pair was re-measured rather than carried over). The 2026-09-08 run of the same measurement reported 15 concepts, and
that number was the defect rather than the result: `okf project` was calling
`build()` as a function and reading its signature's defaults, which disagreed
with argparse's on two flags. Two tests now hold the two default sets equal. Before that day the same result took a `PYTHONPATH`, a snapshot of a

155
docs/2026-09-10-k3-runde15-bundle-mismatch.md Normal file
View file

@ -0,0 +1,155 @@
# K3 round 15: which bundle was this skill generated for?
**Date:** 2026-09-10 · **HEAD at start:** `113b3f8` (`v0.8.1` tagged locally on
`3daf983`) · **Red test:** `b5df335` · No push, no tag, no version bump.
## 1. What was wrong
`okf skill --help` had carried this sentence about its own tooling since
2026-09-08:
> Measured 2026-09-08: the contract checker passes the UNFILLED template
> against a real payload (exit 0, 15 rules, 0 findings), and passes a skill
> built for a different bundle against this one's payload. So the checker
> cannot tell the two forms apart.
Documented, not closed. `okf check` had fifteen rules and none of them asked
whether the skill and the payload were talking about the same bundle.
## 2. Reproduced first, on this HEAD
Every figure below was re-measured on `113b3f8` in a frozen export
(`git archive` + `uv sync --frozen --extra extract`), because the order's
figures were taken on `958e9bc`. **Zero divergence**: all five rows reproduced
byte for byte, including the exact report lines.
Two real artefacts, both read and never written: a SKILL generated by
`okf skill` from the pinned K2 bundle (`k2-trinn1-20260903` at
`sha256-tree:18ae18ab...`, 453 concepts) and a payload from an unrelated corpus
(`vegnormal-n100-2023` at `sha256-tree:da6b8204...`, 446 concepts).
| # | pair | rc before | rc after | line after |
|---|---|---|---|---|
| A | skill(k2) x payload(vegnormal-n100) | 0 | **1** | `NOT conformant: 16 rules over 8 excerpts and 438 withheld entries, 1 findings` |
| B | unfilled template x payload(vegnormal-n100) | 0 | **1** | same denominators, 1 finding |
| C | skill(k2) x payload(k2 id, foreign ref) | 0 | **1** | same denominators, 1 finding, message names **`ref` only** |
| KP | skill(k2) x its own payload | 0 | **0** | `conformant: 16 rules over 8 excerpts and 445 withheld entries, 0 findings` |
| KN | skill(k2) x `{}` | 1 | **1** | `NOT conformant: 16 rules over 0 excerpts and 0 withheld entries, 9 findings` |
Before-rows were measured with the frozen export of `113b3f8`; after-rows with
the working tree. Exit codes were captured directly (`out=$(okf check ...);
rc=$?`) and never through a pipe, which would report `tail`'s status.
## 3. Why the rule compares the ref and not only the id
Three distinct builds on this machine carry the **same** `bundle_id`
`k2-trinn1-20260903` at three different refs (`f14872a0...`, `9a4e5561...`,
`18ae18ab...`). So `bundle_id` does not identify the bytes, and an id-only rule
would pass a stale skill at exit 0 -- the case the generated skill warns about
in its own words: "If the bundle moves, the ref moves with it and this file is
stale -- regenerate". `docs/consumption-contract.md` SS 3.3 is the ground:
"a version is the producer's assertion; a ref is a fact about bytes". SS 3.1 is
the ground for the excerpt-level half: "identity across bundles is the tuple,
never `concept_id` alone".
**Row C is the arm that separates a whole rule from half of one.** It is the
only row where an id comparison and an identity comparison disagree.
## 4. What was built
One new rule, `contract_check.rule_bundle_identity`, one new code
`bundle_mismatch`, `RULES` 15 -> 16. No existing rule weakened, no existing code
renamed, no rule short-circuiting another.
**No new field was needed, and that was verified rather than assumed.** The
identity is already in the generated skill's prose. It was factored into one
authored copy at `skill.py:395` (`identity_line`), which `_rewrite` uses at the
place it was previously inlined, and `contract_check.skill_identity` reads back.
Generated skill bytes are **unchanged**: both tracked bundles produce
byte-identical skills before and after when rendered on the same interpreter
(`e67827a6...` and `890aeac2...` both ways). The earlier apparent 9-byte
difference was the bundle path's length in two different export trees, not the
code -- measured and discarded.
Three cases, in one rule:
1. The skill declares no readable identity -> finding. This is what refuses the
unfilled template, and it is load-bearing: a silent pass here would let row B
through and leave the round undone.
2. The declared identities disagree in `bundle_id`, in `ref`, or in both ->
one finding naming which halves disagreed.
3. An excerpt names a bundle the payload does not -> one finding per excerpt.
Same question one level down (SS 3.1).
A payload declaring no identity at all is left to `rule_bundle_ref`: restating
it would report one hole twice. Measured: the known-negative `{}` is unchanged
at **9 findings**, and `bundle_mismatch` is not among them.
## 5. What the rule found in this repository
`skills/okf-consume/SKILL.md` -- the first instantiated skill, filled in by hand
before `okf skill` existed -- **declares no bundle identity a reader can act
on**, so `okf check` now refuses it against its own shipped example payload:
`NOT conformant: 16 rules over 3 excerpts and 0 withheld entries, 1 findings`,
rc 1. **1 of 1** shipped hand-made instantiated skill. A second fact sits behind
it: that skill's prose states K2 numbers while the payload shipped beside it is
regenerated from the tracked golden bundle, because K2 cannot ship. Repairing it
is a decision about what that artefact is for, and is left to the operator
rather than taken here.
Nine tests asserted the old, false conformance by pairing a skill with a
payload from another bundle (or with the template standing in for a skill).
They now pair correctly -- the checker's control is an INSTANTIATED skill, since
the template can no longer be one.
## 6. Nothing else moved, measured on the bytes
- **Build output identical.** `~/okf-test/dokumenter` built with the old source
and the new source **on the same interpreter**: `diff -r` empty, 52 files, 26
concepts. (Comparing across the two venvs shows one line in `log.md` -- the
converter's own path and Python version -- which is the venv, not this change.)
- **`okf project` is still byte-equal to `okf build`** on that folder: `diff -r`
empty against `<out>/.okf/<id>`.
- **K2 pin untouched and green.** `tests/test_default_bundle_pin.py` is
unmodified and RAN (7 passed, not skipped): 453 concepts, per-row ranks
`(1, 1, 1, 1, 1, 5)`.
- **The repository's own generated pair is conformant at the full 16 rules**:
`conformant: 16 rules over 2 excerpts and 1 withheld entries, 0 findings`, and
the `okf project` skill likewise, `16 rules over 8 excerpts and 18 withheld
entries, 0 findings`.
- **Gates**, ruff 0.16.6: `ruff check src tests tools` clean, `ruff format
--check .` 119 files, `mypy src` clean, `pytest -q` **1593 passed, 1 skipped**
(baseline 1582 + 1 on this HEAD; the frozen export additionally fails
`test_segmented_golden.py::test_the_four_existing_goldens_are_untouched`,
which is the export artefact and was red before this round too).
## 7. Honesty limits
- **The rule compares a DECLARED identity against a DECLARED identity and never
opens the bundle.** A payload that misreports its own `ref` passes. Proving a
ref against bytes is `okf consume --ref`'s job and needs a bundle path this
command deliberately does not take. Nothing here makes a ref true.
- **The denominator for the reproduction is two real SKILL/payload pairs plus
arms built from this repository's own two tracked bundles -- not a corpus.**
Every fixture is generated from bundles already in the tree, so no sentence
from any document, no corpus path and no gold question reaches a tracked file.
- **The excerpt-level arm is built, not found.** Across the two real payloads
reproduced this round, **0 of 16** excerpts carried a `bundle_id` differing
from the payload's own. The arm covers a case that has not been observed in
the wild; it is cheap, and its absence has a denominator.
- **Conformance is the floor, never the proof.** The checker's own docstring
says why: the division of labour (SS 2) and the prohibitions (SS 9) are
properties of a RUN -- whether an agent went looking for context the pre-pass
withheld -- and no static check reaches them. A green `okf check` is not a
quality gate, and this repository measured that directly on
2026-09-10: three arms over one corpus all returned 0 findings from `okf
check` while their hit@k ranged from 6 of 6 to 0 of 6.
- **"Closed" means the three measured forms now fail, not that no fourth form
exists.** A skill whose identity sentence has been edited by hand to name a
bundle it was not generated from would pass. So would a payload and a skill
that agree with each other and disagree with the bytes.
- **The rule count is a contract change downstream.** Every consumer quoting
"15 rules" is quoting a number that has changed, and one pair
`portfolio-optimiser` already runs goes from rc 0 to rc 1. That is the rule
working, not a regression there -- but it is their build's exit code, so it
was sent to them rather than left to be discovered.

86
src/llm_ingestion_okf/contract_check.py
View file

@ -36,6 +36,7 @@ from __future__ import annotations
import argparse
import json
import re
from collections.abc import Callable, Mapping, Sequence
from dataclasses import dataclass
from pathlib import Path
@ -62,6 +63,23 @@ ADJUDICATION_STATES = ("proposed", "adjudicated", "unknown")
#: SS 6.2, from SPEC SS 5.3: derived from `verified`, lowest to highest.
TRUST_TIERS = ("unverified", "machine-confirmed", "human-reviewed")
#: SS 3.1 and SS 3.3, read back out of the skill's own prose. `okf skill` writes
#: the bundle it was generated for in one authored sentence, and this is the
#: pattern that reads it: there is no structured skill model, only its text.
#: Held to the generator by a test, because the two live in different files.
SKILL_IDENTITY = re.compile(r"for one bundle: `([^`<>]+)` at ref\s+`([^`<>]+)`")
def skill_identity(skill_text: str) -> tuple[str, str] | None:
"""The `(bundle_id, ref)` the skill declares, or `None` when it declares
none a reader could act on. `None` is a finding, never a silent pass: the
unfilled template's `<CORPUS>` and `<REF>` are placeholders, and the
template's own rule is that a copy leaving one unfilled is not configured,
it is unfinished."""
match = SKILL_IDENTITY.search(skill_text)
return (match.group(1), match.group(2)) if match else None
#: The level-2 headings a conformant skill carries. Checked by name because the
#: alternative -- searching the prose for the obligation -- passes on any
#: document that happens to use the words.
@ -174,6 +192,73 @@ def rule_bundle_ref(ctx: Context) -> list[Finding]:
return findings
def rule_bundle_identity(ctx: Context) -> list[Finding]:
"""SS 3.1 and SS 3.3: the skill and the payload must name one bundle.
Added 2026-09-10 on a measurement this checker had published about itself
since 2026-09-08 and not closed: it reported `conformant, 15 rules, 0
findings` for a skill generated from one corpus against a payload assembled
from another, for the unfilled template against that payload, and for a
payload sharing the skill's `bundle_id` at a foreign `ref`.
**Both halves are compared, and the `ref` half is the load-bearing one.**
Three distinct builds on one machine were measured carrying the same
`bundle_id`, so an id comparison would pass a stale skill -- exactly the
case the generated skill warns about in its own words. SS 3.3: "a version
is the producer's assertion; a ref is a fact about bytes".
**It compares a DECLARED identity against a DECLARED identity** and never
opens the bundle, so a payload misreporting its own `ref` passes here.
Proving a ref against bytes is `okf consume --ref`'s job and needs a bundle
path this command deliberately does not take.
A payload that declares no identity at all is `rule_bundle_ref`'s defect,
not this one's: restating it would report one hole twice.
"""
declared = skill_identity(ctx.skill)
if declared is None:
return [
Finding(
"bundle_mismatch",
"the skill declares no readable bundle identity, so no payload "
"can be shown to belong to it; a `<PLACEHOLDER>` left unfilled "
"is not an identity, and neither is its absence (SS 3.1, SS 3.3)",
)
]
skill_id, skill_ref = declared
bundle = _mapping(ctx.payload.get("bundle"))
payload_id, payload_ref = _text(bundle.get("bundle_id")), _text(bundle.get("ref"))
disagreements = [
f"{key} (skill {mine!r}, payload {theirs!r})"
for key, mine, theirs in (
("bundle_id", skill_id, payload_id),
("ref", skill_ref, payload_ref),
)
if theirs and theirs != mine
]
if disagreements:
return [
Finding(
"bundle_mismatch",
"the skill was generated for a bundle the payload does not "
f"describe: {'; '.join(disagreements)}. Every number in the "
"skill was measured against its own bundle's bytes (SS 3.1, "
"SS 3.3)",
)
]
return [
Finding(
"bundle_mismatch",
f"excerpt {position} names bundle {found!r}, which is not the "
f"payload's {payload_id!r}; identity across bundles is the "
"(bundle_id, concept_id) tuple, so an excerpt naming another "
"bundle is another bundle's excerpt (SS 3.1)",
)
for position, raw in enumerate(_sequence(ctx.payload.get("excerpts")))
if payload_id and (found := _text(_mapping(raw).get("bundle_id"))) and found != payload_id
]
def rule_excerpt_source_marking(ctx: Context) -> list[Finding]:
if not ctx.payload_is_mapping:
return []
@ -435,6 +520,7 @@ RULES: tuple[Callable[[Context], list[Finding]], ...] = (
rule_payload_shape,
rule_contract_version,
rule_bundle_ref,
rule_bundle_identity,
rule_excerpt_source_marking,
rule_excerpt_named,
rule_excerpt_states,

37
src/llm_ingestion_okf/skill.py
View file

@ -6,11 +6,20 @@ Filling it by hand is what produced `skills/okf-consume/` for one corpus. This
command does the same thing for any bundle, from values it measures rather than
values someone remembered.
**Why a generator rather than one generic skill.** Measured 2026-09-08: the
contract checker passes the UNFILLED template against a real payload (exit 0, 15
rules, 0 findings), and passes a skill built for a different bundle against this
one's payload. So the checker cannot tell the two forms apart, and the choice
could not be made on conformance. It was made on what the skill has to state:
**Why a generator rather than one generic skill.** The measurement this
paragraph used to rest on is CLOSED 2026-09-10. It read: the contract checker
passes the UNFILLED template against a real payload (exit 0, 15 rules, 0
findings), and passes a skill built for a different bundle against this one's
payload. `contract_check.rule_bundle_identity` now compares the identity a
skill declares with the identity its payload declares, so all three measured
pairs are refused at exit 1 with one `bundle_mismatch` finding over 16 rules:
a skill against another bundle's payload, the unfilled template against a real
payload, and -- the arm an id comparison would miss -- a payload sharing the
skill's `bundle_id` at a foreign `ref`. The right pair is untouched at exit 0
with 0 findings.
**The argument for a generator never rested on conformance, and still does
not.** It was made on what the skill has to state:
§ 5's denominators, § 7.6's breaking point and § 6.4's conditional-field list
are all per-bundle numbers. A generic skill can either leave them as holes -- the
template's own definition of unfinished -- or carry another corpus's numbers,
@ -383,6 +392,18 @@ def render(
return header + text, payload
def identity_line(bundle_id: str, ref: str) -> str:
"""The one sentence that says which bundle a generated skill belongs to.
Authored here because the generator writes it, and read back by
`contract_check.skill_identity`, whose `bundle_mismatch` rule is the reason
it has to be findable rather than merely present. Two copies of this
sentence would drift, and the copy nobody reads is the one that goes wrong,
so the coupling has its own test.
"""
return f"generated by `okf skill` for one bundle: `{bundle_id}` at ref\n`{ref}`"
def _description(bundle_id: str, total: int, ref: str) -> str:
return (
f"Answer one question about the OKF bundle `{bundle_id}` ({total} concepts, "
@ -417,9 +438,9 @@ def _rewrite(
(
TEMPLATE_HEADER,
"**This file is an instantiated copy of "
"`skills/okf-consume-template/SKILL.md`,** generated by "
f"`okf skill` for one bundle: `{bundle_id}` at ref\n"
f"`{ref}`. Every value below was measured against those bytes. If the\n"
"`skills/okf-consume-template/SKILL.md`,** "
f"{identity_line(bundle_id, ref)}. Every value below was measured "
"against those bytes. If the\n"
"bundle moves, the ref moves with it and this file is stale — regenerate\n"
"it rather than editing a number here. The section headings are fixed:\n"
"the contract checker reads them by name.",

92
tests/test_contract_check.py
View file

@ -33,6 +33,7 @@ PROJECT_ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(PROJECT_ROOT / "tools"))
import okf_contract_check # noqa: E402
import okf_skill # noqa: E402
TEMPLATE = PROJECT_ROOT / "skills" / "okf-consume-template" / "SKILL.md"
EXAMPLE = TEMPLATE.parent / "references" / "example-payload.json"
@ -44,6 +45,32 @@ def load_example() -> dict[str, Any]:
return payload
def declaring(skill_text: str, payload: dict[str, Any]) -> str:
"""The template with the one hole `bundle_mismatch` reads filled in.
The template alone can no longer be a conformant skill: `<CORPUS>` and
`<REF>` are placeholders, and an identity the checker cannot read is a
finding by design. The sentence is the generator's own, taken from it
rather than copied, so the two cannot drift apart.
"""
bundle = payload["bundle"]
return skill_text.replace(
okf_skill.TEMPLATE_HEADER,
okf_skill.identity_line(bundle["bundle_id"], bundle["ref"]) + ".",
1,
)
def instantiated() -> str:
return declaring(TEMPLATE.read_text(encoding="utf-8"), load_example())
def instantiated_file(tmp_path: Path) -> Path:
target = tmp_path / "SKILL.md"
target.write_text(instantiated(), encoding="utf-8")
return target
def codes(skill_text: str, payload: Any) -> list[str]:
return [finding.code for finding in okf_contract_check.check(skill_text, payload).findings]
@ -51,16 +78,20 @@ def codes(skill_text: str, payload: Any) -> list[str]:
# --- The known-positive, which every negative arm depends on ----------------
def test_shipped_template_and_example_payload_are_conformant() -> None:
"""The CONTROL. A checker that cannot pass anything proves nothing below."""
report = okf_contract_check.check(TEMPLATE.read_text(encoding="utf-8"), load_example())
def test_an_instantiated_skill_and_its_own_example_payload_are_conformant() -> None:
"""The CONTROL. A checker that cannot pass anything proves nothing below.
It is the INSTANTIATED skill, not the template: since 2026-09-10 a skill
declaring no readable bundle identity is `bundle_mismatch`, measured on
the template itself in `test_bundle_identity.py`."""
report = okf_contract_check.check(instantiated(), load_example())
assert report.findings == ()
def test_the_report_carries_its_denominators() -> None:
""" "Conformant" without a denominator is unmeasured, per the contract's own
section 5. The report says how many rules ran over how many units."""
report = okf_contract_check.check(TEMPLATE.read_text(encoding="utf-8"), load_example())
report = okf_contract_check.check(instantiated(), load_example())
assert report.rules_evaluated == len(okf_contract_check.RULES)
assert report.rules_evaluated > 0
assert report.excerpts_examined == 3
@ -76,112 +107,112 @@ def test_the_report_carries_its_denominators() -> None:
def test_broken_denominator_identity_is_named() -> None:
payload = load_example()
payload["denominators"]["considered"] = 6
assert "denominator_identity" in codes(TEMPLATE.read_text(encoding="utf-8"), payload)
assert "denominator_identity" in codes(instantiated(), payload)
def test_delivered_count_disagreeing_with_the_list_is_named() -> None:
payload = load_example()
payload["excerpts"] = payload["excerpts"][:2]
assert "denominator_list_mismatch" in codes(TEMPLATE.read_text(encoding="utf-8"), payload)
assert "denominator_list_mismatch" in codes(instantiated(), payload)
def test_a_missing_adjudication_key_is_not_absence() -> None:
"""The contract's sharpest rule: `unknown` is written, never omitted."""
payload = load_example()
del payload["excerpts"][0]["adjudication"]
assert "state_missing" in codes(TEMPLATE.read_text(encoding="utf-8"), payload)
assert "state_missing" in codes(instantiated(), payload)
def test_an_adjudication_value_outside_the_closed_set_is_named() -> None:
payload = load_example()
payload["excerpts"][0]["adjudication"] = "absent"
assert "state_not_in_set" in codes(TEMPLATE.read_text(encoding="utf-8"), payload)
assert "state_not_in_set" in codes(instantiated(), payload)
def test_a_trust_tier_outside_the_closed_set_is_named() -> None:
payload = load_example()
payload["excerpts"][1]["trust_tier"] = "trusted"
assert "state_not_in_set" in codes(TEMPLATE.read_text(encoding="utf-8"), payload)
assert "state_not_in_set" in codes(instantiated(), payload)
@pytest.mark.parametrize("key", ["bundle_id", "concept_id", "sha256"])
def test_an_excerpt_missing_its_source_marking_is_named(key: str) -> None:
payload = load_example()
del payload["excerpts"][0][key]
assert "source_marking_incomplete" in codes(TEMPLATE.read_text(encoding="utf-8"), payload)
assert "source_marking_incomplete" in codes(instantiated(), payload)
def test_a_bundle_without_a_ref_is_named() -> None:
payload = load_example()
payload["bundle"]["ref"] = ""
assert "ref_missing" in codes(TEMPLATE.read_text(encoding="utf-8"), payload)
assert "ref_missing" in codes(instantiated(), payload)
def test_a_withheld_concept_without_a_rule_is_an_undeclared_cut() -> None:
payload = load_example()
del payload["withheld"][0]["rule"]
assert "cut_undeclared" in codes(TEMPLATE.read_text(encoding="utf-8"), payload)
assert "cut_undeclared" in codes(instantiated(), payload)
def test_spending_over_the_limit_is_the_gate_firing() -> None:
payload = load_example()
payload["budget"]["spent"] = payload["budget"]["limit"] + 1
assert "budget_exceeded" in codes(TEMPLATE.read_text(encoding="utf-8"), payload)
assert "budget_exceeded" in codes(instantiated(), payload)
def test_an_instrument_that_missed_its_known_positive_is_named() -> None:
payload = load_example()
payload["budget"]["known_positive"]["measured"] = 10405
assert "instrument_unvalidated" in codes(TEMPLATE.read_text(encoding="utf-8"), payload)
assert "instrument_unvalidated" in codes(instantiated(), payload)
def test_a_budget_without_an_instrument_is_named() -> None:
payload = load_example()
payload["budget"]["instrument"] = ""
assert "budget_undeclared" in codes(TEMPLATE.read_text(encoding="utf-8"), payload)
assert "budget_undeclared" in codes(instantiated(), payload)
def test_a_payload_naming_no_contract_revision_is_named() -> None:
payload = load_example()
del payload["contract"]
assert "contract_unversioned" in codes(TEMPLATE.read_text(encoding="utf-8"), payload)
assert "contract_unversioned" in codes(instantiated(), payload)
def test_a_payload_that_is_not_a_mapping_is_refused_not_crashed() -> None:
assert "payload_invalid" in codes(TEMPLATE.read_text(encoding="utf-8"), ["not", "a", "mapping"])
assert "payload_invalid" in codes(instantiated(), ["not", "a", "mapping"])
# --- Skill mutations --------------------------------------------------------
def test_a_missing_required_section_is_named() -> None:
text = TEMPLATE.read_text(encoding="utf-8").replace("## Denominators", "## Counts")
text = instantiated().replace("## Denominators", "## Counts")
assert "skill_section_missing" in codes(text, load_example())
def test_a_missing_required_marking_is_named() -> None:
text = TEMPLATE.read_text(encoding="utf-8").replace("[sourced-not-sufficient]", "[thin]")
text = instantiated().replace("[sourced-not-sufficient]", "[thin]")
assert "skill_marking_missing" in codes(text, load_example())
def test_a_translated_marking_literal_does_not_count() -> None:
"""One literal string, no variants, no translations."""
text = TEMPLATE.read_text(encoding="utf-8").replace(
"[unverifiable-from-bundle]", "[ikke-verifiserbar-fra-bundle]"
)
text = instantiated().replace("[unverifiable-from-bundle]", "[ikke-verifiserbar-fra-bundle]")
assert "skill_marking_missing" in codes(text, load_example())
def test_a_skill_omitting_an_adjudication_value_is_named() -> None:
text = TEMPLATE.read_text(encoding="utf-8").replace("`unknown`", "`missing`")
text = instantiated().replace("`unknown`", "`missing`")
assert "skill_state_missing" in codes(text, load_example())
# --- The command line -------------------------------------------------------
def test_the_command_exits_zero_on_the_shipped_pair() -> None:
exit_code = okf_contract_check.main(["--skill", str(TEMPLATE), "--payload", str(EXAMPLE)])
def test_the_command_exits_zero_on_an_instantiated_pair(tmp_path: Path) -> None:
exit_code = okf_contract_check.main(
["--skill", str(instantiated_file(tmp_path)), "--payload", str(EXAMPLE)]
)
assert exit_code == 0
@ -190,7 +221,12 @@ def test_the_command_exits_one_on_a_non_conformant_payload(tmp_path: Path) -> No
payload["denominators"]["considered"] = 6
broken = tmp_path / "payload.json"
broken.write_text(json.dumps(payload), encoding="utf-8")
assert okf_contract_check.main(["--skill", str(TEMPLATE), "--payload", str(broken)]) == 1
assert (
okf_contract_check.main(
["--skill", str(instantiated_file(tmp_path)), "--payload", str(broken)]
)
== 1
)
def test_the_command_separates_could_not_run_from_non_conformant(tmp_path: Path) -> None:
@ -204,13 +240,13 @@ def test_the_command_separates_could_not_run_from_non_conformant(tmp_path: Path)
assert okf_contract_check.main(["--skill", str(TEMPLATE), "--payload", str(unreadable)]) == 2
def test_the_module_runs_as_a_script() -> None:
def test_the_module_runs_as_a_script(tmp_path: Path) -> None:
result = subprocess.run(
[
sys.executable,
str(PROJECT_ROOT / "tools" / "okf_contract_check.py"),
"--skill",
str(TEMPLATE),
str(instantiated_file(tmp_path)),
"--payload",
str(EXAMPLE),
],

53
tests/test_okf_consume.py
View file

@ -32,6 +32,7 @@ import sys
import unicodedata
from collections.abc import Mapping
from pathlib import Path
from typing import Any
import pytest
@ -41,10 +42,28 @@ sys.path.insert(0, str(PROJECT_ROOT / "tools"))
import okf_consume # noqa: E402
import okf_consume_measure # noqa: E402
import okf_contract_check # noqa: E402
import okf_skill # noqa: E402
from llm_ingestion_okf.materialize import parse_frontmatter # noqa: E402
TEMPLATE = PROJECT_ROOT / "skills" / "okf-consume-template" / "SKILL.md"
def _skill_declaring(payload: dict[str, Any]) -> str:
"""A skill declaring the bundle THIS payload declares.
The template cannot stand in for one any more: `<CORPUS>` and `<REF>` are
placeholders, and since 2026-09-10 an identity `okf check` cannot read is
a `bundle_mismatch` finding. The sentence comes from the generator rather
than being copied beside it."""
bundle = payload["bundle"]
return TEMPLATE.read_text(encoding="utf-8").replace(
okf_skill.TEMPLATE_HEADER,
okf_skill.identity_line(bundle["bundle_id"], bundle["ref"]) + ".",
1,
)
GOLDEN = PROJECT_ROOT / "examples" / "ingest-golden-segmented-okf-v0-2" / "expected-bundle"
@ -730,8 +749,9 @@ def _payload(
return okf_consume.build_payload(root, question=question, **kwargs) # type: ignore[arg-type]
def test_the_payload_passes_the_checker_against_the_template_with_zero_findings() -> None:
report = okf_contract_check.check(TEMPLATE.read_text(encoding="utf-8"), _payload())
def test_the_payload_passes_the_checker_against_a_skill_for_its_own_bundle() -> None:
payload = _payload()
report = okf_contract_check.check(_skill_declaring(payload), payload)
assert report.findings == ()
@ -855,7 +875,7 @@ def test_a_question_with_no_answer_returns_a_measured_empty_set_not_a_guess() ->
def test_the_empty_payload_still_passes_the_checker() -> None:
payload = _payload(question="Hva er reglene for sveising av titan i vakuum?")
assert okf_contract_check.check(TEMPLATE.read_text(encoding="utf-8"), payload).findings == ()
assert okf_contract_check.check(_skill_declaring(payload), payload).findings == ()
# --- Corpus-conditional arms --------------------------------------------------
@ -1072,12 +1092,16 @@ def test_the_payload_written_by_the_cli_passes_the_checker(tmp_path: Path) -> No
assert (
_run(str(FIXTURE), "--question", "Hvordan skal prisene fylles ut?", "--out", str(out))
).returncode == 0
skill_file = tmp_path / "SKILL.md"
skill_file.write_text(
_skill_declaring(json.loads(out.read_text(encoding="utf-8"))), encoding="utf-8"
)
checked = subprocess.run(
[
sys.executable,
str(PROJECT_ROOT / "tools" / "okf_contract_check.py"),
"--skill",
str(TEMPLATE),
str(skill_file),
"--payload",
str(out),
],
@ -1134,9 +1158,14 @@ def test_every_rule_the_pre_pass_can_emit_is_named_in_the_skill() -> None:
assert rule in text, rule
def test_the_skill_and_a_real_payload_pass_the_checker_together() -> None:
payload = _payload()
assert okf_contract_check.check(SKILL.read_text(encoding="utf-8"), payload).findings == ()
def test_the_skill_and_a_real_payload_pass_the_checker_together(tmp_path: Path) -> None:
# A GENERATED skill, against a payload from the bundle it was generated for.
# The shipped `skills/okf-consume/SKILL.md` cannot serve here: it predates
# `okf skill` and declares no bundle identity a reader can act on, which is
# a `bundle_mismatch` finding and is recorded as one rather than worked
# around.
text, payload = okf_skill.render(GOLDEN, out=tmp_path / "skill")
assert okf_contract_check.check(text, payload).findings == ()
def test_the_shipped_example_payload_is_current_and_regenerates_byte_for_byte() -> None:
@ -1793,9 +1822,8 @@ def test_a_payload_carrying_a_reservation_still_passes_the_checker(tmp_path: Pat
# SS 8 permits additional members; a declaration the checker refuses would
# buy honesty at the price of conformance.
root = _eviction_bundle(tmp_path / "bundle")
report = okf_contract_check.check(
TEMPLATE.read_text(encoding="utf-8"), _eviction_payload(root, reserve_top_rank=True)
)
payload = _eviction_payload(root, reserve_top_rank=True)
report = okf_contract_check.check(_skill_declaring(payload), payload)
assert report.findings == ()
@ -2492,9 +2520,10 @@ def test_the_checker_refuses_an_excerpt_that_cannot_be_named() -> None:
# `title` is the defect po measured, and a checker that passes it certifies
# a payload a model cannot cite from.
payload = json.loads((SKILL.parent / "references" / "example-payload.json").read_text("utf-8"))
assert okf_contract_check.check(SKILL.read_text(encoding="utf-8"), payload).findings == ()
skill_text = _skill_declaring(payload)
assert okf_contract_check.check(skill_text, payload).findings == ()
del payload["excerpts"][0]["title"]
codes = [f.code for f in okf_contract_check.check(SKILL.read_text("utf-8"), payload).findings]
codes = [f.code for f in okf_contract_check.check(skill_text, payload).findings]
assert codes == ["excerpt_unnamed"]

10
tests/test_okf_skill.py
View file

@ -2,9 +2,15 @@
`tools/okf_skill.py` turns one OKF bundle into one instantiated `SKILL.md` that
`tools/okf_contract_check.py` accepts. The discipline here is the one measurement
that decided the form: **the checker cannot tell an instantiated skill from an
unfilled template**, and passes a skill built for a different bundle against this
that decided the form: the checker could not tell an instantiated skill from an
unfilled template, and passed a skill built for a different bundle against this
one's payload. So every gate the checker does not have is a test here.
**That measurement is closed on its identity half since 2026-09-10.** The
`bundle_mismatch` rule refuses both forms, and `tests/test_bundle_identity.py`
holds the arms. The gates below are the ones it still does not have: what the
generated skill MEASURES -- the per-bundle denominators, the breaking point,
the conditional-field list -- is not something any static pairing check reaches.
"""
from __future__ import annotations