docs(consume): the Claude Code recipe, measured end to end on two bundles

Four questions, two bundles, one run each, in a scratch project outside this
repository with a generated skill per bundle. All four passed, and zero numbers
or identifiers appeared in any answer that were not in the delivered set or in
the payload's own identities (62, 45 and 35 unique numeric tokens checked).

The skill triggered WITHOUT being named in the prompt and selected the right one
of two installed skills from the question alone, so no special invocation syntax
is needed: the generated `description`, which carries the bundle id, the concept
count and the ref, is enough to route on.

One defect the runs found, and it was in the prose rather than the payload. The
citation guidance listed the four locator keys this library writes, so on the
270-concept third-party bundle the model reported "no page locator, the address
is at document level" while the excerpt in front of it carried
`source_element_id` - that bundle's own locator, correctly delivered by the
prefix rule. The guidance now tells the reader to cite whichever `source_*` keys
are present. On the re-run the same question returned the element id. Two runs
of one question, the second measuring a changed artefact and not retrying the
first.

One finding that is not a defect in this chain: the first attempt at a
known-negative was not one. The bundle covers water and frost protection on 17
of its 270 concepts and the ranker put none of them in the cut. The consumer
behaved exactly as the contract asks - refused, named its denominator, reported
its own zero as unmeasured because `withheld` entries carry no titles, and did
not go around the cut. Recorded as a retrieval miss rather than replaced, and
it is the same shape as the open fusion finding.

A correction to this session's own measurement is in the record too: a first
sweep used `grep -rhoE "^source_[a-z_]+:"`, whose character class excludes
digits, and so missed `source_sha256` on 270 of 270 concepts. A pattern that
cannot match what it is looking for returns a zero that reads like a fact.

README gains "Consume in Claude Code": folder to answer in three commands, every
one of them run in this session. A test holds that the recipe invokes only
scripts this repository ships, at the paths it names.

Suite 1373 (1339 at the session baseline), ruff clean, mypy src clean. No
version bump, no tag, no push.

Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 15:32:32 +02:00
commit 171798ed32
5 changed files with 477 additions and 15 deletions

View file

@ -191,6 +191,67 @@ hit@8 was **5 of 6** questions at rank 1 against a chance baseline of **1.35 of
6** — with one control that failed, and both are in
`docs/2026-09-07-okf-konsumskill-maaling.md` with the honesty limits stated.
## Consume in Claude Code
A folder of documents to an answer a model can cite, in three commands. Every
command below was run end to end on 2026-09-08 against a nine-document folder
and a 270-concept third-party bundle; nothing here is untested.
```sh
SRC=/tmp/c1-fresh-src # the folder of documents
BUNDLE=/tmp/c1-fresh-bundle # where the OKF bundle goes
PROJECT=/tmp/c1-scratch # the project you will ask the question from
```
**1. Build the bundle.**
```sh
okf build "$SRC" --bundle "$BUNDLE" --bundle-id c1-fresh-20260908 --okf-version 0.2 --ingested-at 2026-09-08T00:00:00Z
```
**2. Generate a skill for that bundle**, straight into the project's skill
directory. The skill is instantiated for these bytes: its id, ref, concept
count, per-field denominators, whole-bundle cost and breaking point are all
measured from the bundle, and it ships a reference payload the checker accepts.
```sh
python3 tools/okf_skill.py "$BUNDLE" --out "$PROJECT/.claude/skills/c1-fresh-20260908-consume"
```
Repeat for every bundle you want reachable; each one gets its own skill named
after its `bundle_id`, which is what lets a model pick between them. A bundle
you only have read access to is fine — the generator only reads it.
**3. Ask.** From `$PROJECT`, in Claude Code:
```sh
claude -p "Hvordan skal prisene fylles ut?"
```
Measured with two bundles installed side by side: the model selected the right
skill from the question alone, ran the pre-pass and the contract check itself,
quoted the requirement verbatim, and named the document, the requirement number,
the source resource and the locator inside it. Across three questions, **0**
numbers or identifiers appeared in an answer that were not in the delivered set
or in the payload's own identities. On a question the bundle does not cover it
answered `[sourced-not-sufficient]` and reported the denominator rather than
inventing an answer.
The pre-pass and the checker are the same two commands the skill runs for you,
if you want to see the payload first:
```sh
python3 tools/okf_consume.py "$BUNDLE" --question "your question" --out /tmp/payload.json
python3 tools/okf_contract_check.py --skill "$PROJECT/.claude/skills/c1-fresh-20260908-consume/SKILL.md" --payload /tmp/payload.json
```
The honest limits: this was measured on **four questions** across two bundles,
which is a demonstration and not a hit rate. The ranking is lexical, and one of
the four found a topic the bundle **does** cover and did not rank it into the
cut — the skill then said so with its denominator instead of answering, which is
the behaviour the contract asks for, but a miss is still a miss.
`docs/2026-09-08-claude-code-skill-vilkaarlig-bundle.md` has the runs.
## Implemented scope (v1)
The library provides three entry points for getting content into an OKF