llm-ingestion-okf/docs/2026-09-09-o6-forsteskjerm-og-skillmodi.md
Kjell Tore Guttormsen fbad5b0eab docs(o6): the measurement record, including the query that could not find
Restates the order's premises on HEAD (README is 679 lines, not 676; the
`okf project` install line is 320, not 317), records the defect the
reproduction turned up (`okf project` 15 concepts / 30 files against
`okf build`'s 26 / 52, one signature default apart), and states what the
three end-to-end runs and the two-bundle routing measured with the
denominator beside each figure.

Two of this repository's own published figures are corrected: O5's "4
absolute paths -> 0" was measured with `grep -c "^/"` against paths indented
by two spaces, so the zero was never a measurement; and the order's own
"0 occurrences of hypothes/task/write/produce" is false per term (`write` 4,
`produce` 7, all unrelated) even though the modes were genuinely absent.

Honesty limits name N = 1 per row, that routing is 2 questions over 2
bundles and worked on the bundle ID rather than the description, that run 2
overstated one negative (NS 8407 is in 3 of 52 files, all withheld), and
that run 3's document numbers are for a non-default payload it changed and
declared.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-09 18:31:04 +02:00

15 KiB
Raw Blame History

O6: the first screen an agent reads, three modes, and one flag that made two builds

Session 119, order 20260909T155135Z-8080812956-from-.claude, on 9567145. Commit 1260fac, tag v0.7.0 (local, unpushed).

Everything below is a measurement with the command beside it. Where a figure this repository published before is wrong, the correction is stated rather than quietly replaced. § 7 states what was not measured.

1 The red measurement, and the query that could not find

1.1 The order's own premises, restated on HEAD

The order's PM figures (i)(iii) were measured on an export of a364ef4, one commit behind HEAD. Restated on 9567145:

Premise PM on a364ef4 Measured on 9567145 Command
suite 1498 1498 pytest -q
ruff clean clean ruff check src tests tools
mypy clean, 21 files clean, 21 files mypy --strict src
template SKILL.md 164 lines 164 wc -l < skills/okf-consume-template/SKILL.md
a GENERATED skill 231 lines 231 okf skill …/expected-bundle --out <scratch>
README.md 676 lines 679 wc -l < README.md
okf project install line line 317 line 320 grep -n "@v0\." README.md
~/okf-test/dokumenter default 52 md 30 md — see § 2 okf project ~/okf-test/dokumenter

The last row is not a restatement. It is a defect, and it is § 2.

1.2 The pins disagreed

$ grep -n "@v0\." README.md
26:  … llm-ingestion-okf.git@v0.4.0
40:  … llm-ingestion-pipeline-security.git@v0.2.0
41:  … llm-ingestion-okf.git@v0.4.0
320: … llm-ingestion-okf.git@v0.6.0
$ grep -n "@v0\." llms.txt
10:  … llm-ingestion-okf.git@v0.4.0

An agent reading either file from the top installed v0.4.0, which has no okf project. llms.txt — the file an agent fetches first — named neither okf project nor Claude Code at all.

1.3 The query that could not find (face 4)

docs/2026-09-08-o5-okf-project.md lines 16 and 177 measure absolute paths in a generated skill with grep -c "^/" and report 4 → 0. The generator indents those paths by two spaces, so ^/ cannot match one. Control, run before any zero was read:

$ printf 'prose with no path\n  /Users/x/bundle\nokf consume /Users/x/other\n' > kp.txt
$ grep -c "^/" kp.txt                      # O5's query
0                                          # <- cannot find a known positive
$ grep -cE '(^|[[:space:]])/[A-Za-z]' kp.txt
2                                          # <- can

Applied to a generated skill on 9567145:

$ grep -nE '(^|[[:space:]])/[A-Za-z]' <skill>/SKILL.md
26:  /Users/ktg/repos/llm-ingestion-okf/examples/…/expected-bundle \
29:  --out /tmp/payload.json
41:  --skill /private/tmp/o6-red/skillgen/SKILL.md \
42:  --payload /tmp/payload.json
$ grep -n "$HOME" <skill>/SKILL.md
26:  /Users/ktg/repos/…

Two paths into the machine the skill was generated on (26 and 41), confirming the order's (iii). The other two are /tmp/payload.json, a scratch write target that names no checkout. O5's "0" was never a measurement — the query it used could not have produced anything else.

1.4 The modes, counted per term

The order's (ii) is confirmed in substance and corrected in form. Counted with grep -ic <term> skills/okf-consume-template/SKILL.md:

term template generated
hypothes 0 0
task 0 0
oppgave 0 0
notat 0 0
write 4 2
produce 7 15
confirm 1 1
refut 0 0

Every write and produce hit is unrelated ("the producer wrote none", "Write to <OUT>"). The modes were absent; the claim "0 occurrences of hypothes/task/write/produce" was still false as a measurement.

2 The defect the reproduction found: two builds, one command

okf project ~/okf-test/dokumenter wrote 15 concepts / 30 files. okf build on the same folder wrote 26 / 52. CLAUDE.md states the opposite — "it owns NO flag that moves a bundle's bytes and a test holds it byte-equal to okf build" — and a test by that name exists and was green.

Mechanism. cli.build's Python signature carried keep_table_heading: bool = False and sheet_section_rows: bool = False while argparse carried DEFAULT_KEEP_TABLE_HEADING = True and DEFAULT_SHEET_SECTION_ROWS = True. Every other flag reads its DEFAULT_ constant in both places; these two did not. project.create calls build() as a function and passes no flag list, so it read the signature. The whole 22-file difference was inside del-ii-bilag-7-prisskjema/ — the priced spreadsheet, which is the document a question about price has to reach.

Why the test could not see it. test_the_project_bundle_is_the_bytes_okf_build_writes compares project.create against the same build() function, so both sides carried the same wrong value; and its two fixture documents (krav.md, notat.md) contain neither a table nor a sheet, so the difference could not have appeared even against argparse. A test and the code agreeing over a set where the difference cannot appear.

Repair, red first. Two tests, both red before the change:

  • test_the_build_signature_defaults_are_the_build_command_defaults — compares the signature's default against argparse's for every parameter argparse passes through at the same type. Red output named exactly the two: {'keep_table_heading': (False, True), 'sheet_section_rows': (False, True)}. Parameters argparse converts (segments 'on'→bool, pdf_headings 'none'→bool, okf_version caller-owned) are excluded by the type test, not by a list.
  • test_a_sheet_reaches_the_project_bundle_as_it_reaches_the_build_command — the same folder through project.create and through okf_main(["build", …]), on a document whose concept count actually moves with the two flags. On the first draft (a two-row table) it passed; that draft is recorded here because it is the same failure as the invariant it replaces — a green over an empty set. With five numeric rows it went red (2 segments vs 7) and is the shipped form.

After. okf project writes 26 concepts / 52 files, and diff -rq against the okf build bundle of the same folder reports 0 differences.

3 A — the first screen

README.md now opens with, in this order: what it is in one sentence; one uv tool install line; okf project <folder> and claude; the three shapes of request; one line saying where the rest is. The phase-status paragraph that used to be lines 319 moved down under ## What this library is — moved, not deleted.

llms.txt was rewritten in the same shape: the new tag, the okf project form, the three modes, and a pointer into the Claude Code section.

Acceptance, measured:

$ grep -o "@v0\.[0-9.]*" README.md llms.txt | sed 's/.*://' | sort -u
@v0.7.0
$ printf 'x %s/y\n' "$HOME" > kp.txt; grep -c "$HOME" kp.txt   # control: can find
1
$ grep -c "$HOME" README.md llms.txt
README.md:0
llms.txt:0

Earlier tags are kept under ### Earlier tags, as history, labelled as history and not as commands. The guard pairing is measured, not guessed: v0.7.0's own [tool.uv.sources] names llm-ingestion-guard v1.3.0, and § 5 shows the install resolving exactly that.

4 B — three modes, relative paths

## Modes in the template, the instantiated skills/okf-consume/SKILL.md, and therefore in every generated skill. Question (the default), Hypothesis, and Task that produces a document. The five markings are untouched — a test asserts every literal okf check fixes is still present, so the modes added no sixth.

Relative paths: skill.project_root_of(out) reads <root>/.claude/skills/<name> off the path and returns <root>; skill.as_written(path, base=root) writes a path relative to it when it is under it, and leaves it absolute otherwise. A path outside the project root stays absolute on purpose: ../../.. is not more portable, only harder to read.

Measured on a bundle built by okf project in a cold directory:

$ grep -nE '(^|[[:space:]])/[A-Za-z]' .claude/skills/dokumenter-consume/SKILL.md
29:  --out /tmp/payload.json
42:  --payload /tmp/payload.json
$ grep -c "$HOME" .claude/skills/dokumenter-consume/SKILL.md
0
$ grep -n "\.okf/dokumenter\|\.claude/skills/dokumenter-consume/SKILL.md" <skill>
26:  .okf/dokumenter \
41:  --skill .claude/skills/dokumenter-consume/SKILL.md \

Two paths into the machine → 0. Both commands run from the project root:

$ okf consume .okf/dokumenter --question … --ref … --out /tmp/payload.json ; echo $?
0
$ okf check --skill .claude/skills/dokumenter-consume/SKILL.md --payload /tmp/payload.json
conformant: 15 rules over 8 excerpts and 18 withheld entries, 0 findings

5 The install measured from a cold, empty directory

uv tool install cannot point at an unpushed tag over git, so the tag was exported and installed from a local path. This is stated rather than glossed: the bytes are the tag's (git archive v0.7.0), the channel is not.

$ git archive v0.7.0 | tar -x -C /tmp/o6-install/export
$ uv tool install --force "llm-ingestion-okf[extract] @ file:///tmp/o6-install/export"
 + llm-ingestion-guard==1.3.0 (from git+…@44e2b31…)
 + llm-ingestion-okf==0.7.0 (from file:///tmp/o6-install/export)
Installed 1 executable: okf

The guard pairing the README states is what resolved. Then, in an empty directory, with no PYTHONPATH, no VIRTUAL_ENV, and this repository nowhere on PATH (which okf~/.local/bin/okf):

documents in 5
concepts out 26
files out 52 md
left out of the bundle 0 of 5
landed whole as one concept 1 of 5 (named in the output)
okf check conformant, 15 rules, 0 findings
paths into any checkout in the skill 0

Order of operations, as the order required: the tag was created (§ 6) before this was run, because the measurement is of the tag.

6 The three end-to-end runs (C1's instrument)

Skills installed in a project directory outside this repository; cd <project> && claude -p "<the question>"; one run per row, no re-run for a better answer. Sources grep-verified against the bundle tree with the denominator stated.

# Mode Asked Sources verified Denominator
1 question "hva er kravene til pris?" 8 of 8 sha256 prefixes found in the payload 8 excerpts in the payload
2 hypothesis "stemmer det at totalentreprenøren bærer risikoen for massehåndtering, og at massene prises som fastpris i basiskostnaden?" 2 of 2 sha256 prefixes; the verbatim quote found in 1 file 8 excerpts; 52 md files in the bundle
3 document task "lag krav-pris.md med alle krav til pris, ett avsnitt per krav, med dokument og kravnummer" 28 of 28 64-hex digests; 24 of 24 concept paths exist on disk 80 known digests (excerpt fields + 52 concept files + 5 source documents + ref)

Invented numbers: 0 across the three runs, against the denominators above. Every digest and every concept path a run cited was found.

Run 2 answered per premise, which is what the mode asks for: premise 2 refuted with the contradicting sentence quoted, premise 1 split into mengderisiko (refuted, derived), enhetsprisrisiko (confirmed, extracted), the tiltaksklasse question undecidable-from-bundle marked [sourced-not-sufficient] on that premise, and the hypothesis-level verdict stated as derived. No fourth verdict value appeared.

Run 3 wrote the document. 221 lines, 37 requirement paragraphs, and — the part that matters — the discipline is in the file: 24 extracted, 18 derived, 3 [unverifiable-from-bundle], 1 [unread], 2 [sourced-not-sufficient] standing where the requirement belongs rather than dropped, the bundle ref, and considered / withheld / delivered declared in the document's own header table. It also reported that req_number is 0 of 26 in this bundle and marked its own numbering derived rather than inventing requirement numbers — which is the failure the mode exists to prevent.

7 C — two bundles in one project

Two okf project runs with different --id on two topically different document sets, split from ~/okf-test/dokumenter into scratch (~/okf-test read-only):

.okf/pris      13 concepts   ref sha256-tree:477ed72de1d3be18…
.okf/teknikk   10 concepts   ref sha256-tree:e53bed17f7b8a5e2…
.claude/skills/pris-consume   .claude/skills/teknikk-consume

Both exist side by side with different refs. The order's (vii) is confirmed on the descriptions: the two description: lines differ only in the bundle id, the concept count and the ref — there is no topic signal in either.

Routing measured with no skill named in the prompt, one run per question:

Question Belongs to Chosen Ref in the answer
"hva kreves ved prøvedrift?" teknikk teknikk e53bed17f7b8a5e2…
"hva er påslagsprosenten for materialer?" pris pris 477ed72de1d3be18…

2 of 2. The first run's output names both bundles before settling on teknikk, so the choice was made rather than defaulted to. No topic signal was added to the generator, because the measurement did not call for one: the bundle id carried enough. That is a property of ids chosen meaningfully, not of the description line, and § 8 says so.

8 Honesty limits

  • N = 1 per row in § 6 and § 7. Four claude -p runs total, no re-runs. A single run is not a rate.
  • Routing is 2 questions over 2 bundles, on one corpus split in two. It was not measured for more than two bundles, for topically overlapping bundles, or for ids that carry no topic (alfa/beta). The mechanism that worked was the id, and a caller who passes --id a and --id b has removed it. The description line still carries no topic signal.
  • Run 2 overstated one negative. It wrote "NS 8407-bestemmelsene er ikke i bundelen". grep -ril '8407' .okf/dokumenter --include='*.md' finds 3 of 52 files (control: pris matches 21 of 52, so the query can find). All three are in withheld concepts, and the answer did name dokument-for-avtaleinngåelse/tabell-linje-30 as the one withheld concept that might carry contract terms — so the coverage limitation was reported and the sentence above it was still stronger than what was measured.
  • Run 3 changed the instrument mid-task and said so. It re-ran the pre-pass with --k 26 and --cost-vocabulary after the default k=8 delivered 8 of 26, and declared both cuts in the file. The document's numbers are therefore for a non-default payload; the § 6 default figures are the ones for the shipped configuration.
  • The install in § 5 came from a local export, not a git URL. The tag is unpushed by instruction, so the channel differs from the README's line even though the bytes do not.
  • No pptx was in any measured folder. The pptx fasit stands where round 7 left it.
  • The K2 reference bundle was not rebuilt. This session changed no segmentation rule and no proposer default; the cli.build signature change affects only callers who used the Python API without passing the two flags, of which project.create was the one in this repository. tests/test_default_bundle_pin.py is untouched and green.