Kjell Tore Guttormsen
e15de71aa4
Round 16's partition read every concept whose WHOLE title the question accounts for before everything the fusion ranked above it. That is a claim about the covered title's PRECISION, and it overrode the fusion even against a title answering MORE of the question. Measured on a 26-concept bundle of five documents: the question names a section by three title tokens and holds a neighbour's whole one-token title (1 of 9 question tokens); the fusion put the named section at rank 1, the partition moved the neighbour over it. A covered concept now RISES through the fusion's order and stops beneath the first concept whose title answers more question tokens, by equality, than it holds, or beneath a covered concept the fusion put above it. With nothing above it answering more it reaches the top exactly as before. Same flag (--title-covered / --no-title-covered), no new parameter, no new constant. Measured before this commit, delivered ranks from build_payload: - known-negative: rank 2 -> 1; the payload equals --no-title-covered's - three own probes on that bundle: 1, 1, 1 (unchanged from round 16) - R761 XML, 2 761 concepts: hit@1/8/50 6/6 - 6/6 - 6/6 at default k and at --k 50, KP rank 1; 8 of 8 payloads byte-identical to7cca9e0at BOTH k - payloads byte-identical to7cca9e0: K2 pinned 6/6, Arm B 6/6, N100/N200/ N500 15/15; tests/test_default_bundle_pin.py 7 passed, file untouched - candidates measured beside it: min title length (R761 hit@1 3/6), share of the question (holds only for 1/9 < s <= 1/6), order inside the group (group of one: no effect), stop list (no title involved is one) Suite on the staged set: 1600 passed, 1 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1019 lines
61 KiB
Markdown
1019 lines
61 KiB
Markdown
# Changelog
|
||
|
||
All notable changes to this project will be documented in this file.
|
||
|
||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||
|
||
## [Unreleased]
|
||
|
||
### Added
|
||
|
||
- **`okf check` refuses a skill and a payload that name different bundles
|
||
(`bundle_mismatch`).** The checker had published this hole about itself since
|
||
2026-09-08 and not closed it: reproduced on this repository's HEAD, it
|
||
reported `conformant: 15 rules over 8 excerpts and 438 withheld entries, 0
|
||
findings` for a skill generated from one corpus against a payload assembled
|
||
from another; the same line for the UNFILLED template against that payload;
|
||
and the same line again for a payload sharing the skill's `bundle_id` at a
|
||
foreign `ref`. All three now exit **1** with one finding. The right pair is
|
||
untouched at exit 0 with 0 findings, and a payload declaring no identity at
|
||
all stays `ref_missing`'s defect at 9 findings -- no rule restates another.
|
||
- **BOTH halves are compared, and the `ref` half is the load-bearing one.**
|
||
Three distinct builds on one machine were measured carrying the same
|
||
`bundle_id`, so an id comparison alone would pass a stale skill -- the case
|
||
the generated skill warns about in its own words ("if the bundle moves, the
|
||
ref moves with it and this file is stale"). SS 3.3: "a version is the
|
||
producer's assertion; a ref is a fact about bytes".
|
||
- **An identity the rule cannot read is a finding, never a silent pass.**
|
||
That is what refuses the unfilled template, whose `<CORPUS>` and `<REF>`
|
||
are not an identity. It also refuses the repository's own hand-made
|
||
`skills/okf-consume/SKILL.md`, which predates `okf skill` and declares no
|
||
bundle identity a reader can act on -- **1 of 1** shipped hand-made skill,
|
||
a real find and not a fixture.
|
||
- **The rule compares a DECLARED identity against a DECLARED identity and
|
||
never opens the bundle**, so a payload misreporting its own `ref` still
|
||
passes. Proving a ref against bytes is `okf consume --ref`'s job and needs
|
||
a bundle path this command deliberately does not take.
|
||
|
||
### Changed
|
||
|
||
- **The checker's rule count is 16, not 15**, and `Report.rules_evaluated` is
|
||
the denominator every report line quotes -- so `15 rules` becomes `16 rules`
|
||
in every published line. A consumer citing the old number is citing a number
|
||
that has changed. No payload bytes move: this is the checker, not the
|
||
pre-pass.
|
||
- `skill.identity_line` is now the single authored copy of the sentence a
|
||
generated skill declares its bundle in, read back by
|
||
`contract_check.skill_identity` and held to it by a test. Generated skill
|
||
bytes are unchanged -- measured, both tracked bundles byte-identical before
|
||
and after on the same interpreter.
|
||
|
||
### Fixed
|
||
|
||
- **`--title-covered` no longer lifts a short title over a title that answers
|
||
more of the question.** 0.8.1's partition read every concept whose WHOLE
|
||
title the question accounts for before everything the fusion ranked above
|
||
it. That is a claim about the covered title's PRECISION, and it overrode the
|
||
fusion even against a title answering MORE of the question: measured on a
|
||
26-concept bundle of five tender documents, a question naming a section by
|
||
three of its title tokens also held a neighbour's whole one-token title, and
|
||
the neighbour took rank 1 from the section the question names. A covered
|
||
concept now RISES through the fusion's order and stops beneath the first
|
||
concept whose title answers more question tokens, by equality, than it
|
||
holds. Same flag, no new parameter, no new constant.
|
||
- **What the rule was built for does not move.** On the 2 761-concept bundle
|
||
of one standard no covered concept had such a title above it, so all 8
|
||
payloads are byte-identical to 0.8.1's at default `k` AND at `--k 50`;
|
||
hit@1/8/50 stays 6/6 · 6/6 · 6/6 with the known-positive at rank 1.
|
||
- **Where the rule never fires nothing moves either**, measured on the bytes:
|
||
the pinned K2 bundle 6 of 6 payloads identical, Arm B 6 of 6, the three N
|
||
bundles 15 of 15.
|
||
- **Four other repairs were measured and not taken**: a minimum title length
|
||
sold hit@1 back to 3 of 6; a share of the question holds only in a band
|
||
set by the question's word count (1/9 < s <= 1/6); an order inside the
|
||
covered group cannot act on a group of one; a closed stop list touches no
|
||
title involved and would be a new vocabulary to maintain.
|
||
- **0.8.1's unbounded order is reproducible by no flag.** It differs from
|
||
this one only where a covered concept has such a title above it -- 1 of
|
||
the 4 questions measured on that bundle, 0 of 8 on the standard, 0 of 27
|
||
elsewhere. `--no-title-covered` still reproduces the pre-0.8.1 order.
|
||
- **One constructed variant still reads the short title first, and no rule
|
||
reading titles alone separates it.** A shortened form of the same question
|
||
shares ONE token with each of the two titles; a form that blocks on any
|
||
token the covered title lacks fixes it and takes one of the standard's
|
||
scored questions, the same shape with the opposite answer, from rank 1 to
|
||
3. Report: `docs/2026-09-11-k3-runde17-dekningen-stopper-ved-en-bredere-tittel.md`.
|
||
|
||
## [0.8.1] — 2026-09-10
|
||
|
||
### Added
|
||
|
||
- **`--title-covered` (ON by default, opt out with `--no-title-covered`): a
|
||
question that accounts for a concept's WHOLE title reads that concept first.**
|
||
On the 2 761-concept bundle of one standard, the answering section was
|
||
delivered at rank 1 on **3 of 6** scored questions and **none of the reading
|
||
side's six flags moved that number** -- the whole sweep sits at 3/6 or worse.
|
||
Measured on that bundle, before and after: hit@1/8/50 **3/6 - 5/6 - 5/6 ->
|
||
6/6 - 6/6 - 6/6** at default `k` and **3/6 - 5/6 - 6/6 -> 6/6 - 6/6 - 6/6** at
|
||
`--k 50`, with the known-positive holding rank 1 at both and the
|
||
known-negative still not a hit. S1 4 -> 1, S5 not delivered -> 1, S6 3 -> 1.
|
||
- **THE DEFECT IS THAT BOTH LEXICAL SIGNALS ARE UNNORMALISED COVERAGE
|
||
COUNTS.** They measure how much of the QUESTION a candidate answers and
|
||
nothing measures how much of the CANDIDATE the question accounts for, so a
|
||
section titled with the question's subject alone scores what a narrower
|
||
section titled with that subject PLUS a qualifier scores, and then loses on
|
||
the body count. Decomposed per miss: S1 turns on `hvordan`, an interrogative
|
||
pronoun; S5 on `hvilke` and `stilles` in a body 7x the gold's, and on
|
||
`betonghvelv ~ betongkonstruksjoner` through the four-character stem
|
||
`betong`; S6 on an exact tie broken by `concept_id`.
|
||
- **A PARTITION, never a fourth RRF signal, and the arithmetic is why.** RRF
|
||
consumes ranks alone, so with shared ranks a rule whose positive group has
|
||
`m` members is worth `1/61 - 1/(61 + m)` -- a rule firing on ONE concept of
|
||
2 761 is worth 0.00026 against a body gap of 0.0029. **A precise rule is
|
||
worth LEAST under this fusion.** Measured as a signal it moves hit@1 not at
|
||
all (3/6, both as a third and as a fourth signal); as a partition it reaches
|
||
6/6. `lookup_hits` is the same shape for the same measured reason, and it
|
||
still wins: the new partition lands below it, with a test and its control.
|
||
- **By EQUALITY, never by shared prefix.** Four shared leading characters take
|
||
the group from 1 to 6 on one question and 9 to 31 on another, with the
|
||
answering section falling to candidate rank 6 and the known-positive to 2.
|
||
- **TWO CANDIDATE REPAIRS WERE MEASURED AND FELLED FIRST.** Pivoted length
|
||
normalisation of the body signal collapses at every value swept
|
||
(b = 0.25/0.5/0.75/1.0 -> hit@8 3/6, 1/6, 1/6, 0/6, and at b = 1.0 the
|
||
known-positive falls to rank 49): the median concept holds 22 tokens against
|
||
a mean of 60, so length normalisation promotes thousands of tiny concepts.
|
||
Title PRECISION as a signal reaches candidate hit@1 5/6 and takes the
|
||
known-positive from 1 to 4 every time it does.
|
||
- **NOTHING ELSE MOVES AND IT IS MEASURED ON THE BYTES.** The pinned K2 bundle
|
||
keeps `(1,1,1,1,1,5)` and its 7 pin tests, Arm B keeps `(1,1,1,1,1,5)`, and
|
||
the payloads on both are **byte-identical on 6 of 6 questions**; n100/n200/
|
||
n500 payloads are byte-identical on 5 questions each; the 828-file HTML
|
||
corpus still gives 828 plans, 0 unreadable and 6 015 md with `diff -rq`
|
||
empty; the five-document folder is `diff -r`-identical at 26 concepts / 52
|
||
md; `okf project` stays byte-equal to `okf build`. hit@k on n100/n200/n500
|
||
is **NOT MEASURED** -- this repository holds no gold set for them, which is
|
||
0 gold sets and not 0 hits.
|
||
- **THIS IS THE FOURTH READING-SIDE CHANGE THAT MOVES A PAYLOAD WITH NO BUNDLE
|
||
CHANGING.** A consumer pinned to the previous excerpt order needs
|
||
`--no-title-covered`. The rule fires on **0 of 21** measured cells outside
|
||
that one bundle, so "no regression" there means it never fires -- not that
|
||
it fires harmlessly. Report:
|
||
`docs/2026-09-10-k3-runde16-hele-tittelen-tar-ruten.md`.
|
||
- **A MEASURED DOWNSIDE, written as a known limitation and not as a fixed
|
||
defect: a SHORT, GENERIC title is covered in full by more questions than a
|
||
long one is.** On a five-document folder (26 concepts) a constructed
|
||
known-negative question demoted the answering section from delivered rank 1
|
||
to rank 2: a neighbouring concept titled with a single common process word
|
||
has its WHOLE title accounted for by that question, while the answering
|
||
section's longer title does not. The other seven delivered places did not
|
||
move. The repair is a later round's; nothing here narrows the rule, and the
|
||
opt-out is `--no-title-covered`.
|
||
- `build_payload`'s signature defaults are now held equal to the consume CLI's
|
||
argparse defaults by a test, for every same-named parameter. This is O6's
|
||
defect in the other command: `cli.build` defaulted two flags `False` in the
|
||
signature and `True` in argparse, and a caller reaching it as a function read
|
||
the signature.
|
||
|
||
## [0.8.0] — 2026-09-10
|
||
|
||
### Added
|
||
|
||
- **A section the SOURCE DECLARES now takes the route declared structure takes,
|
||
at the shipped defaults.** `.xml` gained a reader in the entry above and the
|
||
reader reached its ceiling -- **2 761 of 2 761** heading lines -- while the
|
||
build delivered **23 concepts and 15 of 2 761 boundaries**. Everything after
|
||
the reader ate it, and both steps are measured: the **orphan check** removed
|
||
**710 of 2 761** (710 of 710 removed headings are followed immediately by
|
||
another heading and **0 of 2 051** delivered ones are -- they are container
|
||
sections), and **Arm F** folded **2 066** more, 2 089 -> 23.
|
||
- `extract.xml_outline` reports the marks the reader wrote itself. There is
|
||
**no bridge** and therefore no tolerance constant and no `unresolved`
|
||
bucket: the reader appended the line it names. That is the difference from
|
||
`pdf_outline`, whose naive nearest-line rule was wrong on 1 840 of 2 762.
|
||
- `propose.RULE_XML_SECTION` (`rule:xml-section`) is its own name in
|
||
`RULE_NAMES` and in `_ORPHAN_EXEMPT`, so an artifact still distinguishes an
|
||
element the reader transcribed from a bridged bookmark
|
||
(`rule:pdf-outline`) and from a heading somebody guessed (`rule:heading`).
|
||
- The route is chosen by the ROW (`DECLARED_STRUCTURE_IDS = {"xml"}`), never
|
||
by the text: the same markdown arriving from a `.md` file is still a guess.
|
||
**No other file type changes one byte** -- `diff -r` on the five-document
|
||
reference folder is empty (52 md, 26 concepts, 0 of 5 rejected, 0 `.xml`
|
||
files in it), `okf project` is still byte-equal to `okf build`, the pinned
|
||
K2 bundle is unchanged, and the PDF arm still proposes 2 762 segments.
|
||
- Measured at SHIPPED DEFAULTS, not behind a flag: **2 761 concepts**;
|
||
**2 761 of 2 761** declared sections became a concept with the source's own
|
||
directory and title; **0** concepts match no declaration; `a)`-points
|
||
**0 of 4 954**; table blocks **10 of 10**; hit@1/8/50 **3/6 · 5/6 · 6/6**
|
||
(from 0/6 · 0/6 · 0/6) with the known-positive at rank 1. Cross-arm,
|
||
**2 761 shared concept ids** -- 100 % of the XML bundle and 2 761 of 2 762
|
||
of the PDF arm's, up from round 13's 2 022.
|
||
- Two directories of 2 738 still hold two concepts (`11`, `12`): the
|
||
publisher reuses a section number for two distinct sections, and it is the
|
||
same 2 the PDF arm has. Round 13's 14 such directories were false positives
|
||
of the text route reading the document's own contents listing, and they are
|
||
gone.
|
||
|
||
- **`.xml` is a core file type, NISO-STS aware, with a generic fallback.** A
|
||
publisher's own viewer delivers a zip that holds 0 html, 1 xml and 109
|
||
images; `okf build` on it was **110 of 110 unreadable, 0 plans, exit 2**, and
|
||
the conservation identity `merged + coded rejections == N` was never written
|
||
because the run aborted earlier. The one xml file is the whole product: 7 715
|
||
`<sec>`, **2 761 with a `<title>`**, 4 954 lettered points, 10
|
||
`<table-wrap>`, and a `<sec>`-nesting depth distribution row-for-row
|
||
identical to the publisher's own structure fasit.
|
||
- The output grammar is MARKDOWN, the same the office and HTML rows reach the
|
||
proposer through: `propose.py` is untouched. `<label>` + `<title>` become
|
||
one ATX line at the section's own depth; a `<sec>` with only a `<label>` is
|
||
a body line and never a heading (**0 of 4 954** became concepts);
|
||
`<table-wrap>` becomes one markdown table (**10 of 10**, against 0 of 10 on
|
||
the PDF path).
|
||
- The reader emits **2 761 of 2 761** heading lines and preserves text
|
||
exactly -- 1 283 395 of 1 283 395 non-whitespace characters, ratio
|
||
**1.000000**. The BUILD reaches 2 065 of 2 761 with `--no-unit-fold` and 15
|
||
of 2 761 on the shipped defaults; the whole distance is two proposer rules,
|
||
decomposed with denominators in the report.
|
||
- hit@k over six questions, k=50: **3/6 · 5/6 · 6/6**, matching the PDF arm
|
||
row for row, with the known-positive moving from **rank 13 to rank 1**.
|
||
2 022 concept ids are shared between the two channels -- 96.8 % of the XML
|
||
bundle.
|
||
- **No new dependency:** `xml.etree.ElementTree` is stdlib and `uv.lock` is
|
||
untouched. A `<!DOCTYPE` is REFUSED unparsed with its own code, which is a
|
||
guarantee about this package rather than about the installed libexpat.
|
||
- `.xml` never routes through the converter, and it is measured about 12x
|
||
faster and about 30x smaller in peak memory than the PDF arm on the same
|
||
document and the same machine.
|
||
- **XML that is not STS gives 0 plans and a FAILED build, and that is not an
|
||
`.xml` defect.** The known-positive that decides it: a folder holding one
|
||
`.txt` of prose with no headings gives exactly the same three lines and the
|
||
same exit 2. This is general `okf build` behaviour for any structureless
|
||
document -- extraction works, 0 unreadable, the text is there, and the
|
||
proposer has nothing to propose. The gate that refuses a run with no plans
|
||
stays: a run replaying zero plans would emit a flat bundle and report it as
|
||
a success. Separating "0 plans, 0 unreadable" from "0 plans because nothing
|
||
could be read" would change the outcome on **0 of the 4** reference
|
||
corpora, so it is not separated.
|
||
- Report: `docs/2026-09-11-k3-runde13-xml-sts.md`.
|
||
|
||
### Fixed
|
||
|
||
- **A PDF bookmark sharing a line with another left no trace.** `pdf_outline`
|
||
collected marks in a dict keyed on the destination line index, so a second
|
||
bookmark on a line was discarded by `setdefault` in silence: measured on a
|
||
701-page document, **2 763 nodes in, 2 762 marks out, `unresolved` = 0**.
|
||
`PdfOutline` now carries `collided`, and the identity `nodes in == marks +
|
||
unresolved + collided` holds. Keeping both nodes was measured and felled --
|
||
the two candidates then open at one offset and the first closes with an empty
|
||
span the orphan check deletes.
|
||
|
||
- **`--pdf-outline` (OFF): cut a PDF at the boundaries its own `/Outlines`
|
||
bookmark tree declares.** Measured outside this repository on one 701-page
|
||
process code whose publisher also ships a NISO-STS structure for it: the
|
||
shipped default recovers **1 967 of 2 761** titled sections, **0 of its 28**
|
||
chapters, and **794 of 794** misses have their heading text present in the
|
||
extracted text -- the line was read, the boundary was never opened. The same
|
||
file carries a 2 763-node bookmark tree that matches **2 761 of 2 761** STS
|
||
titles exactly after `re.sub(r"\s+","",s).lower()`. With the arm on:
|
||
**2 759 of 2 761 boundaries (99.9 %)**, depth 1 **28 of 28**, concept titles
|
||
identical to the publisher's own after that normalisation **2 761 of 2 761**,
|
||
false positives **3 of 2 762** (was 163 of 2 182), directories carrying two
|
||
concept files **2** (was 132, of which 65 were a contents copy and a body
|
||
section under one id), front-matter concepts **2 of 2 762** (was 72). Seven
|
||
of seven consumption fasit now exist in the bundle (was four); hit@1/8/50 is
|
||
**3/6 · 5/6 · 6/6** against **1/6 · 2/6 · 4/6**.
|
||
- It is a SEGMENTATION arm, not a reader option: the extracted text is byte
|
||
for byte the same either way, and `--pdf-headings`/`--ocr` stay the only
|
||
two things that change what a PDF says.
|
||
- A PDF with no bookmark tree is **byte-identical with the flag on**;
|
||
`pdfminer`'s `PDFNoOutlines` is "this file has no index", never an error.
|
||
Measured on the five-document smoke folder: `diff -r` empty against both
|
||
the arm off and the pre-change tree.
|
||
- No new dependency and no second parse of the file's pages: the tree is read
|
||
through `pdfminer.six`'s `PDFDocument.get_outlines()`, which
|
||
`pdfplumber` already ships under the existing `[extract]` extra. Cost on
|
||
the 701-page document: 119.22 s -> 183.31 s wall, peak RSS 3 252 -> 3 251
|
||
MiB.
|
||
- An unresolvable `/Dest` is dropped and COUNTED, never fabricated into a
|
||
boundary and never a refusal of the file. That document has 0 of 2 763;
|
||
one of the eight reference PDFs in this repository's own sample has 2 of 2.
|
||
- **The default does not move in this release.** Reach measured: 1 of the 8
|
||
reference PDFs carries a usable tree at all.
|
||
|
||
- **`.pdf` has a row in `extract._EVIDENCE`, as `measured`.** It was the row
|
||
with the most measurement behind it and no entry in the table, which is the
|
||
one way a table like that misleads while every entry in it is true.
|
||
|
||
|
||
## [0.7.0] — 2026-09-09
|
||
|
||
The first screen an agent reads, the three shapes of request the skill answers,
|
||
and one defect that made the command the first screen recommends build a worse
|
||
bundle than the command it claims to be.
|
||
|
||
> **Note for path importers, added 2026-09-09 after a report from a consumer.**
|
||
> `tools/okf_consume.py` ends by replacing its own `sys.modules` entry with the
|
||
> packaged `llm_ingestion_okf.consume`. A caller importing it with
|
||
> `importlib.util.spec_from_file_location` holds the object `module_from_spec`
|
||
> returned, which that line does not reach: read the module back out of
|
||
> `sys.modules[<name>]` after `exec_module`, or import `llm_ingestion_okf.consume`
|
||
> directly. From the commit that adds this note the file also copies the
|
||
> implementation's public names into its own globals, so a path-imported object
|
||
> carries them -- but that restores attribute ACCESS only, never patch-through.
|
||
|
||
### Fixed
|
||
|
||
- **`okf project` built a bundle two rules behind `okf build`.** `cli.build`'s
|
||
Python signature defaulted `keep_table_heading` and `sheet_section_rows` to
|
||
`False` while argparse defaulted both to `True`; `project.create` calls
|
||
`build()` as a function and passes no flag list, so it read the signature.
|
||
Measured on a five-document folder: `okf project` wrote **15 concepts / 30
|
||
files** where `okf build` on the same folder wrote **26 / 52**, and the whole
|
||
difference was in the priced spreadsheet — the document a question about price
|
||
has to reach. The invariant test that was supposed to catch this could not:
|
||
it compared `project.create` against the same `build()` function, so both
|
||
sides carried the same wrong value, and its two fixture documents had neither
|
||
a table nor a sheet. Both gaps are now tests: one compares the signature's
|
||
defaults against argparse's, the other builds a document whose concept count
|
||
actually moves with the two flags. After the fix the two paths are byte-equal
|
||
on that folder (`diff -rq`, 0 differences).
|
||
|
||
### Added
|
||
|
||
- **A first screen for a reader who has not used this before**, agent or human:
|
||
what it is in one sentence, one install line, two commands, and the three
|
||
shapes of request. The phase-status paragraph that used to open the README
|
||
moved down to `## What this library is`; nothing was deleted.
|
||
- **Three modes in the consumption skill**, stated in the template, the
|
||
instantiated skill and the generator:
|
||
- **Question** — as before, the default.
|
||
- **Hypothesis** — decomposed into premises and answered **per premise** as
|
||
`confirmed` / `refuted` / `undecidable-from-bundle`, three literals with no
|
||
fourth value. A premise whose excerpt is real but does not carry the
|
||
conclusion is `[sourced-not-sufficient]` on **that premise**, not on the
|
||
whole answer: four premises and one weak source is three answers and one
|
||
gap, and reporting it as one refusal throws the three away.
|
||
- **Task that produces a document or a paragraph** — every claim in the
|
||
written artefact carries `(bundle_id, concept_id)`, the excerpt's `sha256`,
|
||
its `title` and whichever `source_*` keys it has; an ungrounded paragraph is
|
||
**written and marked**, never dropped; and the cut (`considered`,
|
||
`withheld`, `delivered`) is declared inside the document, because the
|
||
document travels without the chat.
|
||
The five markings are untouched — the modes add no sixth.
|
||
|
||
### Changed
|
||
|
||
- **The generated skill states relative paths where it can.** In the layout
|
||
`okf project` writes, the commands are now `okf consume .okf/<id>` and
|
||
`okf check --skill .claude/skills/<id>-consume/SKILL.md`, runnable from the
|
||
project root — which is where `okf project`'s own closing line tells the
|
||
reader to start `claude`. A path outside the project root stays absolute on
|
||
purpose: `../../..` is not more portable, only harder to read. The two
|
||
absolute paths a generated skill carried are now zero, measured with a query
|
||
shown capable of finding first — O5's published "4 → 0" used `grep -c "^/"`
|
||
against paths indented by two spaces, which could not have matched either way.
|
||
- **One tag is pinned everywhere.** `README.md` pinned `v0.4.0` on its install
|
||
lines and `v0.6.0` further down, and `llms.txt` pinned `v0.4.0`; an agent
|
||
reading from the top installed a tag without `okf project`. All install lines
|
||
now name `v0.7.0`, and the earlier tags are kept as a labelled history
|
||
section rather than as commands. `llms.txt` gained the `okf project` form and
|
||
a pointer to the Claude Code section.
|
||
|
||
## [0.6.0] — 2026-09-08
|
||
|
||
The first tag since `v0.5.0a2`, so everything that had accumulated as
|
||
Unreleased is in it — those sections are kept below, under their own heading,
|
||
rather than folded together. What follows first is what O5 added, and why the
|
||
release needed a minor of its own: the installed command grew from one
|
||
subcommand to five, and one build default moved.
|
||
|
||
### Added
|
||
|
||
- **`okf consume`, `okf check`, `okf skill` and `okf project` are subcommands
|
||
of the installed `okf` command.** Until now the pre-pass, the contract
|
||
checker and the skill generator lived in `tools/` and were reachable only
|
||
from a clone; a consumer who installed this library could build a bundle and
|
||
had no way to read one back. Measured before the move: a consumption skill
|
||
generated from a checkout carried **four** lines naming that checkout by
|
||
absolute path, two of them the commands the skill tells a reader to run, so
|
||
the skill could not be moved, shared, or run by anyone else. The generated
|
||
skill now names `okf consume` and `okf check` — names on PATH — and a test
|
||
asserts the repository appears in it nowhere, with a known-positive so the
|
||
zero is a measurement rather than a search that could not find.
|
||
`tools/okf_consume.py`, `tools/okf_contract_check.py` and `tools/okf_skill.py`
|
||
remain as thin aliases, so every published reproduction block still runs.
|
||
- **`okf project <folder>`: a folder of documents to a bundle you can ask a
|
||
question of, in one command.** It runs `okf build` with this package's
|
||
default into `<out>/.okf/<id>/`, generates the skill into
|
||
`<out>/.claude/skills/<id>-consume/`, and prints what it read, what it wrote,
|
||
which documents are in the folder but not in the bundle, and which landed
|
||
whole as a single concept — the two cases a question can only be answered
|
||
`[sourced-not-sufficient]` in. `<out>` defaults to the current directory and
|
||
`<id>` to the folder's name reduced to `[a-z0-9-]`. It owns no flag that
|
||
changes a bundle's bytes, and a test holds the project bundle byte-equal to
|
||
the `okf build` bundle of the same folder.
|
||
- **`skills/okf-prosjekt/`**, a Claude Code skill (Norwegian) that wraps
|
||
`okf project` and reads its summary back.
|
||
- **The template and the contract document travel in the wheel.** `okf skill`
|
||
instantiates `skills/okf-consume-template/SKILL.md` and `okf consume`
|
||
measures `docs/consumption-contract.md` as its section 7.4 known-positive;
|
||
neither was installable before. Both are force-included from the file they
|
||
are authored in, so there is still exactly one copy of each.
|
||
|
||
### Changed
|
||
|
||
- **`okf build`'s default now includes Arm E (`--table-grid`), with
|
||
`--no-table-grid` as its opt-out.** The default moved to Arm D plus Arm F
|
||
earlier in the same day; measured afterwards, that combination is Arm F with
|
||
nothing to fold. The fold's table clause folds a table back into the heading
|
||
that introduces it, and with Arm E off a grid table is not one block but one
|
||
block per rule line. On the operator's twelve-document reference the shipped
|
||
default scored **2 of 12** and `docx` **0 of 3**, against the **5 of 12** the
|
||
fold was published with — which had been measured with Arm E on. Three arms
|
||
are now on by default, each with an explicit opt-out; `--outline-run 0
|
||
--no-table-grid --no-unit-fold` reproduces the pre-2026-09-08 bytes.
|
||
- **This tag does not make OKF v0.2 generally available.** `OKF_LATEST` is
|
||
unchanged.
|
||
|
||
### Also in this release: everything that had accumulated since v0.5.0a2
|
||
|
||
|
||
- **The corpus harness (`tools/okf_corpus_run.py`) can replay segmentation
|
||
plans, and it writes the bundle's `log.md`.** `--plans-dir` names the
|
||
proposals to replay and the profile follows from it; `--bundle-id` and
|
||
`--okf-version` are arguments, never constants, because a profile names a key
|
||
and the caller owns its value. `log.md` is written in SPEC section 9 form and
|
||
dated from `ingested_at`, so `merged + sum(coded rejections) == N` is
|
||
checkable from the bundle alone rather than only from a report that does not
|
||
travel with it. Without `--plans-dir` a run is unchanged.
|
||
- **The K2 rebuild that measured all of this.** The harness had passed
|
||
`STRUCTURED_V1` and no plans, so a 43-document corpus arrived as 39 flat
|
||
concepts with no `adjudication` key anywhere. Rebuilt, the same corpus yields
|
||
629 concepts, 618 of them `adjudication: proposed`. Record:
|
||
`docs/2026-09-03-k2-bundle-rebuild.md`.
|
||
|
||
- **`tools/okf_propose_segments.py` takes `--path-prefix`.** Section numbering
|
||
is document-local, so across 39 documents 618 proposed entries claimed only
|
||
601 distinct paths -- 17 collisions that Door B's gate refuses per document.
|
||
Scoping each document's entries under a caller-supplied prefix removes all
|
||
17. Without the flag every artifact already produced is byte-identical.
|
||
|
||
### Changed
|
||
|
||
- **A spreadsheet's tables are written as PIPE tables, so a row survives as a
|
||
row.** The converter's default markdown writer emits SIMPLE tables, which pad
|
||
every cell out to the width of the widest cell in its column. Measured on a
|
||
real 43-document corpus: one 594-character prose cell turned the sheet holding
|
||
the tender's prices into a 67 244-character whitespace carpet with runs of up
|
||
to **887 characters between a label and its amount**, and the header row named
|
||
a single column because only the first cell of the source's row 1 is filled.
|
||
The bytes reached a live consumer's model in two of eleven prompts and
|
||
appeared in none of its eleven answers. The same sheet through the pipe writer
|
||
is **11 048 characters with no whitespace run longer than two**, one row per
|
||
line, each source column its own cell. `--columns=1` is part of the fix and
|
||
not cosmetic: the pipe writer pads to a width computed from that setting, so
|
||
at the default a NARROW table gains runs of up to 45. Spreadsheet-only: the
|
||
other four office rows have the same defect available to the same one-line
|
||
change, but a spreadsheet is a grid with no prose fallback, while moving the
|
||
prose rows would move a corpus denominator nothing has measured.
|
||
`tests/test_extract.py` pins that scoping with three digests. Record:
|
||
`docs/2026-09-08-prisform-og-loggen-k2.md`.
|
||
|
||
- **An integral spreadsheet number loses the converter's trailing `.0`, and the
|
||
shared string table is what makes that safe.** The converter renders a numeric
|
||
cell as a double, so `5647500` arrives as `5647500.0` -- and a TEXT cell
|
||
reading `92.0` arrives as `92.0` too, which the output alone cannot tell
|
||
apart. The rewrite is bounded to a table cell whose entire content is such a
|
||
number, by unescaped pipes on both sides, and it is skipped whenever the same
|
||
literal is in the workbook's shared string table. Any workbook this cannot
|
||
read keeps its converter decimals rather than being guessed at.
|
||
|
||
- **The bundle's root `index.md` no longer links `log.md`.** The harness added
|
||
that link (`95eb271`) so a reader entering at `index.md` could reach the one
|
||
file carrying `N`; it was a LOCAL choice and said so. Consumption contract
|
||
SS 9.2 forbids a consumer from enumerating the bundle directory unless the
|
||
named profile says the index is derived, which makes the index tree the entire
|
||
map a consumer may use -- so everything it links is a document. Measured: a
|
||
consumer walking a 629-concept bundle that way returned **630**, and a corpus
|
||
run's own log became readable and citable as content. `5a0c879` excluded
|
||
`log.md` from OUR walk, which fixed the count on one side of a disagreement
|
||
produced on the other. The log is still written to the bundle root, which is
|
||
where SPEC section 9 puts it, and `tools/okf_consume.py` still excludes a
|
||
linked `log.md` -- every bundle built between `95eb271` and this change
|
||
carries the link.
|
||
|
||
- **The pinned `llm-ingestion-guard` moves to `v1.3.0`** (`[tool.uv.sources]`
|
||
and `uv.lock`; the `>=1.2,<2.0` range in `[project.dependencies]` already
|
||
covered it and is unchanged). What this fixes is that the guard could not
|
||
read back what this library WRITES: at `1.2.0` the flow-form `sources` in the
|
||
OKF v0.2 golden was refused outright, and flow is the only form this library
|
||
is able to emit, because its own line-oriented parser cannot round-trip the
|
||
block form. Measured before the bump so the test discriminates rather than
|
||
merely passes, and pinned by
|
||
`tests/test_guard_adapter.py::test_the_guard_parses_the_flow_form_sources_our_goldens_emit`.
|
||
Two rows of the gate table in `docs/okf-nokkelinventar.md` moved, not one:
|
||
the BLOCK form of `sources` now passes too, which retires G30 -- though it
|
||
changes nothing about what we emit, since our own parser is still the binding
|
||
constraint. `resource` is allowlisted only inside a `sources` entry, so
|
||
section 10.2's `executor`/`attester` resource stays rejected through every
|
||
carrier and the Door C boundary is unmoved. No public API of this library
|
||
changes; a consumer's cost is a re-run.
|
||
- **`uv.lock` also picks up `pypandoc-binary==1.17`**, which is a stale lockfile
|
||
being corrected rather than a new dependency: the package was already declared
|
||
in the `[extract]` extra, and `uv lock --check` reports the lockfile as out of
|
||
date on the untouched tree. Core still has exactly one runtime dependency.
|
||
- **`tools/okf_propose_segments.py` writes no artifact when it has nothing to
|
||
propose**, exiting `1` (distinct from `2`, "could not do the job") instead of
|
||
`0` with an empty plan. An empty plan cannot be replayed -- `process_inbox`
|
||
refuses one, because a plan naming no entry would persist nothing for a
|
||
document that was dropped -- so the file's only possible use was to fail a
|
||
run later, and it did. Measured: 11 of 39 documents in the K2 corpus propose
|
||
zero segments.
|
||
|
||
- **A concept can now record more than one source.** `sources` renders a flow
|
||
sequence of N flow mappings on one line, so a v0.2 profile can express
|
||
multi-source provenance instead of the single entry that was the measured
|
||
ceiling on SPEC 5.1 coverage. **A single source is byte-identical to before**,
|
||
so every golden is unmoved and no existing bundle changes.
|
||
|
||
The form is flow, not the block list PM decision B6 prescribed, and the
|
||
reason is measured: this library's frontmatter parser is line-oriented and
|
||
skips indented lines, so a block list round-trips to an EMPTY value with every
|
||
entry silently gone -- and the consumer the decision was written for accepts
|
||
the multi-entry flow sequence while classifying a block sequence as unreadable
|
||
provenance. Emitting block would have produced records neither side can read.
|
||
A negative-control test pins the block form's data loss so the reason stays
|
||
measurable rather than remembered.
|
||
|
||
New error code `sources_empty`: an empty list is refused, because
|
||
`sources: []` reads as a measured absence when it is the absence of a
|
||
measurement.
|
||
|
||
- **The consumption contract, stated normatively** in
|
||
`docs/consumption-contract.md`, with a copyable skill template
|
||
(`skills/okf-consume-template/`) and a checker (`tools/okf_contract_check.py`)
|
||
that reads its mechanically checkable half: payload shape, source marking per
|
||
excerpt, the closed `adjudication` and `trust_tier` state sets, denominator
|
||
identity, and the budget gate with its validated known-positive. The checker
|
||
ships outside `src/`, so no consumer's install surface changes.
|
||
|
||
- **Five office formats through a vendored converter**, behind the same
|
||
optional `[extract]` extra: `docx`, `xlsx`, `pptx`, `odt`, `rtf`. The
|
||
converter binary travels inside the wheel and is resolved by path rather than
|
||
found on `PATH`, with its version asserted against a pin -- `pypandoc`
|
||
searches `PATH` first and takes the highest version it finds, so a vendored
|
||
binary buys nothing until something resolves it explicitly. A host carrying a
|
||
different converter is refused rather than silently used.
|
||
|
||
**Two of the five rows are measured; three are not.** The corpus this work
|
||
was measured on contains zero `pptx`, `odt` and `rtf` files, so those rows
|
||
work by construction and have never met a document anyone wrote. The
|
||
distinction is asserted in the suite, not left in a comment.
|
||
|
||
`.doc` (Word 97) stays out -- the converter does not read it. Drawn content
|
||
does not survive extraction in any format, and every conversion warns about
|
||
it.
|
||
|
||
- **Four error codes** for the converter path: `extractor_binary_missing`,
|
||
`extractor_binary_version`, `extractor_convert_error` and
|
||
`extractor_empty_conversion`.
|
||
|
||
- **`tests/test_docs_promises.py`**, which asserts the README's published
|
||
format list against the registries it describes. This exists because the
|
||
README's previous promise -- that `docx` and `xlsx` always fail fast -- went
|
||
false silently when the converter landed. A published guarantee is a test
|
||
obligation.
|
||
|
||
|
||
- **Door B extracts `pdf` behind the optional `[extract]` extra.** The extra is
|
||
populated for the first time, with one parser: `pdfplumber>=0.11.10,<0.12`
|
||
(MIT). The default install is unchanged — still exactly one runtime
|
||
dependency, still stdlib otherwise — and a packaging test enforces that.
|
||
|
||
**The parser choice was forced by a measurement, not by preference**
|
||
(`docs/2026-08-21-g2-pdf-extraction-measurement.md`). On a real Vegnormalene
|
||
requirement table, `pdfplumber` keeps 4 of 4 rows with label and value on the
|
||
same line; `pypdf`, `pdfminer.six` and `pymupdf` each keep 0 of 4, emitting
|
||
all labels and then all values. A downstream reader can only re-pair those by
|
||
guessing, and in a requirements document a wrong pairing looks right.
|
||
`pymupdf` was additionally excluded on licence (AGPL-3.0 or commercial):
|
||
this package is MIT and an optional extra must not hand a consumer copyleft
|
||
they did not choose.
|
||
|
||
**`docx` and `xlsx` were unchanged at this release**, shipping no parser and
|
||
failing fast with `extractor_extra_missing`. That is no longer true: a later
|
||
release added a vendored converter reaching five office formats. This entry
|
||
is left as written — a changelog records what a release did — and the
|
||
correction is stated here so a reader arriving at this line is not misled.
|
||
|
||
- **`ExtractionWarning`**, exported from the package. Every `pdf` extraction
|
||
emits one. Text extraction recovers text; anything a PDF *draws* — figures,
|
||
diagrams, images — has no text to recover, so only captions survive and a
|
||
bundle built from drawn documents is **incomplete by construction**. That is
|
||
categorically true rather than document-specific, so it is stated rather than
|
||
detected: deciding "is there a figure on this page" is a layout heuristic this
|
||
library does not own. A named class so it can be filtered deliberately.
|
||
|
||
- **Two error codes**, both mirroring existing patterns rather than inventing
|
||
behaviour: `extractor_empty_pdf` (a PDF yielded no text on any page — a
|
||
scanned or image-only document; refused rather than persisted as an empty
|
||
concept, which would be the silent skip this registry exists to prevent) and
|
||
`extractor_pdf_error` (the parser failed on the bytes; the third-party
|
||
exception is wrapped, never leaked).
|
||
|
||
### Changed
|
||
|
||
- **The `[extract]` gate for `pdf` is now an import probe rather than a
|
||
membership test.** The rejection did not change: without the extra installed,
|
||
`pdf` still raises `ExtractionError` with code `extractor_extra_missing` and
|
||
the same message naming the remedy. A consumer that has not installed the
|
||
extra sees no difference at all. That behaviour is asserted unconditionally,
|
||
including on machines where the parser *is* installed, so it cannot rot into
|
||
a skipped test.
|
||
|
||
**Known limitation, stated rather than worked around:** structured table
|
||
recovery is out of scope. `pdfplumber.extract_tables()` and
|
||
`PyMuPDF.find_tables()` — two independent implementations — return the *same*
|
||
wrong shape for the measured requirement table, and across the whole handbook
|
||
only 45 of 196 detected table objects are clean enough to hand to
|
||
`render_table` unchanged. The breakage is in the documents' ruling geometry,
|
||
not in either library. PDFs therefore enter this library as **prose**, with
|
||
table lines correctly paired.
|
||
|
||
**Extracted PDF text is pinned to an exact parser version.** `pdfplumber`
|
||
pins `pdfminer.six==20260107` exactly, and `pdfminer.six` ships date-stamped
|
||
releases with no stability contract. Extraction is deterministic within a
|
||
parser version (measured across five configurations) and not guaranteed
|
||
across one, so any golden fixture built on extracted PDF text is a fixture
|
||
migration away from a parser upgrade. `tests/test_extract.py` freezes the
|
||
expected text of a committed fixture so that upgrade breaks something visible
|
||
instead of drifting silently; see `tests/fixtures/README.md`.
|
||
|
||
- **`DEFAULT` now stamps `generated: { by: process:okf-ingest, at: <ingested_at> }`
|
||
instead of `generated: true`.** This is a byte change in every concept file Door A
|
||
writes under `DEFAULT`, so a consumer's own golden fixtures will show one changed
|
||
line per generated file.
|
||
|
||
The trigger is not an upstream Google release. `DEFAULT` states the ingest-spec
|
||
layer owned by `portfolio-optimiser-commons`, and they ratified this shape
|
||
(2026-08-02) and executed it in their spec on 2026-08-09. §7 defines the value:
|
||
the actor is the fixed `process:okf-ingest`, `at` repeats `ingested_at` verbatim,
|
||
and it is unquoted because frontmatter is parsed line-oriented — a quote would be
|
||
a character in the value rather than syntax a reader strips.
|
||
|
||
**What it costs a consumer is a re-run, and nothing else.** Ownership recognition
|
||
is one-way: a profile carrying an actor still owns the older literal stamp, so a
|
||
bundle written by an earlier version re-runs in place rather than tripping the §3
|
||
collision gate. No call signature changed and no key was added or removed.
|
||
|
||
Two things it is NOT. It is not a migration of `DEFAULT` onto OKF v0.2 — the
|
||
profile remains v0.1 on every axis upstream owns, and still emits no `sources`.
|
||
And it does not make `DEFAULT` and `OKF_V0_2` the same profile; they now agree on
|
||
the stamp and continue to differ in index root frontmatter, type-conditional
|
||
requirements, and `sources` derivation.
|
||
|
||
Door B (`process_inbox`) is deliberately unchanged: its marker is `generated`
|
||
plus `source_file`, it is not governed by the ingest-spec, and it stays disjoint
|
||
from Door A's `ingest_manifest`.
|
||
|
||
- **The v0.5.0 pilot set gained a fourth member: `portfolio-optimiser`.** Admitted
|
||
2026-08-09 on their request. `v0.5.0a1`'s entry below says "do not pin this tag
|
||
outside the pilot set" and names three repos; that entry is left as written,
|
||
because it records what was true when the tag shipped. This is the amendment,
|
||
and the sentence still binds — the set is now those three plus
|
||
`portfolio-optimiser`.
|
||
|
||
The reason is the producer axis, not the count. `portfolio-optimiser-claude` is
|
||
parked, and with it parked no original member could *emit* a v0.2 bundle at all:
|
||
`claude-code-llm-wiki` is read-only in the pilot and `catalog` is gate-side. The
|
||
new member consumes the same Door A. Nothing about the provisional surface
|
||
changes: `OKF_LATEST` still points at `DEFAULT`, and the v0.2 surface may still
|
||
move on pilot feedback without a deprecation cycle.
|
||
|
||
**If you pin `v0.5.0a2`, its guard pin is `>=0.2,<0.3`** — that tag resolves
|
||
against guard `v0.2.0`, *not* the `v0.3.4` this repo's `main` now uses. `main`
|
||
moved to `>=0.3,<0.4` after the tag. Also note `tool.uv.sources` is not
|
||
inherited transitively: a consumer supplies the guard's source itself.
|
||
|
||
- **The guard pin moved to `>=0.3,<0.4`, resolved against `v0.3.4`.** The
|
||
window is widened only after measurement, never before: the 19-fixture
|
||
guard-surface suite was run against `v0.3.4` in a scratch venv first, and
|
||
reproduced exactly the three deltas measured against `v0.3.3` — no new ones.
|
||
`v0.3.4`'s own fixes are regex-complexity repairs, one of them
|
||
(`okf._MD_LINK_RE`) on Door C's call path, with no disposition changes.
|
||
|
||
- **Door C now passes `allow_reserved=False` to `okf.import_bundle`.** The
|
||
guard added the keyword in the `0.3.x` line and defaults it `True` for the
|
||
received-bundle path, which would merge a sender's `index.md` / `log.md`
|
||
instead of rejecting them. Door C overrides it, keeping the unconditional
|
||
reserved-name refusal committed to before the keyword existed. The reason is
|
||
structural rather than a second opinion on the guard's scan: Door C generates
|
||
the merged bundle's `index.md` from what it merged and writes every merged
|
||
concept verbatim, so a sender's `index.md` would be a second, irreconcilable
|
||
claim on one path.
|
||
|
||
**This is not a behaviour change for anyone on the previous pin.** Under
|
||
`v0.2.0` the keyword did not exist and reserved names were refused by
|
||
construction; the explicit argument preserves that outcome across the bump.
|
||
A consumer sees the same rejections, with the same reasons, before and after.
|
||
|
||
### Fixed
|
||
|
||
- **`tools/okf_adjudicate.py` exits 2 on a malformed plan instead of raising.**
|
||
A `SegmentationError` from the plan grammar used to escape `main()` as a
|
||
traceback and exit 1, while every other malformed-plan case in the same file
|
||
already returned 2. Exit codes are the interface a caller scripts against,
|
||
and exit 1 with a traceback says "this command broke" where the truth is
|
||
"this file is not a plan" -- the two are the same code an unhandled bug would
|
||
produce. The refusal itself is unchanged: nothing is written either way, and
|
||
the grammar is untouched. Now one line on stderr naming the error code, and
|
||
exit 2. A caller that treated a non-zero exit as failure sees no difference;
|
||
one that distinguished 1 from 2 sees a malformed plan move into the class it
|
||
belonged to. Pre-existing since the empty-verdict branch landed, and pinned
|
||
until now by a test that recorded it as a finding rather than fixing it.
|
||
|
||
- **A verdict `okf_adjudicate` cannot read back is no longer reported as a
|
||
malformed plan.** The command parses the plan before building anything and
|
||
parses its own verdict after, so only the first failure is the operator's
|
||
file. The second is reachable -- an empty `--adjudicator` produces a verdict
|
||
the grammar refuses -- and before the change above it surfaced as a
|
||
traceback. Catching the grammar error at the top would have turned it into a
|
||
clean, confident, wrong diagnosis naming the plan file, which is worse than
|
||
the traceback: it sends the operator to fix the one artifact that was fine.
|
||
The verdict parse now raises `AdjudicationError`, so the message names what
|
||
actually failed. Exit 2 either way; nothing is written either way.
|
||
|
||
- **`okf build --ingested-at` alone now stamps every concept, not just the
|
||
unsegmented ones.** `--proposed-at` defaulted independently to
|
||
`DEFAULT_STAMP`, so a caller who set only `--ingested-at` stamped the 11 of
|
||
629 concepts that read the call's value directly, while the 618 segmented
|
||
concepts -- which read `segment.ingested_at`, the plan's `proposed_at` --
|
||
stayed on `1970-01-01T00:00:00Z`. Measured on K2 rebuilt at `fbaac6d`.
|
||
`proposed_at` now defaults to `ingested_at` when omitted; a caller who wants
|
||
the proposal and the replay dated differently still passes both explicitly.
|
||
Neither flag passed still yields `DEFAULT_STAMP` for both, byte-identical to
|
||
before.
|
||
|
||
- **The consumption pre-pass (`tools/okf_consume.py`) no longer counts a
|
||
linked `log.md` as a concept.** `link_log_in_root_index` (`corpus.py`,
|
||
`95eb271`) links a run's own log from the root index for bundle navigation;
|
||
the index walk that enumerates concepts followed that link like any other
|
||
and counted the log as one, inflating a 629-concept K2 rebuild to 630 and
|
||
letting the log rank and get cut like real content. The link stays --
|
||
`docs/consumption-contract.md` is silent on `log.md`, and `95eb271` already
|
||
named the link a LOCAL choice rather than conformance -- but the walk now
|
||
recognises `LOG_NAME` the same way it recognises the index itself: reachable
|
||
for navigation, never a concept. A bundle whose index does not link its log
|
||
(every bundle built before `95eb271`, including the delivered
|
||
`K2-bundle-20260903`) computes the same `sha256-tree` ref before and after.
|
||
|
||
## [0.5.0a2] — 2026-07-31
|
||
|
||
**This is the pre-release the pilots pin. `v0.5.0a1` was tagged and abandoned
|
||
unused — do not pin it.** It carried a `generated.by` actor that the spec owner
|
||
had already excluded, and it was caught before any pilot was notified.
|
||
|
||
### Fixed
|
||
|
||
- **`OKF_V0_2`'s `generated.by` actor is `process:okf-ingest`**, not
|
||
`process:llm-ingestion-okf`. Commons decided the fixed id on this repo's own
|
||
proposal 2026-07-31, superseding option (d) chosen here 2026-07-27. The
|
||
exclusion is `ingest-spec.md:7-8`, frozen on the spec being framework-neutral:
|
||
normalising *our* repo name into the id would force every other conformant
|
||
implementation to write it into its own output.
|
||
|
||
Nothing in the wild carried the excluded value — `OKF_V0_2` did not exist at
|
||
`v0.4.0`, and no pilot had been notified — so this costs a tag rather than a
|
||
migration. It is recorded rather than quietly folded in because the failure it
|
||
avoids is specific: `actor` is both the stamp written and the value owned back
|
||
(`OwnershipPolicy`), and recognition is one-way, so a pilot holding bundles
|
||
stamped with the excluded id would have hit `collision_unstamped` on its own
|
||
files the moment the id was corrected.
|
||
|
||
## [0.5.0a1] — 2026-07-31 — ABANDONED, do not pin
|
||
|
||
**Pre-release. PROVISIONAL surface.** Shipped to a named pilot set —
|
||
`portfolio-optimiser-claude`, the plugin marketplace catalog, and
|
||
`claude-code-llm-wiki` — so that real data can find what fixtures cannot. The
|
||
v0.2 surface may change on their feedback **without a deprecation cycle**.
|
||
Saying so up front is what buys the freedom to act on the feedback; discovering
|
||
it later is what would make a pilot a de-facto release. Do not pin this tag
|
||
outside the pilot set.
|
||
|
||
Support for a new upstream version is **additive — a new profile, never a
|
||
migration**. `OKF_LATEST` still points at `DEFAULT`; flipping it is the GA
|
||
event, not a side effect of this tag.
|
||
|
||
### Added
|
||
|
||
- **`OKF_V0_2` profile.** Google OKF v0.2 as a profile alongside `DEFAULT` and
|
||
`STRICT_V1`. It closes nothing: §14 forbids a conformant consumer to reject on
|
||
an unknown `type` value or on unknown additional keys, so `type` is the only
|
||
required key. It NAMES `okf_version` but never carries its value — that value
|
||
tracks the upstream Google version and belongs to catalog (decision E1), so
|
||
the caller supplies it via `materialize_bundle(..., root_frontmatter_values=…)`
|
||
and the profile fixes only the key and its position.
|
||
- **`root_frontmatter_values`** on `materialize_bundle`, keyword-only: the
|
||
mechanism behind "a profile names a key, a caller owns its value".
|
||
- **`sources` emitted as an inline flow sequence**, populated from the manifest.
|
||
- **The v0.2 golden fixture**, compared byte-for-byte like the others.
|
||
|
||
### Changed
|
||
|
||
- **`profile` on `materialize_bundle` is keyword-only**, so existing positional
|
||
call sites stay source-compatible.
|
||
- **`_is_ingest_owned` is profile-aware.** Ownership is a policy on the profile,
|
||
not a literal; recognition stays one-way.
|
||
|
||
### Fixed
|
||
|
||
- **`root_frontmatter` permitted a key without requiring it — the two had been
|
||
the same profile field.** Upstream says MAY where the field said MUST, so a
|
||
root index that legally omitted an optional key was rejected. `root_frontmatter`
|
||
now permits and orders; the new `root_frontmatter_required` requires.
|
||
`STRICT_V1` keeps requiring its three (unchanged for its consumer), `OKF_V0_2`
|
||
requires none, `DEFAULT` is untouched. Measured against the eight root indexes
|
||
available locally: 7 of 8 failed before, 0 of 8 after.
|
||
|
||
### Notes
|
||
|
||
- **`DEFAULT` and `STRICT_V1` are byte-stable across this release.** The emit
|
||
path is byte-identical; the golden suite is the check.
|
||
- **Known, deliberately unfixed:** `TypePolicy`'s closed branch does not strip
|
||
quotes from a declared type. All three call sites pass `DEFAULT.types`, and
|
||
both `DEFAULT` and `OKF_V0_2` set `allowed=None`, so the branch is
|
||
unreachable in shipped code — it fires only for a caller constructing
|
||
`STRICT_V1` and calling `rejection` directly. Fixing it would repair a write
|
||
path that never sees quotes, and would fix the meaning of a quote character
|
||
without a value model that can express one, in a line-oriented format.
|
||
|
||
## [0.4.0] — 2026-07-25
|
||
|
||
Phase 2. The bundle inbox (Door B) and external-bundle import (Door C) ship,
|
||
and with them this library's first — and only ever — runtime dependency.
|
||
|
||
### Added
|
||
|
||
- **Door B — bundle inbox (`process_inbox`).** Per dropped file: bytes →
|
||
extraction → persist gate → render → collision gate → write → index link.
|
||
Returns an `InboxResult` whose four buckets (`persisted`, `quarantined`,
|
||
`rejected`, `failed`) are disjoint and complete, so a file that vanished
|
||
mid-run surfaces as a missing entry rather than as nothing. One bad file
|
||
never aborts the run: only an invalid `ingested_at`, a reserved `okf_type`,
|
||
and a missing inbox directory fail the whole run, each being wrong for every
|
||
file at once. Concept files are named `inbox-{slug}.md`, disjoint from
|
||
`index.md`, Door A's `ingest-*`, and `promoted-verdict-*`; `source_sha256`
|
||
is taken over the original dropped bytes, so provenance stays re-verifiable
|
||
against the operator's file.
|
||
- **Door C — external bundle import (`import_bundle`).** Reads an external OKF
|
||
bundle and hands it *whole* to an injected gate over the guard's
|
||
`okf.import_bundle` — a bundle-level call, because it resolves the
|
||
cross-link graph across concepts — merging only concepts that clear the
|
||
non-blocking floor. Two invariants, both load-bearing: a merged concept is
|
||
written **verbatim** (stamping it would require round-tripping frontmatter
|
||
through this library's line-oriented parser, which cannot represent the
|
||
block lists the guard's parser accepts, and would persist bytes the guard
|
||
never screened), and ownership is therefore proven by **content identity** —
|
||
an occupied target name is re-used only when the bytes there are already
|
||
identical, never overwritten. Re-importing an unchanged bundle is a no-op.
|
||
- **`extract_text` — the Door B extraction registry.** All file-type → text
|
||
extraction lives in this library, because the guard is text-only. `md`/`txt`
|
||
pass through, `csv` renders the markdown table, `json` is fenced verbatim,
|
||
and `html`/`htm` reduce to text with `html.parser` — stdlib throughout.
|
||
`pdf`, `docx`, and `xlsx` are gated behind the `[extract]` extra, which
|
||
ships no parser yet: those types fail fast with a typed error naming the
|
||
extra, never a silent skip.
|
||
- **`llm_ingestion_okf.guard_adapter`** — the shipped gate over the real
|
||
guard (`inbox_gate`, `import_gate`), and the only module here that imports
|
||
it. Door B screens the exact bytes it persists: the guard's `prepare_input`
|
||
bookend prepares text for a model call this library never makes, so
|
||
`screen_output` alone is used and the screened string is the written string.
|
||
It follows that the gate **refuses rather than repairs** — a file carrying
|
||
an invisible carrier is rejected, not stripped and persisted. The policy is
|
||
`PRESET_USER_UPLOAD`, so any finding at all is held back.
|
||
- **15 new stable error codes**, each registered in the `errors.py` docstrings
|
||
(the stability contract) and pinned by the error-code conformance suite:
|
||
`extractor_decode_error`, `extractor_empty_csv`, `extractor_extra_missing`,
|
||
`extractor_unknown`, `import_label_invalid`, `import_path_empty`,
|
||
`import_path_too_long`, `import_provenance_invalid`,
|
||
`import_slug_collision`, `inbox_slug_collision`, `inbox_slug_empty`,
|
||
`inbox_slug_too_long`, `inbox_source_file_invalid`, `inbox_title_invalid`,
|
||
`okf_type_reserved`.
|
||
|
||
### Changed
|
||
|
||
- **`llm-ingestion-guard>=0.2,<0.3` is now a mandatory runtime dependency.**
|
||
Installing this package installs the guard. Importing it does not: only
|
||
`guard_adapter` imports the guard, so a Door A consumer keeps working
|
||
whatever state the dependency is in, and the doors themselves still take an
|
||
*injected* gate. Until the guard is published to a package index, install it
|
||
from its tag — see the README. A packaging test enforces that this stays the
|
||
only runtime dependency.
|
||
- **An extraction `title` containing `[` or `]` is rejected at manifest load**
|
||
(ingest-spec §4, ratified D1). Previously only single-line was validated. A
|
||
manifest that loaded before and carries a bracket in a title now fails fast
|
||
with code `manifest_schema`: the title renders verbatim into the index link
|
||
label `- [title](target)`, where a bracket breaks index-link and navigation
|
||
parsing downstream.
|
||
|
||
### Fixed
|
||
|
||
- **Materializing one manifest no longer deletes the files another manifest
|
||
stamped into the same bundle.** The §3 ownership scan classified every
|
||
ingest-stamped file as replaceable, so a second manifest sharing a bundle
|
||
removed the first one's concept files and their index links. Ownership is
|
||
now narrowed to files whose stamp names the running manifest by stem — the
|
||
stem is stable across content edits, so an edited manifest still reclaims
|
||
what a prior run of itself wrote. The operator-copy restriction (a generated
|
||
file copied into curated content) remains documented, not enforced.
|
||
|
||
### Notes
|
||
|
||
- Phase 2's binary extraction is **not** in this release: the `[extract]`
|
||
extra is declared but empty, and `pdf`/`docx`/`xlsx` therefore fail fast.
|
||
That is the one outstanding item from the phase, and it ships separately.
|
||
- **The ownership change above is an extension point, not a spec fix.** Filed
|
||
under "Fixed" because it stops silent data loss, but the spec owner
|
||
(`portfolio-optimiser-commons`) has since pointed out that removing *every*
|
||
ingest-stamped file is verbatim what ingest-spec v1 §5 mandates: v1 assumes
|
||
one manifest per bundle and defers multiple manifests feeding one bundle as
|
||
a named extension point. So this release implements that extension point and
|
||
outruns the frozen text rather than conforming to it. The mechanic itself is
|
||
not at risk — matching by manifest stem follows from the spec's own
|
||
`{stem}@{h}` stamp definition, since `{h}` changes on every content edit —
|
||
but the surrounding prose is queued for amendment in commons and is not
|
||
ratified. Treat multi-manifest bundles as ahead of the spec until it is.
|
||
- Exception `__cause__` preservation is now pinned by a conformance suite, one
|
||
test per fail-fast wrap site, alongside the existing `.code` suite.
|
||
|
||
## [0.3.2] — 2026-07-23
|
||
|
||
### Fixed
|
||
|
||
- **Frontmatter and index-label values are emitted verbatim; only
|
||
`source_query` is whitespace-collapsed.** Earlier releases collapsed every
|
||
whitespace run in every frontmatter value and index link label to a single
|
||
space. §5 of `ingest-spec.md` mandates that collapse for `source_query`
|
||
alone — where a multi-line SQL `SELECT` must render on one line — while
|
||
every other value is validated single-line at manifest load and passed
|
||
through unchanged: validation, not repair. A `title` carrying an internal
|
||
whitespace run now survives byte-for-byte at both the `title` frontmatter
|
||
and the index link label, instead of being silently altered. Output bytes
|
||
change only for values that contained a collapsible whitespace run; the
|
||
shipped golden fixtures and both consumers are unaffected.
|
||
|
||
## [0.3.1] — 2026-07-19
|
||
|
||
### Fixed
|
||
|
||
- **Documentation corrected a security claim that did not hold.** The module
|
||
docstring and README stated that this library "calls the guard at every persist
|
||
gate". That described the intended end state in the present tense. Door A — the
|
||
only door shipped — is ungated: the package has zero runtime dependencies and
|
||
calls no guard function before writing to disk. Both places now say so plainly,
|
||
and state that gating external or untrusted content is the caller's
|
||
responsibility (`okf.import_bundle`, or `prepare_input`/`screen_output`) until
|
||
the persist gates land with Doors B and C.
|
||
|
||
No behavior changed in this release. The correction is published because a
|
||
consumer read the earlier wording as safe-by-default and would have persisted
|
||
ungated content on that basis.
|
||
|
||
## [0.3.0] — 2026-07-17
|
||
|
||
### Added
|
||
|
||
- **Stable machine-readable error codes.** `IngestError` gained a keyword-only
|
||
`code` attribute (default `"unspecified"`). Roughly 24 codes are documented in
|
||
the `errors.py` docstrings, and those docstrings are the stability contract.
|
||
Consumers should assert on `exc.value.code`, not on message text.
|
||
|
||
### Changed
|
||
|
||
- **Exception message text is explicitly declared unstable.** It may change in any
|
||
release. Tests matching on message strings (`pytest.raises(match=...)`) should
|
||
migrate to code comparisons.
|
||
|
||
### Notes
|
||
|
||
- Generic schema shape violations deliberately share the single code
|
||
`manifest_schema`. One code per validation rule would have frozen an
|
||
unnecessarily large surface. Finer resolution is a separate decision, not an
|
||
assumed requirement.
|
||
|
||
## [0.2.0] — 2026-07-17
|
||
|
||
### Added
|
||
|
||
- **PEP 561 support.** The `py.typed` marker ships with the package, so consumers
|
||
get the inline type hints without a mypy override.
|
||
- **`IngestResult.stamp`** exposes the spec §5 provenance stamp on the result
|
||
object.
|
||
|
||
## [0.1.0] — 2026-07-16
|
||
|
||
Phase 1 (Door A) implemented against the normative `ingest-spec.md` owned by
|
||
`portfolio-optimiser-commons`. Never tagged; consumers pinned the commit
|
||
`dae0bd1a` directly.
|
||
|
||
### Added
|
||
|
||
- Fail-fast manifest validation (spec §3–§4).
|
||
- Spec §5 body renderers as pure functions.
|
||
- The `file` connector (CSV, fail-closed path boundary), the `sql` connector
|
||
(read-only sqlite, env-resolved credentials), and the `http` connector behind an
|
||
explicit per-run network opt-in.
|
||
- Spec §5 materialization with an in-memory staging collision gate.
|
||
- Index maintenance on re-materialization (spec §6).
|
||
- The spec §11 golden fixtures, compared byte-for-byte.
|
||
- The Door A public surface: `materialize_bundle` plus the typed error hierarchy
|
||
rooted in `IngestError`.
|
||
|
||
[0.6.0]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.5.0a2...v0.6.0
|
||
[0.5.0a2]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.5.0a1...v0.5.0a2
|
||
[0.5.0a1]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.4.0...v0.5.0a1
|
||
[0.4.0]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.3.2...v0.4.0
|
||
[0.3.2]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.3.1...v0.3.2
|
||
[0.3.1]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.3.0...v0.3.1
|
||
[0.3.0]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.2.0...v0.3.0
|
||
[0.2.0]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/dae0bd1a...v0.2.0
|
||
[0.1.0]: https://git.fromaitochitta.com/open/llm-ingestion-okf/src/commit/dae0bd1a
|