A SegmentationError raised by the plan grammar escaped main() as a
traceback and exit 1, while every other malformed-plan case in the same
file already returned 2. Exit codes are the interface a caller scripts
against, and exit 1 with a traceback is the code an unhandled bug
produces -- it says "this command broke" where the truth is "this file
is not a plan".
The refusal itself is unchanged: nothing was written before and nothing
is written now, and the grammar in src/ is untouched. What changes is
one line on stderr naming the error code, and the exit code.
Both branches that can raise are covered: the pre-write parse of a
non-empty plan, and the required-field check reached through the empty
branch.
The old behaviour was pinned by
test_an_entries_value_that_is_not_a_list_is_still_refused, which
asserted that a wrongly-typed `entries` reaches the caller as a raised
SegmentationError and recorded that as a finding rather than fixing it.
That test is rewritten here, in the same commit as the code, to assert
exit 2 plus the code on stderr. A second test pins the one-line stderr
shape on the non-empty branch.
Suite 1072 -> 1073 passed; ruff and mypy --strict clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Re-measures the declared UMÅLT gap from docs/2026-09-02-korpuskjoring-k1-k2.md
against ground truth (find, not STATE). K2/trinn1 and "K3-korpuset" are the
same directory (both K1/K2 02.09 and K3 09.04 run against
~/corpora/okf-telling-20260829/K2/trinn1) -- one corpus, counted once. 0
pptx, 0 odt, 0 rtf, known-positive control (docx: 5) proves the query can
find. Sum 0 across both named corpora ends the order at step 1 per its own
instruction; no K2 step 2 work started, no code added. No public corpus with
these formats is on hand under ~/corpora -- reported as not verified, not as
nonexistent.
Order: 20260904T190727Z-589245685-from-.claude
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M7Qvq71UCQZwkrLT7WbeB9
Order 20260904T172353Z-6290714297-from-.claude. 39/43 K2/trinn1 documents
measured with tools/okf_cid_measure.py; the 4 not measured are the same 4
K1 already named as coded rejections. Over 10 %: 1/39. Over 50 %: 1/39 --
Bilag 9.1 both times, at 95.0743 % (reproduces the published 95.1 %). No
other document in the corpus crosses either line. Known-negative control
holds (a docx gives 0 CID characters on a nonzero total). The word count
(44) does not reproduce the earlier ad hoc 98 -- no script for that figure
was ever committed, so nothing exists to reconcile against; both counts
are gibberish letter runs, not prose, so the substantive finding does not
move. K2 bundle hashed before and after, byte-unchanged (1108 files).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Order 20260904T172353Z-6290714297-from-.claude. Bilag 9.1's 95.1 % CID
share (docs/2026-09-04-k3-arm-c.md) was found ad hoc, with no committed
script -- the same gap this repo's own fidelity instrument criticizes in
Arm A's uncommitted docx/xlsx figures. okf_cid_measure.py runs the exact
extract_text call the door makes and reports per-document CID share and
4+-letter word count, denominator stated for files it cannot measure.
Red-first: tests/test_cid_measure.py pins measure() against fixture text
of known composition before the implementation existed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Arm C changed the proposal on 6 of the 12 sampled documents and moved the
K3 category counts by ZERO. Both arms: too coarse 8, too fine 4, duplicate
0, correct 0. That is the finding, not a null result -- the size rule makes
concepts smaller, which is what the consumer asked for, and K3 does not
measure size. It asks whether a boundary falls where a unit of knowledge
begins, and an arithmetic cut does not know where one begins.
Blind second rater per arm, n_blind = 6, two separate raters so an arm C
rating cannot anchor on an arm B one: 5/6 in each arm, and the two blind
raters' own six-document counts are identical to each other. The single
disagreement is the same document as the baseline's -- with the roles
reversed, and resolved by a specific offset in one direction and a general
impression in the other, exactly as the baseline described. The `too
coarse` count is still a floor.
Reported rather than smoothed: this first rater records 8/4/0/0 on the same
arm B proposals the 2026-09-02 baseline scored 7/3/1/1. The proposals did
not change; the rater did, and both differences are traced.
What did move, and is not K3: max concept body 217 472 -> 19 988,
concepts over 100 000 characters 4 -> 0, documents proposing nothing
11/39 -> 6/39, concepts 629 -> 680.
Found while measuring, not looked for: Bilag 9.1 -- the largest concept in
the bundle and the one the consumer flagged -- is an EXTRACTION failure.
95.1 % of its 217 470 characters are (cid:N) glyph codes and 98 alphabetic
words of 4+ letters survive in the whole document. Arm C divides unreadable
text into 14 pieces of unreadable text. Its verdict in both arms rests on
document identity, not on readable content.
The bundle a consumer is running against right now is byte-unchanged:
whole tree, 1108 files, sha256 9cd74519... before and after.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Arm C is NOT defined in docs/2026-09-02-k3-k4-k5-metode.md -- that file
contains no occurrence of the word, and neither Arm A nor Arm B is defined
there either. The definition implemented here was written for order
20260904T145630Z and is reported as the author's, never as a ratified one.
Arm C = Arm B's mechanical rules, plus one deterministic rule that cuts
any proposed span longer than a declared cap at the nearest paragraph
boundary at or before it, the whole document counting as one span when
the rules find no boundary at all.
One rule and not two, on purpose. The two failure modes the K2 rebuild
measured -- a PDF with no outline (Bilag 9.1, 217 472 characters) and a PDF
whose headings are its table of contents, so the trailing segment absorbs
the body (Bilag 3.1, Bilag 1) -- are the same failure of SIZE, and a second
rule aimed at each would confound which one moved the number.
`--max-segment-chars` defaults to 0, which is OFF: the artifact is then
byte-identical to Arm B's, pinned by a test that writes both and compares
bytes. The standard profile does not move, and the K2 bundle a consumer is
running against right now is not rebuilt.
What Arm C deliberately does NOT change: the region before the first
candidate is still covered by no segment. That is a real coverage defect --
the K3 baseline's blind rater named it -- and fixing it here would put two
changes behind one measurement.
A part carries TWO rule names in `derived`: the heading rule that opened the
span, and `rule:size-split` for the cut. Dropping the first would make a
part traceable to arithmetic and nothing else.
Tests first: 13 red, then green. 1055 -> 1068.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The guard could not read back what this library WRITES. At 1.2.0,
`okf.parse_frontmatter` refused the OKF v0.2 golden outright --
`OKFFrontmatterError: value begins with a disallowed YAML indicator '['`
against `sources: [{ id: golden-v0-2-sales, resource: fixture }]`. Flow is
the only form this library can emit, because its own line-oriented parser
cannot round-trip the block form at all, so a gate that refuses flow
refuses everything Door A produces under `OKF_V0_2`.
The control was run BEFORE the bump, which is the only moment it exists:
the probe raised on 1.2.0, so the new test discriminates rather than
merely passes. `[project.dependencies]` already said `>=1.2,<2.0` and is
unchanged; only `[tool.uv.sources]` and `uv.lock` move.
TWO gate rows moved, not the one the work was scoped around, which is why
the whole documented probe was re-run instead of just the `sources` case:
the BLOCK form of `sources` now passes too, retiring G30. That changes
nothing about what we emit -- our own parser is still the binding
constraint on writing flow -- and `docs/okf-nokkelinventar.md` now carries
a `guard 1.3.0` column beside the 1.2.0 measurement rather than
overwriting it. A third row kept its verdict but changed its reason, so
the quoted message was corrected too.
The Door C boundary is unmoved, verified with a known-positive:
`resource` is allowlisted only inside a `sources` entry, so section
10.2's `executor.resource` and `attester.resource` are still rejected
("not on the OKF mapping allowlist under 'executor'") while top-level
`resource` passes.
`uv.lock` also gains `pypandoc-binary==1.17`. That is a stale lockfile
being corrected, not a new dependency: it was already declared in the
`[extract]` extra, and `uv lock --check` reports the lockfile out of date
on the untouched tree. Core keeps exactly one runtime dependency.
Not addressed, and recorded rather than built: the guard reports that
`sources[].resource` is scanned as text but never URL-validated, because
SPEC 5.1 permits bundle-relative paths and scope descriptions. No
consumer has asked for a gate there.
Guard 1.3.0 installed from 44e2b31, verified anonymously over https
against the remote tag. 1054 -> 1055 tests. `mypy --strict` clean, `ruff`
clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measured on the K2 artifact by a consumer: `log.md` was on disk and no
index named it, so a reader entering the bundle at `index.md` -- the walk
section 8 exists to support -- never reached the one file carrying `N`.
Stated as a LOCAL choice rather than conformance, because it is one.
Upstream's own bundles do not link their log: measured at `9a15b13`, 0 of
the 24 shipped `index.md` files name the single `log.md` in the set, with
the same grep form finding `tables/index.md` in 4 of them as the
known-positive control. That shows the link is not REQUIRED -- not that
it is disallowed. `docs/plan/okf-v0.2-alignment.md` P1-F6 already
recorded the upstream shape; a line there now separates the two claims,
since reserved names still stay out of an `entries_match_directory`
listing and this profile has that off.
It lives in the harness because the library cannot make it. The log's
content IS the run's outcome, so it cannot exist when the indexes are
projected, and an index that enumerated it off the directory would gain
the link only from the second run onward -- breaking
rebuild-equals-incremental, the property the segmented bundle is built
on.
The membership test is load-bearing and was measured, not assumed. The
two reprojections disagree about this line: the per-directory one drops
every managed entry before re-emitting its block, while the flat one
keeps a managed line whose target is not an owned concept, deliberately,
so that a regex cannot delete curated content. Appending unconditionally
therefore doubled the entry on the second unsegmented run, which is why
both run modes are pinned separately.
1052 -> 1054 tests. `mypy --strict` clean, `ruff` clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`log.md` is written INTO a directory Door B enumerates on the next round: it
matches the concept glob and is excluded only by `index.md`'s name, so a
rebuild could have seen it as pre-existing curated content or pruned it.
Rebuild-equals-incremental is the property the segmented bundle rests on.
Measured on the real artifact, not only the synthetic: the K2 corpus was run
a second time into the same bundle and compared against a snapshot with
`diff -r`, exit 0 over all 1108 files. The test pins the same property in
seconds instead of 13 minutes.
Also corrects the report's reproduction command -- it documented plan
filenames the run did not use, and re-running it into the existing plans
directory would leave two files claiming one `source_sha256`, which
`_resolve_plans` refuses.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One cause behind all four of the consumer's findings: the harness ran under
STRUCTURED_V1 with no plans. 629 concepts, 618 of them carrying
`adjudication: proposed`, K1b recoverable from the bundle's own log.md, and
the two >100k-token concepts split into 34 and 18 entries. The largest
concept is now a different document that has no derivable structure at all
-- reported, not fixed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measured on the K2 corpus 2026-09-03: 11 of 39 documents proposed zero
segments -- overwhelmingly PDFs with no declared structure, which Topic 1b
had already measured at 23 of 33. The proposer wrote an artifact for each
of them and exited 0.
An empty plan cannot be replayed. `process_inbox` refuses one by design,
because a plan naming no entry would persist nothing for a document that
was dropped, so the only thing a zero-entry file can do is fail a run
later -- and it did: the first segmented corpus run stopped on
`segmentation_plan_invalid` before writing a single concept.
Exit 1 with no artifact, distinct from 2, so a driver can tell "this
document lands as one flat concept" from "stop". The orphan check's test
now observes the same property through the status.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measured on the K2 corpus 2026-09-03: 39 documents proposed 618 entries
under 601 distinct paths -- 17 paths were claimed by two documents each.
Section numbering is document-local (`1 Innledning` is in most procurement
documents), so this is structural, not unlucky. Every collision reaches
Door B's gate, which refuses per DOCUMENT, so those documents would land as
coded rejections rather than concepts and a corpus run could not be built
at all.
`--path-prefix` is an argument and not something the tool derives: the
proposer sees ONE document and cannot know what else is in the bundle. It
is reduced to the id grammar before anything is read, and a prefix that
reduces to nothing is refused rather than silently producing the unscoped
paths the caller asked to avoid. Without the flag every artifact already
produced is byte-identical.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two defects a consumer measured on 2026-09-03 against the bundle built by
this harness, both with one cause: `measure` passed `profile=STRUCTURED_V1`
and no plans at all, though Step 17 of the plan says the harness reuses
`process_inbox` "with the per-document plan mapping from Step 15".
- `adjudication` was in 0 of 39 concepts, because the key is written only
inside the plan-covered branch and no plan was ever passed. `--plans-dir`
replays proposals produced per document first; the profile follows from
the flag rather than being something the harness may choose on its own.
`--bundle-id` and `--okf-version` are arguments, never constants: a
profile names a key and the caller owns its value (decision E1).
- `log.md` did not exist, so `merged` was countable from the bundle and `N`
was not -- K1b could only be taken on trust from a report that does not
travel with the artifact. Written in SPEC section 9 form and dated from
`ingested_at`, never the wall clock.
Additive: without `--plans-dir` the run stays the flat `STRUCTURED_V1` run
that produced the published K1/K2 numbers.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measured on the K3 corpus: 4 of 12 judgements produced no artifact, because
the verdict was "none of these segments should be persisted" and the plan
grammar refuses zero entries. That refusal is correct for the run path -- an
empty plan replayed would silently persist nothing for a document that was
dropped -- so the grammar is untouched and the recording tool is taught to
record a rejection instead.
Refusing to materialize and refusing to record are different acts. The
rejection artifact is deliberately NOT replayable: parse_segmentation_plan
still refuses it, and the suite asserts that rather than assuming it. The dwell
time rides at the top level because there is no entry to carry it, and a
ratified rejection with no time on it is as unfalsifiable as a ratified
acceptance with none.
Only the empty LIST takes the branch. A missing entries key, or one that is not
a list, stays the grammar's to refuse: "the adjudicator kept nothing" and
"this file is not a plan" must not collapse.
K4a re-run after the change: propose, adjudicate, run the path twice into two
bundles under SEGMENTED_OKF_V0_2, diff -r exit 0 with no output. 1034 -> 1041
tests.
PM decision B6 asked for a list-taking _render_sources so a concept can record
more than one source, and prescribed the block list as the emitted form. The
list is delivered; the block form is not.
Three measurements, not an argument. Our own parse_frontmatter skips indented
lines, so a block list round-trips to an empty value with every entry silently
gone -- and _is_ingest_owned reads through that same parser. The consumer B6
was written for accepts the multi-entry flow sequence and classifies a block
sequence as unreadable provenance, so block would hand it exactly the state it
cannot read. And B6's own acceptance test asks for a round trip through this
parser, which no block form can pass.
A single source renders byte-identically, so all six goldens are unmoved. The
unquotable-value gate now runs on every entry, not just the first. New code
sources_empty refuses an empty list.
1023 -> 1034 tests, including the negative control that pins the block form's
silent data loss.
Fourteen rules, each with its own code: one "invalid" verdict over fourteen
different defects is a diagnostic no caller can act on. The report quotes its
own denominators -- rules run, excerpts and withheld entries examined -- because
a checker that exempted itself from section 5 would be stating the rule it
breaks.
Three exit codes, not two. "The check did not run" and "the check failed" are
different outcomes, and an unread file reported as a failed check is the fourth
face of the verification law.
The shipped template plus its example payload is the known-positive arm, so a
checker that refuses everything cannot be green on the thirteen negative ones.
A last test asserts every literal the checker enforces appears in the contract
document: two copies of a closed set drift, and the copy nobody reads is the
one that goes wrong.
996 -> 1023 tests.
A per-corpus skill copies this directory and fills every <PLACEHOLDER>. None of
them has a default: a corpus's budget, instrument, profile name and conditional
fields are facts about that corpus, and a default would be this repository
asserting one it has not measured.
The section headings are fixed because the checker in the next commit reads
them by name. references/example-payload.json is the known-positive that keeps
a refuse-everything checker from passing.
The plan document decided what the contract is and who owns it, and said of
itself that nothing in it was implemented. This states the same six elements as
numbered paragraphs with MUST and SHOULD, in the form a checker can read, and
adds the three things the plan layer had no way to carry: concept identity as
the (bundle_id, concept_id) tuple with a content digest per excerpt, the closed
adjudication state set with unknown written explicitly rather than collapsed,
and the prohibition on reaching the verdict layer with a query-time retrieval
tool.
The plan document is not rewritten to match. It records why; this binds. Two
copies of the same rules drift, so the plan carries a pointer instead.
A sixth golden directory, never an edit to the five that exist. Its only
difference from the SEGMENTED_V1 golden is the added `okf_version` root key --
measured by diffing the two -- which is exactly what the profile is for and
nothing else moved.
Modelled on tests/test_segmented_golden.py rather than test_golden.py's
`materialize_case`, which is Door A only: it drives materialize_bundle from a
manifest and this bundle comes through Door B's inbox. The plan's pointer to an
"existing goldens list" in test_golden.py is stale -- that file holds only the
Door A parametrize, and `ingest-golden-segmented` is not in it either. Door B
goldens carrying their own test file is the established pattern, and this one
asserts the five priors are untouched from its own side.
CROSS-BUNDLE IDENTITY (PM decision B1) is settled BEFORE this pin, not after.
Byte-pinning a bundle carrying `okf_version` is where its concept IDs stop
being adjustable: a concept ID is bundle-local and stable, identity across
bundles is the tuple (bundle_id, concept_id), and there is no cross-bundle link
form in v0.2. Asserted -- every concept carries its bundle id, so the tuple is
readable from one document, and no foreign bundle id appears anywhere.
Two defects in these tests, both found by running them:
- `parse_frontmatter` takes a Path and returns the dict; three call sites had
it wrong.
- The cross-bundle check used `b-golden-segmented` as the foreign id, which is
a PREFIX of this bundle's own id. It failed on every file for the wrong
reason and would have read as "a cross-bundle reference was emitted". Now a
foreign id that is not a prefix, with a known-positive half asserting the
bundle does name its own id -- so the check is looking at content that could
have carried one.
Negative control: changing one byte of the golden's `okf_version` turns the pin
red.
`okf_version` and `bundle_id` are fixture DATA read from the case, never
constants in the test -- the value belongs to catalog (E1).
Suite 931 -> 941.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Neither shipped profile could do both: SEGMENTED_V1 names `bundle_id` and not
`okf_version`, OKF_V0_2 names `okf_version` and has no segmentation at all. The
sixth profile is where the two intersect. Additive, as upstream support always
is here -- a new profile, never a migration.
THE INDEX POLICY IS DECIDED, NOT INHERITED, and that is the part that was easy
to get wrong. Measured: OKF_V0_2.index has facets=None and per_directory=False,
while SEGMENTED_V1.index has both. Building on OKF_V0_2's index would have
produced a segmented bundle with no faceted index -- structurally valid, SPEC
conformant, and missing the surface a consumer reads. So the index comes from
SEGMENTED_V1 with both root keys named, and the spec declaration from OKF_V0_2.
The premise is asserted in the suite rather than trusted to stay true.
`SegmentationPolicy.adjudication_key` is the discriminator BETWEEN segmented
profiles. It has to be a field with a value: every 1-to-N branch keys on
`profile.segmentation is not None`, which both segmented profiles satisfy, so a
later step surfacing the adjudication marker on that check would write it into
SEGMENTED_V1 too and move a byte-pinned golden. `None` means the profile does
not surface adjudication state at all.
`okf_version`'s VALUE is not in this module and must never be: a profile names
a key, the caller owns its value (decision E1). A constant here would claim a
decision this library does not own and would be the one thing to chase on every
upstream release. Asserted.
Five existing profiles untouched, each still equal to a freshly constructed
copy, root keys pinned literally, and OKF_LATEST still DEFAULT -- adding a
profile is not a GA event.
Suite 926 -> 931.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Arm E, capability only. A profile MAY name a renderer per suffix; no
domain-aware renderer is written here, that stays a Non-Goal, and `_RENDERERS`
is empty on purpose so the emptiness reads as a decision rather than an
omission.
THE LAYERING IS THE DESIGN, not an implementation detail. `extract.py` is the
extraction registry and must not import the contract layer, or the dependency
runs backwards and the registry stops standing on its own. So `extract_text`
gains a keyword-only `renderer: Callable[[str], str] | None`, knowing nothing
about profiles, and `inbox.py` -- which already holds the profile at that call
site -- resolves a NAME to a function. A test asserts extract.py still contains
no reference to the profile layer, because that constraint is the whole reason
the parameter is shaped this way.
The renderer runs AFTER extraction, never instead of it, so it never has to
re-implement a reader and the two cannot drift. The default is identity, which
is what keeps the five byte-pinned goldens byte-pinned -- asserted per suffix
rather than once.
An unknown renderer NAME is refused rather than falling back to identity: a
silent fallback would produce a bundle that looks rendered and is not, which is
the failure mode this arm exists to make visible. That needed a registered code
(`unknown_renderer`) and its test -- slightly beyond the step's named files,
but the capability cannot ship without defining what an unknown name does.
`tests/test_profile.py`'s exact-field-set assertion went red, as the plan's risk
table predicted. Updated deliberately with the reason recorded: that assertion
exists so a field cannot arrive without someone deciding it should, and its red
run is the mechanism working.
Suite 917 -> 926. All five goldens byte-identical.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The README told consumers that `docx` and `xlsx` ship no parser and always fail
fast. True when written; false the moment the converter seam landed -- and
false SILENTLY, because prose has no test. This repository has been bitten by
that exact shape before: a published guarantee is a test obligation.
So the correction comes with `tests/test_docs_promises.py`, which compares the
README's declared format list against the registries it describes and fails on
a format added without touching the README, on the old claim reappearing in any
wording, on an unmeasured row going unnamed, and on the exclusions being
dropped. Negative control: removing one format from the README's marker turns
it red.
The README now states which rows are measured and which are not. Three of the
five office rows have denominator ZERO in the corpus -- they work by
construction and have never met a document anyone wrote. They are not known to
be broken and not known to be right, and a reader should not have to open the
source to learn which.
The CHANGELOG's shipped entry is left as written, because a changelog records
what a release did; the correction is stated at that line instead so a reader
arriving there is not misled.
Suite 913 -> 917.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three hand-laid OOXML containers, every part written out by hand and zipped
with a fixed date_time so they are byte-reproducible. No converter output
anywhere in them: a .docx written by the converter and read by the converter
proves only that the converter agrees with itself, and would stay green through
any conversion defect that is symmetric -- which is most of them.
two-line-krav.docx heading + label/value on one line (the docx mirror of
the PDF fixture)
no-styles-krav.docx the SAME document without word/styles.xml
two-line-krav.xlsx sheet name as heading + label/value on one row
THE FIXTURES FOUND A REAL DEFECT IN THE SEAM THEY WERE MEANT TO PIN. The
converter call used pypandoc's TEXT entry point, which takes an `encoding`
because it treats its source as text -- and that corrupts a zip. The xlsx
fixture failed with `Failed to unpack XLSX archive: not enough bytes` while
reading correctly from disk with the same binary. The docx of the same shape
happened to survive, which is the part worth writing down: the defect is silent
for some inputs and fatal for others, so "it worked on the file I tried" was
never evidence. Input now goes through a temporary file.
Two measurements while building, both the same shape -- structurally valid
input, silently reduced output, exit code 0, no warning:
- Without word/styles.xml the docx extracts as flat prose with no heading. A
fixture lacking that part would pin the body and pin nothing about structure.
Committed as a negative control that RUNS rather than a sentence in a README.
- With inline strings rather than a shared string table, the xlsx extracts with
the sheet name intact and every cell value gone. The fixture uses a dimension
element and a shared string table instead.
The frozen literals are pinned to a NAMED converter version, asserted beside
them: a frozen literal without one says "these bytes" without saying what
produced them.
Suite 908 -> 913. Fixtures regenerate byte-identically.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`_PANDOC_FORMATS` names the rows and no others: docx, xlsx, pptx, odt, rtf.
`.html` stays on its stdlib extractor -- routing it through the converter would
buy nothing and would add CVE-2025-51591 (SSRF via an iframe in HTML input),
unpatched in every converter version. `.epub` is out on the "no gain" half of
that.
`_EVIDENCE` records what each row rests on, asserted in the suite rather than
written in a comment: docx and xlsx are `measured`, and pptx, odt and rtf are
`unmeasured` because the corpus contains ZERO files of those types. Three of
five rows therefore leave this step working by construction and never checked
against a document anyone wrote, and the assertion is what keeps that visible.
Three converter arguments, all measured and none of them hygiene:
`--eol=lf --wrap=none` because the defaults produce different bytes (max line
length 75 against 447), and `-t markdown` never `-t plain` because plain
destroys the headings the segment proposer reads -- 15 entries with two real
headings become 13 with none.
`_UNPARSED_OPTIONAL_EXTENSIONS` is now empty and kept rather than deleted: the
branch still raises, and a future type arriving before its reader belongs there
rather than in a new mechanism. This is what the first step was for -- both
tests for `extractor_extra_missing` were repointed at the import probe before
the set emptied under them.
The converter call is isolated behind `_convert_bytes` so the seam's own logic
is testable without the binary; the conversion itself is pinned by frozen-text
fixtures in the next step. Checked live against a hand-laid docx through the
real vendored binary: heading and body both survive.
Suite 895 -> 908.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`_pandoc.py` hands back a converter whose identity is known, or refuses.
The wheel is not enough on its own. pypandoc searches PATH before its own
bundled binary and keeps the highest version found, so on this host the
vendored 3.9 was silently bypassed for the system 3.10.2 -- measured a third
time before writing this. The resolver reads the installed package's own
`files/pandoc` path and asserts the reported version against a frozen
PANDOC_VERSION literal, raising `extractor_binary_version` naming both,
`extractor_binary_missing` when the wheel carries no binary, and
`extractor_extra_missing` when the extra is absent.
A mismatch is refused rather than used with a warning: extraction is
deterministic within a converter version and not across one, and a byte-pinned
fixture cannot tell "a different converter ran" from "we introduced a defect".
Two defects found by measuring rather than by the suite:
1. The first implementation asked `pypandoc.get_pandoc_version()`, which
answers from a module global that `clean_pandocpath_cache()` does not reset.
It therefore reported whichever binary was probed FIRST in the process --
3.10.2 for the bundled 3.9 binary. The suite was green because nothing in it
probed the host binary first. Now `_get_pandoc_version(path)` probes the
argument, with no cache and no search in the way, and a regression test
poisons the cache before resolving. Negative control: that test fails on the
old mechanism.
2. The module docstring named the process-spawning API in prose, which is
enough to fail the model-free gate -- the gate is a grep. Reworded. The gate
now proves the narrower "no model vendor is reachable from src/", stated in
the module rather than glossed.
os.environ is restored on both the success and the failure path, and a
pre-existing override is put back rather than deleted.
Suite 887 -> 895. mypy --strict clean (pypandoc joins the guard's
ignore_missing_imports override; every value it returns is coerced here).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`pypandoc-binary==1.17` joins the `[extract]` extra, and the extra's contents
are now pinned by a test -- `test_packaging.py` asserted `project.dependencies`
only, so a second package could have arrived in the extra unnoticed, which is
precisely where an unexamined transitive tree shows up.
WHY VENDORED RATHER THAN FOUND ON PATH: the xlsx and pptx readers exist only
from pandoc 3.8.3. Debian 12 ships 2.17.1.1 and Ubuntu 24.04 ships 3.1.3, so a
PATH binary cannot deliver two of the five office formats on current stable
distributions. The pin is exact rather than a range for the same reason
pdfminer.six's is: extraction is deterministic within a converter version and
not across one.
The single-runtime-dependency rule is untouched -- it governs
`project.dependencies`, which still names the guard alone.
Measured after installing, on this host:
bundled binary pandoc 3.9 (inside the wheel, as intended)
pypandoc picks 3.10.2 (the host's PATH binary)
That is the third independent measurement of the trap: pypandoc searches PATH
before its own bundled binary and takes the highest version it finds, so
"vendored" buys nothing until something resolves the path explicitly. The
resolver is the next step; until it lands, the vendoring is a pin without an
effect and should not be described as more than that.
Suite 886 -> 887 (the extra is installed, so the packaging pin runs).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Registered before any step raises them, so no later step invents a spelling:
- `extractor_binary_missing` -- the converter binary is absent (distinct
from the extra not being installed)
- `extractor_binary_version` -- present, but not the pinned version
- `extractor_convert_error` -- the converter failed on this file
- `extractor_empty_conversion` -- the converter returned no text
Also widened the `extractor_extra_missing` type list, which still read
"pdf/docx/xlsx".
Denominators recounted after the change rather than carried from a note --
the stale 49/48 figure is what made the recount a step requirement:
code bullets in errors.py 50 -> 54
distinct codes 49 -> 53 (one code documented twice)
test definitions in this suite 56 -> 57 (one parametrized definition,
four cases -- one per code)
The four tests assert only what is true at this step: the code is documented
and an ExtractionError carries it. The resolver and the seam replace each with
a behavioural raise-site test; a code that never gains one stays visible here
as a test that still only reads a docstring.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both tests reached `extractor_extra_missing` through a `.docx`/`.xlsx`
filename, which works only while `_UNPARSED_OPTIONAL_EXTENSIONS` is non-empty.
Those types are about to gain a converter, which empties the set and makes the
membership branch unreachable -- the tests would have gone red for the right
reason at the worst moment, mid-series.
Repointed both at the import probe, the mechanism the pdf gate already uses and
the one path that stays reachable however many types gain parsers.
Measured negative control: without the probe the same call raises
`extractor_pdf_error`, so the probe is load-bearing and the test can still fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Record the two cross-repo decisions the programme ratified on 2026-09-02, so
they exist on this side of the boundary rather than only in a consumer's plan.
- Concept IDs are bundle-local; identity across bundles is the tuple
(bundle_id, concept_id); no cross-bundle link form in v0.2. Written down
before a golden freezes the IDs, because retrofitting would move a pinned
golden.
- The adjudication state has a fixed wire form: frontmatter key `adjudication`
with a closed value set, three companion keys when adjudicated, and an
`adjudication` index facet -- new profile only, SEGMENTED_V1 byte-identical.
Absence of the key means `unknown`, never `absent`.
The ten amendments from the same judgement are applied to the voyage plan
itself, which is LOCAL-ONLY under this repo's public-remote model and so is
not part of this commit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The converter binary is vendored as a pinned wheel in the [extract] extra
rather than resolved from PATH. Three measurements retired the alternative:
the xlsx and pptx readers exist only from pandoc 3.8.3 while Debian 12 ships
2.17.1.1 and Ubuntu 24.04 ships 3.1.3; the docx reader changed its output in
at least 22 of the 3.x releases; and Quarto and pypandoc-binary already made
this call. Two consequences are recorded because they are easy to get wrong:
the single-runtime-dependency rule is untouched, since the extra sits outside
it by construction, and the model-free gate keeps subprocess in its
alternation, since src/ imports a package instead of spawning a process.
One trap is recorded because it is invisible when it fires: pypandoc searches
PATH before its own bundled binary and takes the highest version it finds, so
on a host with a newer system pandoc the vendored wheel is bypassed without a
word and the determinism guarantee goes with it.
Judgement lives in tools/, outside the run path, as versioned data. The
verdict carries its dwell time, because a ratified flag with no per-item time
cannot be falsified. Rejecting is one keystroke and correcting optional,
because requiring a typed correction was measured to reduce flagging. The
model leg is off by default so its value can be measured.
A sixth profile carries both okf_version and bundle_id rather than extending
SEGMENTED_V1, which would have rested on an unverifiable claim about whether
another repository had adopted it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three facts about Door B's intake were decided but never stated in a tracked
file, which left them looking like gaps someone would eventually close.
OCR is permanently out of scope (operator, 2026-09-01). The consequence is
stated rather than implied: the vegnormal-okf PDFs are rastered 11 of 11 and
will never enter through the inbox, so "Vegnormalene in => bundle out" is not
a goal this library holds. That premise never described anything that happened
either -- that bundle is built from seven JSON files through Door C.
The id grammar keeps lowercasing every path component. It is stricter than
OKF v0.2 section 2 requires, but it is shared with Door A and pinned by
byte-exact goldens, so softening it on the segmentation path alone would put
two grammars in one codebase. Measured consequence: in vegnormal-okf's B3 run,
267 of 270 concept identities matched case-folded and 0 matched verbatim.
Structured sources are recorded as open work rather than a closed boundary:
a span cuts the canonical extracted text, and for .json that text is raw data
in a fenced block, so 0 of 267 bodies came back byte-identical.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`d2a8c43` states in a code comment that Door C's ordering holds within a
run and never re-orders entries an earlier run wrote. That was true and
untested: every test in the new file ran each door exactly once, so the
sentence was prose rather than a pin.
Two imports into one bundle, the second adding the concept whose key
sorts FIRST. It lands last, because `link_in_index` appends what is
absent and leaves what is present. The same three concepts merged in one
run do come out ordered, asserted alongside, so the two assertions cannot
both be trivially true -- the difference is the append bound, not an
ordering that failed.
883 tests.
Co-Authored-By: Claude <claude-opus-5>
An index ordering a profile names must be honoured wherever this library
writes an index. Door B and Door C have separate index writers, so an
ordering built on Door B's `_index_sort_key` seam alone would have been a
profile field Door B obeys and Door C ignores -- silently, because
nothing raises and both files still parse. That is
`IndexPolicy.per_directory` again: a field that reads as global and acts
on one path.
`IndexPolicy` gains `sort_key`, `sort_order` and `sort_missing`. Both
order fields draw from CLOSED sets, and `sort_order` is deliberately not
a caller-supplied callable: a callable cannot be serialised into the
bundle, reproduced from it, or audited by a reader, which is the whole of
what a deterministic bundle claims. A `sort_key` the facet policy does
not name is refused too -- every entry would be missing the key and the
ordering would silently do nothing, which is this row's own defect class.
`IndexPolicy.sort_entries` is the one helper. Four stable passes, so each
is the tie-break of the next: concept path, then the named key, then the
missing group partitioned to whichever end the policy says, then
navigation last. Passes 2 and 3 are separate on purpose -- folding them
into one reversible key tuple would flip the missing group along with the
order, so `sort_missing="last"` would mean "first" under `descending`.
The tie-break is the CONCEPT PATH, not the link target, and that is
measured rather than assumed: `notes-beta.md` precedes `notes/alpha.md`
by concept path and follows it by generated filename, so ordering Door C
on the target would have re-ordered every existing Door C bundle.
`IndexEntry` carries the path for that reason; `parse_entry` leaves it
`None` and the ordering falls back to the target, which costs nothing
because no caller sorts entries it read back off disk.
Door B's two reprojection writers and Door C's index emission all route
through the helper. Door B's unfaceted path is not routed and does not
need to be: `sort_key` requires a facet policy, and a faceted profile
never reaches that writer. Door C's guarantee is bounded and stated in
the code -- `link_in_index` appends what is absent and leaves what is
present, so the order holds within a run and never re-orders entries an
earlier run wrote.
Default ordering, unchanged and now stated: with no `sort_key`, concepts
before navigation, each group ascending by concept path.
TDD, and the red was watched twice. First behaviourally with the fields
inert (both doors emitted the exact reverse of the named order), then
again with Door B routed and Door C not -- the broken world reproduced,
where a Door-B-only test would have passed.
882 tests (868 before). The five byte-pinned goldens are untouched and
green; no shipped profile moved.
Co-Authored-By: Claude <claude-opus-5>
The registry file says one test per code and that this file IS the
conformance suite. During the 1-to-N voyage the byte-stability pin over
tests/ was the verification mechanism, so editing this file was
forbidden and six new codes were covered in the new segmentation
modules instead. That was right then. The end-to-end gate run released
the pin, so the convention applies again -- a conformance suite split
across two files stops being one quietly, which is why this was carried
as an obligation rather than a preference.
All seven segmentation codes now have a test here, including
segmentation_plan_unmatched from this session. The behavioural tests in
test_segmentation.py and test_segmented_inbox.py stay where they are:
they exercise the parser and the door, a different question from
whether every documented code has a raise site.
Measured after: 49 documented codes, 48 covered in this file. The one
gap, source_reference_unquotable, predates this work and is tested in
tests/test_okf_v0_2_profile.py:351 -- reported, not silently closed.
Also records both defects and this migration as closed in the decision
record's known-gaps list.
Co-Authored-By: Claude <claude-opus-5>
A segment's title comes from the plan, so a human adjudicated it. But
structure derivation runs over the segment body, finds no title key and
no usable heading, falls back to a stem, and adds "title" to derived.
The concept then emitted a stated fact under an inferred marker, and a
consumer that distrusts derived fields would distrust exactly the thing
a human decided. An over-marked field is the same defect class as an
unmarked heuristic: the marker is only worth something if it is
accurate in both directions.
Scoped to title alone, and pinned that way by test: number stays in
derived on a segment, because nothing about segmentation makes an
inferred document number declared. Without a segment a derived title is
still marked, so only a plan makes a title declared.
The SEGMENTED_V1 golden moves, which is the intended consequence and
the only golden that may. The four existing goldens are byte-identical
to baseline 770d8d4, measured against the sha rather than inspected.
Co-Authored-By: Claude <claude-opus-5>
A plan is selected by content hash, so a mistyped source_sha256 matched
nothing, every dropped file fell through to the one-concept rule, and
process_inbox returned an ordinary success over a flat bundle. The
operator asked for segmentation, got none, and had no error to read --
the silent skip this library refuses everywhere else. vegnormal-okf is
about to run an N500 corpus through this path, where a silent zero
would read as "the corpus has no concepts".
The refusal asks whether a covering plan was FOUND, not whether every
file was examined, so an unreadable drop cannot mask it; and coverage
is recorded at selection, not after path validation, so a matched plan
with a refused entry path still reports its own per-file code. The
first cut got that second question wrong and an existing collision test
caught it; the case is now pinned by its own test, verified red against
the earlier form.
New code segmentation_plan_unmatched, registered in the SegmentationError
docstring register in the same commit. Fail-fast before any disk
mutation. The four existing goldens are byte-identical to baseline.
Co-Authored-By: Claude <claude-opus-5>
The plan's Verification section held twelve gates that cross step
boundaries; /trekexecute checks per-step manifests and never ran them.
Run against the tree at 63565bd with the baseline at 770d8d4, before
any new commit, so the gates measure the delivered work and not
themselves.
Eleven passed. Gate 7 failed: one hunk in tests/test_profile.py, whose
closed field set on BundleProfile gained "segmentation" after
assumption A4 was falsified during execution and the edit ratified.
The gate's mechanism is byte-equality; the property it stands for was
measured directly instead — removing the member makes the test red, so
the assertion is a closed set with one more legitimate term, not a
loosened one. Recorded as failed rather than rewritten to pass.
Two mandatory known-positive controls fired: the zsh :e modifier ate a
path in git cat-file, and an unquoted for-loop did not word-split under
zsh. Both had produced a false absence that read as an answer.
Co-Authored-By: Claude <claude-opus-5>
