run() parses the plan before building anything and parses its own
verdict after. Only the first failure is the operator's file.
The second is reachable: an empty --adjudicator produces a verdict the
grammar refuses ("adjudication field 'adjudicated_by' must be a
non-empty string"). The plan parsed fine; the fault is in what this
command stamped onto it.
Catching SegmentationError at the top of main() -- the previous commit
-- caught both raise sites and printed "malformed plan" for each. On
this path that is a clean, confident, WRONG diagnosis: it sends the
operator to fix the one artifact that was fine. Worse than the traceback
it replaced, because a traceback at least does not claim to know.
The verdict parse now raises AdjudicationError, which is what "this
command failed" already means in this file and already returns 2. Exit
code unchanged either way, nothing written either way; only the message
changes.
Suite 1073 -> 1074 passed (pytest exit 0, measured without a pipe);
ruff and mypy --strict clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A SegmentationError raised by the plan grammar escaped main() as a
traceback and exit 1, while every other malformed-plan case in the same
file already returned 2. Exit codes are the interface a caller scripts
against, and exit 1 with a traceback is the code an unhandled bug
produces -- it says "this command broke" where the truth is "this file
is not a plan".
The refusal itself is unchanged: nothing was written before and nothing
is written now, and the grammar in src/ is untouched. What changes is
one line on stderr naming the error code, and the exit code.
Both branches that can raise are covered: the pre-write parse of a
non-empty plan, and the required-field check reached through the empty
branch.
The old behaviour was pinned by
test_an_entries_value_that_is_not_a_list_is_still_refused, which
asserted that a wrongly-typed `entries` reaches the caller as a raised
SegmentationError and recorded that as a finding rather than fixing it.
That test is rewritten here, in the same commit as the code, to assert
exit 2 plus the code on stderr. A second test pins the one-line stderr
shape on the non-empty branch.
Suite 1072 -> 1073 passed; ruff and mypy --strict clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Order 20260904T172353Z-6290714297-from-.claude. Bilag 9.1's 95.1 % CID
share (docs/2026-09-04-k3-arm-c.md) was found ad hoc, with no committed
script -- the same gap this repo's own fidelity instrument criticizes in
Arm A's uncommitted docx/xlsx figures. okf_cid_measure.py runs the exact
extract_text call the door makes and reports per-document CID share and
4+-letter word count, denominator stated for files it cannot measure.
Red-first: tests/test_cid_measure.py pins measure() against fixture text
of known composition before the implementation existed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Arm C is NOT defined in docs/2026-09-02-k3-k4-k5-metode.md -- that file
contains no occurrence of the word, and neither Arm A nor Arm B is defined
there either. The definition implemented here was written for order
20260904T145630Z and is reported as the author's, never as a ratified one.
Arm C = Arm B's mechanical rules, plus one deterministic rule that cuts
any proposed span longer than a declared cap at the nearest paragraph
boundary at or before it, the whole document counting as one span when
the rules find no boundary at all.
One rule and not two, on purpose. The two failure modes the K2 rebuild
measured -- a PDF with no outline (Bilag 9.1, 217 472 characters) and a PDF
whose headings are its table of contents, so the trailing segment absorbs
the body (Bilag 3.1, Bilag 1) -- are the same failure of SIZE, and a second
rule aimed at each would confound which one moved the number.
`--max-segment-chars` defaults to 0, which is OFF: the artifact is then
byte-identical to Arm B's, pinned by a test that writes both and compares
bytes. The standard profile does not move, and the K2 bundle a consumer is
running against right now is not rebuilt.
What Arm C deliberately does NOT change: the region before the first
candidate is still covered by no segment. That is a real coverage defect --
the K3 baseline's blind rater named it -- and fixing it here would put two
changes behind one measurement.
A part carries TWO rule names in `derived`: the heading rule that opened the
span, and `rule:size-split` for the cut. Dropping the first would make a
part traceable to arithmetic and nothing else.
Tests first: 13 red, then green. 1055 -> 1068.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The guard could not read back what this library WRITES. At 1.2.0,
`okf.parse_frontmatter` refused the OKF v0.2 golden outright --
`OKFFrontmatterError: value begins with a disallowed YAML indicator '['`
against `sources: [{ id: golden-v0-2-sales, resource: fixture }]`. Flow is
the only form this library can emit, because its own line-oriented parser
cannot round-trip the block form at all, so a gate that refuses flow
refuses everything Door A produces under `OKF_V0_2`.
The control was run BEFORE the bump, which is the only moment it exists:
the probe raised on 1.2.0, so the new test discriminates rather than
merely passes. `[project.dependencies]` already said `>=1.2,<2.0` and is
unchanged; only `[tool.uv.sources]` and `uv.lock` move.
TWO gate rows moved, not the one the work was scoped around, which is why
the whole documented probe was re-run instead of just the `sources` case:
the BLOCK form of `sources` now passes too, retiring G30. That changes
nothing about what we emit -- our own parser is still the binding
constraint on writing flow -- and `docs/okf-nokkelinventar.md` now carries
a `guard 1.3.0` column beside the 1.2.0 measurement rather than
overwriting it. A third row kept its verdict but changed its reason, so
the quoted message was corrected too.
The Door C boundary is unmoved, verified with a known-positive:
`resource` is allowlisted only inside a `sources` entry, so section
10.2's `executor.resource` and `attester.resource` are still rejected
("not on the OKF mapping allowlist under 'executor'") while top-level
`resource` passes.
`uv.lock` also gains `pypandoc-binary==1.17`. That is a stale lockfile
being corrected, not a new dependency: it was already declared in the
`[extract]` extra, and `uv lock --check` reports the lockfile out of date
on the untouched tree. Core keeps exactly one runtime dependency.
Not addressed, and recorded rather than built: the guard reports that
`sources[].resource` is scanned as text but never URL-validated, because
SPEC 5.1 permits bundle-relative paths and scope descriptions. No
consumer has asked for a gate there.
Guard 1.3.0 installed from 44e2b31, verified anonymously over https
against the remote tag. 1054 -> 1055 tests. `mypy --strict` clean, `ruff`
clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measured on the K2 artifact by a consumer: `log.md` was on disk and no
index named it, so a reader entering the bundle at `index.md` -- the walk
section 8 exists to support -- never reached the one file carrying `N`.
Stated as a LOCAL choice rather than conformance, because it is one.
Upstream's own bundles do not link their log: measured at `9a15b13`, 0 of
the 24 shipped `index.md` files name the single `log.md` in the set, with
the same grep form finding `tables/index.md` in 4 of them as the
known-positive control. That shows the link is not REQUIRED -- not that
it is disallowed. `docs/plan/okf-v0.2-alignment.md` P1-F6 already
recorded the upstream shape; a line there now separates the two claims,
since reserved names still stay out of an `entries_match_directory`
listing and this profile has that off.
It lives in the harness because the library cannot make it. The log's
content IS the run's outcome, so it cannot exist when the indexes are
projected, and an index that enumerated it off the directory would gain
the link only from the second run onward -- breaking
rebuild-equals-incremental, the property the segmented bundle is built
on.
The membership test is load-bearing and was measured, not assumed. The
two reprojections disagree about this line: the per-directory one drops
every managed entry before re-emitting its block, while the flat one
keeps a managed line whose target is not an owned concept, deliberately,
so that a regex cannot delete curated content. Appending unconditionally
therefore doubled the entry on the second unsegmented run, which is why
both run modes are pinned separately.
1052 -> 1054 tests. `mypy --strict` clean, `ruff` clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`log.md` is written INTO a directory Door B enumerates on the next round: it
matches the concept glob and is excluded only by `index.md`'s name, so a
rebuild could have seen it as pre-existing curated content or pruned it.
Rebuild-equals-incremental is the property the segmented bundle rests on.
Measured on the real artifact, not only the synthetic: the K2 corpus was run
a second time into the same bundle and compared against a snapshot with
`diff -r`, exit 0 over all 1108 files. The test pins the same property in
seconds instead of 13 minutes.
Also corrects the report's reproduction command -- it documented plan
filenames the run did not use, and re-running it into the existing plans
directory would leave two files claiming one `source_sha256`, which
`_resolve_plans` refuses.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measured on the K2 corpus 2026-09-03: 11 of 39 documents proposed zero
segments -- overwhelmingly PDFs with no declared structure, which Topic 1b
had already measured at 23 of 33. The proposer wrote an artifact for each
of them and exited 0.
An empty plan cannot be replayed. `process_inbox` refuses one by design,
because a plan naming no entry would persist nothing for a document that
was dropped, so the only thing a zero-entry file can do is fail a run
later -- and it did: the first segmented corpus run stopped on
`segmentation_plan_invalid` before writing a single concept.
Exit 1 with no artifact, distinct from 2, so a driver can tell "this
document lands as one flat concept" from "stop". The orphan check's test
now observes the same property through the status.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measured on the K2 corpus 2026-09-03: 39 documents proposed 618 entries
under 601 distinct paths -- 17 paths were claimed by two documents each.
Section numbering is document-local (`1 Innledning` is in most procurement
documents), so this is structural, not unlucky. Every collision reaches
Door B's gate, which refuses per DOCUMENT, so those documents would land as
coded rejections rather than concepts and a corpus run could not be built
at all.
`--path-prefix` is an argument and not something the tool derives: the
proposer sees ONE document and cannot know what else is in the bundle. It
is reduced to the id grammar before anything is read, and a prefix that
reduces to nothing is refused rather than silently producing the unscoped
paths the caller asked to avoid. Without the flag every artifact already
produced is byte-identical.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two defects a consumer measured on 2026-09-03 against the bundle built by
this harness, both with one cause: `measure` passed `profile=STRUCTURED_V1`
and no plans at all, though Step 17 of the plan says the harness reuses
`process_inbox` "with the per-document plan mapping from Step 15".
- `adjudication` was in 0 of 39 concepts, because the key is written only
inside the plan-covered branch and no plan was ever passed. `--plans-dir`
replays proposals produced per document first; the profile follows from
the flag rather than being something the harness may choose on its own.
`--bundle-id` and `--okf-version` are arguments, never constants: a
profile names a key and the caller owns its value (decision E1).
- `log.md` did not exist, so `merged` was countable from the bundle and `N`
was not -- K1b could only be taken on trust from a report that does not
travel with the artifact. Written in SPEC section 9 form and dated from
`ingested_at`, never the wall clock.
Additive: without `--plans-dir` the run stays the flat `STRUCTURED_V1` run
that produced the published K1/K2 numbers.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measured on the K3 corpus: 4 of 12 judgements produced no artifact, because
the verdict was "none of these segments should be persisted" and the plan
grammar refuses zero entries. That refusal is correct for the run path -- an
empty plan replayed would silently persist nothing for a document that was
dropped -- so the grammar is untouched and the recording tool is taught to
record a rejection instead.
Refusing to materialize and refusing to record are different acts. The
rejection artifact is deliberately NOT replayable: parse_segmentation_plan
still refuses it, and the suite asserts that rather than assuming it. The dwell
time rides at the top level because there is no entry to carry it, and a
ratified rejection with no time on it is as unfalsifiable as a ratified
acceptance with none.
Only the empty LIST takes the branch. A missing entries key, or one that is not
a list, stays the grammar's to refuse: "the adjudicator kept nothing" and
"this file is not a plan" must not collapse.
K4a re-run after the change: propose, adjudicate, run the path twice into two
bundles under SEGMENTED_OKF_V0_2, diff -r exit 0 with no output. 1034 -> 1041
tests.
PM decision B6 asked for a list-taking _render_sources so a concept can record
more than one source, and prescribed the block list as the emitted form. The
list is delivered; the block form is not.
Three measurements, not an argument. Our own parse_frontmatter skips indented
lines, so a block list round-trips to an empty value with every entry silently
gone -- and _is_ingest_owned reads through that same parser. The consumer B6
was written for accepts the multi-entry flow sequence and classifies a block
sequence as unreadable provenance, so block would hand it exactly the state it
cannot read. And B6's own acceptance test asks for a round trip through this
parser, which no block form can pass.
A single source renders byte-identically, so all six goldens are unmoved. The
unquotable-value gate now runs on every entry, not just the first. New code
sources_empty refuses an empty list.
1023 -> 1034 tests, including the negative control that pins the block form's
silent data loss.
Fourteen rules, each with its own code: one "invalid" verdict over fourteen
different defects is a diagnostic no caller can act on. The report quotes its
own denominators -- rules run, excerpts and withheld entries examined -- because
a checker that exempted itself from section 5 would be stating the rule it
breaks.
Three exit codes, not two. "The check did not run" and "the check failed" are
different outcomes, and an unread file reported as a failed check is the fourth
face of the verification law.
The shipped template plus its example payload is the known-positive arm, so a
checker that refuses everything cannot be green on the thirteen negative ones.
A last test asserts every literal the checker enforces appears in the contract
document: two copies of a closed set drift, and the copy nobody reads is the
one that goes wrong.
996 -> 1023 tests.
A sixth golden directory, never an edit to the five that exist. Its only
difference from the SEGMENTED_V1 golden is the added `okf_version` root key --
measured by diffing the two -- which is exactly what the profile is for and
nothing else moved.
Modelled on tests/test_segmented_golden.py rather than test_golden.py's
`materialize_case`, which is Door A only: it drives materialize_bundle from a
manifest and this bundle comes through Door B's inbox. The plan's pointer to an
"existing goldens list" in test_golden.py is stale -- that file holds only the
Door A parametrize, and `ingest-golden-segmented` is not in it either. Door B
goldens carrying their own test file is the established pattern, and this one
asserts the five priors are untouched from its own side.
CROSS-BUNDLE IDENTITY (PM decision B1) is settled BEFORE this pin, not after.
Byte-pinning a bundle carrying `okf_version` is where its concept IDs stop
being adjustable: a concept ID is bundle-local and stable, identity across
bundles is the tuple (bundle_id, concept_id), and there is no cross-bundle link
form in v0.2. Asserted -- every concept carries its bundle id, so the tuple is
readable from one document, and no foreign bundle id appears anywhere.
Two defects in these tests, both found by running them:
- `parse_frontmatter` takes a Path and returns the dict; three call sites had
it wrong.
- The cross-bundle check used `b-golden-segmented` as the foreign id, which is
a PREFIX of this bundle's own id. It failed on every file for the wrong
reason and would have read as "a cross-bundle reference was emitted". Now a
foreign id that is not a prefix, with a known-positive half asserting the
bundle does name its own id -- so the check is looking at content that could
have carried one.
Negative control: changing one byte of the golden's `okf_version` turns the pin
red.
`okf_version` and `bundle_id` are fixture DATA read from the case, never
constants in the test -- the value belongs to catalog (E1).
Suite 931 -> 941.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Neither shipped profile could do both: SEGMENTED_V1 names `bundle_id` and not
`okf_version`, OKF_V0_2 names `okf_version` and has no segmentation at all. The
sixth profile is where the two intersect. Additive, as upstream support always
is here -- a new profile, never a migration.
THE INDEX POLICY IS DECIDED, NOT INHERITED, and that is the part that was easy
to get wrong. Measured: OKF_V0_2.index has facets=None and per_directory=False,
while SEGMENTED_V1.index has both. Building on OKF_V0_2's index would have
produced a segmented bundle with no faceted index -- structurally valid, SPEC
conformant, and missing the surface a consumer reads. So the index comes from
SEGMENTED_V1 with both root keys named, and the spec declaration from OKF_V0_2.
The premise is asserted in the suite rather than trusted to stay true.
`SegmentationPolicy.adjudication_key` is the discriminator BETWEEN segmented
profiles. It has to be a field with a value: every 1-to-N branch keys on
`profile.segmentation is not None`, which both segmented profiles satisfy, so a
later step surfacing the adjudication marker on that check would write it into
SEGMENTED_V1 too and move a byte-pinned golden. `None` means the profile does
not surface adjudication state at all.
`okf_version`'s VALUE is not in this module and must never be: a profile names
a key, the caller owns its value (decision E1). A constant here would claim a
decision this library does not own and would be the one thing to chase on every
upstream release. Asserted.
Five existing profiles untouched, each still equal to a freshly constructed
copy, root keys pinned literally, and OKF_LATEST still DEFAULT -- adding a
profile is not a GA event.
Suite 926 -> 931.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Arm E, capability only. A profile MAY name a renderer per suffix; no
domain-aware renderer is written here, that stays a Non-Goal, and `_RENDERERS`
is empty on purpose so the emptiness reads as a decision rather than an
omission.
THE LAYERING IS THE DESIGN, not an implementation detail. `extract.py` is the
extraction registry and must not import the contract layer, or the dependency
runs backwards and the registry stops standing on its own. So `extract_text`
gains a keyword-only `renderer: Callable[[str], str] | None`, knowing nothing
about profiles, and `inbox.py` -- which already holds the profile at that call
site -- resolves a NAME to a function. A test asserts extract.py still contains
no reference to the profile layer, because that constraint is the whole reason
the parameter is shaped this way.
The renderer runs AFTER extraction, never instead of it, so it never has to
re-implement a reader and the two cannot drift. The default is identity, which
is what keeps the five byte-pinned goldens byte-pinned -- asserted per suffix
rather than once.
An unknown renderer NAME is refused rather than falling back to identity: a
silent fallback would produce a bundle that looks rendered and is not, which is
the failure mode this arm exists to make visible. That needed a registered code
(`unknown_renderer`) and its test -- slightly beyond the step's named files,
but the capability cannot ship without defining what an unknown name does.
`tests/test_profile.py`'s exact-field-set assertion went red, as the plan's risk
table predicted. Updated deliberately with the reason recorded: that assertion
exists so a field cannot arrive without someone deciding it should, and its red
run is the mechanism working.
Suite 917 -> 926. All five goldens byte-identical.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The README told consumers that `docx` and `xlsx` ship no parser and always fail
fast. True when written; false the moment the converter seam landed -- and
false SILENTLY, because prose has no test. This repository has been bitten by
that exact shape before: a published guarantee is a test obligation.
So the correction comes with `tests/test_docs_promises.py`, which compares the
README's declared format list against the registries it describes and fails on
a format added without touching the README, on the old claim reappearing in any
wording, on an unmeasured row going unnamed, and on the exclusions being
dropped. Negative control: removing one format from the README's marker turns
it red.
The README now states which rows are measured and which are not. Three of the
five office rows have denominator ZERO in the corpus -- they work by
construction and have never met a document anyone wrote. They are not known to
be broken and not known to be right, and a reader should not have to open the
source to learn which.
The CHANGELOG's shipped entry is left as written, because a changelog records
what a release did; the correction is stated at that line instead so a reader
arriving there is not misled.
Suite 913 -> 917.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three hand-laid OOXML containers, every part written out by hand and zipped
with a fixed date_time so they are byte-reproducible. No converter output
anywhere in them: a .docx written by the converter and read by the converter
proves only that the converter agrees with itself, and would stay green through
any conversion defect that is symmetric -- which is most of them.
two-line-krav.docx heading + label/value on one line (the docx mirror of
the PDF fixture)
no-styles-krav.docx the SAME document without word/styles.xml
two-line-krav.xlsx sheet name as heading + label/value on one row
THE FIXTURES FOUND A REAL DEFECT IN THE SEAM THEY WERE MEANT TO PIN. The
converter call used pypandoc's TEXT entry point, which takes an `encoding`
because it treats its source as text -- and that corrupts a zip. The xlsx
fixture failed with `Failed to unpack XLSX archive: not enough bytes` while
reading correctly from disk with the same binary. The docx of the same shape
happened to survive, which is the part worth writing down: the defect is silent
for some inputs and fatal for others, so "it worked on the file I tried" was
never evidence. Input now goes through a temporary file.
Two measurements while building, both the same shape -- structurally valid
input, silently reduced output, exit code 0, no warning:
- Without word/styles.xml the docx extracts as flat prose with no heading. A
fixture lacking that part would pin the body and pin nothing about structure.
Committed as a negative control that RUNS rather than a sentence in a README.
- With inline strings rather than a shared string table, the xlsx extracts with
the sheet name intact and every cell value gone. The fixture uses a dimension
element and a shared string table instead.
The frozen literals are pinned to a NAMED converter version, asserted beside
them: a frozen literal without one says "these bytes" without saying what
produced them.
Suite 908 -> 913. Fixtures regenerate byte-identically.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`_PANDOC_FORMATS` names the rows and no others: docx, xlsx, pptx, odt, rtf.
`.html` stays on its stdlib extractor -- routing it through the converter would
buy nothing and would add CVE-2025-51591 (SSRF via an iframe in HTML input),
unpatched in every converter version. `.epub` is out on the "no gain" half of
that.
`_EVIDENCE` records what each row rests on, asserted in the suite rather than
written in a comment: docx and xlsx are `measured`, and pptx, odt and rtf are
`unmeasured` because the corpus contains ZERO files of those types. Three of
five rows therefore leave this step working by construction and never checked
against a document anyone wrote, and the assertion is what keeps that visible.
Three converter arguments, all measured and none of them hygiene:
`--eol=lf --wrap=none` because the defaults produce different bytes (max line
length 75 against 447), and `-t markdown` never `-t plain` because plain
destroys the headings the segment proposer reads -- 15 entries with two real
headings become 13 with none.
`_UNPARSED_OPTIONAL_EXTENSIONS` is now empty and kept rather than deleted: the
branch still raises, and a future type arriving before its reader belongs there
rather than in a new mechanism. This is what the first step was for -- both
tests for `extractor_extra_missing` were repointed at the import probe before
the set emptied under them.
The converter call is isolated behind `_convert_bytes` so the seam's own logic
is testable without the binary; the conversion itself is pinned by frozen-text
fixtures in the next step. Checked live against a hand-laid docx through the
real vendored binary: heading and body both survive.
Suite 895 -> 908.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`_pandoc.py` hands back a converter whose identity is known, or refuses.
The wheel is not enough on its own. pypandoc searches PATH before its own
bundled binary and keeps the highest version found, so on this host the
vendored 3.9 was silently bypassed for the system 3.10.2 -- measured a third
time before writing this. The resolver reads the installed package's own
`files/pandoc` path and asserts the reported version against a frozen
PANDOC_VERSION literal, raising `extractor_binary_version` naming both,
`extractor_binary_missing` when the wheel carries no binary, and
`extractor_extra_missing` when the extra is absent.
A mismatch is refused rather than used with a warning: extraction is
deterministic within a converter version and not across one, and a byte-pinned
fixture cannot tell "a different converter ran" from "we introduced a defect".
Two defects found by measuring rather than by the suite:
1. The first implementation asked `pypandoc.get_pandoc_version()`, which
answers from a module global that `clean_pandocpath_cache()` does not reset.
It therefore reported whichever binary was probed FIRST in the process --
3.10.2 for the bundled 3.9 binary. The suite was green because nothing in it
probed the host binary first. Now `_get_pandoc_version(path)` probes the
argument, with no cache and no search in the way, and a regression test
poisons the cache before resolving. Negative control: that test fails on the
old mechanism.
2. The module docstring named the process-spawning API in prose, which is
enough to fail the model-free gate -- the gate is a grep. Reworded. The gate
now proves the narrower "no model vendor is reachable from src/", stated in
the module rather than glossed.
os.environ is restored on both the success and the failure path, and a
pre-existing override is put back rather than deleted.
Suite 887 -> 895. mypy --strict clean (pypandoc joins the guard's
ignore_missing_imports override; every value it returns is coerced here).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`pypandoc-binary==1.17` joins the `[extract]` extra, and the extra's contents
are now pinned by a test -- `test_packaging.py` asserted `project.dependencies`
only, so a second package could have arrived in the extra unnoticed, which is
precisely where an unexamined transitive tree shows up.
WHY VENDORED RATHER THAN FOUND ON PATH: the xlsx and pptx readers exist only
from pandoc 3.8.3. Debian 12 ships 2.17.1.1 and Ubuntu 24.04 ships 3.1.3, so a
PATH binary cannot deliver two of the five office formats on current stable
distributions. The pin is exact rather than a range for the same reason
pdfminer.six's is: extraction is deterministic within a converter version and
not across one.
The single-runtime-dependency rule is untouched -- it governs
`project.dependencies`, which still names the guard alone.
Measured after installing, on this host:
bundled binary pandoc 3.9 (inside the wheel, as intended)
pypandoc picks 3.10.2 (the host's PATH binary)
That is the third independent measurement of the trap: pypandoc searches PATH
before its own bundled binary and takes the highest version it finds, so
"vendored" buys nothing until something resolves the path explicitly. The
resolver is the next step; until it lands, the vendoring is a pin without an
effect and should not be described as more than that.
Suite 886 -> 887 (the extra is installed, so the packaging pin runs).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Registered before any step raises them, so no later step invents a spelling:
- `extractor_binary_missing` -- the converter binary is absent (distinct
from the extra not being installed)
- `extractor_binary_version` -- present, but not the pinned version
- `extractor_convert_error` -- the converter failed on this file
- `extractor_empty_conversion` -- the converter returned no text
Also widened the `extractor_extra_missing` type list, which still read
"pdf/docx/xlsx".
Denominators recounted after the change rather than carried from a note --
the stale 49/48 figure is what made the recount a step requirement:
code bullets in errors.py 50 -> 54
distinct codes 49 -> 53 (one code documented twice)
test definitions in this suite 56 -> 57 (one parametrized definition,
four cases -- one per code)
The four tests assert only what is true at this step: the code is documented
and an ExtractionError carries it. The resolver and the seam replace each with
a behavioural raise-site test; a code that never gains one stays visible here
as a test that still only reads a docstring.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both tests reached `extractor_extra_missing` through a `.docx`/`.xlsx`
filename, which works only while `_UNPARSED_OPTIONAL_EXTENSIONS` is non-empty.
Those types are about to gain a converter, which empties the set and makes the
membership branch unreachable -- the tests would have gone red for the right
reason at the worst moment, mid-series.
Repointed both at the import probe, the mechanism the pdf gate already uses and
the one path that stays reachable however many types gain parsers.
Measured negative control: without the probe the same call raises
`extractor_pdf_error`, so the probe is load-bearing and the test can still fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`d2a8c43` states in a code comment that Door C's ordering holds within a
run and never re-orders entries an earlier run wrote. That was true and
untested: every test in the new file ran each door exactly once, so the
sentence was prose rather than a pin.
Two imports into one bundle, the second adding the concept whose key
sorts FIRST. It lands last, because `link_in_index` appends what is
absent and leaves what is present. The same three concepts merged in one
run do come out ordered, asserted alongside, so the two assertions cannot
both be trivially true -- the difference is the append bound, not an
ordering that failed.
883 tests.
Co-Authored-By: Claude <claude-opus-5>
An index ordering a profile names must be honoured wherever this library
writes an index. Door B and Door C have separate index writers, so an
ordering built on Door B's `_index_sort_key` seam alone would have been a
profile field Door B obeys and Door C ignores -- silently, because
nothing raises and both files still parse. That is
`IndexPolicy.per_directory` again: a field that reads as global and acts
on one path.
`IndexPolicy` gains `sort_key`, `sort_order` and `sort_missing`. Both
order fields draw from CLOSED sets, and `sort_order` is deliberately not
a caller-supplied callable: a callable cannot be serialised into the
bundle, reproduced from it, or audited by a reader, which is the whole of
what a deterministic bundle claims. A `sort_key` the facet policy does
not name is refused too -- every entry would be missing the key and the
ordering would silently do nothing, which is this row's own defect class.
`IndexPolicy.sort_entries` is the one helper. Four stable passes, so each
is the tie-break of the next: concept path, then the named key, then the
missing group partitioned to whichever end the policy says, then
navigation last. Passes 2 and 3 are separate on purpose -- folding them
into one reversible key tuple would flip the missing group along with the
order, so `sort_missing="last"` would mean "first" under `descending`.
The tie-break is the CONCEPT PATH, not the link target, and that is
measured rather than assumed: `notes-beta.md` precedes `notes/alpha.md`
by concept path and follows it by generated filename, so ordering Door C
on the target would have re-ordered every existing Door C bundle.
`IndexEntry` carries the path for that reason; `parse_entry` leaves it
`None` and the ordering falls back to the target, which costs nothing
because no caller sorts entries it read back off disk.
Door B's two reprojection writers and Door C's index emission all route
through the helper. Door B's unfaceted path is not routed and does not
need to be: `sort_key` requires a facet policy, and a faceted profile
never reaches that writer. Door C's guarantee is bounded and stated in
the code -- `link_in_index` appends what is absent and leaves what is
present, so the order holds within a run and never re-orders entries an
earlier run wrote.
Default ordering, unchanged and now stated: with no `sort_key`, concepts
before navigation, each group ascending by concept path.
TDD, and the red was watched twice. First behaviourally with the fields
inert (both doors emitted the exact reverse of the named order), then
again with Door B routed and Door C not -- the broken world reproduced,
where a Door-B-only test would have passed.
882 tests (868 before). The five byte-pinned goldens are untouched and
green; no shipped profile moved.
Co-Authored-By: Claude <claude-opus-5>
The registry file says one test per code and that this file IS the
conformance suite. During the 1-to-N voyage the byte-stability pin over
tests/ was the verification mechanism, so editing this file was
forbidden and six new codes were covered in the new segmentation
modules instead. That was right then. The end-to-end gate run released
the pin, so the convention applies again -- a conformance suite split
across two files stops being one quietly, which is why this was carried
as an obligation rather than a preference.
All seven segmentation codes now have a test here, including
segmentation_plan_unmatched from this session. The behavioural tests in
test_segmentation.py and test_segmented_inbox.py stay where they are:
they exercise the parser and the door, a different question from
whether every documented code has a raise site.
Measured after: 49 documented codes, 48 covered in this file. The one
gap, source_reference_unquotable, predates this work and is tested in
tests/test_okf_v0_2_profile.py:351 -- reported, not silently closed.
Also records both defects and this migration as closed in the decision
record's known-gaps list.
Co-Authored-By: Claude <claude-opus-5>
A segment's title comes from the plan, so a human adjudicated it. But
structure derivation runs over the segment body, finds no title key and
no usable heading, falls back to a stem, and adds "title" to derived.
The concept then emitted a stated fact under an inferred marker, and a
consumer that distrusts derived fields would distrust exactly the thing
a human decided. An over-marked field is the same defect class as an
unmarked heuristic: the marker is only worth something if it is
accurate in both directions.
Scoped to title alone, and pinned that way by test: number stays in
derived on a segment, because nothing about segmentation makes an
inferred document number declared. Without a segment a derived title is
still marked, so only a plan makes a title declared.
The SEGMENTED_V1 golden moves, which is the intended consequence and
the only golden that may. The four existing goldens are byte-identical
to baseline 770d8d4, measured against the sha rather than inspected.
Co-Authored-By: Claude <claude-opus-5>
A plan is selected by content hash, so a mistyped source_sha256 matched
nothing, every dropped file fell through to the one-concept rule, and
process_inbox returned an ordinary success over a flat bundle. The
operator asked for segmentation, got none, and had no error to read --
the silent skip this library refuses everywhere else. vegnormal-okf is
about to run an N500 corpus through this path, where a silent zero
would read as "the corpus has no concepts".
The refusal asks whether a covering plan was FOUND, not whether every
file was examined, so an unreadable drop cannot mask it; and coverage
is recorded at selection, not after path validation, so a matched plan
with a refused entry path still reports its own per-file code. The
first cut got that second question wrong and an existing collision test
caught it; the case is now pinned by its own test, verified red against
the earlier form.
New code segmentation_plan_unmatched, registered in the SegmentationError
docstring register in the same commit. Fail-fast before any disk
mutation. The four existing goldens are byte-identical to baseline.
Co-Authored-By: Claude <claude-opus-5>
The line-oriented frontmatter grammar exists in three copies, each with the
duplication documented at its site: `materialize` reads a path, `structure`
needs a character offset, `profiles` returns body lines. All three keyed on
`key.strip()`, which discards the indentation that is the only thing telling
a nested key from a top-level one. An indented `title:` under a `sources:`
block therefore landed in the same flat namespace as the document's own
`title:` and, arriving later, won.
The failure is substitution, not omission. A dropped value is visible to
whoever reads the concept; a substituted one is not -- the document carries a
title that looks entirely right and belongs to something else. Because
`number` derives from `title` and `parent` derives from `number`, one
substitution walks the hierarchy. Measured, not inferred: a document titled
`N100.2` with a nested source titled `N200.7` came back as N200.7 with parent
N200 instead of N100.2 with parent N100.
Measured incidence across the two corpora, denominators stated:
`_okf-canonical` @ ad30107, 54 documents with parsable frontmatter, 49 carry
a nested key colliding with a top-level name (90.7%); `_okf-upstream` @
9a15b13, 66 documents, 58 collide (87.9%). The colliding key is `title`, and
often `resource` with it -- in `acme_retail/tables/orders.md` the concept's
own BigQuery resource pointer was replaced by a nested one. This is a fix
that clears observed damage, not a hardening without a witness.
The fix refuses indented lines; it does not read them. Block form stays
unreadable -- `sources` and `verified` still come back empty -- so D4's
flow-form emission rule is untouched and the structured reader is still D1b.
Two characterization tests that pinned the old behaviour now pin the new: the
block-list family still DROPS its value, and only the key-space pollution is
gone. That family is not otherwise addressed here.
Test first, red before the code was touched, with known-positive controls for
all three parsers so that a parser returning nothing could not pass.
Order: 20260830T000740Z-4733930312-from-.claude
Co-Authored-By: Claude <claude-opus-5>
