A sixth golden directory, never an edit to the five that exist. Its only
difference from the SEGMENTED_V1 golden is the added `okf_version` root key --
measured by diffing the two -- which is exactly what the profile is for and
nothing else moved.
Modelled on tests/test_segmented_golden.py rather than test_golden.py's
`materialize_case`, which is Door A only: it drives materialize_bundle from a
manifest and this bundle comes through Door B's inbox. The plan's pointer to an
"existing goldens list" in test_golden.py is stale -- that file holds only the
Door A parametrize, and `ingest-golden-segmented` is not in it either. Door B
goldens carrying their own test file is the established pattern, and this one
asserts the five priors are untouched from its own side.
CROSS-BUNDLE IDENTITY (PM decision B1) is settled BEFORE this pin, not after.
Byte-pinning a bundle carrying `okf_version` is where its concept IDs stop
being adjustable: a concept ID is bundle-local and stable, identity across
bundles is the tuple (bundle_id, concept_id), and there is no cross-bundle link
form in v0.2. Asserted -- every concept carries its bundle id, so the tuple is
readable from one document, and no foreign bundle id appears anywhere.
Two defects in these tests, both found by running them:
- `parse_frontmatter` takes a Path and returns the dict; three call sites had
it wrong.
- The cross-bundle check used `b-golden-segmented` as the foreign id, which is
a PREFIX of this bundle's own id. It failed on every file for the wrong
reason and would have read as "a cross-bundle reference was emitted". Now a
foreign id that is not a prefix, with a known-positive half asserting the
bundle does name its own id -- so the check is looking at content that could
have carried one.
Negative control: changing one byte of the golden's `okf_version` turns the pin
red.
`okf_version` and `bundle_id` are fixture DATA read from the case, never
constants in the test -- the value belongs to catalog (E1).
Suite 931 -> 941.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Neither shipped profile could do both: SEGMENTED_V1 names `bundle_id` and not
`okf_version`, OKF_V0_2 names `okf_version` and has no segmentation at all. The
sixth profile is where the two intersect. Additive, as upstream support always
is here -- a new profile, never a migration.
THE INDEX POLICY IS DECIDED, NOT INHERITED, and that is the part that was easy
to get wrong. Measured: OKF_V0_2.index has facets=None and per_directory=False,
while SEGMENTED_V1.index has both. Building on OKF_V0_2's index would have
produced a segmented bundle with no faceted index -- structurally valid, SPEC
conformant, and missing the surface a consumer reads. So the index comes from
SEGMENTED_V1 with both root keys named, and the spec declaration from OKF_V0_2.
The premise is asserted in the suite rather than trusted to stay true.
`SegmentationPolicy.adjudication_key` is the discriminator BETWEEN segmented
profiles. It has to be a field with a value: every 1-to-N branch keys on
`profile.segmentation is not None`, which both segmented profiles satisfy, so a
later step surfacing the adjudication marker on that check would write it into
SEGMENTED_V1 too and move a byte-pinned golden. `None` means the profile does
not surface adjudication state at all.
`okf_version`'s VALUE is not in this module and must never be: a profile names
a key, the caller owns its value (decision E1). A constant here would claim a
decision this library does not own and would be the one thing to chase on every
upstream release. Asserted.
Five existing profiles untouched, each still equal to a freshly constructed
copy, root keys pinned literally, and OKF_LATEST still DEFAULT -- adding a
profile is not a GA event.
Suite 926 -> 931.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Arm E, capability only. A profile MAY name a renderer per suffix; no
domain-aware renderer is written here, that stays a Non-Goal, and `_RENDERERS`
is empty on purpose so the emptiness reads as a decision rather than an
omission.
THE LAYERING IS THE DESIGN, not an implementation detail. `extract.py` is the
extraction registry and must not import the contract layer, or the dependency
runs backwards and the registry stops standing on its own. So `extract_text`
gains a keyword-only `renderer: Callable[[str], str] | None`, knowing nothing
about profiles, and `inbox.py` -- which already holds the profile at that call
site -- resolves a NAME to a function. A test asserts extract.py still contains
no reference to the profile layer, because that constraint is the whole reason
the parameter is shaped this way.
The renderer runs AFTER extraction, never instead of it, so it never has to
re-implement a reader and the two cannot drift. The default is identity, which
is what keeps the five byte-pinned goldens byte-pinned -- asserted per suffix
rather than once.
An unknown renderer NAME is refused rather than falling back to identity: a
silent fallback would produce a bundle that looks rendered and is not, which is
the failure mode this arm exists to make visible. That needed a registered code
(`unknown_renderer`) and its test -- slightly beyond the step's named files,
but the capability cannot ship without defining what an unknown name does.
`tests/test_profile.py`'s exact-field-set assertion went red, as the plan's risk
table predicted. Updated deliberately with the reason recorded: that assertion
exists so a field cannot arrive without someone deciding it should, and its red
run is the mechanism working.
Suite 917 -> 926. All five goldens byte-identical.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The README told consumers that `docx` and `xlsx` ship no parser and always fail
fast. True when written; false the moment the converter seam landed -- and
false SILENTLY, because prose has no test. This repository has been bitten by
that exact shape before: a published guarantee is a test obligation.
So the correction comes with `tests/test_docs_promises.py`, which compares the
README's declared format list against the registries it describes and fails on
a format added without touching the README, on the old claim reappearing in any
wording, on an unmeasured row going unnamed, and on the exclusions being
dropped. Negative control: removing one format from the README's marker turns
it red.
The README now states which rows are measured and which are not. Three of the
five office rows have denominator ZERO in the corpus -- they work by
construction and have never met a document anyone wrote. They are not known to
be broken and not known to be right, and a reader should not have to open the
source to learn which.
The CHANGELOG's shipped entry is left as written, because a changelog records
what a release did; the correction is stated at that line instead so a reader
arriving there is not misled.
Suite 913 -> 917.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three hand-laid OOXML containers, every part written out by hand and zipped
with a fixed date_time so they are byte-reproducible. No converter output
anywhere in them: a .docx written by the converter and read by the converter
proves only that the converter agrees with itself, and would stay green through
any conversion defect that is symmetric -- which is most of them.
two-line-krav.docx heading + label/value on one line (the docx mirror of
the PDF fixture)
no-styles-krav.docx the SAME document without word/styles.xml
two-line-krav.xlsx sheet name as heading + label/value on one row
THE FIXTURES FOUND A REAL DEFECT IN THE SEAM THEY WERE MEANT TO PIN. The
converter call used pypandoc's TEXT entry point, which takes an `encoding`
because it treats its source as text -- and that corrupts a zip. The xlsx
fixture failed with `Failed to unpack XLSX archive: not enough bytes` while
reading correctly from disk with the same binary. The docx of the same shape
happened to survive, which is the part worth writing down: the defect is silent
for some inputs and fatal for others, so "it worked on the file I tried" was
never evidence. Input now goes through a temporary file.
Two measurements while building, both the same shape -- structurally valid
input, silently reduced output, exit code 0, no warning:
- Without word/styles.xml the docx extracts as flat prose with no heading. A
fixture lacking that part would pin the body and pin nothing about structure.
Committed as a negative control that RUNS rather than a sentence in a README.
- With inline strings rather than a shared string table, the xlsx extracts with
the sheet name intact and every cell value gone. The fixture uses a dimension
element and a shared string table instead.
The frozen literals are pinned to a NAMED converter version, asserted beside
them: a frozen literal without one says "these bytes" without saying what
produced them.
Suite 908 -> 913. Fixtures regenerate byte-identically.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`_PANDOC_FORMATS` names the rows and no others: docx, xlsx, pptx, odt, rtf.
`.html` stays on its stdlib extractor -- routing it through the converter would
buy nothing and would add CVE-2025-51591 (SSRF via an iframe in HTML input),
unpatched in every converter version. `.epub` is out on the "no gain" half of
that.
`_EVIDENCE` records what each row rests on, asserted in the suite rather than
written in a comment: docx and xlsx are `measured`, and pptx, odt and rtf are
`unmeasured` because the corpus contains ZERO files of those types. Three of
five rows therefore leave this step working by construction and never checked
against a document anyone wrote, and the assertion is what keeps that visible.
Three converter arguments, all measured and none of them hygiene:
`--eol=lf --wrap=none` because the defaults produce different bytes (max line
length 75 against 447), and `-t markdown` never `-t plain` because plain
destroys the headings the segment proposer reads -- 15 entries with two real
headings become 13 with none.
`_UNPARSED_OPTIONAL_EXTENSIONS` is now empty and kept rather than deleted: the
branch still raises, and a future type arriving before its reader belongs there
rather than in a new mechanism. This is what the first step was for -- both
tests for `extractor_extra_missing` were repointed at the import probe before
the set emptied under them.
The converter call is isolated behind `_convert_bytes` so the seam's own logic
is testable without the binary; the conversion itself is pinned by frozen-text
fixtures in the next step. Checked live against a hand-laid docx through the
real vendored binary: heading and body both survive.
Suite 895 -> 908.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`_pandoc.py` hands back a converter whose identity is known, or refuses.
The wheel is not enough on its own. pypandoc searches PATH before its own
bundled binary and keeps the highest version found, so on this host the
vendored 3.9 was silently bypassed for the system 3.10.2 -- measured a third
time before writing this. The resolver reads the installed package's own
`files/pandoc` path and asserts the reported version against a frozen
PANDOC_VERSION literal, raising `extractor_binary_version` naming both,
`extractor_binary_missing` when the wheel carries no binary, and
`extractor_extra_missing` when the extra is absent.
A mismatch is refused rather than used with a warning: extraction is
deterministic within a converter version and not across one, and a byte-pinned
fixture cannot tell "a different converter ran" from "we introduced a defect".
Two defects found by measuring rather than by the suite:
1. The first implementation asked `pypandoc.get_pandoc_version()`, which
answers from a module global that `clean_pandocpath_cache()` does not reset.
It therefore reported whichever binary was probed FIRST in the process --
3.10.2 for the bundled 3.9 binary. The suite was green because nothing in it
probed the host binary first. Now `_get_pandoc_version(path)` probes the
argument, with no cache and no search in the way, and a regression test
poisons the cache before resolving. Negative control: that test fails on the
old mechanism.
2. The module docstring named the process-spawning API in prose, which is
enough to fail the model-free gate -- the gate is a grep. Reworded. The gate
now proves the narrower "no model vendor is reachable from src/", stated in
the module rather than glossed.
os.environ is restored on both the success and the failure path, and a
pre-existing override is put back rather than deleted.
Suite 887 -> 895. mypy --strict clean (pypandoc joins the guard's
ignore_missing_imports override; every value it returns is coerced here).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`pypandoc-binary==1.17` joins the `[extract]` extra, and the extra's contents
are now pinned by a test -- `test_packaging.py` asserted `project.dependencies`
only, so a second package could have arrived in the extra unnoticed, which is
precisely where an unexamined transitive tree shows up.
WHY VENDORED RATHER THAN FOUND ON PATH: the xlsx and pptx readers exist only
from pandoc 3.8.3. Debian 12 ships 2.17.1.1 and Ubuntu 24.04 ships 3.1.3, so a
PATH binary cannot deliver two of the five office formats on current stable
distributions. The pin is exact rather than a range for the same reason
pdfminer.six's is: extraction is deterministic within a converter version and
not across one.
The single-runtime-dependency rule is untouched -- it governs
`project.dependencies`, which still names the guard alone.
Measured after installing, on this host:
bundled binary pandoc 3.9 (inside the wheel, as intended)
pypandoc picks 3.10.2 (the host's PATH binary)
That is the third independent measurement of the trap: pypandoc searches PATH
before its own bundled binary and takes the highest version it finds, so
"vendored" buys nothing until something resolves the path explicitly. The
resolver is the next step; until it lands, the vendoring is a pin without an
effect and should not be described as more than that.
Suite 886 -> 887 (the extra is installed, so the packaging pin runs).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Registered before any step raises them, so no later step invents a spelling:
- `extractor_binary_missing` -- the converter binary is absent (distinct
from the extra not being installed)
- `extractor_binary_version` -- present, but not the pinned version
- `extractor_convert_error` -- the converter failed on this file
- `extractor_empty_conversion` -- the converter returned no text
Also widened the `extractor_extra_missing` type list, which still read
"pdf/docx/xlsx".
Denominators recounted after the change rather than carried from a note --
the stale 49/48 figure is what made the recount a step requirement:
code bullets in errors.py 50 -> 54
distinct codes 49 -> 53 (one code documented twice)
test definitions in this suite 56 -> 57 (one parametrized definition,
four cases -- one per code)
The four tests assert only what is true at this step: the code is documented
and an ExtractionError carries it. The resolver and the seam replace each with
a behavioural raise-site test; a code that never gains one stays visible here
as a test that still only reads a docstring.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both tests reached `extractor_extra_missing` through a `.docx`/`.xlsx`
filename, which works only while `_UNPARSED_OPTIONAL_EXTENSIONS` is non-empty.
Those types are about to gain a converter, which empties the set and makes the
membership branch unreachable -- the tests would have gone red for the right
reason at the worst moment, mid-series.
Repointed both at the import probe, the mechanism the pdf gate already uses and
the one path that stays reachable however many types gain parsers.
Measured negative control: without the probe the same call raises
`extractor_pdf_error`, so the probe is load-bearing and the test can still fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`d2a8c43` states in a code comment that Door C's ordering holds within a
run and never re-orders entries an earlier run wrote. That was true and
untested: every test in the new file ran each door exactly once, so the
sentence was prose rather than a pin.
Two imports into one bundle, the second adding the concept whose key
sorts FIRST. It lands last, because `link_in_index` appends what is
absent and leaves what is present. The same three concepts merged in one
run do come out ordered, asserted alongside, so the two assertions cannot
both be trivially true -- the difference is the append bound, not an
ordering that failed.
883 tests.
Co-Authored-By: Claude <claude-opus-5>
An index ordering a profile names must be honoured wherever this library
writes an index. Door B and Door C have separate index writers, so an
ordering built on Door B's `_index_sort_key` seam alone would have been a
profile field Door B obeys and Door C ignores -- silently, because
nothing raises and both files still parse. That is
`IndexPolicy.per_directory` again: a field that reads as global and acts
on one path.
`IndexPolicy` gains `sort_key`, `sort_order` and `sort_missing`. Both
order fields draw from CLOSED sets, and `sort_order` is deliberately not
a caller-supplied callable: a callable cannot be serialised into the
bundle, reproduced from it, or audited by a reader, which is the whole of
what a deterministic bundle claims. A `sort_key` the facet policy does
not name is refused too -- every entry would be missing the key and the
ordering would silently do nothing, which is this row's own defect class.
`IndexPolicy.sort_entries` is the one helper. Four stable passes, so each
is the tie-break of the next: concept path, then the named key, then the
missing group partitioned to whichever end the policy says, then
navigation last. Passes 2 and 3 are separate on purpose -- folding them
into one reversible key tuple would flip the missing group along with the
order, so `sort_missing="last"` would mean "first" under `descending`.
The tie-break is the CONCEPT PATH, not the link target, and that is
measured rather than assumed: `notes-beta.md` precedes `notes/alpha.md`
by concept path and follows it by generated filename, so ordering Door C
on the target would have re-ordered every existing Door C bundle.
`IndexEntry` carries the path for that reason; `parse_entry` leaves it
`None` and the ordering falls back to the target, which costs nothing
because no caller sorts entries it read back off disk.
Door B's two reprojection writers and Door C's index emission all route
through the helper. Door B's unfaceted path is not routed and does not
need to be: `sort_key` requires a facet policy, and a faceted profile
never reaches that writer. Door C's guarantee is bounded and stated in
the code -- `link_in_index` appends what is absent and leaves what is
present, so the order holds within a run and never re-orders entries an
earlier run wrote.
Default ordering, unchanged and now stated: with no `sort_key`, concepts
before navigation, each group ascending by concept path.
TDD, and the red was watched twice. First behaviourally with the fields
inert (both doors emitted the exact reverse of the named order), then
again with Door B routed and Door C not -- the broken world reproduced,
where a Door-B-only test would have passed.
882 tests (868 before). The five byte-pinned goldens are untouched and
green; no shipped profile moved.
Co-Authored-By: Claude <claude-opus-5>
The registry file says one test per code and that this file IS the
conformance suite. During the 1-to-N voyage the byte-stability pin over
tests/ was the verification mechanism, so editing this file was
forbidden and six new codes were covered in the new segmentation
modules instead. That was right then. The end-to-end gate run released
the pin, so the convention applies again -- a conformance suite split
across two files stops being one quietly, which is why this was carried
as an obligation rather than a preference.
All seven segmentation codes now have a test here, including
segmentation_plan_unmatched from this session. The behavioural tests in
test_segmentation.py and test_segmented_inbox.py stay where they are:
they exercise the parser and the door, a different question from
whether every documented code has a raise site.
Measured after: 49 documented codes, 48 covered in this file. The one
gap, source_reference_unquotable, predates this work and is tested in
tests/test_okf_v0_2_profile.py:351 -- reported, not silently closed.
Also records both defects and this migration as closed in the decision
record's known-gaps list.
Co-Authored-By: Claude <claude-opus-5>
A segment's title comes from the plan, so a human adjudicated it. But
structure derivation runs over the segment body, finds no title key and
no usable heading, falls back to a stem, and adds "title" to derived.
The concept then emitted a stated fact under an inferred marker, and a
consumer that distrusts derived fields would distrust exactly the thing
a human decided. An over-marked field is the same defect class as an
unmarked heuristic: the marker is only worth something if it is
accurate in both directions.
Scoped to title alone, and pinned that way by test: number stays in
derived on a segment, because nothing about segmentation makes an
inferred document number declared. Without a segment a derived title is
still marked, so only a plan makes a title declared.
The SEGMENTED_V1 golden moves, which is the intended consequence and
the only golden that may. The four existing goldens are byte-identical
to baseline 770d8d4, measured against the sha rather than inspected.
Co-Authored-By: Claude <claude-opus-5>
A plan is selected by content hash, so a mistyped source_sha256 matched
nothing, every dropped file fell through to the one-concept rule, and
process_inbox returned an ordinary success over a flat bundle. The
operator asked for segmentation, got none, and had no error to read --
the silent skip this library refuses everywhere else. vegnormal-okf is
about to run an N500 corpus through this path, where a silent zero
would read as "the corpus has no concepts".
The refusal asks whether a covering plan was FOUND, not whether every
file was examined, so an unreadable drop cannot mask it; and coverage
is recorded at selection, not after path validation, so a matched plan
with a refused entry path still reports its own per-file code. The
first cut got that second question wrong and an existing collision test
caught it; the case is now pinned by its own test, verified red against
the earlier form.
New code segmentation_plan_unmatched, registered in the SegmentationError
docstring register in the same commit. Fail-fast before any disk
mutation. The four existing goldens are byte-identical to baseline.
Co-Authored-By: Claude <claude-opus-5>
The line-oriented frontmatter grammar exists in three copies, each with the
duplication documented at its site: `materialize` reads a path, `structure`
needs a character offset, `profiles` returns body lines. All three keyed on
`key.strip()`, which discards the indentation that is the only thing telling
a nested key from a top-level one. An indented `title:` under a `sources:`
block therefore landed in the same flat namespace as the document's own
`title:` and, arriving later, won.
The failure is substitution, not omission. A dropped value is visible to
whoever reads the concept; a substituted one is not -- the document carries a
title that looks entirely right and belongs to something else. Because
`number` derives from `title` and `parent` derives from `number`, one
substitution walks the hierarchy. Measured, not inferred: a document titled
`N100.2` with a nested source titled `N200.7` came back as N200.7 with parent
N200 instead of N100.2 with parent N100.
Measured incidence across the two corpora, denominators stated:
`_okf-canonical` @ ad30107, 54 documents with parsable frontmatter, 49 carry
a nested key colliding with a top-level name (90.7%); `_okf-upstream` @
9a15b13, 66 documents, 58 collide (87.9%). The colliding key is `title`, and
often `resource` with it -- in `acme_retail/tables/orders.md` the concept's
own BigQuery resource pointer was replaced by a nested one. This is a fix
that clears observed damage, not a hardening without a witness.
The fix refuses indented lines; it does not read them. Block form stays
unreadable -- `sources` and `verified` still come back empty -- so D4's
flow-form emission rule is untouched and the structured reader is still D1b.
Two characterization tests that pinned the old behaviour now pin the new: the
block-list family still DROPS its value, and only the key-space pollution is
gone. That family is not otherwise addressed here.
Test first, red before the code was touched, with known-positive controls for
all three parsers so that a parser returning nothing could not pass.
Order: 20260830T000740Z-4733930312-from-.claude
Co-Authored-By: Claude <claude-opus-5>
STRUCTURED_V1 derived `references` from every number-shaped token in a body.
A consumer measured 12 false references out of 12 on their corpus, ten of them
version numbers in titles read as document numbers. Measured again here before
changing anything, on two corpora, through `derive_document_structure` itself:
corpus A a consumer's normative bundles 2 561 docs -> 2 838 subjects
corpus B this repository's own docs/ 28 docs -> 559 subjects
The reported class reproduces, and two larger ones they did not report turn up:
702 of corpus A's subjects (24.7 %) are hexadecimal fragments of UUIDs read out
of `id="..."` attributes in embedded MathML, and corpus B lifts tokens out of
escape sequences in quoted source (`\n60` -> `N60`), licence identifiers
(`AGPL-3.0` -> `3.0`) and package pins.
A derived subject is now a bundle-local link target, or a number immediately
preceded by a cue from a closed set. Positive rather than a blacklist because
the data forces it: `V221` is a genuine document number in corpus A and
`V0.3.0` is a software version in corpus B, and they are the same token shape,
so only the words in front of them can tell them apart. The cue matches at a
word boundary (a Norwegian compound ending in `-klasse` otherwise satisfies the
cue `se`, which admitted 86 class designations) and the window is NFC-normalised
so a cue survives a decomposed filesystem. Fragment-only and brace-carrying link
targets go too: neither can name a concept, so neither is a pending pointer.
After: 2 838 -> 1 279 (A) and 559 -> 72 (B). Hand-classified against the
occurrence that actually passed the gate: 30 of 30 sampled genuine on A, 60 of
60 on B. Residual known falses: 9 of B's 72, all illustrative link targets in
prose about link syntax.
The prefix-resolution rule stays open, per the order's condition: of 2 589
documents, 2 562 carry a number and 0 of those are dotted, so unique-prefix
match has no data here to be defended against.
Cost stated rather than hidden: a corpus phrasing cross-references outside this
vocabulary derives nothing and must declare `references`. A missing reference is
visible to the reader; a false one is not.
Record: docs/plan/references-cue-rule.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ATyA5Lx53N1rKzwMnuMoem
Four decisions: Door C projects and never derives (D-C1), the projection is
key-agnostic so it is not about numbering (D-C2), an unrenderable value drops
the facet and never the concept (D-C3), and a faceted entry is refreshed
because it is the first entry that can go stale (D-C4).
Also records a measurement about the measuring. The mutation harness first
reported 11 of 11 killed; the number was entirely false, because it invoked
pytest with an unrecognised `--timeout` flag and every run exited non-zero. The
negative control caught it by being reported killed when it provably could not
be. Honest result is 9 of 11, with both survivors analysed as equivalent
mutants and the one real gap they exposed (profile threading pinned only by
coincidence) closed with a direct test.
Closes open item 6 of the Door B record, which asked exactly the question this
answers: whose claim does the frontmatter make.
vegnormal-okf measured the gap on 2026-08-27: the arm reading DEFAULT's
index.md scored 0 hits of 8, the arm reading a faceted index of the same
frontmatter scored 25 of 29. Same bundle, same concepts, same model. The
DEFAULT arm did not answer wrongly, it abstained -- the metadata is in the
bundle and the index throws it away (30 974 characters over 269
requirements, 0 occurrences of any of the eight facts).
FacetPolicy and STRUCTURED_V1 already did this. They did not reach Door C.
`import_bundle` now takes a keyword-only `profile` defaulting to DEFAULT, so
every existing call site emits the bytes it always did, and `link_in_index`
takes the facets to render.
Door C PROJECTS and never DERIVES, which is the answer to the objection this
work opened with: deriving structure for a document a third party wrote would
put our inference into an index entry ABOUT their bytes, where it reads as
their claim. The concept file was already verbatim; the entry describing it
now is too. Where the sender carries `derived`, THEIR list travels unchanged,
so a reader can still see which of the sender's facts the sender inferred.
The projection asks the policy which keys to carry and never what a key
means. That is what makes it work for a meeting note as well as a numbered
norm -- nothing in it can key off a numbering scheme -- and it is why a
consumer whose concepts are named by UUID can get `title` into the index by
naming the key, with no change here.
Two things measured during the work rather than assumed:
- A value carrying the policy's own joiner cannot be rendered. Door C's
tolerance is structural and it refuses no sender on form, so the FACET is
dropped and the concept still merges -- reported per concept and key in
`ImportResult.unrendered_facets`, never dropped silently.
- A faceted entry can go stale where a flat one never could: the collision
gate refuses an updated concept, so the operator's only route is to remove
the merged file and re-import, after which the file said `gjeldende` while
the index still said `utkast`. A faceted entry for a present target is now
refreshed in place instead of skipped. Unfaceted callers keep the early
return byte for byte.
Suite 695 -> 707; ruff and mypy --strict clean.
Order 20260826T224500Z-873805419-from-vegnormal-okf.
Door B takes a profile (keyword-only, DEFAULT) and, under a profile carrying
facets, derives each dropped document's title, number, hierarchy and
cross-references, writes them into the concept's own frontmatter, and projects
them into the index entry.
The additive requirement is answered by one decision rather than by an
algorithm: the index is a PROJECTION of the concept files, recomputed from the
whole bundle each round. Nothing is diffed, so the three invariants hold by
construction -- rebuild-from-scratch equals incremental byte for byte,
re-dropping a document replaces its entry instead of doubling it, and a
relation formed in round 3 UPDATES the round-1 entry it is about, which an
append-only index could never do.
An unresolved pointer is marked '?' in the entry rather than omitted: during
build-up, pointing at something not dropped yet is normal, and the dangerous
version is the one that leaves no trace. Facet values are validated per file
BEFORE the write, so a producer value that breaks the grammar fails that file
and not the run.
DEFAULT is byte-identical with and without the new parameter, and is asserted
so. Door B keeps writing the literal 'generated: true' rather than the
profile's ownership stamp -- routing it through the profile would move
DEFAULT's bytes and orphan every bundle this door has already written; that is
a separate question and answering it here would have answered it silently.
18 new tests; suite 677 -> 695.
The measured defect, as data: the 2026-08-26 bake-off had every arm retrieve
40/40, so quality could not separate them. The only axis that did was trap
exposure -- 18/20 for the OKF-index arm against 8/20 for a frontmatter
head-scan -- and both sides measured the reason independently: the flat index
carries title/date/status/supersedes 0 times while its own documents carry them
55/55/55/5. The metadata is in the bundle; the index throws it away.
FacetPolicy lets an index entry keep it. The grammar is thin on purpose (one
separator, then key: value joined by '; ') because index lines are read by
regex on both sides of this library, and a value carrying either delimiter is
REFUSED rather than escaped -- validation, not repair, as everywhere else here.
Additive by construction, not by caution. entry_pattern IS link_pattern when a
policy carries no facets, so DEFAULT and STRICT_V1 match the same lines and
emit the same bytes; the goldens are the proof. Facets arrive as STRUCTURED_V1,
a new profile, because DEFAULT states commons' ingest-spec index layer and
changing its bytes from here would be this repo editing a contract it does not
own.
17 new tests; suite 660 -> 677.
A bundle a consumer can only look things up in is a filing cabinet. This adds
the derivation half of what lets one REASON over it: per-document title,
number, parent and references, plus bundle-level resolution of every pointer.
Two rules carry the design. Every fact is marked DECLARED or DERIVED, because
an unmarked heuristic is worse than no heuristic -- a consumer cannot know when
to doubt it. And resolution is a PURE function of the whole document set rather
than a diff, which is what makes rebuild-from-scratch equal an incremental
update by construction: there is no diffing algorithm to prove correct.
An unresolved pointer is kept and reported, never dropped: while a bundle is
built up over several rounds, pointing at something not dropped yet is the
normal state, and the dangerous version of it is the one that leaves no trace.
Symmetrically, a parent our own grammar could never admit (4.2 -> 4, a bare
integer) is not emitted at all -- an unresolved list that never clears is one a
consumer learns to ignore.
45 new tests; suite 615 -> 660.
The operator's condition for revisiting the pin was met (v1.2.0 contains
the flow-mapping frontmatter fix, commit 5870483) and the operator has now
approved the move itself, dispatched as its own order because it changes
consumer-visible Requires-Dist and shifts golden-fixture admission.
Floor 1.2: this library needs the flow-mapping support (`generated: { by:
x, at: y }`) that landed there — without it Door C fail-secures every
concept carrying that stamp. Ceiling <2.0, not narrower: the guard's own
1.0.0 release freezes its exported surface until a 2.0.0, and explicitly
keeps calibration (severities, dispositions) free to move within 1.x, so a
tighter ceiling here would claim a stability guarantee neither side needs.
Re-measured through Door C against the guard's own default
(allow_reserved=True, matching how the earlier recommendation measured
it), over the 9 concept documents across all four Door-A goldens:
4/9 admitted at 0.3.4 -> 8/9 admitted at 1.2.0, confirming the number
already reported. The ninth (a `sources` block-list carrying `resource`)
stays refused by design (G30) and is not expected to move.
Failing-test-first: test_guard_adapter.py::test_guard_version_is_inside_the_pin
and test_packaging.py::test_the_only_runtime_dependency_is_the_security_boundary
were updated to the new pin first and confirmed red against the
still-installed 0.3.4, then `uv sync --extra extract` installed 1.2.0 and
both went green. Full suite (615), ruff and mypy --strict clean.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RxcxzNwpX1kDP53n1rLhM5
The operator asked for a job that checks at least weekly whether Google OKF has
moved, and messages the right repo immediately when it has. It belongs here
rather than in `.claude` because knowing what a meaningful spec change IS
requires owning the pin, the runbook and the always-latest policy.
`tools/okf_watch.py`, stdlib only, driving git against the local read-only
mirror. It lives outside `src/` so it never enters a wheel; a new packaging test
holds that as a promise rather than an accident of the build config.
Three properties carry the design, and each closes a failure this repo has
actually met:
1. A failed call is never an empty result. Every git invocation raises on a
non-zero exit and carries stderr, so a caller reading "" knows the query ran.
The precedent is `grep ... | head; echo $?` reporting head's exit status - a
broken query read as a quiet upstream.
2. It proves it can find, every run. Before believing any zero it re-runs the
full detect-and-classify path over `ad30107^1..ad30107`, a range known to have
changed SPEC.md. An empty known-positive aborts loudly rather than reporting a
clean sweep. Network failure likewise raises; it never degrades to "no change".
3. It reports on change, not on state. A pin-keyed state file records what has
been announced; moving the pin resets it, because a pin move means everything
behind it was absorbed.
Quiet is the enumerated list, not signal. Enumerating what counts as normative
can only match what upstream has already invented, so anything new would fall
outside it and the watch would go silent - failing in the direction nobody
notices. A small measured quiet list, everything else reports. README.md is
deliberately not quiet: the repository move was announced in a README commit.
Sixteen tests build their own git repository in tmp_path rather than skipping
when the mirror is absent - a skipped test preserves nothing on the machine
where the dependency exists. All four load-bearing behaviours were mutation-
tested red before this landed.
Two more tests exist because building this fired a real false alarm: running
with `--pin` and without `--dry-run` delivered two live coord messages. The
override now implies dry-run, enforced in argument parsing rather than
remembered, and `.claude` has the correction.
The runbook gains a section stating what the watch CANNOT do, because that is
the part a future session will otherwise assume away: it sees commits, not
meaning. It would have fired on the 2026-08 tightening because SPEC.md changed,
but no commit list says a value that conformed last month no longer does, and
none says is_stale reversed. Its output is "run the runbook", never "here is
your exposure".
Order G2a. Populates the optional `[extract]` extra for the first time with
one parser, `pdfplumber>=0.11.10,<0.12` (MIT), and wires `pdf` through it.
The default install is untouched: exactly one runtime dependency, stdlib
otherwise, enforced by test_packaging.py.
The gate for `pdf` becomes an import probe rather than a frozenset membership
test, exactly as extract.py's docstring had promised. The rejection does not
change: without the extra, `pdf` still raises `extractor_extra_missing` with
the same message. That behaviour is asserted UNCONDITIONALLY via a sys.modules
monkeypatch, so it holds on machines where the parser is installed too — a
skip would have preserved nothing there. Verified in a clean venv without the
extra: 589 passed, 7 skipped; with it, 596 passed.
`docx`/`xlsx` are unchanged and still fail fast — the extra names exactly what
it ships.
The parser choice was forced by measurement, not preference (b73dd9d,
docs/2026-08-21-g2-pdf-extraction-measurement.md): on a real requirement table
pdfplumber keeps 4 of 4 rows with label and value on one line, where pypdf,
pdfminer.six and pymupdf each keep 0 of 4. pymupdf is additionally out on
licence (AGPL-3.0), which an MIT package must not push onto a consumer.
Three facts from that measurement are now carried in code rather than in a
report:
- Extracted text is pinned to an exact transitive parser version
(pdfplumber pins pdfminer.six==20260107; date-stamped, no stability
contract). tests/test_extract.py freezes the expected text of a committed
hand-written fixture so a parser upgrade breaks something visible instead of
drifting silently. Reasoning at the declaration site and in
tests/fixtures/README.md.
- Determinism within a version is now held by a test, not only measured once.
- Drawn content does not survive extraction. Every pdf extraction emits the
new `ExtractionWarning`: figures have no text to recover, so a bundle built
from drawn documents is incomplete by construction. Stated categorically
rather than detected — deciding "is there a figure here" is the layout
heuristic G2b declined.
Two new error codes, both mirroring existing patterns: `extractor_empty_pdf`
(a scanned/image-only PDF, refused rather than persisted as an empty concept)
and `extractor_pdf_error` (parser failure wrapped, never leaked).
Structured table recovery (G2b) is NOT implemented and is documented as out of
scope: two independent parsers return the same wrong shape, so the breakage is
document geometry, not a library choice. PDFs enter as prose.
Also corrects an install promise this change would otherwise have published:
the README no longer presents a bare `pip install 'llm-ingestion-okf[extract]'`
as working, because the package is not on an index.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HtNhsdHnMGtMi7U2mvMU8z
The V1 commit told nine repos, in README, CHANGELOG, the plan and two
coord messages, that upgrading across it costs a consumer a re-run and
nothing more. Nothing exercised that end to end.
What was covered: `owns()` returning True for the legacy literal at the
unit level. What was not: the consumer's actual path -- materializing
into a directory that ALREADY holds pre-V1 `DEFAULT` output. The golden
suite cannot see it, because it materializes into a fresh directory
every time.
`test_second_v0_2_run_into_the_same_directory_succeeds` is the same
shape for `OKF_V0_2`, and its comment already said why one byte-compared
run cannot catch this. V1 made `DEFAULT` change its stamp exactly as D2
made `OKF_V0_2` change its own; no equivalent test followed it until now.
The legacy bundle is derived from a real run rather than hand-authored,
so every byte except the stamp is what an earlier version actually
wrote, and two assertions guard the derivation against silently becoming
a no-op the next time the stamp moves.
Hand-mutated to confirm it can fail: making `owns()` return the legacy
literal only for an actor-less profile turns it red with
`collision_unstamped` -- which is precisely the consumer-visible failure
the promise rules out.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VwcjUXbKySLbEG5WqTNkta
Commons ratified V1 2026-08-02 and executed it at `54e0ec7`; verified
against their tree rather than taken on report. ingest-spec.md:217 now
defines `generated` as `{ by: process:okf-ingest, at: <ingested_at> }`,
unquoted, `at` repeating `ingested_at` verbatim. `generated: true` no
longer appears in the spec.
`DEFAULT` states commons' §5 layer, so its stamp is theirs to decide.
`DEFAULT.ownership` gains the actor; the four goldens this repo's plan
named in advance were regenerated by RUNNING the materializer, each on
its own case's `ingested-at.txt`. The v0.2 golden was untouched, as
predicted -- it has carried the O2 form since D5.
Not a migration onto OKF v0.2: `DEFAULT` stays v0.1 on every axis
upstream owns and still emits no `sources`. Commons' spec and the Google
version are independent axes, and comments that narrated them as one
were rewritten rather than left to mislead. README and CLAUDE.md said
the additive rule without that boundary, which would have told a
consumer their DEFAULT bytes can never move; both now state it.
V-A3 is amended, not dropped. `DEFAULT` must OWN the mapping it now
writes -- a profile refusing its own output fires the collision gate on
files its own previous run wrote -- while a mapping naming a foreign
actor, or §7's `human:` actor on curated content, stays unowned. That
half is what carried the safety and it is asserted directly.
§11's stamp-integrity condition moved with the value: the forgeable
stamp was `true` and is now the mapping naming the ingest actor. The
defence was never the value -- the §3 scan globs `ingest-*.md`, so a
Door C import is unreachable however well it forges. Second spoof test
added; both were hand-mutated (glob widened to `*.md`) to confirm they
can fail.
The characterization test derived its foreign-stamp fixture from the
literal `generated: true`, which V1 leaves without a referent -- a
silent no-op waiting to happen. It now derives the needle from the
profile and asserts the substitution occurred.
Door B is deliberately untouched: not the ingest-spec's, marker is
`generated` + `source_file`, disjoint from Door A's `ingest_manifest`,
and the divergence predates V1.
Nothing released or notified. The pilot set pins `v0.5.0a2`, not `main`,
so this is invisible to portfolio-optimiser's freeze and demo; the
consumer exposure report is owed at the release that carries this.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VwcjUXbKySLbEG5WqTNkta
Measure first, widen after. The 19-fixture guard-surface suite was re-run
against v0.3.4 in a scratch venv before the range moved, and reproduced the
three deltas measured against v0.3.3 exactly, with none added. v0.3.4 is the
tag pinned rather than v0.3.3 because it shipped first and repairs a quadratic
regex (okf._MD_LINK_RE) that sits on Door C's own call path.
Door C now passes allow_reserved=False explicitly. The guard added the keyword
in the 0.3 line and defaults it True for received bundles, which would merge a
sender's index.md / log.md instead of rejecting them. The override keeps the
unconditional reserved-name refusal committed to before the keyword existed,
and the reason is structural rather than a second opinion on the guard's scan:
Door C generates the merged bundle's index.md from what it merged and writes
every merged concept verbatim, so a sender's index.md would be a second and
irreconcilable claim on one path.
This is not a behaviour change for anyone on the previous pin: under v0.2.0
the keyword did not exist and reserved names were refused by construction.
The floor is >=0.3 and not >=0.2 for a measured reason. allow_reserved is
absent in v0.2.0 and present from v0.3.0 onward, checked across all five tags:
a >=0.2 floor would admit a version that raises TypeError on every Door C
import. That measurement also corrects a recorded premise -- the plan said the
keyword "shipped in v0.3.3", which read the first version we ran the suite
against as the version it was introduced in. The conclusion held; the reason
did not, and the reason is what a future bump would have relied on.
test_door_c_pins_allow_reserved_false_against_the_guards_default locks both
halves: that the guard still defaults True, without which the override is a
no-op that would pass forever over nothing, and that Door C overrides it.
586 tests, mypy --strict clean, goldens byte-identical.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V2v1hrDhrff2H3y2TNJHkF
The §3 ownership scan globbed every *.md file regardless of which door
wrote it, then unconditionally unlinked whatever _is_ingest_owned agreed
to. Because _is_ingest_owned reads through the line-oriented parser that
flattens nested blocks (pinned in
test_two_nested_block_mappings_sharing_a_key_collide_in_the_scalar_parser),
a Door B/C file whose nested content happened to share a key name with
the ownership markers (generated, ingest_manifest) could get promoted to
top level and spoof ownership -- silently deleting content this door
never wrote.
Scoping the glob to ingest_prefix closes this by construction: a Door
B/C file is never even a candidate for the scan, regardless of what its
frontmatter parses to. Traced from a coordination tip from
portfolio-optimiser-claude about the same flattening mechanism hitting
their `type` field.
V6, settled by the operator today: import and report, not refuse.
Door C imports the POINTER to executable code and never the code — it
writes concepts verbatim and skips every non-`.md` file. So an imported
Attested Computation can name an `executor`/`attester` resource that did
not arrive, or one that RESOLVES against a file the destination tree
already holds under that path. The second is the outcome worth
surfacing: it looks valid.
Refusing was the plan's leaning and is not what shipped. §14 forbids a
consumer to reject a bundle over a broken cross-link and does not settle
whether `executor.resource` is one; §10.5 asks a consumer to surface
rather than silently drop. Reporting honours the second without testing
the first, and leaves the door's one invariant — verbatim bytes — alone.
`ImportResult.unverified_references` is an advisory over the merged set,
not a fifth bucket: every concept it names has already merged, the
bytes are unchanged, and a refused concept is never named (there is no
imported pointer to check).
The report is at KEY level, and that is a measured limit rather than a
choice. Resolving the resource means reading `executor.resource` — the
value the line-oriented parser cannot recover in either canonical form:
a block mapping flattens and collides, a flow mapping stays one opaque
string. A resource-level report would be empty or wrong on exactly the
forms upstream writes. Precision arrives with the structured reader
(D1b); the key-level signal is robust in both forms today.
[skip-docs] is on the CLAUDE.md half only: README carries the new
public surface (`unverified_references`), and the invariant this work
put in CLAUDE.md — flow form, never block — landed with the previous
commit and needs no restatement.
584 tests, mypy --strict clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KKKMwi7e7PVHoFW6dJK5XP
Upstream §10 adds a concept type carrying a sanctioned way to compute a
value. This lands its FORMAT: the five contract fields for emission and
round-trip, and §10.2's one requirement. No execution — upstream defers
the receipt and verdict wire formats, so there is nothing to build a
runtime against.
Two additions, both additive:
- The five fields (`runtime`, `parameters`, `computation`, `executor`,
`attester`) join `OKF_V0_2`'s emission order as one block, internally
in §10.2's own listing order. Without it they still emit — in `emit`'s
sorted tail, where `attester` precedes `runtime`, alphabetical order
standing in for the contract's own. No bundle that carries none of the
keys changes by a byte, and the v0.1 profiles gain nothing.
- `FrontmatterSchema.required_by_type` expresses "`runtime` is REQUIRED
for this type and no other" — the first rule here keyed off a
frontmatter VALUE rather than a key. It cannot be `required`, which
would demand `runtime` of every document. A type the mapping does not
name carries no extra requirement, which is what keeps it inside §14:
a consumer must not reject on an unknown `type`, so a conditional
keyed on an unknown type stays silent rather than guesses.
Also pinned, measured today: the line-oriented parser cannot read §10's
canonical BLOCK form. `executor` and `attester` both carry a `resource`,
and with no indentation model the second overwrites the first —
`executor.resource` is lost silently, no error. Characterized rather
than fixed: reading that form needs the structured reader (D1b), and a
half-reader that drops half a contract is worse than one that never
claimed to read it. CLAUDE.md gains the invariant that falls out of it:
we emit flow form, never block, or we write bundles we cannot read back.
578 tests, mypy --strict clean, goldens byte-identical.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KKKMwi7e7PVHoFW6dJK5XP
v0.5.0a1 shipped the one actor value the spec owner had already excluded.
Caught before any pilot was notified, so it costs a tag, not a migration.
Commons decided <fast id> = process:okf-ingest on 2026-07-31, on this repo's
own proposal, superseding option (d) (process:llm-ingestion-okf) chosen here on
07-27. The exclusion is ingest-spec.md:7-8, frozen on the spec being
framework-neutral: normalising OUR repo name into the normative id would force
every other conformant implementation to write it into its own output. Verified
against three independent sources before touching anything — commons' coord
message 20260731T154140Z, their plan :215-216/:244, and their STATE :37.
Why this had to land before the pilot notifications rather than after:
actor is both the stamp written and the value owned back (OwnershipPolicy),
and recognition is one-way. A pilot that had run Test A against the excluded
id would hold bundles this library stops recognising the moment the id is
corrected — collision_unstamped on their OWN files. That is the A-E5 failure
mode, and we would have inflicted it.
Worse, it would not have shown up as a failure: the plan's A-E3 expectation
(:854) named the same excluded value as the code, so Test A would have PASSED
and confirmed the error. Expectation and implementation agreeing is not
evidence when both predate the decision.
Nothing in the wild carried the old value: OKF_V0_2 did not exist at v0.4.0, so
the profile has never been released. v0.5.0a1 is abandoned, not moved — a tag
already on a public remote does not get force-pushed, and the history should
say plainly that a1 was wrong.
A-E3 now records both corrections with dates. The V1 paragraph at :1169 is
superseded in place rather than rewritten: its reasoning still holds, only its
outcome moved.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVTup4v7tW9QiVyBENk2LV
The tag was specified to land on 128109b, where the package still declared
0.4.0 in both places a version is written. The pilots install FROM the tag, so
they would have pinned a pre-release that reports the previous release.
Two requirements the "minimally requires" list did not cover, both stated
elsewhere in the plan:
- The CHANGELOG entry is load-bearing, not ceremony: the plan makes the
provisional status real by having the profile docstring AND the entry both
say the surface may change without a deprecation cycle, and name the pilot
repos. The docstring had it (profiles.py:694); the CHANGELOG stopped at
0.4.0.
- pyproject.toml and __init__.py both said 0.4.0.
Nothing in the run path reads __version__, which is why a stale one survives a
green suite. test_the_declared_version_agrees_with_the_packaged_one is the only
machine check on it — written failing first, against a one-sided bump. The tag
NAME remains a human step; no test can catch that one.
README keeps pointing at v0.4.0 on purpose: a pre-release for three named
pilots is not the general install line, and OKF_LATEST does not move until GA.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVTup4v7tW9QiVyBENk2LV
`IndexPolicy.root_frontmatter` carried two claims at once: which keys the
root index may carry and in what order, and which it must carry. The judge
read it as the second, so `OKF_V0_2` — which names `okf_version` only to fix
its position, upstream granting it as a MAY (§8:509-510, §12:773-775) —
reported every bundle exercising that MAY as violating. Measured in P1 over
17 bundles: 14 failed with exactly this one violation, upstream's own four
reference bundles among them, while D5's emitter treated the same key as
optional. We emitted a MAY correctly and graded it a MUST.
`root_frontmatter` now permits and orders; `root_frontmatter_required`
demands. A required key outside the ordered set fails at construction, the
same contradiction `FrontmatterSchema` already refuses when `required`
strays outside `allowed`.
- `OKF_V0_2` requires none — upstream's MAY, stated on the judge side too.
- `STRICT_V1` requires all three: the proving consumer's root index carries
exactly those keys in that order on every bundle measured (`c5141f8`), so
separating the meanings costs them nothing.
- `DEFAULT` names no root keys and is untouched.
Re-measured over the eight bundle-root indexes reachable locally (our four
goldens, upstream's four): 7 of 8 failing under the old semantics, 0 of 8
after. Emit path byte-identical — the golden suite fails otherwise.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JqCmfJ2ukpFXeFjfab8wvy
